I have been in the transformation semigroups game for nearly ten years now, but I still feel that I am finding my feet.
Here is apparently a huge difference between permutation groups and transformation semigroups, one which is still not fully understood. A permutation group may have automorphisms which are not induced by conjugation in the symmetric group. The most spectacular example of this is the symmetric group of degree 6, which has an outer automorphism, as I have discussed here. (This is the only symmetric group, finite or infinite, which has an outer automorphism.) This beautiful configuration is intimately connected with other interesting phenomena; I outlined here how to use this to construct the Mathieu groups M12 (which also has an automorphism not induced by a permutation) and M24, and one can go on to the Conway groups and the Monster. Chapter 6 of my book with Jack van Lint explains also how to use it to construct (and show uniqueness of) the projective plane of order 4 and the Moore graph of valency 7 (the Hoffman–Singleton graph).
Could anything similar happen for transformation semigroups?
It appears not. Forty years ago, R. P. Sullivan proved a very general theorem which implies, in particular, that a transformation semigroup on a set X which contains all the constant maps on X has the property that its automorphisms are all induced by permutations of X, so that its automorphism group is isomorphic to its normaliser in the symmetric group on X. (To show this, observe first that an automorphism of the semigroup must preserve the set of constant maps, which is naturally bijective with X; then we must show that an automorphism which fixes all the constant maps must be the identity.)
Now this is something interesting. The synchronization project, which was my introduction to semigroup theory, is concerned with those permutation groups G with the property that any transformation semigroup containing G and at least one singular transformation necessarily contains all the constant maps. It follows that if a transformation semigroup contains a synchronizing group, then all its automorphisms are induced by permutations. Moreover, we know that the class of synchronizing groups contains the 2-transitive groups (even the 2-homogeneous groups) and is contained in the class of primitive groups; this is a large and interesting class of permutation groups.
So what about transformation semigroups which contain a singular transformation and whose group of units is a non-synchronizing permutation group? For these, João Araújo, Wolfram Bentz and I have made the first small breakthrough. Embarrassingly small, I would say.
Assume that G is a primitive permutation group. We say that G synchronizes the singular map t if the semigroup generated by G and t contains a constant map (and, hence, contains all constants). Now we know, from our paper with Gordon Royle and Artur Schaefer which just appeared in the Proceedings of the London Mathematical Society, that a primitive group of degree n synchronizes any map whose rank (cardinality of the image) is 2 or at least n−4 (and, indeed, we conjecture that the upper value can be improved to n−5, but the labour involved in showing this with our techniques would be prohibitive). So the obvious case to consider is maps of rank 3.
Our theorem says that, if a transformation semigroup contains a primitive group and a map of rank 3, then all its automorphisms are inner.
There are some interesting examples of such groups to be found in the paper. These include the automorphism groups of the Heawood, Tutte–Coxeter, and Biggs–Smith graphs, acting on the edge sets of the graphs, and two different actions of the Mathieu group M12 on 495 points. The last two stand in a curious relation: they are automorphism groups of a pair of graphs where the vertices of one graph correspond to the triangles (images of endomorphisms of minimal rank) of the other, adjacency of vertices corresponding to intersection of triangles.
We go so far as to conjecture that the “rank 3” assumption can be dropped; all we need is a primitive group and a singular map. But we are a long way from the proof of this at the moment. Still, hopefully we have taken the first step, so here is a nice project for the new year.