Subgroups of simple primitive permutation groups defined by unordered relations (Automorphism groups of hypergraphs)

Abstract

The problem of describing the invariance groups of unordered relations, called briefly \emph{relation groups}, goes back to classical work by H. Wielandt. In general, the problem turned out to be hard, and so far it has been settled only for a few special classes of permutation groups. The problem have been solved, in particular, for the class of primitive permutation groups, using the classification of finite simple groups and other deep results of permutation group theory. In this paper we show that, if $G$ is a finite simple primitive permutation group other then the alternating group $A_n$, then each subgroup of $G$, with four exceptions, is a relation group.