The structured Gerstenhaber problem (I)

Abstract

Let $b$ be a symmetric or alternating bilinear form on a finite-dimensional vector space $V$. When the characteristic of the underlying field is not $2$, we determine the greatest dimension for a linear subspace of nilpotent $b$-symmetric or $b$-alternating endomorphisms of $V$, expressing it as a function of the dimension, the rank, and the Witt index of $b$. Similar results are obtained for subspaces of nilpotent $b$-Hermitian endomorphisms when $b$ is a Hermitian form with respect to a non-identity involution. In three situations ($b$-symmetric endomorphisms when $b$ is symmetric, $b$-alternating endomorphisms when $b$ is alternating, and $b$-Hermitian endomorphisms when $b$ is Hermitian and the underlying field has more than $2$ elements), we also characterize the linear subspaces with the maximal dimension. Our results are wide generalizations of results of Meshulam and Radwan, who tackled the case of a non-degenerate symmetric bilinear form over the field of complex numbers, and recent results of Bukov\v{s}ek and Omladi\v{c}, in which the spaces with maximal dimension were determined when the underlying field is the one of complex numbers, the bilinear form $b$ is symmetric and non-degenerate, and one considers $b$-symmetric endomorphisms.