Sequential subspace methods on Stiefel manifold optimization

Abstract

We investigate the minimization of a quadratic function over Stiefel manifolds (the set of all orthogonal $r$- frames in $\mathbf{R}^n$), which has applications in high-dimensional semi-supervised classification tasks. To reduce the computational complexity, we employ sequential subspace methods(SSM) to transform the high-dimensional problem to a series of low-dimensional ones. In this paper, our goal is to achieve an optimal solution of high quality, referred to as a ''qualified critical point". Qualified critical points are defined as those where the associated multiplier matrix meets specific upper-bound conditions. These points exhibit near-global optimality in quadratic optimization problems. In the context of a general quadratic, SSM generates a sequence of qualified critical points through low-dimensional surrogate regularized models. The convergence to a qualified critical point is guaranteed, when each SSM subspace is constructed from the following vectors: (i) a set of orthogonal unit vectors associated with the current iterate, (ii) a set of vectors representing the gradient of the objective, and (iii) a set of eigenvectors links to the smallest $r$ eigenvalues of the system matrix. Furthermore, incorporating Newton direction vectors into the subspaces can significantly accelerate the convergence of SSM.