From Manifold Identification to Newton Acceleration on Intersections: Sparse Stiefel Optimization

Abstract

We study Newton acceleration for sparse composite optimization on the Stiefel manifold. The main difficulty is geometric: the active manifold identified by the nonsmooth regularizer may fail to intersect the Stiefel manifold transversely, which obstructs a Riemannian Newton step on the identified manifold. In the transverse case, we prove local identification of the ManPG tangent proximal mapping. For nontransverse cases, we introduce an off-diagonally perturbed Stiefel family that generically restores the identification geometry while yielding an \(O(\|Δ\|_F)\)-KKT guarantee for the original problem. We also derive verifiable support-level conditions for clean intersection, which cover nontransverse sparse patterns and yield the smooth moving local models used by the Newton correction. Based on these results, we propose MIX, a safeguarded ManPG/Newton-CG method on moving identified intersections. In the general clean-intersection setting, we prove global descent and KKT-residual guarantees for MIX. In the transverse or generically perturbed cases, we further show that, under sequence convergence and second-order sufficient conditions, MIX identifies the active manifold in finite time and then converges locally superlinearly. Numerical experiments on compressed modes and sparse PCA show that MIX substantially improves efficiency while preserving solution quality.