Prox-NAG-GS: A Semi-Implicit Proximal Method for Composite Optimization

Abstract

Composite optimization problems, where a smooth loss is combined with a nonsmooth regularizer, are common in machine learning and inverse problems. In this work, we study a proximal extension of NAG-GS, a semi-implicit accelerated method obtained from a Gauss-Seidel discretization of an inertial dynamics. The proposed method, called Prox-NAG-GS, keeps the coupled structure of NAG-GS for the smooth part and replaces the second update by a proximal step. It therefore applies to objectives of the form $F=f+r$, where $f$ is smooth and $r$ is convex and proximable. We derive deterministic convergence guarantees for this method. The analysis has to account for a specific feature of the scheme. Prox-NAG-GS keeps two coupled sequences: an $x$-sequence, on which the gradient of the smooth term is evaluated, and a $v$-sequence, produced by the proximal update. The gradient is evaluated at $x_{k+1}$, whereas the proximal step returns $v_{k+1}$, which creates a mismatch absent from the standard proximal-gradient analysis. Under the sufficient condition that the proximal quadratic parameter is at least as large as the smoothness constant of $f$, we control this mismatch through an augmented Lyapunov function involving both sequences. This gives a linear convergence result in the strongly convex composite case. In the convex case, the same Lyapunov structure yields an $O(1/k)$ rate for the best iterate and for the averaged iterate. We test the method on deterministic Elastic Net and Group Lasso problems, and on stochastic sparse softmax-regression benchmarks. In the deterministic tests, Prox-NAG-GS reaches the same solutions as the baselines with substantially fewer iterations; for Group Lasso this also gives the best wall-clock time. In the stochastic tests, Prox-NAG-GS compares favorably with Prox-SGD in terms of data-fit reduction and gives similar test accuracies.