On the complexity of proximal gradient and proximal gradient-Newton-CG methods for \(\ell_1\)-regularized Optimization

Abstract

In this paper, we propose two second-order methods for solving the \(\ell_1\)-regularized composite optimization problem, which are developed based on two distinct definitions of approximate second-order stationary points. We introduce a hybrid proximal gradient and negative curvature method, as well as an adaptive hybrid proximal gradient-Newton-CG method with negative curvature directions, to find a strong* approximate second-order stationary point and a weak approximate second-order stationary point for \(\ell_1\)-regularized optimization problems, respectively. Comprehensive analyses are provided regarding the iteration complexity, operation complexity (including gradient evaluations and Hessian-vector products), and the local superlinear convergence rates of the first phases of these two methods under specific error bound conditions. We demonstrate that the proximal gradient-Newton-CG method achieves the best-known iteration complexity for attaining the proposed weak approximate second-order stationary point, which is consistent with results for finding an approximate second-order stationary point in unconstrained optimization. Through a toy example, we show that our proposed methods can effectively escape first-order approximate solution. Numerical experiments implemented on the \(\ell_1\)-regularized Student's t-regression problem validate the effectiveness of both methods.