Adaptive Acceleration Without Strong Convexity Priors Or Restarts

Abstract

A longstanding challenge in optimization is achieving optimal performance when the strong convexity parameter m is unknown. In this paper, we propose NAG-free, a simple extension of Nesterov's accelerated gradient (NAG) which is the first method capable of estimating m directly, without priors or restarts. Our estimator is inexpensive: it requires no additional function or gradient evaluations, only the storage of one extra iterate and gradient already computed by NAG. We prove that, by estimating the smoothness parameter L via backtracking, NAG-free converges globally at least as fast as gradient descent. We also prove that, given an upper bound on L, NAG-free achieves accelerated convergence locally near the minimum under local smoothness of the Hessian and some mild additional assumptions. Finally, we present experiments with smooth and nonsmooth Hessians on both synthetic and real-world data which demonstrate that NAG-free is competitive with restart methods, and naturally adapts to favorable local curvature conditions.