Modern Theory of Gradient-Based Optimization

Abstract

In this review, we offer a comprehensive survey of emerging techniques in gradient-based optimization, with a particular emphasis on the interplay between ordinary differential equation (ODE) perspectives and their extensions into discrete Lyapunov analysis. We begin by examining the acceleration mechanisms underlying Nesterov's accelerated gradient method for strongly convex functions (NAG-SC) and Polyak's heavy-ball method, identifying the gradient-correction term as the primary driver of acceleration. This mechanistic insight is substantiated through high-resolution ODE modeling and the systematic construction of Lyapunov functions. We then synthesize recent advancements in convex optimization regarding NAG and its proximal generalization, the fast iterative shrinkage-thresholding algorithm (FISTA). Key topics include the accelerated convergence of gradient norms, underdamped acceleration, linear convergence under strong convexity, and novel Lyapunov frameworks for establishing convergence and monotonicity properties of generalized accelerated methods. Furthermore, we demonstrate how these ODE approximations and Lyapunov techniques can be extended to provide a unified framework for analyzing advanced optimization algorithms, including the alternating direction method of multipliers (ADMM), the primal-dual hybrid gradient (PDHG) method, and their respective accelerated variants. Finally, we discuss recent progress in minimax optimization and outline future directions for extending Lyapunov-based analysis to saddle-point problems.