A new algorithmic approach for finding flat minima: critically damped Polyak stepsizes
Francesco Caporali, Jason M. Altschuler, Boris Hanin
Abstract
We present a new adaptive stepsize rule that biases gradient descent toward flatter minima in both theory and practice, and establish the first uniform-in-time guarantees for both sharpness and loss reduction for a broad class of locally quadratic objectives. Our critically damped Polyak (CDP) stepsizes multiply the classical Polyak rule by 4 and a mild damping factor. The factor 4 is critical in that it produces equioscillations around the manifold of minimizers M to leading order. Damping provably drives convergence to M. CDP thus operates at the edge of stability by adapting its stepsizes to local curvature. Once the loss is small, its projected iterates follow gradient descent on λ1(∇2 L) along M to leading order, paralleling the flatness bias of normalized gradient descent (Arora et al., 2022). Unlike normalized gradient descent, CDP step lengths shrink with the distance to M, enabling convergence to an interpolating solution that locally minimizes sharpness at finite learning rate. We establish our theory for a broad class of non-negative losses with locally quadratic Morse-Bott structure near M, capturing regular interpolation regimes in smooth overparameterized neural networks. Empirically, CDP stepsizes produce a flattening effect similar to that of Sharpness-Aware Minimization (SAM, Foret et al., 2021), reducing sharpness even in settings beyond our theory, including nonlinear teacher-student networks and fine-tuning of pre-trained LLMs.
Create a lesson
Related papers
Bio-PM-Guided Synthetic IMU Generation
Deepali Singh, Deepesh Parpani, Aviral Gupta
Active Feature Acquisition by Distilling an Oracle Teacher
Yang Meng, Zixin Ding, Robert L. Grossman et al.
The Window Lever: Longer Observation Windows Outweigh the Observed Effects of Data, Capacity, and Architecture in Customer Behavior Forecasting
Tara Sowrirajan, Yanqiao Wang
Hybrid Retrieval for Government Scheme Search in English and Marathi
Aishwarya Gunjal, Kanchan Khemnar, Pawan Poojary et al.
Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning
Élodie Côté-Gauthier, Doina Precup, Isabeau Prémont-Schwarz
Neural Conditional Gaussian Model: Efficient Data Assimilation with Sequential Uncertainty Quantification for Nonlinear Dynamics
Chuanqi Chen, Nan Chen