When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective
Zilin Wang, Binghui Li, Jia-Nan Wang, Lean Wang, Jinbo Wang, Lei Wu
Abstract
We study how the sign operator interacts with gradient noise to shape the learning dynamics and scaling behavior of sign-based optimizers. In power-law kernel regression with diagonal features, we establish a functional scaling law (FSL) for SignSGD under general learning rate schedules. The FSL describes the learning dynamics through two processes: signal learning and noise forgetting. The key mechanism is noise-induced preconditioning: gradient noise smooths the sign operator, inducing an effective preconditioned SGD. Noise-curvature alignment enables this preconditioning to transform the effective Hessian from H to H1/2. The resulting dynamics exhibit faster signal learning but slower noise forgetting than SGD. This tradeoff yields a regime-dependent comparison with SGD: (i) SignSGD achieves improved data efficiency in the hard-task regime; (ii) both optimizers achieve the same data efficiency in the easy-task regime, but SignSGD maintains this efficiency at larger batch sizes, requiring fewer optimization steps for a fixed data budget. Power-law kernel regression experiments support the proposed mechanism and predicted scaling behavior. LLM pre-training experiments further suggest that the SignSGD FSL better captures the batch-size dependence of loss dynamics for sign-based optimizers than the original SGD FSL.
Create a lesson
Related papers
Bayesian optimization on amortized search space
Tristan Fauvel, Federico Pavone, Titouan Guillou et al.
Exact Bayesian layers via characteristic functions
William H. W. Thompson, Guillaume St-Onge, Laurent Hébert-Dufresne et al.
Flow Matching from Non-Gaussian Sources: A Theory of Efficient Sampling
Yuchen Zhou, Lili Zheng, Yuxin Chen et al.
Conformal Selection for Hierarchical Streaming Data
Zhenyuan Sun, Chengyao Yu, Ruixing Ming et al.
A Statistical Framework for Training Data Auditing of Large Language Models
Shengjie Niu, Yeheng Ge, Jian Huang
Hellinger Decision Trees for Density Ratio Estimation: Finite-Sample Guarantees
Ren-Rui Liu, Zheng-Chu Guo