Polyak Stepsize: Estimating Optimal Functional Values Without Parameters or Prior Knowledge

Abstract

The Polyak stepsize for Gradient Descent is known for its fast convergence but requires prior knowledge of the optimal functional value, which is often unavailable in practice. In this paper, we propose a parameter-free approach that estimates this unknown value during the algorithm's execution, enabling a parameter-free stepsize schedule. Our method maintains two sequences of iterates: one with a higher functional value is updated using the Polyak stepsize, and the other one with a lower functional value is used as an estimate of the optimal functional value. We provide a theoretical analysis of the approach and validate its performance through numerical experiments. The results demonstrate that our method achieves competitive performance without relying on prior function-dependent information.