On Convergence and Stability of Two Extended BB-like Step Sizes

Abstract

The Barzilai-Borwein (BB) step sizes have a profound impact on gradient descent methods. In this work, we propose two new gradient step sizes: one longer than the original long BB step size, and the other shorter than the original short BB step size. This extends the bounds of the original BB step sizes. For strictly convex quadratic optimization problems, we employ the dynamics of difference equations to prove that these two new methods achieve R-linear convergence. Regarding stability, we surprisingly find that under certain conditions, the gradient descent method corresponding to the longer step size is stable, whereas the shorter step size consistently leads to instability. Numerical results validate these stability theories. Here, stability refers to whether the gradient norm decreases monotonically.