Extension of the safeguarding stepsize interval in Adaptive Gradient Descent

Abstract

In this paper, inspired by the idea of Adaptive Gradient Descent (AdGD) by Malitsky and Mishchenko, we propose a method that adaptively provides a stepsize interval, which represents the condition for stepsizes ensuring global convergence. By projecting the Barzilai Borwein stepsize onto this interval, we can ensure global convergence and expect further acceleration of convergence. Furthermore, we propose enlarging the stepsize interval obtained by AdGD and show that global convergence can be guaranteed within this enlarged stepsize interval. This extension of the interval enables us to adopt the full Barzilai-Borwein (BB) stepsize more frequently. We report the results of numerical experiments on the stepsize that combines the proposed interval with the BB stepsize.