A Parameter-Free Restart Scheme with Only a Parallelizable $\log\log(1/ε)$ Overhead

Abstract

It is well-known that first-order methods can offer accelerated convergence rates in the presence of growth structures. Restarting schemes provide a general tool for such speed-ups. These schemes typically either require unrealistic problem knowledge, incur logarithmic overhead factors in oracle complexity, and/or have a nontrivial initial burn-in phase. We present a parameter-free approach for restarting any first-order method, avoiding these three drawbacks. Our approach dynamically deploys parallel instances of a given first-order method communicating progress in the style of Renegar and Grimmer. Our optimized scheme avoids expensive burn-ins and only requires $O(\log\log(1/ε))$ parallel processes when the accelerated rate is sublinear.