A Proof of the Exact Convergence Rate of Gradient Descent

Abstract

We prove the exact worst-case convergence rate of gradient descent for smooth strongly convex optimization on $\mathbb{R}^d$. Concretely, assuming that the objective function $f$ is $\mu$-strongly convex and $L$-smooth, we identify the smallest possible value of $\tau$ for which the inequality $f(x_{N})-f_{*}\leq\tau\|x_{0}-x_{*}\|^{2}$ always holds. The result was previously conjectured by Drori and Teboulle for the case $\mu=0$, and by Taylor, Hendrickx, and Glineur for the case $\mu>0$.