Variance Reduction Note 4: The Convergence of SVRG

Abstract

Stochastic variance-reduced gradient (SVRG) uses a full gradient at a snapshot point to correct inexpensive component gradients. This fourth expository note on variance reduction explains that control variate and gives a self-contained proof of linear convergence for the classical strongly convex finite-sum setting. Each component is assumed convex and smooth, while only their average must be strongly convex. A second-moment bound links the estimator to the objective gaps at the current iterate and the snapshot. Summing a distance recursion over an epoch then yields a geometric decrease and an oracle complexity of order $(n+L/μ)\log(1/τ)$ for objective accuracy $τ$, with the initial gap fixed. The result uses a random snapshot update and counts all full-gradient computations. No new algorithm or rate is claimed.