Optimal Convergence Rate for Mirror Descent Methods with special Time-Varying Step Sizes Rules

Abstract

In this paper, the optimal convergence rate $O\left(N^{-1/2}\right)$ (where $N$ is the total number of iterations performed by the algorithm), without the presence of a logarithmic factor, is proved for mirror descent algorithms with special time-varying step sizes, for solving classical constrained non-smooth problems, problems with the composite model and problems with non-smooth functional (inequality types) constraints. The proven result is an improvement on the well-known rate $O\left(\log (N) N^{-1/2}\right)$ for the mirror descent algorithms with the time-varying step sizes under consideration. It was studied a new weighting scheme assigns smaller weights to the initial points and larger weights to the most recent points. This scheme improves the convergence rate of the considered mirror descent methods, which in the conducted numerical experiments outperform the other methods providing a better solution in all the considered test problems.