A Distributed Asynchronous Generalized Momentum Algorithm Without Delay Bounds

Abstract

Asynchronous optimization algorithms often require delay bounds to prove their convergence, though these bounds can be difficult to obtain in practice. Existing algorithms that do not require delay bounds often converge slowly. Therefore, we introduce a novel distributed generalized momentum algorithm that provides fast convergence and allows arbitrary delays. It subsumes Nesterov's accelerated gradient algorithm and the heavy ball algorithm, among others. We first develop conditions on the parameters of this algorithm that ensure asymptotic convergence. Then we show its convergence rate is linear in a function of the number of computations and communications that processors perform (in a way that we make precise). Simulations compare this algorithm to gradient descent, heavy ball, and Nesterov's accelerated gradient algorithm with a classification problem on the Fashion-MNIST dataset. Across a range of scenarios with unbounded delays, convergence of the generalized momentum algorithm requires at least 71% fewer iterations than gradient descent, 41% fewer iterations than the heavy ball algorithm, and 19% fewer iterations that Nesterov's accelerated gradient algorithm.