Compressed Distributed Stochastic Nonconvex Optimization with Differential Privacy

Abstract

This paper studies distributed stochastic nonconvex optimization problems with compressed communication and differential privacy, in which each agent aims to minimize the sum of all agents' cost functions by using local compressed information exchange. To this end, we propose a compressed distributed stochastic gradient descent algorithm, which is robust under a general class of compression operators that allow both relative and absolute compression errors. We then show that the proposed algorithm finds the first-order stationary point for smooth nonconvex functions with the linear speedup convergence rate $\mathcal{O}(1/\sqrt{nT})$ and converges to the optimum if the global cost function additionally satisfies the Polyak--{\L}ojasiewicz (P--\L) condition with the convergence rate $\mathcal{O}(1/(nT^\theta)),\theta\in(0,1)$, where $T$ is the total number of iterations and $n$ is the number of agents. Furthermore, if the P--\L ~constant is known in advance, we show that the proposed algorithm achieves a convergence rate $\mathcal{O}(1/(nT))$. Finally, we show that the proposed algorithm is able to achieve $(0,\delta)$-differential privacy without sacrificing convergence accuracy. Numerical experiments are carried out to