On Federated Averaging with Partial Participation under a General Variance Condition
Ziming Wang, Kun Huang, Karl Henrik Johansson, Shi Pu
Abstract
In this paper, we study the convergence of Federated Averaging (FedAvg) with partial participation under a general variance condition. In each communication round, the server samples S of the n clients uniformly at random without replacement. The stochastic gradients are assumed to satisfy the so-called ABC condition, which allows their variance to grow with the local objective gaps. For smooth nonconvex objectives, we establish a finite-time stationarity bound in terms of the heterogeneity measure σf:=f-1nΣi=1n fi and the client sampling factor IS:=(n-S)/(n-1). Here, f and fi denote the infima of the global average objective and client i's local objective, respectively. The bound exhibits a linear speedup in the number of sampled clients S and recovers the full-participation result when S=n. Under diminishing stepsizes, we further prove that the gradient norm converges to zero both in expectation and almost surely. Notably, compared with existing analyses of FedAvg with partial participation, our results require no additional data heterogeneity assumption such as bounded gradient dissimilarity (BGD), and the sampling-induced term in our bound is not amplified by the number of local updates at the leading order. The importance of removing the BGD assumption is illustrated by an example that is smooth and lower bounded but violates BGD, yet is covered by our analysis.
Create a lesson
Related papers
Optimal Scheduling of Multi-Site AI Data Centers in Day-Ahead Markets
Xinyang Zhou, Guido Cavraro
Minimization of free energy in a Poisson--Boltzmann problem
Shahid Ali, Klemen Bohinc, Angeles Martinez et al.
Lyapunov Theory for Output-to-State Stability of Nonlinear Infinite-Dimensional Systems
Qiaoling Chen, Andrii Mironchenko, Fabian Wirth
Semi-global Convergence and Finite-Sample Error Bounds for Model Predictive Path Integral Control
Christian Fiedler, Giacomo Borghi
When Suspicion Adds Up: A Stochastic Framework for Optimizing Honeypot-Based Insider-Threat Detection
Saharnaz Mehrani, Lujia Zhan, Mohsen Emadikhiav et al.
Hamiltonian Potential Optimization for Partially Observed Stochastic Optimal Control
Christian Bayer, Saifeddine Ben Naamia, Erik von Schwerin et al.