Robust Decentralized Optimization under Node Failures via Adaptive Regularization

Abstract

We study decentralized minimization of a sum of functions over a network where nodes may only leave, the remaining nodes stay connected, and the topology freezes between departures. Standard methods forget departed functions, causing a permanent bias proportional to the heterogeneity of the data. We propose Legacy Gradient Tracking (Legacy-GT): before leaving, a node compresses its function into a gradient-anchored quadratic legacy, bequeaths it to a neighbor, and provides a correction that exactly preserves the gradient-tracking invariant. We prove that the optimal legacy curvature is the average of the strong-convexity and smoothness constants, that legacies compose losslessly across chained departures, and that an adaptive anchor rule yields error bounds that decay geometrically after the last departure to a small residual--the minimum of the network's optimization error at departure and its heterogeneity radius. In contrast, the classic drop-and-forget baseline suffers a bias that never decays. Numerical experiments confirm the theory.