CoNeT-GIANT: A compressed Newton-type fully distributed optimization algorithm

Abstract

Compression techniques are essential in distributed optimization and learning algorithms with high-dimensional model parameters, particularly in scenarios with tight communication constraints such as limited bandwidth. This article presents a communication-efficient second-order distributed optimization algorithm, termed as CoNet-GIANT, equipped with a compression module, designed to minimize the average of local strongly convex functions. CoNet-GIANT incorporates two consensus-based averaging steps at each node: gradient tracking and approximate Newton-type iterations, inspired by the recently proposed Network-GIANT. Under certain sufficient conditions on the step size, CoNet-GIANT achieves significantly faster linear convergence, comparable to that of its first-order counterparts, both in the compressed and uncompressed settings. CoNet-GIANT is efficient in terms of data usage, communication cost, and run-time, making it a suitable choice for distributed optimization over a wide range of wireless networks. Extensive experiments on synthetic data and the widely used CovType dataset demonstrate its superior performance.