Communication-Efficient Distributionally Robust Decentralized Learning

Matteo Zecchin (1) , Marios Kountouris (1) , David Gesbert (1)

1 Eurecom [Sophia Antipolis]

Decentralized learning algorithms empower interconnected devices to share data and computational resources to collaboratively train a machine learning model without the aid of a central coordinator. In the case of heterogeneous data distributions at the network nodes, collaboration can yield predictors with unsatisfactory performance for a subset of the devices. For this reason, in this work we consider the formulation of a distributionally robust decentralized learning task and we propose a decentralized single loop gradient descent/ascent algorithm (AD-GDA) to directly solve the underlying minimax optimization problem. We render our algorithm communication-efficient by employing a compressed consensus scheme and we provide convergence guarantees for smooth convex and non-convex loss functions. Finally, we corroborate the theoretical findings with empirical results that highlight AD-GDA ability to provide unbiased predictors and to greatly improve communication efficiency compared to existing distributionally robust algorithms.

Domaines: Informatique [cs]

Matteo Zecchin, Marios Kountouris, David Gesbert. Communication-Efficient Distributionally Robust Decentralized Learning. Transactions on Machine Learning Research Journal, 2022. ⟨hal-03926701⟩