An Accelerated Decentralized Stochastic Proximal Algorithm for Finite Sums

Hadrien Hendrikx 1, 2, 3 Francis Bach 2 Laurent Massoulié 1, 3
1 DYOGENE - Dynamics of Geometric Networks
DI-ENS - Département d'informatique de l'École normale supérieure, CNRS - Centre National de la Recherche Scientifique : UMR 8548, Inria de Paris
2 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique de l'École normale supérieure, CNRS - Centre National de la Recherche Scientifique, Inria de Paris
Abstract : Modern large-scale finite-sum optimization relies on two key aspects: distribution and stochastic updates. For smooth and strongly convex problems, existing decentralized algorithms are slower than modern accelerated variance-reduced stochastic algorithms when run on a single machine, and are therefore not efficient. Centralized algorithms are fast, but their scaling is limited by global aggregation steps that result in communication bottlenecks. In this work, we propose an efficient Accelerated Decentralized stochastic algorithm for Finite Sums named ADFS, which uses local stochastic proximal updates and randomized pairwise communications between nodes. On n machines, ADFS learns from nm samples in the same time it takes optimal algorithms to learn from m samples on one machine. This scaling holds until a critical network size is reached, which depends on communication delays, on the number of samples m, and on the network topology. We provide a theoretical analysis based on a novel augmented graph approach combined with a precise evaluation of synchronization times and an extension of the accelerated proximal coordinate gradient algorithm to arbitrary sampling. We illustrate the improvement of ADFS over state-of-the-art decentralized approaches with experiments.
Complete list of metadatas

Cited literature [12 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-02280763
Contributor : Hadrien Hendrikx <>
Submitted on : Friday, September 6, 2019 - 3:36:24 PM
Last modification on : Tuesday, September 10, 2019 - 1:10:39 AM

File

1905.11394.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02280763, version 1

Collections

Citation

Hadrien Hendrikx, Francis Bach, Laurent Massoulié. An Accelerated Decentralized Stochastic Proximal Algorithm for Finite Sums. 2019. ⟨hal-02280763⟩

Share

Metrics

Record views

50

Files downloads

71