Asaga: Asynchronous Parallel Saga

Rémi Leblond (1, 2) , Fabian Pedregosa (1, 2) , Simon Lacoste-Julien (1, 2, 3)

1 SIERRA - Statistical Machine Learning and Parsimony
2 DI-ENS - Département d'informatique - ENS Paris
3 MSR - INRIA - Microsoft Research - Inria Joint Centre

We describe Asaga, an asynchronous parallel version of the incremental gradient algorithm Saga that enjoys fast linear convergence rates. We highlight a subtle but important technical issue present in a large fraction of the recent convergence rate proofs for asynchronous parallel optimization algorithms, and propose a simplification of the recently proposed " perturbed iterate " framework that resolves it. We thereby prove that Asaga can obtain a theoretical linear speedup on multi-core systems even without sparsity assumptions. We present results of an implementation on a 40-core architecture illustrating the practical speedup as well as the hardware overhead.

Domaines: Optimisation et contrôle [math.OC]; Apprentissage [cs.LG]

Rémi Leblond, Fabian Pedregosa, Simon Lacoste-Julien. Asaga: Asynchronous Parallel Saga. 2016. ⟨hal-01407833⟩