Distributed Optimization for Deep Learning with Gossip Exchange

Abstract : We address the issue of speeding up the training of convolutional neural networks by studying a distributed method adapted to stochastic gradient descent. Our parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way of sharing information between different threads based on gossip algorithms that show good consensus convergence properties. Our method called GoSGD has the advantage to be fully asynchronous and decentralized.
Complete list of metadatas

Cited literature [10 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01930346
Contributor : David Picard <>
Submitted on : Tuesday, January 15, 2019 - 10:47:43 AM
Last modification on : Friday, July 5, 2019 - 3:26:03 PM

File

2018NeuroComputingcord_sans ma...
Files produced by the author(s)

Identifiers

Citation

Michael Blot, David Picard, Nicolas Thome, Matthieu Cord. Distributed Optimization for Deep Learning with Gossip Exchange. Neurocomputing, Elsevier, 2019, 330, pp.287-296. ⟨10.1016/j.neucom.2018.11.002⟩. ⟨hal-01930346⟩

Share

Metrics

Record views

151

Files downloads

11