Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning

Abstract : A commonly cited inefficiency of neural network training using back-propagation is the update locking problem: each layer must wait for the signal to propagate through the full network before updating. Several alternatives that can alleviate this issue have been proposed. In this context, we consider a simple alternative based on minimal feedback, which we call Decoupled Greedy Learning (DGL). It is based on a classic greedy relaxation of the joint training objective, recently shown to be effective in the context of Convolutional Neural Networks (CNNs) on large-scale image classification. We consider an optimization of this objective that permits us to decouple the layer training, allowing for layers or modules in networks to be trained with a potentially linear parallelization. With the use of a replay buffer we show that this approach can be extended to asynchronous settings, where modules can operate and continue to update with possibly large communication delays. To address bandwidth and memory issues we propose an approach based on online vector quantization. This allows to drastically reduce the communication bandwidth between modules and required memory for replay buffers. We show theoretically and empirically that this approach converges and compare it to the sequential solvers. We demonstrate the effectiveness of DGL against alternative approaches on the CIFAR-10 dataset and on the large-scale ImageNet dataset.
Document type :
Preprints, Working Papers, ...
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03247753
Contributor : Edouard Oyallon Connect in order to contact the contributor
Submitted on : Thursday, June 10, 2021 - 9:34:23 AM
Last modification on : Tuesday, July 13, 2021 - 3:27:39 AM
Long-term archiving on: : Saturday, September 11, 2021 - 6:03:26 PM

Files

example_paper.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03247753, version 1
  • ARXIV : 2106.06401

Citation

Eugene Belilovsky, Louis Leconte, Lucas Caccia, Michael Eickenberg, Edouard Oyallon. Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning. 2021. ⟨hal-03247753⟩

Share

Metrics

Record views

54

Files downloads

83