Equi-normalization of Neural Networks

Abstract : Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and output weights, without changing the rest of the network. Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for minimizing the L2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batch-and group-normalization on CIFAR-10 and ImageNet with a ResNet-18.
Document type :
Conference papers
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-02050408
Contributor : Pierre Stock <>
Submitted on : Wednesday, February 27, 2019 - 10:33:52 AM
Last modification on : Wednesday, March 6, 2019 - 1:17:50 AM

File

paper.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02050408, version 1
  • ARXIV : 1902.10416

Citation

Pierre Stock, Benjamin Graham, Rémi Gribonval, Hervé Jégou. Equi-normalization of Neural Networks. ICLR 2019 - Seventh International Conference on Learning Representations, May 2019, New Orleans, United States. pp.1-20. ⟨hal-02050408⟩

Share

Metrics

Record views

114

Files downloads

32