Exploring Weight Symmetry in Deep Neural Networks

Xu Shell Hu 1, 2, 3 Sergey Zagoruyko 4 Nikos Komodakis 1, 2, 3, 5
3 IMAGINE [Marne-la-Vallée]
LIGM - Laboratoire d'Informatique Gaspard-Monge, ENPC - École des Ponts ParisTech
4 WILLOW - Models of visual object recognition and scene understanding
DI-ENS - Département d'informatique de l'École normale supérieure, Inria de Paris
Abstract : We propose to impose symmetry in neural network parameters to improve parameter usage and make use of dedicated convolution and matrix multiplication routines. Due to significant reduction in the number of parameters as a result of the symmetry constraints, one would expect a dramatic drop in accuracy. Surprisingly, we show that this is not the case, and, depending on network size, symmetry can have little or no negative effect on network accuracy, especially in deep overparameterized networks. We propose several ways to impose local symmetry in recurrent and convolutional neural networks, and show that our symmetry parameterizations satisfy universal approximation property for single hidden layer networks. We extensively evaluate these parameterizations on CIFAR, ImageNet and language modeling datasets, showing significant benefits from the use of symmetry. For instance, our ResNet-101 with channel-wise symmetry has almost 25% less parameters and only 0.2% accuracy loss on ImageNet. Code for our experiments is available at https://github.com/hushell/deep-symmetry
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-01978633
Contributor : Sergey Zagoruyko <>
Submitted on : Friday, January 11, 2019 - 4:16:03 PM
Last modification on : Monday, July 1, 2019 - 4:54:03 PM

Links full text

Identifiers

  • HAL Id : hal-01978633, version 1
  • ARXIV : 1812.11027

Citation

Xu Shell Hu, Sergey Zagoruyko, Nikos Komodakis. Exploring Weight Symmetry in Deep Neural Networks. 2018. ⟨hal-01978633⟩

Share

Metrics

Record views

53