On the Expressive Power of Deep Fully Circulant Neural Networks

Alexandre Araujo (1) , Benjamin Negrevergne (2) , Yann Chevaleyre (1) , Jamal Atif (3)

1 LAMSADE - Laboratoire d'analyse et modélisation de systèmes pour l'aide à la décision
2 LIG - Laboratoire d'Informatique de Grenoble
3 LRI - Laboratoire de Recherche en Informatique

In this paper, we study deep fully circulant neural networks, that is deep neural networks in which all weight matrices are circulant ones. We show that these networks outperform the recently introduced deep networks with other types of structured layers. Besides introducing principled techniques for training these models, we provide theoretical guarantees regarding their expressivity. Indeed, we prove that the function space spanned by circulant networks of bounded depth includes the one spanned by dense networks with specific properties on their rank. We conduct a thorough experimental study to compare the performance of deep fully circulant networks with state of the art models based on structured matrices and with dense models. We show that our models achieve better accuracy than their structured alternatives while required 2x fewer weights as the next best approach. Finally we train deep fully circulant networks to build a compact and accurate models on a real world video classification dataset with over 3.8 million training examples.

Domaines: Apprentissage [cs.LG]; Intelligence artificielle [cs.AI]

Alexandre Araujo, Benjamin Negrevergne, Yann Chevaleyre, Jamal Atif. On the Expressive Power of Deep Fully Circulant Neural Networks. 2019. ⟨hal-02078318⟩