A Random Matrix and Concentration Inequalities framework for Neural Networks Analysis

Cosme Louart (1, 2) , Romain Couillet (3)

1 LVIC - Laboratoire Vision et Ingénierie des Contenus
2 GIPSA-CICS - GIPSA - Communication Information and Complex Systems
3 L2S - Laboratoire des signaux et systèmes

This article provides a theoretical analysis of the asymptotic performance of a regression or classification task performed by a simple random neural network. This result is obtained by leveraging a new framework at the crossroads between random matrix theory and the concentration of measure theory. This approach is of utmost interest for neural network analysis at large in that it naturally dismisses the difficulty induced by the non-linear activation functions, so long that these are Lipschitz functions. As an application, we provide formulas for the limiting law of the random neural network output and compare them conclusively to those obtained practically on handwritten digits databases.

Mots clés: random matrix theory; Neural networks; concentration inequalities; extreme learning machines

Domaines: Informatique [cs]; Réseau de neurones [cs.NE]; Apprentissage [cs.LG]; Mathématiques [math]; Probabilités [math.PR]

Cosme Louart, Romain Couillet. A Random Matrix and Concentration Inequalities framework for Neural Networks Analysis. ICASSP 2018 - IEEE International Conference on Acoustics, Speech and Signal Processing, Apr 2018, Calgary, Canada. ⟨10.1109/ICASSP.2018.8462001⟩. ⟨hal-01962077⟩