Harnessing neural networks: A random matrix approach

Cosme Louart (1) , Romain Couillet (1)

1 L2S - Laboratoire des signaux et systèmes

This article proposes an original approach to the performance understanding of large dimensional neural networks. In this preliminary study, we study a single hidden layer feed-forward network with random input connections (also called extreme learning machine) which performs a simple regression task. By means of a new random matrix result, we prove that, as the size and cardinality of the input data and the number of neurons grow large, the network performance is asymptotically deterministic. This entails a better comprehension of the effects of the hyper-parameters (activation function, number of neurons, etc.) under this simple setting, thereby paving the path to the harnessing of more involved structures. Index Terms-Neural networks, random matrix theory, extreme learning machines.

Domaines: Informatique [cs]; Apprentissage [cs.LG]; Réseau de neurones [cs.NE]; Mathématiques [math]; Probabilités [math.PR]

Cosme Louart, Romain Couillet. Harnessing neural networks: A random matrix approach. The 42nd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), Mar 2017, New Orleans, United States. ⟨10.1109/ICASSP.2017.7952563⟩. ⟨hal-01962073⟩