A RANDOM MATRIX APPROACH TO NEURAL NETWORKS - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue The Annals of Applied Probability Année : 2018

A RANDOM MATRIX APPROACH TO NEURAL NETWORKS

Cosme Louart
  • Fonction : Auteur
  • PersonId : 1040908
  • IdRef : 270239219
Zhenyu Liao
  • Fonction : Auteur
Romain Couillet

Résumé

This article studies the Gram random matrix model G = 1 T Σ T Σ, Σ = σ(W X), classically found in the analysis of random feature maps and random neural networks, where X = [x1,. .. , xT ] ∈ R p×T is a (data) matrix of bounded norm, W ∈ R n×p is a matrix of independent zero-mean unit variance entries, and σ : R → R is a Lipschitz continuous (activation) function-σ(W X) being understood entry-wise. By means of a key concentration of measure lemma arising from non-asymptotic random matrix arguments, we prove that, as n, p, T grow large at the same rate, the resolvent Q = (G + γIT) −1 , for γ > 0, has a similar behavior as that met in sample covariance matrix models, involving notably the moment Φ = T n E[G], which provides in passing a deterministic equivalent for the empirical spectral measure of G. Application-wise, this result enables the estimation of the asymptotic performance of single-layer random neural networks. This in turn provides practical insights into the underlying mechanisms into play in random neural networks, entailing several unexpected consequences, as well as a fast practical means to tune the network hyperparameters.
Fichier principal
Vignette du fichier
1702.05419.pdf (607.65 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01957656 , version 1 (17-12-2018)

Identifiants

Citer

Cosme Louart, Zhenyu Liao, Romain Couillet. A RANDOM MATRIX APPROACH TO NEURAL NETWORKS. The Annals of Applied Probability, 2018, 28 (2), pp.1190-1248. ⟨10.1214/17-AAP1328⟩. ⟨hal-01957656⟩
189 Consultations
166 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More