A RANDOM MATRIX APPROACH TO NEURAL NETWORKS

Cosme Louart 1 Zhenyu Liao Romain Couillet 2
1 LVIC - Laboratoire Vision et Ingénierie des Contenus
DIASI - Département Intelligence Ambiante et Systèmes Interactifs : DRT/LIST/DIASI
Abstract : This article studies the Gram random matrix model G = 1 T Σ T Σ, Σ = σ(W X), classically found in the analysis of random feature maps and random neural networks, where X = [x1,. .. , xT ] ∈ R p×T is a (data) matrix of bounded norm, W ∈ R n×p is a matrix of independent zero-mean unit variance entries, and σ : R → R is a Lipschitz continuous (activation) function-σ(W X) being understood entry-wise. By means of a key concentration of measure lemma arising from non-asymptotic random matrix arguments, we prove that, as n, p, T grow large at the same rate, the resolvent Q = (G + γIT) −1 , for γ > 0, has a similar behavior as that met in sample covariance matrix models, involving notably the moment Φ = T n E[G], which provides in passing a deterministic equivalent for the empirical spectral measure of G. Application-wise, this result enables the estimation of the asymptotic performance of single-layer random neural networks. This in turn provides practical insights into the underlying mechanisms into play in random neural networks, entailing several unexpected consequences, as well as a fast practical means to tune the network hyperparameters.
Complete list of metadatas

Cited literature [40 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01962070
Contributor : Cosme Louart <>
Submitted on : Thursday, December 20, 2018 - 1:13:13 PM
Last modification on : Monday, February 18, 2019 - 7:52:04 PM
Long-term archiving on : Friday, March 22, 2019 - 10:31:00 AM

File

1702.05419.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01962070, version 1
  • ARXIV : 1702.05419

Citation

Cosme Louart, Zhenyu Liao, Romain Couillet. A RANDOM MATRIX APPROACH TO NEURAL NETWORKS. Annals of Applied Probability, Institute of Mathematical Statistics (IMS), 2017. ⟨hal-01962070⟩

Share

Metrics

Record views

107

Files downloads

87