The Dynamics of Learning: A Random Matrix Approach

Zhenyu Liao 1 Romain Couillet 1, 2
2 GIPSA-CICS - CICS
GIPSA-DIS - Département Images et Signal
Abstract : Understanding the learning dynamics of neural networks is one of the key issues for the improvement of optimization algorithms as well as for the theoretical comprehension of why deep neu-ral nets work so well today. In this paper, we introduce a random matrix-based framework to analyze the learning dynamics of a single-layer linear network on a binary classification problem, for data of simultaneously large dimension and size, trained by gradient descent. Our results provide rich insights into common questions in neural nets, such as overfitting, early stopping and the initialization of training, thereby opening the door for future studies of more elaborate structures and models appearing in today's neural networks.
Type de document :
Communication dans un congrès
International Conference on Machine Learning (ICML 2018), Jul 2018, Stockholm, Sweden. Proceedings of the 35th International Conference on Machine Learning, PMLR
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-01957704
Contributeur : Romain Couillet <>
Soumis le : lundi 17 décembre 2018 - 15:06:37
Dernière modification le : mercredi 13 mars 2019 - 13:54:20
Document(s) archivé(s) le : lundi 18 mars 2019 - 15:14:55

Fichier

liao18b.pdf
Fichiers éditeurs autorisés sur une archive ouverte

Identifiants

  • HAL Id : hal-01957704, version 1

Citation

Zhenyu Liao, Romain Couillet. The Dynamics of Learning: A Random Matrix Approach. International Conference on Machine Learning (ICML 2018), Jul 2018, Stockholm, Sweden. Proceedings of the 35th International Conference on Machine Learning, PMLR. 〈hal-01957704〉

Partager

Métriques

Consultations de la notice

26

Téléchargements de fichiers

11