An Inertial Newton Algorithm for Deep Learning

Camille Castera (1, 2) , Jérôme Bolte (3) , Cédric Févotte (1, 2) , Edouard Pauwels (4, 5)

1 IRIT-SC - Signal et Communications
2 CNRS - Centre National de la Recherche Scientifique
3 TSE-R - Toulouse School of Economics
4 IRT Saint Exupéry - Institut de Recherche Technologique
5 IRIT-ADRIA - Argumentation, Décision, Raisonnement, Incertitude et Apprentissage

We introduce an inertial second-order method for machine learning, exploiting the geometry of the loss function while requiring only stochastic approximations function values and generalized gradients. The method features a simple mechanical interpretation and we describe promising numerical results on deep learning benchmarks. We give convergence guarantees in a theoretical framework encompassing most deep learning losses under very mild assumptions.

Domaines: Optimisation et contrôle [math.OC]; Intelligence artificielle [cs.AI]

Camille Castera, Jérôme Bolte, Cédric Févotte, Edouard Pauwels. An Inertial Newton Algorithm for Deep Learning. Thirty-third Conference on Neural Information Processing Systems : Beyond First Order Methods in ML (NeurIPS Workshop2019), Neural Information Processing Systems Foundation, Dec 2019, Vancouver, Canada. ⟨hal-03049921⟩