Skip to Main content Skip to Navigation
New interface
Conference papers

A Principle of Least Action for the Training of Neural Networks

Abstract : Neural networks have been achieving high generalization performance on many tasks despite being highly over-parameterized. Since classical statistical learning theory struggles to explain this behaviour, much effort has recently been focused on uncovering the mechanisms behind it, in the hope of developing a more adequate theoretical framework and having a better control over the trained models. In this work, we adopt an alternative perspective, viewing the neural network as a dynamical system displacing input particles over time. We conduct a series of experiments and, by analyzing the network's behaviour through its displacements, we show the presence of a low kinetic energy bias in the transport map of the network, and link this bias with generalization performance. From this observation, we reformulate the learning problem as follows: find neural networks that solve the task while transporting the data as efficiently as possible. This offers a novel formulation of the learning problem which allows us to provide regularity results for the solution network, based on Optimal Transport theory. From a practical viewpoint, this allows us to propose a new learning algorithm, which automatically adapts to the complexity of the task, and leads to networks with a high generalization ability even in low data regimes.
Document type :
Conference papers
Complete list of metadata
Contributor : skander karkar Connect in order to contact the contributor
Submitted on : Thursday, December 3, 2020 - 3:35:32 PM
Last modification on : Sunday, June 26, 2022 - 2:59:37 AM
Long-term archiving on: : Thursday, March 4, 2021 - 7:25:08 PM


Files produced by the author(s)


  • HAL Id : hal-03038615, version 1


Skander Karkar, Ibrahim Ayed, Emmanuel de Bezenac, Patrick Gallinari. A Principle of Least Action for the Training of Neural Networks. ECML PKDD, Sep 2020, Ghent, Belgium. ⟨hal-03038615⟩



Record views


Files downloads