Skip to Main content Skip to Navigation
Conference papers

A Principle of Least Action for the Training of Neural Networks

Abstract : Neural networks have been achieving high generalization performance on many tasks despite being highly over-parameterized. Since classical statistical learning theory struggles to explain this behaviour, much effort has recently been focused on uncovering the mechanisms behind it, in the hope of developing a more adequate theoretical framework and having a better control over the trained models. In this work, we adopt an alternative perspective, viewing the neural network as a dynamical system displacing input particles over time. We conduct a series of experiments and, by analyzing the network's behaviour through its displacements, we show the presence of a low kinetic energy bias in the transport map of the network, and link this bias with generalization performance. From this observation, we reformulate the learning problem as follows: find neural networks that solve the task while transporting the data as efficiently as possible. This offers a novel formulation of the learning problem which allows us to provide regularity results for the solution network, based on Optimal Transport theory. From a practical viewpoint, this allows us to propose a new learning algorithm, which automatically adapts to the complexity of the task, and leads to networks with a high generalization ability even in low data regimes.
Document type :
Conference papers
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03038615
Contributor : Skander Karkar Connect in order to contact the contributor
Submitted on : Thursday, December 3, 2020 - 3:35:32 PM
Last modification on : Tuesday, March 23, 2021 - 9:28:03 AM
Long-term archiving on: : Thursday, March 4, 2021 - 7:25:08 PM

File

2009.08372.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03038615, version 1

Citation

Skander Karkar, Ibrahim Ayed, Emmanuel de Bezenac, Patrick Gallinari. A Principle of Least Action for the Training of Neural Networks. ECML PKDD, Sep 2020, Ghent, Belgium. ⟨hal-03038615⟩

Share

Metrics

Record views

84

Files downloads

147