A Shooting Formulation of Deep Learning - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2020

A Shooting Formulation of Deep Learning

Résumé

Continuous-depth neural networks can be viewed as deep limits of discrete neural networks whose dynamics resemble a discretization of an ordinary differential equation (ODE). Although important steps have been taken to realize the advantages of such continuous formulations, most current techniques are not truly continuous-depth as they assume \textit{identical} layers. Indeed, existing works throw into relief the myriad difficulties presented by an infinite-dimensional parameter space in learning a continuous-depth neural ODE. To this end, we introduce a shooting formulation which shifts the perspective from parameterizing a network layer-by-layer to parameterizing over optimal networks described only by a set of initial conditions. For scalability, we propose a novel particle-ensemble parametrization which fully specifies the optimal weight trajectory of the continuous-depth neural network. Our experiments show that our particle-ensemble shooting formulation can achieve competitive performance, especially on long-range forecasting tasks. Finally, though the current work is inspired by continuous-depth neural networks, the particle-ensemble shooting formulation also applies to discrete-time networks and may lead to a new fertile area of research in deep learning parametrization.
Fichier principal
Vignette du fichier
main.pdf (7.06 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-02871236 , version 1 (17-06-2020)
hal-02871236 , version 2 (08-12-2020)

Identifiants

Citer

François-Xavier Vialard, Roland Kwitt, Susan Wei, Marc Niethammer. A Shooting Formulation of Deep Learning. 2020. ⟨hal-02871236v2⟩
158 Consultations
41 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More