Stochastic Runge-Kutta methods and adaptive SGD-G2 stochastic gradient descent

Imen Ayadi (1) , Gabriel Turinici (1)

1 CEREMADE - CEntre de REcherches en MAthématiques de la DEcision

The minimization of the loss function is of paramount importance in deep neural networks. On the other hand, many popular optimization algorithms have been shown to correspond to some evolution equation of gradient flow type. Inspired by the numerical schemes used for general evolution equations we introduce a second order stochastic Runge Kutta method and show that it yields a consistent procedure for the minimization of the loss function. In addition it can be coupled, in an adaptive framework, with a Stochastic Gradient Descent (SGD) to adjust automatically the learning rate of the SGD, without the need of any additional information on the Hessian of the loss functional. The adaptive SGD, called SGD-G2, is successfully tested on standard datasets.

Mots clés: Machine Learning; SGD; stochastic gradient descent; adaptive stochastic gradient; deep learning optimization; neural networks optimization

Domaines: Machine Learning [stat.ML]; Intelligence artificielle [cs.AI]; Analyse numérique [math.NA]; Statistiques [math.ST]

Imen Ayadi, Gabriel Turinici. Stochastic Runge-Kutta methods and adaptive SGD-G2 stochastic gradient descent. 25th International Conference on Pattern Recognition (ICPR 2020), Jan 2021, Milano, Italy. ⟨hal-02483988⟩