Tracking the gradients using the Hessian: A new look at variance reducing stochastic methods

Robert M. Gower 1 Nicolas Le Roux 2 Francis Bach 3
3 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique de l'École normale supérieure, CNRS - Centre National de la Recherche Scientifique, Inria de Paris
Abstract : Our goal is to improve variance reducing stochastic methods through better control variates. We first propose a modification of SVRG which uses the Hessian to track gradients over time, rather than to recondition, increasing the correlation of the control variates and leading to faster theoretical convergence close to the optimum. We then propose accurate and computationally efficient approximations to the Hessian, both using a diagonal and a low-rank matrix. Finally, we demonstrate the effectiveness of our method on a wide range of problems.
Type de document :
Pré-publication, Document de travail
17 pages, 2 figures, 1 table. 2017
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-01652152
Contributeur : Francis Bach <>
Soumis le : jeudi 30 novembre 2017 - 07:58:18
Dernière modification le : jeudi 26 avril 2018 - 10:29:06

Lien texte intégral

Identifiants

Collections

Citation

Robert M. Gower, Nicolas Le Roux, Francis Bach. Tracking the gradients using the Hessian: A new look at variance reducing stochastic methods. 17 pages, 2 figures, 1 table. 2017. 〈hal-01652152〉

Partager

Métriques

Consultations de la notice

152