Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning

Francis Bach 1, 2 Eric Moulines 3
2 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique de l'École normale supérieure, ENS Paris - École normale supérieure - Paris, Inria Paris-Rocquencourt, CNRS - Centre National de la Recherche Scientifique : UMR8548
Abstract : In this paper, we consider the minimization of a convex objective function defined on a Hilbert space, which is only available through unbiased estimates of its gradients. This problem includes standard machine learning algorithms such as kernel logistic regression and least-squares regression, and is commonly referred to as a stochastic approximation problem in the operations research community. We provide a non-asymptotic analysis of the convergence of two well-known algorithms, stochastic gradient descent (a.k.a. Robbins-Monro algorithm) as well as a simple modification where iterates are averaged (a.k.a. Polyak-Ruppert averaging). Our analysis suggests that a learning rate proportional to the inverse of the number of iterations, while leading to the optimal convergence rate in the strongly convex case, is not robust to the lack of strong convexity or the setting of the proportionality constant. This situation is remedied when using slower decays together with averaging, robustly leading to the optimal rate of convergence. We illustrate our theoretical results with simulations on synthetic and standard datasets.
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-00608041
Contributeur : Francis Bach <>
Soumis le : mardi 12 juillet 2011 - 09:17:30
Dernière modification le : mercredi 20 février 2019 - 01:28:52
Document(s) archivé(s) le : jeudi 13 octobre 2011 - 02:21:19

Fichiers

gradsto_hal.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-00608041, version 1

Citation

Francis Bach, Eric Moulines. Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning. Neural Information Processing Systems (NIPS), 2011, Spain. 2011. 〈hal-00608041〉

Partager

Métriques

Consultations de la notice

11876

Téléchargements de fichiers

3081