submit
english version rss feed
HAL: hal-00608041, version 1

Detailed view  Export this paper
Neural Information Processing Systems (NIPS), Espagne (2011)
Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning
Francis Bach 1, 2, Eric Moulines 3
(2011)

In this paper, we consider the minimization of a convex objective function defined on a Hilbert space, which is only available through unbiased estimates of its gradients. This problem includes standard machine learning algorithms such as kernel logistic regression and least-squares regression, and is commonly referred to as a stochastic approximation problem in the operations research community. We provide a non-asymptotic analysis of the convergence of two well-known algorithms, stochastic gradient descent (a.k.a. Robbins-Monro algorithm) as well as a simple modification where iterates are averaged (a.k.a. Polyak-Ruppert averaging). Our analysis suggests that a learning rate proportional to the inverse of the number of iterations, while leading to the optimal convergence rate in the strongly convex case, is not robust to the lack of strong convexity or the setting of the proportionality constant. This situation is remedied when using slower decays together with averaging, robustly leading to the optimal rate of convergence. We illustrate our theoretical results with simulations on synthetic and standard datasets.
1:  Laboratoire d'informatique de l'école normale supérieure (LIENS)
CNRS : UMR8548 – École normale supérieure [ENS] - Paris
2:  SIERRA (INRIA Paris - Rocquencourt)
INRIA : PARIS - ROCQUENCOURT – École normale supérieure [ENS] - Paris – CNRS : UMR8548
3:  Laboratoire Traitement et Communication de l'Information [Paris] (LTCI)
Télécom ParisTech – CNRS : UMR5141
Computer Science/Machine Learning

Mathematics/Optimization and Control

Statistics/Machine Learning

Engineering Sciences/Signal and Image processing

Computer Science/Signal and Image Processing
Attached file list to this document: 
PDF
gradsto_hal.pdf(649.2 KB)
PS
gradsto_hal.ps(1.4 MB)

all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...
all articles on CCSd database...