Stochastic Variance Reduction Methods for Saddle-Point Problems

Palaniappan Balamurugan 1, 2 Francis Bach 1, 2
1 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique de l'École normale supérieure, CNRS - Centre National de la Recherche Scientifique, Inria de Paris
Abstract : We consider convex-concave saddle-point problems where the objective functions may be split in many components, and extend recent stochastic variance reduction methods (such as SVRG or SAGA) to provide the first large-scale linearly convergent algorithms for this class of problems which is common in machine learning. While the algorithmic extension is straightforward, it comes with challenges and opportunities: (a) the convex minimization analysis does not apply and we use the notion of monotone operators to prove convergence, showing in particular that the same algorithm applies to a larger class of problems, such as variational inequalities, (b) there are two notions of splits, in terms of functions, or in terms of partial derivatives, (c) the split does need to be done with convex-concave terms, (d) non-uniform sampling is key to an efficient algorithm, both in theory and practice, and (e) these incremental algorithms can be easily accelerated using a simple extension of the "catalyst" framework, leading to an algorithm which is always superior to accelerated batch algorithms.
Type de document :
Communication dans un congrès
Neural Information Processing Systems (NIPS), Dec 2016, Barcelona, Spain. 2016, Advances in Neural Information Processing Systems
Liste complète des métadonnées

Littérature citée [31 références]  Voir  Masquer  Télécharger

https://hal.archives-ouvertes.fr/hal-01319293
Contributeur : Francis Bach <>
Soumis le : mercredi 2 novembre 2016 - 21:00:10
Dernière modification le : jeudi 26 avril 2018 - 10:29:13

Fichiers

sagasaddle.pdf
Fichiers éditeurs autorisés sur une archive ouverte

Identifiants

  • HAL Id : hal-01319293, version 2
  • ARXIV : 1605.06398

Collections

Citation

Palaniappan Balamurugan, Francis Bach. Stochastic Variance Reduction Methods for Saddle-Point Problems. Neural Information Processing Systems (NIPS), Dec 2016, Barcelona, Spain. 2016, Advances in Neural Information Processing Systems. 〈hal-01319293v2〉

Partager

Métriques

Consultations de la notice

1674

Téléchargements de fichiers

1190