Structure-Adaptive, Variance-Reduced, and Accelerated Stochastic Optimization

Junqi Tang 1 Francis Bach 2 Mohammad Golbabaee 1 Mike Davies 1
2 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique de l'École normale supérieure, ENS Paris - École normale supérieure - Paris, CNRS - Centre National de la Recherche Scientifique, Inria de Paris
Abstract : In this work we explore the fundamental structure-adaptiveness of state of the art randomized first order algorithms on regularized empirical risk minimization tasks, where the solution has intrinsic low-dimensional structure (such as sparsity and low-rank). Such structure is often enforced by non-smooth regu-larization or constraints. We start by establishing the fast linear convergence rate of the SAGA algorithm on non-strongly-convex objectives with convex constraints, via an argument of cone-restricted strong convexity. Then for the composite minimization task with a coordinate-wise separable convex regularization term, we propose and analyse a two stage accelerated coordinate descend algorithm (Two-Stage APCG). We provide the convergence analysis showing that the proposed method has a global convergence in general and enjoys a local accelerated linear convergence rate with respect to the low-dimensional structure of the solution. Then based on this convergence result, we proposed an adaptive variant of the two-stage APCG method which does not need to foreknow the restricted strong convexity beforehand, but estimate it on the fly. In numerical experiments we compare the adaptive two-stage APCG with various state of the art variance-reduced stochastic gradient methods on sparse regression tasks, and demonstrate the effectiveness of our approach.
Type de document :
Pré-publication, Document de travail
2017
Liste complète des métadonnées

Littérature citée [30 références]  Voir  Masquer  Télécharger

https://hal.archives-ouvertes.fr/hal-01658487
Contributeur : Junqi Tang <>
Soumis le : jeudi 7 décembre 2017 - 16:14:00
Dernière modification le : jeudi 14 décembre 2017 - 01:15:12

Fichier

Billy Ada-TS-APCG.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01658487, version 1

Citation

Junqi Tang, Francis Bach, Mohammad Golbabaee, Mike Davies. Structure-Adaptive, Variance-Reduced, and Accelerated Stochastic Optimization. 2017. 〈hal-01658487v1〉

Partager

Métriques

Consultations de la notice

69

Téléchargements de fichiers

13