Structure-Adaptive Accelerated Coordinate Descent

Junqi Tang 1 Mohammad Golbabaee 2 Francis Bach 3, 4 Mike Davies 1
4 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique de l'École normale supérieure, CNRS - Centre National de la Recherche Scientifique, Inria de Paris
Abstract : In this work we explore the fundamental structure-adaptiveness of accelerated randomized coordinate descent algorithms on regularized empirical risk minimization tasks, where the solution has intrinsic low-dimensional structure such as sparsity and low-rank, enforced by non-smooth regularization. We propose and analyze a two-stage accelerated coordinate descent algorithm ("two-stage APCG") utilizing the restricted strong-convexity framework. We provide the convergence analysis showing that the proposed method have a local accelerated linear convergence rate with respect to the low-dimensional structure of the solution. We also propose an adaptive variant of the two-stage APCG which does not need to foreknow the restricted strong convexity parameter beforehand, but estimates it on the fly. In our numerical experiments we test the proposed method on a number of machine learning datasets and demonstrate the effectiveness of our approach.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-01889990
Contributor : Junqi Tang <>
Submitted on : Wednesday, October 17, 2018 - 3:23:22 PM
Last modification on : Friday, April 19, 2019 - 4:55:14 PM
Long-term archiving on: Friday, January 18, 2019 - 2:26:19 PM

File

JT_TSAPCG.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01889990, version 2

Collections

Citation

Junqi Tang, Mohammad Golbabaee, Francis Bach, Mike Davies. Structure-Adaptive Accelerated Coordinate Descent. 2018. ⟨hal-01889990v2⟩

Share

Metrics

Record views

381

Files downloads

270