Optimal Computational Trade-Off of Inexact Proximal Methods

Pierre Machart (1) , Sandrine Anthoine (2) , Luca Baldassarre (3)

1 QARMA - éQuipe AppRentissage et MultimediA [Marseille]
2 LATP - Laboratoire d'Analyse, Topologie, Probabilités
3 Centre for Computational Statistics and Machine Learning

In this paper, we investigate the trade-off between convergence rate and computational cost when minimizing a composite functional with proximal-gradient methods, which are a popular optimization tool in machine learning. We consider the case when the proximity operator is approximated via an iterative procedure, which leads to an algorithm with two nested loops. We show that the computationally optimal strategy to reach a desired accuracy in finite time is to set the number of inner iterations to a constant, which differs from the strategy indicated by a convergence rate analysis. In the process, we also present a new procedure called SIP that is both computationally and practically efficient. Our numerical experiments confirm the theoretical findings and suggest that SIP can be a very competitive alternative to the standard procedure.

Mots clés: Convex Optimization; Machine Learning; Signal Processing; Finite-Time Analysis

Domaines: Apprentissage [cs.LG]

Pierre Machart, Sandrine Anthoine, Luca Baldassarre. Optimal Computational Trade-Off of Inexact Proximal Methods. 2012. ⟨hal-00704398v1⟩