Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

On the convergence of a stochastic approximation method for structured bi-level optimization

Abstract : We analyze the convergence of stochastic gradient methods for well structured bi-level optimization problems. We address two specific cases: first when the outer objective function can be expressed as a finite sum of independent terms, and next when both the outer and inner objective functions can be expressed as finite sums of independent terms. We assume Lipschitz continuity and differentiability of both objectives as well as convexity of the inner objective and consider diminishing steps sizes. We show that, under these conditions and some other assumptions on the implicit function and the variance of the gradient errors, both methods converge in expectation to a stationary point of the problem if gradient approximations are chosen so as to satisfy a sufficient decrease condition. We also discuss the satisfaction of our assumptions in machine learning problems where these methods can be nicely applied to automatically tune hyperparameters when the loss functions are very large sums of error terms.
Document type :
Preprints, Working Papers, ...
Complete list of metadata
Contributor : Nicolas Couellan Connect in order to contact the contributor
Submitted on : Friday, November 23, 2018 - 8:54:45 AM
Last modification on : Wednesday, November 3, 2021 - 5:17:12 AM


Files produced by the author(s)


  • HAL Id : hal-01932372, version 1


Nicolas Couellan, Wenjuan Wang. On the convergence of a stochastic approximation method for structured bi-level optimization. 2018. ⟨hal-01932372⟩



Record views


Files downloads