On the convergence of a stochastic approximation method for structured bi-level optimization

Abstract : We analyze the convergence of stochastic gradient methods for well structured bi-level optimization problems. We address two specific cases: first when the outer objective function can be expressed as a finite sum of independent terms, and next when both the outer and inner objective functions can be expressed as finite sums of independent terms. We assume Lipschitz continuity and differentiability of both objectives as well as convexity of the inner objective and consider diminishing steps sizes. We show that, under these conditions and some other assumptions on the implicit function and the variance of the gradient errors, both methods converge in expectation to a stationary point of the problem if gradient approximations are chosen so as to satisfy a sufficient decrease condition. We also discuss the satisfaction of our assumptions in machine learning problems where these methods can be nicely applied to automatically tune hyperparameters when the loss functions are very large sums of error terms.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-01932372
Contributor : Nicolas Couellan <>
Submitted on : Friday, November 23, 2018 - 8:54:45 AM
Last modification on : Friday, January 10, 2020 - 9:09:00 PM

File

Bilevel_preprint.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01932372, version 1

Citation

Nicolas Couellan, Wenjuan Wang. On the convergence of a stochastic approximation method for structured bi-level optimization. 2018. ⟨hal-01932372⟩

Share

Metrics

Record views

71

Files downloads

174