A comparison of continuous-time approximations to stochastic gradient descent

Stefan Ankirchner (1) , Stefan Perko (1)

1 Friedrich-Schiller-Universität = Friedrich Schiller University Jena [Jena, Germany]

Applying a stochastic gradient descent (SGD) method for minimizing an objective gives rise to a discrete-time process of estimated parameter values. In order to better understand the dynamics of the estimated values, many authors have considered continuous-time approximations of SGD. We refine existing results on the weak error of first-order ODE and SDE approximations to SGD for non-infinitesimal learning rates. In particular, we explicitly compute the leading term in the error expansion of gradient flow and two of its stochastic counterparts, with respect to a discretization parameter h. In the example of linear regression, we demonstrate the general inferiority of the deterministic gradient flow approximation in comparison to the stochastic ones. Further, we demonstrate that for Gaussian features both SDE approximations are equally good. However, for leptokurtic features we find that the SDE approximation with state-dependent diffusion coefficient is of higher quality than the approximation with state-independent noise. Moreover, the relationship reverses for platykurtic features.

Mots clés: Stochastic gradient descent, gradient flow, stochastic differential equation, weak approximation, learning rate schedules, Talay-Tubaro expansion

Domaines: Probabilités [math.PR]; Apprentissage [cs.LG]

Stefan Ankirchner, Stefan Perko. A comparison of continuous-time approximations to stochastic gradient descent. 2023. ⟨hal-03262396v3⟩