Averaging Stochastic Gradient Descent on Riemannian Manifolds

Nilesh Tripuraneni (1) , Nicolas Flammarion (1) , Francis Bach (2) , Michael I. Jordan (1)

1 Berkeley EECS - Department of Electrical Engineering and Computer Sciences
2 SIERRA - Statistical Machine Learning and Parsimony

We consider the minimization of a function defined on a Riemannian manifold $\mathcal{M}$ accessible only through unbiased estimates of its gradients. We develop a geometric framework to transform a sequence of slowly converging iterates generated from stochastic gradient descent (SGD) on $\mathcal{M}$ to an averaged iterate sequence with a robust and fast $O(1/n)$ convergence rate. We then present an application of our framework to geodesically-strongly-convex (and possibly Euclidean non-convex) problems. Finally, we demonstrate how these ideas apply to the case of streaming $k$-PCA, where we show how to accelerate the slow rate of the randomized power method (without requiring knowledge of the eigengap) into a robust algorithm achieving the optimal rate of convergence.

Domaines: Apprentissage [cs.LG]

Nilesh Tripuraneni, Nicolas Flammarion, Francis Bach, Michael I. Jordan. Averaging Stochastic Gradient Descent on Riemannian Manifolds. Computational Learning Theory (COLT), Jul 2018, Stockholm, Sweden. ⟨hal-01957015⟩