Adaptive and Optimal Online Linear Regression on L1-balls

Sébastien Gerchinovitz (1, 2) , Jia Yuan Yu (1, 2, 3)

1 DMA - Département de Mathématiques et Applications - ENS Paris
2 CLASSIC - Computational Learning, Aggregation, Supervised Statistical, Inference, and Classification
3 HEC Paris - Recherche - Hors Laboratoire

We consider the problem of online linear regression on individual sequences. The goal in this paper is for the forecaster to output sequential predictions which are, after T time rounds, almost as good as the ones output by the best linear predictor in a given L1-ball in R^d. We consider both the cases where the dimension d is small and large relative to the time horizon T. We first present regret bounds with optimal dependencies on the sizes U, X and Y of the L1-ball, the input data and the observations. The minimax regret is shown to exhibit a regime transition around the point d = sqrt(T) U X / (2 Y). Furthermore, we present efficient algorithms that are adaptive, i.e., they do not require the knowledge of U, X, and Y, but still achieve nearly optimal regret bounds.

Mots clés: online learning; linear regression; adaptive algorithms; minimax regret

Domaines: Machine Learning [stat.ML]; Statistiques [math.ST]; Théorie [stat.TH]; Apprentissage [cs.LG]

Sébastien Gerchinovitz, Jia Yuan Yu. Adaptive and Optimal Online Linear Regression on L1-balls. 2011. ⟨hal-00594399v1⟩