Precise Minimax Regret for Logistic Regression - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Precise Minimax Regret for Logistic Regression

Résumé

We study online logistic regression with binary labels and general feature values in which a learner sequentially tries to predict an outcome/ label based on data/ features received in rounds. Our goal is to evaluate precisely the (maximal) minimax regret which we analyze using a unique and novel combination of information-theoretic and analytic combinatoric tools such as Fourier transform, saddle point method, and Mellin transform in the multi-dimensional settings. To be more precise, the pointwise regret of an online algorithm is defined as the (excess) loss it incurs over a constant comparator (weight vector) that is used for prediction. It depends on the feature values, label sequence, and the learning algorithm. In the maximal minimax scenario we seek the best weights for the worst label sequence over all label distributions. For dimension d = o(T 1/3) we show that the maximal minimax regret grows as d 2 log(2T /π) + C d + O(d 3/2 / √ T) where T is the number of rounds of running a training algorithm and C d is explicitly computable constant that depends on dimension d and data. For features uniformly distributed on a d-dimensional sphere or ball we estimate precisely the constant C d showing that C d ∼ −(d/2) log(d/ √ 2π) leading to the minimax regret growing for large d as (d/2) log(T /d) − (d/2) log(√ 8π) + O(1). We also extend these results to non-binary labels. The precise maximal minimax regret presented here is the first result of this kind for any feature values and wide range of d. This provides a precise answer to the challenge posed in McMahan and Streeter (2010).
Fichier principal
Vignette du fichier
colt-regret.pdf (310.21 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03279834 , version 1 (06-07-2021)

Identifiants

  • HAL Id : hal-03279834 , version 1

Citer

Philippe Jacquet, Gil I. Shamir, Wojciech Szpankowski. Precise Minimax Regret for Logistic Regression. ISIT 2022 - IEEE International Symposium on Information Theory, Jun 2022, Espoo, Finland. ⟨hal-03279834⟩
66 Consultations
100 Téléchargements

Partager

Gmail Facebook X LinkedIn More