Empirical Risk Minimization with Generalized Relative Entropy Regularization - Archive ouverte HAL Accéder directement au contenu
Rapport Année : 2022

Empirical Risk Minimization with Generalized Relative Entropy Regularization

Minimisation du Risque Empirique avec Régularisation par l’Entropie Relative Généralisée

Résumé

The empirical risk minimization (ERM) problem with relative entropy regularization (ERM-RER) is investigated under the assumption that the reference measure is a sigma-finite measure instead of a probability measure. This assumption leads to a generalization of the ERM-RER (g-ERM-RER) problem that allows for a larger degree of flexibility in the incorporation of prior knowledge over the set of models. The solution of the g-ERM-RER problem is shown to be a unique probability measure mutually absolutely continuous with the reference measure and to exhibit a probably-approximately-correct (PAC) guarantee for the ERM problem. For a given dataset, the empirical risk is shown to be a sub-Gaussian random variable when the models are sampled from the solution to the g-ERM-RER problem. Finally, the sensitivity of the expected empirical risk to deviations from the solution of the g-ERM-RER problem is studied. In particular, the expectation of the absolute value of sensitivity is shown to be upper bounded, up to a constant factor, by the square root of the lautum information between the models and the datasets.
Fichier principal
Vignette du fichier
INRIA-RR9454.pdf (1.21 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03560072 , version 1 (07-02-2022)
hal-03560072 , version 2 (21-02-2022)
hal-03560072 , version 3 (21-07-2022)
hal-03560072 , version 4 (10-08-2022)
hal-03560072 , version 5 (12-07-2023)
hal-03560072 , version 6 (21-11-2023)
hal-03560072 , version 7 (27-02-2024)

Identifiants

  • HAL Id : hal-03560072 , version 4

Citer

Samir M. Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, Stefano Rini. Empirical Risk Minimization with Generalized Relative Entropy Regularization. [Research Report] RR-9454, Inria. 2022. ⟨hal-03560072v4⟩
587 Consultations
738 Téléchargements

Partager

Gmail Facebook X LinkedIn More