Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Interpreting a Penalty as the Influence of a Bayesian Prior

Abstract : In machine learning, it is common to optimize the parameters of a probabilistic model, modulated by a somewhat ad hoc regularization term that penalizes some values of the parameters. Regularization terms appear naturally in Variational Inference (VI), a tractable way to approximate Bayesian posteriors: the loss to optimize contains a Kullback--Leibler divergence term between the approximate posterior and a Bayesian prior. We fully characterize which regularizers can arise this way, and provide a systematic way to compute the corresponding prior. This viewpoint also provides a prediction for useful values of the regularization factor in neural networks. We apply this framework to regularizers such as L1 or group-Lasso.
Complete list of metadata

Cited literature [26 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-02466702
Contributor : Pierre Wolinski Connect in order to contact the contributor
Submitted on : Tuesday, February 4, 2020 - 3:46:03 PM
Last modification on : Friday, July 2, 2021 - 11:20:51 AM
Long-term archiving on: : Tuesday, May 5, 2020 - 5:51:55 PM

File

2002.00178.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02466702, version 1
  • ARXIV : 2002.00178

Citation

Pierre Wolinski, Guillaume Charpiat, Yann Ollivier. Interpreting a Penalty as the Influence of a Bayesian Prior. 2020. ⟨hal-02466702⟩

Share

Metrics

Record views

76

Files downloads

58