PAC-Bayesian Theory Meets Bayesian Inference

Pascal Germain (1) , Francis Bach (2, 1) , Alexandre Lacoste (3) , Simon Lacoste-Julien (1, 4, 5)

1 SIERRA - Statistical Machine Learning and Parsimony
2 LIENS - Laboratoire d'informatique de l'école normale supérieure
3 Google Inc
4 DI-ENS - Département d'informatique - ENS Paris
5 MSR - INRIA - Microsoft Research - Inria Joint Centre

We exhibit a strong link between frequentist PAC-Bayesian bounds and the Bayesian marginal likelihood. That is, for the negative log-likelihood loss function, we show that the minimization of PAC-Bayesian generalization bounds maximizes the Bayesian marginal likelihood. This provides an alternative explanation to the Bayesian Occam's razor criteria, under the assumption that the data is generated by a i.i.d. distribution. Moreover, as the negative log-likelihood is an unbounded loss function, we motivate and propose a PAC-Bayesian theorem tailored for the sub-Gamma loss family, and we show that our approach is sound on classical Bayesian linear regression tasks.

Mots clés: PAC-Bayesian Theory; Bayesian Inference

Domaines: Machine Learning [stat.ML]; Apprentissage [cs.LG]

Pascal Germain, Francis Bach, Alexandre Lacoste, Simon Lacoste-Julien. PAC-Bayesian Theory Meets Bayesian Inference. 2016. ⟨hal-01324072v1⟩