PAC-Bayesian Theory Meets Bayesian Inference

Abstract : We exhibit a strong link between frequentist PAC-Bayesian bounds and the Bayesian marginal likelihood. That is, for the negative log-likelihood loss function, we show that the minimization of PAC-Bayesian generalization bounds maximizes the Bayesian marginal likelihood. This provides an alternative explanation to the Bayesian Occam's razor criteria, under the assumption that the data is generated by a i.i.d. distribution. Moreover, as the negative log-likelihood is an unbounded loss function, we motivate and propose a PAC-Bayesian theorem tailored for the sub-Gamma loss family, and we show that our approach is sound on classical Bayesian linear regression tasks.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-01324072
Contributor : Pascal Germain <>
Submitted on : Tuesday, May 31, 2016 - 3:40:42 PM
Last modification on : Tuesday, April 24, 2018 - 5:20:15 PM
Long-term archiving on: Thursday, September 1, 2016 - 11:29:32 AM

File

BPB.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01324072, version 1
  • ARXIV : 1605.08636

Citation

Pascal Germain, Francis Bach, Alexandre Lacoste, Simon Lacoste-Julien. PAC-Bayesian Theory Meets Bayesian Inference. 2016. ⟨hal-01324072v1⟩

Share

Metrics

Record views

67

Files downloads

113