Stochastic thermodynamic integration: efficient Bayesian model selection via stochastic gradient MCMC

Umut Şimşekli (1, 2) , Roland Badeau (2, 1) , Gael Richard (1, 2) , Ali Taylan Cemgil (3)

1 LTCI - Laboratoire Traitement et Communication de l'Information
2 Télécom ParisTech
3 Boǧaziçi üniversitesi = Boğaziçi University [Istanbul]

Model selection is a central topic in Bayesian machine learning, which requires the estimation of the marginal likelihood of the data under the models to be compared. During the last decade, conventional model selection methods have lost their charm as they have high computational requirements. In this study, we propose a computationally efficient model selection method by integrating ideas from Stochastic Gradient Markov Chain Monte Carlo (SG-MCMC) literature and statistical physics. As opposed to conventional methods, the proposed method has very low computational needs and can be implemented almost without modifying existing SG-MCMC code. We provide an upper-bound for the bias of the proposed method. Our experiments show that, our method is 40 times as fast as the baseline method on finding the optimal model order in a matrix factorization problem.

Mots clés: Bayesian model selection; Markov Chain Monte Carlo; Non-negative matrix factorization

Domaines: Traitement du signal et de l'image [eess.SP]; Machine Learning [stat.ML]

Umut Şimşekli, Roland Badeau, Gael Richard, Ali Taylan Cemgil. Stochastic thermodynamic integration: efficient Bayesian model selection via stochastic gradient MCMC. 41st International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016, Shanghai, China. ⟨hal-01248011⟩