Skip to Main content Skip to Navigation
Journal articles

Expectation Propagation in the large-data limit

Abstract : Expectation Propagation (Minka, 2001) is a widely successful algorithm for variational inference. EP is an iterative algorithm that can be used to approximate complicated distributions, most often posterior distributions arising in Bayesian settings. Its most typical use is to find a Gaussian approximation to posterior distributions, and in many applications of this type, EP performs extremely well. Surprisingly, despite its widespread use, there are very few theoretical guarantees on Gaussian EP.A basic requirement of statistical inference methods is that they should perform well in the limit of infinite data, and here we show that it is indeed the case for EP. In the classical large data limit, where the Bernstein-von Mises theorem applies, we prove that EP is exact, meaning that it recovers the correct Gaussian posterior. We prove further that in the same limit EP behaves like a simpler algorithm we call averaged-EP (aEP), and in turn aEP behaves similarly to the Newton algorithm. This correspondence yields interesting insights into the dynamic behavior of EP, for example that it may diverge under poor initialization, just like the Newton algorithm. EP is a simple algorithm to state, but a difficult one to study. Our results should facilitate further research into the theoretical properties of this important method.
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-01235066
Contributor : Simon Barthelmé <>
Submitted on : Friday, November 27, 2015 - 4:55:44 PM
Last modification on : Thursday, July 9, 2020 - 5:02:08 PM

Links full text

Identifiers

Collections

Citation

Guillaume Dehaene, Simon Barthelme. Expectation Propagation in the large-data limit. Journal of the Royal Statistical Society: Series B, Royal Statistical Society, 2017, 80 (Part 1), pp.197-217. ⟨hal-01235066⟩

Share

Metrics

Record views

314