Expectation Propagation in the large-data limit
Résumé
Expectation Propagation (Minka, 2001) is a widely successful algorithm for variational inference. EP is an iterative algorithm that can be used to approximate complicated distributions, most often posterior distributions arising in Bayesian settings. Its most typical use is to find a Gaussian approximation to posterior distributions, and in many applications of this type, EP performs extremely well. Surprisingly, despite its widespread use, there are very few theoretical guarantees on Gaussian EP.A basic requirement of statistical inference methods is that they should perform well in the limit of infinite data, and here we show that it is indeed the case for EP. In the classical large data limit, where the Bernstein-von Mises theorem applies, we prove that EP is exact, meaning that it recovers the correct Gaussian posterior. We prove further that in the same limit EP behaves like a simpler algorithm we call averaged-EP (aEP), and in turn aEP behaves similarly to the Newton algorithm. This correspondence yields interesting insights into the dynamic behavior of EP, for example that it may diverge under poor initialization, just like the Newton algorithm. EP is a simple algorithm to state, but a difficult one to study. Our results should facilitate further research into the theoretical properties of this important method.