On the Equivalence between Herding and Conditional Gradient Algorithms

Francis Bach (1, 2) , Simon Lacoste-Julien (1, 2) , Guillaume Obozinski (1, 2)

1 SIERRA - Statistical Machine Learning and Parsimony
2 LIENS - Laboratoire d'informatique de l'école normale supérieure

We show that the herding procedure of Welling (2009) takes exactly the form of a standard convex optimization algorithm--namely a conditional gradient algorithm minimizing a quadratic moment discrepancy. This link enables us to invoke convergence results from convex optimization and to consider faster alternatives for the task of approximating integrals in a reproducing kernel Hilbert space. We study the behavior of the different variants through numerical simulations. The experiments indicate that while we can improve over herding on the task of approximating integrals, the original herding algorithm tends to approach more often the maximum entropy distribution, shedding more light on the learning bias behind herding.

Domaines: Apprentissage [cs.LG]; Machine Learning [stat.ML]; Optimisation et contrôle [math.OC]

Francis Bach, Simon Lacoste-Julien, Guillaume Obozinski. On the Equivalence between Herding and Conditional Gradient Algorithms. ICML 2012 International Conference on Machine Learning, Jun 2012, Edimburgh, United Kingdom. ⟨hal-00681128v2⟩