Pure Exploration for Multi-Armed Bandit Problems

Sébastien Bubeck (1) , Rémi Munos (1) , Gilles Stoltz (2, 3)

1 SEQUEL - Sequential Learning
2 DMA - Département de Mathématiques et Applications - ENS Paris
3 GREGH - Groupement de Recherche et d'Etudes en Gestion à HEC

We consider the framework of stochastic multi-armed bandit problems and study the possibilities and limitations of strategies that explore sequentially the arms. The strategies are assessed not in terms of their cumulative regrets, as is usually the case, but through quantities referred to as simple regrets. The latter are related to the (expected) gains of the decisions that the strategies would recommend for a new one-shot instance of the same multi-armed bandit problem. Here, exploration is only constrained by the number of available rounds (not necessarily known in advance), in contrast to the case when cumulative regrets are considered and when exploitation needs to be performed at the same time. We start by indicating the links between simple and cumulative regrets. A small cumulative regret entails a small simple regret but too small a cumulative regret prevents the simple regret from decreasing exponentially towards zero, its optimal distribution-dependent rate. We therefore introduce specific strategies, for which we prove both distribution-dependent and distribution-free bounds. A concluding experimental study puts these theoretical bounds in perspective and shows the interest of non-uniform exploration of the arms.

Domaines: Statistiques [math.ST]; Théorie [stat.TH]; Apprentissage [cs.LG]

Sébastien Bubeck, Rémi Munos, Gilles Stoltz. Pure Exploration for Multi-Armed Bandit Problems. 2008. ⟨hal-00257454v3⟩