On Bayesian index policies for sequential resource allocation

Emilie Kaufmann (1, 2)

1 SEQUEL - Sequential Learning
2 CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189

This paper is about index policies for minimizing (frequentist) regret in a stochastic multi-armed bandit model, that are inspired by a Bayesian view on the problem. Our main contribution is to prove the asymptotic optimality of Bayes-UCB, an algorithm based on quantiles of posterior distributions, when the rewards distributions belong to a one-dimensional exponential family, for a large class of prior distributions. We also show that the Bayesian literature gives new insight on what kind of exploration rates could be used in frequentist, UCB-type algorithms. Indeed, approximations of the Bayesian optimal solution or the Finite Horizon Gittins indices suggest the introduction of two algorithms, KL-UCB + and KL-UCB-H + , whose asymptotic optimality is also established.

Mots clés: Bayesian algorithms; multi-armed bandits

Domaines: Machine Learning [stat.ML]

Emilie Kaufmann. On Bayesian index policies for sequential resource allocation. 2016. ⟨hal-01251606v1⟩