Finite-Sample Analysis of Bellman Residual Minimization - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2010

Finite-Sample Analysis of Bellman Residual Minimization

Odalric-Ambrym Maillard
Rémi Munos
  • Fonction : Auteur
  • PersonId : 836863
Alessandro Lazaric
Mohammad Ghavamzadeh
  • Fonction : Auteur
  • PersonId : 868946

Résumé

We consider the Bellman residual minimization approach for solving discounted Markov decision problems, where we assume that a generative model of the dynamics and rewards is available. At each policy iteration step, an approximation of the value function for the current policy is obtained by minimizing an empirical Bellman residual defined on a set of n states drawn i.i.d. from a distribution, the immediate rewards, and the next states sampled from the model. Our main result is a generalization bound for the Bellman residual in linear approximation spaces. In particular, we prove that the empirical Bellman residual approaches the true (quadratic) Bellman residual with a rate of order O(1/sqrt((n)). This result implies that minimizing the empirical residual is indeed a sound approach for the minimization of the true Bellman residual which guarantees a good approximation of the value function for each policy. Finally, we derive performance bounds for the resulting approximate policy iteration algorithm in terms of the number of samples n and a measure of how well the function space is able to approximate the sequence of value functions.)
Fichier principal
Vignette du fichier
brm_acml2010.pdf (214.98 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-00830212 , version 1 (04-06-2013)

Identifiants

  • HAL Id : hal-00830212 , version 1

Citer

Odalric-Ambrym Maillard, Rémi Munos, Alessandro Lazaric, Mohammad Ghavamzadeh. Finite-Sample Analysis of Bellman Residual Minimization. Asian Conference on Machine Learning, 2010, Japan. ⟨hal-00830212⟩
276 Consultations
151 Téléchargements

Partager

Gmail Facebook X LinkedIn More