Is the Bellman residual a bad proxy?

Matthieu Geist 1 Bilal Piot 2 Olivier Pietquin 3
3 SEQUEL - Sequential Learning
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189
Abstract : This paper aims at theoretically and empirically comparing two standard optimization criteria for Reinforcement Learning: i) maximization of the mean value and ii) minimization of the Bellman residual. For that purpose, we place ourselves in the framework of policy search algorithms, that are usually designed to maximize the mean value, and derive a method that minimizes the residual T * v π − v π 1,ν over policies. A theoretical analysis shows how good this proxy is to policy optimization , and notably that it is better than its value-based counterpart. We also propose experiments on randomly generated generic Markov decision processes, specifically designed for studying the influence of the involved concentrability coefficient. They show that the Bellman residual is generally a bad proxy to policy optimization and that directly maximizing the mean value is much better, despite the current lack of deep theoretical analysis. This might seem obvious, as directly addressing the problem of interest is usually better, but given the prevalence of (projected) Bellman residual minimization in value-based reinforcement learning, we believe that this question is worth to be considered.
Type de document :
Communication dans un congrès
NIPS 2017 - Advances in Neural Information Processing Systems, Dec 2017, Long Beach, United States. pp.1-13, 2017
Liste complète des métadonnées

Littérature citée [23 références]  Voir  Masquer  Télécharger
Contributeur : Matthieu Geist <>
Soumis le : lundi 6 novembre 2017 - 17:15:32
Dernière modification le : mardi 3 juillet 2018 - 11:35:56


Fichiers produits par l'(les) auteur(s)


  • HAL Id : hal-01629739, version 1


Matthieu Geist, Bilal Piot, Olivier Pietquin. Is the Bellman residual a bad proxy?. NIPS 2017 - Advances in Neural Information Processing Systems, Dec 2017, Long Beach, United States. pp.1-13, 2017. 〈hal-01629739〉



Consultations de la notice


Téléchargements de fichiers