Skip to Main content Skip to Navigation
Conference papers

Boosted Bellman Residual Minimization Handling Expert Demonstrations

Abstract : This paper addresses the problem of batch Reinforcement Learning with Expert Demonstrations (RLED). In RLED, the goal is to find an optimal policy of a Markov Decision Process (MDP), using a data set of fixed sampled transitions of the MDP as well as a data set of fixed expert demonstrations. This is slightly different from the batch Reinforcement Learning (RL) framework where only fixed sampled transitions of the MDP are available. Thus, the aim of this article is to propose algorithms that leverage those expert data. The idea proposed here differs from the Approximate Dynamic Programming methods in the sense that we minimize the Optimal Bellman Residual (OBR), where the minimization is guided by constraints defined by the expert demonstrations. This choice is motivated by the the fact that controlling the OBR implies controlling the distance between the estimated and optimal quality functions. However, this method presents some difficulties as the criterion to minimize is non-convex, non-differentiable and biased. Those difficulties are overcome via the embedding of distributions in a Reproducing Kernel Hilbert Space (RKHS) and a boosting technique which allows obtaining non-parametric algorithms. Finally, our algorithms are compared to the only state of the art algorithm, Approximate Policy Iteration with Demonstrations (APID) algorithm, in different experimental settings.
Document type :
Conference papers
Complete list of metadata

Cited literature [25 references]  Display  Hide  Download
Contributor : Sébastien Van Luchene Connect in order to contact the contributor
Submitted on : Thursday, September 4, 2014 - 4:21:08 PM
Last modification on : Tuesday, December 15, 2020 - 3:56:44 AM
Long-term archiving on: : Friday, December 5, 2014 - 10:39:44 AM


Files produced by the author(s)




Bilal Piot, Matthieu Geist, Olivier Pietquin. Boosted Bellman Residual Minimization Handling Expert Demonstrations. European Conference, ECML PKDD 2014, Sep 2014, Nancy, France. pp.549-564, ⟨10.1007/978-3-662-44851-9_35⟩. ⟨hal-01060953⟩



Record views


Files downloads