Skip to Main content Skip to Navigation
Journal articles

Bridging the Gap Between Imitation Learning and Inverse Reinforcement Learning

Bilal Piot 1, 2 Matthieu Geist 3, 4 Olivier Pietquin 2, 1
2 SEQUEL - Sequential Learning
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189
Abstract : —Learning from Demonstrations (LfD) is a paradigm by which an apprentice agent learns a control policy for a dynamic environment by observing demonstrations delivered by an expert agent. It is usually implemented as either Imitation Learning (IL) or Inverse Reinforcement Learning (IRL) in the literature. On the one hand, IRL is a paradigm relying on Markov Decision Processes (MDPs), where the goal of the apprentice agent is to find a reward function from the expert demonstrations that could explain the expert behavior. On the other hand, IL consists in directly generalizing the expert strategy, observed in the demonstrations, to unvisited states (and it is therefore close to classification, when there is a finite set of possible decisions). While these two visions are often considered as opposite to each other, the purpose of this paper is to exhibit a formal link between these approaches from which new algorithms can be derived. We show that IL and IRL can be redefined in a way that they are equivalent, in the sense that there exists an explicit bijective operator (namely the inverse optimal Bellman operator) between their respective spaces of solutions. To do so, we introduce the set-policy framework which creates a clear link between IL and IRL. As a result, IL and IRL solutions making the best of both worlds are obtained. In addition, it is a unifying framework from which existing IL and IRL algorithms can be derived and which opens the way for IL methods able to deal with the environment's dynamics. Finally, the IRL algorithms derived from the set-policy framework are compared to algorithms belonging to the more common trajectory-matching family. Experiments demonstrate that the set-policy-based algorithms outperform both standard IRL and IL ones and result in more robust solutions.
Document type :
Journal articles
Complete list of metadata

Cited literature [40 references]  Display  Hide  Download
Contributor : Matthieu Geist Connect in order to contact the contributor
Submitted on : Monday, November 6, 2017 - 4:15:54 PM
Last modification on : Sunday, October 17, 2021 - 3:02:53 AM


Files produced by the author(s)



Bilal Piot, Matthieu Geist, Olivier Pietquin. Bridging the Gap Between Imitation Learning and Inverse Reinforcement Learning. IEEE Transactions on Neural Networks and Learning Systems, IEEE, 2017, 28 (8), pp.1814 - 1826. ⟨10.1109/TNNLS.2016.2543000⟩. ⟨hal-01629654⟩



Record views


Files downloads