Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Concave Utility Reinforcement Learning: the Mean-field Game viewpoint

Matthieu Geist Julien Pérolat Mathieu Laurière Romuald Elie Sarah Perrin 1 Olivier Bachem Rémi Munos Olivier Pietquin
1 Scool - Scool
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189
Abstract : Concave Utility Reinforcement Learning (CURL) extends RL from linear to concave utilities in the occupancy measure induced by the agent's policy. This encompasses not only RL but also imitation learning and exploration, among others. Yet, this more general paradigm invalidates the classical Bellman equations, and calls for new algorithms. Mean-field Games (MFGs) are a continuous approximation of many-agent RL. They consider the limit case of a continuous distribution of identical agents, anonymous with symmetric interests, and reduce the problem to the study of a single representative agent in interaction with the full population. Our core contribution consists in showing that CURL is a subclass of MFGs. We think this important to bridge together both communities. It also allows to shed light on aspects of both fields: we show the equivalence between concavity in CURL and monotonicity in the associated MFG, between optimality conditions in CURL and Nash equilibrium in MFG, or that Fictitious Play (FP) for this class of MFGs is simply Frank-Wolfe, bringing the first convergence rate for discrete-time FP for MFGs. We also experimentally demonstrate that, using algorithms recently introduced for solving MFGs, we can address the CURL problem more efficiently.
Complete list of metadata
Contributor : Sarah Perrin Connect in order to contact the contributor
Submitted on : Friday, November 5, 2021 - 10:44:13 AM
Last modification on : Friday, January 21, 2022 - 3:12:23 AM

Links full text


  • HAL Id : hal-03416247, version 1
  • ARXIV : 2106.03787



Matthieu Geist, Julien Pérolat, Mathieu Laurière, Romuald Elie, Sarah Perrin, et al.. Concave Utility Reinforcement Learning: the Mean-field Game viewpoint. 2021. ⟨hal-03416247⟩



Les métriques sont temporairement indisponibles