A survey of actor-critic reinforcement learning: standard and natural policy gradients

Ivo Grondman (1) , Lucian Busoniu (2) , Gabriel Lopes (1) , Robert Babuska (1)

1 DCSC - Delft Center for Systems and Control [Delft]
2 CRAN - Centre de Recherche en Automatique de Nancy

Policy gradient based actor-critic algorithms are amongst the most popular algorithms in the reinforcement learning framework. Their advantage of being able to do policy search using low-variance gradient estimates has made them useful in several real-life applications, such as robotics, power control and finance. Although general surveys on reinforcement learning techniques already exist, no survey is specifically dedicated to actor-critic algorithms in particular. This paper therefore describes the state of the art of actor-critic algorithms, with a focus on methods that can work in an online setting and use function approximation in order to deal with continuous state and action spaces. After starting with a discussion on the concepts of reinforcement learning and the origins of actor-critic algorithms, this paper describes the workings of the natural gradient, which has made its way into many actor-critic algorithms in the past few years. A review of several standard and natural actor-critic algorithms follows and the paper concludes with an overview of application areas and a discussion on open issues.

Mots clés: actor-critic; natural gradient; policy gradient; reinforcement learning

Domaines: Automatique / Robotique; Automatique; Apprentissage [cs.LG]

Ivo Grondman, Lucian Busoniu, Gabriel Lopes, Robert Babuska. A survey of actor-critic reinforcement learning: standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 2012, 42 (6), pp.1291-1307. ⟨10.1109/TSMCC.2012.2218595⟩. ⟨hal-00756747⟩