Mixed-strategy learning with continuous action sets

Steven Perkins (1) , Panayotis Mertikopoulos (2) , David Stuart Leslie (3)

1 School of Mathematics [Bristol]
2 POLARIS - Performance analysis and optimization of LARge Infrastructures and Systems
3 Department of Mathematics & Statistics [Lancaster]

Motivated by the recent applications of game-theoretical learning to the design of distributed control systems, we study a class of control problems that can be formulated as potential games with continuous action sets. We propose an actor-critic reinforcement learning algorithm that adapts mixed strategies over continuous action spaces. To analyze the algorithm, we extend the theory of finite-dimensional two-timescale stochastic approximation to a Banach space setting, and prove that the continuous dynamics of the process converge to equilibrium in the case of potential games. These results combine to give a provably-convergent learning algorithm in which players do not need to keep track of the controls selected by other agents.

Domaines: Optimisation et contrôle [math.OC]

Steven Perkins, Panayotis Mertikopoulos, David Stuart Leslie. Mixed-strategy learning with continuous action sets. IEEE Transactions on Automatic Control, 2017, 62 (1), pp.379 - 384. ⟨10.1109/TAC.2015.2511930⟩. ⟨hal-01382280⟩