Model-Free Reinforcement Learning with Skew-Symmetric Bilinear Utilities

Hugo Gilbert (1) , Bruno Zanuttini (2) , Paolo Viappiani (1) , Paul Weng (3, 4) , Esther Nicart (5, 2)

1 DECISION
2 Equipe MAD - Laboratoire GREYC - UMR6072
3 SYSU-CMU Joint Institute of Engineering
4 SYSU-CMU Shunde International Joint Research Institute
5 Cordon Electronics DS2i

In reinforcement learning, policies are typically evaluated according to the expectation of cumu-lated rewards. Researchers in decision theory have argued that more sophisticated decision criteria can better model the preferences of a decision maker. In particular, Skew-Symmetric Bilinear (SSB) utility functions generalize von Neumann and Morgenstern's expected utility (EU) theory to encompass rational decision behaviors that EU cannot accommodate. In this paper , we adopt an SSB utility function to compare policies in the reinforcement learning setting. We provide a model-free SSB reinforcement learning algorithm, SSB Q-learning, and prove its convergence towards a policy that is-optimal according to SSB. The proposed algorithm is an adaptation of fictitious play [Brown, 1951] combined with techniques from stochastic approximation [Borkar, 1997]. We also present some experimental results which evaluate our approach in a variety of settings.

Domaines: Informatique [cs]; Intelligence artificielle [cs.AI]; Apprentissage [cs.LG]

Hugo Gilbert, Bruno Zanuttini, Paolo Viappiani, Paul Weng, Esther Nicart. Model-Free Reinforcement Learning with Skew-Symmetric Bilinear Utilities. 32nd Conference on Uncertainty in Artificial Intelligence (UAI~2016), Jun 2016, New York City, United States. ⟨hal-01356085⟩