Solving F(3)MDPs: Collaborative Multiagent Markov Decision Processes with Factored Transitions, Rewards and Stochastic Policies

Julia J. Radoszycki (1) , Nathalie Dubois Peyrard Peyrard (1) , Régis Sabbadin (1)

1 MIAT INRA - Unité de Mathématiques et Informatique Appliquées de Toulouse

Multiagent Markov Decision Processes provide a rich framework to model problems of multiagent sequential decision under uncertainty, as in robotics. However, when the state space is also factored and of high dimension, even dedicated solution algorithms (exact or approximate) do not apply when the dimension of the state space and the number of agents both exceed 30, except under strong assumptions about state transitions or value function. In this paper we introduce the (FMDP)-M-3 framework and associated approximate solution algorithms which can tackle much larger problems. An (FMDP)-M-3 is a collaborative multiagent MDP whose state space is factored, reward function is additively factored and solution policies are constrained to be factored and can be stochastic. The proposed algorithms belong to the family of Policy Iteration (PI) algorithms. On small problems, where the optimal policy is available, they provide policies close to optimal. On larger problems belonging to the subclass of GMDPs they compete well with state-of-the-art resolution algorithms in terms of quality. Finally, we show that our algorithms can tackle very large F(3)MDPs, with 100 agents and a state space of size 2(100).

Mots clés: Multiagent markov decision processes; Policy gradient; Inference in graphical models; Propagation; Complexity

Domaines: Mathématiques [math]; Informatique [cs]

Julia J. Radoszycki, Nathalie Dubois Peyrard Peyrard, Régis Sabbadin. Solving F(3)MDPs: Collaborative Multiagent Markov Decision Processes with Factored Transitions, Rewards and Stochastic Policies. 18th International Conference on Principles and Practice of Multi-Agent (PRIMA 2015), Oct 2015, Bertinorio, Italy. pp.17, ⟨10.1007/978-3-319-25524-8_1⟩. ⟨hal-02743625⟩