Action Decomposable MDP A Linear Programming formulation for queuing network problems

Ariel Waserhole (1) , Vincent Jost (2) , Jean-Philippe Gayon (3)

1 G-SCOP_ROSP - Recherche Opérationnelle pour les Systèmes de Production
2 LIX - Laboratoire d'informatique de l'École polytechnique [Palaiseau]
3 G-SCOP_GCSP - Gestion et Conduite des Systèmes de Production

Markov decision processes (MDP) have provided general frameworks for many control, decision making, and stochastic optimization problems. In this paper we are interested in a class of queuing control that can be modelled with a Continuous-Time MDP but that have an exponential actions state space with classic methods. The Event Based Dynamic Programming approach deals with this problem and provides some algorithms (value iteration) to compute the best policy efficiently. However there is no formal definition of the subclass of MDP models they can tackle. The first contribution of this paper to define this class, naming it "Action Decomposable MDP". The second contribution is to give a new MDP Linear Programming formulation using "Action Decomposability" that contributes to extend MDP solution techniques. Finally we give some examples of application of this framework and give numeric experiments showing the interest of using the action decomposition properties.

Domaines: Recherche opérationnelle [math.OC]

Ariel Waserhole, Vincent Jost, Jean-Philippe Gayon. Action Decomposable MDP A Linear Programming formulation for queuing network problems. 2012. ⟨hal-00727039v1⟩