Limits of Multi-Discounted Markov Decision Processes

Hugo Gimbert (1) , Wieslaw Zielonka (2)

1 LaBRI - Laboratoire Bordelais de Recherche en Informatique
2 LIAFA - Laboratoire d'informatique Algorithmique : Fondements et Applications

Markov decision processes (MDPs) are controllable discrete event systems with stochastic transitions. The payoff received by the controller can be evaluated in different ways, depending on the payoff function the MDP is equipped with. For example a \emph{mean--payoff} function evaluates average performance, whereas a \emph{discounted} payoff function gives more weights to earlier performance by means of a discount factor. Another well--known example is the \emph{parity} payoff function which is used to encode logical specifications~\cite{dagstuhl}. Surprisingly, parity and mean--payoff MDPs share two non--trivial properties: they both have pure stationary optimal strategies~\cite{CourYan:1990,neyman} and they both are approximable by discounted MDPs with multiple discount factors (multi--discounted MDPs)~\cite{dealf:2003,neyman}. In this paper we unify and generalize these results. We introduce a new class of payoff functions called the priority weighted payoff functions, which are generalization of both parity and mean--payoff functions. We prove that priority weighted MDPs admit optimal strategies that are pure and stationary, and that the priority weighted value of an MDP is the limit of the multi--discounted value when discount factors tend to $0$ simultaneously at various speeds.

Domaines: Informatique et théorie des jeux [cs.GT]

Hugo Gimbert, Wieslaw Zielonka. Limits of Multi-Discounted Markov Decision Processes. LICS 07, Jul 2007, Wroclaw, Poland. pp.89-98, ⟨10.1109/LICS.2007.28⟩. ⟨hal-00140148⟩