A distance for probability spaces, and long-term values in Markov Decision Processes and Repeated Games

Jérôme Renault (1) , Xavier Venel (1)

1 GREMAQ - Groupe de recherche en économie mathématique et quantitative

Given a finite set $K$, we denote by $X=\Delta(K)$ the set of probabilities on $K$ and by $Z=\Delta_f(X)$ the set of Borel probabilities on $X$ with finite support. Studying a Markov Decision Process with partial information on $K$ naturally leads to a Markov Decision Process with full information on $X$. We introduce a new metric $d_*$ on $Z$ such that the transitions become $1$-Lipschitz from $(X, \|.\|_1)$ to $(Z,d_*)$. In the first part of the article, we define and prove several properties of the metric $d_*$. Especially, $d_*$ satisfies a Kantorovich-Rubinstein type duality formula and can be characterized by using disintegrations. In the second part, we characterize the limit values in several classes of ''compact non expansive" Markov Decision Processes. In particular we use the metric $d_*$ to characterize the limit value in Partial Observation MDP with finitely many states and in Repeated Games with an informed controller with finite sets of states and actions. Moreover in each case we can prove the existence of a generalized notion of uniform value where we consider not only the Cesàro mean when the number of stages is large enough but any evaluation function $\theta \in \Delta(\N^*)$ when the impatience $I(\theta)=\sum_{t\geq 1} |\theta_{t+1}-\theta_t|$ is small enough.

Mots clés: Markov Decision Process; gambling houses; POMDP; Repeated Games; distance for belief spaces; Kantorovich-Rubinstein duality; limit value; uniform value; general values; characterization of the value.; characterization of the value

Domaines: Optimisation et contrôle [math.OC]

Jérôme Renault, Xavier Venel. A distance for probability spaces, and long-term values in Markov Decision Processes and Repeated Games. 2012. ⟨hal-00674998⟩