Numerical Approximations for Discounted Continuous Time Markov Decision Processes

François Dufour (1, 2, 3) , Tomás Prieto-Rumeau (4)

1 Bordeaux INP - Institut Polytechnique de Bordeaux
2 CQFD - Quality control and dynamic reliability
3 IMB - Institut de Mathématiques de Bordeaux
4 UNED - Universidad Nacional de Educación a Distancia

This paper deals with a continuous-time Markov decision process M, with Borel state and action spaces, under the total expected discounted cost optimality criterion. By suitably approximating an underlying probability measure with a measure with finite support and by discretizing the action sets of the control model, we can construct a finite state and action space Markov decision process that approximates M and that can be solved explicitly. We can derive bounds on the approximation error of the optimal discounted cost function; such bounds are written in terms of Wasserstein and Hausdorff distances. We show a numerical application to a queueing problem.

Domaines: Optimisation et contrôle [math.OC]

François Dufour, Tomás Prieto-Rumeau. Numerical Approximations for Discounted Continuous Time Markov Decision Processes. George Yin; Qing Zhang. Modeling, Stochastic Control, Optimization, and Applications, Springer, pp.147-171, 2019, ⟨10.1007/978-3-030-25498-8_7⟩. ⟨hal-02414376⟩