Optimistic planning for sparsely stochastic systems

Lucian Busoniu 1 Rémi Munos 2 Bart de Schutter 1 Robert Babuska 1
2 SEQUEL - Sequential Learning
LIFL - Laboratoire d'Informatique Fondamentale de Lille, Inria Lille - Nord Europe, LAGIS - Laboratoire d'Automatique, Génie Informatique et Signal
Abstract : We propose an online planning algorithm for finite action, sparsely stochastic Markov decision processes, in which the random state transitions can only end up in a small number of possible next states. The algorithm builds a planning tree by iteratively expanding states, where each expansion exploits sparsity to add all possible successor states. Each state to expand is actively chosen to improve the knowledge about action quality, and this allows the algorithm to return a good action after a strictly limited number of expansions. More specifically, the active selection method is optimistic in that it chooses the most promising states first, so the novel algorithm is called optimistic planning for sparsely stochastic systems. We note that the new algorithm can also be seen as model-predictive (receding-horizon) control. The algorithm obtains promising numerical results, including the successful online control of a simulated HIV infection with stochastic drug effectiveness.
Document type :
Conference papers
Complete list of metadatas

Cited literature [22 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-00830125
Contributor : Rémi Munos <>
Submitted on : Tuesday, June 4, 2013 - 3:49:46 PM
Last modification on : Thursday, February 21, 2019 - 10:52:49 AM
Long-term archiving on : Thursday, September 5, 2013 - 4:22:29 AM

File

adprl2011.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-00830125, version 1

Collections

Citation

Lucian Busoniu, Rémi Munos, Bart de Schutter, Robert Babuska. Optimistic planning for sparsely stochastic systems. IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning, 2011, paris, France. pp.48-55. ⟨hal-00830125⟩

Share

Metrics

Record views

270

Files downloads

259