Variance-Aware Regret Bounds for Undiscounted Reinforcement Learning in MDPs

Mohammad Talebi 1 Odalric-Ambrym Maillard 2
2 SEQUEL - Sequential Learning
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189
Abstract : The problem of reinforcement learning in an unknown and discrete Markov Decision Process (MDP) under the average-reward criterion is considered, when the learner interacts with the system in a single stream of observations, starting from an initial state without any reset. We revisit the minimax lower bound for that problem by making appear the local variance of the bias function in place of the diameter of the MDP. Furthermore, we provide a novel analysis of the KL-Ucrl algorithm establishing a high-probability regret bound scaling as O S s,a V s,a T for this algorithm for ergodic MDPs, where S denotes the number of states and where V s,a is the variance of the bias function with respect to the next-state distribution following action a in state s. The resulting bound improves upon the best previously known regret bound O(DS √ AT) for that algorithm, where A and D respectively denote the maximum number of actions (per state) and the diameter of MDP. We finally compare the leading terms of the two bounds in some benchmark MDPs indicating that the derived bound can provide an order of magnitude improvement in some cases. Our analysis leverages novel variations of the transportation lemma combined with Kullback-Leibler concentration inequalities, that we believe to be of independent interest.
Type de document :
Article dans une revue
Journal of Machine Learning Research, Journal of Machine Learning Research, In press, pp.1-36
Liste complète des métadonnées

Littérature citée [3 références]  Voir  Masquer  Télécharger

https://hal.archives-ouvertes.fr/hal-01737142
Contributeur : Odalric-Ambrym Maillard <>
Soumis le : lundi 19 mars 2018 - 11:49:25
Dernière modification le : mardi 3 juillet 2018 - 11:38:42
Document(s) archivé(s) le : mardi 11 septembre 2018 - 08:53:49

Fichier

ALT18_CameraReady.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01737142, version 1

Collections

Citation

Mohammad Talebi, Odalric-Ambrym Maillard. Variance-Aware Regret Bounds for Undiscounted Reinforcement Learning in MDPs. Journal of Machine Learning Research, Journal of Machine Learning Research, In press, pp.1-36. 〈hal-01737142〉

Partager

Métriques

Consultations de la notice

313

Téléchargements de fichiers

75