Influence of the context of a Reinforcement Learning Technique on the learning performances - A case study

Frédéric Davesne (1) , Claude Barret (2)

1 LPPA - Laboratoire de Physiologie de la Perception et de l'Action
2 LSC - Laboratoire Systèmes Complexes

Statistical learning methods select the model that statistically best fit the data, given a cost function. In this case, learning means finding out a set of internal parameters of the model that minimize (or maximize) the cost function. As an example of such a procedure, reinforcement learning techniques (RLT) may be used in robotics to find the best mapping between sensors and effectors to achieve a goal. A lot of practical issues have been already pointed out to apply RLT in real robotics, and some solutions have been investigated. However, an underlying issue, which is critical for the reliability of the task accomplished by the robot, is the adequacy of the a priori knowledge (design of the states, value of the temperature parameter) used by the RLT with the physical properties of the robot, in order to achieve the goal defined by the experimenter. We call it Context Quality (CQ). Some work has pointed out that bad CQ may lead to poor learning results, but CQ in itself was not really quantified. In this paper, we suggest that the entropy measure taken from the Information Theory is well suited to quantify CQ and to predict the quality of the results obtained by the learning process. Taking the Cart Pole Balancing benchmark, we show that there exists a strong relation between our CQ measure and the performance of the RLT, that is to say the viability duration of the cart/pole. In particular, we investigate the influence of the noisiness of the inputs and the design of the states. In the first case, we show that CQ is linked to performance of recognition of the input states by the system. Moreover,we propose an statistical explanatory model of the influence of CQ on the RLT performance.

Keywords: Machine Learning; Context Quality; State Design Testing; Shannon Entropy

Domains: Machine Learning [cs.LG]

Frédéric Davesne, Claude Barret. Influence of the context of a Reinforcement Learning Technique on the learning performances - A case study. Artificial Intelligence and Applications (AIA 2003), Sep 2003, Benalmádena, Spain. elec. proc. ⟨hal-00377120⟩