Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia

Mark D. Humphries (1, 2) , Mehdi Khamassi (3, 4) , Kevin Gurney (5)

1 Group for Neural Theory [Paris]
2 Department of Psychology [Sheffield]
3 ISIR - Institut des Systèmes Intelligents et de Robotique
4 AMAC
5 Adaptive Behaviour Research Group

We continuously face the dilemma of choosing between actions that gather new information or actions that exploit existing knowledge. This "exploration-exploitation" trade-off depends on the environment: stability favours exploiting knowledge to maximise gains; volatility favours exploring new options and discovering new outcomes. Here we set out to reconcile recent evidence for dopamine's involvement in the exploration-exploitation trade-off with the existing evidence for basal ganglia control of action selection, by testing the hypothesis that the tonic dopamine in the striatum, the basal ganglia's input nucleus, sets the current exploration-exploitation tradeoff. We first advanced the idea of interpreting the basal ganglia output as a probability distribution function for action selection. Using computational models of the full basal ganglia circuit, we showed that, under this interpretation, the actions of dopamine within the striatum change the basal ganglia's output to favour the level of exploration or exploitation encoded in the probability distribution. We also found that our models predict striatal dopamine controls the exploration-exploitation trade-off if we instead read out the probability distribution from the target nuclei of the basal ganglia, where their inhibitory input shapes the cortical input to these nuclei. Finally, by integrating the basal ganglia within a reinforcement learning model, we showed ho dopamine's effect on the exploration-exploitation trade-off could be measurable in a forced two-choice task. These simulations also showed how tonic dopamine can appear to affect learning while only directly altering the trade-off. Thus, our models support the hypothesis that changes in tonic dopamine within the striatum can alter the exploration-exploitation trade-off by modulating the output of the basal ganglia.

Mots clés: decision making; reward; reinforcement learning; meta-parameters; uncertainty

Domaines: Neurosciences [q-bio.NC]

Mark D. Humphries, Mehdi Khamassi, Kevin Gurney. Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia. Frontiers in Neuroscience, 2012, 6 (9), pp.1-14. ⟨10.3389/fnins.2012.00009⟩. ⟨hal-00688928⟩