Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
We continuously face the dilemma of choosing between actions that gather new information or actions that exploit existing knowledge. This "exploration-exploitation" trade-off depends on the environment: stability favors exploiting knowledge to maximize gains; volatility favors exploring new options and discovering new outcomes. Here we set out to reconcile recent evidence for dopamine's involvement in the exploration-exploitation trade-off with the existing evidence for basal ganglia control of action selection, by testing the hypothesis that tonic dopamine in the striatum, the basal ganglia's input nucleus, sets the current exploration-exploitation trade-off. We first advance the idea of interpreting the basal ganglia output as a probability distribution function for action selection. Using computational models of the full basal ganglia circuit, we showed that, under this interpretation, the actions of dopamine within the striatum change the basal ganglia's output to favor the level of exploration or exploitation encoded in the probability distribution. We also found that our models predict striatal dopamine controls the exploration-exploitation trade-off if we instead read-out the probability distribution from the target nuclei of the basal ganglia, where their inhibitory input shapes the cortical input to these nuclei. Finally, by integrating the basal ganglia within a reinforcement learning model, we showed how dopamine's effect on the exploration-exploitation trade-off could be measurable in a forced two-choice task. These simulations also showed how tonic dopamine can appear to affect learning while only directly altering the trade-off. Thus, our models support the hypothesis that changes in tonic dopamine within the striatum can alter the exploration-exploitation trade-off by modulating the output of the basal ganglia.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 2×
“Formal models of reinforcement learning posit a conceptual separation between a system that learns the value of each action through reinforcement and a system that transforms those values into a probability distribution for action selection (Sutton and Barto, 1998).”
BackgroundReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 2×
“(We emphasize that all results here were thus obtained by modeling dopamine’s tonic effect on excitability in the striatum, and not the proposed role of phasic dopamine signals in reinforcement learning; Schultz et al., 1997).”
BackgroundFunctional architecture of basal ganglia circuits: neural substrates of parallel processing
Alexander, G. E., & Crutcher, M. D. (1990) · Trends in Neurosciences · doi:10.1016/0166-2236(90)90107-l · Cited by 4,131 · cited 3×
“However, it is not clear how many channels participate in the basal ganglia microcircuit, illustrated in Figure 2B, that underpins the action selection computation (Alexander and Crutcher, 1990; Middleton and Strick, 2000).”
BackgroundD 1 and D 2 Dopamine Receptor-regulated Gene Expression of Striatonigral and Striatopallidal Neurons
Gerfen, C. R., Engber, T. M., Mahan, L. C., Susel, Z., Chase, T. N., Monsma, F. J., & Sibley, D. R. (1990) · Science · doi:10.1126/science.2147780 · Cited by 2,967 · cited 1×
“…and lesion studies support this functional split into D1- and D2-dominant projection neurons and, further, that the D1-dominant neurons project to SNr, and the D2-dominant neurons project to GP (Gerfen et al., 1990; Surmeier et al., 2007; Matamales et al., 2009; Humphries and Prescott, 2010).”
BackgroundTHE BASAL GANGLIA: FOCUSED SELECTION AND INHIBITION OF COMPETING MOTOR PROGRAMS
MINK, J. W. (1996) · Progress in Neurobiology · doi:10.1016/s0301-0082(96)00042-1 · Cited by 2,486 · cited 2×
“A fundamental assumption of basal ganglia theories is that reduction of its tonic inhibitory output is the signal for selection (Mink, 1996; Redgrave et al., 1999; Hikosaka et al., 2000).”
BackgroundCortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224 · cited 2×
“Prefrontal cortical activity in humans (Daw et al., 2006; Behrens et al., 2007) and primates (Khamassi et al., 2011) appears to contribute to the regulation of exploratory behaviors based on tracking the stability of the environment.”
BackgroundLearning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053 · cited 3×
“Prefrontal cortical activity in humans (Daw et al., 2006; Behrens et al., 2007) and primates (Khamassi et al., 2011) appears to contribute to the regulation of exploratory behaviors based on tracking the stability of the environment.”
BackgroundBy Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964 · cited 7×
“, 1999; Kimchi and Laubach, 2009; Humphries and Prescott, 2010), consisting of a repeated canonical circuit that instantiates a mechanism for selecting between competing inputs (Gurney et al., 2001a; Frank et al., 2004; Humphries et al., 2006; Leblois et al., 2006a; Girard et al., 2008).”
BackgroundMethodInfluentialBasal ganglia and cerebellar loops: motor and cognitive circuits
Middleton, F. (2000) · Brain Research Reviews · doi:10.1016/s0165-0173(99)00040-5 · Cited by 1,940 · cited 1×
“However, it is not clear how many channels participate in the basal ganglia microcircuit, illustrated in Figure 2B, that underpins the action selection computation (Alexander and Crutcher, 1990; Middleton and Strick, 2000).”
Background
Cited by
Active inference: a process theory
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., & Pezzulo, G. (2017) · Neural Computation · doi:10.1162/neco_a_00912 · Cited by 1,156
“…is also consistent with its multiplicative effect in equation 2.7, to nuance the selection among competing policies (Fiorillo et al., 2003; Frank, Scheres, & Sherman, 2007; Humphries, Wood, & Gurney, 2009; Humphries, Khamassi, & Gurney, 2012; Solway & Botvinick, 2012; Mannella & Baldassarre, 2015).”
BackgroundInfluentialAdolescence as a neurobiological critical period for the development of higher-order cognition
Larsen, B., & Luna, B. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.09.005 · Cited by 853
“, 1978; Le Moal and Simon, 1991) as well as computational simulations (Humphries et al., 2012).”
Mesolimbic dopamine signals the value of work
Hamid, A. A., Pettibone, J. R., Mabrouk, O. S., Hetrick, V. L., Schmidt, R., Vander Weele, C. M., Kennedy, R. T., Aragona, B. J., & Berke, J. D. (2016) · Nature Neuroscience · doi:10.1038/nn.4173 · Cited by 824
Active inference and epistemic value
Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T., & Pezzulo, G. (2015) · Cognitive Neuroscience · doi:10.1080/17588928.2015.1020053 · Cited by 731
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., ODoherty, J., & Pezzulo, G. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.022 · Cited by 674
Small-world human brain networks: Perspectives and challenges
Liao, X., Vasilakos, A. V., & He, Y. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.03.018 · Cited by 436
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236
Gruber, A. J., & McDonald, R. J. (2012) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2012.00050 · Cited by 173
Putting desire on a budget: dopamine and energy expenditure, reconciling reward and resources
Beeler, J. A., Frazier, C. R. M., & Zhuang, X. (2012) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2012.00049 · Cited by 136
“(Humphries et al., 2012) demonstrates, the precise operation of this function of dopamine is likely to be complex.”
BackgroundResultInfluentialA systematic review of impulse control disorders in Parkinson’s disease.
Callesen, M. B., Scheel-Krüger, J., Kringelbach, M. L., & Møller, A. (2013) · Journal of Parkinson s Disease · doi:10.3233/jpd-120165 · Cited by 130
“Current neuroanatomical studies strongly suggest that the modulatory influence on cognition via the nucleus accumbens appears to be mediated by its upstream neuronal looping through the ventral pallidum and thalamic projections to the frontal cortex [166, 200].”
BackgroundRole of nicotinic acetylcholine receptors in regulating dopamine neuron activity
Faure, P., Tolu, S., Valverde, S., & Naudé, J. (2014) · Neuroscience · doi:10.1016/j.neuroscience.2014.05.040 · Cited by 125
Dopamine, reward learning, and active inference
FitzGerald, T. H. B., Dolan, R. J., & Friston, K. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00136 · Cited by 112
“This coexists quite happily with alternative “bottom-up” approaches based on what is known about striatal anatomy and physiology (Frank, 2011; Humphries et al., 2012; Collins and Frank, 2014; Fiore et al., 2014).”
BackgroundClarke, H. F., Cardinal, R. N., Rygula, R., Hong, Y. T., Fryer, T. D., Sawiak, S. J., Ferrari, V., Cockcroft, G., Aigbirhio, F. I., Robbins, T. W., et al. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.0718-14.2014 · Cited by 54
“Our results are compatible with Humphries et al. (2012) theory that tonic striatal DA influences the trade-off between exploration and exploitation.”
BackgroundResultNicotinic alteration of decision-making
Naudé, J., Dongelmans, M., & Faure, P. (2015) · Neuropharmacology · doi:10.1016/j.neuropharm.2014.11.021 · Cited by 39
“TonicDA levels have beenproposed to control the exploration/exploitation trade-off (Beeler et al., 2010; Humphries et al., 2012).”
Computational psychopharmacology: a translational and pragmatic approach
Robbins, T. W., & Cardinal, R. N. (2019) · Psychopharmacology · doi:10.1007/s00213-019-05302-3 · Cited by 25
“From a computational perspective, exploration versus exploitation has been related to chang-es in tonic DA function in the basal ganglia (Humphries et al. 2012) and from genetic evidence in humans, exploration has been shown to be affected by polymorphisms of the enzyme catechol-O -methyl…”
BackgroundFrontostriatal contribution to the interplay of flexibility and stability in serial prediction
Trempler, I., Schiffer, A.-M., El-Sourani, N., Ahlheim, C., Fink, G. R., & Schubotz, R. I. (2017) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01040 · Cited by 18
“Furthermore, it has been proposed that tonic DA release balances transitions between different states by modulating the degree to which prior learning biases action selection (Humphries et al. 2012).”
BackgroundMethodTrempler, I., Bürkner, P.-C., El-Sourani, N., Binder, E., Reker, P., Fink, G. R., & Schubotz, R. I. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116674 · Cited by 6
“…and sets transition thresholds between flexible and stable states, favouring either bottom-up sensory input or top-down predictions (Beeler et al., 2010; Beeler et al., 2012; Humphries et al., 2012; Yu et al., 2013; Barter et al., 2015).”
Background
Metadata
- Authors
- Humphries, M.
- Venue
- Frontiers in Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2012
- Volume
- 6
- Pages
- 9-9
- ISSN
- 1662-453X (ISSN-L)
- OpenAlex
- W2014610415
- Cited by
- 206
- References
- 71
- FWCI
- 4.56
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.