Dopamine-Mediated Learning and Switching in Cortico-Striatal Circuit Explain Behavioral Changes in Reinforcement Learning
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
The basal ganglia are thought to play a crucial role in reinforcement learning. Central to the learning mechanism are dopamine (DA) D1 and D2 receptors located in the cortico-striatal synapses. However, it is still unclear how this DA-mediated synaptic plasticity is deployed and coordinated during reward-contingent behavioral changes. Here we propose a computational model of reinforcement learning that uses different thresholds of D1- and D2-mediated synaptic plasticity which are antagonized by DA-independent synaptic plasticity. A phasic increase in DA release caused by a larger-than-expected reward induces long-term potentiation (LTP) in the direct pathway, whereas a phasic decrease in DA release caused by a smaller-than-expected reward induces a cessation of long-term depression, leading to LTP in the indirect pathway. This learning mechanism can explain the robust behavioral adaptation observed in a location-reward-value-association task where the animal makes shorter latency saccades to reward locations. The changes in saccade latency become quicker as the monkey becomes more experienced. This behavior can be explained by a switching mechanism which activates the cortico-striatal circuit selectively. Our model also shows how D1- or D2-receptor blocking experiments affect selectively either reward or no-reward trials. The proposed mechanisms also explain the behavioral changes in Parkinson's disease.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
AGEING AND PARKINSON'S DISEASE: SUBSTANTIA NIGRA REGIONAL SELECTIVITY
FEARNLEY, J. M., & LEES, A. J. (1991) · Brain · doi:10.1093/brain/114.5.2283 · Cited by 3,529
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 1×
“It has also been shown that dopamine (DA) plays a crucial role in learning in the BG. Phasic DA signals, in particular, have been hypothesized to cause reinforcement learning (Montague et al., 1996; Schultz et al., 1997; Schultz, 2007).”
BackgroundBy Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Regulation of parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry
Kravitz, A. V., Freeze, B. S., Parker, P. R. L., Kay, K., Thwin, M. T., Deisseroth, K., & Kreitzer, A. C. (2010) · Nature · doi:10.1038/nature09159 · Cited by 1,885 · cited 2×
“In the BG, there are two opposing pathways: (1) direct pathway which facilitates movement initiation and is under the control of D1 DA receptors, and (2) indirect pathway which suppresses movement initiation and is under the control of D2 DA receptors (Kravitz et al., 2010).”
BackgroundBehavioral Theories and the Neurophysiology of Reward
Schultz, W. (2006) · Annual Review of Psychology · doi:10.1146/annurev.psych.56.091103.070229 · Cited by 1,637 · cited 1×
“It is also known that many neurons in the BG are sensitive to expectation of reward (see Hikosaka et al., 2006; Schultz, 2006, for review).”
BackgroundLateral habenula as a source of negative reward signals in dopamine neurons
Matsumoto, M., & Hikosaka, O. (2007) · Nature · doi:10.1038/nature05860 · Cited by 1,292 · cited 5×
“We also have an extensive set of data that indicates how DA neurons change their activity during the 1DR task (e.g., Kawagoe et al., 2004; Matsumoto and Hikosaka, 2007; Bromberg-Martin et al., 2010), which is a pre-requisite for making a computational model of reinforcement learning.”
BackgroundSelf-citationSurmeier, D. J., Ding, J., Day, M., Wang, Z., & Shen, W. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.03.008 · Cited by 1,203 · cited 1×
“The mechanisms of the synaptic plasticity in the BG have been studied extensively, yet there are conflicting experimental results (for reference, see Calabresi et al., 2007; Surmeier et al., 2007).”
ResultDichotomous dopaminergic control of striatal synaptic plasticity
Shen, W., Flajolet, M., Greengard, P., & Surmeier, D. J. (2008) · Science · doi:10.1126/science.1160575 · Cited by 1,190
Schultz, W. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.03.007 · Cited by 1,028
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939
A cellular mechanism of reward-related learning
Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001) · Nature · doi:10.1038/35092560 · Cited by 879 · cited 3×
“…| Volume 5 | Article 15 | 1 Original research article published: 21 March 2011 doi: 10.3389/fnbeh.2011.00015 promotes long-term potentiation (LTP; Reynolds et al., 2001; Calabresi et al., 2007) whereas D2 receptor-mediated DA signaling induces long-term depression (LTD; Gerdeman et al., 2002;…”
BackgroundInfluentialDopamine-mediated regulation of corticostriatal synaptic plasticity
Calabresi, P., Picconi, B., Tozzi, A., & Di Filippo, M. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.03.001 · Cited by 865
Loss of bidirectional striatal synaptic plasticity in L-DOPA-induced dyskinesia
Picconi, B., Centonze, D., Håkansson, K., Bernardi, G., Greengard, P., Fisone, G., Cenci, M. A., & Calabresi, P. (2003) · Nature Neuroscience · doi:10.1038/nn1040 · Cited by 821 · cited 9×
“(Picconi et al., 2003) showed that LTD (or synaptic depotentiation) seen in control animals was absent in the l-DOPA treated animals that had too much phospho[Thr34]-DARPP-32, an inhibitor of protein phosphatase.”
BackgroundInfluentialDynamic reorganization of striatal circuits during the acquisition and consolidation of a skill
Yin, H. H., Mulcare, S. P., Hilário, M. R. F., Clouse, E., Holloway, T., Davis, M. I., Hansson, A. C., Lovinger, D. M., & Costa, R. M. (2009) · Nature Neuroscience · doi:10.1038/nn.2261 · Cited by 818
Endocannabinoid-mediated rescue of striatal LTD and motor deficits in Parkinson's disease models
Kreitzer, A. C., & Malenka, R. C. (2007) · Nature · doi:10.1038/nature05506 · Cited by 735 · cited 3×
“It is shown that DA-mediated D1 receptor signaling promotes LTP (Reynolds et al., 2001; Calabresi et al., 2007) whereas D2 signaling induces LTD (Gerdeman et al., 2002; Kreitzer and Malenka, 2007).”
BackgroundInfluentialCoincident but Distinct Messages of Midbrain Dopamine and Striatal Tonically Active Neurons
Morris, G., Arkadir, D., Nevet, A., Vaadia, E., & Bergman, H. (2004) · Neuron · doi:10.1016/j.neuron.2004.06.012 · Cited by 574
Cortico–basal ganglia circuit mechanism for a decision threshold in reaction time tasks
Lo, C.-C., & Wang, X.-J. (2006) · Nature Neuroscience · doi:10.1038/nn1722 · Cited by 505 · cited 1×
“Our current model is focused on modulation of saccades by reward-oriented biases rather than by such taskdependent speed–accuracy tradeoffs, which might be implemented by different BG circuit mechanisms (Lo and Wang, 2006).”
MethodBasal ganglia orient eyes to reward
Hikosaka, O., Nakamura, K., & Nakahara, H. (2006) · Journal of Neurophysiology · doi:10.1152/jn.00458.2005 · Cited by 377 · cited 4×
“It is postulated that the basal ganglia (BG) contribute to this kind of reinforcement learning (see Hikosaka et al., 2006 for a review).”
BackgroundSelf-citationDifferential Corticostriatal Plasticity during Fast and Slow Motor Skill Learning in Mice
Costa, R. M., Cohen, D., & Nicolelis, M. A. L. (2004) · Current Biology · doi:10.1016/j.cub.2004.06.053 · Cited by 373 · cited 2×
“…in most of the BG components including dorsal striatum (Hikosaka et al., 1989; Apicella et al., 1992; Kawagoe et al., 1998; Lauwereyns et al., 2002; Costa et al., 2004; Samejima et al., 2005; Oyama et al., 2010), ventral striatum (Schultz et al., 1992; Kalenscher et al., 2010), subthalamic nucleus…”
BackgroundThe globus pallidus sends reward-related signals to the lateral habenula
Hong, S., & Hikosaka, O. (2008) · Neuron · doi:10.1016/j.neuron.2008.09.035 · Cited by 346
Self-citationMechanisms underlying dopamine-mediated reward bias in compulsive behaviors
Voon, V., Pessiglione, M., Brezing, C., Gallea, C., Fernandez, H. H., Dolan, R. J., & Hallett, M. (2010) · Neuron · doi:10.1016/j.neuron.2009.12.027 · Cited by 280
Reward-Dependent Gain and Bias of Visual Responses in Primate Superior Colliculus
Ikeda, T., & Hikosaka, O. (2003) · Neuron · doi:10.1016/s0896-6273(03)00464-1 · Cited by 233 · cited 2×
“Indeed, SC neurons exhibit such pre-target ramp-up activity which is stronger when the contralateral saccade was rewarded than unrewarded (Ikeda and Hikosaka, 2003; Isoda and Hikosaka, 2008).”
BackgroundSelf-citationPizzagalli, D. A., Evins, A. E., Schetter, E. C., Frank, M. J., Pajtas, P. E., Santesso, D. L., & Culhane, M. (2008) · Psychopharmacology · doi:10.1007/s00213-007-0957-y · Cited by 219 · cited 1×
“This hypothesis has been supported by a recent study where suppression of phasic DA release by pharmacological manipulation impairs the acquisition of reward-related behavior in healthy human subjects (Pizzagalli et al., 2008).”
Background
Cited by
Neural Basis of Reinforcement Learning and Decision Making
Lee, D., Seo, H., & Jung, M. W. (2012) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062111-150512 · Cited by 546
The ins and outs of the striatum: Role in drug addiction
Yager, L. M., Garcia, A. F., Wunsch, A. M., & Ferguson, S. M. (2015) · Neuroscience · doi:10.1016/j.neuroscience.2015.06.033 · Cited by 417
Dopamine neurons encode performance error in singing birds
Gadagkar, V., Puzerey, P. A., Chen, R., Baird-Daniel, E., Farhang, A. R., & Goldberg, J. H. (2016) · Science · doi:10.1126/science.aah6837 · Cited by 268
Dopaminergic control of cognitive flexibility in humans and animals
Klanker, M., Feenstra, M., & Denys, D. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00201 · Cited by 209
The functional logic of corticostriatal connections
Shipp, S. (2017) · Brain Structure and Function · doi:10.1007/s00429-016-1250-9 · Cited by 118
“…the alternative output of SPNs to either the GPi/SNr or GPe that, in theory, enable the BG to fractionate a bid for action into positive and negative salience signals (Frank 2005; Hong and Hikosaka 2011; Schroll and Hamker 2013; Collins and Frank 2014; Baladron and Hamker 2015; Gurney et al. 2015).”
Background
Metadata
- Authors
- Hong, S., & Hikosaka, O.
- Venue
- Frontiers in Behavioral Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2011
- Volume
- 5
- Pages
- 15-15
- ISSN
- 1662-5153 (ISSN-L)
- OpenAlex
- W2013314169
- Cited by
- 83
- References
- 23
- FWCI
- 2.02
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.