Changes in corticostriatal connectivity during reinforcement learning in humans
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Many computational models assume that reinforcement learning relies on changes in synaptic efficacy between cortical regions representing stimuli and striatal regions involved in response selection, but this assumption has thus far lacked empirical support in humans. We recorded hemodynamic signals with fMRI while participants navigated a virtual maze to find hidden rewards. We fitted a reinforcement‐learning algorithm to participants' choice behavior and evaluated the neural activity and the changes in functional connectivity related to trial‐by‐trial learning variables. Activity in the posterior putamen during choice periods increased progressively during learning. Furthermore, the functional connections between the sensorimotor cortex and the posterior putamen strengthened progressively as participants learned the task. These changes in corticostriatal connectivity differentiated participants who learned the task from those who did not. These findings provide a direct link between changes in corticostriatal connectivity and learning, thereby supporting a central assumption common to several computational models of reinforcement learning. Hum Brain Mapp 36:793–803, 2015. © 2014 Wiley Periodicals, Inc.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · cited 3×
“…S-R learning [McDonald and Hong, 2004; Packard et al., 1989], as well as the remarkable fit between our findings and the predictions of “modelfree RL” [Daw et al., 2005] models such as the actor-critic (which work by S-R learning), provide some support our interpretation in terms of S-R learning.”
BackgroundResultInfluentialThe role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 1×
“More generally, our findings support the parallels between the dorsal striatum and the “actor” in the actor-critic model of action selection [Barto, 1995; O’Doherty et al., 2004].”
ResultA neostriatal habit learning system in humans
Knowlton, B. J., Mangels, J. A., & Squire, L. R. (1996) · Science · doi:10.1126/science.273.5280.1399 · Cited by 1,777
Balleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775
Learning and Memory Functions of the Basal Ganglia
Packard, M. G., & Knowlton, B. J. (2002) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.25.112701.142937 · Cited by 1,717 · cited 2×
“We used fMRI based on computational modeling (i.e., computational fMRI) [O’Doherty et al., 2007] to study changes in corticostriatal connectivity as human participants performed a task analogous to the “win-stay” version of a radial-arm maze task [Packard et al., 1989; Packard and Knowlton, 2002].”
MethodSelf-citationHaber, S. N., Fudge, J. L., & McFarland, N. R. (2000) · The Journal of Neuroscience · doi:10.1523/jneurosci.20-06-02369.2000 · Cited by 1,610 · cited 4×
“Consistent with the known anatomy of the CBGTC sensorimotor loop [Draganski et al., 2008; Haber et al., 2000; Lehericy et al., 2004], we found that the posterolateral putamen is selectively connected to the premotor cortex-SMA, lateral thalamus, and sensory cortex.”
BackgroundInfluentialDopamine-dependent prediction errors underpin reward-seeking behaviour in humans
Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006) · Nature · doi:10.1038/nature05051 · Cited by 1,584
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · cited 1×
“Such coding of neutral outcomes relative to overall task expectations has been previously demonstrated [O’Doherty et al., 2003].”
BackgroundRapid formation and selective stabilization of synapses for enduring motor memories
Xu, T., Yu, X., Perlik, A. J., Tobin, W. F., Zweig, J. A., Tennant, K., Jones, T., & Zuo, Y. (2009) · Nature · doi:10.1038/nature08389 · Cited by 1,193 · cited 2×
“Because rapid changes in synaptic efficacy accompany learning-related plasticity [Xu et al., 2009], learning likely modifies the functional coupling between presynaptic and postsynaptic neurons—a phenomenon that can be studied using functional-connectivity tools in fMRI.”
BackgroundUnderstanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis
Glimcher, P. W. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1014269108 · Cited by 955 · cited 1×
“Computational models form the backbone of our current theoretical understanding of reinforcement learning (RL) in humans and other animals [Glimcher, 2011; Maia and Frank, 2011].”
BackgroundFrank, M. J. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/0898929052880093 · Cited by 925
A triple dissociation of memory systems: Hippocampus, amygdala, and dorsal striatum.
McDonald, R. J., & White, N. M. (1993) · Behavioral Neuroscience · doi:10.1037/0735-7044.107.1.3 · Cited by 885
A specific role for posterior dorsolateral striatum in human habit learning
Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06796.x · Cited by 769 · cited 2×
“, 1996] to habit formation, showing that putaminal activation at the onset of task blocks increased over the course of each training day and across days of training [Tricomi et al., 2009], and to valuation following extensive training [Wunderlich et al.”
BackgroundPostuma, R. B., & Dagher, A. (2006) · Cerebral Cortex · doi:10.1093/cercor/bhj088 · Cited by 579 · cited 2×
“Consistent with the organization of CBGTC loops [Draganski et al., 2008; Haber et al., 2000; Lehericy et al., 2004; Postuma and Dagher, 2006] and predictions from RL models, in learners, the premotor-motor, somatosensory, visual, and superior parietal cortices increased their connectivity with the…”
BackgroundBrain mediators of cardiovascular responses to social threat
Wager, T. D., Waugh, C. E., Lindquist, M., Noll, D. C., Fredrickson, B. L., & Taylor, S. F. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.05.043 · Cited by 402 · cited 1×
“A mediation analysis [Wager et al., 2009] based on the choice-related activations within the posterolateral putamen, premotor cortex-SMA, and lateral thalamus ROIs in learners showed that the thalamus fully mediated the functional connections between putamen and premotor cortex-SMA (Zab 5 4.61, P 5…”
MethodInfluentialSchönberg, T., Daw, N. D., Joel, D., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.2496-07.2007 · Cited by 390
de Wit, S., Watson, P., Harsay, H. A., Cohen, M. X., van de Vijver, I., & Ridderinkhof, K. R. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.1088-12.2012 · Cited by 373 · cited 1×
“Other studies have instead focused on interindividual differences in structural connections in relation to habitual tendencies [de Wit et al., 2012].”
BackgroundMapping value based planning and extensively trained choice in the human brain
Wunderlich, K., Dayan, P., & Dolan, R. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3068 · Cited by 310 · cited 5×
“, 2009], and to valuation following extensive training [Wunderlich et al., 2012].”
BackgroundMethodInfluentialA Dual Role for Prediction Error in Associative Learning
den Ouden, H. E. M., Friston, K. J., Daw, N. D., McIntosh, A. R., & Stephan, K. E. (2009) · Cerebral Cortex · doi:10.1093/cercor/bhn161 · Cited by 306 · cited 1×
“…assess changes in striatal connectivity as participants learn the relational association between pairs of stimuli (S-S learning) [den Ouden et al., 2010; den Ouden et al., 2009; Wimmer et al., 2012], but none have assessed such changes as a function of S-R learning or other forms of RL.”
BackgroundStriatal Prediction Error Modulates Cortical Coupling
den Ouden, H. E. M., Daunizeau, J., Roiser, J., Friston, K. J., & Stephan, K. E. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4458-09.2010 · Cited by 304 · cited 1×
“…to specifically assess changes in striatal connectivity as participants learn the relational association between pairs of stimuli (S-S learning) [den Ouden et al., 2010; den Ouden et al., 2009; Wimmer et al., 2012], but none have assessed such changes as a function of S-R learning or other forms…”
BackgroundGuitart-Masip, M., Fuentemilla, L., Bach, D. R., Huys, Q. J. M., Dayan, P., Dolan, R. J., & Duzel, E. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.6376-10.2011 · Cited by 216 · cited 1×
“Finally, independent manipulation of actions and rewards in a recent study revealed anticipatory signals in the putamen that represent action and not state value [Guitart-Masip et al., 2011].”
BackgroundGeneralization of value in reinforcement learning by humans
Wimmer, G. E., Daw, N. D., & Shohamy, D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08017.x · Cited by 111 · cited 1×
“…assess changes in striatal connectivity as participants learn the relational association between pairs of stimuli (S-S learning) [den Ouden et al., 2010; den Ouden et al., 2009; Wimmer et al., 2012], but none have assessed such changes as a function of S-R learning or other forms of RL.”
Background
Cited by
Computational psychiatry as a bridge from neuroscience to clinical applications
Huys, Q. J. M., Maia, T. V., & Frank, M. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4238 · Cited by 1,002
“Resource-light MF systems conversely learn values by iteratively updating them with prediction errors through experience, and are thought to capture habits and rely on sensorimotor CSTC loops 98,105–108 .”
BackgroundSelf-citationLearning, reward, and decision making
O'Doherty, J. P., Cockburn, J., & Pauli, W. M. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-010416-044216 · Cited by 475
“Others have reported putative model-free value signals in the posterior putamen (Horga et al. 2015).”
BackgroundEvolutionary diversity as a catalyst for biological discovery
JOHNSON, Z. V., & YOUNG, L. J. (2018) · Integrative Zoology · doi:10.1111/1749-4877.12339 · Cited by 25
Metadata
- Authors
- Horga, G., Maia, T. V., Marsh, R., Hao, X., Xu, D., Duan, Y., Tau, G. Z., Graniello, B., Wang, Z., Kangarlu, A., Martinez, D., Packard, M. G., & Peterson, B. S.
- Venue
- Human Brain Mapping
- Publisher
- Wiley
- Published
- 2014-11-12
- Volume
- 36
- Issue
- 2
- Pages
- 793-803
- ISSN
- 1065-9471 (ISSN-L)
- 1097-0193
- OpenAlex
- W1580587320
- Cited by
- 43
- References
- 23
- FWCI
- 2.21
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.