The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Dissociable neural representations of reinforcement and belief prediction errors underlie strategic learning
Open accessAbstract
Decision-making in the presence of other competitive intelligent agents is fundamental for social and economic behavior. Such decisions require agents to behave strategically, where in addition to learning about the rewards and punishments available in the environment, they also need to anticipate and respond to actions of others competing for the same rewards. However, whereas we know much about strategic learning at both theoretical and behavioral levels, we know relatively little about the underlying neural mechanisms. Here, we show using a multi-strategy competitive learning paradigm that strategic choices can be characterized by extending the reinforcement learning (RL) framework to incorporate agents’ beliefs about the actions of their opponents. Furthermore, using this characterization to generate putative internal values, we used model-based functional magnetic resonance imaging to investigate neural computations underlying strategic learning. We found that the distinct notions of prediction errors derived from our computational model are processed in a partially overlapping but distinct set of brain regions. Specifically, we found that the RL prediction error was correlated with activity in the ventral striatum. In contrast, activity in the ventral striatum, as well as the rostral anterior cingulate (rACC), was correlated with a previously uncharacterized belief-based prediction error. Furthermore, activity in rACC reflected individual differences in degree of engagement in belief learning. These results suggest a model of strategic behavior where learning arises from interaction of dissociable reinforcement and belief-based inputs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, ventral striatum and medial prefrontal cortex signals encode information learned about strategic actions of competitors 72, 88, 90.”
From The social brain and reward: Social information processing in the human striatum (2014) · cites this paper as [90]
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Statistical parametric maps in functional imaging: A general linear approach
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. ‐., Frith, C. D., & Frackowiak, R. S. J. (1994) · Human Brain Mapping · doi:10.1002/hbm.460020402 · Cited by 9,688
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Cognitive and emotional influences in anterior cingulate cortex
Bush, G., Luu, P., & Posner, M. I. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01483-2 · Cited by 6,259
Evolutionary Games and Population Dynamics
Hofbauer, J., & Sigmund, K. (1998) · doi:10.1017/cbo9781139173179 · Cited by 5,502
Holroyd, C. B., & Coles, M. G. H. (2002) · Psychological Review · doi:10.1037/0033-295x.109.4.679 · Cited by 4,027
Meeting of minds: the medial frontal cortex and social cognition
Amodio, D. M., & Frith, C. D. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1884 · Cited by 3,782 · cited 3×
“Such a functional division is consistent with previous proposals in the mentalization literature (18) and may re fl ect differences in the type of information being learned.”
BackgroundResultBehavioral Game Theory: Experiments in Strategic Interaction
Camerer, C. F. (2003) · Cited by 3,722
Conflict monitoring and anterior cingulate cortex: an update
Botvinick, M. M., Cohen, J. D., & Carter, C. S. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.10.003 · Cited by 3,623
Cited by
Sorted by citations
The Anterior Cingulate Gyrus and Social Cognition: Tracking the Motivation of Others
Apps, M. A. J., Rushworth, M. F. S., & Chang, S. W. C. (2016) · Neuron · doi:10.1016/j.neuron.2016.04.018 · Cited by 559
“The responses in the ACCg to the value of behaviors for others and to PEs that were discussed in earlier sections approximate the signals predicted by the MoVE model (Lockwood et al., 2015; Apps and Ramnani, 2014; Zhu et al., 2012; Apps et al., 2012, 2013a, 2015a).”
InfluentialNeural Basis of Reinforcement Learning and Decision Making
Lee, D., Seo, H., & Jung, M. W. (2012) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062111-150512 · Cited by 581
Theory of Mind: A Neural Prediction Problem
Koster-Hale, J., & Saxe, R. (2013) · Neuron · doi:10.1016/j.neuron.2013.08.020 · Cited by 513
“…stimuli (e.g., smiling faces) as reward, or learning about reward based on observation or consideration of others’ experiences (Lin et al., 2012; Zhu et al., 2012; Zaki and Mitchell, 2011; Poore et al., 2012; Jones et al., 2011; Izuma et al., 2008; Chang and Sanfey, 2013; see Dunne and…”
Behavioral Variability through Stochastic Choice and Its Gating by Anterior Cingulate Cortex
Tervo, D. G. R., Proskurin, M., Manakov, M., Kabra, M., Vollmer, A., Branson, K. M., & Karpova, A. Y. (2014) · Cell · doi:10.1016/j.cell.2014.08.037 · Cited by 283
“Indeed, studies in primates support the idea that competition triggers model-based counterprediction rather than stochastic choice (Abe and Lee, 2011; Zhu et al., 2012).”
InfluentialThe social brain and reward: Social information processing in the human striatum
Bhanji, J. P., & Delgado, M. R. (2014) · WIREs Cognitive Science · doi:10.1002/wcs.1266 · Cited by 256
Metadata
- Authors
- Zhu, L., Mathewson, K. E., & Hsu, M.
- Affiliations
- Department of Economics and
- University of Alberta (CA)
- University of California, Berkeley (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2012-01-18
- Volume
- 109
- Issue
- 5
- Pages
- 1419-1424
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC3277161
- OpenAlex
- W2165958648
- Cited by
- 125
- References
- 45
- Citations by year
- 2026: 7 · 2025: 6 · 2024: 6 · 2023: 4 · 2022: 4 · 2021: 14 · 2020: 9 · 2019: 7 · 2018: 7 · 2017: 8 · 2016: 11 · 2015: 11 · 2014: 13 · 2013: 11 · 2012: 7
- FWCI
- 6.87
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- strategic learning, competitive decision-making, opponent modeling, belief learning, ventral striatum, rostral anterior cingulate cortex, model-based reinforcement learning, internal values, individual differences
- MeSH terms
- Decision Making, Humans, Learning, Motivation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
