The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Dissociable neural representations of reinforcement and belief prediction errors underlie strategic learning
Open accessAbstract
Decision-making in the presence of other competitive intelligent agents is fundamental for social and economic behavior. Such decisions require agents to behave strategically, where in addition to learning about the rewards and punishments available in the environment, they also need to anticipate and respond to actions of others competing for the same rewards. However, whereas we know much about strategic learning at both theoretical and behavioral levels, we know relatively little about the underlying neural mechanisms. Here, we show using a multi-strategy competitive learning paradigm that strategic choices can be characterized by extending the reinforcement learning (RL) framework to incorporate agents’ beliefs about the actions of their opponents. Furthermore, using this characterization to generate putative internal values, we used model-based functional magnetic resonance imaging to investigate neural computations underlying strategic learning. We found that the distinct notions of prediction errors derived from our computational model are processed in a partially overlapping but distinct set of brain regions. Specifically, we found that the RL prediction error was correlated with activity in the ventral striatum. In contrast, activity in the ventral striatum, as well as the rostral anterior cingulate (rACC), was correlated with a previously uncharacterized belief-based prediction error. Furthermore, activity in rACC reflected individual differences in degree of engagement in belief learning. These results suggest a model of strategic behavior where learning arises from interaction of dissociable reinforcement and belief-based inputs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, ventral striatum and medial prefrontal cortex signals encode information learned about strategic actions of competitors 72, 88, 90.”
From The social brain and reward: Social information processing in the human striatum (2014) · cites this paper as [90]
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Statistical parametric maps in functional imaging: A general linear approach
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. ‐., Frith, C. D., & Frackowiak, R. S. J. (1994) · Human Brain Mapping · doi:10.1002/hbm.460020402 · Cited by 9,688
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Cognitive and emotional influences in anterior cingulate cortex
Bush, G., Luu, P., & Posner, M. I. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01483-2 · Cited by 6,259
Evolutionary Games and Population Dynamics
Hofbauer, J., & Sigmund, K. (1998) · doi:10.1017/cbo9781139173179 · Cited by 5,502
Holroyd, C. B., & Coles, M. G. H. (2002) · Psychological Review · doi:10.1037/0033-295x.109.4.679 · Cited by 4,027
Meeting of minds: the medial frontal cortex and social cognition
Amodio, D. M., & Frith, C. D. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1884 · Cited by 3,782 · cited 3×
“Such a functional division is consistent with previous proposals in the mentalization literature (18) and may re fl ect differences in the type of information being learned.”
BackgroundResultBehavioral Game Theory: Experiments in Strategic Interaction
Camerer, C. F. (2003) · Cited by 3,722
Conflict monitoring and anterior cingulate cortex: an update
Botvinick, M. M., Cohen, J. D., & Carter, C. S. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.10.003 · Cited by 3,623
Cited by
Sorted by citations
Items 101–125 of 125
Hsu, M., & Chiong, W. (2016) · Springer eBooks · doi:10.1007/978-1-4939-3486-7_4 · Cited by 2
Self-citationA neural fingerprint of adaptive mentalizationPreprint
Bürgi, N., Aydogan, G., Konovalov, A., & Ruff, C. C. (2024) · doi:10.31234/osf.io/2fcpz · Cited by 2
Human–AI social cooperation: convergences, divergences, and challenges
Yao, S., Shan, J., Hu, L., & lu, X. (2025) · Chinese Science Bulletin (Chinese Version) · doi:10.1360/csb-2025-0662 · Cited by 1
Freedom through understanding: instructed knowledge shapes voluntary action choices
Ota, K., Christofilea, E., Charles, L., Jean Daunizeau, A., & Haggard, P. (2026) · Royal Society Open Science · doi:10.1098/rsos.250845 · Cited by 1
The Neural Mechanisms of Strategic Decision-Making
Konovalov, A. (2026) · Springer eBooks · doi:10.1007/978-3-032-02925-6_25 · Cited by 1
Layers of Social Meaning: Innate, Acquired, and Deliberative Levels of Referential Signal Processing
Yang, F., Liu, N., Jiang, Y., & Zhu, L. (2026) · Journal of Neuroscience · doi:10.1523/jneurosci.0535-25.2025 · Cited by 1
Self-citationUnderstanding the Brain as a Mentalizing Machine with a Symbol Grounding Function
Kaneko, T., Ninomiya, T., & Isoda, M. (2026) · Journal of Neuroscience · doi:10.1523/jneurosci.2445-24.2025 · Cited by 1
Culture as a Response to Uncertainty
Tobler, P. N., & Christopoulos, G. I. (2016) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199357376.013.6 · Cited by 1
Hsu, M., & Jenkins, A. C. (2015) · Nature Neuroscience · doi:10.1038/nn.4093 · Cited by 1
Self-citationNeuroeconomics: An Introduction
Wang, J. T. (2015)
Towards an Understanding of How People Build Mental Representations
Valadao, D. F. (2016) · UWSpace (University of Waterloo)
Neural Correlates of Strategic Decision-Making in the Primate Prefrontal Cortex
Seo, H., Kim, S., Cai, X., Abe, H., Donahue, C. H., & Lee, D. (2017) · Springer eBooks · doi:10.1007/978-4-431-56508-6_1
Valdespino, A. (2014) · VTechWorks (Virginia Tech)
A Hierarchical Bayesian Implementation of the Experience-Weighted Attraction Model
Zhang, Z., Chandra, S., Kayser, A. S., Hsu, M., & Warren, J. L. (2020) · Computational Psychiatry · doi:10.1162/cpsy_a_00028
“…to be estimated, we adopt the following simplifications in accordance with established practices in the literature (Ho, Camerer, & Chong, 2007; Zhu, Mathewson, & Hsu, 2012): (a) The V ijk ( 0 ) values are determined with a maximum-likelihood estimation procedure using data collected in the…”
BackgroundMethodInfluentialSelf-citationClassification of Theory of Mind tasks and their computational modelsPreprint
Rusch, T., & Gläscher, J. P. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/uf85z
Exploiting Predictability in Competitive Games: A Reinforcement Learning ApproachPreprint
Howatt, B. C., & Young, M. E. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4829206
Preference-based focal point and belief-based focal point: An fNIRS hyperscanning studyPreprint
zhou, X., & Li, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4940466
Zhang, F., Jin, Y., Yao, X., Liu, C., & Wang, X. (2025) · Behavioural Brain Research · doi:10.1016/j.bbr.2025.115859
Integration of contextual reward motivation with category representations
Seger, C. A., Braunlich, K., & Liu, Z. (2025) · NeuroImage · doi:10.1016/j.neuroimage.2025.121589
People Can Adaptively Exploit Model-free and Model-based Reinforcement Learning in Competitive Games
Howatt, B. C., & Young, M. E. (2026) · Quarterly Journal of Experimental Psychology · doi:10.1177/17470218261437211
“…predicts that participants’ choice behaviors would be best fit by a hybrid model-based RL algorithm over a pure model-free algorithm, a pure model-based algorithm, and a random-choice algorithm, consistent with previous literature (e.g., Hampton et al., 2008; Hunter et al., 2022; Zhu et al., 2012).”
Learning to adaptively cooperate through social interactions during childhood and adolescence
Zhang, R., Zhang, L., Hao, X.-M., Dreher, J., & Qu, C. (2026) · npj Science of Learning · doi:10.1038/s41539-026-00423-9 · Cited by 1
“In this model 15,16,35 , we assumed that participants tracked the opponent ’ s probability p t (with an initial p ¼ 0 : 5) of choosing Swerve .”
Implication of the locus coeruleus in effort processing : electrophysiological studies
Bornert, P. (2021) · theses.fr (ABES) · doi:10.70675/665d6bd7ze2f5z41c9z93f8z764be3c6d822
Neural Dynamics of Belief and Value Computations Guiding Strategic Social DecisionsPreprint
Konovalov, A., Hu, J., & Ruff, C. C. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.08.723744
Metadata
- Authors
- Zhu, L., Mathewson, K. E., & Hsu, M.
- Affiliations
- Department of Economics and
- University of Alberta (CA)
- University of California, Berkeley (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2012-01-18
- Volume
- 109
- Issue
- 5
- Pages
- 1419-1424
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC3277161
- OpenAlex
- W2165958648
- Cited by
- 125
- References
- 45
- Citations by year
- 2026: 7 · 2025: 6 · 2024: 6 · 2023: 4 · 2022: 4 · 2021: 14 · 2020: 9 · 2019: 7 · 2018: 7 · 2017: 8 · 2016: 11 · 2015: 11 · 2014: 13 · 2013: 11 · 2012: 7
- FWCI
- 6.87
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- strategic learning, competitive decision-making, opponent modeling, belief learning, ventral striatum, rostral anterior cingulate cortex, model-based reinforcement learning, internal values, individual differences
- MeSH terms
- Decision Making, Humans, Learning, Motivation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
