The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Dissociable neural representations of reinforcement and belief prediction errors underlie strategic learning
Open accessAbstract
Decision-making in the presence of other competitive intelligent agents is fundamental for social and economic behavior. Such decisions require agents to behave strategically, where in addition to learning about the rewards and punishments available in the environment, they also need to anticipate and respond to actions of others competing for the same rewards. However, whereas we know much about strategic learning at both theoretical and behavioral levels, we know relatively little about the underlying neural mechanisms. Here, we show using a multi-strategy competitive learning paradigm that strategic choices can be characterized by extending the reinforcement learning (RL) framework to incorporate agents’ beliefs about the actions of their opponents. Furthermore, using this characterization to generate putative internal values, we used model-based functional magnetic resonance imaging to investigate neural computations underlying strategic learning. We found that the distinct notions of prediction errors derived from our computational model are processed in a partially overlapping but distinct set of brain regions. Specifically, we found that the RL prediction error was correlated with activity in the ventral striatum. In contrast, activity in the ventral striatum, as well as the rostral anterior cingulate (rACC), was correlated with a previously uncharacterized belief-based prediction error. Furthermore, activity in rACC reflected individual differences in degree of engagement in belief learning. These results suggest a model of strategic behavior where learning arises from interaction of dissociable reinforcement and belief-based inputs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, ventral striatum and medial prefrontal cortex signals encode information learned about strategic actions of competitors 72, 88, 90.”
From The social brain and reward: Social information processing in the human striatum (2014) · cites this paper as [90]
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Statistical parametric maps in functional imaging: A general linear approach
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. ‐., Frith, C. D., & Frackowiak, R. S. J. (1994) · Human Brain Mapping · doi:10.1002/hbm.460020402 · Cited by 9,688
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Cognitive and emotional influences in anterior cingulate cortex
Bush, G., Luu, P., & Posner, M. I. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01483-2 · Cited by 6,259
Evolutionary Games and Population Dynamics
Hofbauer, J., & Sigmund, K. (1998) · doi:10.1017/cbo9781139173179 · Cited by 5,502
Holroyd, C. B., & Coles, M. G. H. (2002) · Psychological Review · doi:10.1037/0033-295x.109.4.679 · Cited by 4,027
Meeting of minds: the medial frontal cortex and social cognition
Amodio, D. M., & Frith, C. D. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1884 · Cited by 3,782 · cited 3×
“Such a functional division is consistent with previous proposals in the mentalization literature (18) and may re fl ect differences in the type of information being learned.”
BackgroundResultBehavioral Game Theory: Experiments in Strategic Interaction
Camerer, C. F. (2003) · Cited by 3,722
Conflict monitoring and anterior cingulate cortex: an update
Botvinick, M. M., Cohen, J. D., & Carter, C. S. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.10.003 · Cited by 3,623
Quantal Response Equilibria for Normal Form Games
McKelvey, R. D., & Palfrey, T. R. (1995) · Games and Economic Behavior · doi:10.1006/game.1995.1023 · Cited by 2,624
The Theory of Learning in Games
Drew Fudenberg, B., & Levine, D. K. (1998) · MIT Press Books · Cited by 2,455
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088
Pain and emotion interactions in subregions of the cingulate gyrus
Vogt, B. A. (2005) · Nature Reviews Neuroscience · doi:10.1038/nrn1704 · Cited by 1,810
Roth, A. E., & Erev, I. (1995) · Games and Economic Behavior · doi:10.1016/s0899-8256(05)80020-x · Cited by 1,685
Experience-weighted Attraction Learning in Normal Form Games
Camerer, C. F., & Ho, T. (1999) · Econometrica · doi:10.1111/1468-0262.00054 · Cited by 1,442
Getting to Know You: Reputation and Trust in a Two-Person Economic Exchange
King-Casas, B., Tomlin, D., Anen, C., Camerer, C. F., Quartz, S. R., & Montague, P. R. (2005) · Science · doi:10.1126/science.1108062 · Cited by 1,345
Behavioral game theory, experiments in strategic interaction
Canté, F. (2004) · Cuadernos de Economía · Cited by 1,265
The Neural Basis of Financial Risk Taking
Kuhnen, C. M., & Knutson, B. (2005) · Neuron · doi:10.1016/j.neuron.2005.08.008 · Cited by 1,182
Associative learning of social value
Behrens, T. E. J., Hunt, L. T., Woolrich, M. W., & Rushworth, M. F. S. (2008) · Nature · doi:10.1038/nature07538 · Cited by 925
Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum
McClure, S. M., Berns, G. S., & Montague, P. R. (2003) · Neuron · doi:10.1016/s0896-6273(03)00154-5 · Cited by 917
Optimized EPI for fMRI studies of the orbitofrontal cortex
Deichmann, R., Gottfried, J. A., Hutton, C., & Turner, R. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00073-9 · Cited by 821
Social neuroeconomics: the neural circuitry of social preferences
Fehr, E., & Camerer, C. F. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.09.002 · Cited by 779
Perceptions of moral character modulate the neural systems of reward during the trust game
Delgado, M. R., Frank, R. H., & Phelps, E. A. (2005) · Nature Neuroscience · doi:10.1038/nn1575 · Cited by 729
Prefrontal cortex and decision making in a mixed-strategy game
Barraclough, D. J., Conroy, M. L., & Lee, D. (2004) · Nature Neuroscience · doi:10.1038/nn1209 · Cited by 678
Regret and its avoidance: a neuroimaging study of choice behavior
Coricelli, G., Critchley, H., Joffily, M., O’Doherty, J. P., Angela Sirigu, B., & Dolan, R. J. (2005) · Nature Neuroscience · doi:10.1038/nn1514 · Cited by 673
Model‐Based fMRI and Its Application to Reward Learning and Decision Making
O’Doherty, J. P., Hampton, A. N., & Kim, H. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.022 · Cited by 525
Neural correlates of mentalizing-related computations during strategic interactions in humans
Hampton, A. N., Bossaerts, P. L., & O’Doherty, J. P. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0711099105 · Cited by 522
Dorris, M. C., & Glimcher, P. W. (2004) · Neuron · doi:10.1016/j.neuron.2004.09.009 · Cited by 440
The Computation of Social Behavior
Behrens, T. E. J., Hunt, L. T., & Rushworth, M. F. S. (2009) · Science · doi:10.1126/science.1169694 · Cited by 407 · cited 2×
“However, studies on reward-guided behavior in social settings have suggested that these computations build on the same brain structures associated with reward, including the striatum, rostral anterior cingulate (rACC), and orbitofrontal cortex (19).”
BackgroundNeural correlates of depth of strategic reasoning in medial prefrontal cortex
Coricelli, G., & Nagel, R. (2009) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.0807721106 · Cited by 385
Neural mechanisms of observational learning
Burke, C. J., Tobler, P. N., Baddeley, M. C., & Schultz, W. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1003111107 · Cited by 356
Neural signature of fictive learning signals in a sequential investment task
Lohrenz, T. M., McCabe, K., Camerer, C. F., & Montague, P. R. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0608842104 · Cited by 276
Fictive Reward Signals in the Anterior Cingulate Cortex
Hayden, B. Y., Pearson, J. M., & Platt, M. L. (2009) · Science · doi:10.1126/science.1168488 · Cited by 246
Separate Neural Mechanisms Underlie Choices and Strategic Preferences in Risky Decision Making
Venkatraman, V., Payne, J. W., Bettman, J. R., Luce, M. F., & Huettel, S. A. (2009) · Neuron · doi:10.1016/j.neuron.2009.04.007 · Cited by 222
Mixed-effects and fMRI studies
Friston, K. J., Stephan, K. E., Lund, T. E., Morcom, A. M., & Kiebel, S. J. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.08.055 · Cited by 204
Self-tuning experience weighted attraction learning in games
Ho, T., Camerer, C. F., & Chong, J.-K. (2006) · Journal of Economic Theory · doi:10.1016/j.jet.2005.12.008 · Cited by 202
Abe, H., & Lee, D. (2011) · Neuron · doi:10.1016/j.neuron.2011.03.026 · Cited by 174
An Evaluation of Econometric Models of Adaptive Learning
Salmon, T. C. (2001) · Econometrica · doi:10.1111/1468-0262.00258 · Cited by 130
Neuroeconomic Approaches to Mental Disorders
Kishida, K. T., King‐Casas, B., & Montague, P. R. (2010) · Neuron · doi:10.1016/j.neuron.2010.07.021 · Cited by 124
Theories of Learning in Games and Heterogeneity Bias
Wilcox, N. T. (2006) · Econometrica · doi:10.1111/j.1468-0262.2006.00704.x · Cited by 109
Rapoport, A., & Amaldoss, W. (2000) · Journal of Economic Behavior & Organization · doi:10.1016/s0167-2681(00)00101-3 · Cited by 98
Experience-Weighted Attraction Learning in Games: A Unifying ApproachPreprint
Camerer, C., & Ho, T. (1997) · RePEc: Research Papers in Economics · Cited by 67
Role of the Superior Colliculus in Choosing Mixed-Strategy Saccades
Thevarajah, D., Mikulić, A., & Dorris, M. C. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.4764-08.2009 · Cited by 39
Cited by
Sorted by citations
The Anterior Cingulate Gyrus and Social Cognition: Tracking the Motivation of Others
Apps, M. A. J., Rushworth, M. F. S., & Chang, S. W. C. (2016) · Neuron · doi:10.1016/j.neuron.2016.04.018 · Cited by 559
“The responses in the ACCg to the value of behaviors for others and to PEs that were discussed in earlier sections approximate the signals predicted by the MoVE model (Lockwood et al., 2015; Apps and Ramnani, 2014; Zhu et al., 2012; Apps et al., 2012, 2013a, 2015a).”
InfluentialNeural Basis of Reinforcement Learning and Decision Making
Lee, D., Seo, H., & Jung, M. W. (2012) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062111-150512 · Cited by 581
Theory of Mind: A Neural Prediction Problem
Koster-Hale, J., & Saxe, R. (2013) · Neuron · doi:10.1016/j.neuron.2013.08.020 · Cited by 513
“…stimuli (e.g., smiling faces) as reward, or learning about reward based on observation or consideration of others’ experiences (Lin et al., 2012; Zhu et al., 2012; Zaki and Mitchell, 2011; Poore et al., 2012; Jones et al., 2011; Izuma et al., 2008; Chang and Sanfey, 2013; see Dunne and…”
Behavioral Variability through Stochastic Choice and Its Gating by Anterior Cingulate Cortex
Tervo, D. G. R., Proskurin, M., Manakov, M., Kabra, M., Vollmer, A., Branson, K. M., & Karpova, A. Y. (2014) · Cell · doi:10.1016/j.cell.2014.08.037 · Cited by 283
“Indeed, studies in primates support the idea that competition triggers model-based counterprediction rather than stochastic choice (Abe and Lee, 2011; Zhu et al., 2012).”
InfluentialThe social brain and reward: Social information processing in the human striatum
Bhanji, J. P., & Delgado, M. R. (2014) · WIREs Cognitive Science · doi:10.1002/wcs.1266 · Cited by 256
Metadata
- Authors
- Zhu, L., Mathewson, K. E., & Hsu, M.
- Affiliations
- Department of Economics and
- University of Alberta (CA)
- University of California, Berkeley (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2012-01-18
- Volume
- 109
- Issue
- 5
- Pages
- 1419-1424
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC3277161
- OpenAlex
- W2165958648
- Cited by
- 125
- References
- 45
- Citations by year
- 2026: 7 · 2025: 6 · 2024: 6 · 2023: 4 · 2022: 4 · 2021: 14 · 2020: 9 · 2019: 7 · 2018: 7 · 2017: 8 · 2016: 11 · 2015: 11 · 2014: 13 · 2013: 11 · 2012: 7
- FWCI
- 6.87
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- strategic learning, competitive decision-making, opponent modeling, belief learning, ventral striatum, rostral anterior cingulate cortex, model-based reinforcement learning, internal values, individual differences
- MeSH terms
- Decision Making, Humans, Learning, Motivation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
