Dissociable neural representations of reinforcement and belief prediction errors underlie strategic learning

Zhu, Lusha; Mathewson, Kyle E.; Hsu, Ming

doi:10.1073/pnas.1116783109

Cited by 125Neural and Behavioral Psychology StudiesNeurotransmitter Receptor Influence on BehaviorDecision-Making and Behavioral Economics

Dissociable neural representations of reinforcement and belief prediction errors underlie strategic learning

Open access

Zhu, Lusha, Mathewson, Kyle E., Hsu, Ming·2012·Proceedings of the National Academy of Sciences

DOI 10.1073/pnas.1116783109PMC PMC3277161

Open this paper in CoChat — free account

Abstract

Decision-making in the presence of other competitive intelligent agents is fundamental for social and economic behavior. Such decisions require agents to behave strategically, where in addition to learning about the rewards and punishments available in the environment, they also need to anticipate and respond to actions of others competing for the same rewards. However, whereas we know much about strategic learning at both theoretical and behavioral levels, we know relatively little about the underlying neural mechanisms. Here, we show using a multi-strategy competitive learning paradigm that strategic choices can be characterized by extending the reinforcement learning (RL) framework to incorporate agents’ beliefs about the actions of their opponents. Furthermore, using this characterization to generate putative internal values, we used model-based functional magnetic resonance imaging to investigate neural computations underlying strategic learning. We found that the distinct notions of prediction errors derived from our computational model are processed in a partially overlapping but distinct set of brain regions. Specifically, we found that the RL prediction error was correlated with activity in the ventral striatum. In contrast, activity in the ventral striatum, as well as the rostral anterior cingulate (rACC), was correlated with a previously uncharacterized belief-based prediction error. Furthermore, activity in rACC reflected individual differences in degree of engagement in belief learning. These results suggest a model of strategic behavior where learning arises from interaction of dissociable reinforcement and belief-based inputs.

Integrity signals

Registry

Crossref · 10.1073/pnas.1116783109

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Ventral striatum and medial prefrontal cortex signals encode information learned about strategic actions of competitors.
“Additionally, ventral striatum and medial prefrontal cortex signals encode information learned about strategic actions of competitors 72, 88, 90.”
From The social brain and reward: Social information processing in the human striatum (2014) · cites this paper as [90]

Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Statistical parametric maps in functional imaging: A general linear approach
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. ‐., Frith, C. D., & Frackowiak, R. S. J. (1994) · Human Brain Mapping · doi:10.1002/hbm.460020402 · Cited by 9,688 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Cognitive and emotional influences in anterior cingulate cortex
Bush, G., Luu, P., & Posner, M. I. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01483-2 · Cited by 6,259 · see citing works
Evolutionary Games and Population Dynamics
Hofbauer, J., & Sigmund, K. (1998) · doi:10.1017/cbo9781139173179 · Cited by 5,502 · see citing works
The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity.
Holroyd, C. B., & Coles, M. G. H. (2002) · Psychological Review · doi:10.1037/0033-295x.109.4.679 · Cited by 4,027 · see citing works
Behavioral Game Theory: Experiments in Strategic Interaction
Camerer, C. F. (2003) · Cited by 3,801 · see citing works
Meeting of minds: the medial frontal cortex and social cognition
Amodio, D. M., & Frith, C. D. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1884 · Cited by 3,782 · see citing works
Conflict monitoring and anterior cingulate cortex: an update
Botvinick, M. M., Cohen, J. D., & Carter, C. S. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.10.003 · Cited by 3,623 · see citing works
Quantal Response Equilibria for Normal Form Games
McKelvey, R. D., & Palfrey, T. R. (1995) · Games and Economic Behavior · doi:10.1006/game.1995.1023 · Cited by 2,624 · see citing works
The Theory of Learning in Games
Fudenberg, D., & Levine, D. K. (1998) · MIT Press Books · Cited by 2,417 · see citing works
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224 · see citing works
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · see citing works
Pain and emotion interactions in subregions of the cingulate gyrus
Vogt, B. A. (2005) · Nature Reviews Neuroscience · doi:10.1038/nrn1704 · Cited by 1,810 · see citing works
Learning in extensive-form games: Experimental data and simple dynamic models in the intermediate term
Roth, A. E., & Erev, I. (1995) · Games and Economic Behavior · doi:10.1016/s0899-8256(05)80020-x · Cited by 1,685 · see citing works
Experience-weighted Attraction Learning in Normal Form Games
Camerer, C. F., & Ho, T. (1999) · Econometrica · doi:10.1111/1468-0262.00054 · Cited by 1,442 · see citing works
Getting to Know You: Reputation and Trust in a Two-Person Economic Exchange
King-Casas, B., Tomlin, D., Anen, C., Camerer, C. F., Quartz, S. R., & Montague, P. R. (2005) · Science · doi:10.1126/science.1108062 · Cited by 1,345 · see citing works
Behavioral game theory, experiments in strategic interaction
Canté, F. (2004) · Cuadernos de Economía · Cited by 1,265 · see citing works
The Neural Basis of Financial Risk Taking
Kuhnen, C. M., & Knutson, B. (2005) · Neuron · doi:10.1016/j.neuron.2005.08.008 · Cited by 1,182 · see citing works
Associative learning of social value
Behrens, T. E. J., Hunt, L. T., Woolrich, M. W., & Rushworth, M. F. S. (2008) · Nature · doi:10.1038/nature07538 · Cited by 925 · see citing works
Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum
McClure, S. M., Berns, G. S., & Montague, P. R. (2003) · Neuron · doi:10.1016/s0896-6273(03)00154-5 · Cited by 917 · see citing works
Optimized EPI for fMRI studies of the orbitofrontal cortex
Deichmann, R., Gottfried, J. A., Hutton, C., & Turner, R. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00073-9 · Cited by 821 · see citing works
Social neuroeconomics: the neural circuitry of social preferences
Fehr, E., & Camerer, C. F. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.09.002 · Cited by 779 · see citing works
Perceptions of moral character modulate the neural systems of reward during the trust game
Delgado, M. R., Frank, R. H., & Phelps, E. A. (2005) · Nature Neuroscience · doi:10.1038/nn1575 · Cited by 729 · see citing works
Prefrontal cortex and decision making in a mixed-strategy game
Barraclough, D. J., Conroy, M. L., & Lee, D. (2004) · Nature Neuroscience · doi:10.1038/nn1209 · Cited by 678 · see citing works
Regret and its avoidance: a neuroimaging study of choice behavior
Coricelli, G., Critchley, H., Joffily, M., O’Doherty, J. P., Angela Sirigu, B., & Dolan, R. J. (2005) · Nature Neuroscience · doi:10.1038/nn1514 · Cited by 673 · see citing works
Neural correlates of mentalizing-related computations during strategic interactions in humans
Hampton, A. N., Bossaerts, P. L., & O’Doherty, J. P. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0711099105 · Cited by 518 · see citing works
Model‐Based fMRI and Its Application to Reward Learning and Decision Making
O’Doherty, J. P., Hampton, A. N., & Kim, H. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.022 · Cited by 509 · see citing works
Activity in Posterior Parietal Cortex Is Correlated with the Relative Subjective Desirability of Action
Dorris, M. C., & Glimcher, P. W. (2004) · Neuron · doi:10.1016/j.neuron.2004.09.009 · Cited by 428 · see citing works
The Computation of Social Behavior
Behrens, T. E. J., Hunt, L. T., & Rushworth, M. F. S. (2009) · Science · doi:10.1126/science.1169694 · Cited by 407 · cited 2× · see citing works
“However, studies on reward-guided behavior in social settings have suggested that these computations build on the same brain structures associated with reward, including the striatum, rostral anterior cingulate (rACC), and orbitofrontal cortex (19).”
Background
Neural correlates of depth of strategic reasoning in medial prefrontal cortex
Coricelli, G., & Nagel, R. (2009) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.0807721106 · Cited by 385 · see citing works
Neural mechanisms of observational learning
Burke, C. J., Tobler, P. N., Baddeley, M. C., & Schultz, W. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1003111107 · Cited by 356 · see citing works
Neural signature of fictive learning signals in a sequential investment task
Lohrenz, T. M., McCabe, K., Camerer, C. F., & Montague, P. R. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0608842104 · Cited by 263 · see citing works
Fictive Reward Signals in the Anterior Cingulate Cortex
Hayden, B. Y., Pearson, J. M., & Platt, M. L. (2009) · Science · doi:10.1126/science.1168488 · Cited by 235 · see citing works
Separate Neural Mechanisms Underlie Choices and Strategic Preferences in Risky Decision Making
Venkatraman, V., Payne, J. W., Bettman, J. R., Luce, M. F., & Huettel, S. A. (2009) · Neuron · doi:10.1016/j.neuron.2009.04.007 · Cited by 222 · see citing works
Self-tuning experience weighted attraction learning in games
Ho, T., Camerer, C. F., & Chong, J.-K. (2006) · Journal of Economic Theory · doi:10.1016/j.jet.2005.12.008 · Cited by 217 · see citing works
Mixed-effects and fMRI studies
Friston, K. J., Stephan, K. E., Lund, T. E., Morcom, A. M., & Kiebel, S. J. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.08.055 · Cited by 204 · see citing works
Distributed Coding of Actual and Hypothetical Outcomes in the Orbital and Dorsolateral Prefrontal Cortex
Abe, H., & Lee, D. (2011) · Neuron · doi:10.1016/j.neuron.2011.03.026 · Cited by 168 · see citing works
An Evaluation of Econometric Models of Adaptive Learning
Salmon, T. C. (2001) · Econometrica · doi:10.1111/1468-0262.00258 · Cited by 139 · see citing works
Theories of Learning in Games and Heterogeneity Bias
Wilcox, N. T. (2006) · Econometrica · doi:10.1111/j.1468-0262.2006.00704.x · Cited by 109 · see citing works
Mixed strategies and iterative elimination of strongly dominated strategies: an experimental investigation of states of knowledge
Rapoport, A., & Amaldoss, W. (2000) · Journal of Economic Behavior & Organization · doi:10.1016/s0167-2681(00)00101-3 · Cited by 98 · see citing works
Experience-Weighted Attraction Learning in Games: A Unifying ApproachPreprint
Camerer, C., & Ho, T. (1997) · RePEc: Research Papers in Economics · Cited by 67 · see citing works
Role of the Superior Colliculus in Choosing Mixed-Strategy Saccades
Thevarajah, D., Mikulić, A., & Dorris, M. C. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.4764-08.2009 · Cited by 40 · see citing works
Neuroeconomic approaches to mental disorders
Kishida, K., King-Casas, B., & Montague, P. R. (2010) · Neuron · doi:10.1016/j.neuron.2010.07.021 · Cited by 9 · see citing works

Flagged first, then by citations

Sorted by citations

The Anterior Cingulate Gyrus and Social Cognition: Tracking the Motivation of Others
Apps, M. A. J., Rushworth, M. F. S., & Chang, S. W. C. (2016) · Neuron · doi:10.1016/j.neuron.2016.04.018 · Cited by 559 · see citing works
“The responses in the ACCg to the value of behaviors for others and to PEs that were discussed in earlier sections approximate the signals predicted by the MoVE model (Lockwood et al., 2015; Apps and Ramnani, 2014; Zhu et al., 2012; Apps et al., 2012, 2013a, 2015a).”
Influential
Neural Basis of Reinforcement Learning and Decision Making
Lee, D., Seo, H., & Jung, M. W. (2012) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062111-150512 · Cited by 542 · see citing works
Theory of Mind: A Neural Prediction Problem
Koster-Hale, J., & Saxe, R. (2013) · Neuron · doi:10.1016/j.neuron.2013.08.020 · Cited by 513 · see citing works
Behavioral Variability through Stochastic Choice and Its Gating by Anterior Cingulate Cortex
Tervo, D. G. R., Proskurin, M., Manakov, M., Kabra, M., Vollmer, A., Branson, K. M., & Karpova, A. Y. (2014) · Cell · doi:10.1016/j.cell.2014.08.037 · Cited by 269 · see citing works
The social brain and reward: Social information processing in the human striatum
Bhanji, J. P., & Delgado, M. R. (2014) · WIREs Cognitive Science · doi:10.1002/wcs.1266 · Cited by 256 · see citing works
Interaction of Candida albicans with host cells: virulence factors, host defense, escape strategies, and the microbiota
Hoefs, S., Mogavero, S., & Hube, B. (2016) · The Journal of Microbiology · doi:10.1007/s12275-016-5514-0 · Cited by 238 · see citing works
Executive control and decision-making in the prefrontal cortex
Domenech, P., & Koechlin, É. (2014) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2014.10.007 · Cited by 237 · see citing works
A causal account of the brain network computations underlying strategic social behavior
Hill, C., Suzuki, S., Polanía, R., Moisa, M., O’Doherty, J. P., & Ruff, C. C. (2017) · Nature Neuroscience · doi:10.1038/nn.4602 · Cited by 196 · see citing works
Learning to minimize efforts versus maximizing rewards: computational principles and neural correlates
Skvortsova, V., Palminteri, S., & Pessiglione, M. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.1350-14.2014 · Cited by 177 · see citing works
Using reinforcement learning models in social neuroscience: frameworks, pitfalls and suggestions of best practices
Zhang, L., Lengersdorff, L., Mikuš, N., Gläscher, J. P., & Lamm, C. (2020) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsaa089 · Cited by 158 · see citing works
Decision Making: From Neuroscience to Psychiatry
Lee, D. (2013) · Neuron · doi:10.1016/j.neuron.2013.04.008 · Cited by 169 · see citing works
Prefrontal executive function and adaptive behavior in complex environments
Koechlin, É. (2015) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2015.11.004 · Cited by 154 · see citing works
Instrumental learning of traits versus rewards: dissociable neural correlates and effects on choice
Hackel, L. M., Doll, B. B., & Amodio, D. M. (2015) · Nature Neuroscience · doi:10.1038/nn.4080 · Cited by 148 · see citing works
Social learning through prediction error in the brain
Joiner, J. A., Piva, M., Turrin, C., & Chang, S. W. C. (2017) · npj Science of Learning · doi:10.1038/s41539-017-0009-2 · Cited by 134 · see citing works
Computational modelling of social cognition and behaviour—a reinforcement learning primer
Lockwood, P. L., & Klein-Flügge, M. C. (2020) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsaa040 · Cited by 125 · see citing works
Causal Control of Medial–Frontal Cortex Governs Electrophysiological and Behavioral Indices of Performance Monitoring and Learning
Reinhart, R. M. G., & Woodman, G. F. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.5421-13.2014 · Cited by 126 · see citing works
Brain Changes Associated with Postural Training in Patients with Cerebellar Degeneration: A Voxel-Based Morphometry Study
Burciu, R. G., Fritsche, N., Granert, O., Schmitz, L., Spönemann, N., Konczak, J., Theysohn, N., Gerwig, M., van Eimeren, T., & Timmann, D. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.3381-12.2013 · Cited by 117 · see citing works
Neural Mechanisms of Social Cognition in Primates
Wittmann, M. K., Lockwood, P. L., & Rushworth, M. F. S. (2018) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-080317-061450 · Cited by 113 · see citing works
The Opponent Matters: Elevated fMRI Reward Responses to Winning Against a Human Versus a Computer Opponent During Interactive Video Game Playing
Kätsyri, J., Hari, R., Ravaja, N., & Nummenmaa, L. (2012) · Cerebral Cortex · doi:10.1093/cercor/bhs259 · Cited by 111 · see citing works
Neural correlates of strategic reasoning during competitive games
Seo, H., Cai, X., Donahue, C. H., & Lee, D. (2014) · Science · doi:10.1126/science.1256254 · Cited by 108 · see citing works
Neural Basis of Strategic Decision Making
Lee, D., & Seo, H. (2015) · Trends in Neurosciences · doi:10.1016/j.tins.2015.11.002 · Cited by 104 · see citing works
Understanding Interpersonal Function in Psychiatric Illness Through Multiplayer Economic Games
King‐Casas, B., & Chiu, P. H. (2012) · Biological Psychiatry · doi:10.1016/j.biopsych.2012.03.033 · Cited by 94 · see citing works
Disrupted prediction errors index social deficits in autism spectrum disorder
Balsters, J. H., Apps, M. A. J., Bolis, D., Lehner, R., Gallagher, L., & Wenderoth, N. (2017) · Brain · doi:10.1093/brain/aww287 · Cited by 85 · see citing works
Computational neuroscience approaches to social cognition
Hackel, L. M., & Amodio, D. M. (2018) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2018.09.001 · Cited by 85 · see citing works
The application of computational models to social neuroscience: promises and pitfalls
Charpentier, C. J., & O’Doherty, J. P. (2018) · Social Neuroscience · doi:10.1080/17470919.2018.1518834 · Cited by 83 · see citing works
Neural Correlates of Reinforcement Learning and Social Preferences in Competitive Bidding
van den Bos, W., Talwar, A., & McClure, S. M. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.3095-12.2013 · Cited by 80 · see citing works
The dorsomedial prefrontal cortex computes task-invariant relative subjective value for self and other
Piva, M., Velnoskey, K. R., Jia, R., Nair, A. R., Levy, I., & Chang, S. W. C. (2019) · eLife · doi:10.7554/elife.44939 · Cited by 78 · see citing works
The computational challenge of social learning
FeldmanHall, O., & Nassar, M. R. (2021) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2021.09.002 · Cited by 75 · see citing works
Vicarious Reinforcement Learning Signals When Instructing Others
Apps, M. A. J., Lesage, E., & Ramnani, N. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.3669-14.2015 · Cited by 69 · see citing works
Treatment approaches for interoceptive dysfunctions in drug addiction
Paulus, M. P., Stewart, J. L., & Haase, L. (2013) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2013.00137 · Cited by 69 · see citing works
Theory of mind and decision science: Towards a typology of tasks and computational models
Rusch, T., Kumar, S., Doshi, P., Spezio, M., & Gläscher, J. P. (2020) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2020.107488 · Cited by 67 · see citing works
Neural Changes with Tactile Learning Reflect Decision-Level Reweighting of Perceptual Readout
Sathian, K., Deshpande, G., & Stilla, R. F. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.3482-12.2013 · Cited by 63 · see citing works
Goal-directed, habitual and Pavlovian prosocial behavior
GÄTMsiarz, F., & Crockett, M. J. (2015) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2015.00135 · Cited by 60 · see citing works
Decision-making: from neuroscience to neuroeconomics—an overview
Serra, D. (2021) · Theory and Decision · doi:10.1007/s11238-021-09830-3 · Cited by 61 · see citing works
Excessive deliberation in social anxiety
Hunter, L. E., Meer, E. A., Gillan, C., Hsu, M., & Daw, N. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01180-y · Cited by 41 · see citing works
Self-citation
Over a Decade of Neuroeconomics: What Have We Learned?
Konovalov, A., & Krajbich, I. M. (2016) · Organizational Research Methods · doi:10.1177/1094428116644502 · Cited by 51 · see citing works
Dissecting functional contributions of the social brain to strategic behavior
Konovalov, A., Hill, C., Jean Daunizeau, A., & Ruff, C. C. (2021) · Neuron · doi:10.1016/j.neuron.2021.07.025 · Cited by 57 · see citing works
Computational substrates of social norm enforcement by unaffected third parties
Zhong, S., Chark, R., Hsu, M., & Chew, S. H. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.01.040 · Cited by 47 · see citing works
Self-citation
Neurocomputational approaches to social behavior
Konovalov, A., Hu, J., & Ruff, C. C. (2018) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2018.04.009 · Cited by 48 · see citing works
Neural basis of learning and preference during social decision-making
Seo, H., & Lee, D. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.05.010 · Cited by 43 · see citing works
The Dynamics of Belief Updating in Human Cooperation: Findings from inter-brain ERP hyperscanning
Zhang, D., Lin, Y., Jing, Y., Feng, C., & Gu, R. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.05.029 · Cited by 43 · see citing works
Breaking human social decision making into multiple components and then putting them together again
Suzuki, S., & O’Doherty, J. P. (2020) · Cortex · doi:10.1016/j.cortex.2020.02.014 · Cited by 43 · see citing works
Dissociable contribution of prefrontal and striatal dopaminergic genes to learning in economic games
Set, E., Sáez, I., Zhu, L., Houser, D. E., Myung, N., Zhong, S., Ebstein, R. P., Chew, S. H., & Hsu, M. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1316259111 · Cited by 35 · see citing works
Self-citation
Neural coding of prior expectations in hierarchical intention inference
Chambon, V., Domenech, P., Jacquet, P. O., Barbalat, G., Bouton, S., Pacherie, É., Koechlin, É., & Farrer, C. (2017) · Scientific Reports · doi:10.1038/s41598-017-01414-y · Cited by 30 · see citing works
Dissociable roles of left and right temporoparietal junction in strategic competitive interaction
Ogawa, A., & Kameda, T. (2019) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsz082 · Cited by 32 · see citing works
Neural dynamics of social tie formation in economic decision-making
Bault, N., Pelloux, B., Fahrenfort, J. J., Ridderinkhof, K. R., & van Winden, F. A. A. M. (2014) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsu138 · Cited by 33 · see citing works
A Review Essay aboutFoundations of Neuroeconomic Analysisby Paul Glimcher
Camerer, C. F. (2013) · Journal of Economic Literature · doi:10.1257/jel.51.4.1155 · Cited by 28 · see citing works
The neuroeconomics of strategic interaction
Griessinger, T., & Coricelli, G. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.01.012 · Cited by 29 · see citing works
Differential Reward Learning for Self and Others Predicts Self-Reported Altruism
Kwak, Y., Pearson, J., & Huettel, S. A. (2014) · PLoS ONE · doi:10.1371/journal.pone.0107621 · Cited by 31 · see citing works
Decoding covert motivations of free riding and cooperation from multi-feature pattern analysis of EEG signals
Chung, D., Yun, K., & Jeong, J. (2015) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsv006 · Cited by 25 · see citing works
Affective evaluation of others’ altruistic decisions under risk and ambiguity
Xiong, W., Gao, X., He, Z., Yu, H., Liu, H., & Zhou, X. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116996 · Cited by 23 · see citing works
Neurocomputational mechanisms involved in adaptation to fluctuating intentions of others
Philippe, R., Janet, R., Khalvati, K., Rao, R. P. N., Lee, D., & Dreher, J. (2024) · Nature Communications · doi:10.1038/s41467-024-47491-2 · Cited by 27 · see citing works
Reading between the lines: Listener’s vmPFC simulates speaker cooperative choices in communication games
Mi, Q., Wang, C., Camerer, C. F., & Zhu, L. (2021) · Science Advances · doi:10.1126/sciadv.abe6276 · Cited by 23 · see citing works
Self-citation
Enhancing models of social and strategic decision making with process tracing and neural data
Konovalov, A., & Ruff, C. C. (2021) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1559 · Cited by 20 · see citing works
Computational Models in Social Neuroscience
Cheong, J. H., Jolly, E., Sul, S., & Chang, L. J. (2017) · Wiley eBooks · doi:10.1002/9781119159193.ch17 · Cited by 20 · see citing works
Efficient learning mechanisms hold in the social domain and are implemented in the medial prefrontal cortex
Seid-Fatemi, A., & Tobler, P. N. (2014) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsu130 · Cited by 20 · see citing works
Computational models as statistical tools
Durstewitz, D., Koppe, G., & Toutounji, H. (2016) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2016.07.004 · Cited by 19 · see citing works
Balancing selfishness and norm conformity can explain human behavior in large-scale prisoner's dilemma games and can poise human groups near criticality
Realpe-Gómez, J., Andrighetto, G., Nardin, L. G., & Montoya, J. A. (2018) · Physical review. E · doi:10.1103/physreve.97.042321 · Cited by 18 · see citing works
From genes to behavior: placing cognitive models in the context of biological pathways
Sáez, I., Set, E., & Hsu, M. (2014) · Frontiers in Neuroscience · doi:10.3389/fnins.2014.00336 · Cited by 17 · see citing works
Self-citation
Political games of attack and defence
Dreu, C. K. W. D., Pliskin, R., Rojek-Giffin, M., Méder, Z. Z., & Gross, J. (2021) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2020.0135 · Cited by 17 · see citing works
CACNA1C gene regulates behavioral strategies in operant rule learning
Koppe, G., Mallien, A. S., Berger, S. M., Bartsch, D., Gass, P., Vollmayr, B., & Durstewitz, D. (2017) · PLoS Biology · doi:10.1371/journal.pbio.2000936 · Cited by 17 · see citing works
Source localized infraslow neurofeedback training in people with chronic painful knee osteoarthritis: A randomized, double-blind, sham-controlled feasibility clinical trial
Mathew, J., Adhia, D. B., Smith, M. L., Ridder, D. D., & Mani, R. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.899772 · Cited by 17 · see citing works
Learning to Resolve Social Dilemmas: A Survey
Fatima, S. S., Jennings, N. R., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15167 · Cited by 16 · see citing works
The Neural Basis of Strategic Choice
Camerer, C. F., & Hare, T. A. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00025-5 · Cited by 15 · see citing works
Belief Updating during Social Interactions: Neural Dynamics and Causal Role of Dorsomedial Prefrontal Cortex
Christian, P., Kaiser, J., Taylor, P. C., George, M., Schütz‐Bosbach, S., & Soutschek, A. (2024) · Journal of Neuroscience · doi:10.1523/jneurosci.1669-23.2024 · Cited by 15 · see citing works
Foraging under conditions of short-term exploitative competition: the case of stock traders
Saavedra, S., Malmgren, R. D., Switanek, N., & Uzzi, B. (2013) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2012.2901 · Cited by 14 · see citing works
Functions of Learning Rate in Adaptive Reward Learning
Wu, X., Wang, T., Liu, C., Wu, T., Jiang, J., Zhou, D., & Zhou, J. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00592 · Cited by 14 · see citing works
Neural correlates of recursive thinking during interpersonal strategic interactions
Zhen, S., & Yu, R. (2021) · Human Brain Mapping · doi:10.1002/hbm.25355 · Cited by 14 · see citing works
Neural mechanisms of social learning and decision-making
Ni, Y., & Li, J. (2020) · Science China Life Sciences · doi:10.1007/s11427-020-1833-8 · Cited by 15 · see citing works
Hierarchical Control of Behaviour in Human Prefrontal Cortex
Duverne, S., & Koechlin, É. (2017) · doi:10.1002/9781118920497.ch12 · Cited by 14 · see citing works
Neurocomputational mechanism of real-time distributed learning on social networks
Jiang, Y., Mi, Q., & Zhu, L. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01258-y · Cited by 14 · see citing works
Self-citation
Does It Pay to Increase Competition in Combinatorial Conservation Auctions?
Iftekhar, M. S., Hailu, A., & Lindner, R. K. (2014) · Canadian Journal of Agricultural Economics/Revue canadienne d agroeconomie · doi:10.1111/cjag.12028 · Cited by 13 · see citing works
Neuroeconomic Measures of Social Decision-Making Across the Lifespan
Zhu, L., Walsh, D., & Hsu, M. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00128 · Cited by 13 · see citing works
Self-citation
Excessive deliberation in social anxietyPreprint
Hunter, L. E., Meer, E. A., Gillan, C. M., Hsu, M., & Daw, N. D. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/522433 · Cited by 13 · see citing works
Self-citation
Patients with basal ganglia damage show preserved learning in an economic game
Zhu, L., Jiang, Y., Scabini, D., Knight, R. T., & Hsu, M. (2019) · Nature Communications · doi:10.1038/s41467-019-08766-1 · Cited by 13 · see citing works
Self-citation
Agency Modulates the Lateral and Medial Prefrontal Cortex Responses in Belief-Based Decision Making
Xue, G., He, Q., Lu, Z., Levin, I. P., Dong, Q., & Bechara, A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0065274 · Cited by 11 · see citing works
The interaction of perceived control and Gambler's fallacy in risky decision making: An f MRI study
Shao, R., Sun, D., & Lee, T. M. C. (2016) · Human Brain Mapping · doi:10.1002/hbm.23098 · Cited by 11 · see citing works
Learning whom to cooperate with: neurocomputational mechanisms for choosing cooperative partners
Jin, T., Zhang, S., Lockwood, P. L., Vilares, I., Wu, H., Liu, C., & Ma, Y. (2022) · Cerebral Cortex · doi:10.1093/cercor/bhac365 · Cited by 10 · see citing works
Opponent Identity Influences Value Learning in Simple Games
Vickery, T. J., Kleinman, M. R., Chun, M. M., & Lee, D. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.3530-14.2015 · Cited by 9 · see citing works
Learning Dynamics and Norm Psychology Supports Human Cooperation in a Large-Scale Prisoner’s Dilemma on Networks
Realpe-Gómez, J., Vilone, D., Andrighetto, G., Nardin, L. G., & Montoya, J. A. (2018) · Games · doi:10.3390/g9040090 · Cited by 10 · see citing works
Latent goal models for dynamic strategic interaction
Iqbal, S., Yin, L., Drucker, C. B., Qian, K., Gariépy, J., Platt, M. L., & Pearson, J. M. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006895 · Cited by 9 · see citing works
Characterizing human random-sequence generation in competitive and non-competitive environments using Lempel–Ziv complexity
Wong, A., Merholz, G., & Maoz, U. (2021) · Scientific Reports · doi:10.1038/s41598-021-99967-6 · Cited by 9 · see citing works
A Neural Mechanism of Strategic Social Choice under Sanction-Induced Norm Compliance
Makwana, A., Grön, G., Fehr, E., & Hare, T. A. (2015) · eNeuro · doi:10.1523/eneuro.0066-14.2015 · Cited by 9 · see citing works
How mood shapes belief updating bias in depression
Bottemanne, H., Frileux, S., Sévoz‐Couche, C., Pelloux, Y., Colle, R., & Corruble, E. (2025) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-025-01297-x · Cited by 8 · see citing works
Reinforcement Learning and Strategic Reasoning During Social Decision-Making
Seo, H., & Lee, D. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-805308-9.00018-x · Cited by 8 · see citing works
Introducing COSMOS: a Web Platform for Multimodal Game-Based Psychological Assessment Geared Towards Open Science Practice
Aeberhard, A., Gschwind, L., Kossowsky, J., Lukšys, G., Papassotiropoulos, A., de Quervain, D. J. ‐., & Vogler, C. (2018) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-018-0071-5 · Cited by 7 · see citing works
Trait Reward Sensitivity Modulates Connectivity with the Temporoparietal Junction and Anterior Insula during Strategic Decision Making
Sazhin, D., Wyngaarden, J. Β., Dennison, J. B., Zaff, O., Fareri, D. S., McCloskey, M. S., Alloy, L. B., Jarcho, J. M., & Smith, D. V. (2023) · Biological Psychology · doi:10.1016/j.biopsycho.2024.108857 · Cited by 9 · see citing works
Navigating uncertainty in human social decision‐making: Consequences and coping strategies
Li, T., Li, S., Li, B., Zhang, Z., Luo, Y., & Feng, C. (2023) · Social and Personality Psychology Compass · doi:10.1111/spc3.12756 · Cited by 7 · see citing works
The neural correlates of reaching focal points
Chowdhury, A., Liu, C., & Yu, R. (2020) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2020.107397 · Cited by 6 · see citing works
Neurocognitive Underpinnings of Aggressive Predation in Economic Contests
Rojek-Giffin, M., Lebreton, M. P., Scholte, H. S., van Winden, F. A. A. M., Ridderinkhof, K. R., & Dreu, C. K. W. D. (2020) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01545 · Cited by 6 · see citing works
Deliberation during online bargaining reveals strategic information
Cotet, M., Zhao, W. J., & Krajbich, I. M. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2410956122 · Cited by 5 · see citing works
Neurocomputations of strategic behavior: From iterated to novel interactions
Jiang, Y., Wu, H., Mi, Q., & Zhu, L. (2022) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1598 · Cited by 5 · see citing works
Self-citation
Orbitofrontal-Hippocampal State Coding Dynamics during Reversal Learning
Sanchez, A. N., Ford, C. F., Balewski, Z. Z., & Wallis, J. D. (2025) · Journal of Neuroscience · doi:10.1523/jneurosci.1228-24.2025 · Cited by 5 · see citing works
A Bayesian cognitive hierarchy model with fixed reasoning levels
Jun, F., Qin, X., & Wang, X. (2021) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2021.10.030 · Cited by 5 · see citing works
Autonomous behaviour and the limits of human volition
Ota, K., Charles, L., & Haggard, P. (2023) · Cognition · doi:10.1016/j.cognition.2023.105684 · Cited by 4 · see citing works
Learning in Games : Neural Computations underlying Strategic Learning
Hsu, M., & Zhu, L. (2012) · Recherches économiques de Louvain · doi:10.3917/rel.783.0047 · Cited by 4 · see citing works
Self-citation
The computational and neural substrates underlying social learning
Li, S., Chen, X., Zhai, Y., Zhang, Y., Zhang, Z., & Chunliang, F. E. N. G. (2021) · Advances in Psychological Science · doi:10.3724/sp.j.1042.2021.00677 · Cited by 3 · see citing works
The neural underpinnings of allocentric thinking in a novel signaling task
Zhen, S.-S., Chowdhury, A., & Yu, R. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.117808 · Cited by 3 · see citing works
How collective reward structure impedes group decision making: An experimental study using the HoneyComb paradigm
Ritter, M., Wang, M., Pritz, J., Menssen, O., & Boos, M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0259963 · Cited by 3 · see citing works
Introducing Cool School
Young, M., Killen, M., Lee–Kim, J., & Park, Y. (2012) · International Journal of Game-Based Learning · doi:10.4018/ijgbl.2012100105 · Cited by 3 · see citing works

Neural Dynamics of Belief and Value Computations Guiding Strategic Social DecisionsPreprint
Konovalov, A., Hu, J., & Ruff, C. C. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.08.723744
Reinforcement Learning Signals in the Human Striatum Distinguish Learners from Nonlearners during Reward-Based Decision Making
Schönberg, T., Daw, N. D., Joel, D., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.2496-07.2007 · Cited by 390 · see citing works
Reward and fictive prediction error signals in ventral striatum: asymmetry between factual and counterfactual processing
Santo-Angles, A., Fuentes‐Claramonte, P., Argila‐Plaza, I., Guardiola-Ripoll, M., Almodóvar-Payá, C., Munuera, J., McKenna, P. J., Pomarol‐Clotet, E., & Raduà, J. (2021) · Brain Structure and Function · doi:10.1007/s00429-021-02270-3 · Cited by 3 · see citing works
Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies
Garrison, J. R., Erdeniz, B., & Done, J. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.03.023 · Cited by 430 · see citing works
Causal Inference Gates Corticostriatal Learning
Dorfman, H. M., Tomov, M., Cheung, B., Clarke, D., Gershman, S. J., & Hughes, B. (2021) · Journal of Neuroscience · doi:10.1523/jneurosci.2796-20.2021 · Cited by 15 · see citing works

Authors: Zhu, L., Mathewson, K. E., & Hsu, M.
Affiliations: Department of Economics and
University of Alberta (CA)
University of California, Berkeley (US)
ORCID: Zhu, Lusha
Mathewson, Kyle E.
Hsu, Ming
Venue: Proceedings of the National Academy of Sciences
Publisher: National Academy of Sciences
Published: 2012-01-18
Volume: 109
Issue: 5
Pages: 1419-1424
ISSN: 0027-8424 (ISSN-L)
1091-6490
DOI: 10.1073/pnas.1116783109
PMC: PMC3277161
OpenAlex: W2165958648
Cited by: 125
References: 45
Citations by year: 2026: 7 · 2025: 6 · 2024: 6 · 2023: 4 · 2022: 4 · 2021: 14 · 2020: 9 · 2019: 7 · 2018: 7 · 2017: 8 · 2016: 11 · 2015: 11 · 2014: 13 · 2013: 11 · 2012: 7
FWCI: 6.87
Type: journal article
Language: en
Open access: green
www.ncbi.nlm.nih.gov
Locations: Proceedings of the National Academy of Sciences (publishedVersion)
Europe PMC (PubMed Central) (publishedVersion)
PubMed Central (publishedVersion)
PubMed (publishedVersion)
Topics: Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
Keywords: strategic learning, competitive decision-making, opponent modeling, belief learning, ventral striatum, rostral anterior cingulate cortex, model-based reinforcement learning, internal values, individual differences
MeSH terms: Decision Making, Humans, Learning, Motivation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Dissociable neural representations of reinforcement and belief prediction errors underlie strategic learning

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks