Cortical mechanisms for reinforcement learning in competitive games
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Game theory analyses optimal strategies for multiple decision makers interacting in a social group. However, the behaviours of individual humans and animals often deviate systematically from the optimal strategies described by game theory. The behaviours of rhesus monkeys (Macaca mulatta) in simple zero-sum games showed similar patterns, but their departures from the optimal strategies were well accounted for by a simple reinforcement-learning algorithm. During a computer-simulated zero-sum game, neurons in the dorsolateral prefrontal cortex often encoded the previous choices of the animal and its opponent as well as the animal's reward history. By contrast, the neurons in the anterior cingulate cortex predominantly encoded the animal's reward history. Using simple competitive games, therefore, we have demonstrated functional specialization between different areas of the primate frontal cortex involved in outcome monitoring and action selection. Temporally extended signals related to the animal's previous choices might facilitate the association between choices and their delayed outcomes, whereas information about the choices of the opponent might be used to estimate the reward expected from a particular action. Finally, signals related to the reward history might be used to monitor the overall success of the animal's current decision-making strategy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Separate neural systems value immediate and delayed monetary rewards
McClure, S. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2004) · Science · doi:10.1126/science.1100907 · Cited by 3,089
Neural correlates of decision variables in parietal cortex
Platt, M. L., & Glimcher, P. W. (1999) · Nature · doi:10.1038/22268 · Cited by 1,721
Neurons in the orbitofrontal cortex encode economic value
Padoa-Schioppa, C., & Assad, J. A. (2006) · Nature · doi:10.1038/nature04676 · Cited by 1,598
A neural basis for social cooperation
Rilling, J. K., Gutman, D. A., Zeh, T. R., Pagnoni, G., Berns, G. S., & Kilts, C. D. (2002) · Neuron · doi:10.1016/s0896-6273(02)00755-9 · Cited by 1,451
Getting to Know You: Reputation and Trust in a Two-Person Economic Exchange
King-Casas, B., Tomlin, D., Anen, C., Camerer, C. F., Quartz, S. R., & Montague, P. R. (2005) · Science · doi:10.1126/science.1108062 · Cited by 1,345
Animal intelligence; experimental studies
Thorndike, E. L. (1911) · doi:10.5962/bhl.title.55072 · Cited by 1,100
Neural Responses to Taxation and Voluntary Giving Reveal Motives for Charitable Donations
Harbaugh, W. T., Mayr, U., & Burghart, D. R. (2007) · Science · doi:10.1126/science.1140738 · Cited by 1,046
Matching Behavior and the Representation of Value in the Parietal Cortex
Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004) · Science · doi:10.1126/science.1094765 · Cited by 1,030
A functional imaging study of cooperation in two-person reciprocal exchange
McCabe, K., Houser, D., Ryan, L., Smith, V., & Trouard, T. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.211415698 · Cited by 984
Optimal decision making and the anterior cingulate cortex
Kennerley, S. W., Walton, M. E., Behrens, T. E. J., Buckley, M. J., & Rushworth, M. F. S. (2006) · Nature Neuroscience · doi:10.1038/nn1724 · Cited by 948
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939
Human fronto–mesolimbic networks guide decisions about charitable donation
Moll, J., Krueger, F., Zahn, R., Pardini, M., de Oliveira-Souza, R., & Grafman, J. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0604475103 · Cited by 900
In All Likelihood: Statistical Modelling and Inference Using Likelihood
Pawitan, Y. (2001) · doi:10.1093/oso/9780198507659.001.0001 · Cited by 874
Choice, uncertainty and value in prefrontal and cingulate cortex
Rushworth, M. F. S., & Behrens, T. E. J. (2008) · Nature Neuroscience · doi:10.1038/nn2066 · Cited by 866
Social neuroeconomics: the neural circuitry of social preferences
Fehr, E., & Camerer, C. F. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.09.002 · Cited by 779
Reward expectancy in primate prefrental neurons
Watanabe, M. (1996) · Nature · doi:10.1038/382629a0 · Cited by 684
Prefrontal cortex and decision making in a mixed-strategy game
Barraclough, D. J., Conroy, M. L., & Lee, D. (2004) · Nature Neuroscience · doi:10.1038/nn1209 · Cited by 678
Self-citationContrasting roles for cingulate and orbitofrontal cortex in decisions and social behaviour
Rushworth, M. F. S., Behrens, T. E. J., Rudebeck, P. H., & Walton, M. E. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.01.004 · Cited by 507
Leon, M. I., & Shadlen, M. N. (1999) · Neuron · doi:10.1016/s0896-6273(00)80854-5 · Cited by 449
Influence of Reward Expectation on Behavior-Related Neuronal Activity in Primate Striatum
Hollerman, J. R., Tremblay, L., & Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.2.947 · Cited by 380
Lewis, D. A., Melchitzky, D. S., Sesack, S. R., Whitehead, R. E., Auh, S., & Sampson, A. (2001) · Journal of Comparative Neurology · doi:10.1002/cne.1092 · Cited by 352
Seo, H., & Lee, D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.2369-07.2007 · Cited by 324
Self-citationOpposing BOLD responses to reciprocated and unreciprocated altruism in putative reward pathways
Rilling, J. K., Sanfey, A. G., Aronson, J. A., Nystrom, L. E., & Cohen, J. D. (2004) · NeuroReport · doi:10.1097/00001756-200411150-00022 · Cited by 281
Meta-learning in Reinforcement Learning
Schweighofer, N., & Doya, K. (2003) · Neural Networks · doi:10.1016/s0893-6080(02)00228-9 · Cited by 261
Reinforcement learning and decision making in monkeys during a competitive game
Lee, D., Conroy, M. L., McGreevy, B. P., & Barraclough, D. J. (2004) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2004.07.007 · Cited by 156
Self-citationNeural mechanism for stochastic behaviour during a competitive game
Soltani, A., Lee, D., & Wang, X.-J. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.05.044 · Cited by 77
Self-citation
Cited by
Frontal cortex and reward-guided learning and decision-making
Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E., & Behrens, T. E. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.014 · Cited by 1,111
Prefrontal cortex as a meta-reinforcement learning system
Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., & Botvinick, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0147-8 · Cited by 596
Neural Basis of Reinforcement Learning and Decision Making
Lee, D., Seo, H., & Jung, M. W. (2012) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062111-150512 · Cited by 546
Self-citationSocial Network Size Affects Neural Circuits in Macaques
Sallet, J., Mars, R. B., Noonan, M. P., Andersson, J. L., O’Reilly, J. X., Jbabdi, S., Croxson, P. L., Jenkinson, M., Miller, K. L., & Rushworth, M. F. S. (2011) · Science · doi:10.1126/science.1210027 · Cited by 514
The Computation of Social Behavior
Behrens, T. E. J., Hunt, L. T., & Rushworth, M. F. S. (2009) · Science · doi:10.1126/science.1169694 · Cited by 407
Metadata
- Authors
- Seo, H., & Lee, D.
- Venue
- Philosophical Transactions of the Royal Society B: Biological Sciences
- Publisher
- The Royal Society
- Published
- 2008-10
- Volume
- 363
- Issue
- 1511
- Pages
- 3845-3857
- ISSN
- 0962-8436 (ISSN-L)
- 1471-2970
- OpenAlex
- W2116722476
- Cited by
- 65
- References
- 26
- FWCI
- 4.38
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.