Neural mechanism for stochastic behaviour during a competitive game
Open accessAbstract
Previous studies have shown that non-human primates can generate highly stochastic choice behaviour, especially when this is required during a competitive interaction with another agent. To understand the neural mechanism of such dynamic choice behaviour, we propose a biologically plausible model of decision making endowed with synaptic plasticity that follows a reward-dependent stochastic Hebbian learning rule. This model constitutes a biophysical implementation of reinforcement learning, and it reproduces salient features of behavioural data from an experiment with monkeys playing a matching pennies game. Due to interaction with an opponent and learning dynamics, the model generates quasi-random behaviour robustly in spite of intrinsic biases. Furthermore, non-random choice behaviour can also emerge when the model plays against a non-interactive opponent, as observed in the monkey experiment. Finally, when combined with a meta-learning algorithm, our model accounts for the slow drift in the animal's strategy based on a process of reward maximization.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Neural correlates of reward valuation have been interpreted in the framework of reinforcement learning (RL) theory.
“and interpreted in the framework of reinforcement learning (RL) theory4-5.”
From A reservoir of time constants for memory traces in cortical neurons (2011) · cites this paper as [4]
Prefrontal cortex plays a major role in dynamic decision processes encoding and updating values.
“Prefrontal cortex plays a major role in dynamic decision processes encoding and updating values1-4.”
From A reservoir of time constants for memory traces in cortical neurons (2011) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524 · see citing works
Model selection and multimodel inference: a practical information-theoretic approach.
Guthery, F. S., Burnham, K. P., & Anderson, D. R. (2003) · The Journal of Wildlife Management · doi:10.2307/3802723 · Cited by 41,495 · see citing works
Prospect theory: An analysis of decision under risk
Kahneman, D. J., & Tversky, A. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511609220.014 · Cited by 32,700 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Theory of games and economic behavior
Garcia, S. R., & Miller, S. J. (2019) · American Mathematical Society eBooks · doi:10.1090/mbk/121/32 · Cited by 16,777 · see citing works
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855 · see citing works
Theory of Games and Economic Behavior
Wald, A., von Neumann, J., & Morgenstern, O. (1947) · The Review of Economics and Statistics · doi:10.2307/1925651 · Cited by 2,349 · see citing works
Behavioral Theories and the Neurophysiology of Reward
Schultz, W. (2006) · Annual Review of Psychology · doi:10.1146/annurev.psych.56.091103.070229 · Cited by 1,637 · see citing works
Theory of Games and Economic Behaviour.
K., M. G., von Neumann, J., & Morgenstern, O. (1944) · Journal Of The Royal Statistical Society · doi:10.2307/2981222 · Cited by 1,414 · see citing works
Multiple reward signals in the brain
Schultz, W. (2000) · Nature Reviews Neuroscience · doi:10.1038/35044563 · Cited by 1,303 · see citing works
Probabilistic decision making by slow reverberation in cortical circuits
Wang, X.-J. (2002) · Neuron · doi:10.1016/s0896-6273(02)01092-9 · Cited by 1,242 · see citing works
Self-citationThe basal ganglia: a vertebrate solution to the selection problem?
Redgrave, P., Prescott, T. J., & Gurney, K. (1999) · Neuroscience · doi:10.1016/s0306-4522(98)00319-4 · Cited by 1,199 · see citing works
Models of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190 · see citing works
Animal intelligence; experimental studies
Thorndike, E. L. (1911) · doi:10.5962/bhl.title.55072 · Cited by 1,100 · see citing works
Matching Behavior and the Representation of Value in the Parietal Cortex
Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004) · Science · doi:10.1126/science.1094765 · Cited by 1,030 · see citing works
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939 · see citing works
A cellular mechanism of reward-related learning
Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001) · Nature · doi:10.1038/35092560 · Cited by 879 · see citing works
Animal Intelligence. Experimental Studies
Thorndike, E. L., & Jelliffe (1912) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-191205000-00016 · Cited by 824 · see citing works
Metalearning and neuromodulation
Doya, K. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00044-8 · Cited by 704 · see citing works
Prefrontal cortex and decision making in a mixed-strategy game
Barraclough, D. J., Conroy, M. L., & Lee, D. (2004) · Nature Neuroscience · doi:10.1038/nn1209 · Cited by 678 · see citing works
Self-citationBrunel, N., & Wang, X.-J. (2001) · Journal of Computational Neuroscience · doi:10.1023/a:1011204814320 · Cited by 620 · see citing works
Self-citationDopamine: a potential substrate for synaptic plasticity and memory mechanisms
Jay, T. M. (2003) · Progress in Neurobiology · doi:10.1016/s0301-0082(03)00085-6 · Cited by 576 · see citing works
Cascade Models of Synaptically Stored Memories
Fusi, S., Drew, P. J., & Abbott, L. F. (2005) · Neuron · doi:10.1016/j.neuron.2005.02.001 · Cited by 569 · see citing works
Brunel, N., & Wang, X. (2014) · Journal of Computational Neuroscience · doi:10.1007/s10827-014-0506-8 · Cited by 538 · see citing works
Self-citationChoosing the greater of two goods: neural currencies for valuation and decision making
Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2005) · Nature Reviews Neuroscience · doi:10.1038/nrn1666 · Cited by 536 · see citing works
Dopamine-dependent plasticity of corticostriatal synapses
Reynolds, J. N. J., & Wickens, J. R. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00045-x · Cited by 528 · see citing works
Dorris, M. C., & Glimcher, P. W. (2004) · Neuron · doi:10.1016/j.neuron.2004.09.009 · Cited by 428 · see citing works
Bar‐Hillel, M., & Wagenaar, W. A. (1991) · Advances in Applied Mathematics · doi:10.1016/0196-8858(91)90029-i · Cited by 402 · see citing works
Risk-sensitive neurons in macaque posterior cingulate cortex
McCoy, A. N., & Platt, M. L. (2005) · Nature Neuroscience · doi:10.1038/nn1523 · Cited by 380 · see citing works
Lewis, D. A., Melchitzky, D. S., Sesack, S. R., Whitehead, R. E., Auh, S., & Sampson, A. (2001) · Journal of Comparative Neurology · doi:10.1002/cne.1092 · Cited by 352 · see citing works
All-or-none potentiation at CA3-CA1 synapses
Petersen, C. C. H., Malenka, R. C., Nicoll, R. A., & Hopfield, J. J. (1998) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.95.8.4732 · Cited by 330 · see citing works
Learning in Neural Networks with Material Synapses
Amit, D., & Fusi, S. (1994) · Neural Computation · doi:10.1162/neco.1994.6.5.957 · Cited by 299 · see citing works
A stochastic reinforcement learning algorithm for learning real-valued functions
Gullapalli, V. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90056-q · Cited by 292 · see citing works
(1997) · Medical Entomology and Zoology · Cited by 289 · see citing works
Models of information processing in the basal ganglia
Alexander, G. E. (1996) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0921-884x(96)89703-7 · Cited by 278 · see citing works
Meta-learning in Reinforcement Learning
Schweighofer, N., & Doya, K. (2003) · Neural Networks · doi:10.1016/s0893-6080(02)00228-9 · Cited by 261 · see citing works
Can people behave "randomly?": The role of feedback.
Neuringer, A. J. (1986) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.115.1.62 · Cited by 261 · see citing works
Learning Behavior in an Experimental Matching Pennies Game
Mookherjee, D., & Sopher, B. (1994) · Games and Economic Behavior · doi:10.1006/game.1994.1037 · Cited by 259 · see citing works
Huang, Y.-Y., Simpson, E., Kellendonk, C., & Kandel, E. R. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0308280101 · Cited by 231 · see citing works
A Computational Model of How the Basal Ganglia Produce Sequences
Berns, G. S., & B, T. J. S. A. (1998) · Journal of Cognitive Neuroscience · doi:10.1162/089892998563815 · Cited by 231 · see citing works
Graded bidirectional synaptic plasticity is composed of switch-like unitary events
O’Connor, D. H., Wittenberg, G. M., & Wang, S. S. ‐. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0502332102 · Cited by 227 · see citing works
Generation of random series in two-person strictly competitive games.
Rapoport, A., & Budescu, D. V. (1992) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.121.3.352 · Cited by 220 · see citing works
Dopaminergic Modulation of Long-term Synaptic Plasticity in Rat Prefrontal Neurons
Otani, S. (2003) · Cerebral Cortex · doi:10.1093/cercor/bhg092 · Cited by 193 · see citing works
Reinforcement learning and decision making in monkeys during a competitive game
Lee, D., Conroy, M. L., McGreevy, B. P., & Barraclough, D. J. (2004) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2004.07.007 · Cited by 156 · see citing works
Self-citationHebbian spike-driven synaptic plasticity for learning patterns of mean firing rates
Fusi, S. (2002) · Biological Cybernetics · doi:10.1007/s00422-002-0356-8 · Cited by 131 · see citing works
A Biophysically Based Neural Model of Matching Law Behavior: Melioration by Stochastic Synapses
Soltani, A., & Wang, X. (2006) · Journal of Neuroscience · doi:10.1523/jneurosci.5159-05.2006 · Cited by 127 · see citing works
Self-citationLearning and decision making in monkeys during a rock–paper–scissors game
Lee, D., McGreevy, B. P., & Barraclough, D. J. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2005.07.003 · Cited by 123 · see citing works
Self-citationMathematical Methods in Small Group Processes.
Stone, M., Criswell, J. H., Solomon, H., et al. (1963) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2982231 · Cited by 107 · see citing works
Neural basis of quasi-rational decision making
Lee, D. (2006) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2006.02.001 · Cited by 98 · see citing works
Self-citationMathematical Methods in Small Group Processes.
Rapoport, A., Criswell, J. H., Solomon, H., & Suppes, P. (1963) · Journal of the American Statistical Association · doi:10.2307/2282732 · Cited by 75 · see citing works
Interdependent decision strategies in zero-sum games: A computer-controlled study
Messick, D. M. (1967) · Systems Research and Behavioral Science · doi:10.1002/bs.3830120105 · Cited by 61 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Camerer CF. Behavioral game theory: Experiments in strategic interaction. Princeton: Princeton Univ. Press; 2003. [Google Scholar]
- Stephens DW, Krebs JR. Foraging theory. Princeton: Princeton Univ. Press; 1986. [Google Scholar]
Cited by
Sorted by citations
Neuronal Reward and Decision Signals: From Theories to Data
Schultz, W. (2015) · Physiological Reviews · doi:10.1152/physrev.00023.2014 · Cited by 1,126 · see citing works
Decision Making in Recurrent Neuronal Circuits
Wang, X.-J. (2008) · Neuron · doi:10.1016/j.neuron.2008.09.034 · Cited by 679 · see citing works
“Therefore, a decision maker does not have a goal to play randomly, but simply tries to play at its best, given the environment or other decision agents in an interactive game (Soltani et al., 2006; Lee and Wang, 2008).”
BackgroundInfluentialSelf-citationLearning to reinforcement learn
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M. (2016) · arXiv · doi:10.48550/arxiv.1611.05763 · Cited by 579 · see citing works
Seo, H., & Lee, D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.2369-07.2007 · Cited by 324 · see citing works
Self-citationA reservoir of time constants for memory traces in cortical neurons
Bernacchia, A., Seo, H., Lee, D., & Wang, X.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2752 · Cited by 292 · see citing works
“For example, f(3,4) describes the effect of a reward after 3 trials during epoch 4.”
BackgroundSelf-citationFusi, S., Asaad, W. F., Miller, E. K., & Wang, X.-J. (2007) · Neuron · doi:10.1016/j.neuron.2007.03.017 · Cited by 199 · see citing works
Self-citationDynamic Signals Related to Choices and Outcomes in the Dorsolateral Prefrontal Cortex
Seo, H., Barraclough, D. J., & Lee, D. (2007) · Cerebral Cortex · doi:10.1093/cercor/bhm064 · Cited by 143 · see citing works
Self-citationLateral Intraparietal Cortex and Reinforcement Learning during a Mixed-Strategy Game
Seo, H., Barraclough, D. J., & Lee, D. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.1479-09.2009 · Cited by 125 · see citing works
Self-citationReinforcement learning: Computational theory and biological mechanisms
Doya, K. (2007) · HFSP journal · doi:10.2976/1.2732246 · Cited by 150 · see citing works
Intelligence plays dice: Stochasticity is essential for machine learning
Sabuncu, M. R. (2020) · arXiv · doi:10.48550/arxiv.2008.07496 · Cited by 3 · see citing works
- Authors
- Soltani, A., Lee, D., & Wang, X.-J.
- Affiliations
- Yale University (US)
- Corresponding author
- Wang, Xiao-Jing
- Venue
- Neural Networks
- Publisher
- Elsevier BV
- Published
- 2006-10
- Volume
- 19
- Issue
- 8
- Pages
- 1075-1090
- ISSN
- 0893-6080 (ISSN-L)
- PMC
- PMC1752206
- OpenAlex
- W2030612334
- Cited by
- 77
- References
- 52
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 4 · 2023: 1 · 2022: 4 · 2021: 2 · 2020: 3 · 2019: 2 · 2018: 4 · 2017: 7 · 2016: 5 · 2015: 3 · 2014: 1 · 2013: 6 · 2012: 4
- FWCI
- 3.24
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Neural Networks (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (MH073246, R01MH073246)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- stochastic choice, competitive games, matching pennies, decision-making, synaptic plasticity, reinforcement learning, non-human primates, meta-learning, reward maximization, neural models
- MeSH terms
- Algorithms, Animals, Haplorhini, Behavior, Animal, Choice Behavior, Competitive Behavior, Humans, Models, Neurological, Neural Inhibition, Neurons, Probability Learning, Reaction Time, Reinforcement, Psychology, Stochastic Processes, Likelihood Functions, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
