Reinforcement learning: Computational theory and biological mechanisms
Open accessAbstract
Reinforcement learning is a computational framework for an active agent to learn behaviors on the basis of a scalar reward signal. The agent can be an animal, a human, or an artificial system such as a robot or a computer program. The reward can be food, water, money, or whatever measure of the performance of the agent. The theory of reinforcement learning, which was developed in an artificial intelligence community with intuitions from animal learning theory, is now giving a coherent account on the function of the basal ganglia. It now serves as the "common language" in which biologists, engineers, and social scientists can exchange their problems and findings. This article reviews the basic theoretical framework of reinforcement learning and discusses its recent and future contributions toward the understanding of animal behaviors and human decision making.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The striatum is ideally suited for reinforcement learning because the conditions required for long-term potentiation at cortico-striatal synapses closely match the conditions for reinforcement learning.
“The striatum is ideally suited for such learning because the conditions required for LTP at cortico-striatal synapses closely match the conditions for reinforcement learning [34–36].”
From Cortical and basal ganglia contributions to habit learning and automaticity (2010) · cites this paper as [34]
Dopamine cells exhibit phasic increases in discharge rate following unexpected primary or conditioned rewards and depress their discharge below baseline following an unexpected failure to receive a reward.
“An extensive literature shows that dopamine cells exhibit phasic increases in discharge rate following unexpected primary or conditioned rewards and depress their discharge below baseline following an unexpected failure to receive a reward [34, 36].”
From Cortical and basal ganglia contributions to habit learning and automaticity (2010) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Watkins, C., & Dayan, P. (1992) · Machine-mediated learning · doi:10.1023/a:1022676722315 · Cited by 14,277 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083 · see citing works
Quantum phase transition from a superfluid to a Mott insulator in a gas of ultracold atoms
Greiner, M., Mandel, O., Esslinger, T., Hänsch, T. W., & Bloch, I. (2002) · Nature · doi:10.1038/415039a · Cited by 5,935 · see citing works
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,406 · see citing works
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · see citing works
Predictive reward signal of dopamine neurons
Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.1.1 · Cited by 4,564 · see citing works
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · see citing works
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,238 · see citing works
Internal models for motor control and trajectory planning.
Kawato, M. (1999) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(99)00028-8 · Cited by 2,685 · see citing works
The Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588 · see citing works
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
Barto, A. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,172 · see citing works
Discrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085 · see citing works
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · see citing works
Zhang, B., Li, C., & Lin, N. (2020) · De Gruyter eBooks · doi:10.1515/9783110595567-011 · Cited by 1,939 · see citing works
Uncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774 · see citing works
Neural correlates of decision variables in parietal cortex
Platt, M. L., & Glimcher, P. W. (1999) · Nature · doi:10.1038/22268 · Cited by 1,721 · see citing works
Animal Intelligence: An Experimental Study of the Associative Processes in Animals
Kline, L. W., & Thorndike, E. L. (1898) · The American Journal of Psychology · doi:10.2307/1412682 · Cited by 1,105 · see citing works
Matching Behavior and the Representation of Value in the Parietal Cortex
Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004) · Science · doi:10.1126/science.1094765 · Cited by 1,030 · see citing works
Dopamine responses comply with basic assumptions of formal learning theory
Waelti, P., Dickinson, A., & Schultz, W. (2001) · Nature · doi:10.1038/35083500 · Cited by 1,018 · see citing works
Complementary roles of basal ganglia and cerebellum in learning and motor control
Doya, K. (2000) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(00)00153-7 · Cited by 964 · see citing works
Self-citationReward processing in primate orbitofrontal cortex and basal ganglia
Schultz, W. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.272 · Cited by 942 · see citing works
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939 · see citing works
Self-citationA cellular mechanism of reward-related learning
Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001) · Nature · doi:10.1038/35092560 · Cited by 879 · see citing works
Opponent interactions between serotonin and dopamine
Daw, N. D., Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00052-7 · Cited by 844 · see citing works
The cerebellum communicates with the basal ganglia
Hoshi, E., Tremblay, L., Féger, J., Carras, P. L., & Strick, P. L. (2005) · Nature Neuroscience · doi:10.1038/nn1544 · Cited by 833 · see citing works
Bayesian brain : probabilistic approaches to neural coding
賢治, 銅., 信, 石., Pouget, A., & Rao, R. P. N. (2006) · The MIT Press eBooks · doi:10.7551/mitpress/9780262042383.001.0001 · Cited by 823 · see citing works
TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play
Tesauro, G. J. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.215 · Cited by 803 · see citing works
Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004) · Nature Neuroscience · doi:10.1038/nn1279 · Cited by 779 · see citing works
Self-citationThe short-latency dopamine signal: a role in discovering novel actions?
Redgrave, P., & Gurney, K. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn2022 · Cited by 762 · see citing works
What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?
Doya, K. (1999) · Neural Networks · doi:10.1016/s0893-6080(99)00046-5 · Cited by 739 · see citing works
Self-citationNeuroeconomics: The Consilience of Brain and Decision
Glimcher, P. W., & Rustichini, A. (2004) · Science · doi:10.1126/science.1102566 · Cited by 725 · see citing works
Metalearning and neuromodulation
Doya, K. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00044-8 · Cited by 704 · see citing works
Self-citationReward expectancy in primate prefrental neurons
Watanabe, M. (1996) · Nature · doi:10.1038/382629a0 · Cited by 684 · see citing works
Prefrontal cortex and decision making in a mixed-strategy game
Barraclough, D. J., Conroy, M. L., & Lee, D. (2004) · Nature Neuroscience · doi:10.1038/nn1209 · Cited by 678 · see citing works
Functional Properties of Brain Areas Associated With Motor Execution and Imagery
Hanakawa, T., Immisch, I., Toma, K., Dimyan, M. A., van Gelderen, P., & Hallett, M. (2003) · Journal of Neurophysiology · doi:10.1152/jn.00132.2002 · Cited by 677 · see citing works
Sawaguchi, T., & Goldman‐Rakic, P. S. (1994) · Journal of Neurophysiology · doi:10.1152/jn.1994.71.2.515 · Cited by 649 · see citing works
Expectation of reward modulates cognitive signals in the basal ganglia
Kawagoe, R., Takikawa, Y., & Hikosaka, O. (1998) · Nature Neuroscience · doi:10.1038/1625 · Cited by 632 · see citing works
Parallel incentive processing: an integrated view of amygdala function
Balleine, B. W., & Killcross, S. (2006) · Trends in Neurosciences · doi:10.1016/j.tins.2006.03.002 · Cited by 622 · see citing works
A Menu of Designs for Reinforcement Learning Over Time
Werbos, P. J. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/4939.003.0007 · Cited by 609 · see citing works
Houk, J. C., & Wise, S. P. (1995) · Cerebral Cortex · doi:10.1093/cercor/5.2.95 · Cited by 594 · see citing works
Neuroeconomics: cross-currents in research on decision-making
Sanfey, A. G., Loewenstein, G. F., McClure, S. M., & Cohen, J. D. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.01.009 · Cited by 557 · see citing works
A Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement
Houk, J. C., Adams, J. L., & Barto, A. G. (1994) · The MIT Press eBooks · doi:10.7551/mitpress/4708.003.0020 · Cited by 555 · see citing works
Minsky, M. (1991) · Artificial Intelligence · doi:10.1016/0004-3702(91)90036-j · Cited by 555 · see citing works
Dopamine-dependent plasticity of corticostriatal synapses
Reynolds, J. N. J., & Wickens, J. R. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00045-x · Cited by 528 · see citing works
Midbrain dopamine neurons encode decisions for future action
Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006) · Nature Neuroscience · doi:10.1038/nn1743 · Cited by 503 · see citing works
Adaptive Critics and the Basal Ganglia
Barto, A. G. (1994) · The MIT Press eBooks · doi:10.7551/mitpress/4708.003.0018 · Cited by 466 · see citing works
The computational neurobiology of learning and reward
Daw, N. D., & Doya, K. (2006) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2006.03.006 · Cited by 462 · see citing works
Self-citationDorris, M. C., & Glimcher, P. W. (2004) · Neuron · doi:10.1016/j.neuron.2004.09.009 · Cited by 428 · see citing works
Neuronal correlates of goal-based motor selection in the prefrontal cortex.
Matsumoto, K., Suzuki, W., & Tanaka, K. (2003) · Science · doi:10.1126/science.1084204 · Cited by 384 · see citing works
Basal ganglia orient eyes to reward
Hikosaka, O., Nakamura, K., & Nakahara, H. (2006) · Journal of Neurophysiology · doi:10.1152/jn.00458.2005 · Cited by 377 · see citing works
Numerical representation for action in the parietal cortex of the monkey
Sawamura, H., Shima, K., & Tanji, J. (2002) · Nature · doi:10.1038/415918a · Cited by 323 · see citing works
Deiber, M.-P., Ibañez, V., Honda, M., Sadato, N., Raman, R., & Hallett, M. (1998) · NeuroImage · doi:10.1006/nimg.1997.0314 · Cited by 304 · see citing works
Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons
Satoh, T., Nakai, S., Sato, T., & Kimura, M. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-30-09913.2003 · Cited by 298 · see citing works
Wickens, J. R., Begg, A., & Arbuthnott, G. W. (1996) · Neuroscience · doi:10.1016/0306-4522(95)00436-m · Cited by 294 · see citing works
Winstanley, C. A., Theobald, D. E. H., Dalley, J. W., Cardinal, R. N., & Robbins, T. W. (2005) · Cerebral Cortex · doi:10.1093/cercor/bhi088 · Cited by 279 · see citing works
Dopamine Neurons Can Represent Context-Dependent Prediction Error
Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004) · Neuron · doi:10.1016/s0896-6273(03)00869-9 · Cited by 276 · see citing works
Cellular, synaptic and network effects of neuromodulation
Marder, E. E., & Thirumalai, V. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00043-6 · Cited by 273 · see citing works
Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning
Morimoto, J., & Doya, K. (2001) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(01)00113-0 · Cited by 262 · see citing works
Self-citationLearning of sequential movements by neural network model with dopamine-like reinforcement signal
Suri, R. E., & Schultz, W. (1998) · Experimental Brain Research · doi:10.1007/s002210050467 · Cited by 198 · see citing works
Resolution of Uncertainty in Prefrontal Cortex
Yoshida, W., & Ishii, S. (2006) · Neuron · doi:10.1016/j.neuron.2006.05.006 · Cited by 196 · see citing works
Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004) · Journal of Neurophysiology · doi:10.1152/jn.00721.2003 · Cited by 158 · see citing works
Reinforcement learning and decision making in monkeys during a competitive game
Lee, D., Conroy, M. L., McGreevy, B. P., & Barraclough, D. J. (2004) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2004.07.007 · Cited by 156 · see citing works
Reward-related Signals Carried by Dopamine Neurons
Schultz, W., Romo, R., Ljungberg, T., Mirenowicz, J., Hollerman, J. R., & Dickinson, A. (1994) · The MIT Press eBooks · doi:10.7551/mitpress/4708.003.0019 · Cited by 154 · see citing works
Houk, J. C. (2005) · Biological Cybernetics · doi:10.1007/s00422-005-0569-8 · Cited by 138 · see citing works
Reynolds, J. N. J., & Wickens, J. R. (2000) · Neuroscience · doi:10.1016/s0306-4522(00)00273-6 · Cited by 131 · see citing works
Learning and decision making in monkeys during a rock–paper–scissors game
Lee, D., McGreevy, B. P., & Barraclough, D. J. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2005.07.003 · Cited by 123 · see citing works
A Predictive Reinforcement Model of Dopamine Neurons for Learning Approach Behavior
Contreras-Vidal, J. L., & Schultz, W. (1999) · Journal of Computational Neuroscience · doi:10.1023/a:1008862904946 · Cited by 102 · see citing works
Exploration Bonuses and Dual Control
Dayan, P., & Sejnowski, T. J. (1996) · Machine Learning · doi:10.1023/a:1018357105171 · Cited by 92 · see citing works
Bayesian brain probabilistic approaches to neural coding
Doya, K. (2007) · Cited by 88 · see citing works
Self-citationNeural mechanism for stochastic behaviour during a competitive game
Soltani, A., Lee, D., & Wang, X.-J. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.05.044 · Cited by 77 · see citing works
Doya, K., & Uchibe, E. (2005) · Adaptive Behavior · doi:10.1177/105971230501300206 · Cited by 70 · see citing works
Self-citationAnimal intelligence: An experimental study of the associate processes in animals.
Thorndike, E. L. (1998) · American Psychologist · doi:10.1037/0003-066x.53.10.1125 · Cited by 69 · see citing works
Hermann Von Helmholtz and the new psychology.
Stumpf, C. (1895) · Psychological Review · doi:10.1037/h0070633 · Cited by 36 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [64]64. Watkins, C JCH, and Dayan, P (1992). “Q-learning.” Mach. Learn. 8, 279–292. [[Google Scholar](https://scholar.google.com/scholar_lookup?journal=Mach.%20Learn.&title=Q-learning&author=C%20JCH%20Watkins&author=P%20Dayan&volume=8&publication_year=1992&pages=279-292&issn=0885-6125&)]
Cited by
Sorted by citations
Doya, K. (2008) · Nature Neuroscience · doi:10.1038/nn2077 · Cited by 642 · see citing works
Self-citationCortical and basal ganglia contributions to habit learning and automaticity
Ashby, F. G., Turner, B. O., & Horvitz, J. C. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.02.001 · Cited by 481 · see citing works
Evidence for model-based action planning in a sequential finger movement task
Fermin, A., Yoshida, T., Ito, M., Yoshimoto, J., & Doya, K. (2010) · Journal of Motor Behavior · doi:10.1080/00222895.2010.526467 · Cited by 36 · see citing works
Self-citationBlaming luck, claiming skill: Self-attribution bias in error assignment
Okamoto, N., Taylor, M., Kubo, T., Ishii, S., De Martino, B., & Cortese, A. (2025) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1013787 · Cited by 2 · see citing works
- Authors
- Doya, K.
- Affiliations
- Okinawa Institute of Science and Technology Graduate University (JP)
- Corresponding author
- Doya, Kenji
- ORCID
- Doya, Kenji
- Venue
- HFSP journal
- Publisher
- Informa UK Limited
- Published
- 2007-05
- Volume
- 1
- Issue
- 1
- Pages
- 30-40
- ISSN
- 1955-205X (ISSN-L)
- 1955-2068
- PMC
- PMC2645553
- OpenAlex
- W2091789872
- Cited by
- 150
- References
- 75
- Citations by year
- 2026: 7 · 2025: 11 · 2024: 8 · 2023: 7 · 2022: 6 · 2021: 9 · 2020: 6 · 2019: 6 · 2018: 10 · 2017: 4 · 2016: 12 · 2015: 5 · 2014: 13 · 2013: 6 · 2012: 12
- FWCI
- 7.06
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- HFSP journal (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- reinforcement learning, basal ganglia, reward signals, animal learning, human decision-making, theory of computation, biological mechanisms, artificial intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
