Dopamine neurons learn to encode the long-term value of multiple future rewards
Open accessAbstract
Midbrain dopamine neurons signal reward value, their prediction error, and the salience of events. If they play a critical role in achieving specific distant goals, long-term future rewards should also be encoded as suggested in reinforcement learning theories. Here, we address this experimentally untested issue. We recorded 185 dopamine neurons in three monkeys that performed a multistep choice task in which they explored a reward target among alternatives and then exploited that knowledge to receive one or two additional rewards by choosing the same target in a set of subsequent trials. An analysis of anticipatory licking for reward water indicated that the monkeys did not anticipate an immediately expected reward in individual trials; rather, they anticipated the sum of immediate and multiple future rewards. In accordance with this behavioral observation, the dopamine responses to the start cues and reinforcer beeps reflected the expected values of the multiple future rewards and their errors, respectively. More specifically, when monkeys learned the multistep choice task over the course of several weeks, the responses of dopamine neurons encoded the sum of the immediate and expected multiple future rewards. The dopamine responses were quantitatively predicted by theoretical descriptions of the value function with time discounting in reinforcement learning. These findings demonstrate that dopamine neurons learn to encode the long-term value of multiple future rewards with distant rewards discounted.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Dopamine responses in elaborate tasks constitute standard reward prediction-error signals.
“However, on closer inspection, dopamine responses in such elaborate tasks constitute standard reward prediction-error signals2,33,59,66,69,75,151–153.”
From Dopamine reward prediction-error signalling: A two-component response (2016) · cites this paper as [75]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524 · see citing works
Prospect theory: An analysis of decision under risk
Kahneman, D. J., & Tversky, A. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511609220.014 · Cited by 32,700 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Separate neural systems value immediate and delayed monetary rewards
McClure, S. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2004) · Science · doi:10.1126/science.1100907 · Cited by 3,089 · see citing works
An adjusting procedure for studying delayed reinforcement.
Mazur, J. E. (1987) · doi:10.4324/9781315825502-11 · Cited by 2,189 · see citing works
Barto, A. (1998) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)38315-5 · Cited by 2,172 · see citing works
Discrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085 · see citing works
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · see citing works
Zhang, B., Li, C., & Lin, N. (2020) · De Gruyter eBooks · doi:10.1515/9783110595567-011 · Cited by 1,939 · see citing works
Two types of dopamine neuron distinctly convey positive and negative motivational signals
Matsumoto, M., & Hikosaka, O. (2009) · Nature · doi:10.1038/nature08028 · Cited by 1,378 · see citing works
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · see citing works
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939 · see citing works
Self-citationPrediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004) · Nature Neuroscience · doi:10.1038/nn1279 · Cited by 779 · see citing works
Roesch, M. R., Calu, D. J., & Schoenbaum, G. (2007) · Nature Neuroscience · doi:10.1038/nn2013 · Cited by 597 · see citing works
Coincident but Distinct Messages of Midbrain Dopamine and Striatal Tonically Active Neurons
Morris, G., Arkadir, D., Nevet, A., Vaadia, E., & Bergman, H. (2004) · Neuron · doi:10.1016/j.neuron.2004.06.012 · Cited by 574 · see citing works
Midbrain Dopamine Neurons Signal Preference for Advance Information about Upcoming Rewards
Bromberg-Martin, E. S., & Hikosaka, O. (2009) · Neuron · doi:10.1016/j.neuron.2009.06.009 · Cited by 548 · see citing works
Is the short-latency dopamine response too short to signal reward error?
Redgrave, P., Prescott, T. J., & Gurney, K. (1999) · Trends in Neurosciences · doi:10.1016/s0166-2236(98)01373-3 · Cited by 506 · see citing works
Midbrain dopamine neurons encode decisions for future action
Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006) · Nature Neuroscience · doi:10.1038/nn1743 · Cited by 503 · see citing works
(1997) · The MIT Press eBooks · doi:10.7551/mitpress/3917.003.0017 · Cited by 497 · see citing works
Influence of Reward Delays on Responses of Dopamine Neurons
Kobayashi, S., & Schultz, W. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.1600-08.2008 · Cited by 431 · see citing works
Quantitative Analyses of Behavior
(2019) · Psychology Press eBooks · doi:10.4324/9781315788999 · Cited by 422 · see citing works
Value Representations in the Primate Striatum during Matching Behavior
Lau, B., & Glimcher, P. W. (2008) · Neuron · doi:10.1016/j.neuron.2008.02.021 · Cited by 411 · see citing works
Kable, J. W., & Glimcher, P. W. (2010) · Journal of Neurophysiology · doi:10.1152/jn.00177.2009 · Cited by 316 · see citing works
The temporal precision of reward prediction in dopamine neurons
Fiorillo, C. D., Newsome, W. T., & Schultz, W. (2008) · Nature Neuroscience · doi:10.1038/nn.2159 · Cited by 310 · see citing works
Dopamine, Time, and Impulsivity in Humans
Pine, A., Shiner, T., Seymour, B., & Dolan, R. J. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.6028-09.2010 · Cited by 300 · see citing works
Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons
Satoh, T., Nakai, S., Sato, T., & Kimura, M. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-30-09913.2003 · Cited by 298 · see citing works
Self-citationEncoding of Time-Discounted Rewards in Orbitofrontal Cortex Is Independent of Value Representation
Roesch, M. R., Taylor, A. R., & Schoenbaum, G. (2006) · Neuron · doi:10.1016/j.neuron.2006.06.027 · Cited by 297 · see citing works
Denk, F., Walton, M. E., Jennings, K. A., Sharp, T., Rushworth, M. F. S., & Bannerman, D. M. (2004) · Psychopharmacology · doi:10.1007/s00213-004-2059-4 · Cited by 294 · see citing works
Dopamine Neurons Can Represent Context-Dependent Prediction Error
Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004) · Neuron · doi:10.1016/s0896-6273(03)00869-9 · Cited by 276 · see citing works
Limbic Corticostriatal Systems and Delayed Reinforcement
Cardinal, R. N., Winstanley, C. A., Robbins, T. W., & Everitt, B. J. (2004) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1308.004 · Cited by 266 · see citing works
Prefrontal Coding of Temporally Discounted Values during Intertemporal Choice
Kim, S., Hwang, J., & Lee, D. (2008) · Neuron · doi:10.1016/j.neuron.2008.05.010 · Cited by 241 · see citing works
Separating Value from Choice: Delay Discounting Activity in the Lateral Intraparietal Area
Louie, K., & Glimcher, P. W. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.5742-09.2010 · Cited by 182 · see citing works
Shaping of Motor Responses by Incentive Values through the Basal Ganglia
Pasquereau, B., Nadjar, A., Arkadir, D., Bézard, E., Goillandeau, M., Bioulac, B. H., Gross, C. E., & Boraud, T. (2007) · Journal of Neuroscience · doi:10.1523/jneurosci.3745-06.2007 · Cited by 146 · see citing works
Normative and Descriptive Models of Decision Making: Time Discounting and Risk Sensitivity
Kacelnik, A. (2007) · Novartis Foundation symposium · doi:10.1002/9780470515372.ch5 · Cited by 115 · see citing works
Dopamine, Learning, and Impulsivity: ABiological Account of Attention-Deficit/Hyperactivity Disorder
Williams, J. H., & Dayan, P. (2005) · Journal of Child and Adolescent Psychopharmacology · doi:10.1089/cap.2005.15.160 · Cited by 78 · see citing works
Neuronal Activity Representing Temporal Prediction of Reward in the Primate Prefrontal Cortex
Tsujimoto, S., & Sawaguchi, T. (2005) · Journal of Neurophysiology · doi:10.1152/jn.01149.2004 · Cited by 64 · see citing works
Dopamine neuronal responses in monkeys performing visually cued reward schedules
Ravel, S., & Richmond, B. J. (2006) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2006.04905.x · Cited by 41 · see citing works
Hyperbolically Discounted Temporal Difference Learning
Alexander, W. H., & Brown, J. W. (2010) · Neural Computation · doi:10.1162/neco.2010.08-09-1080 · Cited by 34 · see citing works
Normative and descriptive models of decision making: time discounting and risk sensitivity
Kacelnik, A. (2000) · doi:10.1093/oso/9780198524151.003.0003 · Cited by 24 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Neuronal Reward and Decision Signals: From Theories to Data
Schultz, W. (2015) · Physiological Reviews · doi:10.1152/physrev.00023.2014 · Cited by 1,126 · see citing works
Dopamine reward prediction-error signalling: A two-component response
Schultz, W. (2016) · Nature Reviews Neuroscience · doi:10.1038/nrn.2015.26 · Cited by 1,036 · see citing works
Updating dopamine reward signals
Schultz, W. (2013) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.11.012 · Cited by 651 · see citing works
Dopamine reward prediction error coding
Schultz, W. (2016) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2016.18.1/wschultz · Cited by 597 · see citing works
Neural Circuitry of Reward Prediction Error
Watabe-Uchida, M., Eshel, N., & Uchida, N. (2017) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-072116-031109 · Cited by 437 · see citing works
Acheson, D. T., Twamley, E. W., & Young, J. W. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00103 · Cited by 26 · see citing works
- Authors
- Enomoto, K., Matsumoto, N., Nakai, S., Satoh, T., Sato, T. K., Ueda, Y., Inokawa, H., Haruno, M., & Kimura, M.
- Affiliations
- Tamagawa University (JP)
- Prefectural University of Kumamoto (JP)
- Tokyo University of Science (JP)
- Kyoto Prefectural University of Medicine (JP)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2011-08-30
- Volume
- 108
- Issue
- 37
- Pages
- 15462-15467
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC3174584
- OpenAlex
- W2003565109
- Cited by
- 83
- References
- 39
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 2 · 2023: 1 · 2022: 1 · 2021: 2 · 2020: 9 · 2019: 4 · 2018: 6 · 2017: 10 · 2016: 10 · 2015: 8 · 2014: 8 · 2013: 7 · 2012: 5
- FWCI
- 2.78
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Japan Society for the Promotion of Science (23700503)
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- dopaminergic neurons, delay discounting, reinforcement learning, prediction error, value function, reward anticipation, midbrain dopamine
- MeSH terms
- Animals, Haplorhini, Behavior, Animal, Dopamine, Learning, Neurons, Reward, Time Factors, Anticipation, Psychological
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
