Dopamine, uncertainty and TD learning
Open accessAbstract
Substantial evidence suggests that the phasic activities of dopaminergic neurons in the primate midbrain represent a temporal difference (TD) error in predictions of future reward, with increases above and decreases below baseline consequent on positive and negative prediction errors, respectively. However, dopamine cells have very low baseline activity, which implies that the representation of these two sorts of error is asymmetric. We explore the implications of this seemingly innocuous asymmetry for the interpretation of dopaminergic firing patterns in experiments with probabilistic rewards which bring about persistent prediction errors. In particular, we show that when averaging the non-stationary prediction errors across trials, a ramping in the activity of the dopamine neurons should be apparent, whose magnitude is dependent on the learning rate. This exact phenomenon was observed in a recent experiment, though being interpreted there in antipodal terms as a within-trial encoding of uncertainty.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
It is exactly these neuromodulatory systems that have been implicated in value-learning, attention and the encoding of uncertainty.
“Furthermore, it is exactly these systems that have been implicated in value-learning [68]–[70], attention and the encoding of uncertainty [63],[71].”
From Hierarchical Models in the Brain (2008) · cites this paper as [71]
The Introduction links dopamine to learning, and the Discussion says uncertainty may influence cortical inference and learning through other neuromodulatory systems. But the paper argues that the observed dopamine ramps may not directly encode uncertainty, and it does not itself establish a role for these systems in attention. No page numbers are available in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredLateral interactions may encode the relative precision of prediction errors and change in a way that is consistent with classical neuromodulation.
“Lateral interactions may encode the relative precision of prediction errors and change in a way that is consistent with classical neuromodulation (c.f., [63],[71]).”
From Hierarchical Models in the Brain (2008) · cites this paper as [71]
The Discussion (page unavailable) addresses asymmetric dopamine coding of prediction errors and mentions a possible opponent neurotransmitter. Searches for lateral interactions, relative precision, and neuromodulation found no passage saying that lateral interactions encode precision or change in the claimed way. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023 · see citing works
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · see citing works
Self-citationModels of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190 · see citing works
Computational roles for dopamine in behavioural control
Montague, P. R., Hyman, S. E., & Cohen, J. D. (2004) · Nature · doi:10.1038/nature03015 · Cited by 966 · see citing works
Temporal difference models describe higher-order learning in humans
Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S. (2004) · Nature · doi:10.1038/nature02581 · Cited by 605 · see citing works
Self-citationCoincident but Distinct Messages of Midbrain Dopamine and Striatal Tonically Active Neurons
Morris, G., Arkadir, D., Nevet, A., Vaadia, E., & Bergman, H. (2004) · Neuron · doi:10.1016/j.neuron.2004.06.012 · Cited by 574 · see citing works
Flagged first, then by citations
Research works printed but not linked: 18
- Ljungberg T Apicella P Schultz W Responses of monkey dopamine neurons during learning of behavioral reactions Journal Neurophysiol 1992 67 145 163 1552316
- Schultz W Predictive reward signal of dopamine neurons Journal of Neurophysiology 1998 80 1 27 9658025
- Montague PR Dayan P Sejnowski TJ A framework for mesencephalic dopamine systems based on predictive Hebbian learning The Journal of Neuroscience 1996 16 1936 1947 8774460 PMC6578666
- Schultz W Dayan P Montague PR A neural substrate of prediction and reward Science 1997 275 1593 1599 9054347 10.1126/science.275.5306.1593 · DOI · Paper page
- Sutton RS Barto AG Reinforcement learning: An introduction 1998 MIT Press
- Hollerman J Schultz W Dopamine neurons report an error in the temporal prediction of reward during learning Nature Neuroscience 1998 1 304 309 10195164 10.1038/1124 · DOI
- Schultz W Apicella P Ljungberg T Responses of monkey dopamine neurons to reward and conditioned stimuli during succesive steps of learning a delayed response task The Journal of Neuroscience 1993 13 900 913 8441015 PMC6576600
- Tobler P Dickinson A Schultz W Coding of Predicted Reward Omission by Dopamine Neurons in a Conditioned Inhibition Paradigm Journal of Neuroscience 2003 23 10402 10410 14614099 PMC6741002
- Takikawa Y Kawagoe R Hikosaka O A possible role of midbrain dopamine neurons in short- and long-term adaptation of saccades to position-reward mapping Journal of Neurophysiology 2004 92 2520 2529 15163669 10.1152/jn.00238.2004 · DOI
- Bayer H A role for the substantia nigra in learning and motor control PhD thesis, New York University 2004
- Fiorillo C Tobler P Schultz W Discrete Coding of Reward Probability and Uncertainty by Dopamine Neurons Science 2003 299 1898 1902 12649484 10.1126/science.1077349 · DOI · Paper page
- Barto A Sutton R Watkins C Gabriel M, Moore J Learning and sequntial decision making Learning and Computational Neuroscience: Foundations of Adaptive Networks 1990 Cambridge, MA: MIT Press 539 602
- Nakahara H Itoh H Kawagoe R Takikawa Y Hikosaka O Dopamine neurons can represent context-dependent prediction error Neuron 2004 41 269 280 14741107 10.1016/S0896-6273(03)00869-9 · DOI
- Gallistel CR Gibbon J Time, rate and conditioning Psychological Review 2000 107 289 344 10789198 10.1037//0033-295X.107.2.289 · DOI
- Daw ND Kakade S Dayan P Opponent interactions between serotonin and dopamine Neural Networks 2002 15 603 616 12371515 10.1016/S0893-6080(02)00052-7 · DOI · Paper page
- Suri RE Schultz W A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task Neuroscience 1999 91 871 890 10391468 10.1016/S0306-4522(98)00697-6 · DOI
- Dayan P Kakade S Montague PR Learning and selective attention Nature Neuroscience 2000 3 1218 1223 11127841 10.1038/81504 · DOI · Paper page
- Dayan P Yu A Dietterich T, Becker S, Ghahramani Z Expected and unexpected uncertainty: Ach and NE in the neocortex Advances in Neural Information Processing Sysytems 2002 14 Cambridge, MA: MIT Press 189 196
Cited by
Sorted by citations
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · see citing works
Habits, Rituals, and the Evaluative Brain
Graybiel, A. M. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.112851 · Cited by 1,697 · see citing works
Neuronal Reward and Decision Signals: From Theories to Data
Schultz, W. (2015) · Physiological Reviews · doi:10.1152/physrev.00023.2014 · Cited by 1,126 · see citing works
Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis
Glimcher, P. W. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1014269108 · Cited by 955 · see citing works
Hierarchical Models in the Brain
Friston, K. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000211 · Cited by 952 · see citing works
Orbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885 · see citing works
Self-citationNeural differentiation of expected reward and risk in human subcortical structures
Preuschoff, K., Bossaerts, P., & Quartz, S. R. (2006) · Neuron · doi:10.1016/j.neuron.2006.06.024 · Cited by 692 · see citing works
Risky business: the neuroeconomics of decision making under uncertainty
Platt, M. L., & Huettel, S. A. (2008) · Nature Neuroscience · doi:10.1038/nn2062 · Cited by 556 · see citing works
Prolonged dopamine signalling in striatum signals proximity and value of distant rewards
Howe, M. W., Tierney, P. L., Sandberg, S. G., Phillips, P. E. M., & Graybiel, A. M. (2013) · Nature · doi:10.1038/nature12475 · Cited by 565 · see citing works
Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain
Niv, Y., Edlund, J. A., Dayan, P., & O'Doherty, J. P. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.5498-10.2012 · Cited by 356 · see citing works
Self-citationKnowing how much you don't know: a neural organization of uncertainty estimates
Bach, D. R., & Dolan, R. J. (2012) · Nature Reviews Neuroscience · doi:10.1038/nrn3289 · Cited by 339 · see citing works
Adaptive learning under expected and unexpected uncertainty
Soltani, A., & Izquierdo, A. (2019) · Nature reviews. Neuroscience · doi:10.1038/s41583-019-0180-y · Cited by 286 · see citing works
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224 · see citing works
Representation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · see citing works
Neural coding of distinct statistical properties of reward information in humans
Dreher, J.-C., Kohn, P., & Berman, K. F. (2006) · Cerebral Cortex · doi:10.1093/cercor/bhj004 · Cited by 180 · see citing works
Stimulus representation and the timing of reward-prediction errors in models of the dopamine system
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008) · Neural Computation · doi:10.1162/neco.2008.11-07-654 · Cited by 148 · see citing works
Adding prediction risk to the theory of reward learning
PREUSCHOFF, K., & BOSSAERTS, P. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.005 · Cited by 133 · see citing works
Computational and dynamic models in neuroimaging
Friston, K. J., & Dolan, R. J. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.12.068 · Cited by 131 · see citing works
Dynamic shaping of dopamine signals during probabilistic Pavlovian conditioning
Hart, A. S., Clark, J. J., & Phillips, P. E. M. (2015) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2014.07.010 · Cited by 82 · see citing works
Dopamine, inference, and uncertainty
Gershman, S. J. (2017) · Neural Computation · doi:10.1162/neco_a_01023 · Cited by 40 · see citing works
- Authors
- Niv, Y., Duff, M. O., & Dayan, P.
- Affiliations
- Hebrew University of Jerusalem (IL)
- Gatsby Computational Neuroscience Unit (GB)
- Corresponding author
- Niv, Yael
- Venue
- Behavioral and Brain Functions
- Published
- 2005-05-04
- Volume
- 1
- Issue
- 1
- Pages
- 6-6
- ISSN
- 1744-9081 (ISSN-L)
- PMC
- PMC1171969
- OpenAlex
- W2159088729
- Cited by
- 153
- References
- 6
- Citations by year
- 2026: 9 · 2025: 4 · 2024: 3 · 2023: 5 · 2022: 9 · 2021: 6 · 2020: 6 · 2019: 4 · 2018: 10 · 2017: 4 · 2016: 4 · 2015: 6 · 2014: 9 · 2013: 7 · 2012: 6
- FWCI
- 4.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Behavioral and Brain Functions (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Funders
- Gatsby Charitable Foundation
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- dopaminergic neurons, temporal difference learning, prediction error, uncertainty, probabilistic reward, learning rate, reward prediction, midbrain dopaminergic neurons
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
