Dopamine, Inference, and Uncertainty
Open accessPublished version of a preprint
The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
The hypothesis that the phasic dopamine response reports a reward prediction error has become deeply entrenched. However, dopamine neurons exhibit several notable deviations from this hypothesis. A coherent explanation for these deviations can be obtained by analyzing the dopamine response in terms of Bayesian reinforcement learning. The key idea is that prediction errors are modulated by probabilistic beliefs about the relationship between cues and outcomes, updated through Bayesian inference. This account can explain dopamine responses to inferred value in sensory preconditioning, the effects of cue preexposure (latent inhibition), and adaptive coding of prediction errors when rewards vary across orders of magnitude. We further postulate that orbitofrontal cortex transforms the stimulus representation through recurrent dynamics, such that a simple error-driven learning rule operating on the transformed representation can implement the Bayesian reinforcement learning update.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Indeed, some dopamine neurons may signal not just PE or precision, but a combined precision-weighted PE(26):”
From Aberrant Salience, Information Processing, and Dopaminergic Signaling in People at Clinical High Risk for Psychosis (2020) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 4×
“(1997) in terms of the temporal difference (TD) learning algorithm (Sutton and Barto, 1998), which posits an error signal of the form:”
BackgroundInfluentialA neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 1×
“The phasic firing of dopamine neurons in the midbrain has long been thought to report a reward prediction error—the discrepancy between observed and expected reward—whose purpose is to correct future reward predictions (Eshel et al., 2015; Glimcher, 2011; Schultz et al., 1997).”
BackgroundAdvances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023 · cited 1×
“This mechanism harkens back to the classic Pearce-Hall theory of attention in associative learning (Pearce and Hall, 1980; Pearce et al., 1982), as well as to more recent Bayesian incarnations (Behrens et al.”
BackgroundDiscrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053 · cited 1×
“, 1982), as well as to more recent Bayesian incarnations (Behrens et al., 2007; Mathys et al., 2011; Nassar et al., 2010; Yu and Dayan, 2005).”
BackgroundUncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774 · cited 2×
“, 1982), as well as to more recent Bayesian incarnations (Behrens et al., 2007; Mathys et al., 2011; Nassar et al., 2010; Yu and Dayan, 2005).”
BackgroundAdaptive Coding of Reward Value by Dopamine Neurons
Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005) · Science · doi:10.1126/science.1105370 · Cited by 1,276 · cited 2×
“This implication disagrees with the work of (Tobler et al., 2005), who paired different cues half the time with a cue-specific reward magnitude (liquid volume) and half the time with no reward.”
Background
Research works printed but not linked: 4
- Dayan P., 2001, Advances in neural information processing systems, 13, 451
- 10.1037/0097-7403.5.1.31 · doi:10.1037/0097-7403.5.1.31 · Paper page
- 10.1037/0033-2909.117.3.363 · doi:10.1037/0033-2909.117.3.363 · Paper page
- Pearce J., 1982, Quantitative Analyses of Behavior, 3, 241
Cited by
Adaptive learning under expected and unexpected uncertainty
Soltani, A., & Izquierdo, A. (2019) · Nature reviews. Neuroscience · doi:10.1038/s41583-019-0180-y · Cited by 286
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236
Self-citationThe successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191
“The linear function approximation architecture described above has been widely used in neural models of reinforcement learning(Schultzetal.,1997;DawandTouretzky,2002;Ludviget al.,2008;Gershman,2017a),butitwillfailforvaluefunctions thatarenonlinear.”
BackgroundMethodSelf-citationModel-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161
“Attempts have been made to explain these results by enhancing TDRL to operate not only on the current state, but on states that are inferred to be related to the current state — a departure from pure model-free reinforcement learning — as in ‘mediated learning,’ [39,40] or the Kalman TD model [32,41].”
Rethinking dopamine as generalized prediction error
Gardner, M. P. H., Schoenbaum, G., & Gershman, S. J. (2018) · Proceedings of the Royal Society B: Biological Sciences · doi:10.1098/rspb.2018.1645 · Cited by 160
Self-citation
Metadata
- Authors
- Gershman, S. J.
- Affiliations
- Harvard University (US)
- Corresponding author
- Gershman, Samuel J.
- ORCID
- Gershman, Samuel J.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2017-12
- Volume
- 29
- Issue
- 12
- Pages
- 3311-3326
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2951636479
- Cited by
- 40
- References
- 66
- Citations by year
- 2026: 3 · 2025: 1 · 2024: 3 · 2023: 3 · 2022: 7 · 2021: 5 · 2020: 3 · 2019: 5 · 2018: 6 · 2017: 4
- FWCI
- 2.41
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- Neural Computation (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- National Science Foundation
- China Scholarship Council
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- dopamine, reward prediction error, Bayesian reinforcement learning, Bayesian inference, uncertainty, sensory preconditioning, latent inhibition, orbitofrontal cortex, recurrent dynamics, error-based learning, probabilistic beliefs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Dopamine, Inference, and Uncertainty. https://app.cochat.ai/papers/10.1162/neco_a_01023
- The paper
- Gershman, S. J. (2017). Dopamine, Inference, and Uncertainty. Neural Computation. https://doi.org/10.1162/neco_a_01023
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.