Dopamine, Inference, and Uncertainty
Open accessPublished version of a preprint
The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
The hypothesis that the phasic dopamine response reports a reward prediction error has become deeply entrenched. However, dopamine neurons exhibit several notable deviations from this hypothesis. A coherent explanation for these deviations can be obtained by analyzing the dopamine response in terms of Bayesian reinforcement learning. The key idea is that prediction errors are modulated by probabilistic beliefs about the relationship between cues and outcomes, updated through Bayesian inference. This account can explain dopamine responses to inferred value in sensory preconditioning, the effects of cue preexposure (latent inhibition), and adaptive coding of prediction errors when rewards vary across orders of magnitude. We further postulate that orbitofrontal cortex transforms the stimulus representation through recurrent dynamics, such that a simple error-driven learning rule operating on the transformed representation can implement the Bayesian reinforcement learning update.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Indeed, some dopamine neurons may signal not just PE or precision, but a combined precision-weighted PE(26):”
From Aberrant Salience, Information Processing, and Dopaminergic Signaling in People at Clinical High Risk for Psychosis (2020) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 4×
“(1997) in terms of the temporal difference (TD) learning algorithm (Sutton and Barto, 1998), which posits an error signal of the form:”
BackgroundInfluentialA neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 1×
“The phasic firing of dopamine neurons in the midbrain has long been thought to report a reward prediction error—the discrepancy between observed and expected reward—whose purpose is to correct future reward predictions (Eshel et al., 2015; Glimcher, 2011; Schultz et al., 1997).”
BackgroundAdvances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023 · cited 1×
“This mechanism harkens back to the classic Pearce-Hall theory of attention in associative learning (Pearce and Hall, 1980; Pearce et al., 1982), as well as to more recent Bayesian incarnations (Behrens et al.”
BackgroundDiscrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053 · cited 1×
“, 1982), as well as to more recent Bayesian incarnations (Behrens et al., 2007; Mathys et al., 2011; Nassar et al., 2010; Yu and Dayan, 2005).”
BackgroundUncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774 · cited 2×
“, 1982), as well as to more recent Bayesian incarnations (Behrens et al., 2007; Mathys et al., 2011; Nassar et al., 2010; Yu and Dayan, 2005).”
BackgroundAdaptive Coding of Reward Value by Dopamine Neurons
Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005) · Science · doi:10.1126/science.1105370 · Cited by 1,276 · cited 2×
“This implication disagrees with the work of (Tobler et al., 2005), who paired different cues half the time with a cue-specific reward magnitude (liquid volume) and half the time with no reward.”
Background
Research works printed but not linked: 2
- Dayan, P. and Kakade, S. (2001). Explaining away in weight space. In Leen, T., Dietterich, T., and Tresp, V., editors, Advances in Neural Information Processing Systems 13 , pages 451-457. MIT Press.
- Pearce, J., Kaye, H., and Hall, G. (1982). Predictive accuracy and stimulus associability: Development of a model for Pavlovian learning. Quantitative analyses of behavior , 3:241-255.
Cited by
Adaptive learning under expected and unexpected uncertainty
Soltani, A., & Izquierdo, A. (2019) · Nature reviews. Neuroscience · doi:10.1038/s41583-019-0180-y · Cited by 286
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236
Self-citationThe successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191
“The linear function approximation architecture described above has been widely used in neural models of reinforcement learning(Schultzetal.,1997;DawandTouretzky,2002;Ludviget al.,2008;Gershman,2017a),butitwillfailforvaluefunctions thatarenonlinear.”
BackgroundMethodSelf-citationModel-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161
“Attempts have been made to explain these results by enhancing TDRL to operate not only on the current state, but on states that are inferred to be related to the current state — a departure from pure model-free reinforcement learning — as in ‘mediated learning,’ [39,40] or the Kalman TD model [32,41].”
Rethinking dopamine as generalized prediction error
Gardner, M. P. H., Schoenbaum, G., & Gershman, S. J. (2018) · Proceedings of the Royal Society B: Biological Sciences · doi:10.1098/rspb.2018.1645 · Cited by 160
Self-citationGershman, S. J. (2019) · Decision · doi:10.1037/dec0000101 · Cited by 145
Self-citationHowes, O. D., Hird, E. J., Adams, R. A., Corlett, P. R., & McGuire, P. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2020.03.012 · Cited by 110
“Indeed, some dopamine neurons may signal not just PE or precision, but a combined pre cision-weighted PE(26): research using functional magnetic resonance imaging (fMRI) has shown that PE signals in the midbrain and striatum in humans are modulated by the variance of the distrib ution generating the PE (27).”
BackgroundThe interpretation of computational model parameters depends on the context
Eckstein, M. K., Master, S. L., Xia, L., Dahl, R. E., Wilbrecht, L., & Collins, A. G. (2022) · eLife · doi:10.7554/elife.75474 · Cited by 112
Explaining dopamine through prediction errors and beyond
Gershman, S. J., Assad, J. A., Datta, S. R., Linderman, S. W., Sabatini, B. L., Uchida, N., & Wilbrecht, L. (2024) · Nature Neuroscience · doi:10.1038/s41593-024-01705-4 · Cited by 126
“Although some aspects of these findings might be explained by the hypothesis that RPEs are normalized by uncertainty 93,94 , which decreases over the course of stimulus pre-exposure, in combination with the hypothesis that dopamine encodes a novelty bonus 35 , there are other aspects that are more…”
Self-citationDopamine role in learning and action inference
Bogacz, R. (2020) · eLife · doi:10.7554/elife.53262 · Cited by 81
Memory as a Computational Resource
Dasgupta, I., & Gershman, S. J. (2021) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.12.008 · Cited by 68
“Moreover, it can be implemented in biologically plausible neural circuits [33 – 35].”
Self-citationDopaminergic genes are associated with both directed and random exploration
Gershman, S. J., & Tzovaras, B. G. (2018) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.10.009 · Cited by 51
Self-citationPreconditioned cues have no value
Sharpe, M. J., Batchelor, H. M., & Schoenbaum, G. (2017) · eLife · doi:10.7554/elife.28362 · Cited by 43
Adams, R. A., Moutoussis, M., Nour, M. M., Dahoun, T., Lewis, D. J., Illingworth, B. J. G., Veronese, M., Mathys, C. D., de Boer, L., Guitart‐Masip, M., et al. (2019) · Cerebral Cortex · doi:10.1093/cercor/bhz327 · Cited by 34
Rethinking dopamine as generalized prediction errorPreprint
Gardner, M. P. H., Schoenbaum, G., & Gershman, S. J. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/239731 · Cited by 30
Self-citationKim, J., Kwon, J. H., Kim, J., Kim, E. J., Kim, H. E., Kyeong, S.-H., & Kim, J. (2021) · Scientific Reports · doi:10.1038/s41598-021-94328-9 · Cited by 59
Evaluation of the hypothesis that phasic dopamine constitutes a cached-value signal
Sharpe, M. J., & Schoenbaum, G. M. (2017) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2017.12.002 · Cited by 24
“Specifically, Nakahara (2014) and Gershman (2017) argue that the dopaminergic error system can be influenced by more than the expectation elicited by the cue which is currently present.”
BackgroundInfluentialBayesian updating during development predicts genotypic differences in plasticity
Stamps, J. A., Biro, P. A., Mitchell, D. J., & Saltz, J. B. (2018) · Evolution · doi:10.1111/evo.13585 · Cited by 25
“…used to study conditioning, associative and reinforcement learning, perceptual learning, causal learning, and decision-making in humans and animals (Holyoak and Cheng 2011; Trimmer et al. 2011; Series and Seitz 2013; Soto et al. 2014; Gershman 2015; Foley and Marjoram 2017; Gershman 2017).”
BackgroundSpee, B. T. M., Sladky, R., Fingerhut, J., Laciny, A., Kraus, C., Carls‐Diamante, S., Brücke, C., Pelowski, M. J., & Treven, M. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.930293 · Cited by 20
“…adjusting the weighting of different forms of prediction errors, including reward prediction errors, but also the weighting of salient sensory-motor cues that signify affordance (Friston K. J. et al., 2012; FitzGerald et al., 2015; Schwartenbeck et al., 2015; Gershman, 2017; Gardner et al., 2018).”
BackgroundMikuš, N., Eisenegger, C., Mathys, C. D., Clark, L., Müller, U., Robbins, T. W., Lamm, C., & Naef, M. (2023) · Nature Communications · doi:10.1038/s41467-023-39823-5 · Cited by 21
Uncertainty–guided learning with scaled prediction errors in the basal ganglia
Möller, M., Manohar, S., & Bogacz, R. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009816 · Cited by 21
“One recent theory—Kalman–TD—explained the scaling of dopamine responses, as well as other phenomena such as preconditioning, as a consequence of volatility tracking [5].”
BackgroundMethodSynthetic surprise as the foundation of the psychedelic experience
Filippo, R. D., & Schmitz, D. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2024.105538 · Cited by 19
“The computing of precision, however, seems to be distributed among several neuromodulators including dopamine and acetylcholine (Feldman and Friston, 2010; Yu and Dayan, 2005; Moran et al., 2013; P ´ erez-Gonz ´ alez et al., 2023; Haarsma et al., 2021; Gershman, 2017; Soltani and Izquierdo, 2019).”
BackgroundRoughley, S., Marcus, A., & Killcross, S. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.740992 · Cited by 15
“This has led some to suggest a role for dopamine in subserving a generalised prediction error term that signals errors in both sensory and reward predictions (Suri, 2001; Gershman, 2017; Gardner et al., 2018).”
BackgroundA probabilistic successor representation for context-dependent learning
Geerts, J. P., Gershman, S. J., Burgess, N., & Stachenfeld, K. L. (2024) · Psychological Review · doi:10.1037/rev0000414 · Cited by 14
“Applying a prediction error update to this “replay vector” approximates the Kalman TD algorithm (Gershman, 2017b).”
BackgroundMethodSelf-citationPaskewitz, S., & Jones, M. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102371 · Cited by 16
The dopamine circuit as a reward-taxis navigation system
Karin, O., & Alon, U. S. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010340 · Cited by 17
“according to the standard deviation of the reward distributions [26,53].”
ResultQuantity versus quality: Convergent findings in effort-based choice tasks
Hart, E. E., & Izquierdo, A. (2019) · Behavioural Processes · doi:10.1016/j.beproc.2019.05.009 · Cited by 15
“1978; Wise, Spindler, and Legault 1978), this idea fell out of favor in light of the many other functions of dopamine, including aversively motivated behaviors (McCullough, Sokolowski, and Salamone 1993), and reward prediction error (RPE) signals that drive learning (Niv, Duff, and Dayan 2005; Gershman 2017).”
BackgroundTonic dopamine, uncertainty and basal ganglia action selection
Gilbertson, T., & Steele, J. D. (2021) · Neuroscience · doi:10.1016/j.neuroscience.2021.05.010 · Cited by 18
Sandholtz, N., Miyamoto, Y. R., Bornn, L., & Smith, M. A. (2022) · Bayesian Analysis · doi:10.1214/21-ba1303 · Cited by 15
Learning Temporal Relationships Between Symbols with Laplace Neural Manifolds
Howard, M. W., Esfahani, Z. G., Le, B., & Sederberg, P. B. (2024) · Computational Brain & Behavior · doi:10.1007/s42113-024-00230-8 · Cited by 13
Experience, memory, and the places they meet.
Cowansage, K. K. (2018) · Behavioral Neuroscience · doi:10.1037/bne0000280 · Cited by 9
Velocity Estimation in Reinforcement Learning
Velázquez, C., Villarreal, M., & Bouzas, A. (2019) · Computational Brain & Behavior · doi:10.1007/s42113-019-00026-1 · Cited by 5
“…to vary as function of the relative uncertainty in the estimates and the outcomes as expressed in the Kalman filter equations (Kalman 1960; Navarro et al. 2018; Zajkowski et al. 2017; Speekenbrink and Konstantinidis 2015; Speekenbrink and Shanks 2010; Gershman 2017, 2015; Kakade and Dayan 2002).”
BackgroundSynthetic Spatial Foraging With Active Inference in a Geocaching Task
Neacsu, V., Convertino, L., & Friston, K. J. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.802396 · Cited by 6
“…simulations, which confirm the biological plausibility of the model and the role of dopamine in foraging and uncertainty reduction, as shown in previous studies (Fiorillo et al., 2003; Niv et al., 2005; Friston et al., 2014; Li et al., 2016; Gershman, 2017; Jo et al., 2018; Le Heron et al., 2020).”
ResultBlocking D2/D3 dopamine receptors increases volatility of beliefs when we learn to trust othersPreprint
Mikuš, N., Eisenegger, C., Mathys, C. D., Clark, L., Müller, U., Robbins, T. W., Lamm, C., & Naef, M. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.06.21.496956 · Cited by 3
“Through encoding of uncertainty of beliefs about the world and what action to perform, dopamine receptors are thought to adjust the weights on PEs and control action selection (Adams, Stephan, Brown, Frith, & Friston, 2013; Babayan, Uchida, & Gershman, 2018).”
BackgroundPrediction error correlates in the striosome-dopamine circuit emerge from information gain
Beck, D. W., & Friedman, A. (2026) · Nature Communications · doi:10.1038/s41467-026-73994-1 · Cited by 2
“For example, the mathematics underpinning policy-IG closely mirrors the Bayesian “ surprise ” term used in active inference accounts of dopamine 152,153 (Supplementary Note 17).”
Computational signatures of uncertainty are reflected in motor cortex excitatory neurochemistry
Jassim, N., Waade, P. T., Parsons, O., Petzschner, F. H., Rua, C., Rodgers, C. T., Baron‐Cohen, S., Suckling, J., Mathys, C. D., & Lawson, R. P. (2025) · Nature Communications · doi:10.1038/s41467-025-64702-6 · Cited by 5
Uncertainty quantification and exploration–exploitation trade-off in humans
Candelieri, A., Ponti, A., & Archetti, F. (2021) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-021-03547-5 · Cited by 13
“Recent results in the line of research related to brain science are discussed in (Gershman, 2017; Friston et al., 2014).”
BackgroundPupil dynamics reflect uncertainty-driven adjustments of probability learning
Greenhouse‐Tucknott, A., Foucault, C., Buot, A., & Meyniel, F. (2026) · PLoS Computational Biology · doi:10.64898/2026.02.26.708148 · Cited by 1
Multi-timescale learning signals in Drosophila dopaminergic neuronsPreprint
Choi, W., Srinivasan, S., & Grover, D. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.08.729969
Metadata
- Authors
- Gershman, S. J.
- Affiliations
- Harvard University (US)
- Corresponding author
- Gershman, Samuel J.
- ORCID
- Gershman, Samuel J.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2017-12
- Volume
- 29
- Issue
- 12
- Pages
- 3311-3326
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2951636479
- Cited by
- 40
- References
- 66
- Citations by year
- 2026: 3 · 2025: 1 · 2024: 3 · 2023: 3 · 2022: 7 · 2021: 5 · 2020: 3 · 2019: 5 · 2018: 6 · 2017: 4
- FWCI
- 2.41
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- Neural Computation (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- National Science Foundation
- China Scholarship Council
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- dopamine, reward prediction error, Bayesian reinforcement learning, Bayesian inference, uncertainty, sensory preconditioning, latent inhibition, orbitofrontal cortex, recurrent dynamics, error-based learning, probabilistic beliefs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.