Dopamine, Inference, and Uncertainty
Open accessPublished version of a preprint
The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
The hypothesis that the phasic dopamine response reports a reward prediction error has become deeply entrenched. However, dopamine neurons exhibit several notable deviations from this hypothesis. A coherent explanation for these deviations can be obtained by analyzing the dopamine response in terms of Bayesian reinforcement learning. The key idea is that prediction errors are modulated by probabilistic beliefs about the relationship between cues and outcomes, updated through Bayesian inference. This account can explain dopamine responses to inferred value in sensory preconditioning, the effects of cue preexposure (latent inhibition), and adaptive coding of prediction errors when rewards vary across orders of magnitude. We further postulate that orbitofrontal cortex transforms the stimulus representation through recurrent dynamics, such that a simple error-driven learning rule operating on the transformed representation can implement the Bayesian reinforcement learning update.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Indeed, some dopamine neurons may signal not just PE or precision, but a combined precision-weighted PE(26):”
From Aberrant Salience, Information Processing, and Dopaminergic Signaling in People at Clinical High Risk for Psychosis (2020) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 4×
“(1997) in terms of the temporal difference (TD) learning algorithm (Sutton and Barto, 1998), which posits an error signal of the form:”
BackgroundInfluentialA neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 1×
“The phasic firing of dopamine neurons in the midbrain has long been thought to report a reward prediction error—the discrepancy between observed and expected reward—whose purpose is to correct future reward predictions (Eshel et al., 2015; Glimcher, 2011; Schultz et al., 1997).”
BackgroundAdvances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023 · cited 1×
“This mechanism harkens back to the classic Pearce-Hall theory of attention in associative learning (Pearce and Hall, 1980; Pearce et al., 1982), as well as to more recent Bayesian incarnations (Behrens et al.”
BackgroundDiscrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053 · cited 1×
“, 1982), as well as to more recent Bayesian incarnations (Behrens et al., 2007; Mathys et al., 2011; Nassar et al., 2010; Yu and Dayan, 2005).”
BackgroundUncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774 · cited 2×
“, 1982), as well as to more recent Bayesian incarnations (Behrens et al., 2007; Mathys et al., 2011; Nassar et al., 2010; Yu and Dayan, 2005).”
BackgroundAdaptive Coding of Reward Value by Dopamine Neurons
Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005) · Science · doi:10.1126/science.1105370 · Cited by 1,276 · cited 2×
“This implication disagrees with the work of (Tobler et al., 2005), who paired different cues half the time with a cue-specific reward magnitude (liquid volume) and half the time with no reward.”
BackgroundUnderstanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis
Glimcher, P. W. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1014269108 · Cited by 955 · cited 1×
“The phasic firing of dopamine neurons in the midbrain has long been thought to report a reward prediction error—the discrepancy between observed and expected reward—whose purpose is to correct future reward predictions (Eshel et al., 2015; Glimcher, 2011; Schultz et al., 1997).”
BackgroundOrbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885 · cited 1×
“One potential neural substrate for this stimulus transformation is the OFC, a critical hub for state representation in RL (Wilson et al., 2014).”
BackgroundRational regulation of learning dynamics by pupil-linked arousal systems
Nassar, M. R., Rumsey, K. M., Wilson, R. C., Parikh, K., Heasly, B., & Gold, J. I. (2012) · Nature Neuroscience · doi:10.1038/nn.3130 · Cited by 722
Reinforcement learning: The Good, The Bad and The Ugly
Dayan, P., & Niv, Y. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.08.003 · Cited by 704 · cited 1×
“The theory presented here does not pretend to be a complete account of dopamine; there remain numerous anomalies that will keep RL theorists busy for a long time (Dayan and Niv, 2008).”
BackgroundMetalearning and neuromodulation
Doya, K. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00044-8 · Cited by 704 · cited 1×
“Instead of modulating learning rates over time (e.g., using neuromodulation; Doya, 2002; Nassar et al., 2012; Yu and Dayan, 2005), the circuit transforms the inputs so that they implicitly encode the learning rate dynamics.”
BackgroundCould information theory provide an ecological theory of sensory processing?
Atick, J. J. (2011) · Network: Computation in Neural Systems · doi:10.3109/0954898x.2011.638888 · Cited by 679
A Bayesian foundation for individual learning under uncertainty
Mathys, C. (2011) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2011.00039 · Cited by 655
Time-Derivative Models of Pavlovian Reinforcement
Sutton, R. S., & Barto, A. G. (1990) · Cited by 623
Frank, M. J., & Claus, E. D. (2006) · Psychological Review · doi:10.1037/0033-295x.113.2.300 · Cited by 585 · cited 1×
“The idea that recurrent dynamics in OFC play an important role in stimulus representation for reinforcement learning and reward expectation has also figured in earlier models (Deco and Rolls, 2005; Frank and Claus, 2006).”
ResultNassar, M. R., Wilson, R. C., Heasly, B., & Gold, J. I. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.0822-10.2010 · Cited by 560
Learning and Computational Neuroscience: Foundations of Adaptive Networks
Gabriel, M., & Moore, J. (1990) · Cited by 518
Learning and selective attention
Dayan, P., Kakade, S., & Montague, P. R. (2000) · Nature Neuroscience · doi:10.1038/81504 · Cited by 495 · cited 1×
“This idea was originally explored by Dayan and colleagues (Dayan and Kakade, 2001; Dayan et al., 2000) using a simple Bayesian extension of the Rescorla-Wagner model (the Kalman filter).”
MethodBayesian theories of conditioning in a changing world
Courville, A. C., Daw, N. D., & Touretzky, D. S. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.004 · Cited by 457
Could information theory provide an ecological theory of sensory processing?
Atick, J. J. (1992) · Network: Computation in Neural Systems · doi:10.1088/0954-898x_3_2_009 · Cited by 446
Assessment of the Rescorla-Wagner model.
Miller, R. R., Barnet, R. C., & Grahame, N. J. (1995) · Psychological Bulletin · doi:10.1037//0033-2909.117.3.363 · Cited by 426 · cited 2×
“First, it predicts that a host of post-training manipulations, identified as problematic for traditional associative learning (Gershman, 2015; Miller et al., 1995), should have systematic effects on dopamine responses.”
Background(2019) · Psychology Press eBooks · doi:10.4324/9781315788999 · Cited by 422
Information processing, dimensionality reduction and reinforcement learning in the basal ganglia
Bar-Gad, I., Morris, G., & Bergman, H. (2003) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2003.12.001 · Cited by 399
Arithmetic and local circuitry underlying dopamine prediction errors
Eshel, N., Bukwich, M., Rao, V., Hemmelder, V., Tian, J., & Uchida, N. (2015) · Nature · doi:10.1038/nature14855 · Cited by 392 · cited 1×
“The phasic firing of dopamine neurons in the midbrain has long been thought to report a reward prediction error—the discrepancy between observed and expected reward—whose purpose is to correct future reward predictions (Eshel et al., 2015; Glimcher, 2011; Schultz et al., 1997).”
BackgroundHighly restricted origin of prefrontal cortical inputs to striosomes in the macaque monkey
Eblen, F., & Graybiel, A. (1995) · The Journal of Neuroscience · doi:10.1523/jneurosci.15-09-05999.1995 · Cited by 391 · cited 1×
“We suggest that the intermediate units correspond to the orbitofrontal cortex (OFC), with feedforward synapses to reward prediction neurons in the ventral striatum (Eblen and Graybiel, 1995).”
BackgroundDopamine, affordance and active inference
Friston, K. J., Shiner, T., FitzGerald, T., Galea, J. M., Adams, R., Brown, H., Dolan, R. J., Moran, R., Stephan, K. E., & Bestmann, S. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002327 · Cited by 381 · cited 1×
“Friston and colleagues have developed a theory grounded in a variational approximation of Bayesian inference, whereby dopamine reports an estimate of inverse variance (FitzGerald et al., 2015; Friston et al., 2012; Schwartenbeck et al., 2014).”
BackgroundLe Pelley, M. E. (2004) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/02724990344000141 · Cited by 360
Orbitofrontal Cortex Supports Behavior and Learning Using Inferred But Not Cached Values
Jones, J. L., Esber, G. R., McDannald, M. A., Gruber, A. J., Hernandez, A., Mirenzi, A., & Schoenbaum, G. (2012) · Science · doi:10.1126/science.1227489 · Cited by 335 · cited 1×
“This prevents the network from exhibiting sensory preconditioning, as has been observed experimentally (Jones et al., 2012).”
MethodDopamine neurons share common response function for reward prediction error
Eshel, N., Tian, J., Bukwich, M., & Uchida, N. (2016) · Nature Neuroscience · doi:10.1038/nn.4239 · Cited by 316
An analytic solution to discrete Bayesian reinforcement learning
Poupart, P., Vlassis, N., Hoey, J., & Regan, K. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143932 · Cited by 278
Latent inhibition of conditioned dopamine release in rat nucleus accumbens
Young, A. M. J., Joseph, M. H., & Gray, J. A. (1993) · Neuroscience · doi:10.1016/0306-4522(93)90378-s · Cited by 261 · cited 2×
“, 2016) and latent inhibition (Young et al., 1993), where the prediction error should theoretically be zero? Second, why do dopamine responses appear to rescale with the range or variance of rewards (Tobler et al.”
BackgroundMethodOptimal learning: computational procedures for bayes-adaptive markov decision processes
Duff, M. O., & Barto, A. G. (2002) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 239
Could information theory provide an ecological theory of sensory processing?
Atick, J. J. (1992) · Network Computation in Neural Systems · doi:10.1088/0954-898x/3/2/009 · Cited by 234
The dopaminergic midbrain encodes the expected certainty about desired outcomes
Schwartenbeck, P., FitzGerald, T. H. B., Mathys, C., Dolan, R., & Friston, K. (2015) · Cerebral Cortex · doi:10.1093/cercor/bhu159 · Cited by 219 · cited 1×
“Friston and colleagues have developed a theory grounded in a variational approximation of Bayesian inference, whereby dopamine reports an estimate of inverse variance (FitzGerald et al., 2015; Friston et al., 2012; Schwartenbeck et al., 2014).”
BackgroundDopamine reward prediction errors reflect hidden-state inference across time
Starkweather, C. K., Babayan, B. M., Uchida, N., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4520 · Cited by 206 · cited 1×
“, 2006; Rao, 2010), and experimental evidence has recently accumulated for this proposal (Lak et al., 2017; Starkweather et al., 2017; Takahashi et al., 2016).”
BackgroundSelf-citationCorticostriatal Combinatorics: The Implications of Corticostriatal Axonal Arborizations
Zheng, T., & Wilson, C. J. (2002) · Journal of Neurophysiology · doi:10.1152/jn.00519.2001 · Cited by 205
Surprise! Neural correlates of Pearce–Hall and Rescorla–Wagner coexist within the brain
Roesch, M. R., Esber, G. R., Li, J., Daw, N. D., & Schoenbaum, G. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07986.x · Cited by 191
Rao, R. P. N. (2010) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2010.00146 · Cited by 191 · cited 1×
“Several authors have proposed that belief states could explain some anomalous patterns of dopamine responses (Daw et al., 2006; Rao, 2010), and experimental evidence has recently accumulated for this proposal (Lak et al.”
BackgroundLatent inhibition of a CS during CS-US pairings.
Hall, G., & Pearce, J. M. (1979) · Journal of Experimental Psychology: Animal Behavior Processes · doi:10.1037//0097-7403.5.1.31 · Cited by 176 · cited 1×
“For example, learning is slower when the cue was previously a reliable predictor of reward (Hall and Pearce, 1979).”
BackgroundRepresentation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · cited 2×
“(1997) showed good agreement between prediction errors and dopamine using the CSC, later work called into question its adequacy (Daw et al., 2006; Gershman et al., 2014; Ludvig et al., 2008).”
BackgroundReversal learning and dopamine: a Bayesian perspective
Costa, V. D., Tran, V. L., Turchi, J., & Averbeck, B. B. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.1989-14.2015 · Cited by 170 · cited 1×
“Other authors have suggested that dopamine is involved in specifying a prior probability distribution (Costa et al., 2015) or influencing uncertainty representation in the striatum (Mikhael and Bogacz, 2016).”
BackgroundMidbrain dopamine neurons signal belief in choice accuracy during a perceptual decision
Lak, A., Nomoto, K., Keramati, M., Sakagami, M., & Kepecs, A. (2017) · Current Biology · doi:10.1016/j.cub.2017.02.026 · Cited by 170 · cited 1×
“, 2006; Rao, 2010), and experimental evidence has recently accumulated for this proposal (Lak et al., 2017; Starkweather et al., 2017; Takahashi et al., 2016).”
BackgroundA Unifying Probabilistic View of Associative Learning
Gershman, S. J. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004567 · Cited by 166 · cited 6×
“The theory presented here was first developed to explain a broad range of associative learning phenomena within a unifying framework (Gershman, 2015).”
BackgroundMethodInfluentialSelf-citationEfficient coding and the neural representation of value
Louie, K., & Glimcher, P. W. (2012) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2012.06496.x · Cited by 159 · cited 1×
“Adaptive coding has been found throughout sensory areas as well as in reward-processing areas (Louie and Glimcher, 2012).”
BackgroundDopamine, uncertainty and TD learning
Niv, Y., Duff, M. O., & Dayan, P. (2005) · Behavioral and Brain Functions · doi:10.1186/1744-9081-1-6 · Cited by 153
Discovering latent causes in reinforcement learning
Gershman, S. J., Norman, K. A., & Niv, Y. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.07.007 · Cited by 151 · cited 1×
“This view allows one to contemplate more complex versions of the model proposed here, for example with nonlinear function approximators or structure learning (Gershman et al., 2015), although inference quickly becomes intractable in these cases.”
BackgroundSelf-citationStimulus representation and the timing of reward-prediction errors in models of the dopamine system
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008) · Neural Computation · doi:10.1162/neco.2008.11-07-654 · Cited by 148 · cited 1×
“(1997) showed good agreement between prediction errors and dopamine using the CSC, later work called into question its adequacy (Daw et al., 2006; Gershman et al., 2014; Ludvig et al., 2008).”
Attention and Associative Learning: From Brain to Behaviour
Mitchell, C. J., & Pelley, M. E. L. (2010) · Cited by 146
Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework
Sadacca, B. F., Jones, J. L., & Schoenbaum, G. (2016) · eLife · doi:10.7554/elife.13665 · Cited by 135 · cited 2×
“First, why do dopamine neurons respond under some conditions, such as sensory preconditioning (Sadacca et al., 2016) and latent inhibition (Young et al.”
BackgroundResultAdding prediction risk to the theory of reward learning
PREUSCHOFF, K., & BOSSAERTS, P. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.005 · Cited by 133
Recovery from blocking achieved by extinguishing the blocking CS
Blaisdell, A. P., Gunther, L. M., & Miller, R. R. (1999) · Animal Learning & Behavior · doi:10.3758/bf03199432 · Cited by 118 · cited 1×
“For example, extinguishing the blocking cue in a blocking paradigm causes recovery of responding to the blocked cue in a subsequent test (Blaisdell et al., 1999); the Kalman TD model predicts that this extinction procedure should cause a positive dopaminergic response to the blocked cue.”
BackgroundTakahashi, Y. K., Langdon, A. J., Niv, Y., & Schoenbaum, G. (2016) · Neuron · doi:10.1016/j.neuron.2016.05.015 · Cited by 116 · cited 1×
“, 2006; Rao, 2010), and experimental evidence has recently accumulated for this proposal (Lak et al., 2017; Starkweather et al., 2017; Takahashi et al., 2016).”
BackgroundSynaptic and Spiking Dynamics underlying Reward Reversal in the Orbitofrontal Cortex
Deco, G. (2004) · Cerebral Cortex · doi:10.1093/cercor/bhh103 · Cited by 114 · cited 1×
“The idea that recurrent dynamics in OFC play an important role in stimulus representation for reinforcement learning and reward expectation has also figured in earlier models (Deco and Rolls, 2005; Frank and Claus, 2006).”
ResultDopamine, reward learning, and active inference
FitzGerald, T. H. B., Dolan, R. J., & Friston, K. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00136 · Cited by 112 · cited 1×
“Friston and colleagues have developed a theory grounded in a variational approximation of Bayesian inference, whereby dopamine reports an estimate of inverse variance (FitzGerald et al., 2015; Friston et al., 2012; Schwartenbeck et al., 2014).”
BackgroundDiederen, K. M. J., Spencer, T., Vestergaard, M. D., Fletcher, P. C., & Schultz, W. (2016) · Neuron · doi:10.1016/j.neuron.2016.04.019 · Cited by 102
Learning Reward Uncertainty in the Basal Ganglia
Mikhael, J. G., & Bogacz, R. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005062 · Cited by 95 · cited 1×
“, 2015) or influencing uncertainty representation in the striatum (Mikhael and Bogacz, 2016).”
BackgroundDynamic shaping of dopamine signals during probabilistic Pavlovian conditioning
Hart, A. S., Clark, J. J., & Phillips, P. E. M. (2015) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2014.07.010 · Cited by 82
Geist, M., & Pietquin, O. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3077 · Cited by 82 · cited 1×
“The same logic that transforms Rescorla-Wagner into the Kalman filter can be applied to transform the TD model into a Bayesian model (Geist and Pietquin, 2010).”
MethodConvergent Algorithm for Sensory Receptive Field Development
Atick, J. J., & Redlich, A. N. (1993) · Neural Computation · doi:10.1162/neco.1993.5.1.45 · Cited by 81
Time representation in reinforcement learning models of the basal ganglia
Gershman, S. J., Moustafa, A. A., & Ludvig, E. A. (2014) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2013.00194 · Cited by 81 · cited 1×
“(1997) showed good agreement between prediction errors and dopamine using the CSC, later work called into question its adequacy (Daw et al., 2006; Gershman et al., 2014; Ludvig et al., 2008).”
Self-citationScaling prediction errors to reward variability benefits error-driven learning in humans
Diederen, K. M. J., & Schultz, W. (2015) · Journal of Neurophysiology · doi:10.1152/jn.00483.2015 · Cited by 69
Performance of a Stochastic Net
Goodall, M. (1960) · Nature · doi:10.1038/185557a0 · Cited by 34
Research works printed but not linked: 2
- Dayan, P. and Kakade, S. (2001). Explaining away in weight space. In Leen, T., Dietterich, T., and Tresp, V., editors, Advances in Neural Information Processing Systems 13 , pages 451-457. MIT Press.
- Pearce, J., Kaye, H., and Hall, G. (1982). Predictive accuracy and stimulus associability: Development of a model for Pavlovian learning. Quantitative analyses of behavior , 3:241-255.
Cited by
Adaptive learning under expected and unexpected uncertainty
Soltani, A., & Izquierdo, A. (2019) · Nature reviews. Neuroscience · doi:10.1038/s41583-019-0180-y · Cited by 286
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236
Self-citationThe successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191
“The linear function approximation architecture described above has been widely used in neural models of reinforcement learning(Schultzetal.,1997;DawandTouretzky,2002;Ludviget al.,2008;Gershman,2017a),butitwillfailforvaluefunctions thatarenonlinear.”
BackgroundMethodSelf-citationModel-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161
“Attempts have been made to explain these results by enhancing TDRL to operate not only on the current state, but on states that are inferred to be related to the current state — a departure from pure model-free reinforcement learning — as in ‘mediated learning,’ [39,40] or the Kalman TD model [32,41].”
Rethinking dopamine as generalized prediction error
Gardner, M. P. H., Schoenbaum, G., & Gershman, S. J. (2018) · Proceedings of the Royal Society B: Biological Sciences · doi:10.1098/rspb.2018.1645 · Cited by 160
Self-citation
Metadata
- Authors
- Gershman, S. J.
- Affiliations
- Harvard University (US)
- Corresponding author
- Gershman, Samuel J.
- ORCID
- Gershman, Samuel J.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2017-12
- Volume
- 29
- Issue
- 12
- Pages
- 3311-3326
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2951636479
- Cited by
- 40
- References
- 66
- Citations by year
- 2026: 3 · 2025: 1 · 2024: 3 · 2023: 3 · 2022: 7 · 2021: 5 · 2020: 3 · 2019: 5 · 2018: 6 · 2017: 4
- FWCI
- 2.41
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- Neural Computation (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- National Science Foundation
- China Scholarship Council
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- dopamine, reward prediction error, Bayesian reinforcement learning, Bayesian inference, uncertainty, sensory preconditioning, latent inhibition, orbitofrontal cortex, recurrent dynamics, error-based learning, probabilistic beliefs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.