Dopamine reward prediction errors reflect hidden-state inference across time
Open accessAbstract
Midbrain dopamine neurons signal reward prediction error (RPE), or actual minus expected reward. The temporal difference (TD) learning model has been a cornerstone in understanding how dopamine RPEs could drive associative learning. Classically, TD learning imparts value to features that serially track elapsed time relative to observable stimuli. In the real world, however, sensory stimuli provide ambiguous information about the hidden state of the environment, leading to the proposal that TD learning might instead compute a value signal based on an inferred distribution of hidden states (a 'belief state'). Here we asked whether dopaminergic signaling supports a TD learning framework that operates over hidden states. We found that dopamine signaling showed a notable difference between two tasks that differed only with respect to whether reward was delivered in a deterministic manner. Our results favor an associative learning rule that combines cached values with hidden-state inference.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous experimental work has shown that the phasic activity of midbrain dopamine neurons resembles the RPEs of TD learning in partially observable environments, where TD learning is performed on belief states rather than observations.
“Previous experimental work has shown that the phasic activity of midbrain dopamine neurons resembles the RPEs of TD learning in partially observable environments, where TD learning is performed on belief states rather than observations [6][7][8][9][10][11].”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [7]
RNNs could be trained to estimate value in the tasks from Starkweather et al. (2017), and their representations became more belief-like as a result of training.
“We showed that RNNs could be trained to estimate value in the tasks from Starkweather et al. (2017) [7], and that the representations of these RNNs became more belief-like as a result of training.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [7]
Beliefs in the Babayan task should have a fixed point at the ITIs for Block 1 and Block 2.
“Similar to Task 1 of Starkweather et al. (2017) [7] (Fig 5A), beliefs in the present task should have a fixed point at the ITIs for Block 1 and Block 2 (Fig 7F).”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [7]
In both tasks, mice were trained to associate an odor cue with probabilistic delivery of a liquid reward 1.2-2.8s later.
“In both tasks, mice were trained to associate an odor cue with probabilistic delivery of a liquid reward 1.2-2.8slater.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [7]
Previous work showed that dopamine activity depended on the reward time differently in the two tasks, with activity decreasing as a function of reward time in Task 1, but increasing as a function of reward time in Task 2.
“Previous work showed that dopamine activity depended on the reward time differently in the two tasks, with activity decreasing as a function of reward time in Task 1, but increasing as a function of reward time in Task 2 [7] (Fig 3A).”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [7]
As a working example, we will consider the probabilistic associative learning paradigm where dopamine RPEs were shown to be consistent with a belief representation.
“As a working example, we will consider the probabilistic associative learning paradigm where dopamine RPEs were shown to be consistent with a belief representation [7,13].”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [7]
RNNs can be trained to optimally estimate value in two previously studied associative learning tasks.
“We first show that RNNs can be trained to optimally estimate value in two previously studied associative learning tasks [7,10], and that the RPEs of these models resemble experimentally observed dopamine neuron activity.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [7]
Previous work has suggested that prefrontal cortex may perform state estimation in tasks with hidden states.
“For instance, previous work has suggested that prefrontal cortex may perform state estimation in tasks with hidden states [6,7,12,13].”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [7]
The tasks analyzed are technically discrete-time semi-Markov processes, and can be formulated equivalently as Markov processes with micro-states defined over each relevant discrete time step.
“The tasks we analyze here are technically discretetime semi-Markov processes, and so we follow previous work in formulating them equivalently as Markov processes with micro-states defined over each relevant discrete time step [6,7].”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [7]
In the Starkweather et al. (2017) task, reward times were sampled from a discretized Gaussian ranging from 1.2 to 2.8 seconds.
“Reward times were sampled from a discretized Gaussian ranging from 1.2-2.8s(see Materials and methods).”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as Starkweather et al. (2017)[7]
In both versions of the Starkweather et al. (2017) task, there were two underlying states: the ITI and the ISI.
“In both versions of the task, there were two underlying states: the ITI and the ISI.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as Starkweather et al. (2017)[7]
In Task 2, rewards were omitted on 10% of trials, so an odor did not reveal whether or not the state transitioned to the ISI.
“In Task 2, rewards were omitted on 10% of trials; as a result, an odor did not reveal whether or not the state transitioned to the ISI.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as Starkweather et al. (2017)[7]
Mouse dopamine reward prediction errors exhibit opposite dependence on reward time across the two tasks.
“RPEs exhibit opposite dependence on reward time across tasks.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as Starkweather et al. (2017) [7]
In Task 1, every trial was rewarded, so an odor always indicated a transition from the ITI to the ISI, while a reward always indicated a transition from the ISI to the ITI.
“In Task 1, every trial was rewarded.As a result, an odor always indicated a transition from the ITI to the ISI, while a reward always indicated a transition from the ISI to the ITI.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as Starkweather et al. (2017)[7]
In Task 2, each macro-state (ISI or ITI) is composed of micro-states denoting elapsed time, allowing for probabilistic reward times and minimum dwell times in the ISI and ITI, respectively.
“Each macro-state (ISI or ITI) is composed of micro-states denoting elapsed time; this allows for probabilistic reward times and minimum dwell times in the ISI and ITI, respectively.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as Starkweather et al. (2017)[7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · see citing works
A theory of cortical responses.
Friston, K. (2005) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2005.1622 · Cited by 4,578 · see citing works
Guide for the Care and Use of Laboratory Animals (8th edn)
Albus, U. (2012) · Laboratory Animals · doi:10.1258/la.2012.150312 · Cited by 1,706 · see citing works
Hierarchical Bayesian inference in the visual cortex
Lee, T. S., & Mumford, D. (2003) · Journal of the Optical Society of America A · doi:10.1364/josaa.20.001434 · Cited by 1,473 · cited 2× · see citing works
“A long-standing idea in modern neuroscience is that the brain computes inferences about the outside world rather than passively observing its environment 27,28 .”
Neuron-type-specific signals for reward and punishment in the ventral tegmental area
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., & Uchida, N. (2012) · Nature · doi:10.1038/nature10754 · Cited by 1,318 · see citing works
Self-citationMidbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · see citing works
Dopamine neurons report an error in the temporal prediction of reward during learning
Hollerman, J. R., & Schultz, W. (1998) · Nature Neuroscience · doi:10.1038/1124 · Cited by 1,103 · see citing works
Mesolimbic dopamine signals the value of work
Hamid, A., Pettibone, J. R., Mabrouk, O. S., Hetrick, V. L., Schmidt, R., Weele, C. M. V., Kennedy, R. T., Aragona, B. J., & Berke, J. D. (2015) · Nature Neuroscience · doi:10.1038/nn.4173 · Cited by 815 · see citing works
Speed and accuracy of olfactory discrimination in the rat
Uchida, N., & Mainen, Z. F. (2003) · Nature Neuroscience · doi:10.1038/nn1142 · Cited by 687 · see citing works
Self-citationQuantitative measures of cluster quality for use in extracellular recordings
Schmitzer-Torbert, N. C., Jackson, J. C., Henze, D. A., Harris, K., & Redish, A. D. (2005) · Neuroscience · doi:10.1016/j.neuroscience.2004.09.066 · Cited by 642 · see citing works
Atasoy, D., Aponte, Y., Su, H. H., & Sternson, S. M. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.1954-08.2008 · Cited by 614 · see citing works
A representation of the hazard rate of elapsed time in macaque area LIP
Janssen, P., & Shadlen, M. N. (2005) · Nature Neuroscience · doi:10.1038/nn1386 · Cited by 601 · see citing works
Distinct behavioural and network correlates of two interneuron types in prefrontal cortex
Kvitsiani, D., Ranade, S., Hangya, B., Taniguchi, H., Huang, J. Z., & Kepecs, A. (2013) · Nature · doi:10.1038/nature12176 · Cited by 516 · see citing works
Bäckman, C. M., Malik, N., Zhang, Y., Shan, L., Grinberg, A., Hoffer, B. J., Westphal, H., & Tomac, A. C. (2006) · genesis · doi:10.1002/dvg.20228 · Cited by 463 · see citing works
Influence of Reward Delays on Responses of Dopamine Neurons
Kobayashi, S., & Schultz, W. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.1600-08.2008 · Cited by 431 · see citing works
A Scalable Population Code for Time in the Striatum
Mello, G. B. M. E., Soares, S., & Paton, J. J. (2015) · Current Biology · doi:10.1016/j.cub.2015.02.036 · Cited by 393 · see citing works
Context, learning, and extinction
Gershman, S. J., Blei, D. M., & Niv, Y. (2010) · Psychological Review · doi:10.1037/a0017808 · Cited by 386 · cited 1× · see citing works
“However, in the real world, stimuli often provide ambiguous information about states; the true underlying states are “hidden” and must therefore be inferred 6,7 .”
Self-citationArithmetic and local circuitry underlying dopamine prediction errors
Eshel, N., Bukwich, M., Rao, V., Hemmelder, V., Tian, J., & Uchida, N. (2015) · Nature · doi:10.1038/nature14855 · Cited by 386 · see citing works
Self-citationLima, S. Q., Hromádka, T., Znamenskiy, P., & Zador, A. M. (2009) · PLoS ONE · doi:10.1371/journal.pone.0006099 · Cited by 356 · see citing works
The temporal precision of reward prediction in dopamine neurons
Fiorillo, C. D., Newsome, W. T., & Schultz, W. (2008) · Nature Neuroscience · doi:10.1038/nn.2159 · Cited by 310 · see citing works
Dopamine neurons share common response function for reward prediction error
Eshel, N., Tian, J., Bukwich, M., & Uchida, N. (2016) · Nature Neuroscience · doi:10.1038/nn.4239 · Cited by 306 · see citing works
Self-citationBrown, J. W., Bullock, D. H., & Grossberg, S. (1999) · Journal of Neuroscience · doi:10.1523/jneurosci.19-23-10502.1999 · Cited by 302 · see citing works
Attentional modulation in visual cortex depends on task timing
Ghose, G. M., & Maunsell, J. H. R. (2002) · Nature · doi:10.1038/nature01057 · Cited by 297 · see citing works
Suri, R. E., & Schultz, W. (1999) · Neuroscience · doi:10.1016/s0306-4522(98)00697-6 · Cited by 294 · see citing works
Learning of sequential movements by neural network model with dopamine-like reinforcement signal
Suri, R. E., & Schultz, W. (1998) · Experimental Brain Research · doi:10.1007/s002210050467 · Cited by 198 · see citing works
Rao, R. P. N. (2010) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2010.00146 · Cited by 190 · see citing works
Representation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · see citing works
Habenula Lesions Reveal that Multiple Mechanisms Underlie Dopamine Prediction Errors
Tian, J., & Uchida, N. (2015) · Neuron · doi:10.1016/j.neuron.2015.08.028 · Cited by 165 · see citing works
Self-citationTemporally Extended Dopamine Responses to Perceptually Demanding Reward-Predictive Stimuli
Nomoto, K., Schultz, W., Watanabe, T., & Sakagami, M. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4828-09.2010 · Cited by 158 · see citing works
Discovering latent causes in reinforcement learning
Gershman, S. J., Norman, K. A., & Niv, Y. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.07.007 · Cited by 151 · see citing works
Self-citationStimulus representation and the timing of reward-prediction errors in models of the dopamine system
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008) · Neural Computation · doi:10.1162/neco.2008.11-07-654 · Cited by 148 · see citing works
Takahashi, Y. K., Langdon, A., Niv, Y., & Schoenbaum, G. M. (2016) · Neuron · doi:10.1016/j.neuron.2016.05.015 · Cited by 116 · see citing works
Acquisition and extinction in autoshaping.
Kakade, S. M., & Dayan, P. (2002) · Psychological Review · doi:10.1037/0033-295x.109.3.533 · Cited by 102 · see citing works
Time representation in reinforcement learning models of the basal ganglia
Gershman, S. J., Moustafa, A. A., & Ludvig, E. A. (2014) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2013.00194 · Cited by 81 · see citing works
Self-citationCholinergic Interneurons Use Orbitofrontal Input to Track Beliefs about Current State
Stalnaker, T. A., Berg, B., Aujla, N., & Schoenbaum, G. M. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.0157-16.2016 · Cited by 76 · see citing works
Dissociable reward and timing signals in human midbrain and ventral striatum
Klein-Flügge, M. C., Hunt, L. T., Bach, D. R., Dolan, R. J., & Behrens, T. E. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.08.024 · Cited by 75 · see citing works
The Time Course of Stimulus Expectation in a Saccadic Decision Task
Oswal, A., Ogden, M. L., & Carpenter, R. H. S. (2007) · Journal of Neurophysiology · doi:10.1152/jn.01238.2006 · Cited by 66 · see citing works
Dopamine neurons encode errors in predicting movement trigger occurrence
Pasquereau, B., & Turner, R. S. (2014) · Journal of Neurophysiology · doi:10.1152/jn.00401.2014 · Cited by 53 · see citing works
Prefrontal Regulation of Neuronal Activity in the Ventral Tegmental Area
Jo, Y. S., & Mizumori, S. J. Y. (2015) · Cerebral Cortex · doi:10.1093/cercor/bhv215 · Cited by 49 · see citing works
Tsunoda, Y., & Kakei, S. (2008) · Neuroscience Letters · doi:10.1016/j.neulet.2007.12.063 · Cited by 22 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [5]Sutton RS Barto AG Time-Derivative Models of Pavlovian Reinforcement Learning and Computational Neuroscience: Foundations of Adaptive Networks Gabriel M Moore J Cambridge, MA MIT Press 497 537
Cited by
Sorted by citations
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236 · see citing works
Self-citationTowards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224 · see citing works
Planning and navigation as active inference
Kaplan, R., & Friston, K. J. (2018) · Biological Cybernetics · doi:10.1007/s00422-018-0753-2 · Cited by 203 · see citing works
Forgetting as a form of adaptive engram cell plasticity
Ryan, T. J., & Frankland, P. W. (2022) · Nature reviews. Neuroscience · doi:10.1038/s41583-021-00548-3 · Cited by 182 · see citing works
Model-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161 · see citing works
Dark control: The default mode network as a reinforcement learning agent
Dohmatob, E., Dumas, G., & Bzdok, D. (2020) · Human Brain Mapping · doi:10.1002/hbm.25019 · Cited by 121 · see citing works
The medial prefrontal cortex shapes dopamine reward prediction errors under state uncertainty
Starkweather, C. K., Gershman, S. J., & Uchida, N. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.036 · Cited by 136 · see citing works
Self-citationWilliams, A. H., Poole, B., Maheswaranathan, N., Dhawale, A. K., Fisher, T., Wilson, C. D., Brann, D. H., Trautmann, E. M., Ryu, S., Shusterman, R., et al. (2020) · Neuron · doi:10.1016/j.neuron.2019.10.020 · Cited by 102 · see citing works
Belief state representation in the dopamine system
Babayan, B. M., Uchida, N., & Gershman, S. J. (2018) · Nature Communications · doi:10.1038/s41467-018-04397-0 · Cited by 101 · see citing works
Self-citationTouch neurons underlying dopaminergic pleasurable touch and sexual receptivity
Elias, L. J., Succi, I. K., Schaffler, M. D., Foster, W., Gradwell, M. A., Bohic, M., Fushiki, A., Upadhyay, A., Ejoh, L. L., Schwark, R., et al. (2023) · Cell · doi:10.1016/j.cell.2022.12.034 · Cited by 85 · see citing works
Clark, L., & Zack, M. (2023) · Addictive Behaviors · doi:10.1016/j.addbeh.2023.107626 · Cited by 59 · see citing works
“Neurophysiological studies of DA signalling confirm that such temporal variability serves to potentiate DA release (Starkweather et al., 2017).”
BackgroundTransdiagnostic computations of uncertainty: towards a new lens on intolerance of uncertainty
Sandhu, T. R., Xiao, B., & Lawson, R. P. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105123 · Cited by 51 · see citing works
Emergence of belief-like representations through reinforcement learning
Hennig, J. A., Romero Pinto, S. A., Yamaguchi, T., Linderman, S. W., Uchida, N., & Gershman, S. J. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011067 · Cited by 41 · see citing works
Self-citationDopamine, inference, and uncertainty
Gershman, S. J. (2017) · Neural Computation · doi:10.1162/neco_a_01023 · Cited by 40 · see citing works
Self-citationGoedhoop, J., Arbab, T., & Willuhn, I. (2023) · The Journal of Neuroscience · doi:10.1523/jneurosci.1527-22.2023 · Cited by 24 · see citing works
- Authors
- Starkweather, C. K., Babayan, B. M., Uchida, N., & Gershman, S. J.
- Affiliations
- Harvard University (US)
- Corresponding author
- Uchida, Naoshige
- Gershman, Samuel J
- Venue
- Nature Neuroscience
- Published
- 2017-03-06
- Volume
- 20
- Issue
- 4
- Pages
- 581-589
- ISSN
- 1097-6256 (ISSN-L)
- 1546-1726
- PMC
- PMC5374025
- OpenAlex
- W2593605634
- Cited by
- 206
- References
- 41
- Citations by year
- 2026: 12 · 2025: 25 · 2024: 24 · 2023: 22 · 2022: 21 · 2021: 23 · 2020: 31 · 2019: 18 · 2018: 24 · 2017: 6
- FWCI
- 10.57
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature Neuroscience (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Funders
- National Institute of Mental Health (F30MH112242, R01MH110404, R01MH101207, R01MH095953, R01MH109177, T32MH020017)
- National Institute of General Medical Sciences (T32GM007753)
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- dopaminergic neurons, reward prediction error, hidden state inference, belief states, temporal difference learning, associative learning, midbrain dopamine, reward contingency
- MeSH terms
- Animals, Association Learning, Male, Models, Neurological, Reward, Time Factors, Ventral Tegmental Area, Mice, Dopaminergic Neurons
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
