Belief state representation in the dopamine system
Open accessAbstract
Learning to predict future outcomes is critical for driving appropriate behaviors. Reinforcement learning (RL) models have successfully accounted for such learning, relying on reward prediction errors (RPEs) signaled by midbrain dopamine neurons. It has been proposed that when sensory data provide only ambiguous information about which state an animal is in, it can predict reward based on a set of probabilities assigned to hypothetical states (called the belief state). Here we examine how dopamine RPEs and subsequent learning are regulated under state uncertainty. Mice are first trained in a task with two potential states defined by different reward amounts. During testing, intermediate-sized rewards are given in rare trials. Dopamine activity is a non-monotonic function of reward size, consistent with RL models operating on belief states. Furthermore, the magnitude of dopamine responses quantitatively predicts changes in behavior. These results establish the critical role of state inference in RL.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous work showed that dopamine activity of animals trained on the Babayan task depended on the number of trials in the current block, similar to what would be expected if animals used a belief representation.
“Previous work showed that the dopamine activity of animals trained on this task depended on the number of trials in the current block (Fig 7C), similar to what you would expect if animals used a belief representation (Fig 7D, dashed lines) [10].”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [10]
In the Babayan task, block identity was hidden to the animal and was resampled uniformly every five trials.
“Importantly, the block identity was hidden to the animal, and was resampled uniformly every five trials.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [10]
In the Babayan task, reward quantity was varied in blocks: Block 1 had a small (1 μL) reward and Block 2 had a large (10 μL) reward.
“In Block 1, each trial consisted of a small (1 μL) reward, while in Block 2 each trial consisted of a large (10 μL) reward.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [10]
RNNs can be trained to optimally estimate value in two previously studied associative learning tasks.
“We first show that RNNs can be trained to optimally estimate value in two previously studied associative learning tasks [7,10], and that the RPEs of these models resemble experimentally observed dopamine neuron activity.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [10]
In the Babayan et al. (2018) task, each trial consisted of an odor followed by a deterministic reward.
“In this task, similar to Task 1 of Starkweather et al. (2017) [7], each trial consisted of an odor followed by a deterministic reward.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [10]
In the Babayan et al. (2018) task, each trial consists of an odor and a subsequent reward, and the reward amount depends on the block identity, which is resampled uniformly every five trials.
“Each trial consists of an odor and a subsequent reward.The reward amount depends on the block identity, which is resampled uniformly every five trials.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as Babayan et al. (2018) [10]
Value RNNs trained on the Babayan et al. (2018) task reproduce Belief RPEs and learn belief-like representations.
“Value RNNs trained on Babayan et al. (2018) [10] reproduce Belief RPEs and learn belief-like representations.”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as Babayan et al. (2018) [10]
In the Babayan et al. (2018) task, average phasic dopamine activity in the VTA of mice trained on the task varies with trial index within the block and current/previous block identity.
“Average phasic dopamine activity in the VTA of mice trained on the task at the time of odor (left) and reward (right) delivery.Activity is shown separately as a function of the trial index within the block (x-axis) and the current/previous block identity (colors).”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as Babayan et al. (2018) [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855 · see citing works
Ultrasensitive fluorescent proteins for imaging neuronal activity
Chen, T.-W., Wardill, T. J., Sun, Y., Pulver, S. R., Renninger, S. L., Baohan, A., Schreiter, E. R., Kerr, R. A., Orger, M. B., Jayaraman, V., et al. (2013) · Nature · doi:10.1038/nature12354 · Cited by 6,474 · see citing works
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
Bayesian model selection for group studies
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.025 · Cited by 1,423 · see citing works
Natural neural projection dynamics underlying social behavior
Gunaydin, L. A., Grosenick, L., Finkelstein, J. C., Kauvar, I. V., Fenno, L. E., Adhikari, A., Lammel, S., Mirzabekov, J. J., Airan, R. D., Zalocusky, K. A., et al. (2014) · Cell · doi:10.1016/j.cell.2014.05.017 · Cited by 1,389 · see citing works
Two types of dopamine neuron distinctly convey positive and negative motivational signals
Matsumoto, M., & Hikosaka, O. (2009) · Nature · doi:10.1038/nature08028 · Cited by 1,378 · see citing works
Neuron-type-specific signals for reward and punishment in the ventral tegmental area
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., & Uchida, N. (2012) · Nature · doi:10.1038/nature10754 · Cited by 1,318 · see citing works
Self-citationMidbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · see citing works
Adaptive Coding of Reward Value by Dopamine Neurons
Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005) · Science · doi:10.1126/science.1105370 · Cited by 1,276 · see citing works
Concurrent activation of striatal direct and indirect pathways during action initiation
Cui, G., Jun, S. B., Jin, X., Pham, M. D., Vogel, S. S., Lovinger, D. M., & Costa, R. M. (2013) · Nature · doi:10.1038/nature11846 · Cited by 1,245 · see citing works
Optimization of a GCaMP Calcium Indicator for Neural Activity Imaging
Akerboom, J., Chen, T., Wardill, T. J., Tian, L., Marvin, J. S., Mutlu, S., Calderón, N. C., Esposti, F., Borghuis, B. G., Sun, X. R., et al. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.2601-12.2012 · Cited by 1,226 · see citing works
Orbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885 · see citing works
Striatal Dopamine Release Is Triggered by Synchronized Activity in Cholinergic Interneurons
Threlfell, S., Lalic, T., Platt, N. J., Jennings, K. A., Deisseroth, K., & Cragg, S. J. (2012) · Neuron · doi:10.1016/j.neuron.2012.04.038 · Cited by 845 · see citing works
Intact-brain analyses reveal distinct information carried by SNc dopamine subcircuits
Lerner, T. N., Shilyansky, C., Davidson, T. J., Evans, K. E., Beier, K. T., Zalocusky, K. A., Crow, A. K., Malenka, R. C., Luo, L., Tomer, R., et al. (2015) · Cell · doi:10.1016/j.cell.2015.07.014 · Cited by 794 · see citing works
Speed and accuracy of olfactory discrimination in the rat
Uchida, N., & Mainen, Z. F. (2003) · Nature Neuroscience · doi:10.1038/nn1142 · Cited by 687 · see citing works
Self-citationBayesian model selection for group studies — Revisited
Rigoux, L., Stephan, K. E., Friston, K. J., & Jean Daunizeau, A. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.08.065 · Cited by 659 · see citing works
Izquierdo, A., Suda, R. K., & Murray, E. A. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.1921-04.2004 · Cited by 588 · see citing works
Decision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580 · see citing works
Cachope, R., Mateo, Y., Mathur, B. N., Irving, J. M., Wang, H., Morales, M., Lovinger, D. M., & Cheer, J. F. (2012) · Cell Reports · doi:10.1016/j.celrep.2012.05.011 · Cited by 519 · see citing works
Bäckman, C. M., Malik, N., Zhang, Y., Shan, L., Grinberg, A., Hoffer, B. J., Westphal, H., & Tomac, A. C. (2006) · genesis · doi:10.1002/dvg.20228 · Cited by 463 · see citing works
Bayesian theories of conditioning in a changing world
Courville, A. C., Daw, N. D., & Touretzky, D. S. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.004 · Cited by 457 · see citing works
Neural Circuitry of Reward Prediction Error
Watabe-Uchida, M., Eshel, N., & Uchida, N. (2017) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-072116-031109 · Cited by 437 · see citing works
Self-citationContext, learning, and extinction
Gershman, S. J., Blei, D. M., & Niv, Y. (2010) · Psychological Review · doi:10.1037/a0017808 · Cited by 386 · see citing works
Self-citationArithmetic and local circuitry underlying dopamine prediction errors
Eshel, N., Bukwich, M., Rao, V., Hemmelder, V., Tian, J., & Uchida, N. (2015) · Nature · doi:10.1038/nature14855 · Cited by 386 · see citing works
Self-citationMeunier, M. (1997) · Neuropsychologia · doi:10.1016/s0028-3932(97)00027-4 · Cited by 348 · see citing works
Real-time chemical responses in the nucleus accumbens differentiate rewarding and aversive stimuli
Roitman, M. F., Wheeler, R. A., Wightman, R. M., & Carelli, R. M. (2008) · Nature Neuroscience · doi:10.1038/nn.2219 · Cited by 343 · see citing works
Reward and choice encoding in terminals of midbrain dopamine neurons depends on striatal target
Parker, N. F., Cameron, C. M., Taliaferro, J. P., Lee, J., Choi, J. Y., Davidson, T. J., Daw, N. D., & Witten, I. B. (2016) · Nature Neuroscience · doi:10.1038/nn.4287 · Cited by 340 · see citing works
Orbitofrontal Cortex Supports Behavior and Learning Using Inferred But Not Cached Values
Jones, J. L., Esber, G. R., McDannald, M. A., Gruber, A. J., Hernández, A., Mirenzi, A., & Schoenbaum, G. M. (2012) · Science · doi:10.1126/science.1227489 · Cited by 332 · see citing works
Puopolo, M., Raviola, E., & Bean, B. P. (2007) · Journal of Neuroscience · doi:10.1523/jneurosci.4341-06.2007 · Cited by 319 · see citing works
The temporal precision of reward prediction in dopamine neurons
Fiorillo, C. D., Newsome, W. T., & Schultz, W. (2008) · Nature Neuroscience · doi:10.1038/nn.2159 · Cited by 310 · see citing works
Dopamine neurons share common response function for reward prediction error
Eshel, N., Tian, J., Bukwich, M., & Uchida, N. (2016) · Nature Neuroscience · doi:10.1038/nn.4239 · Cited by 306 · see citing works
Self-citationHart, A. S., Rutledge, R. B., Glimcher, P. W., & Phillips, P. E. M. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.2489-13.2014 · Cited by 294 · see citing works
Menegas, W., Babayan, B. M., Uchida, N., & Watabe‐Uchida, M. (2017) · eLife · doi:10.7554/elife.21886 · Cited by 265 · see citing works
Self-citationActivity patterns of serotonin neurons underlying cognitive flexibility
Matias, S., Lottem, E., Dugué, G. P., & Mainen, Z. F. (2017) · eLife · doi:10.7554/elife.20552 · Cited by 239 · see citing works
The involvement of the orbitofrontal cortex in learning under changing task contingencies.
Kim, J., & Ragozzino, M. (2005) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2004.10.003 · Cited by 206 · see citing works
Dopamine reward prediction errors reflect hidden-state inference across time
Starkweather, C. K., Babayan, B. M., Uchida, N., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4520 · Cited by 206 · see citing works
Self-citationDopamine Reward Prediction Error Responses Reflect Marginal Utility
Stauffer, W. R., Lak, A., & Schultz, W. (2014) · Current Biology · doi:10.1016/j.cub.2014.08.064 · Cited by 204 · see citing works
Dopamine neurons respond to errors in the prediction of sensory features of expected rewards
Takahashi, Y. K., Batchelor, H. M., Liu, B., Khanna, A., Morales, M., & Schoenbaum, G. (2017) · Neuron · doi:10.1016/j.neuron.2017.08.025 · Cited by 200 · see citing works
Rao, R. P. N. (2010) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2010.00146 · Cited by 190 · see citing works
A Pallidus-Habenula-Dopamine Pathway Signals Inferred Stimulus Values
Bromberg-Martin, E. S., Matsumoto, M., Hong, S., & Hikosaka, O. (2010) · Journal of Neurophysiology · doi:10.1152/jn.00158.2010 · Cited by 182 · see citing works
Representation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · see citing works
Midbrain dopamine neurons signal belief in choice accuracy during a perceptual decision
Lak, A., Nomoto, K., Keramati, M., Sakagami, M., & Kepecs, A. (2017) · Current Biology · doi:10.1016/j.cub.2017.02.026 · Cited by 170 · see citing works
Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework
Sadacca, B. F., Jones, J. L., & Schoenbaum, G. M. (2016) · eLife · doi:10.7554/elife.13665 · Cited by 137 · see citing works
Takahashi, Y. K., Langdon, A., Niv, Y., & Schoenbaum, G. M. (2016) · Neuron · doi:10.1016/j.neuron.2016.05.015 · Cited by 114 · see citing works
Statistical computations underlying the dynamics of memory updating
Gershman, S. J., Radulescu, A., Norman, K. A., & Niv, Y. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003939 · Cited by 101 · see citing works
Self-citationNucleus Accumbens Acetylcholine Receptors Modulate Dopamine and Motivation
Collins, A. L., Aitken, T. J., Greenfield, V. Y., Ostlund, S. B., & Wassum, K. M. (2016) · Neuropsychopharmacology · doi:10.1038/npp.2016.81 · Cited by 94 · see citing works
Cholinergic Interneurons Use Orbitofrontal Input to Track Beliefs about Current State
Stalnaker, T. A., Berg, B., Aujla, N., & Schoenbaum, G. M. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.0157-16.2016 · Cited by 76 · see citing works
Unraveling the Role of the Hippocampus in Reversal Learning
Vilà‐Balló, A., Herrero, E. M., Ripollés, P., Simó, M., Miró, J., Cucurell, D., López-Barroso, D., Juncadella, M., Marco‐Pallarés, J., Falip, M., et al. (2017) · Journal of Neuroscience · doi:10.1523/jneurosci.3212-16.2017 · Cited by 61 · see citing works
Context Learning in the Rodent Hippocampus
Fuhs, M. C., & Touretzky, D. S. (2007) · Neural Computation · doi:10.1162/neco.2007.19.12.3173 · Cited by 57 · see citing works
Structural learning and the hippocampus
Aggleton, J. P., Sanderson, D. J., & Pearce, J. M. (2007) · Hippocampus · doi:10.1002/hipo.20323 · Cited by 42 · see citing works
Kudo, Y., Akita, K., Nakamura, T., Ogura, A., Makino, T., Tamagawa, A., OZAKI, K., & Miyakawa, A. (1992) · Neuroscience · doi:10.1016/0306-4522(92)90451-7 · Cited by 39 · see citing works
Sarno, S., de Lafuente, V., Romo, R., & Parga, N. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1712479114 · Cited by 30 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The Relationship Between Uncertainty and Affect
Anderson, E. C., Carleton, R. N., Diefenbach, M., & Han, P. K. J. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02504 · Cited by 332 · see citing works
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236 · see citing works
Self-citationTowards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224 · see citing works
The successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191 · see citing works
Self-citationDopaminergic basis for signaling belief updates, but not surprise, and the link to paranoia
Nour, M. M., Dahoun, T., Schwartenbeck, P., Adams, R. A., FitzGerald, T. H. B., Coello, C., Wall, M. B., Dolan, R. J., & Howes, O. D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1809298115 · Cited by 101 · see citing works
Emergence of belief-like representations through reinforcement learning
Hennig, J. A., Romero Pinto, S. A., Yamaguchi, T., Linderman, S. W., Uchida, N., & Gershman, S. J. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011067 · Cited by 41 · see citing works
Self-citation
- Authors
- Babayan, B. M., Uchida, N., & Gershman, S. J.
- Affiliations
- Harvard University (US)
- Corresponding author
- Uchida, Naoshige
- Gershman, Samuel. J.
- Venue
- Nature Communications
- Published
- 2018-05-14
- Volume
- 9
- Issue
- 1
- Pages
- 1891-1891
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC5951832
- OpenAlex
- W2801251491
- Cited by
- 101
- References
- 55
- Citations by year
- 2026: 9 · 2025: 14 · 2024: 10 · 2023: 9 · 2022: 10 · 2021: 18 · 2020: 17 · 2019: 6 · 2018: 8
- FWCI
- 5.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Nature Communications (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Institutes of Health
- National Institute of Mental Health (R01MH101207, R01MH095953, R01MH109177)
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- belief states, dopamine, reward prediction error, reinforcement learning, state uncertainty, state inference, midbrain dopaminergic neurons, reward magnitude, mice
- MeSH terms
- Animals, Dopamine, Electrodes, Implanted, Learning, Male, Reinforcement, Psychology, Reward, Stereotaxic Techniques, Uncertainty, Mice, Anticipation, Psychological, Dopaminergic Neurons, Ventral Striatum
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
