Stimulus representation and the timing of reward-prediction errors in models of the dopamine system
Abstract
The phasic firing of dopamine neurons has been theorized to encode a reward-prediction error as formalized by the temporal-difference (TD) algorithm in reinforcement learning. Most TD models of dopamine have assumed a stimulus representation, known as the complete serial compound, in which each moment in a trial is distinctly represented. We introduce a more realistic temporal stimulus representation for the TD model. In our model, all external stimuli, including rewards, spawn a series of internal microstimuli, which grow weaker and more diffuse over time. These microstimuli are used by the TD learning algorithm to generate predictions of future reward. This new stimulus representation injects temporal generalization into the TD model and enhances correspondence between model and data in several experiments, including those when rewards are omitted or received early. This improved fit mostly derives from the absence of large negative errors in the new model, suggesting that dopamine alone can encode the full range of TD errors in these situations.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The temporal precision of tuning decreased with elapsed time.
“We observed that individual units in our RNNs were tuned to elapsed time relative to observations, and that the temporal precision of tuning decreased with elapsed time (S1 Fig), both of which are standard assumptions of microstate representations [41].”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [41]
Individual units in the RNNs were tuned to elapsed time relative to observations.
“We observed that individual units in our RNNs were tuned to elapsed time relative to observations, and that the temporal precision of tuning decreased with elapsed time (S1 Fig), both of which are standard assumptions of microstate representations [41].”
From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [41]
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Self-citationA neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855 · see citing works
Self-citationLearning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · see citing works
Self-citationThe debate over dopamine’s role in reward: the case for incentive salience
Berridge, K. C. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0578-x · Cited by 2,347 · see citing works
Scalar expectancy theory and Weber's law in animal timing.
Gibbon, J. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.279 · Cited by 2,224 · see citing works
Discrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085 · see citing works
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · see citing works
Punishment and aversive behavior
Campbell, B. A., & Church, R. M. (1969) · Appleton-Century-Crofts eBooks · Cited by 1,989 · see citing works
By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964 · see citing works
Predictability, surprise, attention, and conditioning
Kamin, L. J. (1967) · NASA Technical Reports Server (NASA) · Cited by 1,483 · see citing works
Toward a modern theory of adaptive networks: Expectation and prediction.
Sutton, R. S., & Barto, A. G. (1981) · Psychological Review · doi:10.1037/0033-295x.88.2.135 · Cited by 1,450 · see citing works
Self-citationMidbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · see citing works
Dopamine neurons report an error in the temporal prediction of reward during learning
Hollerman, J. R., & Schultz, W. (1998) · Nature Neuroscience · doi:10.1038/1124 · Cited by 1,103 · see citing works
Dopamine responses comply with basic assumptions of formal learning theory
Waelti, P., Dickinson, A., & Schultz, W. (2001) · Nature · doi:10.1038/35083500 · Cited by 1,018 · see citing works
Computational roles for dopamine in behavioural control
Montague, P. R., Hyman, S. E., & Cohen, J. D. (2004) · Nature · doi:10.1038/nature03015 · Cited by 966 · see citing works
Opponent interactions between serotonin and dopamine
Daw, N. D., Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00052-7 · Cited by 844 · see citing works
Addiction as a Computational Process Gone Awry
Redish, A. D. (2004) · Science · doi:10.1126/science.1102384 · Cited by 534 · see citing works
Time-Derivative Models of Pavlovian Reinforcement
Sutton, R. S., & Barto, A. G. (1990) · Cited by 496 · see citing works
Self-citationPan, W.-X., Schmidt, R., Wickens, J. R., & Hyland, B. I. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.1478-05.2005 · Cited by 429 · see citing works
Staddon, J. E. R., & Cerutti, D. T. (2002) · Annual Review of Psychology · doi:10.1146/annurev.psych.54.101601.145124 · Cited by 384 · see citing works
Learning the temporal dynamics of behavior.
Machado, A. D. (1997) · Psychological Review · doi:10.1037/0033-295x.104.2.241 · Cited by 373 · see citing works
Neural dynamics of adaptive timing and temporal discrimination during associative learning
Grossberg, S., & Schmajuk, N. A. (1989) · Neural Networks · doi:10.1016/0893-6080(89)90026-9 · Cited by 367 · see citing works
Redish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.784 · Cited by 346 · see citing works
Brown, J. W., Bullock, D. H., & Grossberg, S. (1999) · Journal of Neuroscience · doi:10.1523/jneurosci.19-23-10502.1999 · Cited by 299 · see citing works
Suri, R. E., & Schultz, W. (1999) · Neuroscience · doi:10.1016/s0306-4522(98)00697-6 · Cited by 294 · see citing works
Smith, M. C. (1968) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0026550 · Cited by 288 · see citing works
Scalar Properties in Animal Timing: Conformity and Violations
Lejeune, H., & Wearden, J. H. (2006) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210600784649 · Cited by 275 · see citing works
Learning of sequential movements by neural network model with dopamine-like reinforcement signal
Suri, R. E., & Schultz, W. (1998) · Experimental Brain Research · doi:10.1007/s002210050467 · Cited by 198 · see citing works
Statistics of Midbrain Dopamine Neuron Spike Trains in the Awake Primate
Bayer, H. M., Lau, B., & Glimcher, P. W. (2007) · Journal of Neurophysiology · doi:10.1152/jn.01140.2006 · Cited by 184 · see citing works
Reinforcement learning, conditioning, and the brain: Successes and challenges
Maia, T. V. (2009) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.9.4.343 · Cited by 184 · see citing works
Representation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · see citing works
Adaptive timing in neural networks: The conditioned response
Desmond, J. E., & Moore, J. W. (1988) · Biological Cybernetics · doi:10.1007/bf00361347 · Cited by 161 · see citing works
Dopamine, uncertainty and TD learning
Niv, Y., Duff, M. O., & Dayan, P. (2005) · Behavioral and Brain Functions · doi:10.1186/1744-9081-1-6 · Cited by 153 · see citing works
PVLV: The Primary Value and Learned Value Pavlovian Learning Algorithm.
O’Reilly, R. C., Frank, M. J., Hazy, T. E., & Watz, B. (2007) · Behavioral Neuroscience · doi:10.1037/0735-7044.121.1.31 · Cited by 144 · see citing works
The role of dopamine in cognitive sequence learning: evidence from Parkinson’s disease
Daphna Shohamy, A., Myers, C. E., Grossman, S., Sage, J., & Gluck, M. A. (2004) · Behavioural Brain Research · doi:10.1016/j.bbr.2004.05.023 · Cited by 114 · see citing works
Why Choose This Book?: How We Make Decisions
Montague, P. R. (2006) · Medical Entomology and Zoology · Cited by 84 · see citing works
Kehoe, E. J., Schreurs, B. G., & Graham, P. (1987) · Animal Learning & Behavior · doi:10.3758/bf03205056 · Cited by 66 · see citing works
Self-citationFiorillo, C. D., Tobler, P. N., & Schultz, W. (2005) · Behavioral and Brain Functions · doi:10.1186/1744-9081-1-7 · Cited by 65 · see citing works
Temporal encoding as a determinant of blocking.
Barnet, R. C., Grahame, N. J., & Miller, R. R. (1993) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037//0097-7403.19.4.327 · Cited by 53 · see citing works
Temporal encoding as a determinant of blocking.
Barnet, R. C., Grahame, N. J., & Miller, R. R. (1993) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.19.4.327 · Cited by 36 · see citing works
Blocking acquisition of the rabbit's nictitating membrane response to serial conditioned stimuli
Kehoe, E. J., Schreurs, B. G., & Amodei, N. (1981) · Learning and Motivation · doi:10.1016/0023-9690(81)90026-6 · Cited by 35 · see citing works
Self-citationStriatal action-learning based on dopamine concentration
Morris, G., Schmidt, R., & Bergman, H. (2009) · Experimental Brain Research · doi:10.1007/s00221-009-2060-6 · Cited by 32 · see citing works
Alternative time representation in dopamine models
Rivest, F., Kalaska, J. F., & Bengio, Y. (2009) · Journal of Computational Neuroscience · doi:10.1007/s10827-009-0191-1 · Cited by 23 · see citing works
Interstimulus interval and delivery cues influence timed conditioned responding in rats
Williams, D. A., Chubala, C. M., Mather, A. A., et al. (2009) · Learning and Motivation · doi:10.1016/j.lmot.2009.07.001 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446 · see citing works
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236 · see citing works
Dopamine reward prediction errors reflect hidden-state inference across time
Starkweather, C. K., Babayan, B. M., Uchida, N., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4520 · Cited by 206 · see citing works
The successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191 · see citing works
Model-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161 · see citing works
The medial prefrontal cortex shapes dopamine reward prediction errors under state uncertainty
Starkweather, C. K., Gershman, S. J., & Uchida, N. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.036 · Cited by 136 · see citing works
Time representation in reinforcement learning models of the basal ganglia
Gershman, S. J., Moustafa, A. A., & Ludvig, E. A. (2014) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2013.00194 · Cited by 81 · see citing works
Self-citationEmergence of belief-like representations through reinforcement learning
Hennig, J. A., Romero Pinto, S. A., Yamaguchi, T., Linderman, S. W., Uchida, N., & Gershman, S. J. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011067 · Cited by 41 · see citing works
Goldstein Ferber, S., Shoval, G., Zalsman, G., Mikulincer, M., & Weller, A. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.680403 · Cited by 40 · see citing works
Dopamine, inference, and uncertainty
Gershman, S. J. (2017) · Neural Computation · doi:10.1162/neco_a_01023 · Cited by 40 · see citing works
Neural control of dopamine neurotransmission: implications for reinforcement learning
Aggarwal, M., Hyland, B. I., & Wickens, J. R. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08055.x · Cited by 32 · see citing works
Addressing altered anticipation as a transdiagnostic target through computational psychiatry
Sepúlveda, P., Aitsahalia, I., Kumar, K., Atkin, T., & Iigaya, K. (2025) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2025.02.014 · Cited by 3 · see citing works
- Authors
- Ludvig, E. A., Sutton, R. S., & Kehoe, E. J.
- Affiliations
- University of Alberta (CA)
- UNSW Sydney (AU)
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2008-12
- Volume
- 20
- Issue
- 12
- Pages
- 3034-3054
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2011868317
- Cited by
- 148
- References
- 45
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 8 · 2023: 8 · 2022: 12 · 2021: 10 · 2020: 10 · 2019: 8 · 2018: 10 · 2017: 12 · 2016: 5 · 2015: 7 · 2014: 9 · 2013: 6 · 2012: 8
- FWCI
- 3.74
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- PubMed (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- dopaminergic neurons, reward prediction error, temporal difference learning, stimulus representation, temporal generalization, reinforcement learning models
- MeSH terms
- Action Potentials, Algorithms, Animals, Cues, Dopamine, Humans, Models, Neurological, Neurons, Reaction Time, Reward, Time Factors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
