CoChat
Sign Up
Cited by 173Neurotransmitter Receptor Influence on BehaviorNeural and Behavioral Psychology StudiesNeural dynamics and brain function

Representation and timing in theories of the dopamine system

Daw, Nathaniel D., Courville, Aaron C., Touretzky, David S.·2006·Neural Computation
DOI 10.1162/neco.2006.18.7.1637
Open this paper in CoChat — free account

Abstract

Although the responses of dopamine neurons in the primate midbrain are well characterized as carrying a temporal difference (TD) error signal for reward prediction, existing theories do not offer a credible account of how the brain keeps track of past sensory events that may be relevant to predicting future reward. Empirically, these shortcomings of previous theories are particularly evident in their account of experiments in which animals were exposed to variation in the timing of events. The original theories mispredicted the results of such experiments due to their use of a representational device called a tapped delay line. Here we propose that a richer understanding of history representation and a better account of these experiments can be given by considering TD algorithms for a formal setting that incorporates two features not originally considered in theories of the dopaminergic response: partial observability (a distinction between the animal's sensory experience and the true underlying state of the world) and semi-Markov dynamics (an explicit account of variation in the intervals between events). The new theory situates the dopaminergic system in a richer functional and anatomical context, since it assumes (in accord with recent computational theories of cortex) that problems of partial observability and stimulus history are solved in sensory cortex using statistical modeling and inference and that the TD system predicts reward using the results of this inference rather than raw sensory data. It also accounts for a range of experimental data, including the experiments involving programmed temporal variability and other previously unmodeled dopaminergic response phenomena, which we suggest are related to subjective noise in animals' interval timing. Finally, it offers new experimental predictions and a rich theoretical framework for designing future experiments.

Integrity signals

Registry

Crossref · 10.1162/neco.2006.18.7.1637

Cites

98 of 98 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Previous experimental work has shown that the phasic activity of midbrain dopamine neurons resembles the RPEs of TD learning in partially observable environments, where TD learning is performed on belief states rather than observations.

    “Previous experimental work has shown that the phasic activity of midbrain dopamine neurons resembles the RPEs of TD learning in partially observable environments, where TD learning is performed on belief states rather than observations [6][7][8][9][10][11].”

    From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [6]

  • The tasks analyzed are technically discrete-time semi-Markov processes, and can be formulated equivalently as Markov processes with micro-states defined over each relevant discrete time step.

    “The tasks we analyze here are technically discretetime semi-Markov processes, and so we follow previous work in formulating them equivalently as Markov processes with micro-states defined over each relevant discrete time step [6,7].”

    From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [6]

  • Previous work has suggested that prefrontal cortex may perform state estimation in tasks with hidden states.

    “For instance, previous work has suggested that prefrontal cortex may perform state estimation in tasks with hidden states [6,7,12,13].”

    From Emergence of belief-like representations through reinforcement learning (2023) · cites this paper as [6]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Daw, N. D., Courville, A. C., & Touretzky, D. S.
Affiliations
  • Gatsby Computational Neuroscience Unit (GB)
  • Center for the Neural Basis of Cognition (US)
ORCID
Venue
Neural Computation
Publisher
MIT Press
Published
2006-07
Volume
18
Issue
7
Pages
1637-1677
ISSN
  • 0899-7667 (ISSN-L)
  • 1530-888X
DOI
10.1162/neco.2006.18.7.1637
OpenAlex
W2092861175
Cited by
173
References
98
Citations by year
2026: 3 · 2025: 7 · 2024: 6 · 2023: 5 · 2022: 10 · 2021: 10 · 2020: 9 · 2019: 12 · 2018: 11 · 2017: 12 · 2016: 11 · 2015: 8 · 2014: 8 · 2013: 7 · 2012: 9
FWCI
3.40
Type
journal article
Language
en
Update notices
correction (2006-10-01)
Open access
closed
Locations
  • Neural Computation (publishedVersion)
  • PubMed (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion)
Topics
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
Keywords
dopaminergic neurons, temporal difference learning, reward prediction, partial observability, interval timing, temporal variation, stimulus history
MeSH terms
Algorithms, Animals, Dopamine, Humans, Models, Neurological, Nerve Net, Neurons, Reaction Time, Reward, Time Factors, Models, Statistical, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review