Tamping Ramping: Algorithmic, Implementational, and Computational Explanations of Phasic Dopamine Signals in the Accumbens
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Substantial evidence suggests that the phasic activity of dopamine neurons represents reinforcement learning's temporal difference prediction error. However, recent reports of ramp-like increases in dopamine concentration in the striatum when animals are about to act, or are about to reach rewards, appear to pose a challenge to established thinking. This is because the implied activity is persistently predictable by preceding stimuli, and so cannot arise as this sort of prediction error. Here, we explore three possible accounts of such ramping signals: (a) the resolution of uncertainty about the timing of action; (b) the direct influence of dopamine over mechanisms associated with making choices; and (c) a new model of discounted vigour. Collectively, these suggest that dopamine ramps may be explained, with only minor disturbance, by standard theoretical ideas, though urgent questions remain regarding their proximal cause. We suggest experimental approaches to disentangling which of the proposed mechanisms are responsible for dopamine ramps.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Some Studies in Machine Learning Using the Game of Checkers
Samuel, A. L. (1959) · IBM Journal of Research and Development · doi:10.1147/rd.33.0210 · Cited by 4,443
The Neural Basis of Decision Making
Gold, J. I., & Shadlen, M. N. (2007) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.113038 · Cited by 3,764 · cited 1×
“While the SPRT and its close associates are usually considered in relation to decision making under state uncertainty, as when there is doubt about whether the overall motion of a random dot field is to the left or right [70], such models have also been applied with some success to memorybased [71] or value-based [72]…”
BackgroundDaw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Self-citationLIBET, B., GLEASON, C. A., WRIGHT, E. W., & PEARL, D. K. (1983) · Brain · doi:10.1093/brain/106.3.623 · Cited by 2,498
The time course of perceptual choice: The leaky, competing accumulator model.
Usher, M., & McClelland, J. L. (2001) · Psychological Review · doi:10.1037/0033-295x.108.3.550 · Cited by 2,152 · cited 1×
“In particular, this can be shown to be a suitable abstraction of a particular sort of neural circuit involving competition between two (or sometimes more) populations of neurons representing the choices [73, 76, 77].”
BackgroundDissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 2×
“This dissociation is consistent with an implementation of actor and critic components in the dorsal and ventral striatum, respectively [1, 36].”
BackgroundSelf-citationDiscrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 4×
“This dissociation is consistent with an implementation of actor and critic components in the dorsal and ventral striatum, respectively [1, 36].”
BackgroundInfluentialSelf-citationPlace Cells, Grid Cells, and the Brain's Spatial Representation System
Moser, E. I., Kropff, E., & Moser, M.-B. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.31.061307.090723 · Cited by 1,934
Grace, A. A. (1991) · Neuroscience · doi:10.1016/0306-4522(91)90196-u · Cited by 1,831
Learning and Memory Functions of the Basal Ganglia
Packard, M. G., & Knowlton, B. J. (2002) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.25.112701.142937 · Cited by 1,717 · cited 1×
“In particular, the ventral striatum (NAc) is implicated in reward and motivation [37], while the dorsal striatum is implicated in motor and cognitive control [38].”
BackgroundThe primate basal ganglia: Parallel and integrative networks
Haber, S. N. (2003) · Journal of Chemical Neuroanatomy · doi:10.1016/j.jchemneu.2003.10.003 · Cited by 1,479 · cited 1×
“, via a more indirect form of efference copy (a′ in Fig 6) which could be transmitted via interacting corticostriatal loops or some more direct means [56].”
BackgroundIkemoto, S. (2007) · Brain Research Reviews · doi:10.1016/j.brainresrev.2007.05.004 · Cited by 1,477
Some studies in machine learning using the game of checkers
Samuel, A. L. (2000) · IBM Journal of Research and Development · doi:10.1147/rd.441.0206 · Cited by 1,439
The mysterious motivational functions of mesolimbic dopamine
Salamone, J. D., & Correa, M. (2012) · Neuron · doi:10.1016/j.neuron.2012.10.021 · Cited by 1,418 · cited 1×
“This suggestion is based on a long literature implicating dopamine in the modulation of behavioural vigour [45].”
BackgroundMidbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · cited 1×
“Indeed, evidence suggests that negative prediction errors are represented differently from positive prediction errors in the activity of midbrain dopaminergic neurons: while positive prediction errors appear to correlate positively with the firing rates of dopaminergic neurons, the magnitude of negative prediction err…”
BackgroundProbabilistic decision making by slow reverberation in cortical circuits
Wang, X.-J. (2002) · Neuron · doi:10.1016/s0896-6273(02)01092-9 · Cited by 1,242
Models of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
Tonic dopamine: Opportunity costs and the control of response vigor
Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0502-4 · Cited by 1,133 · cited 7×
“The key question, though, is why we should expect to see any such quasi-tonic signal in this context? We consider the possibility that this signal is the equivalent in the discounted case of the average reward ρ (for convenience, in this section, we omit the superscript π) which, as we have seen, has previously been a…”
BackgroundResultInfluentialSelf-citationSubsecond dopamine release promotes cocaine seeking
Phillips, P. E. M., Stuber, G. D., Heien, M. L. A. V., Wightman, R. M., & Carelli, R. M. (2003) · Nature · doi:10.1038/nature01476 · Cited by 1,113
Floresco, S. B., West, A. R., Ash, B., Moore, H., & Grace, A. A. (2003) · Nature Neuroscience · doi:10.1038/nn1103 · Cited by 1,106
Regulation of firing of dopaminergic neurons and control of goal-directed behaviors
Grace, A. A., Floresco, S. B., Goto, Y., & Lodge, D. J. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.03.003 · Cited by 1,034 · cited 16×
“In particular, the ventral striatum (NAc) is implicated in reward and motivation [37], while the dorsal striatum is implicated in motor and cognitive control [38].”
BackgroundInfluentialA selective role for dopamine in stimulus-reward learning
Flagel, S. B., Clark, J. J., Robinson, T. E., Mayo, L., Czuj, A., Willuhn, I., Akers, C. A., Clinton, S. M., Phillips, P. E. M., & Akil, H. (2011) · Nature · doi:10.1038/nature09588 · Cited by 1,026
Nucleus accumbens shell and core dopamine: differential role in behavior and addiction
Di Chiara, G. (2002) · Behavioural Brain Research · doi:10.1016/s0166-4328(02)00286-3 · Cited by 996
Responses of monkey dopamine neurons during learning of behavioral reactions
Ljungberg, T., Apicella, P., & Schultz, W. (1992) · Journal of Neurophysiology · doi:10.1152/jn.1992.67.1.145 · Cited by 987
Distinct roles for direct and indirect pathway striatal neurons in reinforcement
Kravitz, A. V., Tye, L. D., & Kreitzer, A. C. (2012) · Nature Neuroscience · doi:10.1038/nn.3100 · Cited by 972
Computational roles for dopamine in behavioural control
Montague, P. R., Hyman, S. E., & Cohen, J. D. (2004) · Nature · doi:10.1038/nature03015 · Cited by 966
Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens
Nicola, S. M., Surmeier, D. J., & Malenka, R. C. (2000) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.23.1.185 · Cited by 931
Opponent interactions between serotonin and dopamine
Daw, N. D., Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00052-7 · Cited by 844
Self-citationPrediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004) · Nature Neuroscience · doi:10.1038/nn1279 · Cited by 779
Dopamine operates as a subsecond modulator of food seeking
Roitman, M. F., Stuber, G. D., Phillips, P. E. M., Wightman, R. M., & Carelli, R. M. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.3823-03.2004 · Cited by 718 · cited 8×
“[10] very similar in structure to the lever pressing case considered above.”
BackgroundResultInfluentialAssociative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens
Day, J. J., Roitman, M. F., Wightman, R. M., & Carelli, R. M. (2007) · Nature Neuroscience · doi:10.1038/nn1923 · Cited by 710
Prolonged dopamine signalling in striatum signals proximity and value of distant rewards
Howe, M. W., Tierney, P. L., Sandberg, S. G., Phillips, P. E. M., & Graybiel, A. M. (2013) · Nature · doi:10.1038/nature12475 · Cited by 565 · cited 10×
“[14] may reflect an average reward-like signal that arises Fig 5.”
BackgroundMethodInfluentialActor–critic models of the basal ganglia: new anatomical and computational perspectives
Joel, D., Niv, Y., & Ruppin, E. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00047-3 · Cited by 500
Functional–anatomical implications of the nucleus accumbens core and shell subterritories
ZAHM, D. S. (1999) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1999.tb09264.x · Cited by 384
Romo, R., & Schultz, W. (1990) · Journal of Neurophysiology · doi:10.1152/jn.1990.63.3.592 · Cited by 353 · cited 2×
“Even in monkeys, Romo and Schultz [91] reported gradual increases in the firing rates of some putative dopaminergic cells (12 out of 104 recorded) up to 1500 before onset of self-initiated arm movements to obtain food.”
BackgroundThe temporal precision of reward prediction in dopamine neurons
Fiorillo, C. D., Newsome, W. T., & Schultz, W. (2008) · Nature Neuroscience · doi:10.1038/nn.2159 · Cited by 310
Corbit, L. H., & Balleine, B. W. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.2711-11.2011 · Cited by 309
Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons
Satoh, T., Nakai, S., Sato, T., & Kimura, M. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-30-09913.2003 · Cited by 298
A normative perspective on motivation
NIV, Y., JOEL, D., & DAYAN, P. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.06.010 · Cited by 251
Self-citationStuber, G. D., Roitman, M. F., Phillips, P. E. M., Carelli, R. M., & Wightman, R. M. (2005) · Neuropsychopharmacology · doi:10.1038/sj.npp.1300619 · Cited by 217
Representation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · cited 3×
“Implications of partial observability for TD models of dopamine have been explored in previous work, notably by Daw and colleagues [48], though that did not address the possibility of partial observability arising between distinct internal agencies, nor the possible relevance to self-initiated action envisaged here.”
BackgroundStuber, G. D., Wightman, R. M., & Carelli, R. M. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.036 · Cited by 168
Dopamine, uncertainty and TD learning
Niv, Y., Duff, M. O., & Dayan, P. (2005) · Behavioral and Brain Functions · doi:10.1186/1744-9081-1-6 · Cited by 153
Self-citationStimulus representation and the timing of reward-prediction errors in models of the dopamine system
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008) · Neural Computation · doi:10.1162/neco.2008.11-07-654 · Cited by 148
Cheer, J. F., Heien, M. L. A. V., Garris, P. A., Carelli, R. M., & Wightman, R. M. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0509607102 · Cited by 143
Owesson‐White, C. A., Ariansen, J., Stuber, G. D., Cleaveland, N. A., Cheer, J. F., Mark Wightman, R., & Carelli, R. M. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06916.x · Cited by 126
Dynamic shaping of dopamine signals during probabilistic Pavlovian conditioning
Hart, A. S., Clark, J. J., & Phillips, P. E. M. (2015) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2014.07.010 · Cited by 82
Time representation in reinforcement learning models of the basal ganglia
Gershman, S. J., Moustafa, A. A., & Ludvig, E. A. (2014) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2013.00194 · Cited by 81 · cited 2×
“Additional complexity, which we leave to future work, arises from the putative connection between dopamine and the speed of an internal clock, as inferred, for example, from the effects of dopamine manipulations on behaviour in interval-timing tasks [102, 103].”
BackgroundResult
Cited by
A Unified Framework for Dopamine Signals across Timescales
Kim, H. R., Malik, A. N., Mikhael, J. G., Bech, P., Tsutsui-Kimura, I., Sun, F., Zhang, Y., Li, Y., Watabe-Uchida, M., Gershman, S. J., et al. (2020) · Cell · doi:10.1016/j.cell.2020.11.013 · Cited by 302
Midbrain dopamine neurons signal belief in choice accuracy during a perceptual decision
Lak, A., Nomoto, K., Keramati, M., Sakagami, M., & Kepecs, A. (2017) · Current Biology · doi:10.1016/j.cub.2017.02.026 · Cited by 170
Howes, O. D., Hird, E. J., Adams, R. A., Corlett, P. R., & McGuire, P. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2020.03.012 · Cited by 110
Addressing altered anticipation as a transdiagnostic target through computational psychiatry
Sepúlveda, P., Aitsahalia, I., Kumar, K., Atkin, T., & Iigaya, K. (2025) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2025.02.014 · Cited by 3
Metadata
- Authors
- Lloyd, K., & Dayan, P.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-12-23
- Volume
- 11
- Issue
- 12
- Pages
- e1004622
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2211629969
- Cited by
- 50
- References
- 49
- FWCI
- 1.37
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.