Influence of Reward Delays on Responses of Dopamine Neurons
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Psychological and microeconomic studies have shown that outcome values are discounted by imposed delays. The effect, called temporal discounting, is demonstrated typically by choice preferences for sooner smaller rewards over later larger rewards. However, it is unclear whether temporal discounting occurs during the decision process when differently delayed reward outcomes are compared or during predictions of reward delays by pavlovian conditioned stimuli without choice. To address this issue, we investigated the temporal discounting behavior in a choice situation and studied the effects of reward delay on the value signals of dopamine neurons. The choice behavior confirmed hyperbolic discounting of reward value by delays on the order of seconds. Reward delay reduced the responses of dopamine neurons to pavlovian conditioned stimuli according to a hyperbolic decay function similar to that observed in choice behavior. Moreover, the stimulus responses increased with larger reward magnitudes, suggesting that both delay and magnitude constituted viable components of dopamine value signals. In contrast, dopamine responses to the reward itself increased with longer delays, possibly reflecting temporal uncertainty and partial learning. These dopamine reward value signals might serve as useful inputs for brain mechanisms involved in economic choices between delayed rewards.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Self-citationLearning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 1×
“Computational models based on temporal difference (TD) theory reproduced the dopamine responses accurately in both temporal and associative aspects (Sutton, 1988; Houk et al., 1995; Montague et al., 1996; Schultz et al., 1997; Suri and Schultz, 1999).”
MethodSeparate neural systems value immediate and delayed monetary rewards
McClure, S. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2004) · Science · doi:10.1126/science.1100907 · Cited by 3,089 · cited 2×
“The human striatum and orbitofrontal cortex (OFC) showed greater hemodynamic response to immediate rather than delayed reward (McClure et al., 2004; Tanaka et al., 2004).”
BackgroundDiscrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085 · cited 2×
“Dopamine responses to conditioned stimuli Previous studies showed that dopamine neurons change their response to conditioned stimuli proportional to magnitude and probability of the associated reward (Fiorillo et al., 2003; Tobler et al., 2005).”
BackgroundResultSelf-citationA framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 3×
“rewards in their prediction error signals (Montague et al., 1996).”
BackgroundMethodThe neural correlates of subjective value during intertemporal choice
Kable, J. W., & Glimcher, P. W. (2007) · Nature Neuroscience · doi:10.1038/nn2007 · Cited by 1,916
A Discounting Framework for Choice With Delayed and Probabilistic Rewards.
Green, L., & Myerson, J. (2004) · Psychological Bulletin · doi:10.1037/0033-2909.130.5.769 · Cited by 1,676 · cited 1×
“Thus, temporal discounting might share the same mechanisms as probability discounting (Green and Myerson, 2004; Hayden and Platt, 2007).”
BackgroundAdaptive Coding of Reward Value by Dopamine Neurons
Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005) · Science · doi:10.1126/science.1105370 · Cited by 1,276 · cited 2×
“Dopamine responses to conditioned stimuli Previous studies showed that dopamine neurons change their response to conditioned stimuli proportional to magnitude and probability of the associated reward (Fiorillo et al., 2003; Tobler et al., 2005).”
BackgroundResultSelf-citationModels of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
Nucleus accumbens D2/3 receptors predict trait impulsivity and cocaine reinforcement
Dalley, J. W., Fryer, T. D., Brichard, L., Robinson, E. S. J., Theobald, D. E. H., Lääne, K., Peña, Y., Murphy, E. R., Shah, Y., Probst, K., et al. (2007) · Science · doi:10.1126/science.1137073 · Cited by 1,188
Dopamine responses comply with basic assumptions of formal learning theory
Waelti, P., Dickinson, A., & Schultz, W. (2001) · Nature · doi:10.1038/35083500 · Cited by 1,018
Self-citationResponses of monkey dopamine neurons during learning of behavioral reactions
Ljungberg, T., Apicella, P., & Schultz, W. (1992) · Journal of Neurophysiology · doi:10.1152/jn.1992.67.1.145 · Cited by 987
Self-citationImpulsive choice induced in rats by lesions of the nucleus accumbens core
Cardinal, R. N., Pennicott, D. R., Lakmali, C., Sugathapala, Robbins, T. W., & Everitt, B. J. (2001) · Science · doi:10.1126/science.1060818 · Cited by 923 · cited 1×
“The preference of rats for small immediate reward over larger delayed reward increases with lesions of the ventral striatum and basolateral amygdala (Cardinal et al., 2001; Winstanley et al., 2004), and decreases with excitotoxic and dopaminergic lesions of the OFC (Kheramin et al., 2004;…”
BackgroundDISCOUNTING OF DELAYED REWARDS: MODELS OF INDIVIDUAL CHOICE
Myerson, J., & Green, L. (1995) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.1995.64-263 · Cited by 841
Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004) · Nature Neuroscience · doi:10.1038/nn1279 · Cited by 779 · cited 2×
“The human striatum and orbitofrontal cortex (OFC) showed greater hemodynamic response to immediate rather than delayed reward (McClure et al., 2004; Tanaka et al., 2004).”
BackgroundAinslie, G. W. (1974) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.1974.21-485 · Cited by 759
Contrasting Roles of Basolateral Amygdala and Orbitofrontal Cortex in Impulsive Choice
Winstanley, C. A., Theobald, D. E. H., Cardinal, R. N., & Robbins, T. W. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.5606-03.2004 · Cited by 675 · cited 1×
“…rats for small immediate reward over larger delayed reward increases with lesions of the ventral striatum and basolateral amygdala (Cardinal et al., 2001; Winstanley et al., 2004), and decreases with excitotoxic and dopaminergic lesions of the OFC (Kheramin et al., 2004; Winstanley et al., 2004).”
BackgroundRoesch, M. R., Calu, D. J., & Schoenbaum, G. (2007) · Nature Neuroscience · doi:10.1038/nn2013 · Cited by 597
DETERMINATION OF DISCOUNT FUNCTIONS IN RATS WITH AN ADJUSTING‐AMOUNT PROCEDURE
Richards, J. B., Mitchell, S. H., de Wit, H., & Seiden, L. S. (1997) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.1997.67-353 · Cited by 492
Dopamine: generalization and bonuses
Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00048-5 · Cited by 480
TEMPORAL DISCOUNTING: BASIC RESEARCH AND THE ANALYSIS OF SOCIALLY IMPORTANT BEHAVIOR
Critchfield, T. S., & Kollins, S. H. (2001) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.2001.34-101 · Cited by 439 · cited 1×
“Excessive discounting of delayed rewards leads to impulsivity, which is a key characteristic of pathological behaviors such as drug addiction, pathological gambling, and attention-deficit/hyperactivity disorder (for review, see Critchfield and Kollins, 2001).”
BackgroundPreference reversal and delayed reinforcement
Ainslie, G., & Herrnstein, R. J. (1981) · Animal Learning & Behavior · doi:10.3758/bf03209777 · Cited by 382
Schultz, W., & Romo, R. (1990) · Journal of Neurophysiology · doi:10.1152/jn.1990.63.3.607 · Cited by 334 · cited 2×
“Given the frequently observed generalization of dopamine responses to stimuli resembling reward predictors (Schultz and Romo, 1990; Ljungberg et al., 1991, 1992), dopamine neurons might receive separate inputs for the initial activation with poor reward discrimination and the later component that…”
BackgroundSelf-citationCoding of Predicted Reward Omission by Dopamine Neurons in a Conditioned Inhibition Paradigm
Tobler, P. N., Dickinson, A., & Schultz, W. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-32-10402.2003 · Cited by 300 · cited 3×
“As dopamine neurons respond to the difference between the delivered reward and its prediction (Ljungberg et al., 1992; Waelti et al., 2001; Tobler et al., 2003), partial reward prediction would generate a graded positive prediction error at the time of the reward.”
BackgroundResultSelf-citationSuri, R. E., & Schultz, W. (1999) · Neuroscience · doi:10.1016/s0306-4522(98)00697-6 · Cited by 294
Self-citationWhen foragers discount the future: constraint or adaptation?
Kagel, J. H., Green, L., & Caraco, T. (1986) · Animal Behaviour · doi:10.1016/0003-3472(86)90032-1 · Cited by 253
Church, R. M., & Gibbon, J. (1982) · Journal of Experimental Psychology: Animal Behavior Processes · doi:10.1037//0097-7403.8.2.165 · Cited by 224
Adjusting delay to reinforcement: Comparing choice in pigeons and humans.
Rodriguez, M. L., & Logue, A. W. (1988) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.14.1.105 · Cited by 211
LINEAR-NONLINEAR-POISSON MODELS OF PRIMATE CHOICE DYNAMICS
Corrado, G. S., Sugrue, L. P., Seung, H. S., & Newsome, W. T. (2005) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.2005.23-05 · Cited by 206
Schultz, W., & Romo, R. (1987) · Journal of Neurophysiology · doi:10.1152/jn.1987.57.1.201 · Cited by 184 · cited 2×
“5 spikes/s) in sharp contrast to the highfrequency firing of neurons in the substantia nigra pars reticulata (Schultz and Romo, 1987).”
BackgroundInfluentialSelf-citationRepresentation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · cited 1×
“Although introducing the scalar expectancy theory into a TD model is one way to explain the present data (cf. Daw et al., 2006), further experiments are required to measure the time sensitivity of dopamine neurons as a function of delay-related uncertainty.”
MethodTemporal Discounting Predicts Risk Sensitivity in Rhesus Macaques
Hayden, B. Y., & Platt, M. L. (2007) · Current Biology · doi:10.1016/j.cub.2006.10.055 · Cited by 162
Adjusting delay to reinforcement: Comparing choice in pigeons and humans.
Rodríguez, M., & Logue, A. W. (1988) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037//0097-7403.14.1.105 · Cited by 158
PREFERENCE REVERSALS WITH FOOD AND WATER REINFORCERS IN RATS
Green, L., & Estle, S. J. (2003) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.2003.79-233 · Cited by 153
Some Aspects of the Pure Theory of Capital
Samuelson, P. A. (1937) · The Quarterly Journal of Economics · doi:10.2307/1884837 · Cited by 147
CS-US interval as a determinant of the form of Pavlovian appetitive conditioned responses.
Holland, P. C. (1980) · Journal of Experimental Psychology: Animal Behavior Processes · doi:10.1037//0097-7403.6.2.155 · Cited by 135
Responses of monkey midbrain dopamine neurons during delayed alternation performance
Ljungberg, T., Apicella, P., & Schultz, W. (1991) · Brain Research · doi:10.1016/0006-8993(91)90816-e · Cited by 128 · cited 1×
“Given the frequently observed generalization of dopamine responses to stimuli resembling reward predictors (Schultz and Romo, 1990; Ljungberg et al., 1991, 1992), dopamine neurons might receive separate inputs for the initial activation with poor reward discrimination and the later component that…”
BackgroundSelf-citationKheramin, S., Body, S., Ho, M.-Y., Vel�zquez-Martinez, D. N., Bradshaw, C. M., Szabadi, E., Deakin, J. F. W., & Anderson, I. M. (2004) · Psychopharmacology · doi:10.1007/s00213-004-1813-y · Cited by 117 · cited 3×
“…rats for small immediate reward over larger delayed reward increases with lesions of the ventral striatum and basolateral amygdala (Cardinal et al., 2001; Winstanley et al., 2004), and decreases with excitotoxic and dopaminergic lesions of the OFC (Kheramin et al., 2004; Winstanley et al., 2004).”
BackgroundTradeoffs among delay, rate, and amount of reinforcement
Mazur, J. (2000) · Behavioural Processes · doi:10.1016/s0376-6357(00)00070-x · Cited by 109
The influence of CS-US interval on several different indices of learning in appetitive conditioning.
Delamater, A. R., & Holland, P. C. (2008) · Journal of Experimental Psychology: Animal Behavior Processes · doi:10.1037/0097-7403.34.2.202 · Cited by 89
Kobayashi, S., Kawagoe, R., Takikawa, Y., Koizumi, M., Sakagami, M., & Hikosaka, O. (2006) · Experimental Brain Research · doi:10.1007/s00221-006-0622-4 · Cited by 51
Self-citation
Cited by
Dopamine in motivational control: rewarding, aversive, and alerting
Bromberg-Martin, E. S., Matsumoto, M., & Hikosaka, O. (2010) · Neuron · doi:10.1016/j.neuron.2010.11.022 · Cited by 2,396
Neuronal Reward and Decision Signals: From Theories to Data
Schultz, W. (2015) · Physiological Reviews · doi:10.1152/physrev.00023.2014 · Cited by 1,126
Self-citationDopamine reward prediction-error signalling: A two-component response
Schultz, W. (2016) · Nature Reviews Neuroscience · doi:10.1038/nrn.2015.26 · Cited by 1,036
Self-citationThe Neurobiology of Decision: Consensus and Controversy
Kable, J. W., & Glimcher, P. W. (2009) · Neuron · doi:10.1016/j.neuron.2009.09.003 · Cited by 928
Mesolimbic dopamine signals the value of work
Hamid, A. A., Pettibone, J. R., Mabrouk, O. S., Hetrick, V. L., Schmidt, R., Vander Weele, C. M., Kennedy, R. T., Aragona, B. J., & Berke, J. D. (2016) · Nature Neuroscience · doi:10.1038/nn.4173 · Cited by 824
Metadata
- Authors
- Kobayashi, S., & Schultz, W.
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2008-07-30
- Volume
- 28
- Issue
- 31
- Pages
- 7837-7846
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- OpenAlex
- W2046479615
- Cited by
- 430
- References
- 41
- FWCI
- 9.58
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.