A Role for Dopamine in Temporal Decision Making and Reward Maximization in Parkinsonism
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Converging evidence implicates striatal dopamine (DA) in reinforcement learning, such that DA increases enhance “Go learning” to pursue actions with rewarding outcomes, whereas DA decreases enhance “NoGo learning” to avoid non-rewarding actions. Here we test whether these effects apply to the response time domain. We employ a novel paradigm which requires the adjustment of response times to a single response. Reward probability varies as a function of response time, whereas reward magnitude changes in the opposite direction. In the control condition, these factors exactly cancel, such that the expected value across time is constant (CEV). In two other conditions, expected value increases (IEV) or decreases (DEV), such that reward maximization requires either speeding up (Go learning) or slowing down (NoGo learning) relative to the CEV condition. We tested patients with Parkinson's disease (depleted striatal DA levels) on and off dopaminergic medication, compared with age-matched controls. While medicated, patients were better at speeding up in the DEV relative to CEV conditions. Conversely, nonmedicated patients were better at slowing down to maximize reward in the IEV condition. These effects of DA manipulation on cumulative Go/NoGo response time adaptation were captured with our a priori computational model of the basal ganglia, previously applied only to forced-choice tasks. There were also robust trial-to-trial changes in response time, but these single trial adaptations were not affected by disease or medication and are posited to rely on extrastriatal, possibly prefrontal, structures.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
Neural systems of reinforcement for drug addiction: from actions to habits to compulsion
Everitt, B. J., & Robbins, T. W. (2005) · Nature Neuroscience · doi:10.1038/nn1579 · Cited by 3,785 · cited 1×
“For example, the tendencies to speed responses to obtain large rewards and to approach a rewardpredicting stimulus are both dependent on striatal D1 receptors (Dalley et al., 2005; Everitt and Robbins, 2005; Nakamura and Hikosaka, 2006).”
BackgroundSeparate neural systems value immediate and delayed monetary rewards
McClure, S. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2004) · Science · doi:10.1126/science.1100907 · Cited by 3,089 · cited 1×
“Perhaps the most relevant intertemporal choice paradigm is that of “delay discounting” (McClure et al., 2004, 2007; Hariri et al., 2006; Scheres et al., 2006; Heerey et al., 2007).”
BackgroundDaw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · cited 1×
“To the extent that such strategies are used in this task, they might be supported by other rule-based brain systems (e.g., Ashby and O’Brien, 2005; Daw et al., 2005).”
MethodThe debate over dopamine’s role in reward: the case for incentive salience
Berridge, K. C. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0578-x · Cited by 2,347
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 1×
“We also examined whether a standard temporal difference (TD[ ]) reinforcement learning model (Montague et al., 1996; Schultz et al., 1997; Sutton and Barto, 1998; McClure et al., 2003) could capture the pattern of behavior observed in the humans and neural network model.”
BackgroundBy Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Self-citationA neostriatal habit learning system in humans
Knowlton, B. J., Mangels, J. A., & Squire, L. R. (1996) · Science · doi:10.1126/science.273.5280.1399 · Cited by 1,777
Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans
Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006) · Nature · doi:10.1038/nature05051 · Cited by 1,584
Adaptive Coding of Reward Value by Dopamine Neurons
Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005) · Science · doi:10.1126/science.1105370 · Cited by 1,276 · cited 3×
“Together these features are meant to reflect the observed property of DA neurons in monkeys, where phasic firing is proportional to reward prediction error rather than reward value itself, and where firing rates are normalized relative to the largest current available reward (Tobler et al., 2005).”
BackgroundSurmeier, D. J., Ding, J., Day, M., Wang, Z., & Shen, W. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.03.008 · Cited by 1,203 · cited 2×
“Furthermore, depleted DA levels as in PD would enhance this effect, because of D2 receptor sensitivity (Seeman, 2008) and enhanced excitability of striatopallidal NoGo cells in the DA-depleted state (Surmeier et al., 2007).”
BackgroundDichotomous dopaminergic control of striatal synaptic plasticity
Shen, W., Flajolet, M., Greengard, P., & Surmeier, D. J. (2008) · Science · doi:10.1126/science.1160575 · Cited by 1,190 · cited 4×
“These D1 and D2 data converge with a recently reported study showing that these receptor types modulate synaptic plasticity in striatal Go and NoGo populations (Shen et al., 2008), and with human genetic data examining polymorphisms within dopaminergic genes and their effects on Go and NoGo learning (Frank et al.”
BackgroundResultInfluentialTonic dopamine: Opportunity costs and the control of response vigor
Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0502-4 · Cited by 1,133 · cited 5×
“…1998; Frank et al., 2001; Gurney et al., 2001; Frank, 2006; Houk et al., 2007; Moustafa and Maida, 2007) and that striatal DA modulates reward-based learning and performance (Suri and Schultz, 1998; Delgado et al., 2000, 2005; Doya, 2000; Frank, 2005; Shohamy et al., 2006; Niv et al., 2007).”
BackgroundMethodResultInfluentialTracking the Hemodynamic Responses to Reward and Punishment in the Striatum
Delgado, M. R., Nystrom, L. E., Fissell, C., Noll, D. C., & Fiez, J. A. (2000) · Journal of Neurophysiology · doi:10.1152/jn.2000.84.6.3072 · Cited by 1,116 · cited 2×
“…1998; Frank et al., 2001; Gurney et al., 2001; Frank, 2006; Houk et al., 2007; Moustafa and Maida, 2007) and that striatal DA modulates reward-based learning and performance (Suri and Schultz, 1998; Delgado et al., 2000, 2005; Doya, 2000; Frank, 2005; Shohamy et al., 2006; Niv et al., 2007).”
BackgroundThe Basal Ganglia and Chunking of Action Repertoires
Graybiel, A. M. (1998) · Neurobiology of Learning and Memory · doi:10.1006/nlme.1998.3843 · Cited by 1,042 · cited 1×
“…adaptations rely on active maintenance of recent outcomes in the orbitofrontal cortex, whereas probabilistic learning depends on incremental synaptic weight changes in the basal ganglia (Knowlton et al., 1996; Packard and McGaugh, 1996; Graybiel, 1998, 2004; Delgado et al., 2000, 2005).”
BackgroundNeural Economics and the Biological Substrates of Valuation
Montague, P. R., & Berns, G. S. (2002) · Neuron · doi:10.1016/s0896-6273(02)00974-1 · Cited by 994
Time Discounting for Primary Rewards
McClure, S. M., Ericson, K. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.4246-06.2007 · Cited by 971
Complementary roles of basal ganglia and cerebellum in learning and motor control
Doya, K. (2000) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(00)00153-7 · Cited by 964
Frank, M. J. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/0898929052880093 · Cited by 925
Self-citationCools, R. (2006) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2005.03.024 · Cited by 903
A cellular mechanism of reward-related learning
Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001) · Nature · doi:10.1038/35092560 · Cited by 879 · cited 1×
“In the models, phasic DA signals modify synaptic plasticity in the corticostriatal pathway (Wickens et al., 1996; Reynolds et al., 2001).”
BackgroundInteractions between frontal cortex and basal ganglia in working memory: A computational model
FRANK, M. J., LOUGHRY, B., & O'REILLY, R. C. (2001) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.1.2.137 · Cited by 879 · cited 1×
“…models suggest that circuits linking basal ganglia with frontal cortex support action selection (Berns and Sejnowski, 1995; Suri and Schultz, 1998; Frank et al., 2001; Gurney et al., 2001; Frank, 2006; Houk et al., 2007; Moustafa and Maida, 2007) and that striatal DA modulates reward-based…”
BackgroundSelf-citationHold your horses: a dynamic computational role for the subthalamic nucleus in decision making
Frank, M. J. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.03.006 · Cited by 706 · cited 9×
“That is, Go signals from the striatum directly inhibit the corresponding column of the globus pallidus (GPi).”
BackgroundMethodInfluentialSelf-citationWhy Don't We Move Faster? Parkinson's Disease, Movement Vigor, and Implicit Motivation
Mazzoni, P., Hristova, A., & Krakauer, J. W. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0264-07.2007 · Cited by 557
Hariri, A. R., Brown, S. M., Williamson, D. E., Flory, J. D., de Wit, H., & Manuck, S. B. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.3446-06.2006 · Cited by 557
Mechanisms of cognitive set flexibility in Parkinson's disease
Cools, R. (2001) · Brain · doi:10.1093/brain/124.12.2503 · Cited by 406
Scheres, A., Dijkstra, M., Ainslie, E., Balkan, J., Reynolds, B., Sonuga-Barke, E., & Castellanos, F. X. (2006) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2005.10.012 · Cited by 327 · cited 1×
“Perhaps the most relevant intertemporal choice paradigm is that of “delay discounting” (McClure et al., 2004, 2007; Hariri et al., 2006; Scheres et al., 2006; Heerey et al., 2007).”
BackgroundCorrelated Coding of Motivation and Outcome of Decision by Dopamine Neurons
Satoh, T., Nakai, S., Sato, T., & Kimura, M. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-30-09913.2003 · Cited by 298
Delay discounting in schizophrenia
Heerey, E. A., Robinson, B. M., McMahon, R. P., & Gold, J. M. (2007) · Cognitive Neuropsychiatry · doi:10.1080/13546800601005900 · Cited by 248 · cited 1×
“Perhaps the most relevant intertemporal choice paradigm is that of “delay discounting” (McClure et al., 2004, 2007; Hariri et al., 2006; Scheres et al., 2006; Heerey et al., 2007).”
BackgroundFrank, M. J., Santamaria, A., O'Reilly, R. C., & Willcutt, E. (2007) · Neuropsychopharmacology · doi:10.1038/sj.npp.1301278 · Cited by 227 · cited 9×
“In choice paradigms, DA medications potentiate Go learning while also impairing NoGo learning (Frank et al., 2004, 2007b; Frank, 2005; Cools et al., 2006; Frank and O’Reilly, 2006; Pessiglione et al., 2006; Shohamy et al., 2008).”
BackgroundMethodResultInfluentialDalley, J. W., Lääne, K., Theobald, D. E. H., Armstrong, H. C., Corlett, P. R., Chudasama, Y., & Robbins, T. W. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0502080102 · Cited by 214
A dopaminergic basis for working memory, learning and attentional shifting in Parkinsonism
Moustafa, A. A., Sherman, S. J., & Frank, M. J. (2008) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2008.07.011 · Cited by 192
Self-citation
Cited by
From prediction error to incentive salience: mesolimbic computation of reward motivation.
Berridge, K. C. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.07990.x · Cited by 700
“Also, dopamine manipulation by drugs have been reported to change learning and learned performance in people (Pessiglione et al., 2006; Moustafa et al., 2008; Samson et al., 2010).”
BackgroundNeurophysiology of performance monitoring and adaptive behavior
Ullsperger, M., Danielmeier, C., & Jocham, G. (2014) · Physiological Reviews · doi:10.1152/physrev.00041.2012 · Cited by 634
How prediction errors shape perception, attention, and motivation
den Ouden, H. E. M., Kok, P., & de Lange, F. P. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2012.00548 · Cited by 512
“Systemic manipulations of dopamine levels in patients and healthy subjects show opposite effects on rewardand punishment-based learning (Frank et al., 2004; Moustafa et al., 2008; Cools et al., 2009; Palminteri et al., 2009).”
BackgroundFrank, M. J., Doll, B. B., Oas-Terpstra, J., & Moreno, F. (2009) · Nature Neuroscience · doi:10.1038/nn.2342 · Cited by 471
Self-citationBodi, N., Keri, S., Nagy, H., Moustafa, A., Myers, C. E., Daw, N., Dibo, G., Takats, A., Bereczki, D., & Gluck, M. A. (2009) · Brain · doi:10.1093/brain/awp094 · Cited by 363
Self-citation
Metadata
- Authors
- Moustafa, A. A., Cohen, M. X., Sherman, S. J., & Frank, M. J.
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2008-11-19
- Volume
- 28
- Issue
- 47
- Pages
- 12294-12304
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- OpenAlex
- W1993550156
- Cited by
- 144
- References
- 32
- FWCI
- 4.22
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.