Dopamine selectively remediates ‘model-based’ reward learning: a computational approach
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Patients with loss of dopamine due to Parkinson’s disease are impaired at learning from reward. However, it remains unknown precisely which aspect of learning is impaired. In particular, learning from reward, or reinforcement learning, can be driven by two distinct computational processes. One involves habitual stamping-in of stimulus-response associations, hypothesized to arise computationally from ‘model-free’ learning. The other, ‘model-based’ learning, involves learning a model of the world that is believed to support goal-directed behaviour. Much work has pointed to a role for dopamine in model-free learning. But recent work suggests model-based learning may also involve dopamine modulation, raising the possibility that model-based learning may contribute to the learning impairment in Parkinson’s disease. To directly test this, we used a two-step reward-learning task which dissociates model-free versus model-based learning. We evaluated learning in patients with Parkinson’s disease tested ON versus OFF their dopamine replacement medication and in healthy controls. Surprisingly, we found no effect of disease or medication on model-free learning. Instead, we found that patients tested OFF medication showed a marked impairment in model-based learning, and that this impairment was remediated by dopaminergic medication. Moreover, model-based learning was positively correlated with a separate measure of working memory performance, raising the possibility of common neural substrates. Our results suggest that some learning deficits in Parkinson’s disease may be related to an inability to pursue reward based on complete representations of the environment. Patients with Parkinson’s disease show impaired reinforcement learning, but the exact nature of this deficit and the role of dopamine remain unclear. Using computational methods, Sharp et al. show that patients with Parkinson’s disease have a specific model-based, i.e. goal-directed, learning deficit that can be completely remediated by dopamine replacement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Self-citationThe role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402
By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
A neostriatal habit learning system in humans
Knowlton, B. J., Mangels, J. A., & Squire, L. R. (1996) · Science · doi:10.1126/science.273.5280.1399 · Cited by 1,777
D1 Dopamine Receptors in Prefrontal Cortex: Involvement in Working Memory
Sawaguchi, T., & Goldman-Rakic, P. S. (1991) · Science · doi:10.1126/science.1825731 · Cited by 1,364 · cited 1×
“For instance, dopamine is known to support working memory through innervation of the prefrontal cortex (Sawaguchi and Goldman-Rakic, 1991), which could in turn support the manipulation of action-outcome and outcome-value representations in computing action values.”
BackgroundModels of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977
Cools, R. (2001) · Cerebral Cortex · doi:10.1093/cercor/11.12.1136 · Cited by 906 · cited 3×
“An important goal for future research, especially in the context of treatment of cognitive symptoms in Parkinson’s disease, is to further understand the interdependence of striatal learning processes and executive function, and to identify where exogenous dopamine is exerting its positive effect on model-based learnin…”
BackgroundDisorders of compulsivity: A common bias towards learning habits
Voon, V, Derbyshire, K., Rück, C., Irvine, M. A., Worbe, Y., Enander, J., Schreiber, L. R. N., Gillan, C., Fineberg, N. A., Sahakian, B. J., et al. (2015) · Molecular Psychiatry · doi:10.1038/mp.2014.44 · Cited by 643 · cited 3×
“Neuroimaging studies in humans have shown that the caudate, medial orbitofrontal and dorsome-dial prefrontal cortex are associated with model-based learning (Daw et al. , 2011; Doll et al. , 2012; Voon et al. , 2015).”
BackgroundSelf-citationThe ubiquity of model-based reinforcement learning
Doll, B. B., Simon, D. A., & Daw, N. D. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.08.003 · Cited by 380
Self-citationOtto (2013) · doi:10.1177/0956797612463080 · Cited by 352
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · cited 1×
“It is also possible that model-free learning is a very robust cognitive function, in keeping with its role as a faster but less accurate system (Keramati et al., 2011), and thus that model-free learning, but not model-based, can withstand a certain degree of dopamine deficiency.”
BackgroundHabits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297 · cited 1×
“…of model-free learning and the psychological category of habits is also more controversial than that between model-based and goal-directed learning (Dezfouli and Balleine, 2012), though on that account (according to which habitual behaviour and seemingly model-free behaviour on the sequential…”
BackgroundDopamine enhances model-based over model-free choice behavior
Wunderlich, K., Smittenaar, P., & Dolan, R. J. (2012) · Neuron · doi:10.1016/j.neuron.2012.03.042 · Cited by 294 · cited 1×
“Our findings are in line with previous studies showing that levodopa administered to healthy, young adults increased model-based learning (Wunderlich et al. , 2012).”
ResultAtomoxetine for depression and other neuropsychiatric symptoms in Parkinson disease
Weintraub, D., Mavandadi, S., Mamikonyan, E., Siderowf, A. D., Duda, J. E., Hurtig, H. I., Colcher, A., Horn, S. S., Nazem, S., Ten Have, T. R., et al. (2010) · Neurology · doi:10.1212/wnl.0b013e3181ebdd79 · Cited by 255 · cited 2×
“…et al. , 2010), and frontostriatal (Cools et al. , 2002; Lewis et al. , 2003, 2005; Monchi et al. , 2007; Ko et al. , 2013; Nagano-Saito et al. , 2014) and extra-striatal non-dopaminergic networks for executive function (Hilker et al. , 2005; Weintraub et al. , 2010; Ye et al. , 2014, 2015).”
BackgroundSmittenaar, P., FitzGerald, T. H. B., Romei, V., Wright, N. D., & Dolan, R. J. (2013) · Neuron · doi:10.1016/j.neuron.2013.08.009 · Cited by 244 · cited 7×
“More generally, normal pre-frontal cortex function (Smittenaar et al. , 2013) and executive functioning (Otto et al. , 2013 a , 2015) are necessary for model-based learning.”
BackgroundMethodInfluentialDeserno, L., Huys, Q. J. M., Boehme, R., Buchert, R., Heinze, H.-J., Grace, A. A., Dolan, R. J., Heinz, A., & Schlagenhauf, F. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1417219112 · Cited by 233
The algorithmic anatomy of model-based evaluation
Daw, N. D., & Dayan, P. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0478 · Cited by 195
Self-citationHabitual versus goal-directed action control in Parkinson disease
de Wit, S., Barker, R. A., Dickinson, A. D., & Cools, R. (2011) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2010.21514 · Cited by 116 · cited 4×
“…learning in patients with Parkinson’s disease used an instrumental conflict task modelled on those used to study habit learning in rodents, and also showed that patients with Parkinson’s disease had intact stimulus-response habit learning, which relates to model-free learning (de Wit et al. , 2011).”
MethodResultInfluential
Cited by
Predictive representations can link model-based reinforcement learning to model-free mechanisms
Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., & Daw, N. D. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005768 · Cited by 326
Self-citationMiller, K. J., Shenhav, A., & Ludvig, E. A. (2019) · Psychological Review · doi:10.1037/rev0000120 · Cited by 280
“Patients with Parkinson’s Disease, in which dopamine neurons die in large numbers, show both impaired model-based behavior (Sharp, Foerde, Daw, & Shohamy, 2015) and increased perseveration (Rutledge et al., 2009), both of which are mitigated by dopamine-restoring drugs.”
BackgroundWhen Does Model-Based Control Pay Off?
Kool, W., Cushman, F. A., & Gershman, S. J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005090 · Cited by 211
“This task has proven to be a useful and popular tool to characterize the neural [8, 10 – 18], behavioral [19 – 31] and clinical [32 – 35] implications of dual-process models within the reinforcement learning framework.”
MethodModel-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161
Influences of dopaminergic system dysfunction on late-life depression
Taylor, W. D., Zald, D. H., Felger, J. C., Christman, S., Claassen, D. O., Horga, G., Miller, J. M., Gifford, K., Rogers, B., Szymkowicz, S. M., et al. (2022) · Molecular Psychiatry · doi:10.1038/s41380-021-01265-0 · Cited by 89
Metadata
- Authors
- Sharp, M. E., Foerde, K., Daw, N. D., & Shohamy, D.
- Venue
- Brain
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-12-17
- Volume
- 139
- Issue
- 2
- Pages
- 355-364
- ISSN
- 0006-8950 (ISSN-L)
- 1460-2156
- OpenAlex
- W2222166274
- Cited by
- 128
- References
- 19
- FWCI
- 5.36
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.