Reinforcement Learning and Dopamine in Schizophrenia: Dimensions of Symptoms or Specific Features of a Disease Group?
Open accessThe DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Abstract
Abnormalities in reinforcement learning are a key finding in schizophrenia and have been proposed to be linked to elevated levels of dopamine neurotransmission. Behavioral deficits in reinforcement learning and their neural correlates may contribute to the formation of clinical characteristics of schizophrenia. The ability to form predictions about future outcomes is fundamental for environmental interactions and depends on neuronal teaching signals, like reward prediction errors. While aberrant prediction errors, that encode non-salient events as surprising, have been proposed to contribute to the formation of positive symptoms, a failure to build neural representations of decision values may result in negative symptoms. Here, we review behavioral and neuroimaging research in schizophrenia and focus on studies that implemented reinforcement learning models. In addition, we discuss studies that combined reinforcement learning with measures of dopamine. Thereby, we suggest how reinforcement learning abnormalities in schizophrenia may contribute to the formation of psychotic symptoms and may interact with cognitive deficits. These ideas point toward an interplay of more rigid versus flexible control over reinforcement learning. Pronounced deficits in the flexible or model-based domain may allow for a detailed characterization of well-established cognitive deficits in schizophrenia patients based on computational models of learning. Finally, we propose a framework based on the potentially crucial contribution of dopamine to dysfunctional reinforcement learning on the level of neural networks. Future research may strongly benefit from computational modeling but also requires further methodological improvement for clinical group studies. These research tools may help to improve our understanding of disease-specific mechanisms and may help to identify clinically relevant subgroups of the heterogeneous entity schizophrenia.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kapur, S. (2003) · American Journal of Psychiatry · doi:10.1176/appi.ajp.160.1.13 · Cited by 2,813 · cited 2×
“We begin with the“aberrant salience”hypothesis: schizophrenia patients may attribute salience to otherwise neutral environmental stimuli, and those stimuli may ultimately appear meaningful and evoke delusional mood in patients (9, 10).”
BackgroundAnticipation of Increasing Monetary Reward Selectively Recruits Nucleus Accumbens
Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-16-j0002.2001 · Cited by 2,244
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 1×
“Furthermore, prediction errors are involved in learningrelated changes in synaptic plasticity (12), and aberrant plasticity has been suggested as a potential common biological mechanism characterizing the schizophrenia spectrum (13, 14).”
BackgroundLearning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Grace, A. A. (1991) · Neuroscience · doi:10.1016/0306-4522(91)90196-u · Cited by 1,831
Inverted-U–Shaped Dopamine Actions on Human Working Memory and Cognitive Control
Cools, R., & D'Esposito, M. (2011) · Biological Psychiatry · doi:10.1016/j.biopsych.2011.03.028 · Cited by 1,673 · cited 1×
“Therefore, dopamine effects in learning and cognition appear to be a fine-tuned and optimized non-linear system where rather low and rather high levels result in inefficient neural processing (127, 128).”
BackgroundThe Psychosis High-Risk State: A Comprehensive State-of-the-Art Review
Fusar-Poli, P., Borgwardt, S., Bechdolf, A., Addington, J., Riecher-Rössler, A., Schultze-Lutter, F., Keshavan, M., Wood, S., Ruhrmann, S., Seidman, L. J., et al. (2013) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2013.269 · Cited by 1,483
Bayesian model selection for group studies
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.025 · Cited by 1,423
Functional imaging of neural responses to expectancy and experience of monetary gains and losses
Breiter, H. C., Aharon, I., Kahneman, D., Dale, A., & Shizgal, P. (2001) · Neuron · doi:10.1016/s0896-6273(01)00303-8 · Cited by 1,404
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304
Gläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
Dysconnection in schizophrenia: from abnormal synaptic plasticity to failures of self-monitoring
Stephan, K. E., Friston, K. J., & Frith, C. D. (2009) · Schizophrenia Bulletin · doi:10.1093/schbul/sbn176 · Cited by 1,134 · cited 1×
“As a potential mechanism behind this, a disrupted cortico-cortical synaptic plasticity was suggested by a comprehensive biological hypothesis of schizophrenia, the “dysconnectivity” hypothesis (100, 101).”
BackgroundDistinct roles for direct and indirect pathway striatal neurons in reinforcement
Kravitz, A. V., Tye, L. D., & Kreitzer, A. C. (2012) · Nature Neuroscience · doi:10.1038/nn.3100 · Cited by 972
Synaptic plasticity and dysconnection in schizophrenia
Stephan, K. E., Baldeweg, T., & Friston, K. J. (2006) · Biological Psychiatry · doi:10.1016/j.biopsych.2005.10.005 · Cited by 841 · cited 2×
“As a potential mechanism behind this, a disrupted cortico-cortical synaptic plasticity was suggested by a comprehensive biological hypothesis of schizophrenia, the “dysconnectivity” hypothesis (100, 101).”
BackgroundTen simple rules for dynamic causal modeling
Stephan, K. E., Penny, W. D., Moran, R. J., den Ouden, H. E. M., Daunizeau, J., & Friston, K. J. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.015 · Cited by 803 · cited 1×
“Consequently, it appears to be desirable to develop methodological guidelines for these techniques, as it was published for other modeling approaches, for example for dynamic causal modeling for fMRI (116).”
BackgroundA Bayesian foundation for individual learning under uncertainty
Mathys, C. (2011) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2011.00039 · Cited by 655
Substantia nigra/ventral tegmental reward prediction error disruption in psychosis
Murray, G. K., Corlett, P. R., Clark, L., Pessiglione, M., Blackwell, A. D., Honey, G., Jones, P. B., Bullmore, E. T., Robbins, T. W., & Fletcher, P. C. (2008) · Molecular Psychiatry · doi:10.1038/sj.mp.4002058 · Cited by 532 · cited 3×
“(43) and found that methamphetamine blunts both reward prediction errors in VS and expected value representation in ventro-medial PFC (119).”
BackgroundMethodHampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498
Expected value and prediction error abnormalities in depression and schizophrenia
Gradin, V. B., Kumar, P., Waiter, G., Ahearn, T., Stickle, C., Milders, M., Reid, I., Hall, J., & Steele, J. D. (2011) · Brain · doi:10.1093/brain/awr059 · Cited by 485
Dopaminergic dysfunction in schizophrenia: Salience attribution revisited
Heinz, A., & Schlagenhauf, F. (2010) · Schizophrenia Bulletin · doi:10.1093/schbul/sbq031 · Cited by 412
Self-citationTseng, K. Y., & O'Donnell, P. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.1021-04.2004 · Cited by 402
Bello, E. P., Mateo, Y., Gelman, D. M., Noaín, D., Shin, J. H., Low, M. J., Alvarez, V. A., Lovinger, D. M., & Rubinstein, M. (2011) · Nature Neuroscience · doi:10.1038/nn.2862 · Cited by 364 · cited 1×
“Animal studies (122, 123) and other functional human imaging studies (36, 124, 125) also support the idea of this interplay of differential aspects of dopamine neurotransmission.”
BackgroundOtto (2013) · doi:10.1177/0956797612463080 · Cited by 352 · cited 1×
“There is indeed evidence that working memory capacity limits the ability to learn multiple stimulus values to guide decisions and the degree of model-based behavior (81, 82), while, at the same time, possibly directing patients toward more inflexible aspects of learning, which themselves may be affected or spared in s…”
BackgroundLinking nucleus accumbens dopamine and blood oxygenation
Knutson, B., & Gibbs, S. E. B. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0686-7 · Cited by 337 · cited 2×
“In addition, ventral striatal reward anticipation as in the MID task appears to be conveyed by reward-induced dopamine release (27) and can be blunted by massive dopamine release, based on dose-dependent effects of psychostimulants (26, 28).”
BackgroundWaltz, J. A., & Gold, J. M. (2007) · Schizophrenia Research · doi:10.1016/j.schres.2007.03.010 · Cited by 331
Juckel, G., Schlagenhauf, F., Koslowski, M., Filonov, D., Wüstenberg, T., Villringer, A., Knutson, B., Kienast, T., Gallinat, J., Wrase, J., et al. (2006) · Psychopharmacology · doi:10.1007/s00213-006-0405-4 · Cited by 323
Self-citationA Dual Role for Prediction Error in Associative Learning
den Ouden, H. E. M., Friston, K. J., Daw, N. D., McIntosh, A. R., & Stephan, K. E. (2009) · Cerebral Cortex · doi:10.1093/cercor/bhn161 · Cited by 306
Striatal Prediction Error Modulates Cortical Coupling
den Ouden, H. E. M., Daunizeau, J., Roiser, J., Friston, K. J., & Stephan, K. E. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4458-09.2010 · Cited by 304 · cited 2×
“This may indicate the need to extend from standard reinforcement learning models to other types of models, for example Bayesian learners (94, 113, 114).”
BackgroundHeinz, A. (2002) · European Psychiatry · doi:10.1016/s0924-9338(02)00628-4 · Cited by 270
Self-citationNeuropsychological evidence for frontostriatal dysfunction in schizophrenia
Elliott, R., McKenna, P. J., Robbins, T. W., & Sahakian, B. J. (1995) · Psychological Medicine · doi:10.1017/s0033291700033523 · Cited by 267
Do patients with schizophrenia exhibit aberrant salience?
Roiser, J. P., Stephan, K. E., den Ouden, H. E. M., Barnes, T. R. E., Friston, K. J., & Joyce, E. M. (2009) · Psychological Medicine · doi:10.1017/s0033291708003863 · Cited by 262
Striatal dysfunction during reversal learning in unmedicated schizophrenia patients
Schlagenhauf, F., Huys, Q. J. M., Deserno, L., Rapp, M. A., Beck, A., Heinze, H.-J., Dolan, R., & Heinz, A. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.11.034 · Cited by 253 · cited 10×
“Modeling results revealed increased switching in patients due to false beliefs with respect to feedback-conveyed information about the state of the task, which are based on reversals of reward contingencies (77).”
BackgroundMethodInfluentialSelf-citationDelay discounting in schizophrenia
Heerey, E. A., Robinson, B. M., McMahon, R. P., & Gold, J. M. (2007) · Cognitive Neuropsychiatry · doi:10.1080/13546800601005900 · Cited by 248 · cited 1×
“In line with another study (59), patients responded faster on neutral trials in the study by Murray et al.”
ResultFusar-Poli, P., Howes, O. D., Allen, P., Broome, M., Valli, I., Asselin, M.-C., Grasby, P. M., & McGuire, P. K. (2010) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2010.77 · Cited by 234
The formation of abnormal associations in schizophrenia: Neural and behavioral evidence
Jensen, J., Willeit, M., Zipursky, R. B., Savina, I., Smith, A. J., Menon, M., Crawley, A. P., & Kapur, S. (2008) · Neuropsychopharmacology · doi:10.1038/sj.npp.1301437 · Cited by 225
Insel, T. R. (2010) · JAMA · doi:10.1001/jama.2010.555 · Cited by 224
Leeson, V. C., Robbins, T. W., Matheson, E., Hutton, S. B., Ron, M. A., Barnes, T. R. E., & Joyce, E. M. (2009) · Biological Psychiatry · doi:10.1016/j.biopsych.2009.05.016 · Cited by 219
Reward system activation in schizophrenic patients switched from typical neuroleptics to olanzapine
Schlagenhauf, F., Juckel, G., Koslowski, M., Kahnt, T., Knutson, B., Dembler, T., Kienast, T., Gallinat, J., Wrase, J., & Heinz, A. (2008) · Psychopharmacology · doi:10.1007/s00213-007-1016-4 · Cited by 204
Self-citationReward feedback alterations in unmedicated schizophrenia patients: Relevance for delusions
Schlagenhauf, F., Sterzer, P., Schmack, K., Ballmaier, M., Rapp, M., Wrase, J., Juckel, G., Gallinat, J., & Heinz, A. (2009) · Biological Psychiatry · doi:10.1016/j.biopsych.2008.12.016 · Cited by 195
Waltz, J. A., Schweitzer, J. B., Gold, J. M., Kurup, P. K., Ross, T. J., Jo Salmeron, B., Rose, E. J., McClure, S. M., & Stein, E. A. (2009) · Neuropsychopharmacology · doi:10.1038/npp.2008.214 · Cited by 186
Anhedonia and Emotional Experience in Schizophrenia: Neural and Behavioral Indicators
Dowd, E. C., & Barch, D. M. (2010) · Biological Psychiatry · doi:10.1016/j.biopsych.2009.10.020 · Cited by 184
Amphetamine modulates human incentive processing
Knutson, B., Bjork, J. M., Fong, G. W., Hommer, D., Mattay, V. S., & Weinberger, D. R. (2004) · Neuron · doi:10.1016/j.neuron.2004.06.030 · Cited by 174 · cited 2×
“In addition, ventral striatal reward anticipation as in the MID task appears to be conveyed by reward-induced dopamine release (27) and can be blunted by massive dopamine release, based on dose-dependent effects of psychostimulants (26, 28).”
BackgroundDissecting psychiatric spectrum disorders by generative embedding
Brodersen, K. H., Deserno, L., Schlagenhauf, F., Lin, Z., Penny, W. D., Buhmann, J. M., & Stephan, K. E. (2014) · NeuroImage: Clinical · doi:10.1016/j.nicl.2013.11.002 · Cited by 173
Self-citationGenerative embedding for model-based classification of fMRI data
Brodersen, K. H., Schofield, T. M., Leff, A. P., Ong, C. S., Lomakina, E. I., Buhmann, J. M., & Stephan, K. E. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002079 · Cited by 165
Abnormal Responses to Monetary Outcomes in Cortex, but not in the Basal Ganglia, in Schizophrenia
Waltz, J. A., Schweitzer, J. B., Ross, T. J., Kurup, P. K., Salmeron, B. J., Rose, E. J., Gold, J. M., & Stein, E. A. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2010.126 · Cited by 151 · cited 3×
“Activation to reward delivery in lateral PFC was negatively correlated with the degree of positive and negative symptoms (25).”
BackgroundResultDisambiguating ventral striatum fMRI-related bold signal during reward prediction in schizophrenia
Morris, R. W., Vercammen, A., Lenroot, R., Moore, L., Langton, J. M., Short, B., Kulkarni, J., Curtis, J., O'Donnell, M., Weickert, C. S., et al. (2012) · Molecular Psychiatry · doi:10.1038/mp.2011.75 · Cited by 149 · cited 3×
“was exaggerated, response to unexpected outcomes in left VS was found to be blunted (33).”
BackgroundWaltz, J. A., Frank, M. J., Wiecki, T. V., & Gold, J. M. (2011) · Neuropsychology · doi:10.1037/a0020882 · Cited by 132
Cerebral Correlates of Salient Prediction Error for Different Rewards and Punishments
Metereau, E., & Dreher, J.-C. (2013) · Cerebral Cortex · doi:10.1093/cercor/bhs037 · Cited by 114
Midbrain activation during pavlovian conditioning and delusional symptoms in schizophrenia
Romaniuk, L., Honey, G. D., King, J. R. L., Whalley, H. C., McIntosh, A. M., Levita, L., Hughes, M., Johnstone, E. C., Day, M., Lawrie, S. M., et al. (2010) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2010.169 · Cited by 113
Koch, K., Schachtzabel, C., Wagner, G., Schikora, J., Schultz, C., Reichenbach, J. R., Sauer, H., & Schlösser, R. G. M. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.12.031 · Cited by 101 · cited 1×
“study that investigated 20 medicated patients while performing a guessing–gambling paradigm at different levels of uncertainty but analyzed expectation-related and reward-related activation separately (103).”
BackgroundSchmidt, A., Smieskova, R., Aston, J., Simon, A., Allen, P., Fusar-Poli, P., McGuire, P. K., Riecher-Rössler, A., Stephan, K. E., & Borgwardt, S. (2013) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2013.117 · Cited by 93 · cited 1×
“So far, there is converging evidence that dysconnectivity may indeed account for the repeatedly described prefrontal inefficiency observed in schizophrenia during the performance of cognitive tasks: Using models of effective connectivity for fMRI, reduced working-memory-dependent prefrontal-parietal connectivity was r…”
BackgroundSalience network-midbrain dysconnectivity and blunted reward signals in schizophrenia
Gradin, V. B., Waiter, G., O'Connor, A., Romaniuk, L., Stickle, C., Matthews, K., Hall, J., & Douglas Steele, J. (2013) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2012.06.003 · Cited by 82 · cited 1×
“The analysis focused on reward delivery and found blunted ventral striatal activation in patients to be correlated with negative symptoms (97).”
BackgroundImagining the future: degraded representations of future rewards and events in schizophrenia
Heerey, E. A., Matveeva, T. M., & Gold, J. M. (2011) · Journal of Abnormal Psychology · doi:10.1037/a0021810 · Cited by 78
Bernacer, J., Corlett, P. R., Ramachandra, P., McFarlane, B., Turner, D. C., Clark, L., Robbins, T. W., Fletcher, P. C., & Murray, G. K. (2013) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2013.12070978 · Cited by 77
General functioning predicts reward and punishment learning in schizophrenia
Somlai, Z., Moustafa, A. A., Kéri, S., Myers, C. E., & Gluck, M. A. (2011) · Schizophrenia Research · doi:10.1016/j.schres.2010.07.028 · Cited by 45 · cited 1×
“A deficit in reward-based learning, but not in the avoidance of punishment, which was associated with negative symptoms, was also found in two other independent studies, both in patients treated with antipsychotic medication (65, 66).”
ResultAberrant effective connectivity in schizophrenia patients during appetitive conditioning
Diaconescu, A. O., Jensen, J., Wang, H., Willeit, M., Menon, M., Kapur, S., & McIntosh, A. R. (2011) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2010.00239 · Cited by 43
Cited by
Millan, M. J., Fone, K., Steckler, T., & Horan, W. P. (2014) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2014.03.008 · Cited by 347
Ventral Striatal Activation During Reward Processing in Psychosis: A Neurofunctional Meta-Analysis
Radua, J., Schmidt, A., Borgwardt, S., Heinz, A., Schlagenhauf, F., McGuire, P., & Fusar-Poli, P. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2015.2196 · Cited by 342
Self-citationAn integrative perspective on the role of dopamine in schizophrenia
Maia, T. V., & Frank, M. J. (2017) · Biological Psychiatry · doi:10.1016/j.biopsych.2016.05.021 · Cited by 293
Computational psychiatry: towards a mathematically informed understanding of mental illness
Adams, R. A., Huys, Q. J. M., & Roiser, J. P. (2015) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp-2015-310737 · Cited by 285
Linking RDoC and HiTOP: A new interface for advancing psychiatric nosology and neuroscience
Michelini, G., Palumbo, I. M., DeYoung, C. G., Latzman, R. D., & Kotov, R. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102025 · Cited by 273
Metadata
- Authors
- Deserno, L., Boehme, R., Heinz, A., & Schlagenhauf, F.
- Venue
- Frontiers in Psychiatry
- Publisher
- Frontiers Media SA
- Published
- 2013
- Volume
- 4
- Pages
- 172-172
- ISSN
- 1664-0640 (ISSN-L)
- OpenAlex
- W2141640976
- Cited by
- 88
- References
- 57
- FWCI
- 6.80
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.