Individual differences in reinforcement learning: Behavioral, electrophysiological, and neuroimaging correlates
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
During reinforcement learning, phasic modulations of activity in midbrain dopamine neurons are conveyed to the dorsal anterior cingulate cortex (dACC) and basal ganglia and serve to guide adaptive responding. While the animal literature supports a role for the dACC in integrating reward history over time, most human electrophysiological studies of dACC function have focused on responses to single positive and negative outcomes. The present electrophysiological study investigated the role of the dACC in probabilistic reward learning in healthy subjects using a task that required integration of reinforcement history over time. We recorded the feedback-related negativity (FRN) to reward feedback in subjects who developed a response bias toward a more frequently rewarded (“rich”) stimulus (“learners”) versus subjects who did not (“non-learners”). Compared to non-learners, learners showed more positive (i.e., smaller) FRNs and greater dACC activation upon receiving reward for correct identification of the rich stimulus. In addition, dACC activation and a bias to select the rich stimulus were positively correlated. The same participants also completed a monetary incentive delay (MID) task administered during functional magnetic resonance imaging. Compared to non-learners, learners displayed stronger basal ganglia responses to reward in the MID task. These findings raise the possibility that learners in the probabilistic reinforcement task were characterized by stronger dACC and basal ganglia responses to rewarding outcomes. Furthermore, these results highlight the importance of the dACC to probabilistic reward learning in humans.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fischl, B., Salat, D. H., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., van der Kouwe, A., Killiany, R., Kennedy, D., Klaveness, S., et al. (2002) · Neuron · doi:10.1016/s0896-6273(02)00569-x · Cited by 8,610
(2001) · doi:10.1017/cbo9780511546396 · Cited by 4,482
Holroyd, C. B., & Coles, M. G. H. (2002) · Psychological Review · doi:10.1037/0033-295x.109.4.679 · Cited by 4,027
Getting formal with dopamine and reward
Schultz, W. (2002) · Neuron · doi:10.1016/s0896-6273(02)00967-4 · Cited by 2,574
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 1×
“Potentiated recruitment of BG regions in subjects developing a response bias toward the rich stimulus is consistent with the hypothesis that BG regions are critically implicated in feedback-based learning (Delgado, 2007; O'Doherty et al., 2004; Seymour et al., 2007).”
ResultThe Medial Frontal Cortex and the Rapid Processing of Monetary Gains and Losses
Gehring, W. J., & Willoughby, A. R. (2002) · Science · doi:10.1126/science.1066893 · Cited by 1,884 · cited 2×
“…ms following feedback and has been localized to various regions of the cingulate cortex, including the dorsal ACC (dACC; Miltner et al., 1997; Gehring and Willoughby, 2002), medial prefrontal cortex (Muller et al., 2005; Nieuwenhuis et al., 2005; Van Veen et al., 2004), and the posterior…”
MethodInfluentialPain and emotion interactions in subregions of the cingulate gyrus
Vogt, B. A. (2005) · Nature Reviews Neuroscience · doi:10.1038/nrn1704 · Cited by 1,810 · cited 2×
“The human homologue for rCMA is the anterior motor cingulate cortex (BA 24b; Vogt, 2005), thus these findings suggest that the human dACC — specifically, BA 24 — might play an important role in updating response selection based on reward feedback.”
BackgroundMiltner, W. H. R., Braun, C. H., & Coles, M. G. H. (1997) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1997.9.6.788 · Cited by 1,530
Watson, D., Weber, K., Assenheimer, J. S., Clark, L. A., Strauss, M. E., & McCormick, R. A. (1995) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.104.1.3 · Cited by 1,423
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · cited 3×
“Replicating prior findings from an independent sample (Pizzagalli et al., 2005), non-learners reported higher anhedonic symptoms at the EEG session, as assessed by an anhedonic BDI-II subscore [loss of pleasure (item #4), loss of interest (item #12), loss of energy (item #15), and loss of interest in sex (item #21)] (…”
MethodInfluentialDorsal anterior cingulate cortex: A role in reward-based decision making
Bush, G., Vogt, B. A., Holmes, J., Dale, A. M., Greve, D., Jenike, M. A., & Rosen, B. R. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.012470999 · Cited by 1,096
Schultz, W. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.03.007 · Cited by 1,028
Computational roles for dopamine in behavioural control
Montague, P. R., Hyman, S. E., & Cohen, J. D. (2004) · Nature · doi:10.1038/nature03015 · Cited by 966
Optimal decision making and the anterior cingulate cortex
Kennerley, S. W., Walton, M. E., Behrens, T. E. J., Buckley, M. J., & Rushworth, M. F. S. (2006) · Nature Neuroscience · doi:10.1038/nn1724 · Cited by 948 · cited 3×
“In non-human primates, ACC lesions impair the ability to integrate reinforcement history over time and choose advantageous responses (Kennerley et al., 2006).”
BackgroundResultOptimized EPI for fMRI studies of the orbitofrontal cortex
Deichmann, R., Gottfried, J. A., Hutton, C., & Turner, R. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00073-9 · Cited by 821
Reduced hedonic capacity in major depressive disorder: evidence from a probabilistic reward task
Pizzagalli, D. A., Iosifescu, D., Hallett, L. A., Ratner, K. G., & Fava, M. (2008) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2008.03.001 · Cited by 780
Self-citationToward an objective characterization of an anhedonic phenotype: A signal-detection approach
Pizzagalli, D. A., Jahn, A. L., & O’Shea, J. P. (2005) · Biological Psychiatry · doi:10.1016/j.biopsych.2004.11.026 · Cited by 738
Self-citationFunctional magnetic resonance imaging of reward prediction
Knutson, B., & Cooper, J. C. (2005) · Current Opinion in Neurology · doi:10.1097/01.wco.0000173463.24758.f6 · Cited by 736 · cited 2×
“…on data from four BG ROIs (nucleus accumbens, caudate, putamen, and globus pallidus), consistent with prior fMRI studies implicating BG regions in reward processing and reinforcement learning (e.g., Delgado et al., 2005; Dillon et al., 2008; Knutson and Cooper, 2005; Schonberg et al., 2007).”
MethodResultReward‐Related Responses in the Human Striatum
DELGADO, M. R. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.002 · Cited by 736
Performance monitoring by the anterior cingulate cortex during saccade countermanding
Ito, S., Stuphorn, V., Brown, J. W., & Schall, J. D. (2003) · Science · doi:10.1126/science.1087847 · Cited by 581
Optimization of experimental design in fMRI: a general framework using a genetic algorithm
Wager, T. D., & Nichols, T. E. (2003) · NeuroImage · doi:10.1016/s1053-8119(02)00046-0 · Cited by 458 · cited 2×
“The task design and timing were optimized using a genetic algorithm that maximized the statistical orthogonality of the conditions under investigation (Wager and Nichols, 2003).”
MethodInfluentialSchönberg, T., Daw, N. D., Joel, D., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.2496-07.2007 · Cited by 390
Choice selection and reward anticipation: an fMRI study
Ernst, M., Nelson, E. E., McClure, E. B., Monk, C. S., Munson, S., Eshel, N., Zarahn, E., Leibenluft, E., Zametkin, A., Towbin, K., et al. (2004) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2004.05.011 · Cited by 377
Acute stress reduces reward responsiveness: Implications for depression
Bogdan, R., & Pizzagalli, D. A. (2006) · Biological Psychiatry · doi:10.1016/j.biopsych.2006.03.037 · Cited by 364 · cited 2×
“Because reward learning has been found to negatively correlate with anhedonic symptoms (e.g., Pizzagalli et al., 2005; Bogdan and Pizzagalli, 2006), fluctuations in mood across the two sessions might diminish the test-retest estimates.”
BackgroundSelf-citationFunctional organization of the medial frontal cortex
Rushworth, M. F., Buckley, M. J., Behrens, T. E., Walton, M. E., & Bannerman, D. M. (2007) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2007.03.001 · Cited by 353
Dissociation of dopamine release in the nucleus accumbens from intracranial self-stimulation
Garris, P. A., Kilpatrick, M., Bunin, M. A., Michael, D., Walker, Q. D., & Wightman, R. M. (1999) · Nature · doi:10.1038/18019 · Cited by 327 · cited 1×
“…are consistent with two of the model’s main hypotheses: (1) that phasic DA bursts act as signals that reinforce rewarding behaviors (Bayer and Glimacher, 2005; Garris et al., 1999), and (2) that these signals “teach” the dACC to select among various response options (Holroyd and Coles, 2002).”
ResultPizzagalli, D. A., Evins, A. E., Schetter, E. C., Frank, M. J., Pajtas, P. E., Santesso, D. L., & Culhane, M. (2008) · Psychopharmacology · doi:10.1007/s00213-007-0957-y · Cited by 219
Self-citationAkitsuki, Y., Sugiura, M., Watanabe, J., Yamashita, K., Sassa, Y., Awata, S., Matsuoka, H., Maeda, Y., Matsue, Y., Fukuda, H., et al. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00250-7 · Cited by 70
Cited by
Pizzagalli, D. A., Holmes, A. J., Dillon, D. G., Goetz, E. L., Birk, J. L., Bogdan, R., Dougherty, D. D., Iosifescu, D. V., Rauch, S. L., & Fava, M. (2009) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2008.08081201 · Cited by 1,232
Self-citationWacker, J., Dillon, D. G., & Pizzagalli, D. A. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.01.058 · Cited by 405
“A previous publication on this sample has focused on event-related potential data collected during a reinforcement task (Santesso et al., 2008), and a report on links between candidate genes and the fMRI data is in Neural Correlates of Anhedonia preparation (Dillon, Bogdan, Fagerness, Holmes,…”
InfluentialSelf-citationOliveira, F. T. P., McDonald, J. J., & Goodman, D. (2007) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2007.19.12.1994 · Cited by 387
Neural Processing of Reward and Loss in Girls at Risk for Major Depression
Gotlib, I. H., Hamilton, J. P., Cooney, R. E., Singh, M. K., Henry, M. L., & Joormann, J. (2010) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2010.13 · Cited by 333
Trivedi, M. H., McGrath, P. J., Fava, M., Parsey, R. V., Kurian, B. T., Phillips, M. L., Oquendo, M. A., Bruder, G., Pizzagalli, D., Toups, M., et al. (2016) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2016.03.001 · Cited by 286
Self-citation
Metadata
- Authors
- Santesso, D. L., Dillon, D. G., Birk, J. L., Holmes, A. J., Goetz, E., Bogdan, R., & Pizzagalli, D. A.
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2008-08
- Volume
- 42
- Issue
- 2
- Pages
- 807-816
- ISSN
- 1053-8119 (ISSN-L)
- OpenAlex
- W2105239131
- Cited by
- 135
- References
- 28
- FWCI
- 5.03
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.