Individual differences and the neural representations of reward expectation and reward prediction error
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Reward expectation and reward prediction errors are thought to be critical for dynamic adjustments in decision-making and reward-seeking behavior, but little is known about their representation in the brain during uncertainty and risk-taking. Furthermore, little is known about what role individual differences might play in such reinforcement processes. In this study, it is shown behavioral and neural responses during a decision-making task can be characterized by a computational reinforcement learning model and that individual differences in learning parameters in the model are critical for elucidating these processes. In the fMRI experiment, subjects chose between high- and low-risk rewards. A computational reinforcement learning model computed expected values and prediction errors that each subject might experience on each trial. These outputs predicted subjects' trial-to-trial choice strategies and neural activity in several limbic and prefrontal regions during the task. Individual differences in estimated reinforcement learning parameters proved critical for characterizing these processes, because models that incorporated individual learning parameters explained significantly more variance in the fMRI data than did a model using fixed learning parameters. These findings suggest that the brain engages a reinforcement learning process during risk-taking and that individual differences play a crucial role in modeling this process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Holroyd, C. B., & Coles, M. G. H. (2002) · Psychological Review · doi:10.1037/0033-295x.109.4.679 · Cited by 4,027
Emotion, Decision Making and the Orbitofrontal Cortex
Bechara, A. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.295 · Cited by 3,589
Deciding advantageously before knowing the advantageous strategy
Bechara, A., Damasio, H., Tranel, D., & Damasio, A. R. (1997) · Science · doi:10.1126/science.275.5304.1293 · Cited by 3,547
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 4×
“, 2004) or very simple decision-making situations in which one response is optimal and another suboptimal (O’Doherty et al., 2004).”
BackgroundMethodResultInfluentialA framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 1×
“However, this finding seems consistent with the presumed role of the caudate as the ‘actor’ in actor–critic models of reinforcement learning (Montague et al., 1996; O’Doherty et al., 2004).”
ResultTemporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · cited 9×
“In particular, phasic increases in activity are observed when reinforcements are better than expected (a positive prediction error), and phasic decreases in activity are observed when reinforcements are worse than expected or not given (a negative prediction error) (Schultz et al., 1997; Waelti et al., 2001; Daw et al…”
BackgroundMethodResultInfluentialRelative reward preference in primate orbitofrontal cortex
Tremblay, L., & Schultz, W. (1999) · Nature · doi:10.1038/19525 · Cited by 1,314 · cited 3×
“For example, orbitofrontal and amygdala neurons show increased firing rates to preferred rewards compared to less preferred rewards (Everitt et al., 1991; Tremblay and Schultz, 1999; Hikosaka and Watanabe, 2000; Baxter and Murray, 2002; Gilbert et al., 2003).”
BackgroundMidbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · cited 3×
“The location of this activation is also consistent with coordinates reported in previous fMRI studies of reinforcement learning processes (Seymour et al., 2004; O’Doherty et al., 2006) and with direct recordings of single unit activity in monkeys (Schultz, 1998; Bayer and Glimcher, 2005).”
BackgroundMethodResultBaxter, M. G., & Murray, E. A. (2002) · Nature Reviews Neuroscience · doi:10.1038/nrn875 · Cited by 1,200
Matching Behavior and the Representation of Value in the Parietal Cortex
Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004) · Science · doi:10.1126/science.1094765 · Cited by 1,030
Dopamine responses comply with basic assumptions of formal learning theory
Waelti, P., Dickinson, A., & Schultz, W. (2001) · Nature · doi:10.1038/35083500 · Cited by 1,018 · cited 3×
“These activations confirm those reported in previous studies (Schultz et al., 1997; Waelti et al., 2001; Daw et al., 2002; Holroyd and Coles, 2002; O’Doherty et al., 2003; Schultz, 2004; Seymour et al., 2004; Rodriguez et al., 2005; Abler et al., 2006).”
BackgroundResultComputational roles for dopamine in behavioural control
Montague, P. R., Hyman, S. E., & Cohen, J. D. (2004) · Nature · doi:10.1038/nature03015 · Cited by 966
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939
Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum
McClure, S. M., Berns, G. S., & Montague, P. R. (2003) · Neuron · doi:10.1016/s0896-6273(03)00154-5 · Cited by 917 · cited 1×
“Such relationships were observed primarily in the ventral striatum and orbitofrontal cortex, consistent with previous reports that these regions are sensitive to prediction errors (McClure et al., 2003; O’Doherty et al., 2003; Abler et al., 2006; Haruno and Kawato, 2006; Jensen et al., 2006).”
ResultOpponent interactions between serotonin and dopamine
Daw, N. D., Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00052-7 · Cited by 844
Kringelbach, M. L. (2003) · Cerebral Cortex · doi:10.1093/cercor/13.10.1064 · Cited by 824
Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops
Tanaka, S. C., Doya, K., Okada, G., Ueda, K., Okamoto, Y., & Yamawaki, S. (2004) · Nature Neuroscience · doi:10.1038/nn1279 · Cited by 779 · cited 1×
“This method has been previously used to study the putative neural correlates of prediction errors (O’Doherty et al., 2003; O’Doherty et al., 2004; Seymour et al., 2004; Tanaka et al., 2004; Glascher and Buchel, 2005; Haruno and Kawato, 2006).”
MethodTemporal difference models describe higher-order learning in humans
Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S. (2004) · Nature · doi:10.1038/nature02581 · Cited by 605 · cited 10×
“In particular, phasic increases in activity are observed when reinforcements are better than expected (a positive prediction error), and phasic decreases in activity are observed when reinforcements are worse than expected or not given (a negative prediction error) (Schultz et al., 1997; Waelti et al., 2001; Daw et al…”
BackgroundMethodResultInfluentialSchultz, W. (2004) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2004.03.017 · Cited by 517
Everitt, B. J., Morris, K. A., O'Brien, A., & Robbins, T. W. (1991) · Neuroscience · doi:10.1016/0306-4522(91)90145-e · Cited by 507 · cited 1×
“For example, orbitofrontal and amygdala neurons show increased firing rates to preferred rewards compared to less preferred rewards (Everitt et al., 1991; Tremblay and Schultz, 1999; Hikosaka and Watanabe, 2000; Baxter and Murray, 2002; Gilbert et al., 2003).”
BackgroundD2 dopamine receptor gene in psychiatric and neurologic disorders and its phenotypes
Noble, E. P. (2003) · American Journal of Medical Genetics Part B: Neuropsychiatric Genetics · doi:10.1002/ajmg.b.10005 · Cited by 501 · cited 1×
“…preferences have been linked to a number of neurobiological and psychosocial factors such as the concentration of dopamine D2 receptors in the limbic system (Noble, 1998, 2003), socioeconomic status (Diala et al., 2004), or personality (Craig, 1979; Zuckerman and Kuhlman, 2000; Petry, 2001).”
BackgroundActor–critic models of the basal ganglia: new anatomical and computational perspectives
Joel, D., Niv, Y., & Ruppin, E. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00047-3 · Cited by 500
Substance abuse, pathological gambling, and impulsiveness
Petry, N. M. (2001) · Drug and Alcohol Dependence · doi:10.1016/s0376-8716(00)00188-5 · Cited by 462 · cited 2×
“, 2004), or personality (Craig, 1979; Zuckerman and Kuhlman, 2000; Petry, 2001).”
BackgroundHaruno, M., & Kawato, M. (2006) · Journal of Neurophysiology · doi:10.1152/jn.00382.2005 · Cited by 433
Prediction error as a linear function of reward probability is coded in human nucleus accumbens
Abler, B., Walter, H., Erk, S., Kammerer, H., & Spitzer, M. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.01.001 · Cited by 376 · cited 4×
“Such relationships were observed primarily in the ventral striatum and orbitofrontal cortex, consistent with previous reports that these regions are sensitive to prediction errors (McClure et al., 2003; O’Doherty et al., 2003; Abler et al., 2006; Haruno and Kawato, 2006; Jensen et al., 2006).”
BackgroundResultInfluentialO'Doherty, J. P., Buchanan, T. W., Seymour, B., & Dolan, R. J. (2006) · Neuron · doi:10.1016/j.neuron.2005.11.014 · Cited by 322 · cited 1×
“The location of this activation is also consistent with coordinates reported in previous fMRI studies of reinforcement learning processes (Seymour et al., 2004; O’Doherty et al., 2006) and with direct recordings of single unit activity in monkeys (Schultz, 1998; Bayer and Glimcher, 2005).”
ResultWörgötter, F., & Porr, B. (2005) · Neural Computation · doi:10.1162/0899766053011555 · Cited by 203
Ventral–striatal/nucleus–accumbens sensitivity to prediction errors during classification learning
Rodriguez, P. F., Aron, A. R., & Poldrack, R. A. (2006) · Human Brain Mapping · doi:10.1002/hbm.20186 · Cited by 122
Cited by
Liu, X., Hairston, J., Schrier, M., & Fan, J. (2011) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2010.12.012 · Cited by 916
Informatic parcellation of the network involved in the computation of subjective value
Clithero, J. A., & Rangel, A. (2014) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nst106 · Cited by 706
Reinforcement Learning Signal Predicts Social Conformity
Klucharev, V., Hytönen, K., Rijpkema, M., Smidts, A., & Fernández, G. (2009) · Neuron · doi:10.1016/j.neuron.2008.11.027 · Cited by 594
Frontal theta links prediction errors to behavioral adaptation in reinforcement learning
Cavanagh, J. F., Frank, M. J., Klein, T. J., & Allen, J. J. B. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.080 · Cited by 469
Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies
Garrison, J., Erdeniz, B., & Done, J. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.03.023 · Cited by 438
Metadata
- Authors
- Cohen, M. X.
- Venue
- Social Cognitive and Affective Neuroscience
- Publisher
- Oxford University Press (OUP)
- Published
- 2006-09-18
- Volume
- 2
- Issue
- 1
- Pages
- 20-30
- ISSN
- 1749-5016 (ISSN-L)
- 1749-5024
- OpenAlex
- W2125951016
- Cited by
- 78
- References
- 28
- FWCI
- 2.15
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.