Testing the Reward Prediction Error Hypothesis with an Axiomatic Model
Open accessAbstract
Neuroimaging studies typically identify neural activity correlated with the predictions of highly parameterized models, like the many reward prediction error (RPE) models used to study reinforcement learning. Identified brain areas might encode RPEs or, alternatively, only have activity correlated with RPE model predictions. Here, we use an alternate axiomatic approach rooted in economic theory to formally test the entire class of RPE models on neural data. We show that measurements of human neural activity from the striatum, medial prefrontal cortex, amygdala, and posterior cingulate cortex satisfy necessary and sufficient conditions for the entire class of RPE models. However, activity measured from the anterior insula falsifies the axiomatic model, and therefore no RPE model can account for measured activity. Further analysis suggests the anterior insula might instead encode something related to the salience of an outcome. As cognitive neuroscience matures and models proliferate, formal approaches of this kind that assess entire model classes rather than specific model exemplars may take on increased significance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When participants attributed outcomes to the neutral committee (their own performance), MBD activity was positively correlated with outcome.
“When participants attributed outcomes to the neutral committee (their own performance), MBD activity was, as has been previously reported,50”
From Two prefrontal-subcortical circuits estimate controllability and reflect its impact on learning (2026) · cites this paper as [51]
The Results (Assessing the robustness of axiomatic RPE model tests; page unavailable) report no evidence of a reward-prediction-error signal in the midbrain regions tested. The paper does not address outcomes attributed to a neutral committee or participants’ own performance, so it cannot verify the claimed correlation in that condition (searched: neutral committee, own performance, midbrain activity). (LLM, conf=0.76)
Experimental — accuracy not yet measuredThe ventral striatum region of interest was derived from the nucleus accumbens ROI reported by Rutledge et al. (2010).
“The VS ROI was based on Rutledge et al. (2010)9 nucleus accumbens ROI and was comprised of three 5 mm spheres for each hemisphere ([8 13 –3; 12 13 –8; 9 13 –7; –8 13 –3; –12 13 –8; –9 13 –7 ]).”
From Representation of aversive prediction errors in the human periaqueductal gray (2014) · cites this paper as Rutledge et al. (2010)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
The Logic of Scientific Discovery.
Hutchison, T. W., & Popper, K. R. (1959) · Economica · doi:10.2307/2550489 · Cited by 7,628 · see citing works
Dissociable intrinsic connectivity networks for salience processing and executive control
Seeley, W. W., Menon, V., Schatzberg, A. F., Keller, J., Glover, G. H., Kenna, H., Reiss, A. L., & Greicius, M. D. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.5587-06.2007 · Cited by 7,426 · see citing works
What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?
Berridge, K. C., & Robinson, T. E. (1998) · Brain Research Reviews · doi:10.1016/s0165-0173(98)00019-8 · Cited by 4,127 · see citing works
<i>The Logic of Scientific Discovery</i>
Popper, K. R., & Weiss, G. (1959) · Physics Today · doi:10.1063/1.3060577 · Cited by 2,717 · see citing works
Wald, A. (1943) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1943-0012401-3 · Cited by 2,522 · see citing works
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · see citing works
Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans
Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006) · Nature · doi:10.1038/nature05051 · Cited by 1,584 · see citing works
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · see citing works
Two types of dopamine neuron distinctly convey positive and negative motivational signals
Matsumoto, M., & Hikosaka, O. (2009) · Nature · doi:10.1038/nature08028 · Cited by 1,378 · see citing works
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · see citing works
Self-citationLateral habenula as a source of negative reward signals in dopamine neurons
Matsumoto, M., & Hikosaka, O. (2007) · Nature · doi:10.1038/nature05860 · Cited by 1,292 · see citing works
Subsecond dopamine release promotes cocaine seeking
Phillips, P. E. M., Stuber, G. D., Heien, M. L. A. V., Wightman, R. M., & Carelli, R. M. (2003) · Nature · doi:10.1038/nature01476 · Cited by 1,113 · see citing works
Dopamine neurons report an error in the temporal prediction of reward during learning
Hollerman, J. R., & Schultz, W. (1998) · Nature Neuroscience · doi:10.1038/1124 · Cited by 1,103 · see citing works
Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events
Horvitz, J. C. (2000) · Neuroscience · doi:10.1016/s0306-4522(00)00019-1 · Cited by 967 · see citing works
Associative learning of social value
Behrens, T. E. J., Hunt, L. T., Woolrich, M. W., & Rushworth, M. F. S. (2008) · Nature · doi:10.1038/nature07538 · Cited by 925 · see citing works
Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum
McClure, S. M., Berns, G. S., & Montague, P. R. (2003) · Neuron · doi:10.1016/s0896-6273(03)00154-5 · Cited by 917 · see citing works
Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli
Brischoux, F., Chakraborty, S., Brierley, D. I., & Ungless, M. A. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0811507106 · Cited by 801 · see citing works
Hare, T. A., O'Doherty, J., Camerer, C. F., Schultz, W., & Rangel, A. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1309-08.2008 · Cited by 792 · see citing works
Human insula activation reflects risk prediction errors as well as risk
Preuschoff, K., Quartz, S. R., & Bossaerts, P. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.4286-07.2008 · Cited by 784 · see citing works
Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens
Day, J. J., Roitman, M. F., Wightman, R. M., & Carelli, R. M. (2007) · Nature Neuroscience · doi:10.1038/nn1923 · Cited by 710 · see citing works
Neural differentiation of expected reward and risk in human subcortical structures
Preuschoff, K., Bossaerts, P., & Quartz, S. R. (2006) · Neuron · doi:10.1016/j.neuron.2006.06.024 · Cited by 692 · see citing works
BOLD responses reflecting dopaminergic signals in the human ventral tegmental area
D'Ardenne, K., McClure, S. M., Nystrom, L. E., & Cohen, J. D. (2008) · Science · doi:10.1126/science.1150605 · Cited by 661 · see citing works
Nonlinear event‐related responses in fMRI
Friston, K. J., Josephs, O., Rees, G. E., & Turner, R. (1998) · Magnetic Resonance in Medicine · doi:10.1002/mrm.1910390109 · Cited by 629 · see citing works
Temporal difference models describe higher-order learning in humans
Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S. (2004) · Nature · doi:10.1038/nature02581 · Cited by 605 · see citing works
Yacubian, J., Gläscher, J., Schroeder, K., Sommer, T., Braus, D. F., & Büchel, C. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.2915-06.2006 · Cited by 537 · see citing works
Is the short-latency dopamine response too short to signal reward error?
Redgrave, P., Prescott, T. J., & Gurney, K. (1999) · Trends in Neurosciences · doi:10.1016/s0166-2236(98)01373-3 · Cited by 506 · see citing works
Ullsperger, M., & von Cramon, D. Y. (2003) · Journal of Neuroscience · doi:10.1523/jneurosci.23-10-04308.2003 · Cited by 484 · see citing works
Huettel, S. A., Song, A. W., & McCarthy, G. (2005) · Journal of Neuroscience · doi:10.1523/jneurosci.5070-04.2005 · Cited by 457 · see citing works
Medial prefrontal cell activity signaling prediction errors of action values
Matsumoto, M., Matsumoto, K., Abe, H., & Tanaka, K. (2007) · Nature Neuroscience · doi:10.1038/nn1890 · Cited by 440 · see citing works
Human Striatal Responses to Monetary Reward Depend On Saliency
Zink, C. F., Pagnoni, G., Martin-Skurski, M. E., Chappelow, J., & Berns, G. S. (2004) · Neuron · doi:10.1016/s0896-6273(04)00183-7 · Cited by 432 · see citing works
Wald, A. (1943) · Transactions of the American Mathematical Society · doi:10.2307/1990256 · Cited by 386 · see citing works
Human Cerebral Cortex: Localization, Parcellation, and Morphometry with Magnetic Resonance Imaging
Rademacher, J., Galaburda, A. M., Kennedy, D. N., Filipek, P. A., & Caviness, V. S. (1992) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1992.4.4.352 · Cited by 376 · see citing works
Prediction error as a linear function of reward probability is coded in human nucleus accumbens
Abler, B., Walter, H., Erk, S., Kammerer, H., & Spitzer, M. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.01.001 · Cited by 376 · see citing works
Nonlinear Aspects of the BOLD Response in Functional MRI
Vazquez, A. L., & Noll, D. C. (1998) · NeuroImage · doi:10.1006/nimg.1997.0316 · Cited by 364 · see citing works
Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala
Belova, M. A., Paton, J. J., Morrison, S. E., & Salzman, C. D. (2007) · Neuron · doi:10.1016/j.neuron.2007.08.004 · Cited by 349 · see citing works
A Neural Representation of Categorization Uncertainty in the Human Brain
Grinband, J., Hirsch, J., & Ferrera, V. P. (2006) · Neuron · doi:10.1016/j.neuron.2006.01.032 · Cited by 330 · see citing works
Human striatal response to salient nonrewarding stimuli
Zink, C. F., Pagnoni, G., Martin, M. E., Dhamala, M., & Berns, G. S. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-22-08092.2003 · Cited by 312 · see citing works
Caviness, V. S., Meyer, J. W., Makris, N., & Kennedy, D. N. (1996) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1996.8.6.566 · Cited by 298 · see citing works
Mechanisms underlying dopamine-mediated reward bias in compulsive behaviors
Voon, V., Pessiglione, M., Brezing, C., Gallea, C., Fernandez, H. H., Dolan, R. J., & Hallett, M. (2010) · Neuron · doi:10.1016/j.neuron.2009.12.027 · Cited by 280 · see citing works
Striatal activity underlies novelty-based choice in humans
Wittmann, B. C., Daw, N. D., Seymour, B., & Dolan, R. J. (2008) · Neuron · doi:10.1016/j.neuron.2008.04.027 · Cited by 277 · see citing works
Dopamine Neurons Can Represent Context-Dependent Prediction Error
Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., & Hikosaka, O. (2004) · Neuron · doi:10.1016/s0896-6273(03)00869-9 · Cited by 276 · see citing works
Joshua, M., Adler, A., Mitelman, R., Vaadia, E., & Bergman, H. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.3839-08.2008 · Cited by 273 · see citing works
Human substantia nigra neurons encode unexpected financial rewards
Zaghloul, K. A., Blanco, J. A., Weidemann, C. T., McGill, K., Jaggi, J. L., Baltuch, G. H., & Kahana, M. J. (2009) · Science · doi:10.1126/science.1167342 · Cited by 264 · see citing works
Rutledge, R. B., Lazzaro, S. C., Lau, B., Myers, C. E., Gluck, M. A., & Glimcher, P. W. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.3524-09.2009 · Cited by 255 · see citing works
Self-citationSaccade Reward Signals in Posterior Cingulate Cortex
McCoy, A. N., Crowley, J. C., Haghighian, G., Dean, H. L., & Platt, M. L. (2003) · Neuron · doi:10.1016/s0896-6273(03)00719-0 · Cited by 193 · see citing works
Separate brain regions code for salience vs. valence during reward prediction in humans
Jensen, J., Smith, A. J., Willeit, M., Crawley, A. P., Mikulis, D. J., Vitcu, I., & Kapur, S. (2006) · Human Brain Mapping · doi:10.1002/hbm.20274 · Cited by 178 · see citing works
de Bruijn, E. R. A., de Lange, F. P., von Cramon, D. Y., & Ullsperger, M. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.1751-09.2009 · Cited by 129 · see citing works
Dopamine, Reward Prediction Error, and Economics*
Caplin, A., & Dean, M. R. W. (2008) · The Quarterly Journal of Economics · doi:10.1162/qjec.2008.123.2.663 · Cited by 106 · see citing works
Self-citationAxiomatic methods, dopamine and reward prediction error
Caplin, A., & Dean, M. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.07.007 · Cited by 88 · see citing works
Self-citationMeasuring Beliefs and Rewards: A Neuroeconomic Approach*
Caplin, A., Dean, M. R. W., Glimcher, P. W., & Rutledge, R. B. (2010) · The Quarterly Journal of Economics · doi:10.1162/qjec.2010.125.3.923 · Cited by 63 · see citing works
Self-citationPolicy Adjustment in a Dynamic Economic Game
Li, J., McClure, S. M., King‐Casas, B., & Montague, P. R. (2006) · PLoS ONE · doi:10.1371/journal.pone.0000103 · Cited by 63 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Sutton RS Barto AG Gabriel M Moore J Time-derivative models of Pavlovian reinforcement. Learning and computational neuroscience: foundations of adaptive networks 1990 497 537 MIT Cambridge, MA
Cited by
Sorted by citations
Sescousse, G., Caldú, X., Segura, B., & Dreher, J.-C. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.02.002 · Cited by 782 · see citing works
Significance of the insula for the evolution of human awareness of feelings from the body
(Bud) Craig, A. D. (2011) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2011.05990.x · Cited by 646 · see citing works
Neurophysiology of performance monitoring and adaptive behavior
Ullsperger, M., Danielmeier, C., & Jocham, G. (2014) · Physiological Reviews · doi:10.1152/physrev.00041.2012 · Cited by 634 · see citing works
Inflammation Effects on Motivation and Motor Activity: Role of Dopamine
Felger, J. C., & Treadway, M. T. (2016) · Neuropsychopharmacology · doi:10.1038/npp.2016.143 · Cited by 453 · see citing works
A computational and neural model of momentary subjective well-being
Rutledge, R. B., Skandali, N., Dayan, P., & Dolan, R. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1407535111 · Cited by 444 · see citing works
Self-citationPrediction error in reinforcement learning: A meta-analysis of neuroimaging studies
Garrison, J. R., Erdeniz, B., & Done, J. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.03.023 · Cited by 430 · see citing works
Hauser, T. U., Iannaccone, R., Stämpfli, P., Drechsler, R., Brandeis, D., Walitza, S., & Brem, S. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.08.028 · Cited by 423 · see citing works
Rostrolateral Prefrontal Cortex and Individual Differences in Uncertainty-Driven Exploration
Badre, D., Doll, B. B., Long, N. M., & Frank, M. J. (2012) · Neuron · doi:10.1016/j.neuron.2011.12.025 · Cited by 309 · see citing works
Hart, A. S., Rutledge, R. B., Glimcher, P. W., & Phillips, P. E. M. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.2489-13.2014 · Cited by 294 · see citing works
Self-citationFrontal Theta Reflects Uncertainty and Unexpectedness during Exploration and Exploitation
Cavanagh, J. F., y Figueroa, C. M. L. F., Cohen, M. X., & Frank, M. J. (2011) · Cerebral Cortex · doi:10.1093/cercor/bhr332 · Cited by 294 · see citing works
Miller, K. J., Shenhav, A., & Ludvig, E. A. (2019) · Psychological Review · doi:10.1037/rev0000120 · Cited by 280 · see citing works
Anticipatory reward processing in addicted populations: a focus on the monetary incentive delay task
Balodis, I. M., & Potenza, M. N. (2015) · Biological Psychiatry · doi:10.1016/j.biopsych.2014.08.020 · Cited by 267 · see citing works
Valuation of knowledge and ignorance in mesolimbic reward circuitry
Charpentier, C. J., Bromberg-Martin, E. S., & Sharot, T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1800547115 · Cited by 237 · see citing works
Representation of aversive prediction errors in the human periaqueductal gray
Roy, M., Shohamy, D., Daw, N., Jepma, M., Wimmer, G. E., & Wager, T. D. (2014) · Nature Neuroscience · doi:10.1038/nn.3832 · Cited by 246 · see citing works
Critical Roles for Anterior Insula and Dorsal Striatum in Punishment-Based Avoidance Learning
Palminteri, S., Justo, D., Jauffret, C., Pavlicek, B., Dauta, A., Delmaire, C., Czernecki, V., Karachi, C., Capelle, L., Dürr, A., et al. (2012) · Neuron · doi:10.1016/j.neuron.2012.10.017 · Cited by 237 · see citing works
Chase, H. W., Kumar, P., Eickhoff, S. B., & Dombrovski, A. Y. (2015) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-015-0338-7 · Cited by 225 · see citing works
Haber, S. N., & Behrens, T. E. J. (2014) · Neuron · doi:10.1016/j.neuron.2014.08.031 · Cited by 230 · see citing works
Hauser, T. U., Iannaccone, R., Walitza, S., Brandeis, D., & Brem, S. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.09.018 · Cited by 203 · see citing works
Erdeniz, B., Rohe, T., Done, J., & Seidler, R. D. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00116 · Cited by 215 · see citing works
Association of neural and emotional impacts of reward prediction errors with major depression
Rutledge, R. B., Moutoussis, M., Smittenaar, P., Zeidman, P., Taylor, T., Hrynkiewicz, L., Lam, J., Skandali, N., Siegel, J. Z., Ousdal, O. T., et al. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.1713 · Cited by 209 · see citing works
Self-citationWhat is mood? A computational perspective
Clark, J. E., Watson, S., & Friston, K. J. (2018) · Psychological Medicine · doi:10.1017/s0033291718000430 · Cited by 204 · see citing works
Mechanisms of Hierarchical Reinforcement Learning in Cortico-Striatal Circuits 2: Evidence from fMRI
Badre, D., & Frank, M. (2011) · Cerebral Cortex · doi:10.1093/cercor/bhr117 · Cited by 203 · see citing works
Musical reward prediction errors engage the nucleus accumbens and motivate learning
Gold, B. P., Herrero, E. M., Zeighami, Y., Benovoy, M., Dagher, A., & Zatorre, R. J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1809855116 · Cited by 182 · see citing works
Reward Processing, Neuroeconomics, and Psychopathology
Zald, D. H., & Treadway, M. T. (2017) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-044957 · Cited by 181 · see citing works
Skvortsova, V., Palminteri, S., & Pessiglione, M. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.1350-14.2014 · Cited by 177 · see citing works
Hauser, T. U., Iannaccone, R., Ball, J., Mathys, C., Brandeis, D., Walitza, S., & Brem, S. (2014) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2014.1093 · Cited by 168 · see citing works
Effects of early life stress on amygdala and striatal development
Fareri, D. S., & Tottenham, N. (2016) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2016.04.005 · Cited by 166 · see citing works
What’s in a word? How instructions, suggestions, and social information change pain and emotion
Koban, L., Jepma, M., Geuter, S., & Wager, T. D. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.02.014 · Cited by 155 · see citing works
Brain networks for confidence weighting and hierarchical inference during probabilistic learning
Meyniel, F., & Dehaene, S. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1615773114 · Cited by 138 · see citing works
Disambiguating ventral striatum fMRI-related bold signal during reward prediction in schizophrenia
Morris, R. W., Vercammen, A., Lenroot, R., Moore, L., Langton, J. M., Short, B., Kulkarni, J., Curtis, J., O'Donnell, M., Weickert, C. S., et al. (2012) · Molecular Psychiatry · doi:10.1038/mp.2011.75 · Cited by 149 · see citing works
Separate mesocortical and mesolimbic pathways encode effort and reward learning signals
Hauser, T. U., Eldar, E., & Dolan, R. J. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1705643114 · Cited by 135 · see citing works
Collins, A. G. E., & Frank, M. J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720963115 · Cited by 146 · see citing works
Anatomical and functional organization of the human substantia nigra and its connections
Zhang, Y., Larcher, K. M.-H., Mišić, B., & Dagher, A. (2017) · eLife · doi:10.7554/elife.26653 · Cited by 148 · see citing works
Behavioral, Physiological, and Neural Signatures of Surprise during Naturalistic Sports Viewing
Antony, J. W., Hartshorne, T. H., Pomeroy, K., Gureckis, T. M., Hasson, U., McDougle, S. D., & Norman, K. A. (2020) · Neuron · doi:10.1016/j.neuron.2020.10.029 · Cited by 146 · see citing works
The role of beta-gamma oscillations in unexpected rewards processing
HajiHosseini, A., Rodríguez‐Fornells, A., & Marco‐Pallarés, J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.125 · Cited by 146 · see citing works
Neural Evidence of Regret and Its Implications for Investor Behavior
Frydman, C., & Camerer, C. F. (2016) · Review of Financial Studies · doi:10.1093/rfs/hhw010 · Cited by 142 · see citing works
Amygdala and Ventral Striatum Make Distinct Contributions to Reinforcement Learning
Costa, V. D., Monte, O. D., Lucas, D. R., Murray, E. A., & Averbeck, B. B. (2016) · Neuron · doi:10.1016/j.neuron.2016.09.025 · Cited by 133 · see citing works
Positive reward prediction errors during decision-making strengthen memory encoding
Jang, A. I., Nassar, M. R., Dillon, D. G., & Frank, M. J. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0597-3 · Cited by 129 · see citing works
Alderson‐Day, B., Diederen, K. M. J., Fernyhough, C., Ford, J. M., Horga, G., Margulies, D. S., McCarthy‐Jones, S., Northoff, G., Shine, J. M., Turner, J. A., et al. (2016) · Schizophrenia Bulletin · doi:10.1093/schbul/sbw078 · Cited by 129 · see citing works
Computational Psychiatry of ADHD: Neural Gain Impairments across Marrian Levels of Analysis
Hauser, T. U., Fiore, V. G., Moutoussis, M., & Dolan, R. J. (2016) · Trends in Neurosciences · doi:10.1016/j.tins.2015.12.009 · Cited by 118 · see citing works
Cerebral Correlates of Salient Prediction Error for Different Rewards and Punishments
Metereau, E., & Dreher, J.-C. (2013) · Cerebral Cortex · doi:10.1093/cercor/bhs037 · Cited by 114 · see citing works
Dopamine, reward learning, and active inference
FitzGerald, T. H. B., Dolan, R. J., & Friston, K. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00136 · Cited by 112 · see citing works
Cai, L. X., Pizano, K., Gundersen, G. W., Hayes, C. L., Fleming, W., Holt, S., Cox, J., & Witten, I. B. (2020) · eLife · doi:10.7554/elife.54936 · Cited by 104 · see citing works
Deficits in attentional control: Cholinergic mechanisms and circuitry-based treatment approaches.
Sarter, M. F., & Paolone, G. (2011) · Behavioral Neuroscience · doi:10.1037/a0026227 · Cited by 98 · see citing works
Neural mechanisms for learning self and other ownership
Lockwood, P. L., Wittmann, M. K., Apps, M. A. J., Klein-Flügge, M. C., Crockett, M. J., Humphreys, G. W., & Rushworth, M. F. S. (2018) · Nature Communications · doi:10.1038/s41467-018-07231-9 · Cited by 95 · see citing works
Hauser, T. U., Iannaccone, R., Dolan, R. J., Ball, J., Hättenschwiler, J., Drechsler, R., Rufer, M., Brandeis, D., Walitza, S., & Brem, S. (2017) · Psychological Medicine · doi:10.1017/s0033291716003305 · Cited by 86 · see citing works
Vigor in the Face of Fluctuating Rates of Reward: An Experimental Examination
Guitart‐Masip, M., Beierholm, U., Dolan, R. J., Düzel, E., & Dayan, P. (2011) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00090 · Cited by 86 · see citing works
The role of learning-related dopamine signals in addiction vulnerability
Huys, Q. J. M., Tobler, P. N., Hasler, G., & Flagel, S. B. (2014) · Progress in brain research · doi:10.1016/b978-0-444-63425-2.00003-9 · Cited by 87 · see citing works
Value Learning through Reinforcement
Daw, N. D., & Tobler, P. N. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00015-2 · Cited by 96 · see citing works
Dynamic shaping of dopamine signals during probabilistic Pavlovian conditioning
Hart, A. S., Clark, J. J., & Phillips, P. E. M. (2015) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2014.07.010 · Cited by 82 · see citing works
Fareri, D. S., Gabard-Durnam, L. J., Goff, B., Flannery, J., Gee, D. G., Lumian, D. S., Caldera, C., & Tottenham, N. (2017) · Development and Psychopathology · doi:10.1017/s0954579417001456 · Cited by 83 · see citing works
Pessiglione, M., & Delgado, M. R. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.08.006 · Cited by 82 · see citing works
Taming the beast: extracting generalizable knowledge from computational models of cognition
Nassar, M. R., & Frank, M. J. (2016) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2016.04.003 · Cited by 91 · see citing works
Dissociating neural learning signals in human sign- and goal-trackers
Schad, D. J., Rapp, M. A., Garbusow, M., Nebe, S., Sebold, M., Obst, E., Sommer, C., Deserno, L., Rabovsky, M., Friedel, E., et al. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0765-5 · Cited by 81 · see citing works
Motivational Deficits in Schizophrenia and the Representation of Expected Value
Waltz, J. A., & Gold, J. M. (2015) · Current topics in behavioral neurosciences · doi:10.1007/7854_2015_385 · Cited by 80 · see citing works
Past Makes Future: Role of pFC in Prediction
Fuster, J. M., & Bressler, S. L. (2014) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00746 · Cited by 79 · see citing works
Jocham, G., Klein, T. A., & Ullsperger, M. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.0757-14.2014 · Cited by 70 · see citing works
Anterior insula reflects surprise in value-based decision-making and perception
Loued-Khenissi, L., Pfeuffer, A., Einhäuser, W., & Preuschoff, K. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116549 · Cited by 66 · see citing works
Gueguen, M. C. M., Lopez‐Persem, A., Billeke, P. B., Lachaux, J.-P., Rheims, S., Kahane, P., Minotti, L., Olivier David, A., Pessiglione, M., & Bastin, J. (2021) · Nature Communications · doi:10.1038/s41467-021-23704-w · Cited by 68 · see citing works
Amiez, C., Sallet, J., Procyk, E., & Petrides, M. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.06.023 · Cited by 74 · see citing works
Bo, K., Kraynak, T. E., Kwon, M., Sun, M. M. C., Gianaros, P. J., & Wager, T. D. (2024) · Nature Neuroscience · doi:10.1038/s41593-024-01605-7 · Cited by 76 · see citing works
Knowledge acquisition is governed by striatal prediction errors
Pine, A., Sadeh, N., Ben-Yakov, A., Dudai, Y., & Mendelsohn, A. (2018) · Nature Communications · doi:10.1038/s41467-018-03992-5 · Cited by 70 · see citing works
Scholl, J., & Klein-Flügge, M. C. (2017) · Behavioural Brain Research · doi:10.1016/j.bbr.2017.09.050 · Cited by 62 · see citing works
Neural mechanisms of the link between giving social support and health
Inagaki, T. K. (2018) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13703 · Cited by 62 · see citing works
Fouragnan, E., Queirazza, F., Retzler, C., Mullinger, K. J., & Philiastides, M. G. (2017) · Scientific Reports · doi:10.1038/s41598-017-04507-w · Cited by 61 · see citing works
Sambrook, T. D., & Goslin, J. (2014) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2014.06.004 · Cited by 60 · see citing works
Woodworth, H. L., Perez‐Bonilla, P., Beekly, B. G., Lewis, T. J., & Leinninger, G. M. (2018) · eNeuro · doi:10.1523/eneuro.0191-17.2018 · Cited by 64 · see citing works
Anatomy and Connectivity of the Reward Circuit
Haber, S. N. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-805308-9.00001-4 · Cited by 61 · see citing works
Decision-making: from neuroscience to neuroeconomics—an overview
Serra, D. (2021) · Theory and Decision · doi:10.1007/s11238-021-09830-3 · Cited by 61 · see citing works
Scholl, J., Kolling, N., Nelissen, N., Browning, M., Rushworth, M. F. S., & Harmer, C. J. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000756 · Cited by 58 · see citing works
The role of dopamine in risk taking: a specific look at Parkinson’s disease and gambling
Clark, C. A., & Dagher, A. (2014) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2014.00196 · Cited by 60 · see citing works
Ghrelin Enhances Food Odor Conditioning in Healthy Humans: An fMRI Study
Han, J. E., Frasnelli, J., Zeighami, Y., Larcher, K. M.-H., Boyle, J., McConnell, T., Malik, S., Jones‐Gotman, M., & Dagher, A. (2018) · Cell Reports · doi:10.1016/j.celrep.2018.11.026 · Cited by 56 · see citing works
Kiverstein, J. D., Miller, M., & Rietveld, E. (2020) · Neuroscience of Consciousness · doi:10.1093/nc/niaa003 · Cited by 56 · see citing works
Over a Decade of Neuroeconomics: What Have We Learned?
Konovalov, A., & Krajbich, I. M. (2016) · Organizational Research Methods · doi:10.1177/1094428116644502 · Cited by 51 · see citing works
Skvortsova, V., Degos, B., Welter, M., Vidailhet, M., & Pessiglione, M. (2017) · Journal of Neuroscience · doi:10.1523/jneurosci.2081-16.2017 · Cited by 51 · see citing works
Dopamine-independent effect of rewards on choices through hidden-state inference
Blanco-Pozo, M., Akam, T. E., & Walton, M. E. (2024) · Nature Neuroscience · doi:10.1038/s41593-023-01542-x · Cited by 70 · see citing works
A unidirectional but not uniform striatal landscape of dopamine signaling for motivational stimuli
van Elzelingen, W., Goedhoop, J. N., Warnaar, P., Denys, D. A. J. P., Arbab, T., & Willuhn, I. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2117270119 · Cited by 51 · see citing works
Annual Research Review: Developmental computational psychiatry
Hauser, T. U., Will, G., Dubois, M., & Dolan, R. J. (2018) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12964 · Cited by 49 · see citing works
Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation
Kato, A., & Morita, K. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005145 · Cited by 57 · see citing works
Schaefer, L. M., & STEINGLASS, J. E. (2021) · Current Psychiatry Reports · doi:10.1007/s11920-020-01213-9 · Cited by 51 · see citing works
Worhunsky, P. D., Potenza, M. N., & Rogers, R. D. (2017) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2017.05.025 · Cited by 49 · see citing works
The human insula processes both modality-independent and pain-selective learning signals
Horing, B., & Büchel, C. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001540 · Cited by 49 · see citing works
Distinct Processing of Aversive Experience in Amygdala Subregions
Michely, J., Rigoli, F., Rutledge, R. B., Hauser, T. U., & Dolan, R. J. (2019) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2019.07.008 · Cited by 46 · see citing works
Self-citationReinforcement Learning and Tourette Syndrome
Palminteri, S., & Pessiglione, M. (2013) · International review of neurobiology · doi:10.1016/b978-0-12-411546-0.00005-6 · Cited by 46 · see citing works
An MEG signature corresponding to an axiomatic model of reward prediction error
Talmi, D., Fuentemilla, L., Vladimir Litvak, A., Düzel, E., & Dolan, R. J. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.06.051 · Cited by 46 · see citing works
Advanced Reinforcement Learning
Daw, N. D. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00016-4 · Cited by 44 · see citing works
Waltz, J. A., Xu, Z., Brown, E. C., Ruiz, R., Frank, M. J., & Gold, J. M. (2017) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2017.07.008 · Cited by 45 · see citing works
Klein-Flügge, M. C., Wittmann, M. K., Shpektor, A., Jensen, D. E. A., & Rushworth, M. F. S. (2019) · Nature Communications · doi:10.1038/s41467-019-12557-z · Cited by 41 · see citing works
Expected Subjective Value Theory (ESVT): A Representation of Decision Under Risk and Certainty
Glimcher, P. W., & Tymula, A. A. (2016) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2022.12.013 · Cited by 53 · see citing works
Self-citationMotivational Context Modulates Prediction Error Response in Schizophrenia
Reinen, J., Snellenberg, J. X. V., Horga, G., Abi‐Dargham, A., Daw, N. D., & Daphna Shohamy, A. (2016) · Schizophrenia Bulletin · doi:10.1093/schbul/sbw045 · Cited by 43 · see citing works
Breaking human social decision making into multiple components and then putting them together again
Suzuki, S., & O’Doherty, J. P. (2020) · Cortex · doi:10.1016/j.cortex.2020.02.014 · Cited by 43 · see citing works
Moutoussis, M., Rutledge, R. B., Prabhu, G., Hrynkiewicz, L., Lam, J., Ousdal, O. T., Guitart‐Masip, M., Fonagy, P., & Dolan, R. J. (2018) · PLoS ONE · doi:10.1371/journal.pone.0201451 · Cited by 42 · see citing works
Self-citationDecision neuroscience and neuroeconomics: Recent progress and ongoing challenges
Dennison, J. B., Sazhin, D., & Smith, D. V. (2022) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1589 · Cited by 44 · see citing works
Navigating complex decision spaces: Problems and paradigms in sequential choice.
Walsh, M. M., & Anderson, J. R. (2013) · Psychological Bulletin · doi:10.1037/a0033455 · Cited by 39 · see citing works
Beyond Reward Prediction Errors: Human Striatum Updates Rule Values During Learning
Ballard, I. C., Miller, E. M., Piantadosi, S. T., Goodman, N. D., & McClure, S. M. (2017) · Cerebral Cortex · doi:10.1093/cercor/bhx259 · Cited by 33 · see citing works
Tivadar, R. I., Knight, R. T., & Tzovara, A. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.702520 · Cited by 28 · see citing works
A multivariate brain signature for reward
Speer, S. P. H., Keysers, C., Barrios, J. C., Teurlings, C. J. S., Smidts, A., Boksem, M. A. S., Wager, T. D., & Gazzola, V. (2023) · NeuroImage · doi:10.1016/j.neuroimage.2023.119990 · Cited by 31 · see citing works
Sallet, J., Camille, N., & Procyk, E. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00209 · Cited by 30 · see citing works
Insel, C., Reinen, J., Weber, J., Wager, T. D., Jarskog, L. F., Daphna Shohamy, A., & Smith, E. E. (2014) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-014-0261-3 · Cited by 32 · see citing works
Of BOLD Claims and Excessive Fears: A Call for Caution and Patience Regarding Political Neuroscience
Theodoridis, A. G., & Nelson, A. J. (2012) · Political Psychology · doi:10.1111/j.1467-9221.2011.00860.x · Cited by 31 · see citing works
- Authors
- Rutledge, R. B., Dean, M., Caplin, A., & Glimcher, P. W.
- Affiliations
- New York University (US)
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2010-10-06
- Volume
- 30
- Issue
- 40
- Pages
- 13525-13536
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- PMC
- PMC2957369
- OpenAlex
- W2035219495
- Cited by
- 244
- References
- 51
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 9 · 2023: 23 · 2022: 11 · 2021: 11 · 2020: 40 · 2019: 10 · 2018: 16 · 2017: 25 · 2016: 15 · 2015: 12 · 2014: 23 · 2013: 13 · 2012: 9
- FWCI
- 5.61
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Neuroscience (publishedVersion, cc-by-nc-sa)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- National Institutes of Health (5r01ns054775-05)
- National Defense Science and Engineering Graduate
- National Institute on Aging (F31AG031656)
- National Institute of Neurological Disorders and Stroke (R01-NS-054775)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- reward prediction error, axiomatic model, reinforcement learning, model falsifiability, striatum, medial prefrontal cortex, amygdala, posterior cingulate cortex, anterior insula, neural coding, model comparison
- MeSH terms
- Adult, Algorithms, Behavior, Brain Mapping, Cognition, Female, Forecasting, Humans, Image Processing, Computer-Assisted, Functional Laterality, Learning, Male, Models, Neurological, Models, Psychological, Predictive Value of Tests, Regression Analysis, Reinforcement, Psychology, Reward, Prosencephalon
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
