Temporal difference models describe higher-order learning in humans
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
The ability to use environmental stimuli to predict impending harm is critical for survival. Such predictions should be available as early as they are reliable. In pavlovian conditioning, chains of successively earlier predictors are studied in terms of higher-order relationships, and have inspired computational theories such as temporal difference learning. However, there is at present no adequate neurobiological account of how this learning occurs. Here, in a functional magnetic resonance imaging (fMRI) study of higher-order aversive conditioning, we describe a key computational strategy that humans use to learn predictions about pain. We show that neural activity in the ventral striatum and the anterior insula displays a marked correspondence to the signals for sequential learning predicted by temporal difference models. This result reveals a flexible aversive learning process ideally suited to the changing and uncertain nature of real-world environments. Taken with existing data on reward learning, our results suggest a critical role for the ventral striatum in integrating complex appetitive and aversive predictions to coordinate behaviour.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One study specifically showed that AIC encodes both prediction of pain and pain prediction error in the same task (Seymour et al., 2004), which directly supports the functional integration and predictive encoding account of AIC.”
From Anterior insular cortex and emotional awareness (2013) · cites this paper as Seymour et al., 2004
In the citing paper
“AIC responses encode both predicted pain and pain prediction error within a single task [33]”
From Interoceptive inference, emotion, and the embodied self (2013) · cites this paper as [33]
In the citing paper
“This phasic signal represents the difference between the predicted and the observed outcome, which is used to update the model such that future predictions are more accurate (3).”
From Aberrant Salience, Information Processing, and Dopaminergic Signaling in People at Clinical High Risk for Psychosis (2020) · cites this paper as [3]
In the citing paper
“Additionally, we hypothesized the anterior insular activity to be altered, because this region is crucially involved in RPE processing (Pessiglione et al., 2006; Seymour et al., 2004; Voon et al., 2010; Wittmann et al., 2008)”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
In the citing paper
“It is well known that aIns activity often coincides with activation in the dmPFC (cf. Hauser et al., 2014a, 2014b; Nelson et al., 2010; Seymour et al., 2004).”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 1×
“, which has close links with methods for prediction, and optimal action selection, in engineerin”
MethodReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Self-citationFunctional profiling of the Saccharomyces cerevisiae genome
Giaever, G., Chu, A. M., Ni, L., Connelly, C., Riles, L., Véronneau, S., Dow, S., Lucau-Danila, A., Anderson, K., André, B., et al. (2002) · Nature · doi:10.1038/nature00935 · Cited by 4,316
Systematic functional analysis of the Caenorhabditis elegans genome using RNAi
Kamath, R. S., Fraser, A. G., Dong, Y., Poulin, G., Durbin, R., Gotta, M., Kanapin, A., Le Bot, N., Moreno, S., Sohrmann, M., et al. (2003) · Nature · doi:10.1038/nature01278 · Cited by 3,616
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Self-citationAzmitia, E. C., & Segal, M. (1978) · Journal of Comparative Neurology · doi:10.1002/cne.901790311 · Cited by 1,909 · cited 1×
“) suggestion that dopamine neurons in the ventral tegmental area and in the substantia nigra report appetitive prediction error, it has been suggested that serotonin-producing neurons of the dorsal raphe nucleus, which send strong projections to the ventral striatu”
BackgroundAn opponent-process theory of motivation. I. Temporal dynamics of affect
Solomon, R. L., & Corbit, J. D. (1974) · Psychological Review · doi:10.1037/h0036128 · Cited by 1,704
Research works printed but not linked: 1
- Barto, A. G., Sutton, R. S. & Watkins, C. J. C. H. in Learning and Computational Neuroscience: Foundations of Adaptive Networks (eds Gabriel, M. & Moor, J.) 539–602 (MIT, Cambridge, Massachusetts, 1990)
Cited by
How do you feel — now? The anterior insula and human awareness
(Bud) Craig, A. D. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2555 · Cited by 6,657
Separate neural systems value immediate and delayed monetary rewards
McClure, S. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2004) · Science · doi:10.1126/science.1100907 · Cited by 3,089
POSNER, J., RUSSELL, J. A., & PETERSON, B. S. (2005) · Development and Psychopathology · doi:10.1017/s0954579405050340 · Cited by 2,602
The integration of negative affect, pain and cognitive control in the cingulate cortex
Shackman, A. J., Salomons, T. V., Slagter, H. A., Fox, A. S., Winter, J. J., & Davidson, R. J. (2011) · Nature Reviews Neuroscience · doi:10.1038/nrn2994 · Cited by 2,082
Uncertainty and anticipation in anxiety: an integrated neurobiological and psychological perspective
Grupe, D. W., & Nitschke, J. B. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3524 · Cited by 2,007
Interoceptive inference, emotion, and the embodied self
Seth, A. K. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.09.007 · Cited by 1,816
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779
Self-citationOptimality principles in sensorimotor control
Todorov, E. (2004) · Nature Neuroscience · doi:10.1038/nn1309 · Cited by 1,746
Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans
Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006) · Nature · doi:10.1038/nature05051 · Cited by 1,584
Self-citationReward representations and reward-related learning in the human brain: Insights from neuroimaging
O’Doherty, J. P. (2004) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2004.10.016 · Cited by 1,426
Self-citationGetting to Know You: Reputation and Trust in a Two-Person Economic Exchange
King-Casas, B., Tomlin, D., Anen, C., Camerer, C. F., Quartz, S. R., & Montague, P. R. (2005) · Science · doi:10.1126/science.1108062 · Cited by 1,345
Eye movements in natural behavior
Hayhoe, M., & Ballard, D. (2005) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2005.02.009 · Cited by 1,326
“Although dopamine is acknowledged to be the main reward signal, attention has turned to the other neurotransmitters as having a role in second order reward statistics such as uncertainty and risk [53, 54 ].”
BackgroundA common role of insula in feelings, empathy and uncertainty
Singer, T., Critchley, H. D., & Preuschoff, K. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.05.001 · Cited by 1,270
Neural mechanisms of autonomic, affective, and cognitive integration
Critchley, H. D. (2005) · Journal of Comparative Neurology · doi:10.1002/cne.20749 · Cited by 1,214
“From Singer et al. (2004) and Seymour et al. (2004).”
BackgroundGläscher, J., Daw, N., Dayan, P., & O'Doherty, J. P. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.016 · Cited by 1,166
“This is consistent with many previous accounts (McClure et al., 2003; O’Doherty et al., 2003; Seymour et al., 2004).”
BackgroundResultSelf-citationCorticostriatal functional connectivity predicts transition to chronic back pain
Baliki, M. N., Petre, B., Torbey, S., Herrmann, K. M., Huang, L., Schnitzer, T. J., Fields, H. L., & Apkarian, A. V. (2012) · Nature Neuroscience · doi:10.1038/nn.3153 · Cited by 1,050
The cognitive functions of the caudate nucleus
Grahn, J. A., Parkinson, J. A., & Owen, A. M. (2008) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2008.09.004 · Cited by 940
“When responses do not determine the outcome (or only non-effortful responses are necessary) then the caudate does not respond (Berns et al., 2001; Breiter et al., 2001; Elliott et al., 2003; McClure et al., 2003; O’Doherty et al., 2006; O’Doherty et al., 2003; Seymour et al., 2004).”
BackgroundCerebellar networks with the cerebral cortex and basal ganglia
Bostan, A. C., Dum, R. P., & Strick, P. L. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.03.003 · Cited by 759
The basal ganglia and the cerebellum: nodes in an integrated network
Bostan, A. C., & Strick, P. L. (2018) · Nature Reviews Neuroscience · doi:10.1038/s41583-018-0002-7 · Cited by 786
“Part b is adapted from reF.(98), Macmillan Publishers Limited.”
The basal ganglia communicate with the cerebellum
Bostan, A. C., Dum, R. P., & Strick, P. L. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1000496107 · Cited by 780
“For example, several imaging studies have examined whether regions of the basal ganglia and related cortical areas display functional activation consistent with their involvement in “temporal difference” models of reward-related learning (27, 28).”
BackgroundThe Neuroscience of Social Decision-Making
Rilling, J. K., & Sanfey, A. G. (2011) · Annual Review of Psychology · doi:10.1146/annurev.psych.121208.131647 · Cited by 775
“Finally, recent fMRI data implicate right anterior insula in aversive conditioning (Seymour et al. 2004).”
Anterior insular cortex and emotional awareness
Gu, X., Hof, P. R., Friston, K. J., & Fan, J. (2013) · Journal of Comparative Neurology · doi:10.1002/cne.23368 · Cited by 712
“One study specifically showed that AIC encodes both prediction of pain and pain prediction error in the same task (Seymour et al., 2004), which directly supports the functional integration and predictive encoding account of AIC.”
Self-citationNeurocognitive aspects of pain perception
Wiech, K., Ploner, M., & Tracey, I. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.05.005 · Cited by 737
Reward‐Related Responses in the Human Striatum
DELGADO, M. R. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.002 · Cited by 736
Reinforcement learning in the brain
Niv, Y. (2009) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2008.12.005 · Cited by 717
“…afferents), and might not differentiate inhibitory from excitatory afferent activity, as is illustrated by the fact that BOLD correlates of positive prediction errors for pain and punishment have also been found in the striatum (Jensen et al., 2007;Menon et al., 2007; Seymour et al., 2004).”
BackgroundReinforcement learning: The Good, The Bad and The Ugly
Dayan, P., & Niv, Y. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.08.003 · Cited by 704
“Further, other studies found a more unitary neural representation of discounting [65], at least in the BOLD signal, and recent results confirm that dopaminergic prediction errors indeed show the expected effects of discounting [58 ].”
InfluentialSelf-citationBOLD responses reflecting dopaminergic signals in the human ventral tegmental area
D'Ardenne, K., McClure, S. M., Nystrom, L. E., & Cohen, J. D. (2008) · Science · doi:10.1126/science.1150605 · Cited by 661
Doya, K. (2008) · Nature Neuroscience · doi:10.1038/nn2077 · Cited by 642
Craig, A. D. (. (2010) · Brain Structure and Function · doi:10.1007/s00429-010-0248-y · Cited by 600
Emotion, Cognition, and Mental State Representation in Amygdala and Prefrontal Cortex
Salzman, C. D., & Fusi, S. (2010) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.051508.135256 · Cited by 588
“…similar tasks are often used to examine human valuation processes using fMRI (Breiter et al. 2001, Gottfried et al. 2003, Kable & Glimcher 2007, Knutson et al. 2001, Knutson & Cooper 2005, McClure et al. 2004, Montague et al. 2006, O’Doherty et al. 2001, Rangel et al. 2008, Seymour et al. 2004).”
MethodDecision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580
“Figure 1A shows a case of prediction learning (Seymour et al., 2004).”
BackgroundInfluentialSelf-citationPlacebo effects on human μ-opioid activity during pain
Wager, T. D., Scott, D. J., & Zubieta, J.-K. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0702413104 · Cited by 562
Baliki, M. N., Geha, P. Y., Fields, H. L., & Apkarian, A. V. (2010) · Neuron · doi:10.1016/j.neuron.2010.03.002 · Cited by 555
“…have established that circuitry typically involved in valuation is activated by acute noxious stimuli (Becerra and Borsook, 2008; Becerra et al., 2001; Delgado et al., 2008; Harris et al., 2007; Jensen et al., 2003, 2007; Scott et al., 2006, 2007; Seymour et al., 2004, 2005; Zubieta et al., 2005).”
Palaniyappan, L., & Liddle, P. F. (2012) · Journal of Psychiatry and Neuroscience · doi:10.1503/jpn.100176 · Cited by 548
Neural correlates of mentalizing-related computations during strategic interactions in humans
Hampton, A. N., Bossaerts, P., & O'Doherty, J. P. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0711099105 · Cited by 515
Self-citationA unified framework for addiction: Vulnerabilities in the decision process
Redish, A. D., Jensen, S., & Johnson, A. (2008) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0800472x · Cited by 501
Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498
“…this update signal (posterior–prior) in the ventral striatum and mPFC, regions that have been associated previously with prediction error coding in neuroimaging studies (McClure et al., 2003; J. P. O'Doherty et al., 2003; O'Doherty et al., 2004; Seymour et al., 2004; Tanaka et al., 2004) (Fig.”
BackgroundSelf-citationEmotional moments across time: a possible neural basis for time perception in the anterior insula
Craig, A. D. (. (2009) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2009.0008 · Cited by 499
Kalisch, R., Korenfeld, E., Stephan, K. E., Weiskopf, N., Seymour, B., & Dolan, R. J. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.2021-06.2006 · Cited by 493
“We observed activation in areas previously implicated in fear responses [bilateral striatum (Jensen et al., 2003; Phelps et al., 2004; Seymour et al., 2004; Wager et al., 2004), left temporal cortex and cerebellum (Ploghaus et al., 1999; Wager et al., 2004), hypothalamus (Simpson et al., 2001;…”
BackgroundMethodSelf-citationRecent advances in Tourette syndrome research
Albin, R. L., & Mink, J. W. (2006) · Trends in Neurosciences · doi:10.1016/j.tins.2006.01.001 · Cited by 478
Action, Outcome, and Value: A Dual-System Framework for Morality
Cushman, F. (2013) · Personality and Social Psychology Review · doi:10.1177/1088868313495594 · Cited by 473
The computational neurobiology of learning and reward
Daw, N. D., & Doya, K. (2006) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2006.03.006 · Cited by 465
“Because this system projects to the striatum, it could be the source of BOLD responses correlated with aversive prediction error in the ventral striatum [15].”
BackgroundAtlas, L. Y., & Wager, T. D. (2012) · Neuroscience Letters · doi:10.1016/j.neulet.2012.03.039 · Cited by 463
Differential Encoding of Losses and Gains in the Human Striatum
Seymour, B., Daw, N., Dayan, P., Singer, T., & Dolan, R. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0400-07.2007 · Cited by 462
“In experiments involving primary rewards and punishments, the blood oxygenation level-dependent (BOLD) signal in the human striatum measured using functional magnetic resonance imaging (fMRI) covaries closely with key learning signals used by abstract learning models (O’Doherty et al., 2003; Haruno et al., 2004; Seymo…”
BackgroundMethodInfluentialSelf-citationKim, H., Shimojo, S., & O'Doherty, J. P. (2006) · PLoS Biology · doi:10.1371/journal.pbio.0040233 · Cited by 453
“This is in direct contradiction of previous studies that have reported such signals during aversive learning with pain or even a least preferred food stimulus [31,47–49].”
ResultSelf-citationRoitman, M. F., Wheeler, R. A., & Carelli, R. M. (2005) · Neuron · doi:10.1016/j.neuron.2004.12.055 · Cited by 453
Opponent appetitive-aversive neural processes underlie predictive learning of pain relief
Seymour, B., O'Doherty, J. P., Koltzenburg, M., Wiech, K., Frackowiak, R., Friston, K., & Dolan, R. (2005) · Nature Neuroscience · doi:10.1038/nn1527 · Cited by 439
Self-citationFrom Fear to Safety and Back: Reversal of Fear in the Human Brain
Schiller, D., Levy, I., Niv, Y., LeDoux, J. E., & Phelps, E. A. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.2265-08.2008 · Cited by 429
“…been observed in aversive learning paradigm in humans (LaBar et al., 1998; Ploghaus et al., 2000; Jensen et al., 2003, 2007; Phelps et al., 2004; Seymour et al., 2004; Menon et al., 2007), and animals (Horvitz 2000; Schoenbaum and Setlow, 2003; Pezze and Feldon, 2004), the role of this region…”
BackgroundResultInfluentialPan, W.-X., Schmidt, R., Wickens, J. R., & Hyland, B. I. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.1478-05.2005 · Cited by 429
“Because the underlying algorithmic processes of TD are well understood (Sutton, 1988; Sutton and Barto, 1998), this link between biological and machine learning processes offers a powerful way to progress our understanding of the neural mechanisms of reward-mediated learning (Barto, 1995; Houk et al., 1995; Montague e…”
BackgroundOxytocin Attenuates Affective Evaluations of Conditioned Faces and Amygdala Activity
Petrovic, P., Kalisch, R., Singer, T., & Dolan, R. J. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.4572-07.2008 · Cited by 424
Self-citationThe Influence of Affect on Beliefs, Preferences, and Financial Decisions
Kuhnen, C. M., & Knutson, B. (2011) · Journal of Financial and Quantitative Analysis · doi:10.1017/s0022109011000123 · Cited by 422
Collins, A. G. E., & Frank, M. J. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07980.x · Cited by 417
“Activity and plasticity in striatal neurons, a major target of dopaminergic efferents, are dynamically sensitive to these dopaminergic prediction error signals, which enable the striatum to represent RL values (O’Doherty et al. , 2004; Frank, 2005; Daw & Doya, 2006).”
BackgroundMethodInfluentialA meta-analysis of instructed fear studies: implications for conscious appraisal of threat
Mechias, M.-L., Etkin, A., & Kalisch, R. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.09.040 · Cited by 392
“One may speculate that the anterior insula is involved in generating the subjective feeling of fear based on interoceptive input (Critchley et al., 2004) or participates in reinforcement learning (Seymour et al., 2004, 2005).”
BackgroundGläscher, J., Hampton, A. N., & O'Doherty, J. P. (2009) · Cerebral Cortex · doi:10.1093/cercor/bhn098 · Cited by 387
“2002; AAL template for amygdata only) and from previous imaging studies that have investigated reversal learning and value-related signals derived from computational learning theory (Daw et al. 2006; Hampton et al. 2007; Hampton et al. 2006; Kim et al. 2006; O’Doherty et al. 2003; Seymour et al. 2004; Seymour et al. 2…”
MethodSelf-citationBrooks, J., & Tracey, I. (2005) · Journal of Anatomy · doi:10.1111/j.1469-7580.2005.00428.x · Cited by 378
“…both the intensity (Coghill et al. 1999; Craig et al. 2000) and the laterality (Brooks et al. 2002; Bingel et al. 2003) of painful and non-painful thermal stimuli, but may also have a role in affective pain processing (Craig, 2003b; Critchley et al. 2004; Seymour et al. 2004; Singer et al. 2004).”
BackgroundIMAGING VALUATION MODELS IN HUMAN CHOICE
Montague, P. R., King-Casas, B., & Cohen, J. D. (2006) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.112903 · Cited by 368
Neural basis of the undermining effect of monetary reward on intrinsic motivation
Murayama, K., Matsumoto, M., Izuma, K., & Matsumoto, K. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1013305107 · Cited by 365
Bodi, N., Keri, S., Nagy, H., Moustafa, A., Myers, C. E., Daw, N., Dibo, G., Takats, A., Bereczki, D., & Gluck, M. A. (2009) · Brain · doi:10.1093/brain/awp094 · Cited by 363
“, 2001), and the striatum may also be implicated in aversive learning and negative prediction errors (Seymour et al., 2004; for review, see Delgado et al., 2008).”
BackgroundNeural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain
Niv, Y., Edlund, J. A., Dayan, P., & O'Doherty, J. P. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.5498-10.2012 · Cited by 356
“Furthermore, at least in the BOLD signal, we did not find risk-sensitivity-related asymmetry in the prediction errors at the time of the outcome (Seymour et al., 2004).”
BackgroundSelf-citationExpectation modulates neural responses to pleasant and aversive stimuli in primate amygdala
Belova, M. A., Paton, J. J., Morrison, S. E., & Salzman, C. D. (2007) · Neuron · doi:10.1016/j.neuron.2007.08.004 · Cited by 349
“…BOLD responses reflect both reward (Haruno and Kawato, 2006; O'Doherty et al., 2003; Seymour et al., 2005) and punishment (Ploghaus et al., 2000; Seymour et al., 2004, 2005) prediction error in the cerebellum, orbitofrontal, parietal, right insula and cingulate cortices, ventral striatum, and…”
Redish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.784 · Cited by 346
Dialogues on prediction errors
Niv, Y., & Schoenbaum, G. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.03.006 · Cited by 344
Hebbian learning of cognitive control: Dealing with specific and nonspecific adaptation.
Verguts, T., & Notebaert, W. (2008) · Psychological Review · doi:10.1037/0033-295x.115.2.518 · Cited by 337
The role of the striatum in aversive learning and aversive prediction errors
Delgado, M. R., Li, J., Schiller, D., & Phelps, E. A. (2008) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0161 · Cited by 328
“(2007) used a secondary reinforcer (monetary loss or gain) in a probabilistic first-order classical delay conditioning task, but used a high-order aversive conditioning task (Seymour et al. 2004) when examining the effect of a primary reinforcer (thermal pain).”
BackgroundMethodResultInfluentialAn Unexpected Sequence of Events: Mismatch Detection in the Human Hippocampus
Kumaran, D., & Maguire, E. A. (2006) · PLoS Biology · doi:10.1371/journal.pbio.0040424 · Cited by 327
Emotion, decision making, and the amygdala.
Seymour, B., & Dolan, R. (2008) · Neuron · doi:10.1016/j.neuron.2008.05.020 · Cited by 324
“…the case for both appetitive learning, thought to be guided by dopaminergic projections from the ventral tegmental area in the midbrain (Nakahara et al., 2004; Satoh et al., 2003; Schultz et al., 1997), and aversive learning, evidenced in humans by fMRI (Jensen et al., 2007; Seymour et al., 2004).”
BackgroundSelf-citationO'Doherty, J. P., Buchanan, T. W., Seymour, B., & Dolan, R. J. (2006) · Neuron · doi:10.1016/j.neuron.2005.11.014 · Cited by 322
“Significant prediction error-related activity has also been observed in ventral striatum during aversive learning (Seymour et al., 2004).”
Self-citationDecision theory: what “should” the nervous system do?
Körding, K. (2007) · Science · doi:10.1126/science.1142998 · Cited by 313
Mapping value based planning and extensively trained choice in the human brain
Wunderlich, K., Dayan, P., & Dolan, R. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3068 · Cited by 310
Self-citationA Dual Role for Prediction Error in Associative Learning
den Ouden, H. E. M., Friston, K. J., Daw, N. D., McIntosh, A. R., & Stephan, K. E. (2009) · Cerebral Cortex · doi:10.1093/cercor/bhn161 · Cited by 306
“…(key dopaminergic targets) has been shown to covary with both predictions and prediction errors (Fletcher et al. 2001; McClure et al. 2003; Corlett et al. 2004; O’Doherty et al. 2004; Seymour et al. 2004; Turner et al. 2004; Gläscher and Büchel 2005; Pessiglione et al. 2006; Jensen et al. 2007).”
InfluentialSelf-citationA computational framework for the study of confidence in humans and animals
Kepecs, A., & Mainen, Z. F. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2012.0037 · Cited by 305
Striatal Prediction Error Modulates Cortical Coupling
den Ouden, H. E. M., Daunizeau, J., Roiser, J., Friston, K. J., & Stephan, K. E. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4458-09.2010 · Cited by 304
“…of learned contingencies, regardless of whether these contingencies signal reinforcement (McClure et al., 2003; O’Doherty et al., 2003, 2004; Seymour et al., 2004; Jensen et al., 2007; Menon et al., 2007), guide decision making (Corlett et al., 2004), or predict target stimuli (as in the…”
InfluentialSelf-citationDynamic Interaction between Reinforcement Learning and Attention in Multidimensional Environments
Leong, Y. C., Radulescu, A., Daniel, R., DeWoskin, V., & Niv, Y. (2017) · Neuron · doi:10.1016/j.neuron.2016.12.040 · Cited by 294
“Another neural signal guidingdecisionmaking is the reward prediction error signal, which is reflected in BOLD activity in the striatum, a major site of efferent dopaminergic connections (O’Doherty et al., 2004; Seymour et al., 2004).”
BackgroundInfluentialEngelmann, J. B. (2009) · Frontiers in Human Neuroscience · doi:10.3389/neuro.09.004.2009 · Cited by 300
“Note that although the coordinates for the midbrain site that we observed are consistent with the SN (Seymour et al., 2004; Tobler et al., 2007), this link must remain speculative because of the spatial resolution of fMRI and the size of this structure.”
BackgroundPlacing prediction into the fear circuit
McNally, G. P., Johansen, J. P., & Blair, H. T. (2011) · Trends in Neurosciences · doi:10.1016/j.tins.2011.03.005 · Cited by 298
Ochsner, K. N. (2008) · Biological Psychiatry · doi:10.1016/j.biopsych.2008.04.024 · Cited by 298
Interoception and emotion: A neuroanatomical perspective.
Craig, A. D. (2008) · Cited by 370
Mechanisms underlying dopamine-mediated reward bias in compulsive behaviors
Voon, V., Pessiglione, M., Brezing, C., Gallea, C., Fernandez, H. H., Dolan, R. J., & Hallett, M. (2010) · Neuron · doi:10.1016/j.neuron.2009.12.027 · Cited by 280
“This finding highlights the role of the anterior insula, a key region implicated in learning from negative outcomes from both monetary loss (Pessiglione et al., 2006) and painful stimuli (Seymour et al., 2004).”
Self-citationNociceptive processing in the human brain
Tracey, I. (2005) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2005.06.010 · Cited by 280
Action versus valence in decision making
Guitart-Masip, M., Duzel, E., Dolan, R., & Dayan, P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.01.003 · Cited by 270
Self-citation(2004) · PubMed · Cited by 272
Cognitive-motor interactions of the basal ganglia in development
Leisman, G., Braun-Benjamin, O., & Melillo, R. (2014) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2014.00016 · Cited by 267
Testing the Reward Prediction Error Hypothesis with an Axiomatic Model
Rutledge, R. B., Dean, M., Caplin, A., & Glimcher, P. W. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.1747-10.2010 · Cited by 244
“Many studies identifying RPE model correlations in the anterior insula involved pain (Seymour et al., 2004) or financial losses (Pessiglione et al., 2006; Voon et al., 2010), events that may be particularly salient.”
BackgroundInfluentialBlunted response to feedback information in depressive illness
Steele, J. D., Kumar, P., & Ebmeier, K. P. (2007) · Brain · doi:10.1093/brain/awm150 · Cited by 261
“This signal has also been reported in the ventral striatum of healthy human subjects learning to predict rewarding and aversive events (O'Doherty et al., 2003, 2004; Seymour et al., 2004, 2005).”
BackgroundValence and salience contribute to nucleus accumbens activation
Cooper, J. C., & Knutson, B. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.08.009 · Cited by 256
“At least two studies have reported NAcc activation correlated with salience prediction error models of conditioning for painful stimuli that included decreases during unpredicted avoidance of painful stimuli (Jensen et al., 2007; Seymour et al., 2004).”
BackgroundSerotonin Selectively Modulates Reward Value in Human Decision-Making
Seymour, B., Daw, N. D., Roiser, J. P., Dayan, P., & Dolan, R. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.0053-12.2012 · Cited by 252
“…any action) have shown an aversive prediction error signal, oriented positively (opposite to that seen in the present study) in ventral and dorsal striatum (Jensen et al., 2003; Seymour et al., 2004, 2007), suggesting a site of convergence with a (putatively dopaminergic) reward prediction error.”
ResultSelf-citationBetter or worse than expected? Aging, learning, and the ERN
Eppinger, B., Kray, J., Mock, B., & Mecklinger, A. (2008) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2007.09.001 · Cited by 248
Representation of aversive prediction errors in the human periaqueductal gray
Roy, M., Shohamy, D., Daw, N., Jepma, M., Wimmer, G. E., & Wager, T. D. (2014) · Nature Neuroscience · doi:10.1038/nn.3832 · Cited by 246
Critical Roles for Anterior Insula and Dorsal Striatum in Punishment-Based Avoidance Learning
Palminteri, S., Justo, D., Jauffret, C., Pavlicek, B., Dauta, A., Delmaire, C., Czernecki, V., Karachi, C., Capelle, L., Durr, A., et al. (2012) · Neuron · doi:10.1016/j.neuron.2012.10.017 · Cited by 242
“Indeed, the anterior insula was found to represent cues predicting primary punishments, such as electric shocks, fearful pictures, or bad tastes, and these punishments themselves (Büchel et al., 1998; Seymour et al., 2004; Nitschke et al., 2006).”
Anxiety, Depression, and Decision Making: A Computational Perspective
Bishop, S. J., & Gagne, C. (2018) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-080317-062007 · Cited by 240
“Similar prediction error signals are generated when aversive outcomes differ from expectation (Li et al. 2011, Mirenowicz & Schultz 1996, Seymour et al. 2004) (though see Schultz 2016 for caveats regarding the interpretation of dopaminergic prediction errors for aversive outcomes and on our current…”
BackgroundNeurocomputational models of basal ganglia function in learning, memory and choice
Cohen, M. X., & Frank, M. J. (2009) · Behavioural Brain Research · doi:10.1016/j.bbr.2008.09.029 · Cited by 237
“These correlations are often observed to be significant in the striatum and frontal cortex, as well as other regions (discussed in more depth below), and are taken to reflect reward prediction error signals from the midbrain to striatal circuitry [34,124,123,148].”
BackgroundResultNOW vs LATER brain circuits: implications for obesity and addiction
Volkow, N. D., & Baler, R. D. (2015) · Trends in Neurosciences · doi:10.1016/j.tins.2015.04.002 · Cited by 237
Chronic pain: The role of learning and brain plasticity
Mansour, A. R., Farmer, M. A., Baliki, M. N., & Apkarian, A. V. (2014) · Restorative Neurology and Neuroscience · doi:10.3233/rnn-139003 · Cited by 237
“…mesolimbic-prefrontal circuit (Ambroggi et al., 2008; Carlezon & Thomas, 2009), which is critical in Accumulating evidence by us, and others, now shows this system is also engaged with pain, its salience, its negative reinforcement value (Baliki et al., al., 2011; Seymour et al., 2004, 2005).”
Rolls, E. T., McCabe, C., & Redoute, J. (2008) · Cerebral Cortex · doi:10.1093/cercor/bhm097 · Cited by 236
“…ventral striatum and a part of the insula was related to the TD error, which arose, for example, at the transition between the first and second visual stimulus if the first visual stimulus had predicted low pain but the second informed the subject that the pain would be high (Seymour et al. 2004).”
Predicting transition to chronic pain
Apkarian, A. V., Baliki, M. N., & Farmer, M. A. (2013) · Current Opinion in Neurology · doi:10.1097/wco.0b013e32836336ad · Cited by 232
Computational Principles of Supervised Learning in the Cerebellum
Raymond, J. L., & Medina, J. F. (2018) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-080317-061948 · Cited by 230
The neurobiology of social decision-making
Rilling, J. K., King-Casas, B., & Sanfey, A. G. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.06.003 · Cited by 227
Hauser, T. U., Iannaccone, R., Walitza, S., Brandeis, D., & Brem, S. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.09.018 · Cited by 203
Is a bird in the hand worth two in the future? The neuroeconomics of intertemporal decision-making
Kalenscher, T., & Pennartz, C. M. A. (2008) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2007.11.004 · Cited by 222
“Especially noteworthy in the present context is the ability of TD-models to mimic a number of phenomena encountered in conditioning paradigms, such as blocking, inhibitory backwards conditioning, stronger conditioning induced by forward CS–US pairings rather than simultaneous CS–US presentation, second-order condition…”
BackgroundBrown, C. A., Seymour, B., Boyle, Y., El-Deredy, W., & Jones, A. K. P. (2008) · Pain · doi:10.1016/j.pain.2007.05.022 · Cited by 218
Self-citation
Metadata
- Authors
- Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S.
- Affiliations
- Wellcome Centre for Human Neuroimaging (GB)
- Gatsby Computational Neuroscience Unit (GB)
- University College London (GB)
- University of Manchester (GB)
- Corresponding author
- Seymour, Ben
- ORCID
- Venue
- Nature
- Published
- 2004-06
- Volume
- 429
- Issue
- 6992
- Pages
- 664-667
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2084912121
- Cited by
- 605
- References
- 60
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 8 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 22 · 2019: 25 · 2018: 24 · 2017: 21 · 2016: 24 · 2015: 31 · 2014: 32 · 2013: 47 · 2012: 37
- FWCI
- 15.70
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- other-oa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Max Planck Digital Library (submittedVersion, other-oa)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- temporal difference learning, higher-order conditioning, classical conditioning, aversive conditioning, pain prediction, ventral striatum, anterior insula, sequential learning, computational models of learning, reward learning
- MeSH terms
- Conditioning, Classical, Cues, Electric Stimulation, Hand, Humans, Learning, Magnetic Resonance Imaging, Models, Neurological, Pain, Punishment, Time Factors, Neostriatum
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.