Temporal difference models describe higher-order learning in humans
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
The ability to use environmental stimuli to predict impending harm is critical for survival. Such predictions should be available as early as they are reliable. In pavlovian conditioning, chains of successively earlier predictors are studied in terms of higher-order relationships, and have inspired computational theories such as temporal difference learning. However, there is at present no adequate neurobiological account of how this learning occurs. Here, in a functional magnetic resonance imaging (fMRI) study of higher-order aversive conditioning, we describe a key computational strategy that humans use to learn predictions about pain. We show that neural activity in the ventral striatum and the anterior insula displays a marked correspondence to the signals for sequential learning predicted by temporal difference models. This result reveals a flexible aversive learning process ideally suited to the changing and uncertain nature of real-world environments. Taken with existing data on reward learning, our results suggest a critical role for the ventral striatum in integrating complex appetitive and aversive predictions to coordinate behaviour.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One study specifically showed that AIC encodes both prediction of pain and pain prediction error in the same task (Seymour et al., 2004), which directly supports the functional integration and predictive encoding account of AIC.”
From Anterior insular cortex and emotional awareness (2013) · cites this paper as Seymour et al., 2004
In the citing paper
“AIC responses encode both predicted pain and pain prediction error within a single task [33]”
From Interoceptive inference, emotion, and the embodied self (2013) · cites this paper as [33]
In the citing paper
“This phasic signal represents the difference between the predicted and the observed outcome, which is used to update the model such that future predictions are more accurate (3).”
From Aberrant Salience, Information Processing, and Dopaminergic Signaling in People at Clinical High Risk for Psychosis (2020) · cites this paper as [3]
In the citing paper
“Additionally, we hypothesized the anterior insular activity to be altered, because this region is crucially involved in RPE processing (Pessiglione et al., 2006; Seymour et al., 2004; Voon et al., 2010; Wittmann et al., 2008)”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
In the citing paper
“It is well known that aIns activity often coincides with activation in the dmPFC (cf. Hauser et al., 2014a, 2014b; Nelson et al., 2010; Seymour et al., 2004).”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 1×
“, which has close links with methods for prediction, and optimal action selection, in engineerin”
MethodReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Self-citationFunctional profiling of the Saccharomyces cerevisiae genome
Giaever, G., Chu, A. M., Ni, L., Connelly, C., Riles, L., Véronneau, S., Dow, S., Lucau-Danila, A., Anderson, K., André, B., et al. (2002) · Nature · doi:10.1038/nature00935 · Cited by 4,316
Systematic functional analysis of the Caenorhabditis elegans genome using RNAi
Kamath, R. S., Fraser, A. G., Dong, Y., Poulin, G., Durbin, R., Gotta, M., Kanapin, A., Le Bot, N., Moreno, S., Sohrmann, M., et al. (2003) · Nature · doi:10.1038/nature01278 · Cited by 3,616
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Self-citationAzmitia, E. C., & Segal, M. (1978) · Journal of Comparative Neurology · doi:10.1002/cne.901790311 · Cited by 1,909 · cited 1×
“) suggestion that dopamine neurons in the ventral tegmental area and in the substantia nigra report appetitive prediction error, it has been suggested that serotonin-producing neurons of the dorsal raphe nucleus, which send strong projections to the ventral striatu”
BackgroundAn opponent-process theory of motivation. I. Temporal dynamics of affect
Solomon, R. L., & Corbit, J. D. (1974) · Psychological Review · doi:10.1037/h0036128 · Cited by 1,704
Research works printed but not linked: 1
- Barto, A. G., Sutton, R. S. & Watkins, C. J. C. H. in Learning and Computational Neuroscience: Foundations of Adaptive Networks (eds Gabriel, M. & Moor, J.) 539–602 (MIT, Cambridge, Massachusetts, 1990)
Cited by
Items 201–300 of 605
Expectations and outcomes: decision-making in the primate brain
McCoy, A. N., & Platt, M. L. (2004) · Journal of Comparative Physiology A · doi:10.1007/s00359-004-0565-9 · Cited by 99
“2004), ventral striatum (Pagnoni et al. 2002; Seymour et al. 2004), CGp (Smith et al.”
BackgroundInfluentialDissociable Learning Processes Underlie Human Pain Conditioning
Zhang, S., Mano, H., Ganesh, G., Robbins, T., & Seymour, B. (2016) · Current Biology · doi:10.1016/j.cub.2015.10.066 · Cited by 87
“C two keymeasures: a prediction error term that records the difference between pain expectations and outcomes [5] and an ‘‘associability’’ term that keeps track of the uncertainty of predictions [7, 8].”
BackgroundSelf-citationSensitivity to pain expectations: A Bayesian model of individual differences
Hoskin, R., Berzuini, C., Acosta-Kane, D., El-Deredy, W., Guo, H., & Talmi, D. (2019) · Cognition · doi:10.1016/j.cognition.2018.08.022 · Cited by 87
Pauli, W. M., Larsen, T., Collette, S., Tyszka, J. M., Seymour, B., & O'Doherty, J. P. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.2277-15.2015 · Cited by 87
“1B), and capturing the core feature of the temporal difference algorithm: learning via prediction errors induced by sequential predictors (Seymour et al., 2004).”
BackgroundSelf-citationGoode, T. D., & Maren, S. (2014) · ILAR Journal · doi:10.1093/ilar/ilu008 · Cited by 87
“Fear is highly adaptive; it is essential in motivating defensive behavior in the face of threat (Cantor 2009; Ellis 1982; Giske et al. 2013; Öhman and Mineka 2001; Seymour et al. 2004).”
BackgroundNormative development of ventral striatal resting state connectivity in humans
Fareri, D. S., Gabard-Durnam, L., Goff, B., Flannery, J., Gee, D. G., Lumian, D. S., Caldera, C., & Tottenham, N. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.06.022 · Cited by 87
“…et al., 2008), important in evaluating risk (Kuhnen and Knutson, 2005; Niv et al., 2012; Pessiglione et al., 2006; Preuschoff et al., 2008; Seymour et al., 2004), learning from aversive incentives (Chang et al., 2013; Pessiglione et al., 2006; Preuschoff et al., 2008; Seymour et al.,…”
BackgroundFinger, E. C., Marsh, A. A., Buzas, B., Kamel, N., Rhodes, R., Vythilingham, M., Pine, D. S., Goldman, D., & Blair, J. R. (2007) · Neuropsychopharmacology · doi:10.1038/sj.npp.1301182 · Cited by 86
“Based on animal studies, both stimulusreward and stimulus-punishment learning recruit the amygdala, striatum and middle frontal cortex (Baxter and Murray, 2002; Everitt et al, 1987; Seymour et al, 2004).”
BackgroundValue Learning through Reinforcement
Daw, N. D., & Tobler, P. N. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00015-2 · Cited by 138
The Caudate Nucleus Mediates Learning of Stimulus–Control State Associations
Chiu, Y., Jiang, J., & Egner, T. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.0778-16.2016 · Cited by 98
Brown, C. A., Seymour, B., El-Deredy, W., & Jones, A. K. P. (2008) · Pain · doi:10.1016/j.pain.2008.04.028 · Cited by 83
“Firstly, activity in this area shows a greater association with subjective pain ratings than actual increases in temperature [12]; secondly, anticipatory activity here has been seen immediately preceding a predictable pain stimulus [8,36,43] which overlaps with subsequent pain-related activity [25]; and thirdly, antic…”
BackgroundSelf-citationEfficient reinforcement learning: computational theories, neuroscience and robotics
Kawato, M., & Samejima, K. (2007) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2007.03.004 · Cited by 94
The anatomy of fear learning in the cerebellum: A systematic meta-analysis
Lange, I., Kasanova, Z., Goossens, L., Leibold, N. K., Zeeuw, C. I. D., van Amelsvoort, T., & Schruers, K. R. J. (2015) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2015.09.019 · Cited by 91
van den Bos, W., McClure, S. M., Harris, L. T., Fiske, S. T., & Cohen, J. D. (2007) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.7.4.337 · Cited by 95
“Previous studies have also found an increased BOLD signal to aversive stimuli in the ventral striatum (Jensen et al., 2003; Seymour et al., 2004).”
BackgroundA temporal difference account of avoidance learning
Moutoussis, M., Bentall, R. P., Williams, J., & Dayan, P. (2008) · Network: Computation in Neural Systems · doi:10.1080/09548980802192784 · Cited by 80
“2006) although some studies did not detect such activity (Seymour et al. 2004).”
BackgroundResultInfluentialSelf-citationNociception and autonomic nervous system
Cortelli, P., Giannini, G., Favoni, V., Cevoli, S., & Pierangeli, G. (2013) · Neurological Sciences · doi:10.1007/s10072-013-1391-z · Cited by 96
Owen, D. G., Clarke, C. F., Ganapathy, S., Prato, F. S., & St. Lawrence, K. S. (2010) · Pain · doi:10.1016/j.pain.2009.10.003 · Cited by 80
“Traditionally implicated in sensory-discriminative processing of pain [8,10], more recent studies suggest the insula may be involved in affective or attentional processing as well, with the roles divided between anterior (affective) and posterior (sensory) regions [5,9,12,13,17,42,44,45].”
ResultGrow, D. A., McCarrey, J. R., & Navara, C. S. (2016) · Stem Cell Research · doi:10.1016/j.scr.2016.08.013 · Cited by 95
Striatal structure and function predict individual biases in learning to avoid pain
Eldar, E., Hauser, T. U., Dayan, P., & Dolan, R. J. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1519829113 · Cited by 79
“, learning what gambles should be avoided), striatal response to painful outcomes was consistent with an aversive prediction error, as seen in fear conditioning (14).”
ResultSelf-citationLucantonio, F., Caprioli, D., & Schoenbaum, G. (2014) · Neuropharmacology · doi:10.1016/j.neuropharm.2013.05.033 · Cited by 78
“A rich body of work using these computational tools has already broadened our understanding of the microstructure of behavior and the neural circuits underlying valuebased decision-making (O'Doherty et al., 2003, Seymour et al., 2004).”
BackgroundPremonitory urges and tics in Tourette syndrome: computational mechanisms and neural correlates
Conceição, V. A., Dias, Â. J. B., Farinha, A. C., & Maia, T. V. (2017) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.08.009 · Cited by 85
“Indeed, the insula has been implicated both in premonitory urges and in the coding of aversive state values [V(s) < 0] [74,75], aversive prediction errors (d < 0) [69,75–77], and aversive outcomes, even when they are fully predicted [78–80], which suggests coding of primary aversive reinforcers (r < 0).”
BackgroundTourette syndrome and other tic disorders
Singer, H. S. (2011) · Handbook of clinical neurology · doi:10.1016/b978-0-444-52014-2.00046-x · Cited by 91
Cohen, M. X. (2007) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsl021 · Cited by 78
“In particular, phasic increases in activity are observed when reinforcements are better than expected (a positive prediction error), and phasic decreases in activity are observed when reinforcements are worse than expected or not given (a negative prediction error) (Schultz et al., 1997; Waelti et al., 2001; Daw et al…”
BackgroundMethodResultInfluentialBrain mechanism of reward prediction under predictable and unpredictable environmental dynamics
Tanaka, S., Samejima, K., Okada, G., Ueda, K., Okamoto, Y., Yamawaki, S., & Doya, K. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.05.039 · Cited by 89
The neural correlates of negative prediction error signaling in human fear conditioning
Spoormaker, V. I., Andrade, K. C., Schröter, M. S., Sturm, A., Goya‐Maldonado, R., Sämann, P. G., & Czisch, M. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.09.042 · Cited by 94
“…classical fear conditioning paradigms (Büchel and Dolan, 2000) to study PE signaling have also revealed increased activation in dopaminergic midbrain regions (Menon et al., 2007; Seymour et al., 2004) and the striatum and insula (Jensen et al., 2007; Schiller et al., 2008; Seymour et al., 2004).”
BackgroundResultInfluentialDissociable reward and timing signals in human midbrain and ventral striatum
Klein-Flügge, M. C., Hunt, L. T., Bach, D. R., Dolan, R. J., & Behrens, T. E. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.08.024 · Cited by 75
“, 2008); this is also true when learning about the probability of aversive events (Seymour et al., 2004; Jensen et al., 2007; Seymour et al., 2007).”
BackgroundSelf-citationHelfinstein, S. M., Benson, B., Perez-Edgar, K., Bar-Haim, Y., Detloff, A., Pine, D. S., Fox, N. A., & Ernst, M. (2011) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2010.12.015 · Cited by 75
Gueguen, M. C. M., Lopez‐Persem, A., Billeke, P. B., Lachaux, J.-P., Rheims, S., Kahane, P., Minotti, L., David, O., Pessiglione, M., & Bastin, J. (2021) · Nature Communications · doi:10.1038/s41467-021-23704-w · Cited by 86
Neural systems analysis of decision making during goal-directed navigation
Penner, M. R., & Mizumori, S. J. Y. (2011) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2011.08.010 · Cited by 95
“…2009, 2010; Ludvig et al., 2008; Maia, 2009; Montague et al., 1996; Nakahara et al., 2004; O'Doherty et al., 2003; Pan et al., 2005, 2008; Schultz et al., 1997; Seymour et al., 2004) such that reward predictions are constantly improved by comparing them to actual rewards (Sutton and Barto, 1998).”
BackgroundEffects of Monetary Incentives on Task Switching
Kleinsorge, T., & Rinkenauer, G. (2012) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1027/1618-3169/a000146 · Cited by 84
“A large body of experimental work has shown that reward-motivated behavior depends on the activity of dopamine (DA) neurons from the midbrain ventral tegmental area and substantia nigra pars compacta (e.g., O’Doherty, Dayan, Friston, Critchley, Dolan, 2003; Schultz, 1998; Seymour et al., 2004).”
BackgroundInstrumental vigour in punishment and reward
Dayan, P. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08026.x · Cited by 72
“…between blood oxygen level-dependent signals in structures such as the ventral striatum that are targets of neuromodulatory systems, and prediction error signals (McClure et al., 2003; O’Doherty et al., 2003; Haruno et al., 2004; Seymour et al., 2004; Tom et al., 2007; Delgado et al., 2008).”
BackgroundSelf-citationEcstatic Epileptic Seizures: A Glimpse into the Multiple Roles of the Insula
Gschwind, M., & Picard, F. (2016) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2016.00021 · Cited by 85
Rowe, J. B., Hughes, L., & Nimmo-Smith, I. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.02.045 · Cited by 74
“In other tasks, where such differential rewards are known to subjects (Glimcher and Rustichini, 2004; Seymour et al., 2007 ; Seymour et al., 2004; Yang and Shadlen, 2007) or are likely (Daw et al., 2006; Dickhaut et al., 2003; Lee et al., 2005), then neuroeconomic theory and reward predictions can…”
BackgroundDopamine, prediction error and associative learning: A model-based account
Smith, A., Li, M., Becker, S., & Kapur, S. (2006) · Network: Computation in Neural Systems · doi:10.1080/09548980500361624 · Cited by 73
“…implicate ventral striatal activity not just in learning about rewards (Pennartz 1996; Schultz 1998; Wilkinson et al. 1998; Wise 2004), but also in the anticipation of appetitive and aversive outcomes (Schultz et al. 2000; Jensen et al. 2003; O’Doherty et al. 2003, 2004; Seymour et al. 2004).”
BackgroundValues and Actions in Aversion
Dayan, P., & Seymour, B. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374176-9.00012-9 · Cited by 78
“Slightly clearer is the representation of the cached aversive values themselves, which evidently involves the amygdala and anterior insula cortex (Paton et al., 2006; Seymour et al., 2004).”
BackgroundResultInfluentialSelf-citationComputational models of reinforcement learning: the role of dopamine as a reward signal
Samson, R. D., Frank, M. J., & Fellous, J. (2010) · Cognitive Neurodynamics · doi:10.1007/s11571-010-9109-x · Cited by 83
A common neural scale for the subjective pleasantness of different primary rewards
Grabenhorst, F., D’Souza, A., Parris, B. A., Rolls, E. T., & Passingham, R. E. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.03.043 · Cited by 118
“, 2004) and respond to both pleasant and unpleasant salient stimuli (Jensen et al., 2007; Seymour et al., 2004).”
Tracking Plasticity: Effects of Long-Term Rehearsal in Expert Dancers Encoding Music to Movement
Bar, R. J., & DeSouza, J. F. X. (2016) · PLoS ONE · doi:10.1371/journal.pone.0147731 · Cited by 77
Mattfeld, A. T., Gluck, M. A., & Stark, C. E. L. (2011) · Learning & Memory · doi:10.1101/lm.022889.111 · Cited by 69
“…Breiter et al. 2001; Yacubian et al. 2006; Liu et al. 2007; Tom et al. 2007), Pavlovian conditioning (Büchel et al. 1998; Jensen et al. 2003, 2007; Seymour et al. 2004, 2005, 2007), “oddball” tasks (Tricomi et al. 2004), reversal learning (Klein et al. 2007; Wheeler and Fellows 2008; Kahnt et…”
BackgroundResultAversive Prediction Error Signals in the Amygdala
McHugh, S. B., Barkus, C., Huber, A., Capitao, L., Lima, J., Lowry, J. P., & Bannerman, D. M. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.4465-13.2014 · Cited by 68
Online learning of task-driven object-based visual attention control
Borji, A., Ahmadabadi, M. N., Araabi, B. N., & Hamidi‐Haines, M. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2009.10.006 · Cited by 75
Apicella, P., Deffains, M., Ravel, S., & Legallet, E. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06872.x · Cited by 66
“…prediction in striatal and prefrontal cortical areas, which are the primary target structures of DA neurons (McClure et al., 2003; Ullsperger & von Cramon, 2003; O’Doherty et al., 2003; Seymour et al., 2004; Abler et al., 2006; Rodriguez et al., 2006; Tobler et al., 2006; Tanaka et al., 2006).”
BackgroundDifferentiable Neural Substrates for Learned and Described Value and Risk
FitzGerald, T. H. B., Seymour, B., Bach, D. R., & Dolan, R. J. (2010) · Current Biology · doi:10.1016/j.cub.2010.08.048 · Cited by 66
“Neuroscientific studies of human decision making tend to situate themselves conceptually within oneof two frameworks: learning theory (most commonly reinforcement learning [1, 19]) andbehavioral economics (most often in the shape of prospect theory [12, 27–29]).”
Self-citationRight Hemisphere Brain Damage Impairs Strategy Updating
Danckert, J. A., Stöttinger, E., Quehl, N., & Anderson, B. (2011) · Cerebral Cortex · doi:10.1093/cercor/bhr351 · Cited by 71
“fMRI studies have shown activation in the insula for a range of tasks from interoceptive perceptions of pain and heartbeat, to time perception, maternal affection, musical perception, perceptual decision making, and risk processing (Brooks et al. 2002; Critchley et al. 2004; Leibenluft et al. 2004; Seymour et al. 2004…”
BackgroundLearning from delayed feedback: neural responses in temporal credit assignment
Walsh, M. M., & Anderson, J. R. (2011) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-011-0027-0 · Cited by 75
“These results are also consistent with a recent fMRI study of higher-order aversive conditioning in humans (Seymour et al., 2004).”
Rapid learning of predictive maps with STDP and theta phase precession
George, T. M., de Cothi, W., Stachenfeld, K., & Barry, C. (2023) · eLife · doi:10.7554/elife.80663 · Cited by 80
Adaptive artificial limbs: a real-time approach to prediction and anticipation
Pilarski, P. M., Dawson, M. R. W., Degris, T., Carey, J. P., Chan, K. M., Hebert, J. S., & Sutton, R. S. (2013) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2012.2229948 · Cited by 76
“Since their development in the 1980s, RL algorithms have been widely used in robotics and have found the best-known approximate solutions to many games; they have also become the standard model of reward processing in the brain [15].”
BackgroundPolicy Adjustment in a Dynamic Economic Game
Li, J., McClure, S. M., King-Casas, B., & Read Montague, P. (2006) · PLoS ONE · doi:10.1371/journal.pone.0000103 · Cited by 63
“that reward associated with each choice depends not only on the current choice but also subject's previous choice history, while most of the previous studies adopted fixed action-reward contingencies paradigms”
ResultTheoretical and Empirical Studies of Learning
Niv, Y., & Montague, P. R. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374176-9.00022-1 · Cited by 75
“…been suggested as the counterpart to dopamine in the domain of punishment ( Daw et al ., 2002 ), and might explain why BOLD correlates of positive prediction errors for pain and punishment have also been found in the striatum ( Seymour et al ., 2004 ; Jensen et al ., 2007 ; Menon et al ., 2007 ).”
BackgroundOpponent Brain Systems for Reward and Punishment Learning
Palminteri, S., & Pessiglione, M. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-805308-9.00023-3 · Cited by 81
The role of executive function in shaping reinforcement learning
Rmus, M., McDougle, S. D., & Collins, A. G. E. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.10.003 · Cited by 78
The influence of feedback valence in associative learning
BISCHOFFGRETHE, A., HAZELTINE, E., BERGREN, L., IVRY, R., & GRAFTON, S. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.08.038 · Cited by 62
“However, growing evidence suggests that salience may also invoke a response (Cooper and Knutson, 2008; Tricomi et al., 2004; Zink et al., 2004), such that negative events, if motivationally salient, will also induce a response in the same region (Seymour et al., 2004).”
InfluentialModular deep reinforcement learning from reward and punishment for robot navigation
Wang, J., Elfwing, S., & Uchibe, E. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.12.001 · Cited by 68
Love-related changes in the brain: a resting-state functional magnetic resonance imaging study
Song, H., Zou, Z., Kou, J., Liu, Y., Yang, L., Zilverstand, A., d’Oleire Uquillas, F., & Zhang, X. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00071 · Cited by 62
Hierarchical models of pain: Inference, information-seeking, and adaptive control.
Seymour, B., & Mancini, F. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117212 · Cited by 64
Self-citationChoi, Y., Shin, E. Y., & Kim, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2003963117 · Cited by 65
Contingency learning in human fear conditioning involves the ventral striatum
Klucken, T., Tabbert, K., Schweckendiek, J., Merz, C. J., Kagerer, S., Vaitl, D., & Stark, R. (2009) · Human Brain Mapping · doi:10.1002/hbm.20791 · Cited by 59
“Yet, there are several studies confirming the involvement of the VS in prediction errors for aversive outcomes also [McNally and Westbrook, 2006; Niv and Schoenbaum, 2008; Seymour et al., 2004].”
BackgroundPrzyrembel, M., Smallwood, J., Pauen, M., & Singer, T. (2012) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2012.00190 · Cited by 76
“In line with substantial evidence for predictive coding in the human brain (e.g., Schultz et al., 1997; Seymour et al., 2004; O’Doherty et al., 2006; Frith, 2007), it is more likely that, for example, activation in anterior insula when empathizing with the pain of others rather reflects predictive…”
BackgroundNeural substrates of appetitive and aversive prediction error
Iordanova, M. D., Yau, J. O.-Y., McDannald, M. A., & Corbit, L. H. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.10.029 · Cited by 66
“, 2005), nucleus accumbens (Delgado et al., 2008; Li et al., 2011; Schiller et al., 2008; Seymour et al., 2004) and the anterior subdivision of the insular cortex (Fazeli and Büchel, 2018).”
BackgroundFormal Learning Theory Dissociates Brain Regions with Different Temporal Integration
Gläscher, J. P., & Büchel, C. (2005) · Neuron · doi:10.1016/j.neuron.2005.06.008 · Cited by 66
“This short-term inte- 1997; Seymour et al., 2004).”
InfluentialAn imaging genetics approach to understanding social influence
Falk, E. B., Way, B. M., & Jasinska, A. J. (2012) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2012.00168 · Cited by 58
“Similarly, the VS has also been found to be activated for anticipation of aversive outcomes (Jensen et al., 2003; Zink et al., 2003, 2004, 2006; Seymour et al., 2004; Menon et al., 2007) and physical pain (Becerra et al., 2001; Baliki et al., 2010).”
BackgroundSuppression of Striatal Prediction Errors by the Prefrontal Cortex in Placebo Hypoalgesia
Schenk, L. A., Sprenger, C., Onat, S., Colloca, L., & Büchel, C. (2017) · The Journal of Neuroscience · doi:10.1523/jneurosci.1101-17.2017 · Cited by 58
“…prediction errors in the VS have initially been associated with appetitive or reward learning (O’Doherty et al., 2003); however, there is substantial evidence that the VS also shows prediction error signals during aversive or pain learning (Seymour et al., 2004, 2005, 2007; Delgado et al., 2008b).”
BackgroundResultBayesian Learning Models of Pain: A Call to Action
Tabor, A., & Burr, C. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2018.10.006 · Cited by 57
Striatal outcome processing in healthy aging
Cox, K. M., Aizenstein, H. J., & Fiez, J. A. (2008) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.8.3.304 · Cited by 60
“…signaling outcome valence. esting target for future aging investigations; recent fMRI studies have documented aversive prediction error signals in the striatum, in addition to the traditional reward prediction error signals (e.g., Seymour et al., 2004; Seymour, Daw, Dayan, Singer, & Dolan, 2007).”
ResultGoal-Directed Decision Making with Spiking Neurons
Friedrich, J., & Lengyel, M. (2016) · The Journal of Neuroscience · doi:10.1523/jneurosci.2854-15.2016 · Cited by 57
“In particular, temporal-difference algorithms of model-free learning account for both behavioral and neuroimaging data regarding habitbased decision making (O’Doherty et al., 2004; Seymour et al., 2004).”
MethodPain-related nucleus accumbens function: modulation by reward and sleep disruption
Seminowicz, D. A., Remeniuk, B., Krimmel, S. R., Smith, M. T., Barrett, F. S., Wulff, A. B., Furman, A. J., Geuter, S., Lindquist, M. A., Irwin, M. R., et al. (2019) · Pain · doi:10.1097/j.pain.0000000000001498 · Cited by 65
“It adds to the mixed body of work that has investigated NAc function specifically during pain, supporting prior work showing increased NAc activation at thermal pain onset(2,5) and increased activation in posterior striatal areas during aversive learning.(43,44) Two other studies have reported decreased NAc activation…”
Météreau, É., & Dreher, J. (2014) · Cortex · doi:10.1016/j.cortex.2014.08.012 · Cited by 64
“…Menon et al., 2007; Metereau & Dreher, 2013; McClure et al., 2003; O'Doherty, Dayan, Friston, Critchley, & Dolan, 2003; O'Doherty et al., 2004; Seymour et al., 2004; Tobler, O'doherty, Dolan, & Schultz, 2006) or on outcome-related activity (Francis et al., 1999; O'Doherty, Rolls, Francis,…”
InfluentialLaufer, O., & Paz, R. (2012) · Journal of Neuroscience · doi:10.1523/jneurosci.6281-11.2012 · Cited by 60
“Recent imaging, lesion, and electrophysiological studies have identified potential neural networks that are associated with assigning subjective value and making choices, and hence might underlie such biases (Bechara et al., 1999; Breiter et al., 2001; Seymour et al., 2004; De Martino et al., 2006, 2010; Yacubian et a…”
BackgroundShih, Y.-W., Tsai, H.-Y., Lin, F.-S., Lin, Y., Chiang, C.-Y., Lu, Z., & Tseng, M. (2018) · Journal of Neuroscience · doi:10.1523/jneurosci.2154-18.2018 · Cited by 63
“Consistent with previous research (Seymour et al., 2004, 2005), we found that the aIC involved in positive expectation-induced pain modulation represented the positive prediction error of pain.”
BackgroundMethodResultGhrelin Enhances Food Odor Conditioning in Healthy Humans: An fMRI Study
Han, J. E., Frasnelli, J., Zeighami, Y., Larcher, K. M.-H., Boyle, J., McConnell, T., Malik, S., Jones‐Gotman, M., & Dagher, A. (2018) · Cell Reports · doi:10.1016/j.celrep.2018.11.026 · Cited by 60
Kosson, D. S., Budhani, S., Nakic, M., Chen, G., Saad, Z. S., Vythilingam, M., Pine, D. S., & Blair, J. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2005.07.060 · Cited by 66
An Imperfect Dopaminergic Error Signal Can Drive Temporal-Difference Learning
Potjans, W., Diesmann, M., & Morrison, A. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001133 · Cited by 61
“Imaging experiments have found that the activity in the ventral striatum is correlated with the TD error during a prediction and action selection task, whereas the activity in the dorsal striatum is correlated with the TD error only during the action selection task [10,86].”
Neural Mechanisms of Gain–Loss Asymmetry in Temporal Discounting
Tanaka, S., Yamada, K., Yoneda, H., & Ohtake, F. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.5169-12.2014 · Cited by 57
“…in tasks with both gains and losses in primary and monetary outcomes, a correlation between striatal activity and negative outcomes has been observed when participants are forced to receive negative outcomes while the potential to receive positive outcomes is present (Seymour et al., 2004, 2007b).”
BackgroundSkvortsova, V., Degos, B., Welter, M., Vidailhet, M., & Pessiglione, M. (2017) · Journal of Neuroscience · doi:10.1523/jneurosci.2081-16.2017 · Cited by 54
“…et al., 2006; Rutledge et al., 2010; Chowdhury et al., 2013), whereas correlates of punishment prediction and prediction errors have been observed in the anterior insula and dorsal anterior cingulate (Seymour et al., 2004; Nitschke et al., 2006; Palminteri et al., 2012; Harrison et al., 2016).”
BackgroundValue generalization in human avoidance learning
Norbury, A., Robbins, T. W., & Seymour, B. (2018) · eLife · doi:10.7554/elife.34779 · Cited by 65
“…(more ventrally) have previously been implicated in representing expected value and prediction error signals in higher-order pain conditioning (Seymour et al., 2004), and the dorsal striatum is implicated in prediction error signals in avoidance learning (Palminteri et al., 2012; Seymour et…”
BackgroundSelf-citationComputational mechanisms underlying the dynamics of physical and cognitive fatigue
Matthews, J., Pisauro, M. A., Jurgelis, M., Müller, T., Vassena, E., Chong, T. T. ‐., & Apps, M. A. J. (2023) · Cognition · doi:10.1016/j.cognition.2023.105603 · Cited by 75
Insights from the application of computational neuroimaging to social neuroscience
Dunne, S., & O’Doherty, J. P. (2013) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2013.02.007 · Cited by 54
“The computational approach to neuroimaging was brought to bear initially on the domains of value-based learning and decision-making in nonsocial situations, as a means of gaining insight into the computations performed by different brain regions in learning to predict rewarding and punishing outcomes and in using thos…”
Self-citationDecoding the Formation of Reward Predictions across Learning
Kahnt, T., Heinzle, J., Park, S. Q., & Haynes, J.-D. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.3412-11.2011 · Cited by 52
“Because reaction time (RT) latencies to conditioned stimuli have been shown to correlate with the expected values derived from RL models (Critchley et al., 2002; Gottfried et al., 2003; Seymour et al., 2004; O’Doherty et al., 2006), the free parameter in the RL model (the learning rate ) was estimated for all subjects…”
BackgroundMethodAnnual Research Review: Developmental computational psychiatry
Hauser, T. U., Will, G., Dubois, M., & Dolan, R. J. (2018) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12964 · Cited by 57
Self-citationPatterned mechanical feedback establishes a global myosin gradient
Gustafson, H. J., Claussen, N. H., Renzis, S. D., & Streichan, S. J. (2022) · Nature Communications · doi:10.1038/s41467-022-34518-9 · Cited by 59
Neurocomputational mechanisms of reinforcement-guided learning in humans: A review
Cohen, M. X. (2008) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.8.2.113 · Cited by 56
“…and/or putamen) (Abler et al., 2006; Knutson & Wimmer, 2007; McClure et al., 2003; O'Doherty et al., 2004; Rolls et al., 2007; Seymour et al., 2007; Seymour et al., 2004), dorsal striatum (Delgado, Miller, Inati, & Phelps, 2005; Filoteo et al., 2005; Haruno & Kawato, 2006), regions of the…”
BackgroundInfluentialShao, R., Read, J., Behrens, T. E. J., & Rogers, R. D. (2013) · Translational Psychiatry · doi:10.1038/tp.2012.134 · Cited by 50
Competition between learned reward and error outcome predictions in anterior cingulate cortex
Alexander, W. H., & Brown, J. W. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.065 · Cited by 50
Value of freedom to choose encoded by the human brain
Fujiwara, J., Usui, N., Park, S. Q., Williams, T., Iijima, T., Taira, M., Tsutsui, K., & Tobler, P. N. (2013) · Journal of Neurophysiology · doi:10.1152/jn.01057.2012 · Cited by 62
“This interpretation is in agreement with the finding that striatal activation underpins higherorder pain conditioning (Seymour et al. 2004), but direct experiments are necessary to test it.”
ResultKawamichi, H., Yoshihara, K., Sasaki, A. T., Sugawara, S. K., Tanabe, H. C., Shinohara, R., Sugisawa, Y., Tokutake, K., Mochizuki, Y., Anme, T., et al. (2014) · Social Neuroscience · doi:10.1080/17470919.2014.954732 · Cited by 66
“By utilizing the learning mechanisms in the anterior insula (Seymour et al., 2004), humans might alter emotional appraisals based on present contextual information, such as a current positive feeling aroused by receiving an active listening attitude.”
BackgroundNees, F., & Becker, S. (2017) · Neuroscience · doi:10.1016/j.neuroscience.2017.08.051 · Cited by 55
Insula and Striatum Mediate the Default Bias
Yu, R., Mobbs, D., Seymour, B., & Calder, A. J. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.3772-10.2010 · Cited by 52
“By contrast, staying with the default engaged the same area as winning (ventral striatum) and might be rewarding in itself (Seymour et al., 2004).”
BackgroundSelf-citationAn MEG signature corresponding to an axiomatic model of reward prediction error
Talmi, D., Fuentemilla, L., Litvak, V., Duzel, E., & Dolan, R. J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.06.051 · Cited by 46
“These midbrain neurons project heavily to the striatum (Haber and Knutson, 2010), and in humans a striatal signal seen in functional magnetic resonance imaging (fMRI) data has been interpreted in terms of a prediction error signal as captured in computational models (O'Doherty et al., 2003; Seymour et al., 2004).”
BackgroundSelf-citationOettl, L., Scheller, M. F., Filosa, C., Wieland, S., Haag, F., Loeb, C., Durstewitz, D., Shusterman, R., Russo, E., & Kelsch, W. (2020) · Nature Communications · doi:10.1038/s41467-020-17257-7 · Cited by 50
Network mechanisms of intentional learning
Hampshire, A., Hellyer, P. J., Parkin, B., Hiebert, N., MacDonald, P., Owen, A. M., Leech, R., & Rowe, J. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2015.11.060 · Cited by 43
“…focused regions of interest (ROI) and psychophysiological interaction (PPI) analyses to test the hypothesis that striatocortical connections are engaged when the stimulus – response rules are being established based on feedback (O'Doherty et al., 2003; Schonberg et al., 2007; Seymour et al., 2004).”
Waltz, J. A., Xu, Z., Brown, E. C., Ruiz, R., Frank, M. J., & Gold, J. M. (2017) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2017.07.008 · Cited by 50
Approach–withdrawal and the role of the striatum in the temperament of behavioral inhibition.
Helfinstein, S. M., Fox, N. A., & Pine, D. S. (2012) · Developmental Psychology · doi:10.1037/a0026402 · Cited by 44
Klein-Flügge, M. C., Wittmann, M. K., Shpektor, A., Jensen, D. E. A., & Rushworth, M. F. S. (2019) · Nature Communications · doi:10.1038/s41467-019-12557-z · Cited by 49
Experience and Choice Shape Expected Aversive Outcomes
Sharot, T., Shiner, T., & Dolan, R. J. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.4770-09.2010 · Cited by 43
“In keeping with this idea, the putamen has previously been shown to track expected aversive consequences (Seymour et al., 2004, 2005) and, in our case, reflected participants’ later choice of the lesser of two evils.”
ResultSelf-citationEfficient statistics, common currencies and the problem of reward-harvesting
Montague, P. R., & King‐Casas, B. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.10.002 · Cited by 52
AI-Empowered Virtual Network Embedding: A Comprehensive Survey
Wu, S., Chen, N., Xiao, A., Zhang, P., Jiang, C., & Zhang, W. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3424533 · Cited by 56
“Representative meth-ods include Q-learning [124], Deep Q-Network (DQN) [125], State-Action-Reward-State-Action (SARSA) [126], Temporal Difference (TD) [127], etc .”
Evidence for model-based encoding of Pavlovian contingencies in the human brain
Pauli, W. M., Gentile, G., Collette, S., Tyszka, J. M., & O’Doherty, J. P. (2019) · Nature Communications · doi:10.1038/s41467-019-08922-7 · Cited by 49
Self-citationEncoding of reward expectation by monkey anterior insular neurons
Mizuhiki, T., Richmond, B. J., & Shidara, M. (2012) · Journal of Neurophysiology · doi:10.1152/jn.00282.2011 · Cited by 41
Path dependence in risky choice: Affective and deliberative processes in brain and behavior
Hytönen, K., Baltussen, G., van den Assem, M. J., Klucharev, V. A., Sanfey, A. G., & Smidts, A. (2014) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2014.01.016 · Cited by 46
“Both of these brain areas have also been implicated in the contexts of negative monetary outcomes in uncertain situations (Kuhnen and Knutson, 2005; Yacubian et al., 2006) and receiving worse outcomes than expected (Seymour et al., 2004).”
Brown, V. M., Price, R. B., & Dombrovski, A. Y. (2023) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-023-01080-w · Cited by 69
“However, computational theories of temporal difference reinforcement learning (Sutton & Barto, 1987, 1998) are able to capture similar processes fully within a learning framework (Maia, 2010; Moutoussis et al., 2008; Seymour et al., 2004).”
BackgroundRole of delay-based reward in the spatial cooperation
Wang, X., Nie, S., Jiang, L.-L., Wang, B.-H., & Chen, S. (2016) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2016.08.014 · Cited by 42
Metadata
- Authors
- Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S.
- Affiliations
- Wellcome Centre for Human Neuroimaging (GB)
- Gatsby Computational Neuroscience Unit (GB)
- University College London (GB)
- University of Manchester (GB)
- Corresponding author
- Seymour, Ben
- ORCID
- Venue
- Nature
- Published
- 2004-06
- Volume
- 429
- Issue
- 6992
- Pages
- 664-667
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2084912121
- Cited by
- 605
- References
- 60
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 8 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 22 · 2019: 25 · 2018: 24 · 2017: 21 · 2016: 24 · 2015: 31 · 2014: 32 · 2013: 47 · 2012: 37
- FWCI
- 15.70
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- other-oa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Max Planck Digital Library (submittedVersion, other-oa)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- temporal difference learning, higher-order conditioning, classical conditioning, aversive conditioning, pain prediction, ventral striatum, anterior insula, sequential learning, computational models of learning, reward learning
- MeSH terms
- Conditioning, Classical, Cues, Electric Stimulation, Hand, Humans, Learning, Magnetic Resonance Imaging, Models, Neurological, Pain, Punishment, Time Factors, Neostriatum
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.