Temporal difference models describe higher-order learning in humans
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
The ability to use environmental stimuli to predict impending harm is critical for survival. Such predictions should be available as early as they are reliable. In pavlovian conditioning, chains of successively earlier predictors are studied in terms of higher-order relationships, and have inspired computational theories such as temporal difference learning. However, there is at present no adequate neurobiological account of how this learning occurs. Here, in a functional magnetic resonance imaging (fMRI) study of higher-order aversive conditioning, we describe a key computational strategy that humans use to learn predictions about pain. We show that neural activity in the ventral striatum and the anterior insula displays a marked correspondence to the signals for sequential learning predicted by temporal difference models. This result reveals a flexible aversive learning process ideally suited to the changing and uncertain nature of real-world environments. Taken with existing data on reward learning, our results suggest a critical role for the ventral striatum in integrating complex appetitive and aversive predictions to coordinate behaviour.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One study specifically showed that AIC encodes both prediction of pain and pain prediction error in the same task (Seymour et al., 2004), which directly supports the functional integration and predictive encoding account of AIC.”
From Anterior insular cortex and emotional awareness (2013) · cites this paper as Seymour et al., 2004
In the citing paper
“AIC responses encode both predicted pain and pain prediction error within a single task [33]”
From Interoceptive inference, emotion, and the embodied self (2013) · cites this paper as [33]
In the citing paper
“This phasic signal represents the difference between the predicted and the observed outcome, which is used to update the model such that future predictions are more accurate (3).”
From Aberrant Salience, Information Processing, and Dopaminergic Signaling in People at Clinical High Risk for Psychosis (2020) · cites this paper as [3]
In the citing paper
“Additionally, we hypothesized the anterior insular activity to be altered, because this region is crucially involved in RPE processing (Pessiglione et al., 2006; Seymour et al., 2004; Voon et al., 2010; Wittmann et al., 2008)”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
In the citing paper
“It is well known that aIns activity often coincides with activation in the dmPFC (cf. Hauser et al., 2014a, 2014b; Nelson et al., 2010; Seymour et al., 2004).”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 1×
“, which has close links with methods for prediction, and optimal action selection, in engineerin”
MethodReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Self-citationFunctional profiling of the Saccharomyces cerevisiae genome
Giaever, G., Chu, A. M., Ni, L., Connelly, C., Riles, L., Véronneau, S., Dow, S., Lucau-Danila, A., Anderson, K., André, B., et al. (2002) · Nature · doi:10.1038/nature00935 · Cited by 4,316
Systematic functional analysis of the Caenorhabditis elegans genome using RNAi
Kamath, R. S., Fraser, A. G., Dong, Y., Poulin, G., Durbin, R., Gotta, M., Kanapin, A., Le Bot, N., Moreno, S., Sohrmann, M., et al. (2003) · Nature · doi:10.1038/nature01278 · Cited by 3,616
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Self-citationAzmitia, E. C., & Segal, M. (1978) · Journal of Comparative Neurology · doi:10.1002/cne.901790311 · Cited by 1,909 · cited 1×
“) suggestion that dopamine neurons in the ventral tegmental area and in the substantia nigra report appetitive prediction error, it has been suggested that serotonin-producing neurons of the dorsal raphe nucleus, which send strong projections to the ventral striatu”
BackgroundAn opponent-process theory of motivation. I. Temporal dynamics of affect
Solomon, R. L., & Corbit, J. D. (1974) · Psychological Review · doi:10.1037/h0036128 · Cited by 1,704
Research works printed but not linked: 1
- Barto, A. G., Sutton, R. S. & Watkins, C. J. C. H. in Learning and Computational Neuroscience: Foundations of Adaptive Networks (eds Gabriel, M. & Moor, J.) 539–602 (MIT, Cambridge, Massachusetts, 1990)
Cited by
Items 301–400 of 605
The Nucleus Accumbens Core is Necessary to Scale Fear to Degree of Threat
Ray, M. H., Russ, A., Walker, R. A., & McDannald, M. A. (2020) · Journal of Neuroscience · doi:10.1523/jneurosci.0299-20.2020 · Cited by 47
Learning affective values for faces is expressed in amygdala and fusiform gyrus
Petrovic, P., Kalisch, R., Pessiglione, M., Singer, T., & Dolan, R. J. (2008) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsn002 · Cited by 41
“This class of model is known to be relevant for learning about rewards (O’Doherty et al., 2003, 2006), and punishments (Seymour et al., 2004, 2005).”
Self-citationImaging Models of Valuation During Social Interaction in Humans
Kishida, K. T., & Montague, P. R. (2012) · Biological Psychiatry · doi:10.1016/j.biopsych.2012.02.037 · Cited by 40
Striatal dopamine D1 receptor is essential for contextual fear conditioning
Ikegami, M., Uemura, T., Kishioka, A., Sakimura, K., & Mishina, M. (2014) · Scientific Reports · doi:10.1038/srep03976 · Cited by 44
Laricchiuta, D., Petrosini, L., Piras, F., Cutuli, D., Macci, E., Picerni, E., Chiapponi, C., Caltagirone, C., & Spalletta, G. (2013) · Brain Structure and Function · doi:10.1007/s00429-013-0535-5 · Cited by 42
“As for FA analyses, no significant association between NS or HA scores and bilateral basal structures was found (details are reported in Online Resource Table 1).”
BackgroundBrown, C. A., Matthews, J. C., Fairclough, M., McMahon, A. W., Barnett, E., Al-Kaysi, A., El‐Deredy, W., & Jones, A. K. P. (2015) · Pain · doi:10.1097/j.pain.0000000000000299 · Cited by 42
“The above networks are involvedwith the generation of prediction errors for rewarding and aversive stimuli.(46,53) However, prediction error signals in the nucleus accumbens have been specifically associated with Pavlovian conditioning,”
BackgroundSelf-citationThe prefrontal cortex in the neurology clinic
Fuster, J. M. (2019) · Handbook of clinical neurology · doi:10.1016/b978-0-12-804281-6.00001-x · Cited by 49
Wittmann, B. C., Tan, G. C.-Y., Lisman, J. E., Dolan, R. J., & Düzel, E. (2013) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2013.07.018 · Cited by 41
“In humans, striatal activity has been shown to correlate with aversive predictions (Carter, Macinnes, Huettel, & Adcock, 2009; Delgado, Li, Schiller, & Phelps, 2008; Seymour et al., 2004; Seymour, Daw, Dayan, Singer, & Dolan, 2007).”
Self-citationModular inverse reinforcement learning for visuomotor behavior
Rothkopf, C. A., & Ballard, D. H. (2013) · Biological Cybernetics · doi:10.1007/s00422-013-0562-6 · Cited by 56
“…empirical and theoretical work has shown that visuomotor behavior and visuomotor learning in animals and humans can be explained through mechanisms of rewardmediated learning (e.g., Glimcher 2004; Graybiel et al. 1994; Gold and Shadlen 2007; Daw and Doya 2006; Seymour et al. 2004; Haber 2003).”
Hester, R., Murphy, K., Brown, F. L., & Skilleter, A. J. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.2565-10.2010 · Cited by 37
“Previous studies have identified insular cortex activity during punishment (Sanfey et al., 2003; Ullsperger and von Cramon, 2003; Seymour et al., 2004; Wächter et al., 2009), with the level of activity linked to the magnitude of (Elliott et al.”
BackgroundLewis, A. H., Porcelli, A. J., & Delgado, M. R. (2014) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2014.00179 · Cited by 41
Human aging reduces the neurobehavioral influence of motivation on episodic memory
Geddes, M. R., Mattfeld, A. T., Angeles, C. D. L., Keshavan, A., & Gabrieli, J. D. E. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.12.053 · Cited by 43
How Instructions, Learning, and Expectations Shape Pain and Neurobiological Responses
Atlas, L. Y. (2023) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-101822-122427 · Cited by 42
Sands, L. P., Jiang, A., Liebenow, B., DiMarco, E. K., Laxton, A. W., Tatter, S. B., Montague, P. R., & Kishida, K. T. (2023) · Science Advances · doi:10.1126/sciadv.adi4927 · Cited by 43
Elfwing, S., & Seymour, B. (2017) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2017.8329799 · Cited by 47
“And in studies of Pavlovian conditioning to pain, punishment prediction errors are observed, also in the striatum [14].”
BackgroundSelf-citationLearning from social rewards predicts individual differences in self-reported social ability.
Heerey, E. A. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0031511 · Cited by 36
Damiano, C. R., Cockrell, D. C., Dunlap, K., Hanna, E. K., Miller, S., Bizzell, J., Kovac, M., Turner‐Brown, L., Sideris, J. H., Kinard, J. L., et al. (2015) · Journal of Neurodevelopmental Disorders · doi:10.1186/s11689-015-9107-8 · Cited by 39
“This mismatch is referred to as a prediction error [72-76], and it reflects the integration of information about current and predicted rewards.”
BackgroundFitzGerald, T. H. B., Schwartenbeck, P., & Dolan, R. J. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.4389-13.2014 · Cited by 35
“Outcome signals in the ventral striatum In the accept condition, activity in bilateral ventral striatum correlated positively with rewarded outcomes (Right: [9 9 6], Z 2.82, psvc 0.045; Left: [ 9 9 6], Z 2.95, psvc 0.033), as predicted by previous findings (Seymour et al., 2004).”
ResultSelf-citationPriming for self-esteem influences the monitoring of one’s own performance
Bengtsson, S., Dolan, R. J., & Passingham, R. E. (2010) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsq048 · Cited by 40
“The insula is commonly activated by aversive stimuli, as seen with pain (Seymour et al., 2004) or following monetary loss (O’Doherty et al.”
BackgroundSelf-citationBelief about Nicotine Modulates Subjective Craving and Insula Activity in Deprived Smokers
Gu, X., Lohrenz, T. M., Salas, R., Baldwin, P. R., Soltani, A., Kirk, U., Cinciripini, P. M., & Montague, P. R. (2016) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2016.00126 · Cited by 34
“Using learning and decision-making paradigms, recent studies have shown that anterior insula activation correlates with value prediction and prediction errors (30), risk and risk prediction errors (31), and subjective feelings during decision-making (32).”
BackgroundHabenular connectivity may predict treatment response in depressed psychiatric inpatients
Gosnell, S. N., Curtis, K. N., Velasquez, K. M., Fowler, J. C., Madan, A., Goodman, W. K., & Salas, R. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.08.026 · Cited by 34
How collective representations can change the structure of the brain
Turner, R. (2010) · Max Planck Digital Library · Cited by 33
Beyond arousal: Prediction error related to aversive events promotes episodic memory formation.
Kalbe, F., & Schwabe, L. (2020) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000728 · Cited by 37
“…prediction errors, on the other hand, have mainly been localized in the striatum (Li, Schiller, Schoenbaum, Phelps, & Daw, 2011; Robinson, Frank, Sahakian, & Cools, 2010; Robinson, Overstreet, Charney, Vytal, & Grillon, 2013; Seymour, Daw, Dayan, Singer, & Dolan, 2007; Seymour et al., 2004).”
Dunsmoor, J. E., & LaBar, K. S. (2012) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2012.02.003 · Cited by 38
“Finally, the striatum is strongly implicated in coding rewardrelated PEs, and several human neuoroimaging studies have found that activity in the striatum tracks PE signals computed from reinforcement learning algorithms during reward-related (Daw, Gershman, Seymour, Dayan, & Dolan, 2011; O’Doherty, Dayan, Friston, Cr…”
MethodSong, A., Jung, W. H., Jang, J. H., Kim, E., Shim, G., Park, H. Y., Choi, C.-H., & Kwon, J. S. (2011) · PLoS ONE · doi:10.1371/journal.pone.0022361 · Cited by 32
“The AIC, on the other hand, has been related to subjective evaluations associated with the visceral sensations [3,6,7].”
BackgroundBortolato, M., Frau, R., Godar, S. C., Mosher, L. J., Paba, S., Marrosu, F., & Devoto, P. (2013) · Journal of Neuroendocrinology · doi:10.1111/jne.12066 · Cited by 34
Dissociable roles of left and right temporoparietal junction in strategic competitive interaction
Ogawa, A., & Kameda, T. (2019) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsz082 · Cited by 36
“As many studies have revealed that the ventral striatum (VS) and ventromedial prefrontal cortex (VMPFC) are involved in learning behaviors based on RL (O’Doherty et al., 2003; O’Doherty et al., 2004; Seymour et al., 2004; see also Sutton and Barto, 1998), we examined whether the VS and VMPFC were associated with the p…”
BackgroundWith you or against you: Social orientation dependent learning signals guide actions made for others
Christopoulos, G. I., & King‐Casas, B. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.09.011 · Cited by 36
“neuroim through experience by serially comparing expectations with outcomes (Krugel et al., 2009; Seymour et al., 2004; Sutton and Barto, 1998).”
Striatal action-learning based on dopamine concentration
Morris, G., Schmidt, R., & Bergman, H. (2009) · Experimental Brain Research · doi:10.1007/s00221-009-2060-6 · Cited by 35
“This poses the problem of ‘temporal credit assignment’: what action was crucial to obtain the reward? To address this problem, an extension to the Rescorla–Wagner model was put forth by Sutton (1988), which came to be known as TD learning and has been widely used in modeling behavioral and neural aspects of reward-rel…”
BackgroundMethodInfluentialThe Neurobiology of Cooperation and Altruism
Rilling, J. K. (2011) · Springer eBooks · doi:10.1007/978-1-4419-9520-9_17 · Cited by 37
The neurophysiological basis of reward effects on backward inhibition processes
Zhang, R., Stock, A., & Beste, C. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.05.080 · Cited by 34
“It is shown that the reward effect depends on the activity of dopamine (DA) neurons (O’Doherty et al., 2003; Seymour et al., 2004) which contributes to overcoming response costs as well as enhancing motivation towards effortful actions (Botvinick et al., 2004; Kurniawan et al., 2011, 2010; Nicola,…”
Atlas, L. Y. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2018.12.008 · Cited by 36
Learning Outside the Brain: Integrating Cognitive Science and Systems Biology
Gunawardena, J. (2022) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3162791 · Cited by 39
“Similar results have been found for more complex, higher order learning tasks in other animals, including humans [53].”
ResultPain, from perception to action: A computational perspective
Chen, Z. S., & Wang, J. (2022) · iScience · doi:10.1016/j.isci.2022.105707 · Cited by 48
“60 In contrast, the amygdala and midbrain tended to reflect reward-like signals predicted by RL models, where humans learned to generate expectations of pain relief.”
BackgroundAlves, F. D. S., Schmitz, N., Figee, M., Abeling, N. G. G. M., Hasler, G., van der Meer, J. N., Nederveen, A. J., de Haan, L., LINSZEN, D. H., & van Amelsvoort, T. (2010) · Journal of Psychopharmacology · doi:10.1177/0269881110367731 · Cited by 36
“These studies reported a relationship between prediction error (PE) (i.e. the difference between actual and predicted reward), the firing of dopamine neurons with phasic activity and the BOLD response (McClure et al., 2003; O’Doherty et al., 2006; Seymour et al., 2004).”
BackgroundThe computational psychiatry of reward: broken brains or misguided minds?
Moutoussis, M., Story, G. W., & Dolan, R. J. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01445 · Cited by 30
“…to the computational psychiatry of reward are arguably those that posit the basal ganglia as representing reward-based learning prediction errors (Seymour et al., 2004) and of the ventral and medial prefrontal cortex representing the values of different actions available to the subject (Rushworth…”
BackgroundSelf-citationThe control of tonic pain by active relief learning
Zhang, S., Mano, H., Lee, M., Yoshida, W., Kawato, M., Robbins, T. W., & Seymour, B. (2018) · eLife · doi:10.7554/elife.31949 · Cited by 42
Self-citationJepma, M., Roy, M., Ramlakhan, K., van Velzen, M., & Dahan, A. (2022) · eLife · doi:10.7554/elife.74149 · Cited by 30
“Previous studies have made considerable progress in our understanding of the neural basis of passive cue–pain association learning (Ploghaus et al., 2000; Seymour et al., 2004; Seymour et al., 2005) and—more recently—active pain- avoidance and -relief learning (Roy et al.”
ResultNeural coding of computational factors affecting decision making
Dreher, J. (2013) · Progress in brain research · doi:10.1016/b978-0-444-62604-2.00016-2 · Cited by 29
Tsukiura, T. (2012) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2012.00050 · Cited by 31
“The insular activity has also been linked to the processing of pain (Critchley et al., 2000) and aversive conditioning (Seymour et al., 2004).”
BackgroundCole, S., & McNally, G. P. (2007) · Behavioral Neuroscience · doi:10.1037/0735-7044.121.5.1043 · Cited by 28
“…as well as Jennings and Kirkpatrick (2006) showed that temporal primacy prevented the development of blocking (i.e., Stage I then Stage III), and Seymour et al. (2004), using human subjects, showed a role for temporal primacy in aversive conditioning using a single stage serial compound design,…”
BackgroundLengersdorff, L., Wagner, I. C., Lockwood, P. L., & Lamm, C. (2020) · Journal of Neuroscience · doi:10.1523/jneurosci.0842-20.2020 · Cited by 29
Haufler, D., Liran, O., Buchanan, R. J., & Paré, D. (2022) · Journal of Neurophysiology · doi:10.1152/jn.00106.2022 · Cited by 30
Tomasi, D. G., Wang, G., Studentsova, Y., & Volkow, N. D. (2014) · Cerebral Cortex · doi:10.1093/cercor/bhu269 · Cited by 32
“Activation in the NAc and anterior insula was significantly greater in the TD learning condition than in the outcome condition of the TOP task, pointing to the importance of these regions for learning associated with reward, as has been shown with reinforcement learning (O’Doherty et al. 2003; Seymour et al. 2004; O’D…”
BackgroundResultDopamine-induced changes in neural network patterns supporting aversive conditioning
Diaconescu, A. O., Menon, M., Jensen, J., Kapur, S., & McIntosh, A. R. (2009) · Brain Research · doi:10.1016/j.brainres.2009.11.064 · Cited by 29
Changes in expectation consequent on experience, modeled by a simple, forgetful neural circuit
Anderson, A. J., & Carpenter, R. H. S. (2006) · Journal of Vision · doi:10.1167/6.8.5 · Cited by 30
“Finally, TD models are primarily designed to learn across multiple sequences to predict a certain outcome z (Schultz, Dayan, & Montague, 1997; Seymour et al., 2004; Sutton, 1988; Sutton & Barto, 1981).”
BackgroundInfluentialNear-Optimal Human Adaptive Control across Different Noise Environments
Chhabra, M., & Jacobs, R. A. (2006) · Journal of Neuroscience · doi:10.1523/jneurosci.2238-06.2006 · Cited by 28
“Interestingly, neuroscientists have found that the activities of some neurons are closely related to quantities that appear in reinforcement learning equations (Seymour et al., 2004; Knutson et al., 2005; Lee, 2006).”
Jessup, R. K., & O’Doherty, J. P. (2014) · European Journal of Neuroscience · doi:10.1111/ejn.12625 · Cited by 32
“…(Delgado et al., 2000; Knutson et al., 2001; Kim et al., 2006), others have found increased activity in the striatum in response to aversive events such as aversive outcomes, aversive predictions and aversive prediction errors (Becerra et al., 2001; Jensen et al., 2003; Seymour et al., 2004, 2005).”
BackgroundInfluentialSelf-citationInsular and hippocampal contributions to remembering people with an impression of bad personality
Tsukiura, T., Shigemune, Y., Nouchi, R., Kambara, T., & Kawashima, R. (2012) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nss025 · Cited by 30
“, 2000) and aversive conditioning (Seymour et al., 2004).”
Neural correlates of taste reward value across eating disorders
Olsavsky, A. K., Shott, M. E., DeGuzman, M. C., & Frank, G. (2018) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2018.08.010 · Cited by 28
Two sides of the same coin: Learning via positive and negative reinforcers in the human striatum
Niznikiewicz, M. A., & Delgado, M. R. (2011) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2011.07.006 · Cited by 28
“, 2007) or even shock itself (Seymour et al., 2004).”
Neural basis of decision making guided by emotional outcomes
Katahira, K., Matsuda, Y.-T., Fujimura, T., Ueno, K., Asamizuya, T., Suzuki, C., Cheng, K., Okanoya, K., & Okada, M. (2015) · Journal of Neurophysiology · doi:10.1152/jn.00564.2014 · Cited by 33
“However, several reports suggest that the activities of our regions of interest (ROIs), the insula and the striatum, are correlated with negative valence with a positively signed value (Seymour et al. 2004, 2005, 2007).”
InfluentialTemporal dissociation of salience and prediction error responses to appetitive and aversive taste
Hird, E., El‐Deredy, W., Jones, A. K. P., & Talmi, D. (2017) · Psychophysiology · doi:10.1111/psyp.12976 · Cited by 35
Self-citationBetween- and within-scanner variability in the CaliBrain study n-back cognitive task
Gradin, V. B., Gountouna, V., Waiter, G. D., Ahearn, T. S., Brennan, D., Condon, B. R., Marshall, I., McGonigle, D. J., Murray, A. D., Whalley, H. C., et al. (2010) · Psychiatry Research Neuroimaging · doi:10.1016/j.pscychresns.2010.08.010 · Cited by 27
Psychophysiological correlates of interpersonal cooperation and aggression
Wang, Y., Roberts, K., Yuan, B., Zhang, W., Shen, D., & Simons, R. F. (2013) · Biological Psychology · doi:10.1016/j.biopsycho.2013.04.008 · Cited by 29
“…in particular is associated with activation of the insula, which alongside evidence that the insula is implicated in aversive conditioning, suggests that it may mark negative social interactions in an effort to learn to avoid them in the future (Rilling et al., 2008; Seymour et al., 2004).”
BackgroundModulation of alpha oscillations in insular cortex reflects the threat of painful stimuli
Franciotti, R., Ciancetta, L., Penna, S. D., Belardinelli, P., Pizzella, V., & Romani, G. L. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.034 · Cited by 30
“…electrophysiological investigation may better separate neural processes that take place within a few hundred milliseconds before pain onset from pain experience, although earlier anticipatory processes could be resolved with fMRI (Ploghaus et al., 1999; Porro et al., 2003; Seymour et al., 2004).”
BackgroundMethodNeural Mechanisms for Adaptive Learned Avoidance of Mental Effort
Nagase, A. M., Onoda, K., Foo, J. C., Haji, T., Akaishi, R., Yamaguchi, S., Sakai, K., & Morita, K. (2018) · Journal of Neuroscience · doi:10.1523/jneurosci.1995-17.2018 · Cited by 30
“Whereas RL of reward values (McClure et al., 2003; O’Doherty et al., 2003; Daw et al., 2006), pain (Seymour et al., 2004; Roy et al., 2014; Zhang et al., 2016), physical effort (Skvortsova et al., 2014), or sustained effort (selecting circles on the screen) concurrently with reward-learning (Scholl…”
BackgroundWinkelmeier, L., Filosa, C., Hartig, R., Scheller, M. F., Sack, M., Reinwald, J., Becker, R., Wolf, D., Gerchen, M. F., Sartorius, A., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30978-1 · Cited by 26
Bratec, S. M., Xie, X., Schmid, G., Doll, A., Schilbach, L., Zimmer, C., Wohlschläger, A. M., Riedl, V., & Sorg, C. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.08.038 · Cited by 32
“…and insula is consistent with previous studies, where they have repeatedly been implicated as the neural correlates of aversive prediction errors (Delgado et al., 2008a; Metereau and Dreher, 2013; Robinson et al., 2010; Schiller et al., 2008; Seymour et al., 2004, 2007; Spoormaker et al., 2011).”
Changes in emotional processing following interoceptive network stimulation with rTMS
Mai, S., Braun, J., Probst, V., Kammer, T., & Pollatos, O. (2019) · Neuroscience · doi:10.1016/j.neuroscience.2019.03.014 · Cited by 30
Conceptual representations in goal-directed decision making
Shea, N. J., Krug, K., & Tobler, P. N. (2008) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.8.4.418 · Cited by 29
“The habitbased decision-making system also shows up in striatal activations during appetitive and aversive conditioning in humans (O’Doherty, Dayan, Friston, Critchley, & Dolan, 2003; Seymour et al., 2004).”
BackgroundChen, S., & Li, X. (2012) · Computational and Mathematical Methods in Medicine · doi:10.1155/2012/613465 · Cited by 28
“In [8], by fMRI study of higher-order aversive conditioning, a computational strategy is described that humans use to learn predictions about pain.”
A Novel Form of Memory for Auditory Fear Conditioning at a Low-Intensity Unconditioned Stimulus
Kishioka, A., Fukushima, F., Ito, T., Kataoka, H., Mori, H., Ikeda, T., Itohara, S., Sakimura, K., & Mishina, M. (2009) · PLoS ONE · doi:10.1371/journal.pone.0004157 · Cited by 29
Caruso, P., Moretti, R., Crisman, E., & Gazzin, S. (2017) · Neurology India · doi:10.4103/neuroindia.ni_850_16 · Cited by 25
Imaginative Reinforcement Learning: Computational Principles and Neural Mechanisms
Gershman, S. J., Zhou, J., & Kommers, C. (2017) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01170 · Cited by 26
“All of these models have in common the well-accepted idea that cached values are updated using temporal difference learning (Daw et al., 2011; Gläscher, Daw, Dayan, & O ’ Doherty, 2010; Seymour et al., 2004; Schultz, Dayan, & Montague, 1997).”
BackgroundMethodWhen is a loss a loss? Excitatory and inhibitory processes in loss-related decision-making
Seymour, B., Maruyama, M., & Martino, B. D. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.09.003 · Cited by 27
Self-citationThe role of moral utility in decision making: An interdisciplinary framework
Tobler, P. N., Kalis, A., & Kalenscher, T. (2008) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/cabn.8.4.390 · Cited by 31
“Activation in the ventral striatum and the anterior insula reflects second-order conditioning of pain (Seymour et al., 2004).”
Environmental statistics and the trade-off between model-based and TD learning in humans
Simon, D. A., & Daw, N. D. (2011) · neural information processing systems · Cited by 23
Clinical Motor and Cognitive Neurobehavioral Relationships in the Basal Ganglia
Leisman, G., Melillo, R. J., & Frederick, R. (2013) · InTech eBooks · doi:10.5772/55227 · Cited by 31
“They identified the ventral striatum as a 1 key locus of such sequential learning [90].”
BackgroundBradfield, L. A., & McNally, G. P. (2010) · Learning & Memory · doi:10.1101/lm.1798810 · Cited by 24
Reactivation of Single-Episode Pain Patterns in the Hippocampus and Decision Making
Wimmer, G. E., & Büchel, C. (2021) · Journal of Neuroscience · doi:10.1523/jneurosci.1350-20.2021 · Cited by 31
“In contrast, the gradual learning of stimulus-value associations over multiple experiences is known to involve systems including the dopaminergic midbrain, striatum, insula, and amygdala (Schultz et al., 1997; LeDoux, 2000; Seymour et al., 2004; Schiller et al., 2008).”
BackgroundThe scent of salience — Is there olfactory-trigeminal conditioning in humans?
Moessnang, C., Pauly, K. D., Kellermann, T., Krämer, J., Finkelmeyer, A. E., Hummel, T. R., Siegel, S. J., Schneider, F. W., & Habel, U. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.03.049 · Cited by 22
“, 2012), and has been shown to apply to the AI as well (d’Acremont et al., 2009; Seymour et al., 2004).”
From the Reward Circuit to the Valuation System: How the Brain Motivates Behavior
Pessiglione, M., & Lebreton, M. P. (2014) · Springer eBooks · doi:10.1007/978-1-4939-1236-0_11 · Cited by 26
“There is also evidence for the converse relation in the anterior insula, in which deactivation might represent higher value (Palminteri et al. 2012; Seymour et al. 2004).”
BackgroundPain and the PAG: learning from painful mistakes
Eippert, F., & Tracey, I. (2014) · Nature Neuroscience · doi:10.1038/nn.3844 · Cited by 24
Neural Correlates of Dynamically Evolving Interpersonal Ties Predict Prosocial Behavior
Fahrenfort, J. J., van Winden, F. A. A. M., Pelloux, B., Stallen, M., & Ridderinkhof, K. R. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00028 · Cited by 23
“…striatum has traditionally been implicated in planning and modulation of movement (Rolls, 1994), but recently more prominently in reward processing (Moll et al., 2006; Haber and Knutson, 2010), novelty-based choice behavior (Wittmann et al., 2008), and higher-order learning (Seymour et al., 2004).”
BackgroundGain–loss situation modulates neural responses to self–other decision making under risk
Zhang, X., Li, S., Liu, Y., Chen, X., Shang, X., Qi, F., Wang, X., Guo, X., & Chen, J. (2019) · Scientific Reports · doi:10.1038/s41598-018-37236-9 · Cited by 25
Perceptual bias in pain: A switch looks closer when it will relieve pain than when it won’t
Tabor, A., Catley, M. J., Gandevia, S. C., Thacker, M. A., & Moseley, G. L. (2013) · Pain · doi:10.1016/j.pain.2013.05.024 · Cited by 22
“The mismatch between the expected and the actual outcome is described as prediction error [10,29,33]; a concept not only essential in learning, but also in understanding how our perceptions are not accurate read-outs of the sensory signals we receive.”
Decision-making in brains and robots — the case for an interdisciplinary approach
Lee, S. W., & Seymour, B. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2018.12.012 · Cited by 22
Self-citationCongruence of Inherent and Acquired Values Facilitates Reward-Based Decision-Making
Chien, S., Wiehler, A., Spezio, M., & Gläscher, J. P. (2016) · Journal of Neuroscience · doi:10.1523/jneurosci.3084-15.2016 · Cited by 19
“Our a priori predictions, based on previous work on the RPE signal in the lateral putamen (Seymour et al., 2004), on the EV signal in the vmPFC (Wunderlich et al., 2010; McNamee et al., 2013), and on the reward outcome in the ventromedial striatum (Reuter et al., 2005) allowed us to perform small…”
MethodThe benefits of negative yet informative feedback
Kim, S., Hwang, S.-Y., & Lee, M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0205183 · Cited by 22
van de Ven, V. G., Lee, C., Lifanov, J., Kochs, S., Jansma, H., & Weerd, P. D. (2020) · Hippocampus · doi:10.1002/hipo.23205 · Cited by 22
“, and fits with the more general notion that dorsal striatum monitors the difference between temporal expectancies and experiences, possibly to optimize future action selection (Seymour et al., 2004).”
BackgroundSocial training reconfigures prediction errors to shape Self-Other boundaries
Ereira, S., Hauser, T. U., Moran, R., Story, G. W., Dolan, R. J., & Kurth-Nelson, Z. L. (2020) · Nature Communications · doi:10.1038/s41467-020-16856-8 · Cited by 22
Self-citationFrontostriatal contribution to the interplay of flexibility and stability in serial prediction
Trempler, I., Schiffer, A.-M., El-Sourani, N., Ahlheim, C., Fink, G. R., & Schubotz, R. I. (2017) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01040 · Cited by 18
“Moreover, there is evidence for a basic stimulus selection function of the striatumwhen faced with unpredicted sensory input (e.g., den Ouden, Danizeau, Roiser, Friston, & Stephan, 2010; Corlett et al., 2004; O’Doherty et al., 2004; Seymour et al., 2004).”
BackgroundThe Neural Correlates of Cued Reward Omission
Mollick, J., Chang, L. J., Krishnan, A., Hazy, T. E., Krueger, K. A., Frank, G., Wager, T. D., & O’Reilly, R. C. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.615313 · Cited by 20
“This may reflect a prediction error if the conditioned inhibition procedure caused the inhibitor to acquire negative value, consistent with other studies that have found putamen regions respond to prediction error signals (O’Doherty et al., 2003; Seymour et al., 2004).”
ResultPain Relativity in Motor Control
Kurniawan, I. T., Seymour, B., Vlaev, I., Trommershäuser, J., Dolan, R. J., & Chater, N. (2010) · Psychological Science · doi:10.1177/0956797610370160 · Cited by 21
“Both action choice and action implementation are central themes in theories of optimal control: action choice is formalized, for example, by reinforcement learning theory (Seymour et al., 2004), and action implementation is formalized by theories of motor control.”
BackgroundSelf-citationAn association between prediction errors and risk-seeking: Theory and behavioral evidence
Möller, M., Grohn, J., Manohar, S., & Bogacz, R. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009213 · Cited by 20
The learning of action sequences through social transmission
Whalen, A., Cownden, D., & Laland, K. N. (2015) · Animal Cognition · doi:10.1007/s10071-015-0877-x · Cited by 22
“TD learning has received broad empirical support both at a behavioral level and at a neurological level (Seymour et al. 2004; Sutton and Barto 1990; Glimcher 2011; Dayan and Niv 2008).”
BackgroundParieto-frontal circuitry in visuomotor control
Grol, M. (2008) · Utrecht University Repository (Utrecht University) · Cited by 129
Wittmann, B. C., & D’Esposito, M. (2014) · Psychopharmacology · doi:10.1007/s00213-014-3646-7 · Cited by 19
“2003) and that the striatum is part of a network responding to salience prediction errors (Delgado et al. 2008; Jensen et al. 2007; Metereau and Dreher 2013; Seymour et al. 2004), possibly with an anteroposterior gradient dissociating reward and punishment-based activations (Seymour et al.”
InfluentialSuzuki, S., Zhang, X., Dezfouli, A., Braganza, L., Fulcher, B., Parkes, L., Fontenelle, L. F., Harrison, B. J., Murawski, C., Yücel, M., et al. (2023) · PLoS Biology · doi:10.1371/journal.pbio.3002031 · Cited by 21
“They are encoded in a network of brain regions, including the midbrain [10,11], striatum [12,13], medial prefrontal cortex (mPFC) [14,15], and anterior insula [16,17] (see [18] for a recent meta-analysis).”
Predictive mechanisms linking brain opioids to chronic pain vulnerability and resilience
Jones, A. K. P., & Brown, C. A. (2017) · British Journal of Pharmacology · doi:10.1111/bph.13840 · Cited by 19
“Critically, empirical evidence supports RL models in that both reward and aversive prediction errors are closely related to phasic activation of the ventral striatum consistent with dopaminergic signalling (O’Doherty et al., 2003; Seymour et al., 2004).”
BackgroundSelf-citationPerceptual Decision Parameters and Their Relation to Self-Reported Pain: A Drift Diffusion Account
Zaman, J., Wiech, K., & Vlaeyen, J. W. S. (2019) · Journal of Pain · doi:10.1016/j.jpain.2019.06.009 · Cited by 16
A Computational Role for Top–Down Modulation from Frontal Cortex in Infancy
Jaffe‐Dax, S., Boldin, A. M., Daw, N. D., & Emberson, L. L. (2019) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01497 · Cited by 19
“Computational associative learning models have been extensively used to characterize the functional role of neural responses in various learning tasks in adults (Li, Schiller, Schoenbaum, Phelps, & Daw, 2011; Seymour et al., 2004) and in nonhuman animals (Schultz, Dayan, & Montague, 1997).”
Reinforcement Learning, High-Level Cognition, and the Human Brain
Silvetti, M., & Verguts, T. (2012) · InTech eBooks · doi:10.5772/23471 · Cited by 20
Lee, T., & Shao, R. (2014) · Neuroscience and Neuroeconomics · doi:10.2147/nan.s35914 · Cited by 23
“Additionally, the ventral striatum is involved in predicting and signaling aversive stimuli, 42 whereas the caudate nucleus may be primarily involved in reinforcement-directed action selection.”
Espresso Reward Learning, Hold the Dopamine: Theoretical Comment on Robinson et al. (2005).
Berridge, K. C. (2005) · Behavioral Neuroscience · doi:10.1037/0735-7044.119.1.336 · Cited by 28
“Elegant studies, originally pioneered by Schultz and colleagues and later extended by others, showed that the activation of mesolimbic–striatal neurons often conforms to temporal difference and prediction error models of reward learning (Cromwell & Schultz, 2003; Ljungberg, Apicella, & Schultz, 1992; Roitman, Stuber,…”
BackgroundMethodThe integration of negative affect, pain and cognitive control in the cingulate cortex
Fig., A. (2011) · Cited by 19
Higher-order conditioning: A critical review and computational model.
Honey, R. C., & Dwyer, D. M. (2022) · Psychological Review · doi:10.1037/rev0000368 · Cited by 20
Spoormaker, V. I., Blechert, J., Goya‐Maldonado, R., Sämann, P. G., Wilhelm, F. H., & Czisch, M. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.08.050 · Cited by 18
Gu, X., Kirk, U., Lohrenz, T. M., & Montague, P. R. (2013) · Human Brain Mapping · doi:10.1002/hbm.22433 · Cited by 17
“, 2002], a dopaminoceptive structure that is commonly implicated in decisionmaking tasks and works closely with a network of brain regions such as the anterior insular cortex (AIC), orbitofrontal cortex (OFC), and the amygdala [Hsu et al., 2005; Li et al., 2011; Seymour et al., 2004].”
Background
Metadata
- Authors
- Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S.
- Affiliations
- Wellcome Centre for Human Neuroimaging (GB)
- Gatsby Computational Neuroscience Unit (GB)
- University College London (GB)
- University of Manchester (GB)
- Corresponding author
- Seymour, Ben
- ORCID
- Venue
- Nature
- Published
- 2004-06
- Volume
- 429
- Issue
- 6992
- Pages
- 664-667
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2084912121
- Cited by
- 605
- References
- 60
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 8 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 22 · 2019: 25 · 2018: 24 · 2017: 21 · 2016: 24 · 2015: 31 · 2014: 32 · 2013: 47 · 2012: 37
- FWCI
- 15.70
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- other-oa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Max Planck Digital Library (submittedVersion, other-oa)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- temporal difference learning, higher-order conditioning, classical conditioning, aversive conditioning, pain prediction, ventral striatum, anterior insula, sequential learning, computational models of learning, reward learning
- MeSH terms
- Conditioning, Classical, Cues, Electric Stimulation, Hand, Humans, Learning, Magnetic Resonance Imaging, Models, Neurological, Pain, Punishment, Time Factors, Neostriatum
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.