Temporal difference models describe higher-order learning in humans
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
The ability to use environmental stimuli to predict impending harm is critical for survival. Such predictions should be available as early as they are reliable. In pavlovian conditioning, chains of successively earlier predictors are studied in terms of higher-order relationships, and have inspired computational theories such as temporal difference learning. However, there is at present no adequate neurobiological account of how this learning occurs. Here, in a functional magnetic resonance imaging (fMRI) study of higher-order aversive conditioning, we describe a key computational strategy that humans use to learn predictions about pain. We show that neural activity in the ventral striatum and the anterior insula displays a marked correspondence to the signals for sequential learning predicted by temporal difference models. This result reveals a flexible aversive learning process ideally suited to the changing and uncertain nature of real-world environments. Taken with existing data on reward learning, our results suggest a critical role for the ventral striatum in integrating complex appetitive and aversive predictions to coordinate behaviour.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One study specifically showed that AIC encodes both prediction of pain and pain prediction error in the same task (Seymour et al., 2004), which directly supports the functional integration and predictive encoding account of AIC.”
From Anterior insular cortex and emotional awareness (2013) · cites this paper as Seymour et al., 2004
In the citing paper
“AIC responses encode both predicted pain and pain prediction error within a single task [33]”
From Interoceptive inference, emotion, and the embodied self (2013) · cites this paper as [33]
In the citing paper
“This phasic signal represents the difference between the predicted and the observed outcome, which is used to update the model such that future predictions are more accurate (3).”
From Aberrant Salience, Information Processing, and Dopaminergic Signaling in People at Clinical High Risk for Psychosis (2020) · cites this paper as [3]
In the citing paper
“Additionally, we hypothesized the anterior insular activity to be altered, because this region is crucially involved in RPE processing (Pessiglione et al., 2006; Seymour et al., 2004; Voon et al., 2010; Wittmann et al., 2008)”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
In the citing paper
“It is well known that aIns activity often coincides with activation in the dmPFC (cf. Hauser et al., 2014a, 2014b; Nelson et al., 2010; Seymour et al., 2004).”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 1×
“, which has close links with methods for prediction, and optimal action selection, in engineerin”
MethodReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Self-citationFunctional profiling of the Saccharomyces cerevisiae genome
Giaever, G., Chu, A. M., Ni, L., Connelly, C., Riles, L., Véronneau, S., Dow, S., Lucau-Danila, A., Anderson, K., André, B., et al. (2002) · Nature · doi:10.1038/nature00935 · Cited by 4,316
Systematic functional analysis of the Caenorhabditis elegans genome using RNAi
Kamath, R. S., Fraser, A. G., Dong, Y., Poulin, G., Durbin, R., Gotta, M., Kanapin, A., Le Bot, N., Moreno, S., Sohrmann, M., et al. (2003) · Nature · doi:10.1038/nature01278 · Cited by 3,616
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Self-citationAzmitia, E. C., & Segal, M. (1978) · Journal of Comparative Neurology · doi:10.1002/cne.901790311 · Cited by 1,909 · cited 1×
“) suggestion that dopamine neurons in the ventral tegmental area and in the substantia nigra report appetitive prediction error, it has been suggested that serotonin-producing neurons of the dorsal raphe nucleus, which send strong projections to the ventral striatu”
BackgroundAn opponent-process theory of motivation. I. Temporal dynamics of affect
Solomon, R. L., & Corbit, J. D. (1974) · Psychological Review · doi:10.1037/h0036128 · Cited by 1,704
Research works printed but not linked: 1
- Barto, A. G., Sutton, R. S. & Watkins, C. J. C. H. in Learning and Computational Neuroscience: Foundations of Adaptive Networks (eds Gabriel, M. & Moor, J.) 539–602 (MIT, Cambridge, Massachusetts, 1990)
Cited by
Items 401–500 of 605
Functions of Learning Rate in Adaptive Reward Learning
Wu, X., Wang, T., Liu, C., Wu, T., Jiang, J., Zhou, D., & Zhou, J. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00592 · Cited by 20
“Examples include subcortical structures such as the striatum (O’Doherty et al., 2003; Seymour et al., 2004; Pessiglione et al., 2006; Glascher et al., 2010; Jocham et al., 2011; Zhu et al., 2012; Eppinger et al., 2013) and nucleus accumbens (Niv et al., 2012).”
BackgroundResultTanaka, S., Yahata, N., Todokoro, A., Kawakubo, Y., Kano, Y., Nishimura, Y., Ishii‐Takahashi, A., Ohtake, F., & Kasai, K. (2018) · Scientific Reports · doi:10.1038/s41598-018-24944-5 · Cited by 14
Stimulus-Dependent Adjustment of Reward Prediction Error in the Midbrain
Takemura, H., Samejima, K., Vogels, R., Sakagami, M., & Okuda, J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028337 · Cited by 18
“Computational studies have described these reward prediction error activities using reinforcement learning models such as the Rescorla–Wagner model and the temporal difference (TD) model [2,7,9,10,11,12,13,14].”
MethodEthical Artificial IntelligencePreprint
Hibbard, B. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.1373 · Cited by 36
“When studying the ability of animals to learn, scientists have found detailed correlations between patterns of neural activity and known learning algorithms designed by computer scientists (Schultz, Dayan and Montague 1997; Brown, Bullock and Grossberg 1999; Seymour et al. 2004).”
BackgroundHypothalamic interaction with reward-related regions during subjective evaluation of foods
Ogawa, A., Osada, T., Tanaka, M., Suda, A., Nakajima, K., Oka, S., Kamagata, K., Aoki, S., Oshima, Y., Tanaka, S., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119744 · Cited by 16
Safety behaviors toward innocuous stimuli can maintain or increase threat beliefs
van Dis, E. A. M., Krypotos, A., Zondervan-Zwijnenburg, M. A. J., Tinga, A. M., & Engelhard, I. M. (2022) · Behaviour Research and Therapy · doi:10.1016/j.brat.2022.104142 · Cited by 14
Unsigned value prediction-error modulates the motor system in absence of choice
Vassena, E., Cobbaert, S., Andres, M., Fias, W., & Verguts, T. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.07.081 · Cited by 16
“Discrepancies between predicted and actual rewards lead to what is called value predictionerror, which drives decision-making as well as learning (Den Ouden et al., 2009; O’Doherty, 2004; Schultz et al., 1997; Seymour et al., 2004; Silvetti et al., 2014; Sutton and Barto, 1998).”
Understanding decision neuroscience: A multidisciplinary perspective and neural substrates
Miyapuram, K. P., & Pammi, V. S. C. (2013) · Progress in brain research · doi:10.1016/b978-0-444-62604-2.00014-9 · Cited by 19
Single Medial Prefrontal Neurons Cope with Error
Michelet, T., Bioulac, B. H., Guehl, D., Goillandeau, M., & Burbaud, P. (2009) · PLoS ONE · doi:10.1371/journal.pone.0006240 · Cited by 19
“This new finding of a neuronal adjustment mechanism is consistent with other studies [30,31,32,33,34,35] utilizing different experimental paradigms, suggesting that the ACC provides an error compensation system that serves to modify a subject’s behavioral response following an unfavorable and unexpected outcome.”
ResultHelmchen, C., Mohr, C. H., Roehl, M., Bingel, U., Lorenz, J., & Büchel, C. (2007) · Human Brain Mapping · doi:10.1002/hbm.20447 · Cited by 18
“, 2004a] and might be used for pain related motor responses or learning predictions about pain [Seymour et al., 2004].”
BackgroundEich, T. S., & Smith, E. E. (2011) · Oxford University Press eBooks · doi:10.1093/obo/9780199828340-0015 · Cited by 15
Back to the future: Progressing memory research in eating disorders
Forester, G., Johnson, J. S., Reilly, E. E., Lloyd, E. C., Johnson, E., & Schaefer, L. M. (2023) · International Journal of Eating Disorders · doi:10.1002/eat.24045 · Cited by 14
Experience and Abstract Reasoning in Learning Backward Induction
Hawes, D. R., Vostroknutov, A., & Rustichini, A. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00023 · Cited by 17
“This activation is consistent with the Insula’s involvement in processing negative affect, and it’s role in signaling negative prediction errors (Seymour et al., 2004)”
ResultAssessing the role of the amygdala in fear of pain: Neural activation under threat of shock
Sambuco, N., Costa, V. D., Lang, P. J., & Bradley, M. M. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.07.110 · Cited by 18
“In 2015, Mesulam et al., 1982, Ploghaus et al., 1999, Ploner et al., 2010, Preuschoff et al., 2008, Seymour et al., 2004, Weinstein, 1990 Declaration of Competing Interest None Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication.”
BackgroundInfluentialReward, Context, and Human Behaviour
Blaukopf, C. L., & DiGirolamo, G. J. (2007) · The Scientific World JOURNAL · doi:10.1100/tsw.2007.122 · Cited by 20
“In humans, activity is consistent with reward prediction errors for appetitive events in the ventral striatum[19] and also the orbitofrontal cortex[20], and for aversive events in the ventral striatum[21].”
BackgroundCreativity: An Organizational Schema
Caselli, R. J. (2009) · Cognitive and Behavioral Neurology · doi:10.1097/wnn.0b013e318192cce0 · Cited by 18
“Stimuli that we learn to associate with an outcome may be single and immediately paired (eg, the sound of a dinner bell means dinner is served), or complex and multiple steps removed from the salient outcome(143) but both act through activation of the same reward system.(144,145) Prefrontal cortex is a multimodal regi…”
BackgroundPreuschoff, K., Quartz, S. R., & Bossaerts, P. L. (2008) · Revue d économie politique · doi:10.3917/redp.181.0075 · Cited by 19
Double Prioritized State Recycled Experience Replay
Bu, F., & Chang, D. E. (2020) · IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia) · doi:10.1109/icce-asia49877.2020.9276975 · Cited by 13
Correlates of reward‐predictive value in learning‐related hippocampal neural activity
Okatan, M. (2009) · Hippocampus · doi:10.1002/hipo.20535 · Cited by 15
“…higher-order learning in humans, in the activity of brain areas involved in reward processing, including the ventral tegmental area, substantia nigra, and ventral striatum (McClure et al., 2003, 2004; O’Doherty et al., 2003; Montague et al., 2004, 2006; Seymour et al., 2004; Tanaka et al., 2004).”
InfluentialDolan, R. J., & Seymour, B. (2013) · Max Planck Digital Library · Cited by 13
Self-citationAdaptive coding of pain prediction error in the anterior insula
Hoskin, R., & Talmi, D. (2023) · European Journal of Pain · doi:10.1002/ejp.2093 · Cited by 12
“Computational pain PE correlates with activation in the human ventral striatum and insula (Geuter et al., 2017; Seymour et al., 2004; TABLE 1 Example of the calculation of computational and adaptively-coded PE.”
BackgroundMethodThe effects of mental fatigue on effort allocation: Modeling and estimation.
Wang, Z., Chang, Y., Schmeichel, B. J., & García, A. (2022) · Psychological Review · doi:10.1037/rev0000365 · Cited by 16
Asymmetric representation of aversive prediction errors in Pavlovian threat conditioning
Ojala, K. E., Tzovara, A., Poser, B. A., Lutti, A., & Bach, D. R. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119579 · Cited by 14
Wittmann, B. C., Tan, G. C.-Y., Lisman, J. E., Dolan, R. J., & Düzel, E. (2013) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2013.09.031 · Cited by 11
Self-citationAlcázar‐Córcoles, M. Á., Verdejo‐García, A., & Bouso, J. C. (2008) · PubMed · Cited by 10
The Technology of Mind and a New Social Contract
Hibbard, B. (2008) · Cited by 11
Gu, Y., Hu, X., Pan, W., Yang, C., Wang, L., Li, Y., & Chen, A. (2016) · Scientific Reports · doi:10.1038/srep34032 · Cited by 12
The contribution of striatal pseudo-reward prediction errors to value-based decision-making
Herrero, E. M., Sescousse, G., Cools, R., & Marco‐Pallarés, J. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.02.052 · Cited by 11
“These action values are learned following the same rule than V r : The learning rate ( α Þ was set to 0.5 (Seymour et al., 2004; Wilson and Niv, 2015).”
InfluentialRules for aversive learning and decision-making
Yau, J. O.-Y., & McNally, G. P. (2018) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2018.08.006 · Cited by 16
Vassiliadis, P., Derosière, G., & Duqué, J. (2019) · eNeuro · doi:10.1523/eneuro.0458-18.2019 · Cited by 11
“In the same vein, neuroimaging studies in humans have reported activity related to RPEs in the cerebellum (O’Doherty, 2004; Ramnani et al., 2004; Seymour et al., 2004; Tanaka et al., 2004; Tobler et al., 2006; Garrison et al., 2013), suggesting that this structure is functionally involved in…”
BackgroundSuboptimal learning of tactile-spatial predictions in patients with complex regional pain syndrome
Brown, C. A., Scholtes, I., Shenker, N., & Lee, M. (2019) · Pain · doi:10.1097/j.pain.0000000000001730 · Cited by 13
“There is preliminary evidence that anterior insula cortex (ACC), which codes both tactile and pain-related PEs (Allen et al., 2015; Seymour et al., 2004), exhibits abnormal functional and structural (white-matter) connections in the brains of patients with CRPS (Geha et al.”
BackgroundStrengths of social ties modulate brain computations for third-party punishment
Tang, Z., Qu, C., Hu, Y., Bénistant, J., Moisan, F., Derrington, E. A., & Dreher, J. (2023) · Scientific Reports · doi:10.1038/s41598-023-37286-8 · Cited by 14
Clinical neuroscience and imaging studies of core psychoanalytic constructs
Peterson, B. S. (2005) · Clinical Neuroscience Research · doi:10.1016/j.cnr.2005.03.011 · Cited by 11
“Activity in the nucleus accumbens has also been shown to change as subjects anticipate and respond to aversive stimuli [19,20], suggesting directly that differential activity within this region may signal differing responses within the mesolimbic system to both positive and negative emotional valences.”
BackgroundEgocentric value maps of the near-body environment
Bufacchi, R. J., Somervail, R., Fitzpatrick, A. M., Murayama, Y., Logothetis, N. K., Caminiti, R., & Iannetti, G. D. (2025) · Nature Neuroscience · doi:10.1038/s41593-025-01958-7 · Cited by 12
Neural representations of aversive value encoding in pain catastrophizers
Brown, C. A., Almarzouki, A. F., Brown, R. J., & Jones, A. K. P. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.09.052 · Cited by 8
“While there has been substantial interest in identifying biomarkers that increase the precision of psychiatric classification and predict outcomes in mental health disorders (Singh and Rose, 2009), more research is needed into biomarkers of cognitive risk factors for physical health conditions.”
Self-citationSpike-based Decision Learning of Nash Equilibria in Two-Player Games
Friedrich, J., & Senn, W. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002691 · Cited by 10
“Classical models in neuroeconomics are based on temporal difference (TD) learning [1], an algorithm to maximize the total expected reward [2] with potential neuronal implementations [3,4].”
BackgroundPrefrontal Neural Activity When Feedback Is Not Relevant to Adjust Performance
Özyurt, J., Rietze, M., & Thiel, C. M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0036509 · Cited by 12
Bratec, S. M., Xie, X., Wang, Y., Schilbach, L., Zimmer, C., Wohlschläger, A. M., Riedl, V., & Sorg, C. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.12.078 · Cited by 9
“…2000), and a number of studies have demonstrated the encoding of aversive prediction errors (aPEs; PEs related to aversive situations) by the ventral striatum during aversive Pavlovian conditioning (Garrison et al., 2013; Menon et al., 2007; Robinson et al., 2013; Seymour et al., 2007, 2004).”
BackgroundPark, J., Lee, S., Sul, S., & Chung, D. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.650042 · Cited by 9
Kimball, M., & Willis, R. (2023) · National Bureau of Economic Research · doi:10.3386/w31707 · Cited by 13
Couto-Ovejero, S., Ye, J., Kind, P. C., Till, S. M., & Watson, T. C. (2023) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2023.1229627 · Cited by 10
Reinstatement of Cortical Outcome Representations during Higher-Order Learning
Luettgau, L., Porcu, E., Tempelmann, C., & Jocham, G. (2021) · Cerebral Cortex · doi:10.1093/cercor/bhab196 · Cited by 9
Patel, A. N., Bartlett, C. E., & Ichim, T. E. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-801888-0.00011-4 · Cited by 11
Vicario-Feliciano, R. M., Murray, E. A., & Averbeck, B. B. (2017) · Behavioral Neuroscience · doi:10.1037/bne0000211 · Cited by 9
A Workflow for Building Computationally Rational Models of Human Behavior
Chandramouli, S., Shi, D., Putkonen, A., Peuter, S. D., Zhang, S., Jokinen, J., Howes, A., & Oulasvirta, A. (2024) · Computational Brain & Behavior · doi:10.1007/s42113-024-00208-6 · Cited by 20
“We note that computational rationality models built with reinforcement learning are distinct from reinforcement learning-based models of human learning such as Pavlovian learning (Rescorla, 1972; Zhang et al., 2020) and error-driven learning (Seymour et al., 2004; Sutton, 1988).”
Neural Correlates of Economic and Moral Decision-Making
Anen, C. (2007) · CaltechTHESIS (California Institute of Technology) · doi:10.7907/w3vs-mw78 · Cited by 7
A computational neural model of goal-directed utterance selection
Klein, M., Kamp, H., Palm, G., & Doya, K. (2010) · Neural Networks · doi:10.1016/j.neunet.2010.01.003 · Cited by 7
“Brain activity correlating to reward prediction was found in the striatum in fMRI studies (Breiter et al., 2001; Haruno et al., 2004; Seymour et al., 2004; Tanaka et al., 2004).”
BackgroundStrigo, I. A., Spadoni, A. D., & Simmons, A. N. (2022) · Frontiers in Pain Research · doi:10.3389/fpain.2022.871961 · Cited by 10
Vlaev, I., Seymour, B., Chater, N., Winston, J. S., Yoshida, W., Wright, N. D., Symmonds, M., & Dolan, R. J. (2012) · Health Psychology · doi:10.1037/a0030372 · Cited by 8
“As in previous experiments (Seymour et al., 2004), shocks with step increases in amplitude were administered to one site (arm or leg), and participants provided a simple visual analogue scale rating of each shock on a scale from 0 (not painful) through 10 (worst imaginable pain).”
Self-citationHow schizophrenia and depression disrupt reward circuitry
Holcomb, H. H., & Rowland, L. M. (2007) · Current Treatment Options in Neurology · doi:10.1007/s11940-007-0021-6 · Cited by 9
Modelle der Substanzabhängigkeit
Martin-Söelch, C. (2010) · Zeitschrift für Neuropsychologie · doi:10.1024/1016-264x/a000015 · Cited by 11
Neurocomputational basis of learning when choices simultaneously affect both oneself and others
Rhoads, S. A., Gan, L., Berluti, K., O’Connell, K., Cutler, J., Lockwood, P. L., & Marsh, A. A. (2025) · Nature Communications · doi:10.1038/s41467-025-64424-9 · Cited by 12
“…parameter estimates from a priori regions of interest that had been previously linked to prediction error coding in self and social contexts 40 – 42 : ventral striatum 8,14 – 16 , subgenual anterior cingulate 21,43 , anterior insula 15 , amygdala 41,44 , and pregenual anterior cingulate 43,45 .”
Hunting for evidence of cognitive planning: Archaeological signatures versus psychological realities
Rogers, N., Killcross, S., & Curnoe, D. (2015) · Journal of Archaeological Science Reports · doi:10.1016/j.jasrep.2015.11.002 · Cited by 6
Takahashi, Y. K., & Schoenbaum, G. M. (2016) · Behavioral Neuroscience · doi:10.1037/bne0000169 · Cited by 7
Mesocorticolimbic Interactions Mediate fMRI-Guided Regulation of Self-Generated Affective States
Carìa, A. (2020) · Brain Sciences · doi:10.3390/brainsci10040223 · Cited by 5
“In addition, reinforcement learning accounts of emotion regulation [23] also postulate that the midbrain would support regulatory processes by mediating dopaminergic prediction error signals resulting from the mismatch between expected and actual emotional state [24,25].”
BackgroundCerebellar Outputs in Non-human Primates: An Anatomical Perspective Using Transsynaptic Tracers
Bostan, A. C., & Strick, P. L. (2012) · Springer eBooks · doi:10.1007/978-94-007-1333-8_25 · Cited by 7
Rossini, L., & Rossini, P. (2006) · Cited by 6
When Optimal Control Meets Neural Network: A Comprehensive Survey
Wang, X., Dou, Y., Yi, X., Zhang, Y., Li, X., Li, B., Peng, H., Wang, L., & Teo, K. L. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10499-6 · Cited by 8
Prével, A., & Krebs, R. M. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.749517 · Cited by 7
Proximal policy optimization with model-based methods
Li, S., Zhang, W., Zhang, H., Zhang, X., & Leng, Y. (2022) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-211935 · Cited by 7
Prediction error in models of adaptive behavior
Navarro, V. M., Dwyer, D. M., & Honey, R. C. (2023) · Current Biology · doi:10.1016/j.cub.2023.08.043 · Cited by 7
Moret, A. (2025) · Philosophical Studies · doi:10.1007/s11098-025-02343-7 · Cited by 25
“For instance, RL algorithms predict blood activity in the ventral putamen in anticipation of pain, playing the role of a punishment signal (Seymour et al., 2004).”
Smith, J. L., Allen, J. W., Fleischer, C. C., & Harper, D. (2022) · Frontiers in Pain Research · doi:10.3389/fpain.2022.966398 · Cited by 6
Time-dependent neural arbitration between cue associative and episodic fear memories
Cortese, A., Ohata, R., Alemany-González, M., Kitagawa, N., Imamizu, H., & Koizumi, A. (2024) · Nature Communications · doi:10.1038/s41467-024-52733-4 · Cited by 7
“Giventhattemporalcon fi gurationssuchascueorder or gap between CS and US (or lack thereof) modulate threat conditioning 52,53 , our paradigm may be more optimal for studying the encoding of sequentialorder during fearful experiences.”
Elemental Causal Learning from Transitions.
Soo, K. W., & Rottman, B. M. (2015) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 5
Coricelli, G., & Rustichini, A. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374620-7.00020-0 · Cited by 6
Neural correlates of opponent processes for financial gains and losses
Erdeniz, B., & Done, J. (2019) · Neurological Sciences and Neurophysiology · doi:10.5152/nsn.2019.10942 · Cited by 4
Wang, X., Wang, X., & Wilkes, D. M. (2019) · Springer eBooks · doi:10.1007/978-981-13-9217-7_15 · Cited by 5
Using signals associated with safety in avoidance learning: computational model of sex differences
Radell, M. L., Beck, K. D., Pang, K. C. H., & Myers, C. E. (2015) · PeerJ · doi:10.7717/peerj.1081 · Cited by 8
“While the actor and critic modules may correspond to the dorsal and ventral striatum, respectively, with prediction error related to dopaminergic (Houk, Adams & Barto, 1995) and possibly serotonergic inputs to the striatum (Moutoussis et al., 2008; Seymour et al., 2004), multiple other regions have been implicated in…”
BackgroundIntracranial electrophysiology of the human orbitofrontal cortex
Adolphs, R., Kawasaki, H., Oya, H., & Howard, M. A. (2006) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780198565741.003.0014 · Cited by 6
Associative Reactivation of Place–Reward Information in the Hippocampal–Ventral Striatal Circuitry
Lansink, C. S., & Pennartz, C. M. A. (2014) · Springer series in computational neuroscience · doi:10.1007/978-1-4939-1969-7_4 · Cited by 4
“pain [ 50 – 53 ].”
Decomposing brain signals involved in value-based decision making
Dreher, J. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374620-7.00006-6 · Cited by 8
Economic probes of mental function and the extraction of computational phenotypes
Kishida, K. T., & Montague, P. R. (2013) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2013.07.009 · Cited by 7
Estimating learning rates in evolution and TDL: Results on a simple grid-world problem
Lucas, S. M. (2010) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/itw.2010.5593332 · Cited by 5
“The true picture is more subtle than this, with some evolutionary algorithms exploiting experience gained during the lifetime of an individual to bias the variation operators [1].”
BackgroundAttentional Modulation of Brain Responses to Primary Appetitive and Aversive Stimuli
Field, B. A., Buck, C. L., McClure, S. M., Nystrom, L. E., Kahneman, D. J., & Cohen, J. D. (2015) · PLoS ONE · doi:10.1371/journal.pone.0130880 · Cited by 6
Model-Based Decision Making in the Human Brain
Hampton, A. N. (2007) · CaltechTHESIS (California Institute of Technology) · doi:10.7907/k9z8-e248 · Cited by 6
Orienting and preference: an enquiry into the mechanisms underlying human decision making
Simion, C. (2005) · CaltechTHESIS (California Institute of Technology) · doi:10.7907/pwan-xk37 · Cited by 7
Shen, Y., Zeng, L., Kang, H., Sun, X., Chen, Q., & Feng, C. (2026) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2026.117990 · Cited by 4
Aberrations in temporal dynamics of cognitive processing induced by Parkinson’s disease and Levodopa
Kiani, M. M., Beni, M. H. H., & Aghajan, H. K. (2023) · Scientific Reports · doi:10.1038/s41598-023-47410-3 · Cited by 4
“The ventral striatum (VS) has been found to play an important role in reward prediction and conditioned behavior 29–31 .”
BackgroundNeurophysiological encoding of aversive prediction errors
Gopalakrishnan, R., Sonneborn, C., Baillet, S., Machado, A. G., Wager, T. D., & Roy, M. (2025) · Pain · doi:10.1097/j.pain.0000000000003712 · Cited by 4
“Neuroimaging studies 10,26,29,39,43,44,55 have identified candidate regions of pain-related PEs, including the periaqueductal gray area (PAG), amygdala, orbitofrontal cortex, insula, and anterior cingulate cortex (ACC).”
What Can Fear and Reward Learning Teach Us About Depression?
Collins, K. A., & Schiller, D. (2013) · Current topics in behavioral neurosciences · doi:10.1007/7854_2012_236 · Cited by 4
Brown, C. A. (2017) · Springer eBooks · doi:10.1007/978-3-319-48046-6_6 · Cited by 6
Data-Driven Grinding Control Using Reinforcement Learning
Guo, L., Wang, H., & Zhang, J. (2019) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc/smartcity/dss.2019.00395 · Cited by 4
“Instead of collecting samples for the whole episode with a Monte Carlo approach, we can alternatively update the policy at each step using the temporal difference (TD) learning [39] approach, ∇ ( ) = ~ , ~ ∇ log (π ( | )) ∗ ( , ) (3) where ( , ) is an approximated state-action value function that replaces the determin…”
MethodThe orchestration of conscious experience by subcortical structures
Ressler, N. (2010) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/j.1469-185x.2009.00102.x · Cited by 5
Singer, H. S., Mink, J. W., Gilbert, D. L., & Jankovic, J. J. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-411573-6.00007-3 · Cited by 5
FAYMONVILLE, M.-E., TEUWIS, S., VERSCHEURE, S., & FONTAINE, R. (2006) · Springer eBooks · doi:10.1007/978-3-540-29805-2_784 · Cited by 4
Developmental Computational PsychiatryPreprint
Hauser, T. U., Will, G., Dubois, M., & Dolan, R. J. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/85prq · Cited by 4
Self-citationSands, L. P., Jiang, A., Jones, R., Trattner, J. D., & Kishida, K. T. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.03.17.533213 · Cited by 7
“…values would be tracked by regions of dorsal and ventral striatum, cingulate cortex, and insula (O’Doherty et al., 2003; McClure et al., 2003; Pessiglione et al., 2006; Palminteri and Pessiglione, 2017; Seymour et al., 2004, 2005, 2007a, 2012; Delgado et al., 2008, 2009; Garrison et al., 2013).”
Addressing altered anticipation as a transdiagnostic target through computational psychiatry
Sepúlveda, P., Aitsahalia, I., Kumar, K., Atkin, T., & Iigaya, K. (2025) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2025.02.014 · Cited by 3
Expectation generation and its effect on subsequent pain and visual perception
Botvinik‐Nezer, R., Geuter, S., Lindquist, M. A., & Wager, T. D. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013053 · Cited by 4
“Predictive processing theories also emphasize coding of PEs (‘predictive coding’ [6,67,90]), a contrast with predictions that provide efficient representations of salient changes in the environment and drive the updating of internal models (learning) [32,82,91].”
Meng, Y., Wang, S., Li, J., Gao, H., Shang, Y., Wang, J., Zhuang, J., & Zhang, D. (2025) · Analytical Chemistry · doi:10.1021/acs.analchem.5c05454 · Cited by 3
Singer, H. S. (2012) · Elsevier eBooks · doi:10.1016/b978-1-4377-0435-8.00070-6 · Cited by 3
Reinforcement Learning as a Context for Integrating AI Research.
Hibbard, B. (2004) · National Conference on Artificial Intelligence · Cited by 3
Westwood, S., & Philiastides, M. G. (2024) · Human Brain Mapping · doi:10.1002/hbm.70072 · Cited by 5
“Activity in the aINS has been directly related to a computational PPE (Kim, Shimojo, and O'Doherty 2006; Seymour et al. 2004; Skvortsova, Palminteri, and Pessiglione 2014), while damage to the amygdala is known to inhibit salience processing of arousing stimuli (e.g., Anderson and Phelps 2001) as…”
Cerebellar Outputs in Non-human Primates: An Anatomical Perspective Using Transsynaptic Tracers
Bostan, A. C., & Strick, P. L. (2021) · Springer eBooks · doi:10.1007/978-3-030-23810-0_25 · Cited by 3
Xu, Y., D’Lauro, C. J., Pyles, J. A., Kass, R. E., & Tarr, M. J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00684 · Cited by 3
“While extensive research (Gehring et al., 1993; Ashby et al., 1998; Holroyd and Coles, 2002; Seymour et al., 2004; Holroyd et al., 2005) indicates that basal ganglia and anterior cingulate cortex are crucial in trial-and-error learning and decision making processes such as that employed in our…”
BackgroundStimulus sampling as an exploration mechanism for fast reinforcement learning
Vladimirskiy, B., Vasilaki, E., Urbanczik, R., & Senn, W. (2009) · Biological Cybernetics · doi:10.1007/s00422-009-0305-x · Cited by 2
“…response to a stimulus (Montague et al. 1996), generally addresses the temporal credit assignment problem for state-action sequences (Sutton and Barto 1981, 1998; Schultz et al. 1997; McClure et al. 2003; Seymour et al. 2004; Doya 2008) without offering a simple neural network implementation.”
BackgroundTHE VON ECONOMO NEURONS: FROM CELLS TO BEHAVIOR
Watson, K. K. (2006) · doi:10.7907/fz94-dw76 · Cited by 3
Timm, D. N. (2013) · DUT Open Scholar (Durban University of Technology) · doi:10.51415/10321/919 · Cited by 8
Free-Lunch Learning: Modeling Spontaneous Recovery of Memory
Stone, J. V., & Jupp, P. E. (2006) · Neural Computation · doi:10.1162/neco.2007.19.1.194 · Cited by 3
“This temporal difference rule has also been used to model learning in humans (Seymour et al., 2004), and (under mild conditions) is equivalent to the standard delta rule (Sutton, 1988).”
Method
Metadata
- Authors
- Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S.
- Affiliations
- Wellcome Centre for Human Neuroimaging (GB)
- Gatsby Computational Neuroscience Unit (GB)
- University College London (GB)
- University of Manchester (GB)
- Corresponding author
- Seymour, Ben
- ORCID
- Venue
- Nature
- Published
- 2004-06
- Volume
- 429
- Issue
- 6992
- Pages
- 664-667
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2084912121
- Cited by
- 605
- References
- 60
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 8 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 22 · 2019: 25 · 2018: 24 · 2017: 21 · 2016: 24 · 2015: 31 · 2014: 32 · 2013: 47 · 2012: 37
- FWCI
- 15.70
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- other-oa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Max Planck Digital Library (submittedVersion, other-oa)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- temporal difference learning, higher-order conditioning, classical conditioning, aversive conditioning, pain prediction, ventral striatum, anterior insula, sequential learning, computational models of learning, reward learning
- MeSH terms
- Conditioning, Classical, Cues, Electric Stimulation, Hand, Humans, Learning, Magnetic Resonance Imaging, Models, Neurological, Pain, Punishment, Time Factors, Neostriatum
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.