Temporal difference models describe higher-order learning in humans
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
The ability to use environmental stimuli to predict impending harm is critical for survival. Such predictions should be available as early as they are reliable. In pavlovian conditioning, chains of successively earlier predictors are studied in terms of higher-order relationships, and have inspired computational theories such as temporal difference learning. However, there is at present no adequate neurobiological account of how this learning occurs. Here, in a functional magnetic resonance imaging (fMRI) study of higher-order aversive conditioning, we describe a key computational strategy that humans use to learn predictions about pain. We show that neural activity in the ventral striatum and the anterior insula displays a marked correspondence to the signals for sequential learning predicted by temporal difference models. This result reveals a flexible aversive learning process ideally suited to the changing and uncertain nature of real-world environments. Taken with existing data on reward learning, our results suggest a critical role for the ventral striatum in integrating complex appetitive and aversive predictions to coordinate behaviour.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One study specifically showed that AIC encodes both prediction of pain and pain prediction error in the same task (Seymour et al., 2004), which directly supports the functional integration and predictive encoding account of AIC.”
From Anterior insular cortex and emotional awareness (2013) · cites this paper as Seymour et al., 2004
In the citing paper
“AIC responses encode both predicted pain and pain prediction error within a single task [33]”
From Interoceptive inference, emotion, and the embodied self (2013) · cites this paper as [33]
In the citing paper
“This phasic signal represents the difference between the predicted and the observed outcome, which is used to update the model such that future predictions are more accurate (3).”
From Aberrant Salience, Information Processing, and Dopaminergic Signaling in People at Clinical High Risk for Psychosis (2020) · cites this paper as [3]
In the citing paper
“Additionally, we hypothesized the anterior insular activity to be altered, because this region is crucially involved in RPE processing (Pessiglione et al., 2006; Seymour et al., 2004; Voon et al., 2010; Wittmann et al., 2008)”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
In the citing paper
“It is well known that aIns activity often coincides with activation in the dmPFC (cf. Hauser et al., 2014a, 2014b; Nelson et al., 2010; Seymour et al., 2004).”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 1×
“, which has close links with methods for prediction, and optimal action selection, in engineerin”
MethodReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Self-citationFunctional profiling of the Saccharomyces cerevisiae genome
Giaever, G., Chu, A. M., Ni, L., Connelly, C., Riles, L., Véronneau, S., Dow, S., Lucau-Danila, A., Anderson, K., André, B., et al. (2002) · Nature · doi:10.1038/nature00935 · Cited by 4,316
Systematic functional analysis of the Caenorhabditis elegans genome using RNAi
Kamath, R. S., Fraser, A. G., Dong, Y., Poulin, G., Durbin, R., Gotta, M., Kanapin, A., Le Bot, N., Moreno, S., Sohrmann, M., et al. (2003) · Nature · doi:10.1038/nature01278 · Cited by 3,616
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Self-citationAzmitia, E. C., & Segal, M. (1978) · Journal of Comparative Neurology · doi:10.1002/cne.901790311 · Cited by 1,909 · cited 1×
“) suggestion that dopamine neurons in the ventral tegmental area and in the substantia nigra report appetitive prediction error, it has been suggested that serotonin-producing neurons of the dorsal raphe nucleus, which send strong projections to the ventral striatu”
BackgroundAn opponent-process theory of motivation. I. Temporal dynamics of affect
Solomon, R. L., & Corbit, J. D. (1974) · Psychological Review · doi:10.1037/h0036128 · Cited by 1,704
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522
Self-citationToward a modern theory of adaptive networks: Expectation and prediction.
Sutton, R. S., & Barto, A. G. (1981) · Psychological Review · doi:10.1037/0033-295x.88.2.135 · Cited by 1,450 · cited 1×
“These latter studies led to a computational account called temporal difference learnin”
BackgroundAnalysis of optimality in natural and perturbed metabolic networks
Segrè, D., Vitkup, D., & Church, G. M. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.232349399 · Cited by 1,362
Essential Bacillus subtilis genes
Kobayashi, K., Ehrlich, S. D., Albertini, A., Amati, G., Andersen, K. K., Arnaud, M., Asai, K., Ashikaga, S., Aymerich, S., Bessieres, P., et al. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0730515100 · Cited by 1,328
MIPS: a database for genomes and protein sequences
Mewes, H. W. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.37 · Cited by 1,272
Models of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
Dissociating pain from its anticipation in the human brain
Ploghaus, A., Tracey, I., Gati, J. S., Clare, S., Menon, R. S., Matthews, P. M., & Rawlins, J. N. P. (1999) · Science · doi:10.1126/science.284.5422.1979 · Cited by 1,151
Genome-Scale Reconstruction of the Saccharomyces cerevisiae Metabolic Network
Förster, J., Famili, I., Fu, P., Palsson, B. Ø., & Nielsen, J. (2003) · Genome Research · doi:10.1101/gr.234503 · Cited by 1,069
Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events
Horvitz, J. C. (2000) · Neuroscience · doi:10.1016/s0306-4522(00)00019-1 · Cited by 967 · cited 2×
“52); right anterior insula (ins; (46, 22, 24), z 1⁄4 3.”
Contemporary Animal Learning Theory.
Abrahamson, D. (1981) · Europe PMC (PubMed Central) · Cited by 945
Global Transposon Mutagenesis and a Minimal Mycoplasma Genome
Hutchison, C. A., Peterson, S. N., Gill, S. R., Cline, R. T., White, O., Fraser, C. M., Smith, H. O., & Craig Venter, J. (1999) · Science · doi:10.1126/science.286.5447.2165 · Cited by 932
Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum
McClure, S. M., Berns, G. S., & Montague, P. R. (2003) · Neuron · doi:10.1016/s0896-6273(03)00154-5 · Cited by 917
Role of duplicate genes in genetic robustness against null mutations
Gu, Z., Steinmetz, L. M., Gu, X., Scharfe, C., Davis, R. W., & Li, W.-H. (2003) · Nature · doi:10.1038/nature01198 · Cited by 889
Contemporary Animal Learning Theory
LoLordo, V. M., & Dickinson, A. (1982) · The American Journal of Psychology · doi:10.2307/1422669 · Cited by 888 · cited 1×
“Most of this work has concentrated on its simplest realization, namely first-order pavlovian fear conditioning; however, the predictions in this paradigm are rudimentary, showing little of the complexities associated with sequences of predictors that are critical in psychological investigations of prognosticatio”
BackgroundEdwards, J. S., & Palsson, B. O. (2000) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.97.10.5528 · Cited by 875
Opponent interactions between serotonin and dopamine
Daw, N. D., Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00052-7 · Cited by 844 · cited 1×
“In appetitive conditioning the dopamine projection to the ventral striatum is believed to be a critical substrate for this signal, although apart from theoretical speculations about opponent processin”
BackgroundSelf-citationPreferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli
Mirenowicz, J., & Schultz, W. (1996) · Nature · doi:10.1038/379449a0 · Cited by 829 · cited 1×
“There is an active debate about the involvement of dopamine in aversive conditionin”
BackgroundOptimized EPI for fMRI studies of the orbitofrontal cortex
Deichmann, R., Gottfried, J. A., Hutton, C., & Turner, R. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00073-9 · Cited by 821 · cited 1×
“5 tesla Siemens Sonata magnetic resonance scanner (imaging parameters: 280 volumes; 2 mm slice thickness; 1 mm inter-slice gap; tilted plane acquisition sequenc”
BackgroundMetabolic network structure determines key aspects of functionality and regulation
Stelling, J., Klamt, S., Bettenbrock, K., Schuster, S., & Gilles, E. D. (2002) · Nature · doi:10.1038/nature01166 · Cited by 802
Fear and the brain: where have we been, and where are we going?
LeDoux, J. (1998) · Biological Psychiatry · doi:10.1016/s0006-3223(98)00282-0 · Cited by 782
Dosage sensitivity and the evolution of gene families in yeast
Papp, B., Pál, C., & Hurst, L. D. (2003) · Nature · doi:10.1038/nature01771 · Cited by 753
Experimental Determination and System Level Analysis of Essential Genes in Escherichia coli MG1655
Gerdes, S. Y., Scholle, M. D., Campbell, J. W., Balázsi, G., Ravasz, E., Daugherty, M. D., Somera, A. L., Kyrpides, N. C., Anderson, I., Gelfand, M. S., et al. (2003) · Journal of Bacteriology · doi:10.1128/jb.185.19.5673-5684.2003 · Cited by 725
Associative processes in addiction and reward. The role of amygdala-ventral striatal subsystems
EVERITT, B. J., PARKINSON, J. A., OLMSTEAD, M. C., ARROYO, M., ROBLEDO, P., & ROBBINS, T. W. (1999) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1999.tb09280.x · Cited by 708
The role of the basal ganglia in nociception and pain
Chudler, E. H., & Dong, W. K. (1995) · Pain · doi:10.1016/0304-3959(94)00172-b · Cited by 599 · cited 1×
“Our findings add to the existing pharmacological, electrophysiological, functional imaging and clinical evidence for the involvement of the striatum in aversive processing and learnin”
ResultA POPULATION GENETIC THEORY OF CANALIZATION
Wagner, G. P., Booth, G., & Bagheri‐Chaichian, H. (1997) · Evolution · doi:10.1111/j.1558-5646.1997.tb02420.x · Cited by 560
Systematic screen for human disease genes in yeast
Steinmetz, L. M., Scharfe, C., Deutschbauer, A. M., Mokranjac, D., Herman, Z. S., Jones, T., Chu, A. M., Giaever, G., Prokisch, H., Oefner, P. J., et al. (2002) · Nature Genetics · doi:10.1038/ng929 · Cited by 544
Learning and Computational Neuroscience: Foundations of Adaptive Networks
Gabriel, M., & Moore, J. (1990) · Cited by 518
An Opponent-Process Theory of Motivation
Solomon, R. L., & Corbit, J. D. (2008) · Cited by 461
Robustness against mutations in genetic networks of yeast
Wagner, A. (2000) · Nature Genetics · doi:10.1038/74174 · Cited by 411
Büchel, C., Dolan, R. J., Armony, J. L., & Friston, K. J. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-24-10869.1999 · Cited by 405
Self-citationNeural circuitry underlying pain modulation: expectation, hypnosis, placebo
Ploghaus, A., Becerra, L., Borras, C., & Borsook, D. (2003) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(03)00061-5 · Cited by 384 · cited 1×
“Our findings have important implications for our understanding of human pain, given that substantial evidence indicates that the experience of pain is modified by prior conditionin”
BackgroundMetabolic pathways in the post-genome era
Papin, J. A., Price, N. D., Wiback, S. J., Fell, D. A., & Palsson, B. O. (2003) · Trends in Biochemical Sciences · doi:10.1016/s0968-0004(03)00064-1 · Cited by 364
Famili, I., Förster, J., Nielsen, J., & Palsson, B. O. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2235812100 · Cited by 353
Classical fear conditioning in functional neuroimaging
Büchel, C. (2000) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(00)00078-7 · Cited by 348
Mechanisms of Learning and Motivation
(2014) · Psychology Press eBooks · doi:10.4324/9781315802435 · Cited by 301
Daran-Lapujade, P., Jansen, M. L. A., Daran, J.-M., van Gulik, W., de Winde, J. H., & Pronk, J. T. (2004) · Journal of Biological Chemistry · doi:10.1074/jbc.m309578200 · Cited by 274
Value-dependent selection in the brain: Simulation in a synthetic neural model
Friston, K. J., Tononi, G., Reeke, G. N., Sporns, O., & Edelman, G. M. (1994) · Neuroscience · doi:10.1016/0306-4522(94)90592-4 · Cited by 268
Self-citationMetabolic Flux Responses to Pyruvate Kinase Knockout in Escherichia coli
Emmerling, M., Dauner, M., Ponti, A., Fiaux, J., Hochuli, M., Szyperski, T., Wüthrich, K., Bailey, J. E., & Sauer, U. (2002) · Journal of Bacteriology · doi:10.1128/jb.184.1.152-164.2002 · Cited by 267
Learning about pain: The neural substrate of the prediction error for aversive events
Ploghaus, A., Tracey, I., Clare, S., Gati, J. S., Rawlins, J. N. P., & Matthews, P. M. (2000) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.160266497 · Cited by 248 · cited 1×
“93); right head of caudate (caud; (18, 20, 6), z 1⁄4 3.”
Horvitz, J. C. (2002) · Behavioural Brain Research · doi:10.1016/s0166-4328(02)00285-1 · Cited by 202 · cited 1×
“There is an active debate about the involvement of dopamine in aversive conditionin”
BackgroundSchilling, C. H., Edwards, J. S., Letscher, D., & Palsson, B. Ø. (2000) · Biotechnology and Bioengineering · doi:10.1002/1097-0290(2000)71:4<286::aid-bit1018>3.3.co;2-i · Cited by 178
Marginal fitness contributions of nonessential genes in yeast
Thatcher, J. W., Shaw, J. M., & Dickinson, W. J. (1998) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.95.1.253 · Cited by 177
Yeast genome evolution in the post-genome era
Seoighe, C., & Wolfe, K. H. (1999) · Current Opinion in Microbiology · doi:10.1016/s1369-5274(99)00015-6 · Cited by 175
Temporal Difference Model Reproduces Anticipatory Neural Activity
Suri, R. E., & Schultz, W. (2001) · Neural Computation · doi:10.1162/089976601300014376 · Cited by 166 · cited 1×
“89); and anterior cingulate cortex (acc; (8, 12, 32), z 1⁄4 3.”
Rate of evolution and gene dispensability
Pál, C., Papp, B., & Hurst, L. D. (2003) · Nature · doi:10.1038/421496b · Cited by 159
Large-Scale Evaluation of In Silico Gene Deletions in Saccharomyces cerevisiae
Förster, J., Famili, I., Palsson, B. Ø., & Nielsen, J. (2003) · OMICS: A Journal of Integrative Biology · doi:10.1089/153623103322246584 · Cited by 140
Evolution of duplicate genes versus genetic robustness against null mutations
Gu, X. (2003) · Trends in Genetics · doi:10.1016/s0168-9525(03)00139-2 · Cited by 114
Stimulus representation in SOP: I
Brandon, S. E., Vogel, E. H., & Wagner, A. R. (2003) · Behavioural Processes · doi:10.1016/s0376-6357(03)00016-0 · Cited by 98
Kitami, T., & Nadeau, J. H. (2002) · Nature Genetics · doi:10.1038/ng945 · Cited by 77
Genome-scale Gene Expression Analysis and Pathway Reconstruction in KEGG.
Nakao, M., Bono, H. U., Kawashima, S., Kamiya, T., Sato, K., Goto, S., & Kanehisa, M. I. (1999) · PubMed · Cited by 63
Research works printed but not linked: 1
- Barto, A. G., Sutton, R. S. & Watkins, C. J. C. H. in Learning and Computational Neuroscience: Foundations of Adaptive Networks (eds Gabriel, M. & Moor, J.) 539–602 (MIT, Cambridge, Massachusetts, 1990)
Cited by
How do you feel — now? The anterior insula and human awareness
(Bud) Craig, A. D. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2555 · Cited by 6,657
Separate neural systems value immediate and delayed monetary rewards
McClure, S. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2004) · Science · doi:10.1126/science.1100907 · Cited by 3,089
POSNER, J., RUSSELL, J. A., & PETERSON, B. S. (2005) · Development and Psychopathology · doi:10.1017/s0954579405050340 · Cited by 2,602
The integration of negative affect, pain and cognitive control in the cingulate cortex
Shackman, A. J., Salomons, T. V., Slagter, H. A., Fox, A. S., Winter, J. J., & Davidson, R. J. (2011) · Nature Reviews Neuroscience · doi:10.1038/nrn2994 · Cited by 2,082
Uncertainty and anticipation in anxiety: an integrated neurobiological and psychological perspective
Grupe, D. W., & Nitschke, J. B. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3524 · Cited by 2,007
Metadata
- Authors
- Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S.
- Affiliations
- Wellcome Centre for Human Neuroimaging (GB)
- Gatsby Computational Neuroscience Unit (GB)
- University College London (GB)
- University of Manchester (GB)
- Corresponding author
- Seymour, Ben
- ORCID
- Venue
- Nature
- Published
- 2004-06
- Volume
- 429
- Issue
- 6992
- Pages
- 664-667
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2084912121
- Cited by
- 605
- References
- 60
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 8 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 22 · 2019: 25 · 2018: 24 · 2017: 21 · 2016: 24 · 2015: 31 · 2014: 32 · 2013: 47 · 2012: 37
- FWCI
- 15.70
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- other-oa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Max Planck Digital Library (submittedVersion, other-oa)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- temporal difference learning, higher-order conditioning, classical conditioning, aversive conditioning, pain prediction, ventral striatum, anterior insula, sequential learning, computational models of learning, reward learning
- MeSH terms
- Conditioning, Classical, Cues, Electric Stimulation, Hand, Humans, Learning, Magnetic Resonance Imaging, Models, Neurological, Pain, Punishment, Time Factors, Neostriatum
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.