Temporal difference models describe higher-order learning in humans
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
The ability to use environmental stimuli to predict impending harm is critical for survival. Such predictions should be available as early as they are reliable. In pavlovian conditioning, chains of successively earlier predictors are studied in terms of higher-order relationships, and have inspired computational theories such as temporal difference learning. However, there is at present no adequate neurobiological account of how this learning occurs. Here, in a functional magnetic resonance imaging (fMRI) study of higher-order aversive conditioning, we describe a key computational strategy that humans use to learn predictions about pain. We show that neural activity in the ventral striatum and the anterior insula displays a marked correspondence to the signals for sequential learning predicted by temporal difference models. This result reveals a flexible aversive learning process ideally suited to the changing and uncertain nature of real-world environments. Taken with existing data on reward learning, our results suggest a critical role for the ventral striatum in integrating complex appetitive and aversive predictions to coordinate behaviour.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One study specifically showed that AIC encodes both prediction of pain and pain prediction error in the same task (Seymour et al., 2004), which directly supports the functional integration and predictive encoding account of AIC.”
From Anterior insular cortex and emotional awareness (2013) · cites this paper as Seymour et al., 2004
In the citing paper
“AIC responses encode both predicted pain and pain prediction error within a single task [33]”
From Interoceptive inference, emotion, and the embodied self (2013) · cites this paper as [33]
In the citing paper
“This phasic signal represents the difference between the predicted and the observed outcome, which is used to update the model such that future predictions are more accurate (3).”
From Aberrant Salience, Information Processing, and Dopaminergic Signaling in People at Clinical High Risk for Psychosis (2020) · cites this paper as [3]
In the citing paper
“Additionally, we hypothesized the anterior insular activity to be altered, because this region is crucially involved in RPE processing (Pessiglione et al., 2006; Seymour et al., 2004; Voon et al., 2010; Wittmann et al., 2008)”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
In the citing paper
“It is well known that aIns activity often coincides with activation in the dmPFC (cf. Hauser et al., 2014a, 2014b; Nelson et al., 2010; Seymour et al., 2004).”
From Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development (2015) · cites this paper as Seymour et al., 2004
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · cited 1×
“, which has close links with methods for prediction, and optimal action selection, in engineerin”
MethodReinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Self-citationFunctional profiling of the Saccharomyces cerevisiae genome
Giaever, G., Chu, A. M., Ni, L., Connelly, C., Riles, L., Véronneau, S., Dow, S., Lucau-Danila, A., Anderson, K., André, B., et al. (2002) · Nature · doi:10.1038/nature00935 · Cited by 4,316
Systematic functional analysis of the Caenorhabditis elegans genome using RNAi
Kamath, R. S., Fraser, A. G., Dong, Y., Poulin, G., Durbin, R., Gotta, M., Kanapin, A., Le Bot, N., Moreno, S., Sohrmann, M., et al. (2003) · Nature · doi:10.1038/nature01278 · Cited by 3,616
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066
Self-citationAzmitia, E. C., & Segal, M. (1978) · Journal of Comparative Neurology · doi:10.1002/cne.901790311 · Cited by 1,909 · cited 1×
“) suggestion that dopamine neurons in the ventral tegmental area and in the substantia nigra report appetitive prediction error, it has been suggested that serotonin-producing neurons of the dorsal raphe nucleus, which send strong projections to the ventral striatu”
BackgroundAn opponent-process theory of motivation. I. Temporal dynamics of affect
Solomon, R. L., & Corbit, J. D. (1974) · Psychological Review · doi:10.1037/h0036128 · Cited by 1,704
Research works printed but not linked: 1
- Barto, A. G., Sutton, R. S. & Watkins, C. J. C. H. in Learning and Computational Neuroscience: Foundations of Adaptive Networks (eds Gabriel, M. & Moor, J.) 539–602 (MIT, Cambridge, Massachusetts, 1990)
Cited by
Items 101–200 of 605
How Humans Integrate the Prospects of Pain and Reward during Choice
Talmi, D., Dayan, P., Kiebel, S. J., Frith, C. D., & Dolan, R. J. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2026-09.2009 · Cited by 217
“We assessed how pain and reward are integrated using a reinforcement learning model (Sutton and Barto, 1998), a variant of temporal difference models that have provided a compelling account of a wealth of psychological, electrophysiological, and neuroimaging data (Schultz et al., 1997; Dayan and Balleine, 2002; O’Dohe…”
BackgroundMethodSelf-citationFunctional imaging of the human dopaminergic midbrain
Düzel, E., Bunzeck, N., Guitart-Masip, M., Wittmann, B., Schott, B. H., & Tobler, P. N. (2009) · Trends in Neurosciences · doi:10.1016/j.tins.2009.02.005 · Cited by 216
Guitart-Masip, M., Fuentemilla, L., Bach, D. R., Huys, Q. J. M., Dayan, P., Dolan, R. J., & Duzel, E. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.6376-10.2011 · Cited by 216
“These results are not incompatible with previous studies reporting reward prediction errors for state values under experimental conditions controlling action requirements indirectly through the use of explicit foil actions (Delgado et al., 2000, 2003, 2004; O’Doherty et al., 2003; Seymour et al., 2004; Tricomi et al.,…”
InfluentialSelf-citationPredictability modulates the affective and sensory-discriminative neural processing of pain
Carlsson, K., Andersson, J., Petrovic, P., Petersson, K. M., Öhman, A., & Ingvar, M. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.05.027 · Cited by 208
“A change in sequencing, that is, a violation of expectation, has previously been related to activity in the ventral striatum (Berns et al., 1997; Pagnoni et al., 2002; Schultz, 2002; Seymour et al., 2004; Forkstam and Petersson, 2005).”
BackgroundA universal role of the ventral striatum in reward-based learning: Evidence from human studies
Daniel, R., & Pollmann, S. (2014) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2014.05.002 · Cited by 208
“…2007; Litt, Plassmann, Shiv, & Rangel, 2011; Menon et al., 2007; Metereau & Dreher, 2013; Robinson, Overstreet, Charney, Vytal, & Grillon, 2013; Seymour et al., 2004; Zink, Pagnoni, Chappelow, Martin-Skurski, & Berns, 2006), this approach has to be interpreted with extreme care (Poldrack,…”
BackgroundTsukiura, T., & Cabeza, R. (2011) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsq025 · Cited by 206
“The insula has also been linked to the processing of pain (Critchley et al., 2000), and aversive conditioning (Seymour et al., 2004).”
Brain imaging of pain: state of the art
Morton, D., Jones, A., & Sandhu, J. (2016) · Journal of Pain Research · doi:10.2147/jpr.s60433 · Cited by 199
“76 78 The amygdala, orbitofrontal cortex, and putamen are areas previously associated with aversive conditioning, 79 reward, and fear, and their activation in OA implies that time activation of this fear-related circuitry is perhaps related to fear of additional injury and disability.”
Self-citationCerebellar and Prefrontal Cortex Contributions to Adaptation, Strategies, and Reinforcement Learning
Taylor, J. A., & Ivry, R. B. (2014) · Progress in Brain Research · doi:10.1016/b978-0-444-63356-9.00009-1 · Cited by 205
“While studies of reinforcement learning have focused on the basal ganglia and orbitofrontal cortex, the BOLD response in the cerebellum has also been found to be correlated with reward prediction error (O’Doherty et al., 2003; Seymour et al., 2004).”
Differential Effect of Reward and Punishment on Procedural Learning
Wächter, T., Lungu, O. V., Liu, T., Willingham, D. T., & Ashe, J. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.4132-08.2009 · Cited by 205
Levels of appraisal: a medial prefrontal role in high-level appraisal of emotional material.
Kalisch, R., Wiech, K., Critchley, H. D., & Dolan, R. J. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2005.11.011 · Cited by 206
“For example, MPFC/ACC activity during high-level appraisal may be a result of parallel lowlevel appraisals potentially involving the MPFC/ACC (Seymour et al., 2004).”
BackgroundSelf-citationDelgado, M. R. (2009) · Frontiers in Behavioral Neuroscience · doi:10.3389/neuro.08.033.2009 · Cited by 204
The neurobiology of punishment
Seymour, B., Singer, T., & Dolan, R. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2119 · Cited by 240
Self-citationEncoding of Marginal Utility across Time in the Human Brain
Pine, A., Seymour, B., Roiser, J. P., Bossaerts, P., Friston, K. J., Curran, H. V., & Dolan, R. J. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.1126-09.2009 · Cited by 169
“For instance, the well recognized role of this region in reinforcement learning (Robbins et al., 1989; O’Doherty et al., 2004, Seymour et al., 2004; Haruno and Kawato, 2006) does not necessarily speak to a role in constructing value and choice.”
BackgroundSelf-citationServaas, M. N., van der Velde, J., Costafreda, S. G., Horton, P., Ormel, J., Riese, H., & Aleman, A. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.05.005 · Cited by 197
Resolution of Uncertainty in Prefrontal Cortex
Yoshida, W., & Ishii, S. (2006) · Neuron · doi:10.1016/j.neuron.2006.05.006 · Cited by 197
“…without hidden states is formulated as a Markov decision process (MDP), and reinforcement learning, which is an approach to solving MDPs, may be realized within brain networks primarily including the PFC and the basal ganglia (Daw et al., 2005; O'Doherty et al., 2004; and Seymour et al., 2004).”
BackgroundA functional endophenotype for sexual orientation in humans
Ponseti, J., Bosinski, H. A., Wolff, S., Peller, M., Jansen, O., Mehdorn, H. M., Büchel, C., & Siebner, H. R. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.08.002 · Cited by 197
“This view would be compatible with the observation that neuronal activity in the ventral striatum is modulated by high-order contingencies (Seymour et al., 2004).”
ResultA Neural Signature of Hierarchical Reinforcement Learning
Ribas-Fernandes, J. J. F., Solway, A., Diuk, C., McGuire, J. T., Barto, A. G., Niv, Y., & Botvinick, M. M. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.042 · Cited by 195
“In the NAcc, where some studies have observed deactivation accompanying negative RPEs (Knutson et al., 2005), no signifi-cant PPE effect was observed.”
BackgroundInfluentialActivation in the VTA and nucleus accumbens increases in anticipation of both gains and losses
Carter, R. M. (2009) · Frontiers in Behavioral Neuroscience · doi:10.3389/neuro.08.021.2009 · Cited by 195
“In fact, there is evidence that relief from pain (Seymour et al., 2005) and even avoiding potential negative outcomes (Kim et al., 2006) can be viewed as rewards.”
BackgroundBehavioral and Neural Properties of Social Reinforcement Learning
Jones, R. M., Somerville, L. H., Li, J., Ruberry, E. J., Libby, V., Glover, G., Voss, H. U., Ballon, D. J., & Casey, B. J. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.2972-11.2011 · Cited by 191
“Reaction time has been shown in previous studies to be a reliable indicator of learning contingencies and speeding or slowing in reaction times has been associated with conditioning as predicted by reinforcement learning models (Seymour et al., 2004; Bray and O’Doherty, 2007).”
MethodUbiquity and Specificity of Reinforcement Signals throughout the Human Brain
Vickery, T. J., Chun, M. M., & Lee, D. (2011) · Neuron · doi:10.1016/j.neuron.2011.08.011 · Cited by 189
“In particular, interest has centered on the relationship between basal ganglia activity and errors in prediction of rewards (Glä scher et al., 2010; Schultz et al., 1997; Sutton and Barto, 1998) or punishments (Delgado et al., 2008; Seymour et al., 2004).”
BackgroundPain, decisions, and actions: a motivational perspective
Wiech, K., & Tracey, I. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00046 · Cited by 188
“In a second-order cue learning task, participant were presented with two consecutive visual cues that predicted the application of a high or low-intensity noxious stimulus (Seymour et al., 2004).”
MethodWaltz, J. A., Schweitzer, J. B., Gold, J. M., Kurup, P. K., Ross, T. J., Jo Salmeron, B., Rose, E. J., McClure, S. M., & Stein, E. A. (2009) · Neuropsychopharmacology · doi:10.1038/npp.2008.214 · Cited by 186
Tops, M., & Boksem, M. A. S. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00330 · Cited by 185
Cognition–Emotion Integration in the Anterior Insular Cortex
Gu, X., Liu, X., Van Dam, N. T., Hof, P. R., & Fan, J. (2013) · Cerebral Cortex · doi:10.1093/cercor/bhr367 · Cited by 186
“2007), risk and uncertainty (Seymour et al. 2004), error detection (Klein et al.”
BackgroundBayesian Prediction and Evaluation in the Anterior Cingulate Cortex
Ide, J. S., Shenoy, P., Yu, A. J., & Li, C.-S. R. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.2201-12.2013 · Cited by 185
“…and not rewards, which are important for Bayesian learning or model-based reinforcement learning; it contrasts with reward prediction error, important for model-free reinforcement learning and putatively encoded in the striatal dopaminergic system (O’Doherty et al., 2004; Seymour et al., 2004).”
BackgroundA neurocomputational account of how inflammation enhances sensitivity to punishments versus rewards
Harrison, N. A., Voon, V., Cercignani, M., Cooper, E. A., Pessiglione, M., & Critchley, H. D. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.07.018 · Cited by 185
Appetitive conditioning: Neural bases and implications for psychopathology
Martin-Soelch, C., Linthicum, J., & Ernst, M. (2007) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2006.11.002 · Cited by 185
Functional dissociation of stimulus intensity encoding and predictive coding of pain in the insula
Geuter, S., Boll, S., Eippert, F., & Büchel, C. (2017) · eLife · doi:10.7554/elife.24770 · Cited by 184
Effort and Valuation in the Brain: The Effects of Anticipation and Execution
Kurniawan, I. T., Guitart-Masip, M., Dayan, P., & Dolan, R. J. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.4777-12.2013 · Cited by 177
“Our results are consistent with previous suggestions that the insula robustly represents aversive events such as punishment (Seymour et al., 2004), pain for others (Singer et al.”
ResultSelf-citationSeparate brain regions code for salience vs. valence during reward prediction in humans
Jensen, J., Smith, A. J., Willeit, M., Crawley, A. P., Mikulis, D. J., Vitcu, I., & Kapur, S. (2007) · Human Brain Mapping · doi:10.1002/hbm.20274 · Cited by 178
“In contrast, and within recent fMRI contexts [Jensen et al., 2003; Seymour et al., 2004], unexpected aversive events have been shown to recruit areas traditionally associated with TD-like prediction errors in human fMRI experiments, e.”
Neural Correlates of Forward Planning in a Spatial Decision Task in Humans
Simon, D. A., & Daw, N. D. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.4647-10.2011 · Cited by 177
“…although there has been much work quantitatively investigating TD characterizations of learning (O’Doherty et al., 2003, 2006; Lee et al., 2004; Seymour et al., 2004), much less research has analogously investigated the neural and computational substrates for model-based learning and planning.”
BackgroundResultKim, H., Shimojo, S., & O'Doherty, J. P. (2011) · Cerebral Cortex · doi:10.1093/cercor/bhq145 · Cited by 176
“…Huettel et al. 2006; Preuschoff et al. 2006), in response to disgusting odors (Wicker et al. 2003) and aversive tastes (Small et al. 1999) or faces displaying disgust (Phillips et al. 1997), and during the anticipation and receipt of painful stimulation (Ploghaus et al. 1999; Seymour et al. 2004).”
BackgroundInfluentialSelf-citationThe neural correlates of the affective response to unreciprocated cooperation
Rilling, J. K., Goldsmith, D. R., Glenn, A. L., Jairam, M. R., Elfenbein, H. A., Dagenais, J. E., Murdock, C. D., & Pagnoni, G. (2008) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2007.11.033 · Cited by 172
“and support in agonistic encounters ( Seyfarth & Cheney, 1984; de Waal, 1997), the number of documented cases is limited.”
BackgroundNeural Coding of Reward-Prediction Error Signals During Classical Conditioning With Attractive Faces
Bray, S., & O'Doherty, J. (2007) · Journal of Neurophysiology · doi:10.1152/jn.01211.2006 · Cited by 172
“Reaction times have previously been shown to be modulated as a function of conditioning, and changes in reaction times over time have previously been found to correlate with reinforcement learning models (Critchley et al. 2002; Gottfried et al. 2002, 2003; Seymour et al. 2004).”
BackgroundResultSelf-citationNeural basis of impaired safety signaling in obsessive compulsive disorder
Apergis-Schoute, A. M., Gillan, C. M., Fineberg, N. A., Fernandez-Egea, E., Sahakian, B. J., & Robbins, T. W. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1609194114 · Cited by 168
“Previous work has established important roles for the amygdala-striatal network in aversive learning in humans (2, 20, 21), including for this paradigm (7).”
BackgroundPain: A Precision Signal for Reinforcement Learning and Control
Seymour, B. (2019) · Neuron · doi:10.1016/j.neuron.2019.01.055 · Cited by 167
Self-citationComputational neuroimaging strategies for single patient predictions
Stephan, K. E., Schlagenhauf, F., Huys, Q. J. M., Raman, S., Aponte, E. A., Brodersen, K. H., Rigoux, L., Moran, R. J., Daunizeau, J., Dolan, R. J., et al. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.06.038 · Cited by 166
Self-citationAltered neural reward and loss processing and prediction error signalling in depression
Ubl, B., Kuehner, C., Kirsch, P., Ruttorf, M., Diener, C., & Flor, H. (2015) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsu158 · Cited by 165
“This result is in line with a study of Stoy et al. (2012) who found similar dysfunctions during the anticipation but not during the receipt of reward.”
BackgroundResultBault, N., Joffily, M., Rustichini, A., & Coricelli, G. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1100892108 · Cited by 162
“001 in left caudate and nucleus accumbens (−9, 9, −3); F(1, 23) = 13.”
BackgroundAtlas, L. Y., & Wager, T. D. (2014) · Handbook of Experimental Pharmacology · doi:10.1007/978-3-662-44519-8_3 · Cited by 162
“Indeed, quantitative modeling supports this account (Seymour et al. 2004, 2005), but the relationship between such signals and perceived pain has not been formally tested.”
Modulation of pain processing in hyperalgesia by cognitive demand
Wiech, K., Seymour, B., Kalisch, R., Enno Stephan, K., Koltzenburg, M., Driver, J., & Dolan, R. J. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.03.044 · Cited by 161
“Recent studies have highlighted value-dependent learning activity in the cerebellum (Ploghaus et al., 2000; Seymour et al., 2004), and an interaction between intensity and attention is thus not unexpected in the light of contemporary learning theory (Dayan et al.”
BackgroundSelf-citationLearning to predict and control harmful events: chronic pain and conditioning
Vlaeyen, J. W. S. (2015) · Pain · doi:10.1097/j.pain.0000000000000107 · Cited by 159
“Participants responded with greater potentiated eyeblink startle, heightened skin conductance, and cardiac deceleration in the presence of the CS1, as compared with CS2, in the absence of the US.12 Classical conditioning has also thought to be the underlying mechanism for the increased muscular reactivity often seen i…”
BackgroundInfluentialChase, H. W., Nusslock, R., Almeida, J. R., Forbes, E. E., LaBarbara, E. J., & Phillips, M. L. (2013) · Bipolar Disorders · doi:10.1111/bdi.12132 · Cited by 157
“Activation coupled to reward prediction errors is frequently observed within the VS, elicited both by errors of reward expectancy and delivery (12, 13, 47).”
Behavioural and neural evidence for self-reinforcing expectancy effects on pain
Jepma, M., Koban, L., van Doorn, J., Jones, M., & Wager, T. D. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0455-8 · Cited by 155
The habenula encodes negative motivational value associated with primary punishment in humans
Lawson, R. P., Seymour, B., Loh, E., Lutti, A., Dolan, R. J., Dayan, P., Weiskopf, N., & Roiser, J. P. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1323586111 · Cited by 155
“Furthermore, we found that the habenula is functionally coupled with the striatum, including the medial wall of the caudate, which is strongly innervated by dopamine neurons (15) and has previously been implicated in fMRI studies of Pavlovian aversive learning (13).”
Self-citationSpoormaker, V. I., Schröter, M. S., Andrade, K. C., Dresler, M., Kiem, S. A., Goya‐Maldonado, R., Wetter, T. C., Holsboer, F., Sämann, P. G., & Czisch, M. (2012) · Human Brain Mapping · doi:10.1002/hbm.21369 · Cited by 153
“…conditioning paradigms [Büchel and Dolan, 2000] to study prediction error signaling have also revealed increased activation in dopaminergic midbrain regions [Menon et al., 2007; Seymour et al., 2004] and the striatum and insula [Jensen et al., 2007; Schiller et al., 2008; Seymour et al., 2004].”
BackgroundJimura, K., Chushak, M. S., & Braver, T. S. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.0919-12.2013 · Cited by 153
“…some computational models do not even postulate that value representations are continuously maintained during delay periods (e.g., Brown et al., 1999; O’Reilly, 2006), and/or that value updates occur only in a cue-triggered fashion (O’Doherty et al., 2003; Redish, 2004; Seymour et al., 2004).”
BackgroundDopamine, uncertainty and TD learning
Niv, Y., Duff, M. O., & Dayan, P. (2005) · Behavioral and Brain Functions · doi:10.1186/1744-9081-1-6 · Cited by 153
Self-citationOn the role of general system theory for functional neuroimaging
Stephan, K. E. (2004) · Journal of Anatomy · doi:10.1111/j.0021-8782.2004.00359.x · Cited by 152
“With the availability of fMRI, conventional analyses using a General Linear Model (GLM) are ideal to inform this choice (see below).”
MethodInfluentialNeuroimaging of reward mechanisms in Gambling disorder: an integrative review
Clark, L., Boileau, I., & Zack, M. H. (2018) · Molecular Psychiatry · doi:10.1038/s41380-018-0230-2 · Cited by 177
Menon, M., Jensen, J., Vitcu, I., Graff‐Guerrero, A., Crawley, A. P., Smith, M. A., & Kapur, S. (2007) · Biological Psychiatry · doi:10.1016/j.biopsych.2006.10.020 · Cited by 158
Reinforcement learning-assisted evolutionary algorithm: A survey and research opportunities
Song, Y., Wu, Y., Guo, Y., Yan, R., Suganthan, P. N., Zhang, Y., Pedrycz, W., Das, S., Mallipeddi, R., Ajani, O. S., et al. (2024) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2024.101517 · Cited by 168
Neural systems underlying aversive conditioning in humans with primary and secondary reinforcers
Delgado, M. R., Jou, R. L., & Phelps, E. A. (2011) · Frontiers in Neuroscience · doi:10.3389/fnins.2011.00071 · Cited by 141
“There is evidence of striatum signals correlating with prediction errors when the context is aversive using both primary (Seymour et al., 2004; Delgado et al., 2008) and secondary reinforcers (Seymour et al., 2007).”
BackgroundFrom Bad to Worse: Striatal Coding of the Relative Value of Painful Decisions
Brooks, A. M., Pammi, V. S. C., Noussair, C., Capra, C. M., Engelmann, J. B., & Berns, G. S. (2010) · Frontiers in Neuroscience · doi:10.3389/fnins.2010.00176 · Cited by 136
“Some research suggests that anticipation and experience of aversive stimuli activate the striatum (LaBar et al., 1998; Becerra et al., 2001; Jensen et al., 2003; Seymour et al., 2004).”
BackgroundSpoormaker, V. I., Sturm, A., Andrade, K. C., Schröter, M. S., Goya‐Maldonado, R., Holsboer, F., Wetter, T. C., Sämann, P. G., & Czisch, M. (2010) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2010.04.017 · Cited by 150
“, 1997) and that the caudate is involved in prediction error signaling in aversive conditioning (Schiller et al., 2008; Seymour et al., 2004; but see Menon et al., 2007).”
BackgroundReduced Striatal Responses to Reward Prediction Errors in Older Compared with Younger Adults
Eppinger, B., Schuck, N. W., Nystrom, L. E., & Cohen, J. D. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.2942-12.2013 · Cited by 134
Adolescent-specific patterns of behavior and neural activity during social reinforcement learning
Jones, R. M., Somerville, L. H., Li, J., Ruberry, E. J., Powers, A., Mehta, N., Dyke, J., & Casey, B. J. (2014) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-014-0257-z · Cited by 127
“slowing to cues predicting lesser value has been associated with conditioning as predicted by reinforcement learning models (Bray & O'Doherty, 2007; Seymour et al., 2004).”
InfluentialFunctional neuroimaging of intertemporal choice models: a review
Carter, R. M., Meyer, J. R., & Huettel, S. A. (2010) · Journal of Neuroscience Psychology and Economics · doi:10.1037/a0018046 · Cited by 124
“Specifically, delay discounting is related to recent neuroscience work positing an actor/critic model of decision making (O’Doherty et al., 2004; Paulus, Feinstein, Tapert, & Liu, 2004; Preuschoff, Bossaerts, & Quartz, 2006; Seymour et al., 2004; Sutton & Barto, 1998).”
BackgroundNeurobiology of Value Integration: When Value Impacts Valuation
Park, S. Q., Kahnt, T., Rieskamp, J., & Heekeren, H. R. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.4973-10.2011 · Cited by 125
“This approach is related to some model-based fMRI studies (Breiter et al., 2001; O’Doherty et al., 2004; Seymour et al., 2004; Kim et al., 2006; Talmi et al., 2009) and provides a concrete example of using neural signals with modeling to identify integration computations that cannot be easily identified by behavioral…”
The neural underpinnings of cognitive flexibility and their disruption in psychotic illness
Waltz, J. A. (2017) · Neuroscience · doi:10.1016/j.neuroscience.2016.06.005 · Cited by 122
“…accumbens (VS/NAcc; Cools et al., 2002) and habenular complex (or epithalamus; Ullsperger and von Cramon, 2003), two areas thought to primarily signal mismatches between expected and obtained outcomes, called reward prediction errors (RPEs; Seymour et al., 2004, Matsumoto and Hikosaka, 2007).”
BackgroundVentral–striatal/nucleus–accumbens sensitivity to prediction errors during classification learning
Rodriguez, P. F., Aron, A. R., & Poldrack, R. A. (2006) · Human Brain Mapping · doi:10.1002/hbm.20186 · Cited by 122
“Imaging studies have also shown activity in the ventral striatum for aversive stimulation [e.g., Becerra et al., 2001; Jensen et al., 2003; Seymour et al., 2004].”
BackgroundWaltz, J. A., Kasanova, Z., Ross, T. J., Salmeron, B. J., McMahon, R. P., Gold, J. M., & Stein, E. A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057257 · Cited by 122
“negative feedback included the vmPFC and neostriatum (Figure 3C–F), which have been typically associated with the integration of outcomes and the signaling of reward prediction errors [4,5,54].”
BackgroundSinger, H. S. (2019) · CONTINUUM Lifelong Learning in Neurology · doi:10.1212/con.0000000000000752 · Cited by 139
Neuroeconomics and aging: Neuromodulation of economic decision making in old age
Mohr, P. N. C., Li, S., & Heekeren, H. R. (2009) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2009.05.010 · Cited by 131
“For instance, derived from the temporal difference model of reward prediction (Seymour et al., 2004) and the stochastic gain tuning model of neuromodulation of cognitive aging, it could be expected that both the reward prediction signal and the mapping between reward predictions and choice actions…”
BackgroundCerebral Changes during Performance of Overlearned Arbitrary Visuomotor Associations
Grol, M. J., de Lange, F. P., Verstraten, F. A. J., Passingham, R. E., & Toni, I. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.2786-05.2006 · Cited by 118
“It has been suggested that reward-prediction signals processed in the striatum (Seymour et al., 2004; Tobler et al., 2005) might support the generation of rapid stimulus–response associations during the early stages of learning (Pasupathy and Miller, 2005).”
BackgroundPredicting danger: The nature, consequences, and neural mechanisms of predictive fear learning
McNally, G. P., & Westbrook, R. F. (2006) · Learning & Memory · doi:10.1101/lm.196606 · Cited by 117
“For example, consider a recent experiment that studied predictive error during serial compound conditioning in humans participants (Seymour et al. 2004).”
BackgroundTic Disorders: Neural Circuits, Neurochemistry, and Neuroimmunology
Harris, K., & Singer, H. S. (2006) · Journal of Child Neurology · doi:10.1177/08830738060210080901 · Cited by 117
Howes, O. D., Hird, E. J., Adams, R. A., Corlett, P. R., & McGuire, P. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2020.03.012 · Cited by 110
“This phasi c signal represents the difference between the predicted and the observed outcome, which is used t o update the model such that future predictions are more accurate (3).”
Neural correlates of unconditioned response diminution during Pavlovian conditioning
Dunsmoor, J. E., Bandettini, P. A., & Knight, D. C. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2007.11.042 · Cited by 127
“Although UCR diminution has not been systematically investigated in previous fMRI research, brain imaging studies of temporal difference learning indicate that prediction errors may alter the fMRI signal produced by aversive stimuli (Jensen et al., 2007; Ploghaus et al., 2000; Seymour et al., 2004).”
Takahashi, Y. K., Langdon, A. J., Niv, Y., & Schoenbaum, G. (2016) · Neuron · doi:10.1016/j.neuron.2016.05.015 · Cited by 116
“…and experimental work has suggested that the ventral striatum (VS) is an important source of these predictions, particularly to dopamine neurons in the ventral tegmental area (VTA) (Daw et al., 2005, 2006; Joel et al., 2002; O’Doherty et al., 2003, 2004; Seymour et al., 2004; Willuhn et al., 2012).”
InfluentialCerebral Correlates of Salient Prediction Error for Different Rewards and Punishments
Metereau, E., & Dreher, J.-C. (2013) · Cerebral Cortex · doi:10.1093/cercor/bhs037 · Cited by 114
“First, we identified several brain regions that have established roles during aversive and appetitive conditioning: the striatum, the insula, the anterior cingulate cortex (ACC) and the amygdala (Jensen et al. 2003; Seymour et al. 2004; Bray and O’Doherty 2007; Herwig et al. 2007; Sescousse et al. 2010).”
InfluentialThe basal ganglia: motor and cognitive relationships in a clinical neurobehavioral context
Leisman, G., & Melillo, R. J. (2012) · Reviews in the Neurosciences · doi:10.1515/revneuro-2012-0067 · Cited by 123
“In the study of non-human primates, electrophysiological studies of the intralaminar thalamic nuclei have revealed that these nuclei influence striatal attentional mechanisms and the processing of reward information (Seymour et al., 2004).”
BackgroundMidbrain activation during pavlovian conditioning and delusional symptoms in schizophrenia
Romaniuk, L., Honey, G. D., King, J. R. L., Whalley, H. C., McIntosh, A. M., Levita, L., Hughes, M., Johnstone, E. C., Day, M., Lawrie, S. M., et al. (2010) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2010.169 · Cited by 113
The periaqueductal gray and Bayesian integration in placebo analgesia
Grahl, A., Onat, S., & Büchel, C. (2018) · eLife · doi:10.7554/elife.32930 · Cited by 112
“…other regions such as lateral orbitofrontal (OFC) and rostral anterior cingulate cortex (rACC) to be involved in aversive prediction error coding (Seymour et al., 2004, 2005; Shackman et al., 2011; Zeidan et al., 2015), yet, no other regions were specifically related to precision level or…”
BackgroundMultiple Forms of Value Learning and the Function of Dopamine
Balleine, B. W., Daw, N. D., & O’Doherty, J. P. (2009) · Elsevier eBooks · doi:10.1016/b978-0-12-374176-9.00024-5 · Cited by 140
Self-citationNeuroscientific approaches and applications within anthropology
Rilling, J. K. (2008) · American Journal of Physical Anthropology · doi:10.1002/ajpa.20947 · Cited by 112
“…(Preuss and Coleman, 2002), to document biochemical differences between human and nonhuman primates in the neuronal populations within primary motor cortex (Sherwood et al., 2004), and to compare serotonergic innervation of frontal cortex in humans and nonhuman primates (Raghanti et al., 2008).”
BackgroundFazeli, S., & Büchel, C. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0671-18.2018 · Cited by 109
“Such a strong connection of the vAI to these limbic structures makes it an ideal mediator between areas representing pain intensity and expectation to evaluate the expectation against the sensory input and estimating a prediction error (Downar et al., 2002; Seymour et al., 2004; Geuter et al., 2017).”
BackgroundCaouette, J. D., & Guyer, A. E. (2014) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2013.10.003 · Cited by 108
“Connections between the striatum and orbitofrontal cortex have been implicated in continuous reward valuation and in the adjustment of behavioral responses based on errant predictions of aversive and appetitive events (Seymour et al., 2004).”
Fear of losing money? Aversive conditioning with secondary reinforcers
Delgado, M. R., Labouliere, C. D., & Phelps, E. A. (2006) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nsl025 · Cited by 107
“In contrast, other brain regions such as the human striatum, involved in updating representations of values during appetitive and aversive learning (Gottfried et al., 2002; Jensen et al., 2003; McClure et al., 2003; O’Doherty et al., 2003; O’Doherty 2004; Seymour et al., 2004; Delgado et al., 2005; Knutson and Cooper,…”
BackgroundThe perception of probability.
Gallistel, C. R., Krishan, M., Liu, Y., Miller, R., & Latham, P. E. (2014) · Psychological Review · doi:10.1037/a0035232 · Cited by 105
Learning and motivation in the human striatum
Shohamy, D. (2011) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2011.05.009 · Cited by 113
“Of particular importance for neurobiological theories of learning, recent findings in both animals and humans indicate that the striatum supports learning to predict both positive and negative outcomes [37–42].”
BackgroundKahnt, T., Park, S. Q., Cohen, M. X., Beck, A., Heinz, A., & Wrase, J. (2009) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2009.21092 · Cited by 102
“It has been shown that these regions represent the value of aversive and punishing stimuli (Seymour et al., 2004; O’Doherty, Kringelbach, Rolls, Hornak, & Andrews, 2001).”
BackgroundThe NMDA Agonist D-Cycloserine Facilitates Fear Memory Consolidation in Humans
Kalisch, R., Holt, B., Petrovic, P., De Martino, B., Klöppel, S., Büchel, C., & Dolan, R. J. (2009) · Cerebral Cortex · doi:10.1093/cercor/bhn076 · Cited by 102
“More recently, researchers have tended to use behavioral data-driven approaches (e.g., Seymour et al. 2004; Carter et al. 2006; Kalisch et al. 2006a) as these generate biologically more plausible predictions of activity time courses.”
BackgroundSelf-citationEmpirical support for an involvement of the mesostriatal dopamine system in human fear extinction
Raczka, K. A., Mechias, M.-L., Gartmann, N., Reif, A., Deckert, J., Pessiglione, M., & Kalisch, R. (2011) · Translational Psychiatry · doi:10.1038/tp.2011.10 · Cited by 101
Seger, C. A., Peterson, E. J., Cincotta, C. M., Lopez-Paniagua, D., & Anderson, C. W. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.083 · Cited by 100
“In the present study we examined signed prediction error and found that this predicted activity in the ventral striatum, consistent with previous studies (Schönberg et al., 2007; Seymour et al., 2004; Brovelli et al., 2008; Cohen, 2008; Rodriguez et al., 2006).”
ResultThe Feeling of Action Tendencies: On the Emotional Regulation of Goal-Directed Behavior
Lowe, R., & Ziemke, T. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00346 · Cited by 99
“TD learning can explain much animal neurobehavioral data (e.g., Suri and Schultz, 1998; Suri, 2002; Roesch et al., 2007) but also higher order learning in humans (cf. Seymour et al., 2004).”
BackgroundUnconditioned responses and functional fear networks in human classical conditioning
Linnman, C., Rougemont-Bücking, A., Beucke, J. C., Zeffiro, T. A., & Milad, M. R. (2011) · Behavioural Brain Research · doi:10.1016/j.bbr.2011.02.045 · Cited by 100
A Spiking Neural Network Model of an Actor-Critic Learning Agent
Potjans, W., Morrison, A., & Diesmann, M. (2008) · Neural Computation · doi:10.1162/neco.2008.08-07-593 · Cited by 118
“Moreover, in human functional neuroimaging, a high correlation was found between the blood-oxygenation-level-dependent (BOLD) activity and the error signal required by a TD learning algorithm (O’Doherty, Dayan, Friston, Critchley, & Dolan, 2003; Seymour et al., 2004).”
BackgroundEffort increases sensitivity to reward and loss magnitude in the human brain
Hernandez Lallement, J., Kuss, K., Trautner, P., Weber, B., Falk, A., & Fliessbach, K. (2014) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nss147 · Cited by 98
Individual differences in risk preference predict neural responses during financial decision-making
Engelmann, J. B., & Tamir, D. (2009) · Brain Research · doi:10.1016/j.brainres.2009.06.078 · Cited by 115
“…the basis for adjustments of future behavior and reward expectation (e.g. Hollerman et al., 1998; McClure et al., 2003; O'Doherty et al., 2003b; Seymour et al., 2004) and has been shown to engage ventral striatum (McClure et al., 2003; Menon et al., 2007; O'Doherty et al., 2003b) and…”
BackgroundEmotional and autonomic consequences of spinal cord injury explored using functional brain imaging
Nicotra, A., Critchley, H. D., Mathias, C. J., & Dolan, R. J. (2006) · Brain · doi:10.1093/brain/awh699 · Cited by 97
Self-citationSerotonin Modulates the Effects of Pavlovian Aversive Predictions on Response Vigor
Crockett, M. J., Clark, L., Apergis-Schoute, A. M., Morein-Zamir, S., & Robbins, T. W. (2012) · Neuropsychopharmacology · doi:10.1038/npp.2012.75 · Cited by 96
“…that specifically modeled the neural representation of Pavlovian aversive predictions Neuropsychopharmacology during learning found aversive value-related activity in the insula, as well as a region of the brainstem consistent with the location of the dorsal raphé nucleus (Seymour et al, 2004).”
BackgroundThe Price of Pain and the Value of Suffering
Vlaev, I., Seymour, B., Dolan, R. J., & Chater, N. (2009) · Psychological Science · doi:10.1111/j.1467-9280.2009.02304.x · Cited by 94
“The study was approved by the joint ethics committee of The National Hospital for Neurology and Neurosurgery and the Institute of Neurology.”
Self-citationNeural activation in the “reward circuit” shows a nonlinear response to facial attractiveness
Liang, X., Zebrowitz, L. A., & Zhang, Y. (2010) · Social Neuroscience · doi:10.1080/17470911003619916 · Cited by 94
“, 2003), and activation of putative reward regions in striatum by both rewarding and aversive stimuli (Jensen et al., 2007; O’Doherty, Buchanan, Seymour, & Dolan, 2006; Seymour et al., 2004).”
BackgroundAtlas, L. Y., Doll, B. B., Li, J., Daw, N. D., & Phelps, E. A. (2016) · eLife · doi:10.7554/elife.15192 · Cited by 93
“, 2011b), although we note that we found a mixture of appetitive and aversive signals here, consistent with previous work (Seymour et al., 2005).”
BackgroundResultMoutoussis, M., Williams, J., Dayan, P., & Bentall, R. P. (2007) · Cognitive Neuropsychiatry · doi:10.1080/13546800701566686 · Cited by 91
“These models already include accounts of functional aspects of neuromodulators and especially of dopamine (Daw, Kakade, & Dayan, 2002; Montague, Dayan, & Sejnowski, 1996; Schultz, Dayan, & Montague, 1997; Seymour et al., 2003).”
MethodSelf-citationStress increases aversive prediction error signal in the ventral striatum
Robinson, O. J., Overstreet, C., Charney, D. R., Vytal, K., & Grillon, C. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1213923110 · Cited by 91
Johnson, A. C., & Redish, A. D. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.08.009 · Cited by 100
“Recent fMRI experiments have also found that the hemodynamic (BOLD) signal in the striatum correlates with changes in the predicted value function in humans making economic choices (O’Doherty, Dayan, Critchley, & Dolan, 2003; Seymour et al., 2004; Tanaka et al., 2004).”
The role of learning-related dopamine signals in addiction vulnerability
Huys, Q. J. M., Tobler, P. N., Hasler, G., & Flagel, S. B. (2014) · Progress in brain research · doi:10.1016/b978-0-444-63425-2.00003-9 · Cited by 107
Gu, X., Wang, X., Hula, A., Wang, S., Xu, S., Lohrenz, T. M., Knight, R. T., Gao, Z., Dayan, P., & Montague, P. R. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.2906-14.2015 · Cited by 99
“Insular activity correlate swith predictions and prediction errors of value (Seymour et al., 2004) and risk (Preuschoff et al.”
BackgroundSelf-citation
Metadata
- Authors
- Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S.
- Affiliations
- Wellcome Centre for Human Neuroimaging (GB)
- Gatsby Computational Neuroscience Unit (GB)
- University College London (GB)
- University of Manchester (GB)
- Corresponding author
- Seymour, Ben
- ORCID
- Venue
- Nature
- Published
- 2004-06
- Volume
- 429
- Issue
- 6992
- Pages
- 664-667
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2084912121
- Cited by
- 605
- References
- 60
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 8 · 2023: 16 · 2022: 21 · 2021: 17 · 2020: 22 · 2019: 25 · 2018: 24 · 2017: 21 · 2016: 24 · 2015: 31 · 2014: 32 · 2013: 47 · 2012: 37
- FWCI
- 15.70
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- other-oa
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Max Planck Digital Library (submittedVersion, other-oa)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Keywords
- temporal difference learning, higher-order conditioning, classical conditioning, aversive conditioning, pain prediction, ventral striatum, anterior insula, sequential learning, computational models of learning, reward learning
- MeSH terms
- Conditioning, Classical, Cues, Electric Stimulation, Hand, Humans, Learning, Magnetic Resonance Imaging, Models, Neurological, Pain, Punishment, Time Factors, Neostriatum
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.