The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Surprise! Neural correlates of Pearce–Hall and Rescorla–Wagner coexist within the brain
Open accessAbstract
Learning theory and computational accounts suggest that learning depends on errors in outcome prediction as well as changes in processing of or attention to events. These divergent ideas are captured by models, such as Rescorla–Wagner (RW) and temporal difference (TD) learning on the one hand, which emphasize errors as directly driving changes in associative strength, vs. models such as Pearce–Hall (PH) and more recent variants on the other hand, which propose that errors promote changes in associative strength by modulating attention and processing of events. Numerous studies have shown that phasic firing of midbrain dopamine (DA) neurons carries a signed error signal consistent with RW or TD learning theories, and recently we have shown that this signal can be dissociated from attentional correlates in the basolateral amygdala and anterior cingulate. Here we will review these data along with new evidence: (i) implicating habenula and striatal regions in supporting error signaling in midbrain DA neurons; and (ii) suggesting that the central nucleus of the amygdala and prefrontal regions process the amygdalar attentional signal. However, while the neural instantiations of the RW and PH signals are dissociable and complementary, they may be linked. Any linkage would have implications for understanding why one signal dominates learning in some situations and not others, and also for appreciating the potential impact on learning of neuropathological conditions involving altered DA or amygdalar function, such as schizophrenia, addiction or anxiety disorders.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 3×
“…in many influential models by the notion of ‘prediction error’, or discrepancy between the actual outcome and the outcome predicted on that trial (Rescorla & Wagner, 1972; Pearce & Hall, 1980; Pearce et al., 1982; Sutton, 1988; Lepelley, 2004; Pearce & Mackintosh, 2010; Esber & Haselgrove, 2011).”
BackgroundHolroyd, C. B., & Coles, M. G. H. (2002) · Psychological Review · doi:10.1037/0033-295x.109.4.679 · Cited by 4,027
Anterior Cingulate Cortex, Error Detection, and the Online Monitoring of Performance
Carter, C. S., Braver, T. S., Barch, D. M., Botvinick, M. M., Noll, D., & Cohen, J. D. (1998) · Science · doi:10.1126/science.280.5364.747 · Cited by 3,273
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023
The amygdala: vigilance and emotion
Davis, M., & Whalen, P. J. (2001) · Molecular Psychiatry · doi:10.1038/sj.mp.4000812 · Cited by 2,882
Kapur, S. (2003) · American Journal of Psychiatry · doi:10.1176/appi.ajp.160.1.13 · Cited by 2,813 · cited 1×
“For example, some symptoms of schizophrenia have been proposed to reflect the spurious attribution of salience to cues (Kapur, 2003).”
BackgroundA theory of attention: Variations in the associability of stimuli with reinforcement
Mackintosh, N. J. (1975) · Psychological Review · doi:10.1037/h0076778 · Cited by 2,733
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Self-citationDopamine in motivational control: rewarding, aversive, and alerting
Bromberg-Martin, E. S., Matsumoto, M., & Hikosaka, O. (2010) · Neuron · doi:10.1016/j.neuron.2010.11.022 · Cited by 2,396
Dissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 2×
“information about the summed expected value given current circumstances to the midbrain (O’Doherty et al., 2004).”
BackgroundResultDiscrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 3×
“These features of DA firing are entirely consistent with prediction error encoding (Mirenowicz & Schultz, 1994; Montague et al., 1996; Hollerman & Schultz, 1998; Waelti et al., 2001; Fiorillo et al., 2003; Tobler et al., 2003; Bayer & Glimcher, 2005; Pan et al., 2005; Morris et al., 2006; Roesch et…”
ResultLearning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
Primate anterior cingulate cortex: Where motor control, drive and cognition interface
Paus, T. (2001) · Nature Reviews Neuroscience · doi:10.1038/35077500 · Cited by 1,722 · cited 8×
“Mathematically, according to one variant of PH (Pearce et al., 1982), the associability of a cue on trial n (an) is updated relative to its previous level (an)1) using the absolute value of the prediction error on that trial.”
BackgroundMethodInfluentialTemporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522
Two types of dopamine neuron distinctly convey positive and negative motivational signals
Matsumoto, M., & Hikosaka, O. (2009) · Nature · doi:10.1038/nature08028 · Cited by 1,378
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304
Lateral habenula as a source of negative reward signals in dopamine neurons
Matsumoto, M., & Hikosaka, O. (2007) · Nature · doi:10.1038/nature05860 · Cited by 1,292
Dopamine responses comply with basic assumptions of formal learning theory
Waelti, P., Dickinson, A., & Schultz, W. (2001) · Nature · doi:10.1038/35083500 · Cited by 1,018
Action sets and decisions in the medial frontal cortex
RUSHWORTH, M., WALTON, M., KENNERLEY, S., & BANNERMAN, D. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.07.009 · Cited by 1,004
Unique properties of mesoprefrontal neurons within a dual mesocorticolimbic dopamine system
Lammel, S., Hetzel, A., Häckel, O., Jones, I., Liss, B., & Roeper, J. (2008) · Neuron · doi:10.1016/j.neuron.2008.01.022 · Cited by 976
Optimal decision making and the anterior cingulate cortex
Kennerley, S. W., Walton, M. E., Behrens, T. E. J., Buckley, M. J., & Rushworth, M. F. S. (2006) · Nature Neuroscience · doi:10.1038/nn1724 · Cited by 948 · cited 3×
“…& Coles, 2000; Paus, 2001; Holroyd & Coles, 2002; Ito et al., 2003; Rushworth et al., 2004, 2007; Walton et al., 2004; Amiez et al., 2005, 2006; Kennerley et al., 2006, 2009; Magno et al., 2006; Matsumoto et al., 2007; Oliveira et al., 2007; Sallet et al., 2007; Quilodran et al., 2008;…”
ResultTemporal Prediction Errors in a Passive Learning Task Activate Human Striatum
McClure, S. M., Berns, G. S., & Montague, P. R. (2003) · Neuron · doi:10.1016/s0896-6273(03)00154-5 · Cited by 917 · cited 2×
“For example, functional imaging results from many groups indicate that the blood oxygen level-dependent (BOLD) response in the human ventral striatum also displays all the hallmarks of prediction errors seen in primate DA recordings, such as elevation for unexpected rewards and suppression when expected rewards are om…”
BackgroundAmygdala inhibitory circuits and the control of fear memory
Ehrlich, I., Humeau, Y., Grenier, F., Ciocchi, S., Herry, C., & Lüthi, A. (2009) · Neuron · doi:10.1016/j.neuron.2009.05.026 · Cited by 878
Choice, uncertainty and value in prefrontal and cingulate cortex
Rushworth, M. F. S., & Behrens, T. E. J. (2008) · Nature Neuroscience · doi:10.1038/nn2066 · Cited by 866
Hare, T. A., O'Doherty, J., Camerer, C. F., Schultz, W., & Rangel, A. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1309-08.2008 · Cited by 792
The short-latency dopamine signal: a role in discovering novel actions?
Redgrave, P., & Gurney, K. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn2022 · Cited by 762
Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens
Day, J. J., Roitman, M. F., Wightman, R. M., & Carelli, R. M. (2007) · Nature Neuroscience · doi:10.1038/nn1923 · Cited by 710
Jhou, T. C., Fields, H. L., Baxter, M. G., Saper, C. B., & Holland, P. C. (2009) · Neuron · doi:10.1016/j.neuron.2009.02.001 · Cited by 696
BOLD responses reflecting dopaminergic signals in the human ventral tegmental area
D'Ardenne, K., McClure, S. M., Nystrom, L. E., & Cohen, J. D. (2008) · Science · doi:10.1126/science.1150605 · Cited by 661
Amygdala circuitry in attentional and representational processes
Holland, P. C., & Gallagher, M. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(98)01271-6 · Cited by 618
Corbit, L. H., & Balleine, B. W. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.4507-04.2005 · Cited by 589
Izquierdo, A., Suda, R. K., & Murray, E. A. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.1921-04.2004 · Cited by 588
Performance monitoring by the anterior cingulate cortex during saccade countermanding
Ito, S., Stuphorn, V., Brown, J. W., & Schall, J. D. (2003) · Science · doi:10.1126/science.1087847 · Cited by 581
Importance of unpredictability for reward responses in primate dopamine neurons
Mirenowicz, J., & Schultz, W. (1994) · Journal of Neurophysiology · doi:10.1152/jn.1994.72.2.1024 · Cited by 569
Contrasting roles for cingulate and orbitofrontal cortex in decisions and social behaviour
Rushworth, M. F. S., Behrens, T. E. J., Rudebeck, P. H., & Walton, M. E. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.01.004 · Cited by 507
Midbrain dopamine neurons encode decisions for future action
Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006) · Nature Neuroscience · doi:10.1038/nn1743 · Cited by 503 · cited 3×
“…1994; Houk et al., 1995; Montague et al., 1996; Hollerman & Schultz, 1998; Waelti et al., 2001; Fiorillo et al., 2003; Tobler et al., 2003; Ungless et al., 2004; Bayer & Glimcher, 2005; Pan et al., 2005; Morris et al., 2006; Roesch et al., 2007; D’Ardenne et al., 2008; Matsumoto & Hikosaka, 2009).”
ResultScheffers, M. K., & Coles, M. G. H. (2000) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.26.1.141 · Cited by 493
Neurons in the frontal lobe encode the value of multiple decision variables
Kennerley, S. W., Dahmubed, A. F., Lara, A. H., & Wallis, J. D. (2009) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2009.21100 · Cited by 453
Pan, W.-X., Schmidt, R., Wickens, J. R., & Hyland, B. I. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.1478-05.2005 · Cited by 429
Hong, S., Jhou, T. C., Smith, M., Saleem, K. S., & Hikosaka, O. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.1384-11.2011 · Cited by 390
Ji, H., & Shepard, P. D. (2007) · Journal of Neuroscience · doi:10.1523/jneurosci.0958-07.2007 · Cited by 388 · cited 1×
“…the VTA and the adjacent rostromedial tegmental nucleus, which has similar response properties as LHb neurons and heavy inhibitory projections to midbrain DA neurons (Ji & Shepard, 2007; Jhou et al., 2009; Kaufling et al., 2009; Omelchenko et al., 2009; Brinschwitz et al., 2010; Hong et al., 2011).”
BackgroundOrbitofrontal Cortex Mediates Outcome Encoding in Pavlovian But Not Instrumental Conditioning
Ostlund, S. B., & Balleine, B. W. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.5443-06.2007 · Cited by 379 · cited 1×
“…input from the orbitofrontal cortex, a key region in encoding such model-based associative structures (Pickens et al., 2003; Izquierdo et al., 2004; Ostlund & Balleine, 2007; Burke et al., 2008), is necessary for expectancy-related changes in phasic firing in midbrain DA neurons (Takahashi et al.,…”
BackgroundHayden, B. Y., Heilbronner, S. R., Pearson, J. M., & Platt, M. L. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.4652-10.2011 · Cited by 379 · cited 2×
“These changes in ACC firing occurred regardless of valence at the single-cell level, consistent with encoding of surprise or attentional variables (Hayden et al., 2011).”
BackgroundLe Pelley, M. E. (2004) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/02724990344000141 · Cited by 360 · cited 5×
“One possibility, which we have suggested several times here and elsewhere (Lepelley, 2004; Courville et al., 2006; Li et al., 2011), is that the RW ⁄ TD and PH mechanisms effectively comprise the associative learning and learning rate control components of a single hybrid learner.”
BackgroundResultInfluentialExpectation modulates neural responses to pleasant and aversive stimuli in primate amygdala
Belova, M. A., Paton, J. J., Morrison, S. E., & Salzman, C. D. (2007) · Neuron · doi:10.1016/j.neuron.2007.08.004 · Cited by 349
Afferents to the GABAergic tail of the ventral tegmental area in the rat
Kaufling, J., Veinante, P., Pawlowski, S. A., Freund‐Mercier, M., & Barrot, M. (2009) · Journal of Comparative Neurology · doi:10.1002/cne.21983 · Cited by 347 · cited 2×
“…the VTA and the adjacent rostromedial tegmental nucleus, which has similar response properties as LHb neurons and heavy inhibitory projections to midbrain DA neurons (Ji & Shepard, 2007; Jhou et al., 2009; Kaufling et al., 2009; Omelchenko et al., 2009; Brinschwitz et al., 2010; Hong et al., 2011).”
BackgroundThe globus pallidus sends reward-related signals to the lateral habenula
Hong, S., & Hikosaka, O. (2008) · Neuron · doi:10.1016/j.neuron.2008.09.035 · Cited by 346
Linking nucleus accumbens dopamine and blood oxygenation
Knutson, B., & Gibbs, S. E. B. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0686-7 · Cited by 337 · cited 1×
“In addition, activity was also high for delayed reward, consistent with recent reports in primates (Fiorillo et al., 2008; Kobayashi & Schultz, 2008).”
ResultNeural substrates for expectation-modulated fear learning in the amygdala and periaqueductal gray
Johansen, J. P., Tarpley, J. W., LeDoux, J. E., & Blair, H. T. (2010) · Nature Neuroscience · doi:10.1038/nn.2594 · Cited by 312 · cited 1×
“Modulation of neural activity in lateral and basal nucleus of the amygdala by expectation has also been described during fear conditioning in rats (Johansen et al., 2010).”
BackgroundThe temporal precision of reward prediction in dopamine neurons
Fiorillo, C. D., Newsome, W. T., & Schultz, W. (2008) · Nature Neuroscience · doi:10.1038/nn.2159 · Cited by 310
Coding of Predicted Reward Omission by Dopamine Neurons in a Conditioned Inhibition Paradigm
Tobler, P. N., Dickinson, A., & Schultz, W. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-32-10402.2003 · Cited by 300 · cited 3×
“…models (Mirenowicz & Schultz, 1994; Houk et al., 1995; Montague et al., 1996; Hollerman & Schultz, 1998; Waelti et al., 2001; Fiorillo et al., 2003; Tobler et al., 2003; Ungless et al., 2004; Bayer & Glimcher, 2005; Pan et al., 2005; Morris et al., 2006; Roesch et al., 2007; D’Ardenne et al.,…”
BackgroundResultExpectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex
Takahashi, Y. K., Roesch, M. R., Wilson, R. C., Toreson, K., O'Donnell, P., Niv, Y., & Schoenbaum, G. (2011) · Nature Neuroscience · doi:10.1038/nn.2957 · Cited by 280 · cited 1×
“…from the orbitofrontal cortex, a key region in encoding such model-based associative structures (Pickens et al., 2003; Izquierdo et al., 2004; Ostlund & Balleine, 2007; Burke et al., 2008), is necessary for expectancy-related changes in phasic firing in midbrain DA neurons (Takahashi et al., 2011).”
BackgroundSelf-citationMoment-to-moment tracking of state value in the amygdala
Belova, M. A., Paton, J. J., & Salzman, C. D. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.1400-08.2008 · Cited by 221 · cited 1×
“Other accounts have suggested that value predictions may also be derived from the amygdala (Belova et al., 2008).”
BackgroundRapid strengthening of thalamo-amygdala synapses mediates cue–reward learning
Tye, K. M., Stuber, G. D., de Ridder, B., Bonci, A., & Janak, P. H. (2008) · Nature · doi:10.1038/nature06963 · Cited by 216 · cited 2×
“…neurons signal shifts in attention While the amygdala has often been viewed as critical for learning to predict aversive outcomes (Davis, 2000; LeDoux, 2000), the last two decades have revealed a more general involvement in associative learning (Gallagher, 2000; Murray, 2007; Tye et al., 2008).”
BackgroundHeterogeneous reward signals in prefrontal cortex
Wallis, J. D., & Kennerley, S. W. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.02.009 · Cited by 186 · cited 1×
“…Magno et al., 2006; Matsumoto et al., 2007; Oliveira et al., 2007; Sallet et al., 2007; Quilodran et al., 2008; Rudebeck et al., 2008; Rushworth & Behrens, 2008; Kennerley & Wallis, 2009; Totah et al., 2009; Hillman & Bilkey, 2010; Wallis & Kennerley, 2010; Hayden et al., 2011; Rothe et al., 2011).”
BackgroundAdding prediction risk to the theory of reward learning
PREUSCHOFF, K., & BOSSAERTS, P. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.005 · Cited by 133 · cited 2×
“…sound statistical reasoning; these considerations typically lead to models containing both a RW-like error-driven update, scaled by a PH-like associability mechanism (Sutton, 1992; Dayan & Long, 1998; Kakade & Dayan, 2002; Courville et al., 2006; Behrens et al., 2007; Bossaerts & Preuschoff, 2007).”
Background
Cited by
Frontal theta as a mechanism for cognitive control
Cavanagh, J. F., & Frank, M. J. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.04.012 · Cited by 2,726
From circuits to behaviour in the amygdala
Janak, P. H., & Tye, K. M. (2015) · Nature · doi:10.1038/nature14188 · Cited by 2,011
The ecology of human fear: survival optimization and the nervous system
Mobbs, D., Hagan, C. C., Dalgleish, T., Silston, B., & Prévost, C. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00055 · Cited by 498
Neural mechanisms and temporal dynamics of performance monitoring
Ullsperger, M., Fischer, A. G., Nigbur, R., & Endrass, T. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.009 · Cited by 435
Emotion and Decision Making: Multiple Modulatory Neural Circuits
Phelps, E. A., Lempert, K. M., & Sokol-Hessner, P. (2014) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071013-014119 · Cited by 430
Metadata
- Authors
- Roesch, M. R., Esber, G. R., Li, J., Daw, N. D., & Schoenbaum, G.
- Venue
- European Journal of Neuroscience
- Publisher
- Wiley
- Published
- 2012-04
- Volume
- 35
- Issue
- 7
- Pages
- 1190-1200
- ISSN
- 0953-816X (ISSN-L)
- 1460-9568
- OpenAlex
- W1595744040
- Cited by
- 191
- References
- 58
- FWCI
- 5.83
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.