The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Contextual modulation of value signals in reward and punishment learning
Open accessAbstract
Compared with reward seeking, punishment avoidance learning is less clearly understood at both the computational and neurobiological levels. Here we demonstrate, using computational modelling and fMRI in humans, that learning option values in a relative—context-dependent—scale offers a simple computational solution for avoidance learning. The context (or state) value sets the reference point to which an outcome should be compared before updating the option value. Consequently, in contexts with an overall negative expected value, successful punishment avoidance acquires a positive value, thus reinforcing the response. As revealed by post-learning assessment of options values, contextual influences are enhanced when subjects are informed about the result of the forgone alternative (counterfactual information). This is mirrored at the neural level by a shift in negative outcome encoding from the anterior insula to the ventral striatum, suggesting that value contextualization also limits the need to mobilize an opponent punishment learning system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Cohen, J. (1992) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep10768783 · Cited by 5,870
Regret Theory: An Alternative Theory of Rational Choice Under Uncertainty
Loomes, G., & Sugden, R. (1982) · The Economic Journal · doi:10.2307/2232669 · Cited by 4,042 · cited 1×
“…of the task served our purposes: first, the task contrasted reward seeking with punishment avoidance learning; second, in specific choice contexts, we provided the information about the outcome of the foregone alternative – counterfactual information – to enhance relative value encoding 24–26 .”
Circular analysis in systems neuroscience: the dangers of double dipping
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009) · Nature Neuroscience · doi:10.1038/nn.2303 · Cited by 2,696
By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Bartra, O., McGuire, J. T., & Kable, J. W. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.02.063 · Cited by 1,927
An opponent-process theory of motivation. I. Temporal dynamics of affect
Solomon, R. L., & Corbit, J. D. (1974) · Psychological Review · doi:10.1037/h0036128 · Cited by 1,704
Mowrer, O. H. (1960) · doi:10.1037/10802-000 · Cited by 1,670
Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans
Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., & Frith, C. D. (2006) · Nature · doi:10.1038/nature05051 · Cited by 1,584
Gonzalez, R. C., & Mowrer, O. H. (1962) · The American Journal of Psychology · doi:10.2307/1419634 · Cited by 1,583
Bayesian model selection for group studies
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.025 · Cited by 1,423
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304
Tversky, A., & Simonson, I. (1993) · Management Science · doi:10.1287/mnsc.39.10.1179 · Cited by 1,290
Brain systems mediating aversive conditioning: an event-related fMRI study
Büchel, C., Morris, J., Dolan, R. J., & Friston, K. J. (1998) · Neuron · doi:10.1016/s0896-6273(00)80476-6 · Cited by 951
Liu, X., Hairston, J., Schrier, M., & Fan, J. (2011) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2010.12.012 · Cited by 916
Opponent interactions between serotonin and dopamine
Daw, N. D., Kakade, S., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00052-7 · Cited by 844
Estimating sample size in functional MRI (fMRI) neuroimaging studies: Statistical power analyses
Desmond, J. E., & Glover, G. H. (2002) · Journal of Neuroscience Methods · doi:10.1016/s0165-0270(02)00121-8 · Cited by 716
Informatic parcellation of the network involved in the computation of subjective value
Clithero, J. A., & Rangel, A. (2014) · Social Cognitive and Affective Neuroscience · doi:10.1093/scan/nst106 · Cited by 706
Regret and its avoidance: a neuroimaging study of choice behavior
Coricelli, G., Critchley, H. D., Joffily, M., O'Doherty, J. P., Sirigu, A., & Dolan, R. J. (2005) · Nature Neuroscience · doi:10.1038/nn1514 · Cited by 671
Self-citationNeural computations underlying arbitration between model-based and model-free learning
Lee, S. W., Shimojo, S., & O’Doherty, J. P. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.028 · Cited by 642
Preference by Association: How Memory Mechanisms in the Hippocampus Bias Decisions
Wimmer, G. E., & Shohamy, D. (2012) · Science · doi:10.1126/science.1223252 · Cited by 515
Model‐Based fMRI and Its Application to Reward Learning and Decision Making
O'DOHERTY, J. P., HAMPTON, A., & KIM, H. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.022 · Cited by 508
An Opponent-Process Theory of Motivation
Solomon, R. L., & Corbit, J. D. (2008) · Cited by 461
Trial-by-trial data analysis using computational models
Daw, N. D. (2011) · Decision Making, Affect, and Learning · doi:10.1093/acprof:oso/9780199600434.003.0001 · Cited by 456
Kim, H., Shimojo, S., & O'Doherty, J. P. (2006) · PLoS Biology · doi:10.1371/journal.pbio.0040233 · Cited by 453
Opponent appetitive-aversive neural processes underlie predictive learning of pain relief
Seymour, B., O'Doherty, J. P., Koltzenburg, M., Wiech, K., Frackowiak, R., Friston, K., & Dolan, R. (2005) · Nature Neuroscience · doi:10.1038/nn1527 · Cited by 439
Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies
Garrison, J., Erdeniz, B., & Done, J. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.03.023 · Cited by 438
Cognitive control over learning: Creating, clustering, and generalizing task-set structure.
Collins, A. G. E., & Frank, M. J. (2013) · Psychological Review · doi:10.1037/a0030852 · Cited by 426
Go and no-go learning in reward and punishment: Interactions between affect and effect
Guitart-Masip, M., Huys, Q. J. M., Fuentemilla, L., Dayan, P., Duzel, E., & Dolan, R. J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.04.024 · Cited by 421
Pitt, M. A., & Myung, I. J. (2002) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(02)01964-2 · Cited by 409
Context, learning, and extinction
Gershman, S. J., Blei, D. M., & Niv, Y. (2010) · Psychological Review · doi:10.1037/a0017808 · Cited by 386 · cited 1×
“…the most parsimonious algorithm implementation of a model that, departing from experienced raw values, learns to identify, for each situation (context) the “best” and the “worst” possible outcomes, based on an explicit representation of the underlying generative process (the task structure) 49,50 .”
VBA: A Probabilistic Treatment of Nonlinear Models for Neurobiological and Behavioural Data
Daunizeau, J., Adam, V., & Rigoux, L. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003441 · Cited by 385
Neural mechanisms of observational learning
Burke, C. J., Tobler, P. N., Baddeley, M., & Schultz, W. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1003111107 · Cited by 367
The role of the striatum in aversive learning and aversive prediction errors
Delgado, M. R., Li, J., Schiller, D., & Phelps, E. A. (2008) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2008.0161 · Cited by 328
Subliminal Instrumental Conditioning Demonstrated in the Human Brain
Pessiglione, M., Petrovic, P., Daunizeau, J., Palminteri, S., Dolan, R. J., & Frith, C. D. (2008) · Neuron · doi:10.1016/j.neuron.2008.07.005 · Cited by 308
Self-citationActivity in human reward-sensitive brain areas is strongly context dependent
Nieuwenhuis, S., Heslenfeld, D. J., Alting von Geusau, N. J., Mars, R. B., Holroyd, C. B., & Yeung, N. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.12.043 · Cited by 289
Does the brain calculate value?
Vlaev, I., Chater, N., Stewart, N., & Brown, G. D. A. (2011) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.09.008 · Cited by 289
Action versus valence in decision making
Guitart-Masip, M., Duzel, E., Dolan, R., & Dayan, P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.01.003 · Cited by 270
A normative perspective on motivation
NIV, Y., JOEL, D., & DAYAN, P. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.06.010 · Cited by 251
Critical Roles for Anterior Insula and Dorsal Striatum in Punishment-Based Avoidance Learning
Palminteri, S., Justo, D., Jauffret, C., Pavlicek, B., Dauta, A., Delmaire, C., Czernecki, V., Karachi, C., Capelle, L., Durr, A., et al. (2012) · Neuron · doi:10.1016/j.neuron.2012.10.017 · Cited by 242
Self-citationThe neurobiology of punishment
Seymour, B., Singer, T., & Dolan, R. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2119 · Cited by 240
Counterfactual Choice and Learning in a Neural Network Centered on Human Lateral Frontopolar Cortex
Boorman, E. D., Behrens, T. E., & Rushworth, M. F. (2011) · PLoS Biology · doi:10.1371/journal.pbio.1001093 · Cited by 218
Twenty-Five Lessons from Computational Neuromodulation
Dayan, P. (2012) · Neuron · doi:10.1016/j.neuron.2012.09.027 · Cited by 190
Individual Differences in Insular Sensitivity During Loss Anticipation Predict Avoidance Learning
Samanez-Larkin, G. R., Hollon, N. G., Carstensen, L. L., & Knutson, B. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02087.x · Cited by 188
Grossberg, S., & Schmajuk, N. A. (1987) · Psychobiology · doi:10.3758/bf03327274 · Cited by 184
Real and Fictive Outcomes Are Processed Differently but Converge on a Common Adaptive Mechanism
Fischer, A. G., & Ullsperger, M. (2013) · Neuron · doi:10.1016/j.neuron.2013.07.006 · Cited by 178
Skvortsova, V., Palminteri, S., & Pessiglione, M. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.1350-14.2014 · Cited by 177
Self-citationReinforcement learning in continuous time: advantage updating
Baird, L. C. (1994) · Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) · doi:10.1109/icnn.1994.374604 · Cited by 177
Weiskopf, N., Hutton, C., Josephs, O., Turner, R., & Deichmann, R. (2007) · Magnetic Resonance Materials in Physics, Biology and Medicine · doi:10.1007/s10334-006-0067-6 · Cited by 177
Value normalization in decision making: theory and evidence
Rangel, A., & Clithero, J. A. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.07.011 · Cited by 166
Signals in Human Striatum Are Appropriate for Policy Update Rather than Value Prediction
Li, J., & Daw, N. D. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.6316-10.2011 · Cited by 163
Efficient coding and the neural representation of value
Louie, K., & Glimcher, P. W. (2012) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2012.06496.x · Cited by 159
Anchors, scales and the relative coding of value in the brain
Seymour, B., & McClure, S. M. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.07.010 · Cited by 158
How instructed knowledge modulates the neural systems of reward learning
Li, J., Delgado, M. R., & Phelps, E. A. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1014938108 · Cited by 153
Inferring affect from fMRI data
Knutson, B. D., Katovich, K., & Suri, G. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.04.006 · Cited by 141
Two-factor theory, the actor--critic model, and conditioned avoidance
Maia, T. V. (2010) · Learning & Behavior · doi:10.3758/lb.38.1.50 · Cited by 139
Kahnt, T., Park, S. Q., Cohen, M. X., Beck, A., Heinz, A., & Wrase, J. (2008) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2009.21092 · Cited by 108
Brain Hemispheres Selectively Track the Expected Value of Contralateral Options
Palminteri, S., Boraud, T., Lafargue, G., Dubois, B., & Pessiglione, M. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.1500-09.2009 · Cited by 99
Self-citationKhamassi, M., Quilodran, R., Enel, P., Dominey, P. F., & Procyk, E. (2014) · Cerebral Cortex · doi:10.1093/cercor/bhu114 · Cited by 96
Self-citationA temporal difference account of avoidance learning
Moutoussis, M., Bentall, R. P., Williams, J. H., & Dayan, P. (2008) · Network Computation in Neural Systems · doi:10.1080/09548980802192784 · Cited by 86
Aversive stimuli and loss in the mesocorticolimbic dopamine system
Brooks, A. M., & Berns, G. S. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.04.001 · Cited by 85
Valence-dependent influence of serotonin depletion on model-based choice strategy
Worbe, Y., Palminteri, S., Savulich, G., Daw, N. D., Fernandez-Egea, E., Robbins, T. W., & Voon, V (2016) · Molecular Psychiatry · doi:10.1038/mp.2015.46 · Cited by 75
Self-citationWorbe, Y. (2011) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2011.137 · Cited by 63
Morris, R. W., Dezfouli, A., Griffiths, K. R., & Balleine, B. W. (2014) · Nature Communications · doi:10.1038/ncomms5390 · Cited by 62
Relative Valuation of Pain in Human Orbitofrontal Cortex
Winston, J. S., Vlaev, I., Seymour, B., Chater, N., & Dolan, R. J. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.1706-14.2014 · Cited by 62
Rational Attention and Adaptive Coding: A Puzzle and a Solution
Padoa‐Schioppa, C., & Rustichini, A. (2014) · American Economic Review · doi:10.1257/aer.104.5.507 · Cited by 48
Advanced Reinforcement Learning
Daw, N. D. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-416008-8.00016-4 · Cited by 46
Jessup, R. K., & O’Doherty, J. P. (2014) · European Journal of Neuroscience · doi:10.1111/ejn.12625 · Cited by 30
From the Reward Circuit to the Valuation System: How the Brain Motivates Behavior
Pessiglione, M., & Lebreton, M. P. (2014) · Springer eBooks · doi:10.1007/978-1-4939-1236-0_11 · Cited by 23
Cited by
The Importance of Falsification in Computational Cognitive Modeling
Palminteri, S., Wyart, V., & Koechlin, E. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.03.011 · Cited by 448
Self-citationBehavioural and neural characterization of optimistic reinforcement learning
Lefebvre, G., Lebreton, M., Meyniel, F., Bourgeois-Gironde, S., & Palminteri, S. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0067 · Cited by 290
Self-citationAssessing inter individual differences with task related functional neuroimaging
Lebreton, M., Bavard, S., Daunizeau, J., & Palminteri, S. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0681-8 · Cited by 160
Self-citationThe case against economic values in the orbitofrontal cortex (or anywhere else in the brain).
Hayden, B. Y., & Niv, Y. (2021) · Behavioral Neuroscience · doi:10.1037/bne0000448 · Cited by 127
Fouragnan, E., Queirazza, F., Retzler, C., Mullinger, K. J., & Philiastides, M. G. (2017) · Scientific Reports · doi:10.1038/s41598-017-04507-w · Cited by 61
Metadata
- Authors
- Palminteri, S., Khamassi, M., Joffily, M., & Coricelli, G.
- Venue
- Nature Communications
- Published
- 2015-08-25
- Volume
- 6
- Issue
- 1
- Pages
- 8096-8096
- ISSN
- 2041-1723 (ISSN-L)
- OpenAlex
- W1152374494
- Cited by
- 286
- References
- 70
- FWCI
- 8.30
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.