Hedging Your Bets by Learning Reward Correlations in the Human Brain
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Human subjects are proficient at tracking the mean and variance of rewards and updating these via prediction errors. Here, we addressed whether humans can also learn about higher-order relationships between distinct environmental outcomes, a defining ecological feature of contexts where multiple sources of rewards are available. By manipulating the degree to which distinct outcomes are correlated, we show that subjects implemented an explicit model-based strategy to learn the associated outcome correlations and were adept in using that information to dynamically adjust their choices in a task that required a minimization of outcome variance. Importantly, the experimentally generated outcome correlations were explicitly represented neuronally in right midinsula with a learning prediction error signal expressed in rostral anterior cingulate cortex. Thus, our data show that the human brain represents higher-order correlation structures between rewards, a core adaptive ability whose immediate benefit is optimized sampling.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 18,497 · cited 2×
“Commonly observed risk aversion in animals (Kacelnik and Bateson, 1996) and humans (Tversky and Kahneman, 1981) is rational in an evolutionary context, as a small but constant supply ron 71, 1141–1152, September 22, 2011 ª2011 Elsevier Inc. 1145 ACC, anterior cingulate; DMPFC, dorsomedial…”
How do you feel — now? The anterior insula and human awareness
(Bud) Craig, A. D. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2555 · Cited by 6,657 · cited 4×
“The finding of a slightly posterior encoding of correlation relative to risk also tallies with a structural model for how unconscious state representations might be integrated into a sentient self along a posterior to anterior insula (Craig, 2009).”
InfluentialDissociable Roles of Ventral and Dorsal Striatum in Instrumental Conditioning
O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004) · Science · doi:10.1126/science.1094285 · Cited by 2,088 · cited 1×
“There is now good evidence that fundamental computational mechanisms underlying valuebased learning and decision-making are well captured by reinforcement learning algorithms (Sutton and Barto, 1998) where option values are updated on a trial by trial basis via prediction errors (PE) (Knutson and Cooper, 2005; Montagu…”
BackgroundSelf-citationLearning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
Neurons in the orbitofrontal cortex encode economic value
Padoa-Schioppa, C., & Assad, J. A. (2006) · Nature · doi:10.1038/nature04676 · Cited by 1,598 · cited 2×
“…inputs to the decision process such as action or goal values, and representations of expected outcome after a choice (Hampton et al., 2006; Knutson et al., 2005; Lau and Glimcher, 2007; Padoa-Schioppa and Assad, 2006; Plassmann et al., 2007; Samejima et al., 2005; Wunderlich et al., 2009, 2010).”
Bayesian model selection for group studies
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.025 · Cited by 1,423
The Neural Basis of Financial Risk Taking
Kuhnen, C. M., & Knutson, B. (2005) · Neuron · doi:10.1016/j.neuron.2005.08.008 · Cited by 1,182
Investigating emotion with music: An fMRI study
Koelsch, S., Fritz, T., v. Cramon, D. Y., Müller, K., & Friederici, A. D. (2006) · Human Brain Mapping · doi:10.1002/hbm.20180 · Cited by 1,037
Neural Economics and the Biological Substrates of Valuation
Montague, P. R., & Berns, G. S. (2002) · Neuron · doi:10.1016/s0896-6273(02)00974-1 · Cited by 994
Distributed Neural Representation of Expected Value
Knutson, B., Taylor, J., Kaufman, M., Peterson, R., & Glover, G. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.0642-05.2005 · Cited by 955
Optimal decision making and the anterior cingulate cortex
Kennerley, S. W., Walton, M. E., Behrens, T. E. J., Buckley, M. J., & Rushworth, M. F. S. (2006) · Nature Neuroscience · doi:10.1038/nn1724 · Cited by 948 · cited 2×
“Several previous studies report a role for anterior cingulate in an error-driven reinforcement learning system (Kennerley et al., 2006), and in prediction errors for actions (Matsumoto et al.”
Representation of Action-Specific Reward Values in the Striatum
Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005) · Science · doi:10.1126/science.1115270 · Cited by 939 · cited 2×
“…inputs to the decision process such as action or goal values, and representations of expected outcome after a choice (Hampton et al., 2006; Knutson et al., 2005; Lau and Glimcher, 2007; Padoa-Schioppa and Assad, 2006; Plassmann et al., 2007; Samejima et al., 2005; Wunderlich et al., 2009, 2010).”
Associative learning of social value
Behrens, T. E. J., Hunt, L. T., Woolrich, M. W., & Rushworth, M. F. S. (2008) · Nature · doi:10.1038/nature07538 · Cited by 925 · cited 4×
“This method for plotting the effect size timecourse of a parametrically modulated regressor is also described in detail elsewhere (Behrens et al., 2008).”
BackgroundMethodInfluentialOrbitofrontal Cortex Encodes Willingness to Pay in Everyday Economic Transactions
Plassmann, H., O'Doherty, J., & Rangel, A. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.2131-07.2007 · Cited by 855
Paulus, M. P., Rogalsky, C., Simmons, A., Feinstein, J. S., & Stein, M. B. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00251-9 · Cited by 745
Functional magnetic resonance imaging of reward prediction
Knutson, B., & Cooper, J. C. (2005) · Current Opinion in Neurology · doi:10.1097/01.wco.0000173463.24758.f6 · Cited by 736 · cited 1×
“…learning and decision-making are well captured by reinforcement learning algorithms (Sutton and Barto, 1998) where option values are updated on a trial by trial basis via prediction errors (PE) (Knutson and Cooper, 2005; Montague and Berns, 2002; O’Doherty et al., 2004; Schultz et al., 1997).”
Neural differentiation of expected reward and risk in human subcortical structures
Preuschoff, K., Bossaerts, P., & Quartz, S. R. (2006) · Neuron · doi:10.1016/j.neuron.2006.06.024 · Cited by 692
Self-citationDoya, K. (2008) · Nature Neuroscience · doi:10.1038/nn2077 · Cited by 642 · cited 1×
“Research in decision neuroscience provides extensive evidence for a neural representation of key decision variables (Doya, 2008) with a focus heretofore on value signals, putative inputs to the decision process such as action or goal values, and representations of expected outcome after a choice…”
Risky business: the neuroeconomics of decision making under uncertainty
Platt, M. L., & Huettel, S. A. (2008) · Nature Neuroscience · doi:10.1038/nn2062 · Cited by 556
Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498
Self-citationNeural correlates of value, risk, and risk aversion contributing to decision making under risk
Christopoulos, G. I., Tobler, P. N., Bossaerts, P., Dolan, R. J., & Schultz, W. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2614-09.2009 · Cited by 451 · cited 2×
“, 2006; Platt and Huettel, 2008) and higher statistical moments of outcomes such as variance (Christopoulos et al., 2009; Mohr et al., 2010; Preuschoff et al., 2006, 2008; Tobler et al., 2009) and skewness (Symmonds et al.”
Self-citationNeural computations underlying action-based decision making in the human brain
Wunderlich, K., Rangel, A., & O'Doherty, J. P. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0901077106 · Cited by 301 · cited 1×
“…inputs to the decision process such as action or goal values, and representations of expected outcome after a choice (Hampton et al., 2006; Knutson et al., 2005; Lau and Glimcher, 2007; Padoa-Schioppa and Assad, 2006; Plassmann et al., 2007; Samejima et al., 2005; Wunderlich et al., 2009, 2010).”
Self-citationRange-adapting representation of economic value in the orbitofrontal cortex
Padoa-Schioppa, C. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.3751-09.2009 · Cited by 276
Cited by
Hierarchical Prediction Errors in Midbrain and Basal Forebrain during Sensory Learning
Iglesias, S., Mathys, C., Brodersen, K. H., Kasper, L., Piccirelli, M., den Ouden, H. E. M., & Stephan, K. E. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.009 · Cited by 342
The ubiquity of model-based reinforcement learning
Doll, B. B., Simon, D. A., & Daw, N. D. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.08.003 · Cited by 380
Fouragnan, E., Retzler, C., & Philiastides, M. G. (2018) · Human Brain Mapping · doi:10.1002/hbm.24047 · Cited by 176
Metadata
- Authors
- Wunderlich, K., Symmonds, M., Bossaerts, P., & Dolan, R. J.
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2011-09
- Volume
- 71
- Issue
- 6
- Pages
- 1141-1152
- ISSN
- 0896-6273 (ISSN-L)
- OpenAlex
- W2152462403
- Cited by
- 40
- References
- 23
- FWCI
- 2.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.