The DOI matches its Crossref record; 50 of the 52 references it cites were found in a registry or an index.
Risk, Unexpected Uncertainty, and Estimation Uncertainty: Bayesian Learning in Unstable Settings
Open accessAbstract
Recently, evidence has emerged that humans approach learning using Bayesian updating rather than (model-free) reinforcement algorithms in a six-arm restless bandit problem. Here, we investigate what this implies for human appreciation of uncertainty. In our task, a Bayesian learner distinguishes three equally salient levels of uncertainty. First, the Bayesian perceives irreducible uncertainty or risk: even knowing the payoff probabilities of a given arm, the outcome remains uncertain. Second, there is (parameter) estimation uncertainty or ambiguity: payoff probabilities are unknown and need to be estimated. Third, the outcome probabilities of the arms change: the sudden jumps are referred to as unexpected uncertainty. We document how the three levels of uncertainty evolved during the course of our experiment and how it affected the learning rate. We then zoom in on estimation uncertainty, which has been suggested to be a driving force in exploration, in spite of evidence of widespread aversion to ambiguity. Our data corroborate the latter. We discuss neural evidence that foreshadowed the ability of humans to distinguish between the three levels of uncertainty. Finally, we investigate the boundaries of human capacity to implement Bayesian learning. We repeat the experiment with different instructions, reflecting varying levels of structural uncertainty. Under this fourth notion of uncertainty, choices were no better explained by Bayesian updating than by (model-free) reinforcement learning. Exit questionnaires revealed that participants remained unaware of the presence of unexpected uncertainty and failed to acquire the right model with which to implement Bayesian updating.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Here, deviations from expected uncertainty (“unexpected uncertainty”) should increase learning rates.8,12,13,15–18”
From Serotonin neurons modulate learning rate through uncertainty (2022) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Nudge: improving decisions about health, wealth, and happiness
(2008) · Choice Reviews Online · doi:10.5860/choice.46-0977 · Cited by 11,624
Knight, F. H. (1921) · Cited by 9,967
“Risk, Ambiguity, and the Savage Axioms.”
Ellsberg, D. (1961) · The Quarterly Journal of Economics · doi:10.2307/1884324 · Cited by 7,891
Statistical Decision Theory and Bayesian Analysis
Berger, J. O. (1985) · Springer Series in Statistics · doi:10.1007/978-1-4757-4286-2 · Cited by 7,722
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 7,268
Nudge: Improving Decisions About Health, Wealth and Happiness
Adkisson, R. V. (2008) · The Social Science Journal · doi:10.1016/j.soscij.2008.09.003 · Cited by 5,706
Aston-Jones, G., & Cohen, J. D. (2005) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.28.061604.135709 · Cited by 4,472
Classical conditioning II: Current research and theory.
Black, A. H., & Prokasy, W. F. (1972) · Appleton-Century-Crofts eBooks · Cited by 4,449
Allais, M. (1953) · Econometrica · doi:10.2307/1907921 · Cited by 4,054
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023
Keynes, J. M. (1973) · Palgrave Macmillan UK eBooks · doi:10.1007/978-1-349-00843-8 · Cited by 2,576
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Stochastic Volatility: Likelihood Inference and Comparison with ARCH Models
Kim, S., Shepherd, N., & Chib, S. (1998) · The Review of Economic Studies · doi:10.1111/1467-937x.00050 · Cited by 2,228
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224
Discrete coding of reward probability and uncertainty by dopamine neurons
Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003) · Science · doi:10.1126/science.1077349 · Cited by 2,085
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
Craik, K. J. W. (1943) · Cited by 1,845
Uncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774
Assessment and Propagation of Model Uncertainty
Draper, D. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02015.x · Cited by 1,599
Neural systems responding to degrees of uncertainty in human decision-making
Hsu, M., Bhatt, M., Adolphs, R., Tranel, D., & Camerer, C. F. (2005) · Science · doi:10.1126/science.1115327 · Cited by 1,411
Cohen, J. D., McClure, S. M., & Yu, A. J. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2098 · Cited by 1,121
Choice, uncertainty and value in prefrontal and cingulate cortex
Rushworth, M. F. S., & Behrens, T. E. J. (2008) · Nature Neuroscience · doi:10.1038/nn2066 · Cited by 866
Human insula activation reflects risk prediction errors as well as risk
Preuschoff, K., Quartz, S. R., & Bossaerts, P. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.4286-07.2008 · Cited by 784
Uncertainty Aversion, Risk Aversion, and the Optimal Choice of Portfolio
Dow, J., & Werlang, S. R. D. C. (1992) · Econometrica · doi:10.2307/2951685 · Cited by 754
Paulus, M. P., Rogalsky, C., Simmons, A., Feinstein, J. S., & Stein, M. B. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00251-9 · Cited by 745
Neural Signatures of Economic Preferences for Risk and Ambiguity
Huettel, S. A., Stowe, C. J., Gordon, E. M., Warner, B. T., & Platt, M. L. (2006) · Neuron · doi:10.1016/j.neuron.2006.01.024 · Cited by 732
Neural differentiation of expected reward and risk in human subcortical structures
Preuschoff, K., Bossaerts, P., & Quartz, S. R. (2006) · Neuron · doi:10.1016/j.neuron.2006.06.024 · Cited by 692
Doya, K. (2008) · Nature Neuroscience · doi:10.1038/nn2077 · Cited by 642
On the Consistency of Bayes Estimates
Diaconis, P. W., & Freedman, D. A. (1986) · The Annals of Statistics · doi:10.1214/aos/1176349830 · Cited by 614
Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498
Dopamine: generalization and bonuses
Kakade, S. M., & Dayan, P. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00048-5 · Cited by 495
Huettel, S. A., Song, A. W., & McCarthy, G. (2005) · Journal of Neuroscience · doi:10.1523/jneurosci.5070-04.2005 · Cited by 475
Reward Value Coding Distinct From Risk Attitude-Related Uncertainty Coding in Human Reward Systems
Tobler, P. N., O'Doherty, J. P., Dolan, R. J., & Schultz, W. (2007) · Journal of Neurophysiology · doi:10.1152/jn.00745.2006 · Cited by 472
Bayesian theories of conditioning in a changing world
Courville, A. C., Daw, N. D., & Touretzky, D. S. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.004 · Cited by 457
Neural correlates of value, risk, and risk aversion contributing to decision making under risk
Christopoulos, G. I., Tobler, P. N., Bossaerts, P., Dolan, R. J., & Schultz, W. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2614-09.2009 · Cited by 451
Rutishauser, U., Mamelak, A. N., & Schuman, E. M. (2006) · Neuron · doi:10.1016/j.neuron.2006.02.015 · Cited by 303
Information theory, novelty and hippocampal responses: unpredicted or unpredictable?
Strange, B. A., Duggins, A., Penny, W. D., Dolan, R. J., & Friston, K. J. (2005) · Neural Networks · doi:10.1016/j.neunet.2004.12.004 · Cited by 260
Control of exploitation–exploration meta-parameter in reinforcement learning
Ishii, S., Yoshida, W., & Yoshimoto, J. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00056-4 · Cited by 214
Acknowledging Misspecification in Macroeconomic Theory
Hansen, L. P., & Sargent, T. J. (2001) · Review of Economic Dynamics · doi:10.1006/redy.2001.0132 · Cited by 211
Resolution of Uncertainty in Prefrontal Cortex
Yoshida, W., & Ishii, S. (2006) · Neuron · doi:10.1016/j.neuron.2006.05.006 · Cited by 201
Adding prediction risk to the theory of reward learning
PREUSCHOFF, K., & BOSSAERTS, P. (2007) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1390.005 · Cited by 133 · cited 1×
“There, uncertainty is monolithic: it is the expected magnitude of the prediction error [6].”
BackgroundNeural correlates of risk prediction error during reinforcement learning in humans
d'Acremont, M., Lu, Z.-L., Li, X., Van der Linden, M., & Bechara, A. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.04.096 · Cited by 94
Perceptual Set as an Artifact of Recency
EPSTEIN, W. M., & Rock, I. (1960) · The American Journal of Psychology · doi:10.2307/1419898 · Cited by 66
Vanni-Mercier, G., Mauguı̀ere, F., Isnard, J., & Dreher, J. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.5298-08.2009 · Cited by 50
Ambiguity and uncertainty in Ellsberg and Shackle
Basili, M., & Zappia, C. (2009) · Cambridge Journal of Economics · doi:10.1093/cje/bep008 · Cited by 46
Statistical Models of Conditioning
Dayan, P., & Long, T. W. (1997) · Cited by 34
Learning for non‐stationary Dirichlet processes
Quinn, A., & Kárný, M. (2007) · International Journal of Adaptive Control and Signal Processing · doi:10.1002/acs.949 · Cited by 10
Bayesian Learning in UnstableSettings: Experimental Evidence Based on the Bandit ProblemPreprint
Nestour, É. P. L. (2010) · RePEc: Research Papers in Economics · Cited by 8
Bayesian Learning in an Unstable World: Experimental Evidence Based on the Bandit ProblemPreprint
Payzan-LeNestour, É. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.1628657 · Cited by 6
Self-citation
Flagged first, then by citations
2 of the 52 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Neural computations underlying arbitration between model-based and model-free learning
Lee, S. W., Shimojo, S., & O’Doherty, J. P. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.028 · Cited by 642
Humans use directed and random exploration to solve the explore–exploit dilemma.
Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A., & Cohen, J. D. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0038199 · Cited by 580
Uncertainty in perception and the Hierarchical Gaussian Filter
Mathys, C. D., Lomakina, E. I., Daunizeau, J., Iglesias, S., Brodersen, K. H., Friston, K. J., & Stephan, K. E. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00825 · Cited by 465
PhysiCell: An open source physics-based cell simulator for 3-D multicellular systems
Ghaffarizadeh, A., Heiland, R. W., Friedman, S. H., Mumenthaler, S. M., & Macklin, P. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005991 · Cited by 486
Confidence as Bayesian probability: From neural origins to behavior
Meyniel, F., Sigman, M., & Mainen, Z. F. (2015) · Neuron · doi:10.1016/j.neuron.2015.09.039 · Cited by 349
Metadata
- Authors
- Payzan-LeNestour, E., & Bossaerts, P.
- Affiliations
- California Institute of Technology (US)
- ORCID
- Bossaerts, Peter
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2011-01-20
- Volume
- 7
- Issue
- 1
- Pages
- e1001048
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC3024253
- OpenAlex
- W2027554764
- Cited by
- 267
- References
- 52
- Citations by year
- 2026: 10 · 2025: 31 · 2024: 24 · 2023: 16 · 2022: 16 · 2021: 18 · 2020: 22 · 2019: 22 · 2018: 19 · 2017: 20 · 2016: 19 · 2015: 18 · 2014: 11 · 2013: 12 · 2012: 8
- FWCI
- 5.59
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- Funders
- National Science Foundation
- Swiss Finance Institute
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
- Topics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Keywords
- Bayesian learning, restless bandits, risk, estimation uncertainty, ambiguity, structural uncertainty, exploration, learning rate, model-free reinforcement learning, human decision-making, perceived uncertainty, ambiguity aversion
- MeSH terms
- Bayes Theorem, Probability, Risk, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
