Comparing Bayesian and non-Bayesian accounts of human confidence reports
Open accessPublished version of a preprint
The DOI matches its Crossref record; 107 of the 110 references it cites were found in a registry or an index; 19 of the 38 sources its claims cite were checked for support.
Abstract
Humans can meaningfully report their confidence in a perceptual or cognitive decision. It is widely believed that these reports reflect the Bayesian probability that the decision is correct, but this hypothesis has not been rigorously tested against non-Bayesian alternatives. We use two perceptual categorization tasks in which Bayesian confidence reporting requires subjects to take sensory uncertainty into account in a specific way. We find that subjects do take sensory uncertainty into account when reporting confidence, suggesting that brain areas involved in reporting confidence can access low-level representations of sensory uncertainty, a prerequisite of Bayesian inference. However, behavior is not fully consistent with the Bayesian hypothesis and is better described by simple heuristic models that use uncertainty in a non-Bayesian way. Both conclusions are robust to changes in the uncertainty manipulation, task, response modality, model comparison metric, and additional flexibility in the Bayesian model. Our results suggest that adhering to a rational account of confidence behavior may require incorporating implementational constraints.
Overview
CoChat Summary
## TL;DR
This paper tests whether human confidence reports in perceptual categorization are Bayesian, using two tasks that require subjects to use trial-to-trial sensory uncertainty. The authors find that people do use uncertainty, but their confidence behavior is better fit by simple non-Bayesian heuristic models than by Bayesian models.
## Key findings
- In Experiment 1, the Bayes model beat the Fixed model by a summed LOO difference of 2265 [498, 4253], showing that subjects take trial-to-trial sensory uncertainty into account rather than relying only on a point estimate.
- Adding decision noise improved the Bayesian fit by 804 [510, 1134], but the Bayesian model with decision noise (Bayes-dN) was still outperformed by heuristic models: Lin by 1398 [571, 2644] and Quad by 1167 [858, 2698].
- The heuristic models remained better after multiple robustness checks: with non-parametric reliability–uncertainty mapping, Quad still beat Bayes-dN by 1676 [839, 2730]; with incorrect generative-model assumptions, Quad still beat Bayes-dN by 768 [399, 1144].
- Results were consistent across the paper’s controls and replications: in Task A and Task B separately, Quad beat Bayes-dN by 581 [278, 938] and 978 [406, 1756], respectively; in a separate-response experiment, Lin beat Bayes-dN by 396 [186, 622]; and in a Task-B-only experiment, Quad beat Bayes-dN by 1361 [777, 2022].
## Methods
- The paper ran three experiments on perceptual category judgment and confidence: Experiment 1 used a single button press for combined category-confidence reports; Experiment 2 separated category and confidence responses and added trial-by-trial feedback; Experiment 3 used Task B only.
- Experiment 1 had 11 subjects; Experiment 2 had 8 subjects; Experiment 3 had 15 subjects. Tasks used drifting Gabors or ellipses, with reliability manipulated by contrast or ellipse elongation, and participants were trained extensively before testing.
- The authors fit 75 models from 6 distinct model families using MCMC, compared models mainly with leave-one-out cross-validated log likelihood (LOO), and also checked AIC, BIC, AICc, and WAIC; they performed model recovery and several robustness analyses.
## Limitations
- The paper states that, because of the short presentation time, the results may not generalize to tasks requiring integration of evidence over time.
- The authors state that, because the stimuli are very low-level, they cannot say much about high-level stimuli like faces.
- They also state that they only considered explicit confidence ratings, so the findings may not extend to implicit confidence measures. As evident from the text, the tasks were tightly controlled lab perceptual tasks with trained subjects and specific stimulus manipulations, so broader ecological generalization remains limited.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
12 Valid · 2 Not valid · 7 Not checked · 96 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Recent studies have attempted to test the BCH [19, 20] but, because of their experimental designs, are unable to meaningfully distinguish the Bayesian model from any other model of confidence.”
§ Comparing Bayesian and non-Bayesian accounts of human confidence reports > Introduction
What the sources say
- Disputedcites [19] L Aitchison, 2015, Doubly Bayesian analysis of confidence in perceptual decision-making, PLoS Comput Biol sourceExperimental — accuracy not yet measured
The Introduction says that one-dimensional designs cannot distinguish heuristic from Bayes-optimal confidence, but explicitly describes this study’s multidimensional design as a way to test that distinction. Results > Model selection reports that the two-response data significantly favored the Bayesian model over the Max and Difference models. The paper therefore makes the contrary point about its own experimental design, while acknowledging that it cannot test every possible heuristic model. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
- Partly supportedcites [20] JI Sanders, 2016, Signatures of a statistical computation in the human sense of confidence, Neuron sourceExperimental — accuracy not yet measured
The Discussion reports that the study compared human confidence with predictions from a statistical confidence model, supporting its identification as an attempt to test Bayesian confidence. It also notes that other algorithms can approximate Bayes-optimal predictions, but does not say the experimental design was unable to meaningfully distinguish the Bayesian model from any other confidence model. The cited passages have section labels but no page numbers. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Because we are interested in subjects’ intrinsic computation of confidence, we did not instruct or incentivize them to assign probability ranges to each button (e.g., by using a scoring rule [34–36]).”
§ Comparing Bayesian and non-Bayesian accounts of human confidence reports > Results > Experiment 1 During each session, each subject completed two orientation categorization tasks, Tasks A and B. On each trial, a category C was selected randomly (both categories were equally probable), and a stimulus s was drawn from the corresponding stimulus distribution and displayed. The subject categorized the stimulus and simultaneously reported their confidence on a 4-point scale, with a single button press (Fig 1a). Using a single button press for choice and confidence may prevent post-choice influences on the confidence judgment ([25], but see [26]) and emphasized that confidence should reflect the observer’s perception rather than a preceding motor response. The categories were defined by normal distributions on orientation, which differed by task (Fig 1b). In Task A, the distributions had different means (±μC) and the same standard deviation (σC); leftward-tilting stimuli were more likely to be from category 1. Variants of Task A are common in decision-making studies [27]. In Task B, the distributions had the same mean (0°) and different standard deviations (σ1, σ2); stimuli around the horizontal were more likely to be from category 1. Variants of Task B are less common [28–30] but have some properties of perceptual organization tasks; for example, a subject may have to detect when a stimulus belongs to a narrow category (e.g., in which two line segments are collinear) that is embedded in a a broader category (e.g., in which two line segments are unrelated). Subjects were highly trained on the categories; during training, we only used highest-reliability stimuli, and we provided trial-to-trial category correctness feedback. Subjects were then tested with 6 different reliability levels, which were chosen randomly on each trial. During testing, correctness feedback was withheld to avoid the possibility that confidence simply reflects a learned mapping between stimulus orientation and reliability and the probability of being correct [30–33].
What the sources say
- Supportedcites [34] S Massoni, 2014, Confidence measurement in the light of signal detection theory, Front Psychol source
The citation identifies a scoring rule as an example of a way to incentivize confidence reports; the citing authors’ statement about what they did not do is their own methodological description. In “Elicitation Rules,” the source explicitly identifies and uses a Quadratic Scoring Rule, so it supports the cited example. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites [35] GW Brier, 1950, Verification of forecasts expressed in terms of probability, Monthly Weather Review doi:10.1175/1520-0493(1950)078
- Not checked — no full text of the sourcecites [36] T Gneiting, 2007, Strictly proper scoring rules, prediction, and estimation, J Am Stat Assoc source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After fitting our core models, our results did not differ substantially from experiment 1: Lin fits better than Bayes-dN by 396 [186, 622] and fits better than Fixed by 2095 [1344, 2889] (S6 Fig and S6 Table).”
§ Comparing Bayesian and non-Bayesian accounts of human confidence reports > Results > Model comparison
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This feature improves Bayes-dN’s fit by 908 [318, 1661], but it still underperforms Quad by 768 [399, 1144] (S1 Table).”
§ Comparing Bayesian and non-Bayesian accounts of human confidence reports > Results > Model comparison
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This feature improves Bayes-dN’s fit by 1310 [580, 2175], but it still underperforms Quad by 362 [162, 602] (S1 Table).”
§ Comparing Bayesian and non-Bayesian accounts of human confidence reports > Results > Model comparison
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
75 more claims
Create a free account to see all 117 claims and every source verdict.
- It may state two separate claims as one. 14
- Probably not a substantial claim. 26
- Borderline: one of the tests for a substantial claim is unclear. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Other work has revealed how people adjust their confidence criteria in the face of changing stimulus uncertainty (Adler & Ma 2018, Aitchison et al. 2015, Denison et al. 2018).”
From Metacognition and Confidence: A Review and Synthesis (2024) · cites this paper as Adler & Ma 2018
Citations
Cites
Brainard, D. H. (1997) · Spatial Vision · doi:10.1163/156856897x00357 · Cited by 19,702
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
The VideoToolbox software for visual psychophysics: transforming numbers into movies
Pelli, D. G. (1997) · Spatial Vision · doi:10.1163/156856897x00366 · Cited by 11,098
(1966) · Medical Entomology and Zoology · Cited by 10,293
Signal detection theory and psychophysics
Ingleby, J. D. (1967) · Journal of Sound and Vibration · doi:10.1016/0022-460x(67)90197-6 · Cited by 8,509
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
(2015) · doi:10.1007/s11222-016-9696-4 · Cited by 5,611 · cited 1×
“−1295 [−2051,−734] −2105 [−3353,−1281] −77 [−1881, 1321] −953 [−2922, 565] −3545 [−5177,−2401] −3799 [−5455,−2521] −361 [−675,−135] −1581 [−2477,−986] −2401 [−3750,−1551] −290 [−2170, 1090] −1172 [−3186, 358] −3549 [−5189,−2402] −4136 [−5848,−2777] 1910 [72, 3813] −847 [−2103, 660] −2361 [−3442,−1517] −3043 [−4489,−19…”
Interplay between Cortical Spreading Depolarization and Seizures
Kramer, D. R., Fujii, T., Ohiorhenuan, I., & Liu, C. Y. (2017) · Stereotactic and Functional Neurosurgery · doi:10.1159/000452841 · Cited by 3,696
An Energy Budget for Signaling in the Grey Matter of the Brain
Attwell, D., & Laughlin, S. B. (2001) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1097/00004647-200110000-00001 · Cited by 3,543 · cited 1×
“−345 [−933, 262] −3497 [−4860,−1817] −2836 [−4544,−1122] −3127 [−4549,−1575] −2449 [−4200,−969] 606 [6, 1269]”
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.2307/2291091 · Cited by 2,312
Research works printed but not linked: 1
- Orhan AE, Jacobs RA. Are performance limitations in visual short-term memory tasks due to capacity limitations of model mismatch? arXiv. 2014 Jul;.
Cited by
Suboptimality in perceptual decision making
Rahnev, D., & Denison, R. N. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000936 · Cited by 342
Metacognition and Confidence: A Review and Synthesis
Fleming, S. M. (2024) · Annual Review of Psychology · doi:10.1146/annurev-psych-022423-032425 · Cited by 256
“Other work has revealed how people adjust their confidence criteria in the face of changing stimulus uncertainty (Aitchison et al., 2015, Adler & Ma, 2018, Denison et al., 2018).”
Sources of Metacognitive Inefficiency
Shekhar, M., & Rahnev, D. (2021) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2020.10.007 · Cited by 119
“Of course, it is possible that some of them are also due to processing or resource limitations, especially when computing the probability of being correct is complex or resource demanding [75,77].”
BackgroundHumans incorporate attention-dependent uncertainty into perceptual decisions and confidence
Denison, R. N., Adler, W. T., Carrasco, M., & Ma, W. J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1717720115 · Cited by 122
“In the Bayesian model, decisions depend on the relative posterior probabilities of the two categories, leading the observer to shift his or her decision boundaries in measurement space, based on the attention condition (4, 5) (Fig.”
BackgroundMethodResultInfluentialSelf-citationAcerbi, L., Dokka, K., Angelaki, D. E., & Ma, W. J. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006110 · Cited by 111
Self-citation
Metadata
- Authors
- Adler, W. T., & Ma, W. J.
- Affiliations
- New York University (US)
- Corresponding author
- Adler, William T.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2018-11-13
- Volume
- 14
- Issue
- 11
- Pages
- e1006572
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC6258566
- OpenAlex
- W2950975704
- Cited by
- 118
- References
- 110
- Citations by year
- 2026: 5 · 2025: 19 · 2024: 19 · 2023: 23 · 2022: 12 · 2021: 12 · 2020: 14 · 2019: 8 · 2018: 6
- FWCI
- 5.95
- Type
- journal article
- Language
- en
- Update notices
- new_version (2018-11-27)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- National Science Foundation (DGE-1342536)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- confidence reports, Bayesian inference, sensory uncertainty, perceptual categorization, decision-making, heuristics, rational model, metacognition
- MeSH terms
- Adult, Bayes Theorem, Behavior, Decision Making, Female, Humans, Male, Probability, Models, Statistical, Observer Variation, Normal Distribution, Poisson Distribution, Uncertainty, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Comparing Bayesian and non-Bayesian accounts of human confidence reports. https://app.cochat.ai/papers/10.1371/journal.pcbi.1006572
- The paper
- Adler, W. T., & Ma, W. J. (2018). Comparing Bayesian and non-Bayesian accounts of human confidence reports. PLoS Computational Biology. https://doi.org/10.1371/journal.pcbi.1006572
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.