CoChat

Comparing Bayesian and non-Bayesian accounts of human confidence reports

Open access
Adler, William T., Ma, Wei Ji · 2018 · PLoS Computational Biology

Published version of a preprint

The DOI matches its Crossref record; 107 of the 110 references it cites were found in a registry or an index; 19 of the 38 sources its claims cite were checked for support.

Abstract

Humans can meaningfully report their confidence in a perceptual or cognitive decision. It is widely believed that these reports reflect the Bayesian probability that the decision is correct, but this hypothesis has not been rigorously tested against non-Bayesian alternatives. We use two perceptual categorization tasks in which Bayesian confidence reporting requires subjects to take sensory uncertainty into account in a specific way. We find that subjects do take sensory uncertainty into account when reporting confidence, suggesting that brain areas involved in reporting confidence can access low-level representations of sensory uncertainty, a prerequisite of Bayesian inference. However, behavior is not fully consistent with the Bayesian hypothesis and is better described by simple heuristic models that use uncertainty in a non-Bayesian way. Both conclusions are robust to changes in the uncertainty manipulation, task, response modality, model comparison metric, and additional flexibility in the Bayesian model. Our results suggest that adhering to a rational account of confidence behavior may require incorporating implementational constraints.

Overview

CoChat Summary

## TL;DR

This paper tests whether human confidence reports in perceptual categorization are Bayesian, using two tasks that require subjects to use trial-to-trial sensory uncertainty. The authors find that people do use uncertainty, but their confidence behavior is better fit by simple non-Bayesian heuristic models than by Bayesian models.

## Key findings

  • In Experiment 1, the Bayes model beat the Fixed model by a summed LOO difference of 2265 [498, 4253], showing that subjects take trial-to-trial sensory uncertainty into account rather than relying only on a point estimate.
  • Adding decision noise improved the Bayesian fit by 804 [510, 1134], but the Bayesian model with decision noise (Bayes-dN) was still outperformed by heuristic models: Lin by 1398 [571, 2644] and Quad by 1167 [858, 2698].
  • The heuristic models remained better after multiple robustness checks: with non-parametric reliability–uncertainty mapping, Quad still beat Bayes-dN by 1676 [839, 2730]; with incorrect generative-model assumptions, Quad still beat Bayes-dN by 768 [399, 1144].
  • Results were consistent across the paper’s controls and replications: in Task A and Task B separately, Quad beat Bayes-dN by 581 [278, 938] and 978 [406, 1756], respectively; in a separate-response experiment, Lin beat Bayes-dN by 396 [186, 622]; and in a Task-B-only experiment, Quad beat Bayes-dN by 1361 [777, 2022].

## Methods

  • The paper ran three experiments on perceptual category judgment and confidence: Experiment 1 used a single button press for combined category-confidence reports; Experiment 2 separated category and confidence responses and added trial-by-trial feedback; Experiment 3 used Task B only.
  • Experiment 1 had 11 subjects; Experiment 2 had 8 subjects; Experiment 3 had 15 subjects. Tasks used drifting Gabors or ellipses, with reliability manipulated by contrast or ellipse elongation, and participants were trained extensively before testing.
  • The authors fit 75 models from 6 distinct model families using MCMC, compared models mainly with leave-one-out cross-validated log likelihood (LOO), and also checked AIC, BIC, AICc, and WAIC; they performed model recovery and several robustness analyses.

## Limitations

  • The paper states that, because of the short presentation time, the results may not generalize to tasks requiring integration of evidence over time.
  • The authors state that, because the stimuli are very low-level, they cannot say much about high-level stimuli like faces.
  • They also state that they only considered explicit confidence ratings, so the findings may not extend to implicit confidence measures. As evident from the text, the tasks were tightly controlled lab perceptual tasks with trained subjects and specific stimulus manipulations, so broader ecological generalization remains limited.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pcbi.1006572

Cites

107 of 107 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig 1 Task design. (a) Schematic of a test block trial. After stimulus offset, subjects reported category and confidence level with a single button press. (b) Stimulus distributions for Tasks A and B. (c) Examples of low and high reliability stimuli. Six (out of eleven) subjects saw drifting Gabors, and five subjects saw ellipses. (d) Generative model. (e) Example measurement distributions at different reliability levels. In all models (except Linear Neural), the measurement is assumed to be drawn from a Gaussian distribution centered on the true stimulus, with s.d. dependent on reliability.
Fig 1 Task design. (a) Schematic of a test block trial. After stimulus offset, subjects reported category and confidence level with a single button press. (b) Stimulus distributions for Tasks A and B. (c) Examples of low and high reliability stimuli. Six (out of eleven) subjects saw drifting Gabors, and five subjects saw ellipses. (d) Generative model. (e) Example measurement distributions at different reliability levels. In all models (except Linear Neural), the measurement is assumed to be drawn from a Gaussian distribution centered on the true stimulus, with s.d. dependent on reliability.CC BY · PubMed Central
Fig 2 Decision rules/mappings in four models. Each model corresponds to a different mapping from a measurement and uncertainty level to a category and confidence response. Colors correspond to category and confidence response, as in Fig 1a. Plots were generated using parameter values that were roughly similar to those found after fitting subject data but were chosen primarily to illustrate the different features of the models.
Fig 2 Decision rules/mappings in four models. Each model corresponds to a different mapping from a measurement and uncertainty level to a category and confidence response. Colors correspond to category and confidence response, as in Fig 1a. Plots were generated using parameter values that were roughly similar to those found after fitting subject data but were chosen primarily to illustrate the different features of the models.CC BY · PubMed Central
Fig 3 Behavioral data and fits from best model (Quad), experiment 1. Error bars represent ±1 s.e.m. across 11 subjects. Shaded regions represent ±1 s.e.m. on model fits. (a,b) Proportion “category 1” reports as a function of stimulus reliability and true category. (c,d) Mean button press as a function of stimulus reliability and true category. (e,f) Normalized histogram of confidence reports for both true categories. (g) Proportion correct category reports as a function of confidence report and task. (h,i) Mean confidence as a function of stimulus reliability and correctness. (j,k) Mean confidence as a function of stimulus orientation and reliability. (l,m) Proportion “category 1” reports as a function of stimulus orientation and reliability. (n,o) Mean button press as a function of stimulus orientation and reliability. (c,d,n,o) Vertical axis label colors correspond to button presses, as in Fig 1a. (l–o) For clarity, only 3 of 6 reliability levels are shown, although models were fit to all reliability levels.
Fig 3 Behavioral data and fits from best model (Quad), experiment 1. Error bars represent ±1 s.e.m. across 11 subjects. Shaded regions represent ±1 s.e.m. on model fits. (a,b) Proportion “category 1” reports as a function of stimulus reliability and true category. (c,d) Mean button press as a function of stimulus reliability and true category. (e,f) Normalized histogram of confidence reports for both true categories. (g) Proportion correct category reports as a function of confidence report and task. (h,i) Mean confidence as a function of stimulus reliability and correctness. (j,k) Mean confidence as a function of stimulus orientation and reliability. (l,m) Proportion “category 1” reports as a function of stimulus orientation and reliability. (n,o) Mean button press as a function of stimulus orientation and reliability. (c,d,n,o) Vertical axis label colors correspond to button presses, as in Fig 1a. (l–o) For clarity, only 3 of 6 reliability levels are shown, although models were fit to all reliability levels.CC BY · PubMed Central
Fig 4 Model fits and model comparison for models Fixed and Bayes. Bayes provides a better fit, but both models have large deviations from the data. Left and middle columns: model fits to mean button press as a function of reliability, true category, and task. Error bars represent ±1 s.e.m. across 11 subjects. Shaded regions represent ±1 s.e.m. on model fits, with each model on a separate row. Right column: LOO model comparison. Bars represent individual subject LOO scores for Bayes, relative to Fixed. Negative (leftward) values indicate that, for that subject, Bayes had a higher (better) LOO score than Fixed. Blue lines and shaded regions represent, respectively, medians and 95% CI of bootstrapped mean LOO differences across subjects. These values are equal to the summed LOO differences reported in the text divided by the number of subjects. Although we plot data as a function of the true category here, the model only takes in measurement and reliability as an input; it is not free to treat individual trials from each true category differently.
Fig 4 Model fits and model comparison for models Fixed and Bayes. Bayes provides a better fit, but both models have large deviations from the data. Left and middle columns: model fits to mean button press as a function of reliability, true category, and task. Error bars represent ±1 s.e.m. across 11 subjects. Shaded regions represent ±1 s.e.m. on model fits, with each model on a separate row. Right column: LOO model comparison. Bars represent individual subject LOO scores for Bayes, relative to Fixed. Negative (leftward) values indicate that, for that subject, Bayes had a higher (better) LOO score than Fixed. Blue lines and shaded regions represent, respectively, medians and 95% CI of bootstrapped mean LOO differences across subjects. These values are equal to the summed LOO differences reported in the text divided by the number of subjects. Although we plot data as a function of the true category here, the model only takes in measurement and reliability as an input; it is not free to treat individual trials from each true category differently.CC BY · PubMed Central
Fig 5 Model fits and model comparison for Bayes-dN and heuristic models. In both tasks, Bayes-dN fails to describe the data at high reliabilities; Lin and Quad provides a good fit at most reliabilities. Left and middle columns: as in Fig 4. Right column: bars represent individual subject LOO scores for each model, relative to Bayes-dN. Negative (leftward) values indicate that, for that subject, the model in the corresponding row had a higher (better) LOO score than Bayes-dN. Blue lines and shaded regions: as in Fig 4.
Fig 5 Model fits and model comparison for Bayes-dN and heuristic models. In both tasks, Bayes-dN fails to describe the data at high reliabilities; Lin and Quad provides a good fit at most reliabilities. Left and middle columns: as in Fig 4. Right column: bars represent individual subject LOO scores for each model, relative to Bayes-dN. Negative (leftward) values indicate that, for that subject, the model in the corresponding row had a higher (better) LOO score than Bayes-dN. Blue lines and shaded regions: as in Fig 4.CC BY · PubMed Central
Fig 6 Comparison of core models, experiment 1. Models were fit jointly to Task A and B category and confidence responses. Blue lines and shaded regions represent, respectively, medians and 95% CI of bootstrapped summed LOO differences across subjects. LOO differences for these and other models are shown in S1a Fig.
Fig 6 Comparison of core models, experiment 1. Models were fit jointly to Task A and B category and confidence responses. Blue lines and shaded regions represent, respectively, medians and 95% CI of bootstrapped summed LOO differences across subjects. LOO differences for these and other models are shown in S1a Fig.CC BY · PubMed Central
Fig 7 (a) In experiment 1, Task B, on trials in which the subject chose category 2, mean confidence increases with the absolute value of stimulus orientation. (b) The “positive evidence” in favor of category 2, however, decreases with the absolute value of stimulus orientation. This plot depicts the category-conditioned stimulus distribution p(s ∣ C = 2); positive evidence in this experiment is equivalent to the likelihood p(x ∣ C = 2), which is just p(s ∣ C = 2) convolved with the subject’s measurement noise.
Fig 7 (a) In experiment 1, Task B, on trials in which the subject chose category 2, mean confidence increases with the absolute value of stimulus orientation. (b) The “positive evidence” in favor of category 2, however, decreases with the absolute value of stimulus orientation. This plot depicts the category-conditioned stimulus distribution p(s ∣ C = 2); positive evidence in this experiment is equivalent to the likelihood p(x ∣ C = 2), which is just p(s ∣ C = 2) convolved with the subject’s measurement noise.CC BY · PubMed Central
Fig 8 Performance as a function of number of trials, for both tasks and for all experiments. Performance was computed as a moving average over test trials (200 trials wide). Shaded regions represent ±1 s.e.m. over subjects. Performance did not change significantly over the course of each experiment.
Fig 8 Performance as a function of number of trials, for both tasks and for all experiments. Performance was computed as a moving average over test trials (200 trials wide). Shaded regions represent ±1 s.e.m. over subjects. Performance did not change significantly over the course of each experiment.CC BY · PubMed Central
Fig 9 Distributions of posterior probabilities of being correct, with confidence criteria for Bayesian models with three different levels of strength. Solid lines represent the distributions of posterior probabilities for each category and task in the absence of measurement noise and sensory uncertainty. Dashed lines represent confidence criteria, generated from the mean of subject 4’s posterior distribution over parameters. Each model has a different number of sets of mappings between posterior probability and confidence report. In BayesUltrastrong, there is one set of mappings. In BayesStrong, there is one set for Task A, and another for Task B. In BayesWeak, as in the non-Bayesian models, there is one set for Task A, and one set for each reported category in Task B. Plots were generated from the mean of subject 4’s posterior distribution over parameters as in Fig 2.
Fig 9 Distributions of posterior probabilities of being correct, with confidence criteria for Bayesian models with three different levels of strength. Solid lines represent the distributions of posterior probabilities for each category and task in the absence of measurement noise and sensory uncertainty. Dashed lines represent confidence criteria, generated from the mean of subject 4’s posterior distribution over parameters. Each model has a different number of sets of mappings between posterior probability and confidence report. In BayesUltrastrong, there is one set of mappings. In BayesStrong, there is one set for Task A, and another for Task B. In BayesWeak, as in the non-Bayesian models, there is one set for Task A, and one set for each reported category in Task B. Plots were generated from the mean of subject 4’s posterior distribution over parameters as in Fig 2.CC BY · PubMed Central
Fig 10 Posterior distributions over parameter values for an example model. Each subplot represents a parameter of the model. Each colored histogram represents the sampled posterior distribution for a parameter and a subject in experiment 1, with colors consistent for each subject. The limits of the x-axis indicates the allowable range for each parameter. Black triangles indicate the overall mean parameter value.
Fig 10 Posterior distributions over parameter values for an example model. Each subplot represents a parameter of the model. Each colored histogram represents the sampled posterior distribution for a parameter and a subject in experiment 1, with colors consistent for each subject. The limits of the x-axis indicates the allowable range for each parameter. Black triangles indicate the overall mean parameter value.CC BY · PubMed Central
Fig 11 Example analysis of a bootstrapped confidence interval. (a) Uncertainty estimates for bootstrapped confidence intervals, as a function of the number of subjects included. Blue line represents the median bootstrapped mean of LOO differences, and black lines indicate the lower and upper bounds of the 95% CI. Error bars represent ±1 s.d. (b) For comparison to a, the standard style of plot used to show model comparison results (e.g., Fig 4).
Fig 11 Example analysis of a bootstrapped confidence interval. (a) Uncertainty estimates for bootstrapped confidence intervals, as a function of the number of subjects included. Blue line represents the median bootstrapped mean of LOO differences, and black lines indicate the lower and upper bounds of the 95% CI. Error bars represent ±1 s.d. (b) For comparison to a, the standard style of plot used to show model comparison results (e.g., Fig 4).CC BY · PubMed Central
Fig 12 Model recovery analysis. Shade represents the difference between the mean AIC score (across datasets) for each fitted model and for the one with the lowest mean AIC score. White squares indicate the model that had the lowest mean AIC score when fitted to data generated from each model. The squares on the diagonal indicate that the true generating model was the best-fitting model, on average, in all cases.
Fig 12 Model recovery analysis. Shade represents the difference between the mean AIC score (across datasets) for each fitted model and for the one with the lowest mean AIC score. White squares indicate the model that had the lowest mean AIC score when fitted to data generated from each model. The squares on the diagonal indicate that the true generating model was the best-fitting model, on average, in all cases.CC BY · PubMed Central

Verification

Claims the paper makes

12 Valid · 2 Not valid · 7 Not checked · 96 Own results and interpretations

75 more claims

Create a free account to see all 117 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 14
  • Probably not a substantial claim. 26
  • Borderline: one of the tests for a substantial claim is unclear. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 107 references

Research works printed but not linked: 1

  1. Orhan AE, Jacobs RA. Are performance limitations in visual short-term memory tasks due to capacity limitations of model mismatch? arXiv. 2014 Jul;.

Cited by

See all 125

Metadata

Authors
Adler, W. T., & Ma, W. J.
Affiliations
New York University (US)
Corresponding author
Adler, William T.
Venue
PLoS Computational Biology
Published
2018-11-13
Volume
14
Issue
11
Pages
e1006572
ISSN
  • 1553-734X (ISSN-L)
  • 1553-7358
OpenAlex
W2950975704
Cited by
118
References
110
Citations by year
2026: 5 · 2025: 19 · 2024: 19 · 2023: 23 · 2022: 12 · 2021: 12 · 2020: 14 · 2019: 8 · 2018: 6
FWCI
5.95
Type
journal article
Language
en
Open access
Locations
  • PLoS Computational Biology (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Funders
National Science Foundation (DGE-1342536)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
Keywords
confidence reports, Bayesian inference, sensory uncertainty, perceptual categorization, decision-making, heuristics, rational model, metacognition
MeSH terms
Adult, Bayes Theorem, Behavior, Decision Making, Female, Humans, Male, Probability, Models, Statistical, Observer Variation, Normal Distribution, Poisson Distribution, Uncertainty, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Comparing Bayesian and non-Bayesian accounts of human confidence reports. https://app.cochat.ai/papers/10.1371/journal.pcbi.1006572
The paper
Adler, W. T., & Ma, W. J. (2018). Comparing Bayesian and non-Bayesian accounts of human confidence reports. PLoS Computational Biology. https://doi.org/10.1371/journal.pcbi.1006572

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.