How experimental procedures influence estimates of metacognitive ability
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
It is becoming widely appreciated that higher stimulus sensitivity trivially increases estimates of metacognitive sensitivity. Therefore, meaningful comparisons of metacognitive ability across conditions and observers necessitates equating stimulus sensitivity. To achieve this, one common approach is to use a continuous staircase that runs throughout the duration of the experiment under the assumption that this procedure has no influence on the estimated metacognitive ability. Here we critically examine this assumption. Using previously published data, we find that, compared to using a single level of stimulus contrast, staircase techniques lead to inflated estimates of metacognitive ability across a wide variety of measures including area under the type 2 ROC curve, the confidence-accuracy correlation phi, meta-d′, meta-d′/d′, and meta-d′–d′. Furthermore, this metacognitive inflation correlates with the degree of stimulus variability experienced by each subject. These results suggest that studies using a staircase approach are likely to report inflated estimates of metacognitive ability. Furthermore, we argue that similar inflation likely occurs in the presence of variability in task difficulty caused by other factors such as fluctuations in alertness or gradual improvement on the task. We offer practical solutions to these issues, both in the design and analysis of metacognition experiments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fleming, S. M., & Lau, H. C. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00443 · Cited by 1,408 · cited 2×
“ures of “metacognitive efficiency” (Fleming and Lau 2014) such as meta-d0/d0.”
Self-citationManiscalco, B., & Lau, H. (2012) · Consciousness and Cognition · doi:10.1016/j.concog.2011.09.021 · Cited by 883
Relating Introspective Accuracy to Individual Differences in Brain Structure
Fleming, S. M., Weil, R. S., Nagy, Z., Dolan, R. J., & Rees, G. (2010) · Science · doi:10.1126/science.1191883 · Cited by 809
Self-citationConfidence in value-based choice
De Martino, B., Fleming, S. M., Garrett, N., & Dolan, R. J. (2013) · Nature Neuroscience · doi:10.1038/nn.3279 · Cited by 534
Self-citationGalvin, S. J., Podd, J. V., Drga, V., & Whitmore, J. (2003) · Psychonomic Bulletin & Review · doi:10.3758/bf03196546 · Cited by 405
Rouault, M., Seow, T., Gillan, C. M., & Fleming, S. M. (2018) · Biological Psychiatry · doi:10.1016/j.biopsych.2017.12.017 · Cited by 393
Self-citationPrefrontal Contributions to Metacognition in Perceptual Decision Making
Fleming, S. M., Huijgen, J., & Dolan, R. J. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.6489-11.2012 · Cited by 391
Self-citationMetacognition: computation, biology and function
Fleming, S. M., Dolan, R. J., & Frith, C. D. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2012.0021 · Cited by 344
Self-citationSuboptimality in perceptual decision making
Rahnev, D., & Denison, R. N. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000936 · Cited by 342 · cited 2×
“Metacognitive noise is noise that affects confidence estimates but not perceptual decisions (Mueller and Weidemann 2008; Jang et al. 2012; De Martino et al. 2013; Maniscalco and Lau 2016; Rahnev et al. 2016; van den Berg et al. 2017; Shekhar and Rahnev 2018).”
Self-citationTransfer of metacognitive skills and hint seeking in monkeys
Kornell, N., Son, L. K., & Terrace, H. S. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01850.x · Cited by 341 · cited 1×
“Second, we computed phi (Kornell et al. 2007): the Pearson correlation between confidence and accuracy.”
Cited by
How Local and Global Metacognition Shape Mental Health
Seow, T. X. F., Rouault, M., Gillan, C. M., & Fleming, S. M. (2021) · Biological Psychiatry · doi:10.1016/j.biopsych.2021.05.013 · Cited by 159
“, (8)], although this method has caveats (21).”
MethodSelf-citationModeling perceptual confidence and the confidence forced-choice paradigm.
Mamassian, P., & de Gardelle, V. (2022) · Psychological Review · doi:10.1037/rev0000312 · Cited by 70
“Failure to meet this assumption leads to overestimations of metacognitive sensitivity (see e.g. Rahnev & Fleming, 2019), because participants could be using variations of performance that cannot be used in the meta-d’ estimation procedure.”
BackgroundMeasuring metacognitive performance: type 1 performance dependence and test-retest reliability
Guggenmos, M. (2021) · Neuroscience of Consciousness · doi:10.1093/nc/niab040 · Cited by 66
The future of metacognition research: Balancing construct breadth with measurement rigor
Katyal, S., & Fleming, S. M. (2024) · Cortex · doi:10.1016/j.cortex.2023.11.002 · Cited by 63
“In what follows, we suggest approaches to redressing this balance.”
BackgroundMethodSelf-citationCalibration in consciousness science
Michel, M. (2023) · Erkenntnis · doi:10.1007/s10670-021-00383-z · Cited by 62
Metadata
- Authors
- Rahnev, D., & Fleming, S. M.
- Venue
- Neuroscience of Consciousness
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-01-01
- Volume
- 2019
- Issue
- 1
- Pages
- niz009-niz009
- ISSN
- 2057-2107 (ISSN-L)
- OpenAlex
- W2948368838
- Cited by
- 69
- References
- 19
- FWCI
- 3.92
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How experimental procedures influence estimates of metacognitive ability. https://app.cochat.ai/papers/10.1093/nc/niz009
- The paper
- Rahnev, D., & Fleming, S. M. (2019). How experimental procedures influence estimates of metacognitive ability. Neuroscience of Consciousness. https://doi.org/10.1093/nc/niz009
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.