CoChat
Sign Up
Cited by 66Memory Processes and InfluencesNeural and Behavioral Psychology StudiesVisual perception and processing mechanisms

Measuring metacognitive performance: type 1 performance dependence and test-retest reliability

Open access
Guggenmos, Matthias·2021·Neuroscience of Consciousness
DOI 10.1093/nc/niab040PMC PMC8633424

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Research on metacognition—thinking about thinking—has grown rapidly and fostered our understanding of human cognition in healthy individuals and clinical populations. Of central importance is the concept of metacognitive performance, which characterizes the capacity of an individual to estimate and report the accuracy of primary (type 1) cognitive processes or actions ensuing from these processes. Arguably one of the biggest challenges for measures of metacognitive performance is their dependency on objective type 1 performance, although more recent methods aim to address this issue. The present work scrutinizes the most popular metacognitive performance measures in terms of two critical characteristics: independence of type 1 performance and test-retest reliability. Analyses of data from the Confidence Database (total N = 6912) indicate that no current metacognitive performance measure is independent of type 1 performance. The shape of this dependency is largely reproduced by extending current models of metacognition with a source of metacognitive noise. Moreover, the reliability of metacognitive performance measures is highly sensitive to the combination of type 1 performance and trial number. Importantly, trial numbers frequently employed in metacognition research are too low to achieve an acceptable level of test-retest reliability. Among common task characteristics, simultaneous choice and confidence reports most strongly improved reliability. Finally, general recommendations about design choices and analytical remedies for studies investigating metacognitive performance are provided.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/nc/niab040

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The assumption that meta-d'/d' is fully independent of metacognitive bias and performance has been challenged.

    “However, the assumption that meta-d ′ /d ′ is fully independent of metacognitive bias and performance has been challenged (Guggenmos 2021, Xue et al. 2021).”

    From Metacognition and Confidence: A Review and Synthesis (2024) · cites this paper as Guggenmos 2021

    The Results section reports that meta-dʹ/dʹ (Mratio) is largely stable across type 1 performance levels but varies at some levels of metacognitive noise and type 1 performance, challenging full performance independence. The paper does not establish that Mratio depends on metacognitive bias; its Introduction discusses bias insensitivity for a different measure, AUROC2. No page numbers are available in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. Bahrami B, Olsen K, Bang D et al. What failure in collective decision-making tells us about metacognition. Philos Trans R Soc B Biol Sci 2012; 367: 1350– 65. PMC3318766 22492752
  2. Baird B, Cieslak M, Smallwood J et al. Regional white matter variation associated with domain-specific metacognitive accuracy. J Cogn Neurosci 2015; 27: 440– 52. 25313660
  3. Baird B, Mrazek MD, Phillips DT et al. Domain-specific enhancement of metacognitive ability following meditation training. J Exp Psychol Gen 2014; 143: 1972– 9. 24820248
  4. Baird B, Smallwood J, Gorgolewski KJ et al. Medial and lateral networks in anterior prefrontal cortex support metacognitive ability for memory and perception. J Neurosci 2013; 33: 16657– 65. 24133268 PMC6618531
  5. Bang JW, Shekhar M, Rahnev D. Sensory noise increases metacognitive efficiency. J Exp Psychol Gen 2019; 148: 437– 52. 30382720
  6. Clarke FR, Birdsall TG, Tanner WP. Two types of ROC curves and definitions of parameters. J Acoust Soc Am 1959; 31: 629– 30.
  7. Critchfield TS. Signal-detection properties of verbal self-reports. J Exp Anal Behav 1993; 60: 495– 514. 8283146 PMC1322161
  8. Faivre N, Roger M, Pereira M et al. Confidence in visual motion discrimination is preserved in individuals with schizophrenia. J Psychiatry Neurosci 2020; 45: 200022. PMC7955841 33009905
  9. Filevich E, Koß C, Faivre N. Response-related signals increase confidence but not metacognitive performance. eNeuro 2020; 7.doi: 10.1523/ENEURO.0326-19.2020. PMC7240286 32327471 · DOI
  10. Fitzgerald LM, Arvaneh M, Dockree PM. Domain-specific and domain-general processes underlying metacognitive judgments. Conscious Cogn 2017; 49: 264– 77. 28222381
  11. Fleming SM. HMeta-d: hierarchical Bayesian estimation of metacognitive efficiency from confidence ratings Neurosci Conscious 2017; 2017: 1– 14. PMC5858026 29877507
  12. Fleming SM, Dolan RJ, Frith CD. Metacognition: computation, biology and function. Philos Trans R Soc B Biol Sci 2012; 367: 1280– 6. PMC3318771 22492746
  13. Galvin SJ, Podd JV, Drga V et al. Type 2 tasks in the theory of signal detectability: discrimination between correct and incorrect decisions. Psychon Bull Rev 2003; 10: 843– 76. 15000533
  14. García-Pérez MA. Forced-choice staircases with fixed step sizes: asymptotic and small-sample properties. Vision Res 1998; 38: 1861– 81. 9797963
  15. Harrison OK, Garfinkel SN, Marlow L et al. The Filter Detection Task for measurement of breathing-related interoception and metacognition. Neuroscience 2020. http://biorxiv.org/lookup/doi/10.1101/2020.06.29.176941 (2 September 2021, date last accessed). 34487805 · DOI
  16. Hauser TU, Allen M, Consortium NSPN et al. Metacognitive impairments extend perceptual decision making weaknesses in compulsivity. Sci Rep 2017; 7: 6614. PMC5529539 28747627
  17. Hertz U, Bell V, Barnby J et al. The communication of metacognition for social strategy in psychosis: an exploratory study. Schizophr Bull Open 2020; 1:sgaa058.doi: 10.1093/schizbullopen/sgaa058. · DOI
  18. Hosseini J, Ferrell WR. Detectability of correctness: a measure of knowing that one knows. Instr Sci 1982; 11: 113– 27.
  19. Jansen RA, Rafferty AN, Griffiths TL. A rational model of the Dunning–Kruger effect supports insensitivity to evidence in low performers. Nat Hum Behav 2021; 5: 756– 63. 33633375
  20. Konishi M, Compain C, Berberian B et al. Resilience of perceptual metacognition in a dual-task paradigm. Psychon Bull Rev 2020; 27: 1259– 68. 32705620
  21. Lee ALF, Ruby E, Giles N et al. Cross-domain association in metacognitive efficiency depends on first-order task types. Front Psychol 2018; 9: 2464. PMC6288301 30564183
  22. Maniscalco B, Lau H. A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings. Conscious Cogn 2012; 21: 422– 30. 22071269
  23. Maniscalco B Lau H. Signal detection theory analysis of type 1 and type 2 data: meta-d’, response- specific meta-d’, and the unequal variance SDT model. In: Fleming SM, Frith CD (eds), The Cognitive Neuroscience of Metacognition. Basel: Springer-Verlag Publishing, 2014, 25– 66.doi: 10.1007/978-3-642-45190-4_3. · DOI
  24. Maniscalco B, Lau H. Manipulation of working memory contents selectively impairs metacognitive sensitivity in a concurrent visual discrimination task. Neurosci Conscious 2015; 2015. https://academic.oup.com/nc/article/doi/10.1093/nc/niv002/2757110 (24 November 2020, date last accessed). PMC5989484 29877509 · DOI
  25. Mazor M, Fleming SM. The Dunning–Kruger effect revisited. Nat Hum Behav 2021; 5: 677– 8. 33833426
  26. Mei N, Rankine S, Olafsson E et al. Similar history biases for distinct prospective decisions of self-performance. Sci Rep 2020; 10: 5854. PMC7125132 32246029
  27. Odegaard B, Chang MY, Lau H et al. Inflation versus filling-in: why we feel we see more than we actually do in peripheral vision. Philos Trans R Soc B Biol Sci 2018; 373: 20170345. PMC6074087 30061459
  28. Ordin M, Polyanskaya L. The role of metacognition in recognition of the content of statistical learning. Psychon Bull Rev 2021; 28: 333– 40. 32869190
  29. Ordin M, Polyanskaya L, Soto D. Metacognitive processing in language learning tasks is affected by bilingualism. J Exp Psychol Learn Mem Cogn 2020; 46: 529– 38. 31282726
  30. Pollack I. On indices of signal and response discriminability. J Acoust Soc Am 1959; 31: 1031.
  31. Rahnev D, Fleming SM. How experimental procedures influence estimates of metacognitive ability. Neurosci Conscious 2019; 5: 1– 9. PMC6556214 31198586
  32. Reyes G, Vivanco-Carlevari A, Medina F et al. Hydrocortisone decreases metacognitive efficiency independent of perceived stress. Sci Rep 2020; 10: 14100. PMC7445749 32839468
  33. Rouault M, Seow T, Gillan CM et al. Psychiatric symptom dimensions are associated with dissociable shifts in metacognition but not task performance. Biol Psychiatry 2018; 84: 443– 51. 29458997 PMC6117452
  34. Rounis E, Maniscalco B, Rothwell JC et al. Theta-burst transcranial magnetic stimulation to the prefrontal cortex impairs metacognitive visual awareness. Cogn Neurosci 2010; 1: 165– 75. 24168333
  35. Sadeghi S, Ekhtiari H, Bahrami B et al. Metacognitive deficiency in a perceptual but not a memory task in methadone maintenance patients. Sci Rep 2017; 7: 7052. PMC5539220 28765624
  36. Samaha J, Postle BR. Correlated individual differences suggest a common mechanism underlying metacognition in visual perception and visual short-term memory. Proc R Soc B Biol Sci 2017; 284: 10. PMC5719179 29167365
  37. Shapiro SS, Wilk MB. An analysis of variance test for normality (complete samples). Biometrika 1965; 52: 591– 611.
  38. Shekhar M, Rahnev D. Distinguishing the roles of dorsolateral and anterior PFC in visual metacognition. J Neurosci 2018; 38: 5078– 87. 29720553 PMC6705938
  39. Shekhar M, Rahnev D. The nature of metacognitive inefficiency in perceptual decision making. Psychol Rev 2021; 128: 45– 70. 32673034 PMC7883626
  40. Ye Q, Zou F, Dayan M et al. Individual susceptibility to TMS affirms the precuneal role in meta-memory upon recollection. Brain Struct Funct 2019; 224: 2407– 19. 31254060

Cited by

Sorted by citations

Authors
Guggenmos, M.
Affiliations
Humboldt-Universität zu Berlin (DE)
Corresponding author
Guggenmos, Matthias
ORCID
Guggenmos, Matthias
Venue
Neuroscience of Consciousness
Publisher
Oxford University Press (OUP)
Published
2021-02-10
Volume
2021
Issue
1
Pages
niab040-niab040
ISSN
2057-2107 (ISSN-L)
DOI
10.1093/nc/niab040
PMC
PMC8633424
OpenAlex
W3216979255
Cited by
66
References
7
Citations by year
2026: 8 · 2025: 16 · 2024: 12 · 2023: 20 · 2022: 8 · 2021: 2
FWCI
5.52
Type
journal article
Language
en
Open access
Licence
Locations
  • Neuroscience of Consciousness (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Refubium (Universitätsbibliothek der Freien Universität Berlin) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion)
  • Universitätsbibliothek der FU Berlin Hochschulschriftenstelle u. Dokumentenserver (cc-by)
Funders
Deutsche Forschungsgemeinschaft (GU 1845/1-1)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
Keywords
metacognition, test-retest reliability, confidence reports, metacognitive confidence, metacognitive model, simultaneous choice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review