Measures of metacognition on signal-detection theoretic models
Open accessThe DOI matches its Crossref record; 54 of the 55 references it cites were found in a registry or an index.
Abstract
Analyzing metacognition, specifically knowledge of accuracy of internal perceptual, memorial, or other knowledge states, is vital for many strands of psychology, including determining the accuracy of feelings of knowing and discriminating conscious from unconscious cognition. Quantifying metacognitive sensitivity is however more challenging than quantifying basic stimulus sensitivity. Under popular signal-detection theory (SDT) models for stimulus classification tasks, approaches based on Type II receiver-operating characteristic (ROC) curves or Type II d-prime risk confounding metacognition with response biases in either the Type I (classification) or Type II (metacognitive) tasks. A new approach introduces meta-d': The Type I d-prime that would have led to the observed Type II data had the subject used all the Type I information. Here, we (a) further establish the inconsistency of the Type II d-prime and ROC approaches with new explicit analyses of the standard SDT model and (b) analyze, for the first time, the behavior of meta-d' under nontrivial scenarios, such as when metacognitive judgments utilize enhanced or degraded versions of the Type I evidence. Analytically, meta-d' values typically reflect the underlying model well and are stable under changes in decision criteria; however, in relatively extreme cases, meta-d' can become unstable. We explore bias and variance of in-sample measurements of meta-d' and supply MATLAB code for estimation in general cases. Our results support meta-d' as a useful measure of metacognition and provide rigorous methodology for its application. Our recommendations are useful for any researchers interested in assessing metacognitive accuracy.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 2×
“An important future challenge lies in integrating SDT with the increasingly influential framework of 'predictive coding' or the so-called 'Bayesian Brain' hypothesis (Rao & Ballard, 1999; Bubic, Cramon, & Schubotz, 2010; Friston, 2010; Clark, in press).”
BackgroundWhatever next? Predictive brains, situated agents, and the future of cognitive science
Clark, A. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000477 · Cited by 5,924
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 1×
“An important future challenge lies in integrating SDT with the increasingly influential framework of 'predictive coding' or the so-called 'Bayesian Brain' hypothesis (Rao & Ballard, 1999; Bubic, Cramon, & Schubotz, 2010; Friston, 2010; Clark, in press).”
BackgroundApproximate is Better than “Exact” for Interval Estimation of Binomial Proportions
Agresti, A., & Coull, B. A. (1998) · The American Statistician · doi:10.1080/00031305.1998.10480550 · Cited by 3,223 · cited 1×
“6 Such a prior, though vague, adds some information and can increase the accuracy of estimates when data are limited (Agresti & Coull, 1998; Greenland, 2006).”
BackgroundPragmatics of measuring recognition memory: Applications to dementia and amnesia.
Snodgrass, J. G., & Corwin, J. (1988) · Journal of Experimental Psychology General · doi:10.1037//0096-3445.117.1.34 · Cited by 3,089 · cited 1×
“An alternative way of dealing with outlying subjects is to add 0.5, i.e. a 'flattening constant', to every data cell for every subject (Snodgrass & Corwin, 1988).”
BackgroundInfluentialDetection theory: A user's guide
Lories, G. (1993) · Behavioural Processes · doi:10.1016/0376-6357(93)90083-4 · Cited by 2,835
A Reference Bayesian Test for Nested Hypotheses and its Relationship to the Schwarz Criterion
Kass, R. E., & Wasserman, L. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476592 · Cited by 1,253 · cited 1×
“…one observation for each of the hit and false alarm rates and corresponding miss and correct rejection rates, i.e. it is a unit information prior (Kass & Wasserman, 1995), corresponding to the belief that with approximately 95% probability all of the hit and false alarm rates lie between 0.05…”
BackgroundInfluentialRosenthal, D. M. (2005) · doi:10.1093/oso/9780198236979.001.0001 · Cited by 993
A comparison of current measures of the accuracy of feeling-of-knowing predictions
Nelson, T. O. (1984) · Psychological Bulletin · doi:10.1037/0033-2909.95.1.109 · Cited by 976
Approximate Is Better than "Exact" for Interval Estimation of Binomial Proportions
Agresti, A., & Coull, B. A. (1998) · The American Statistician · doi:10.2307/2685469 · Cited by 972
Cited by
Knowing your own heart: Distinguishing interoceptive accuracy from interoceptive awareness
Garfinkel, S. N., Seth, A. K., Barrett, A. B., Suzuki, K., & Critchley, H. D. (2015) · Biological Psychology · doi:10.1016/j.biopsycho.2014.11.004 · Cited by 1,624
“…of the degree to which accuracy of (objective) heartbeat detection is predicted by subjective confidence in the task judgement, e.g. using area under an ROC curve (Barrett et al., 2013; Fleming, Weil, Nagy, Dolan, & Rees, 2010; Galvin, Podd, Drga, & Whitmore, 2003; Garfinkel & Critchley, 2013).”
Self-citationFleming, S. M., & Lau, H. C. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00443 · Cited by 1,408
“Barrett et al. (2013) have conducted extensive normative tests of meta-d′, finding that it is robust to changes in bias and that it recovers simulated changes in metacognitive sensitivity (see also Maniscalco and Lau, 2014).”
BackgroundSelf-Evaluation of Decision-Making: A General Bayesian Framework for Metacognitive Computation
Fleming, S. M., & Daw, N. D. (2017) · Psychological Review · doi:10.1037/rev0000045 · Cited by 584
“Suboptimal or hypo-metacognitive sensitivity re-sults in values of meta-d = / d = (cid:5) 1 (Barrett, Dienes, & Seth, 2013; Maniscalco & Lau, 2014).”
Domain-specific impairment in metacognitive accuracy following anterior prefrontal lesions
Fleming, S. M., Ryu, J., Golfinos, J. G., & Blackmon, K. E. (2014) · Brain · doi:10.1093/brain/awu221 · Cited by 312
“Using this ratio as a measure of metacognitive accuracy effectively eliminates performance and response bias confounds typically affecting other measures (Barrett et al., 2013).”
BackgroundHMeta-d: hierarchical Bayesian estimation of metacognitive efficiency from confidence ratings
Fleming, S. M. (2017) · Neuroscience of Consciousness · doi:10.1093/nc/nix007 · Cited by 309
“The properties of the meta -d’ model have been thoroughly explored in previous articles (Maniscalco and Lau 2012; Barrett et al. 2013; Fleming and Lau 2014;Maniscalco and Lau 2014).”
Method
Metadata
- Authors
- Barrett, A. B., Dienes, Z., & Seth, A. K.
- Affiliations
- University of Sussex (GB)
- Venue
- Psychological Methods
- Published
- 2013
- Volume
- 18
- Issue
- 4
- Pages
- 535-552
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2006647339
- Cited by
- 155
- References
- 55
- Citations by year
- 2026: 2 · 2025: 13 · 2024: 6 · 2023: 16 · 2022: 18 · 2021: 16 · 2020: 13 · 2019: 11 · 2018: 12 · 2017: 13 · 2016: 12 · 2015: 9 · 2014: 11 · 2013: 3
- FWCI
- 6.37
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Licence
- cc-by
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Sussex Research Online (University of Sussex) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Funders
- Dr. Mortimer and Dame Theresa Sackler Foundation
- Engineering and Physical Sciences Research Council (EP/G007543/1)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- metacognition, signal detection theory, metacognitive sensitivity, response bias, metacognitive judgments, metacognitive accuracy, metacognitive confidence, metacognitive monitoring
- MeSH terms
- Cognition, Data Interpretation, Statistical, Humans, Models, Psychological, Signal Detection, Psychological
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.