Measures of metacognition on signal-detection theoretic models
Open accessThe DOI matches its Crossref record; 54 of the 55 references it cites were found in a registry or an index.
Abstract
Analyzing metacognition, specifically knowledge of accuracy of internal perceptual, memorial, or other knowledge states, is vital for many strands of psychology, including determining the accuracy of feelings of knowing and discriminating conscious from unconscious cognition. Quantifying metacognitive sensitivity is however more challenging than quantifying basic stimulus sensitivity. Under popular signal-detection theory (SDT) models for stimulus classification tasks, approaches based on Type II receiver-operating characteristic (ROC) curves or Type II d-prime risk confounding metacognition with response biases in either the Type I (classification) or Type II (metacognitive) tasks. A new approach introduces meta-d': The Type I d-prime that would have led to the observed Type II data had the subject used all the Type I information. Here, we (a) further establish the inconsistency of the Type II d-prime and ROC approaches with new explicit analyses of the standard SDT model and (b) analyze, for the first time, the behavior of meta-d' under nontrivial scenarios, such as when metacognitive judgments utilize enhanced or degraded versions of the Type I evidence. Analytically, meta-d' values typically reflect the underlying model well and are stable under changes in decision criteria; however, in relatively extreme cases, meta-d' can become unstable. We explore bias and variance of in-sample measurements of meta-d' and supply MATLAB code for estimation in general cases. Our results support meta-d' as a useful measure of metacognition and provide rigorous methodology for its application. Our recommendations are useful for any researchers interested in assessing metacognitive accuracy.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 2×
“An important future challenge lies in integrating SDT with the increasingly influential framework of 'predictive coding' or the so-called 'Bayesian Brain' hypothesis (Rao & Ballard, 1999; Bubic, Cramon, & Schubotz, 2010; Friston, 2010; Clark, in press).”
BackgroundWhatever next? Predictive brains, situated agents, and the future of cognitive science
Clark, A. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000477 · Cited by 5,924
Rao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 1×
“An important future challenge lies in integrating SDT with the increasingly influential framework of 'predictive coding' or the so-called 'Bayesian Brain' hypothesis (Rao & Ballard, 1999; Bubic, Cramon, & Schubotz, 2010; Friston, 2010; Clark, in press).”
BackgroundApproximate is Better than “Exact” for Interval Estimation of Binomial Proportions
Agresti, A., & Coull, B. A. (1998) · The American Statistician · doi:10.1080/00031305.1998.10480550 · Cited by 3,223 · cited 1×
“6 Such a prior, though vague, adds some information and can increase the accuracy of estimates when data are limited (Agresti & Coull, 1998; Greenland, 2006).”
BackgroundPragmatics of measuring recognition memory: Applications to dementia and amnesia.
Snodgrass, J. G., & Corwin, J. (1988) · Journal of Experimental Psychology General · doi:10.1037//0096-3445.117.1.34 · Cited by 3,089 · cited 1×
“An alternative way of dealing with outlying subjects is to add 0.5, i.e. a 'flattening constant', to every data cell for every subject (Snodgrass & Corwin, 1988).”
BackgroundInfluentialDetection theory: A user's guide
Lories, G. (1993) · Behavioural Processes · doi:10.1016/0376-6357(93)90083-4 · Cited by 2,835
A Reference Bayesian Test for Nested Hypotheses and its Relationship to the Schwarz Criterion
Kass, R. E., & Wasserman, L. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476592 · Cited by 1,253 · cited 1×
“…one observation for each of the hit and false alarm rates and corresponding miss and correct rejection rates, i.e. it is a unit information prior (Kass & Wasserman, 1995), corresponding to the belief that with approximately 95% probability all of the hit and false alarm rates lie between 0.05…”
BackgroundInfluentialRosenthal, D. M. (2005) · doi:10.1093/oso/9780198236979.001.0001 · Cited by 993
A comparison of current measures of the accuracy of feeling-of-knowing predictions
Nelson, T. O. (1984) · Psychological Bulletin · doi:10.1037/0033-2909.95.1.109 · Cited by 976
Approximate Is Better than "Exact" for Interval Estimation of Binomial Proportions
Agresti, A., & Coull, B. A. (1998) · The American Statistician · doi:10.2307/2685469 · Cited by 972
Maniscalco, B., & Lau, H. (2012) · Consciousness and Cognition · doi:10.1016/j.concog.2011.09.021 · Cited by 883
Relating Introspective Accuracy to Individual Differences in Brain Structure
Fleming, S. M., Weil, R. S., Nagy, Z., Dolan, R. J., & Rees, G. (2010) · Science · doi:10.1126/science.1191883 · Cited by 809
The neural basis of metacognitive ability
Fleming, S. M., & Dolan, R. J. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2011.0417 · Cited by 747 · cited 1×
“One useful avenue to integrating consciousness theories with SDT will be to identify the extent to which separable brain networks subserve type I and type II discriminations (Fleming, Weil, Nagy, Dolan, & Rees, 2010; Fleming & Dolan, 2012).”
BackgroundA theory of implicit and explicit knowledge
Dienes, Z., & Perner, J. (1999) · Behavioral and Brain Sciences · doi:10.1017/s0140525x99002186 · Cited by 672
Self-citationPrediction, cognition and the brain
Bubic (2010) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2010.00025 · Cited by 647 · cited 1×
“An important future challenge lies in integrating SDT with the increasingly influential framework of 'predictive coding' or the so-called 'Bayesian Brain' hypothesis (Rao & Ballard, 1999; Bubic, Cramon, & Schubotz, 2010; Friston, 2010; Clark, in press).”
BackgroundLanguage, Thought and Consciousness: An Essay in Philosophical Psychology
Carruthers, P. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511583360 · Cited by 554
Relative blindsight in normal observers and the neural correlate of visual consciousness
Lau, H. C., & Passingham, R. E. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0607716103 · Cited by 518 · cited 1×
“For example, in consciousness research, metacognition is often interpreted as reflecting conscious processing (Kolb & Braun, 1995; Lau & Passingham, 2006; Szczepanowski & Pessoa, 2007; Rounis et al., 2010; Dienes & Perner, 1999; Dienes & Scott, 2005; Persaud, McLeod, & Cowey, 2007).”
BackgroundMeasuring consciousness: relating behavioural and neurophysiological approaches
Seth, A. K., Dienes, Z., Cleeremans, A., Overgaard, M., & Pessoa, L. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.04.008 · Cited by 458 · cited 1×
“All theories of consciousness rely either explicitly or implicitly on subjective reports as data, and delineating the boundaries between what happens implicitly and what is (reportably) conscious provides important constraints on any theory (Seth et al., 2008).”
BackgroundSelf-citationPost-decision wagering objectively measures awareness
Persaud, N., McLeod, P., & Cowey, A. (2007) · Nature Neuroscience · doi:10.1038/nn1840 · Cited by 454
Consciousness lost and found: a neuropsychological exploration
(1998) · Choice Reviews Online · doi:10.5860/choice.35-3011 · Cited by 429
Rounis, E., Maniscalco, B., Rothwell, J. C., Passingham, R. E., & Lau, H. (2010) · Cognitive Neuroscience · doi:10.1080/17588921003632529 · Cited by 407
The role of metacognition in human social interactions
Frith, C. D. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2012.0123 · Cited by 407
Galvin, S. J., Podd, J. V., Drga, V., & Whitmore, J. (2003) · Psychonomic Bulletin & Review · doi:10.3758/bf03196546 · Cited by 405 · cited 6×
“Previously, this outcome had been shown to be impossible given the assumption that F = f (Galvin et al., 2003); our results show that this result does not generalize to the more general case of F = f .”
BackgroundMethodResultInfluentialBayesian perspectives for epidemiological research: I. Foundations and basic methods
Greenland, S. (2006) · International Journal of Epidemiology · doi:10.1093/ije/dyi312 · Cited by 383 · cited 1×
“6 Such a prior, though vague, adds some information and can increase the accuracy of estimates when data are limited (Agresti & Coull, 1998; Greenland, 2006).”
BackgroundRelations between Prototype, Exemplar, and Decision Bound Models of Categorization
Ashby, F. G., & Maddox, W. T. (1993) · Journal of Mathematical Psychology · doi:10.1006/jmps.1993.1023 · Cited by 361
Measuring unconscious knowledge: distinguishing structural knowledge and judgment knowledge
Dienes, Z., & Scott, R. (2005) · Psychological Research Psychologische Forschung · doi:10.1007/s00426-004-0208-3 · Cited by 308 · cited 1×
“For example, in consciousness research, metacognition is often interpreted as reflecting conscious processing (Kolb & Braun, 1995; Lau & Passingham, 2006; Szczepanowski & Pessoa, 2007; Rounis et al., 2010; Dienes & Perner, 1999; Dienes & Scott, 2005; Persaud, McLeod, & Cowey, 2007).”
BackgroundSelf-citationConfidence and accuracy of near-threshold discrimination responses
Kunimoto, C., Miller, J., & Pashler, H. (2001) · Consciousness and Cognition · doi:10.1006/ccog.2000.0494 · Cited by 291
Dissociable prior influences of signal probability and relevance on visual contrast sensitivity
Wyart, V., Nobre, A. C., & Summerfield, C. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1120118109 · Cited by 268
Blindsight in normal observers
Kolb, F. C., & Braun, J. (1995) · Nature · doi:10.1038/377336a0 · Cited by 198 · cited 1×
“For example, in consciousness research, metacognition is often interpreted as reflecting conscious processing (Kolb & Braun, 1995; Lau & Passingham, 2006; Szczepanowski & Pessoa, 2007; Rounis et al., 2010; Dienes & Perner, 1999; Dienes & Scott, 2005; Persaud, McLeod, & Cowey, 2007).”
BackgroundSpatial attention increases performance but not subjective confidence in a discrimination task
Wilimzig, C., Tsuchiya, N., Fahle, M., Einhäuser, W., & Koch, C. (2008) · Journal of Vision · doi:10.1167/8.5.7 · Cited by 196
Masson, M. E. J., & Rotello, C. M. (2009) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/a0014876 · Cited by 173
A higher order Bayesian decision theory of consciousness
Lau, H. C. (2007) · Progress in Brain Research · doi:10.1016/s0079-6123(07)68004-2 · Cited by 172
Two Types of ROC Curves and Definitions of Parameters
Clarke, F. R., Birdsall, T. G., & Tanner, W. P. (1959) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907764 · Cited by 168
Trends and Prospects in Metacognition Research
(2010) · Springer eBooks · doi:10.1007/978-1-4419-6546-2 · Cited by 163 · cited 1×
“…(type I) performance is a key challenge in any situation where metacognitive accuracy is important to assess in cognitive, developmental, social, educational, comparative, or abnormal psychology (Macmillan & Creelman, 2005, Beran, Brandl, Perner & Proust, 2012, Efklides & Misailidi 2010).”
BackgroundDienes, Z., & Seth, A. (2010) · Consciousness and Cognition · doi:10.1016/j.concog.2009.09.009 · Cited by 158
Self-citationSignal detection with criterion noise: Applications to recognition memory.
Benjamin, A. S., Diaz, M., & Wee, S. (2009) · Psychological Review · doi:10.1037/a0014351 · Cited by 153
Reconciling predictive coding and biased competition models of cortical function
Spratling, M. W. (2008) · Frontiers in Computational Neuroscience · doi:10.3389/neuro.10.004.2008 · Cited by 153 · cited 2×
“Although in some situations predictive coding and evidence accumulation are mathematically equivalent (Spratling, 2008), when considered from the perspective of SDT there remain many possible ways in which top-down expectations may shape evidence distributions at type I and type II levels (Turner,…”
BackgroundKanai, R., Walsh, V., & Tseng, C. (2010) · Consciousness and Cognition · doi:10.1016/j.concog.2010.06.003 · Cited by 146 · cited 1×
“Interestingly, when applied to a perceptual detection task these measures could dissect differences in metacognition separately for 'seen' and 'unseen' trials (c.f. Kanai et al., 2010).”
Decision noise: An explanation for observed violations of signal detection theory
Mueller, S. T., & Weidemann, C. T. (2008) · Psychonomic Bulletin & Review · doi:10.3758/pbr.15.3.465 · Cited by 136
Hamm, J. A., Renard, S., Fogley, R. L., Leonhardt, B. L., Dimaggio, G., Buck, K. D., & Lysaker, P. H. (2012) · Journal of Clinical Psychology · doi:10.1002/jclp.21906 · Cited by 136
Mazzoni, G., Scoboria, A., & Harvey, L. (2010) · Psychological Science · doi:10.1177/0956797610379865 · Cited by 128
Assumptions of subjective measures of unconscious mental states: higher order thoughts and bias
Dienes, Z. P. (2004) · Figshare · Cited by 114
Self-citationThe relationships between monitoring, regulation and performance
Koriat, A. (2012) · Learning and Instruction · doi:10.1016/j.learninstruc.2012.01.002 · Cited by 101
Ko, Y., & Lau, H. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2011.0380 · Cited by 96 · cited 9×
“However, in experimental data, departure from this behaviour has been observed, most notably with meta-d ′ less than d ′ , interpreted as signalling imperfect metacognition (Maniscalco & Lau, 2012; Rounis et al., 2010).”
BackgroundMethodInfluentialFear perception: Can objective and subjective awareness measures be dissociated?
Szczepanowski, R., & Pessoa, L. (2007) · Journal of Vision · doi:10.1167/7.4.10 · Cited by 92 · cited 1×
“For example, in consciousness research, metacognition is often interpreted as reflecting conscious processing (Kolb & Braun, 1995; Lau & Passingham, 2006; Szczepanowski & Pessoa, 2007; Rounis et al., 2010; Dienes & Perner, 1999; Dienes & Scott, 2005; Persaud, McLeod, & Cowey, 2007).”
BackgroundGetting technical about awareness
Clifford, C. W. G., Arabzadeh, E., & Harris, J. A. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.11.009 · Cited by 73 · cited 1×
“Clifford et al. (2008) proposed assessing metacognitive sensitivity by comparing the standard type I ROC curve with alternative type I ROC curves derived from confidence ratings.”
MethodA dynamic stimulus-driven model of signal detection.
Turner, B. M., Zandt, T. V., & Brown, S. D. (2011) · Psychological Review · doi:10.1037/a0025191 · Cited by 71
Evaluation of a 'bias-free' measure of awareness
Azzopardi, P., & Evans, S. (2007) · Spatial Vision · doi:10.1163/156856807779369742 · Cited by 68 · cited 4×
“…three distinct approaches to measuring metacognitive sensitivity within the framework of SDT. Corroborating previous analyses (Galvin et al., 2003; Evans & Azzopardi, 2007), we found that the type II d-prime and ROC curve approaches both risk confounding metacognition with (types I and II)…”
BackgroundResultInfluentialAnimal Metacognition: Problems and Prospects
Smith, J. D., Beran, M. J., Couchman, J. J., Coutinho, M. V. C., & Boomer, J. B. (2009) · Comparative Cognition & Behavior Reviews · doi:10.3819/ccbr.2009.40004 · Cited by 68
Barttfeld, P., Wicker, B., McAleer, P., Belin, P., Cojan, Y., Graziano, M., Leiguarda, R., & Sigman, M. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1301353110 · Cited by 41
Imagining what might be: Why children underestimate uncertainty
Beck, S. R., McColgan, K. L., Robinson, E. J., & Rowley, M. G. (2011) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2011.06.010 · Cited by 39
On the measurement of criterion noise in signal detection theory: The case of recognition memory.
Kellen, D., Klauer, K. C., & Singmann, H. (2012) · Psychological Review · doi:10.1037/a0027727 · Cited by 35 · cited 1×
“The Supplemental Material also presents analyses of a model with criterion jitter (Kellen et al., 2012).”
MethodLueddeke, S. E., & Higham, P. A. (2011) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2011.584631 · Cited by 18
(2005)
Cited by
Knowing your own heart: Distinguishing interoceptive accuracy from interoceptive awareness
Garfinkel, S. N., Seth, A. K., Barrett, A. B., Suzuki, K., & Critchley, H. D. (2015) · Biological Psychology · doi:10.1016/j.biopsycho.2014.11.004 · Cited by 1,624
“…of the degree to which accuracy of (objective) heartbeat detection is predicted by subjective confidence in the task judgement, e.g. using area under an ROC curve (Barrett et al., 2013; Fleming, Weil, Nagy, Dolan, & Rees, 2010; Galvin, Podd, Drga, & Whitmore, 2003; Garfinkel & Critchley, 2013).”
Self-citationFleming, S. M., & Lau, H. C. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00443 · Cited by 1,408
“Barrett et al. (2013) have conducted extensive normative tests of meta-d′, finding that it is robust to changes in bias and that it recovers simulated changes in metacognitive sensitivity (see also Maniscalco and Lau, 2014).”
BackgroundSelf-Evaluation of Decision-Making: A General Bayesian Framework for Metacognitive Computation
Fleming, S. M., & Daw, N. D. (2017) · Psychological Review · doi:10.1037/rev0000045 · Cited by 584
“Suboptimal or hypo-metacognitive sensitivity re-sults in values of meta-d = / d = (cid:5) 1 (Barrett, Dienes, & Seth, 2013; Maniscalco & Lau, 2014).”
Domain-specific impairment in metacognitive accuracy following anterior prefrontal lesions
Fleming, S. M., Ryu, J., Golfinos, J. G., & Blackmon, K. E. (2014) · Brain · doi:10.1093/brain/awu221 · Cited by 312
“Using this ratio as a measure of metacognitive accuracy effectively eliminates performance and response bias confounds typically affecting other measures (Barrett et al., 2013).”
BackgroundHMeta-d: hierarchical Bayesian estimation of metacognitive efficiency from confidence ratings
Fleming, S. M. (2017) · Neuroscience of Consciousness · doi:10.1093/nc/nix007 · Cited by 309
“The properties of the meta -d’ model have been thoroughly explored in previous articles (Maniscalco and Lau 2012; Barrett et al. 2013; Fleming and Lau 2014;Maniscalco and Lau 2014).”
Method
Metadata
- Authors
- Barrett, A. B., Dienes, Z., & Seth, A. K.
- Affiliations
- University of Sussex (GB)
- Venue
- Psychological Methods
- Published
- 2013
- Volume
- 18
- Issue
- 4
- Pages
- 535-552
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2006647339
- Cited by
- 155
- References
- 55
- Citations by year
- 2026: 2 · 2025: 13 · 2024: 6 · 2023: 16 · 2022: 18 · 2021: 16 · 2020: 13 · 2019: 11 · 2018: 12 · 2017: 13 · 2016: 12 · 2015: 9 · 2014: 11 · 2013: 3
- FWCI
- 6.37
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Licence
- cc-by
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Sussex Research Online (University of Sussex) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Funders
- Dr. Mortimer and Dame Theresa Sackler Foundation
- Engineering and Physical Sciences Research Council (EP/G007543/1)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- metacognition, signal detection theory, metacognitive sensitivity, response bias, metacognitive judgments, metacognitive accuracy, metacognitive confidence, metacognitive monitoring
- MeSH terms
- Cognition, Data Interpretation, Statistical, Humans, Models, Psychological, Signal Detection, Psychological
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.