The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
HMeta-d: hierarchical Bayesian estimation of metacognitive efficiency from confidence ratings
Open accessAbstract
Metacognition refers to the ability to reflect on and monitor one’s cognitive processes, such as perception, memory and decision-making. Metacognition is often assessed in the lab by whether an observer’s confidence ratings are predictive of objective success, but simple correlations between performance and confidence are susceptible to undesirable influences such as response biases. Recently an alternative approach to measuring metacognition has been developed (Maniscalco & Lau, 2012) that characterises metacognitive sensitivity (meta- d ′) by assuming a generative model of confidence within the framework of signal detection theory. However, current estimation routines require an abundance of confidence rating data to recover robust parameters, and only provide point estimates of meta- d ’. In contrast, hierarchical Bayesian estimation methods provide opportunities to enhance statistical power, incorporate uncertainty in group-level parameter estimates and avoid edge-correction confounds. Here I introduce such a method for estimating metacognitive efficiency (meta- d ’/ d ’) from confidence ratings and demonstrate its application for assessing group differences. A tutorial is provided on both the meta- d ’ model and the preparation of behavioural data for model fitting. Through numerical simulations I show that a hierarchical approach outperforms alternative fitting methods in situations where limited data are available, such as when quantifying metacognition in patient populations. In addition, the model may be flexibly expanded to estimate parameters encoding other influences on metacognitive efficiency. MATLAB software and documentation for implementing hierarchical meta- d ’ estimation (HMeta-d) can be downloaded at https://github.com/smfleming/HMeta-d .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Inference from iterative simulation using multiple sequences
Gelman, A., & Rubin, D. B. (1992) · Statistical Science · doi:10.1214/ss/1177011136 · Cited by 16,627
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry
Flavell, J. H. (1979) · American Psychologist · doi:10.1037/0003-066x.34.10.906 · Cited by 11,282 · cited 1×
“Metacognition is defined as ‘knowledge of one’s own cognitive processes’ (Flavell 1979).”
BackgroundStatistical Methods for Psychology.
Gudgeon, A. C., & Howell, D. C. (1994) · The Statistician · doi:10.2307/2348956 · Cited by 8,278
Judgment Under Uncertainty: Heuristics and Biases.
Shafer, G. R., Kahnerman, D., Slovic, P., & Tversky, A. (1984) · Journal of the American Statistical Association · doi:10.2307/2288362 · Cited by 5,657
Detection Theory: A User's Guide
Macmillan, N. A., & Creelman, C. D. (1991) · Cited by 5,415
JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling
Plummer, M. (2003) · Cited by 4,720
Judgment under uncertainty: Heuristics and biases
Tversky, A., & Kahneman, D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.002 · Cited by 3,643
Detection theory: A user's guide
Lories, G. (1993) · Behavioural Processes · doi:10.1016/0376-6357(93)90083-4 · Cited by 2,835
Calibration of probabilities: the state of the art to 1980
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.023 · Cited by 1,967
Bayesian Cognitive Modeling: A Practical Course
Lee, M. D., & Wagenmakers, E.-J. (2014) · doi:10.1017/cbo9781139087759 · Cited by 1,795
VISUAL CAPACITY IN THE HEMIANOPIC FIELD FOLLOWING A RESTRICTED OCCIPITAL ABLATION
WEISKRANTZ, L., WARRINGTON, E. K., SANDERS, M. D., & MARSHALL, J. (1974) · Brain · doi:10.1093/brain/97.1.709 · Cited by 1,465
Fleming, S. M., & Lau, H. C. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00443 · Cited by 1,408 · cited 4×
“The properties of the meta -d’ model have been thoroughly explored in previous articles (Maniscalco and Lau 2012; Barrett et al. 2013; Fleming and Lau 2014;Maniscalco and Lau 2014).”
BackgroundMethodInfluentialSelf-citation(2013) · Cited by 1,351
Macmillan, N. A., & Creelman, C. D. (2004) · doi:10.4324/9781410611147 · Cited by 1,269
HDDM: Hierarchical Bayesian estimation of the Drift-Diffusion Model in Python
Wiecki, T. V., Sofer, I., & Frank, M. J. (2013) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2013.00014 · Cited by 1,047
A comparison of current measures of the accuracy of feeling-of-knowing predictions
Nelson, T. O. (1984) · Psychological Bulletin · doi:10.1037/0033-2909.95.1.109 · Cited by 976
arm: Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Su, Y.-S. (2007) · CRAN: Contributed Packages · doi:10.32614/cran.package.arm · Cited by 955
Corrections for extreme proportions and their biasing effects on estimated values ofd′
Hautus, M. J. (1995) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03203619 · Cited by 894
Maniscalco, B., & Lau, H. (2012) · Consciousness and Cognition · doi:10.1016/j.concog.2011.09.021 · Cited by 883
Empirical support for higher-order theories of conscious awareness
Lau, H., & Rosenthal, D. (2011) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.05.009 · Cited by 869 · cited 1×
“It is plausible that a joint lack of metacognition and conscious visual experience are both consequences of disruptions to higher-order representations (Lau and Rosenthal 2011; Ko and Lau 2012).”
BackgroundRelating Introspective Accuracy to Individual Differences in Brain Structure
Fleming, S. M., Weil, R. S., Nagy, Z., Dolan, R. J., & Rees, G. (2010) · Science · doi:10.1126/science.1191883 · Cited by 809
Self-citationSelf-Evaluation of Decision-Making: A General Bayesian Framework for Metacognitive Computation
Fleming, S. M., & Daw, N. D. (2017) · Psychological Review · doi:10.1037/rev0000045 · Cited by 584
Self-citationStatistical methods for psychology
Howell, D. C. (2007) · Cited by 564
Confidence and certainty: Distinct probabilistic quantities for different goals
Pouget, A., Drugowitsch, J., & Kepecs, A. (2016) · Nature Neuroscience · doi:10.1038/nn.4240 · Cited by 528
The log transformation is special
Keene, O. N. (1995) · Statistics in Medicine · doi:10.1002/sim.4780140810 · Cited by 498
Re-representing consciousness: dissociations between experience and meta-consciousness
Schooler, J. W. (2002) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(02)01949-6 · Cited by 485
Post-decision wagering objectively measures awareness
Persaud, N., McLeod, P., & Cowey, A. (2007) · Nature Neuroscience · doi:10.1038/nn1840 · Cited by 454
Neuroscience of Self and Self-Regulation
Heatherton, T. F. (2011) · Annual Review of Psychology · doi:10.1146/annurev.psych.121208.131616 · Cited by 433
The development of metacognitive ability in adolescence
Weil, L. G., Fleming, S. M., Dumontheil, I., Kilford, E. J., Weil, R. S., Rees, G., Dolan, R. J., & Blakemore, S.-J. (2013) · Consciousness and Cognition · doi:10.1016/j.concog.2013.01.004 · Cited by 429
Self-citation(2015) · Elsevier eBooks · doi:10.1016/c2012-0-00477-2 · Cited by 416
Galvin, S. J., Podd, J. V., Drga, V., & Whitmore, J. (2003) · Psychonomic Bulletin & Review · doi:10.3758/bf03196546 · Cited by 405
Prefrontal Contributions to Metacognition in Perceptual Decision Making
Fleming, S. M., Huijgen, J., & Dolan, R. J. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.6489-11.2012 · Cited by 391
Self-citationBaird, B., Smallwood, J., Gorgolewski, K. J., & Margulies, D. S. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.0786-13.2013 · Cited by 315
Domain-specific impairment in metacognitive accuracy following anterior prefrontal lesions
Fleming, S. M., Ryu, J., Golfinos, J. G., & Blackmon, K. E. (2014) · Brain · doi:10.1093/brain/awu221 · Cited by 312
Self-citationAnatomical coupling between distinct metacognitive systems for memory and visual perception
McCurdy, L. Y., Maniscalco, B., Metcalfe, J., Liu, K. Y., de Lange, F. P., & Lau, H. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.1890-12.2013 · Cited by 299
Individual differences in metacognition: Evidence against a general metacognitive ability
Kelemen, W. L., Frost, P. J., & Weaver, C. A. (2000) · Memory & Cognition · doi:10.3758/bf03211579 · Cited by 252
Failures of metacognition and lack of insight in neuropsychiatric disorders.
David, A. S., Bedford, N., Wiffen, B., & Gilleen, J. (2012) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2012.0002 · Cited by 227 · cited 1×
“…and cognitive processes also has broader application in work on judgment and decision-making (Lichtenstein et al. 1982), developmental psychology (Weil et al. 2013; Goupil et al. 2016), social psychology (Heatherton 2011) and clinical disorders (David et al. 2012; Moeller and Goldstein 2014).”
BackgroundInfants ask for help when they know they don’t know
Goupil, L., Romand-Monnier, M., & Kouider, S. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1515129113 · Cited by 226
Distinct brain mechanisms for conscious versus subliminal error detection
Charles, L., Van Opstal, F., Marti, S., & Dehaene, S. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.01.054 · Cited by 208 · cited 2×
“2014) further processing of stimulus information (Rabbitt and Vyas 1981; Charles et al. 2013) or knowledge of other influences on task performance when making their metacognitive judgments (Fleming and Daw 2017).”
Maniscalco, B., & Lau, H. (2014) · The Cognitive Neuroscience of Metacognition · doi:10.1007/978-3-642-45190-4_3 · Cited by 190
Confidence Leak in Perceptual Decision Making
Rahnev, D., Koizumi, A., McCurdy, L. Y., D’Esposito, M., & Lau, H. C. (2015) · Psychological Science · doi:10.1177/0956797615595037 · Cited by 184
The Cognitive Neuroscience of Metacognition
(2014) · Springer eBooks · doi:10.1007/978-3-642-45190-4 · Cited by 183
Individual consistency in the accuracy and distribution of confidence judgments
Ais, J., Zylberberg, A., Barttfeld, P., & Sigman, M. (2016) · Cognition · doi:10.1016/j.cognition.2015.10.006 · Cited by 174
Masson, M. E. J., & Rotello, C. M. (2009) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/a0014876 · Cited by 173
Two Types of ROC Curves and Definitions of Parameters
Clarke, F. R., Birdsall, T. G., & Tanner, W. P. (1959) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907764 · Cited by 168
Effects of age on metacognitive efficiency
Palmer, E. C., David, A. S., & Fleming, S. M. (2014) · Consciousness and Cognition · doi:10.1016/j.concog.2014.06.007 · Cited by 161
Self-citationMeasures of metacognition on signal-detection theoretic models
Barrett, A. B., Dienes, Z., & Seth, A. K. (2013) · Psychological Methods · doi:10.1037/a0033268 · Cited by 155
Bayesian Measures of Model Complexity and Fit
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van Der Linde, A. (2002) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00353 · Cited by 151
Impaired self-awareness in human addiction: deficient attribution of personal relevance
Moeller, S. J., & Goldstein, R. Z. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.09.003 · Cited by 140
Ko, Y., & Lau, H. C. (2012) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2011.0380 · Cited by 107
Processing a Display Even after you Make a Response to it. How Perceptual Errors can be Corrected
Rabbitt, P., & Vyas, S. (1981) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640748108400790 · Cited by 97
Blind Insight : Metacognitive Discrimination Despite Chance Task Performance
Scott, R. B. Y., Dienes, Z. P., Barrett, A. B., Bor, D., & Seth, A. K. (2014) · Psychological Science · doi:10.1177/0956797614553944 · Cited by 66
Visibility Is Not Equivalent to Confidence in a Low Contrast Orientation Discrimination Task
Rausch, M., & Zehetleitner, M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00591 · Cited by 64
A stochastic detection and retrieval model for the study of metacognition
Jang, Y., Wallsten, T. S., & Huber, D. E. (2012) · Psychological Review · doi:10.1037/a0025960 · Cited by 59
BayesSDT: Software for Bayesian inference with signal detection theory
Lee, M. (2008) · Behavior Research Methods · doi:10.3758/brm.40.2.450 · Cited by 36
Denison, R. N. (2017) · Analytic Philosophy · doi:10.1111/phib.12092 · Cited by 32
Suboptimal Criterion Learning in Static and Dynamic Environments
Norton, E. H., Fleming, S. M., Daw, N. D., & Landy, M. S. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005304 · Cited by 25
Self-citation
Cited by
Towards a comprehensive assessment of interoception in a multi-dimensional framework
Suksasilp, C., & Garfinkel, S. N. (2022) · Biological Psychology · doi:10.1016/j.biopsycho.2022.108262 · Cited by 288
Morales, J., Lau, H., & Fleming, S. M. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.2360-17.2018 · Cited by 274
Self-citationMetacognitive Failure as a Feature of Those Holding Radical Beliefs
Rollwage, M., Dolan, R. J., & Fleming, S. M. (2018) · Current Biology · doi:10.1016/j.cub.2018.10.053 · Cited by 193
Self-citationHuman Metacognition Across Domains: Insights from Individual Differences and Neuroimaging
Rouault, M., McWilliams, A., Allen, M. G., & Fleming, S. M. (2018) · Personality Neuroscience · doi:10.1017/pen.2018.16 · Cited by 176
Self-citationAbnormalities of confidence in psychiatry: an overview and future perspectives
Hoven, M., Lebreton, M., Engelmann, J. B., Denys, D., Luigjes, J., & van Holst, R. J. (2019) · Translational Psychiatry · doi:10.1038/s41398-019-0602-7 · Cited by 171
Metadata
- Authors
- Fleming, S. M.
- Venue
- Neuroscience of Consciousness
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-01-01
- Volume
- 2017
- Issue
- 1
- Pages
- nix007-nix007
- ISSN
- 2057-2107 (ISSN-L)
- OpenAlex
- W2953219509
- Cited by
- 309
- References
- 58
- FWCI
- 8.45
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.