CoChat
Sign Up
Cited by 58Memory Processes and InfluencesVisual perception and processing mechanismsDecision-Making and Behavioral Economics

Confidence measurement in the light of signal detection theory

Open access
Massoni, Sébastien, Gajdos, Thibault, Vergnaud, Jean-Christophe·2014·Frontiers in Psychology
DOI 10.3389/fpsyg.2014.01455PMC PMC4263084
Open this paper in CoChat — free accountOpen PDF

Abstract

We compare three alternative methods for eliciting retrospective confidence in the context of a simple perceptual task: the Simple Confidence Rating (a direct report on a numerical scale), the Quadratic Scoring Rule (a post-wagering procedure), and the Matching Probability (MP; a generalization of the no-loss gambling method). We systematically compare the results obtained with these three rules to the theoretical confidence levels that can be inferred from performance in the perceptual task using Signal Detection Theory (SDT). We find that the MP provides better results in that respect. We conclude that MP is particularly well suited for studies of confidence that use SDT as a theoretical framework.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fpsyg.2014.01455

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Subjects were not instructed or incentivized to assign probability ranges to each button.

    “Because we are interested in subjects’ intrinsic computation of confidence, we did not instruct or incentivize them to assign probability ranges to each button (e.g., by using a scoring rule [34–36]).”

    From Comparing Bayesian and non-Bayesian accounts of human confidence reports (2018) · cites this paper as [34]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 45 · Cited by 58

Research works printed but not linked: 37

  1. Becker G. M. DeGroot M. H. Marschak J. (1964). Measuring utility by a single-response sequential method. Behav. Sci. 9 226– 232 10.1002/bs.3830090304 5888778 · DOI
  2. Brainard D. H. (1997). The psychophysics toolbox. Spat. Vis. 10 433– 436 10.1163/156856897X00357 9176952 · DOI
  3. Bogacz R. Brown E. Moehlis J. Holmes P. Cohen J. D. (2006). The physic of optimal decision making: a formal analysis of models of performance in two-alternative forced choice tasks. Psychol. Rev. 113 700– 765 10.1037/0033-295X.113.4.700 17014301 · DOI
  4. Clarke F. Birdsall T. Tanner W. (1959). Two types of ROC curves and definition of parameters. J. Acoust. Soc. Am. 31 629– 630 10.1121/1.1907764 · DOI
  5. Dienes Z. (2007). Subjective measures of unconscious knowledge. Prog. Brain Res. 168 49– 269 10.1016/S0079-6123(07)68005-4 18166385 · DOI
  6. Dienes Z. Seth A. (2010). Gambling on the unconscious: a comparison of wagering and confidence ratings as measures of awareness in an artificial grammar task. Conscious. Cogn. 19 674– 681 10.1016/j.concog.2009.09.009 19828331 · DOI
  7. Dunlosky J. Metcalfe J. (2008). Metacognition. Thousand Oaks: SAGE Publications.
  8. Fleming S. M. Dolan R. J. (2010). Effect of loss aversion on post-decision wagering: implications for measures of awareness. Conscious. Cogn. 19 352– 363 10.1016/j.concog.2009.11.002 20005133 PMC2842936 · DOI
  9. Fleming S. M. Weil R. S. Nagy Z. Dolan R. J. Rees G. (2010). Relating introspective accuracy to individual differences in brain structure. Science 329 1541– 1543 10.1126/science.1191883 20847276 PMC3173849 · DOI · Paper page
  10. Galvin S. Podd J. Drga V. Whitmore J. (2003). Type 2 tasks in the theory of signal detectability: discrimination between correct and incorrect decisions. Psychon. Bull. Rev. 10 843– 876 10.3758/BF03196546 15000533 · DOI · Paper page
  11. Gneiting T. Raftery A. E. (2007). Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102 359– 378 10.1198/016214506000001437 · DOI · Paper page
  12. Green D. A. Swets J. A. (1966). Signal Detection Theory and Psychophysics. New-York: John Wiley and Sons.
  13. Kanai R. Walsh V. Tseng C. (2010). Subjective discriminability of invisibility: a framework for distinguishing perceptual and attentional failures of awareness. Conscious. Cogn. 19 1045– 1057 10.1016/j.concog.2010.06.003 20598906 · DOI
  14. Kepecs A. Uchida N. Zariwala H. Mainen Z. (2008). Neural correlates, computation and behavioural impact of decision confidence. Nature 455 227– 231 10.1038/nature07200 18690210 · DOI · Paper page
  15. Levitt H. (1971). Transformed up-down methods in psychoacoustics. J. Acoust. Soc. Am. 49 467– 477 10.1121/1.1912375 5541744 · DOI
  16. Maniscalco B. Lau H. (2012). A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings. Conscious. Cogn. 21 422– 430 10.1016/j.concog.2011.09.021 22071269 · DOI · Paper page
  17. Massoni S. (2009). A direct revelation mechanism for elicitating confidence in perceptual and cognitive tasks. Master’s Thesis, Université Paris 1 – Panthéon-Sorbonne, Paris.
  18. Middlebrooks P. G. Sommer M. A. (2011). Metacognition in monkeys during an oculomotor task. J. Exp. Psychol. Learn. Mem. Cogn. 37 325 10.1037/a0021611 PMC3649877 21171807 · DOI · Paper page
  19. Nieder A. (2005). Counting on neurons: the neurobiology of numerical competence. Nat. Rev. Neurosci. 6 177– 190 10.1038/nrn1626 15711599 · DOI
  20. Nieder A. Miller E. K. (2004). A parieto-frontal network for visual numerical information in the monkey. Proc. Natl. Acad. Sci. U.S.A. 101 7457– 7462 10.1073/pnas.0402239101 15123797 PMC409940 · DOI
  21. Nyarko Y. Schotter A. (2002). An experimental study of belief learning using elicited beliefs. Econometrica 70 971– 1005 10.1111/1468-0262.00316 · DOI
  22. Offerman T. Sonnemans J. Van de Kuilen G. Wakker P. P. (2009). A truth-serum for non-bayesian: correction proper scoring rules for risk attitudes. Rev. Econ. Stud. 76 1461– 1489 10.1111/j.1467-937X.2009.00557.x · DOI
  23. Overgaard M. Sandberg K. (2012). Kinds of access: different methods for report reveal different kinds of metacognitive access. Philos. Trans. R. Soc. B Biol. Sci. 367 1287– 1296 10.1098/rstb.2011.0425 PMC3318768 22492747 · DOI
  24. Palfrey T. Wang S. (2009). On eliciting beliefs in strategic games. J. Econ. Behav. Organ. 71 98– 109 10.1016/j.jebo.2009.03.025 · DOI
  25. Palmer T. N. Hagedorn R. (Eds) (2006). Predictability of Weather and Climate. Cambridge: Cambridge University Press.
  26. Persaud N. McLeod P. Cowey A. (2007). Post-decision wagering objectively measures awareness. Nat. Neurosci. 10 257– 261 10.1038/nn1840 17237774 · DOI · Paper page
  27. Piazza M. Izard V. Pinel P. Le Bihan D. Dehaene S. (2004). Tuning curves for approximate numerosity in the human intraparietal sulcus. Neuron 44 547– 555 10.1016/j.neuron.2004.10.014 15504333 · DOI
  28. Pica P. Lemer C. Izard V. Dehaene S. (2004). Exact and approximate arithmetic in Amazonian indigene group. Science 306 499– 503 10.1126/science.1102085 15486303 · DOI
  29. Pollack I. (1959). On indices of signal and response discriminability. J. Acoust. Soc. Am. 31 1031– 1031 10.1121/1.1907802 · DOI
  30. Ratcliff R. McKoon G. (2008). The diffusion decision model: theory and data for two-choice decision tasks. Neural Comput. 20 873– 922 10.1162/neco.2008.12-06-420 18085991 PMC2474742 · DOI · Paper page
  31. Ratcliff R. Starns J. J. (2009). Modeling confidence and response time in recognition memory. Psychol. Rev. 116 59– 83 10.1037/a0014086 19159148 PMC2693899 · DOI
  32. Rounis E. Maniscalco B. Rothwell J. Passingham R. Lau H. (2010). Theta-burst transcranial magnetic stimulation to the prefrontal cortex impairs metacognitive visual awareness. Cogn. Neurosci. 1 165– 175 10.1080/17588921003632529 24168333 · DOI · Paper page
  33. Sandberg K. Timmermans B. Overgaard M. Cleeremans A. (2010). Measuring consciousness: is one measure better than the other? Conscious. Cogn. 19 1069– 1078 10.1016/j.concog.2009.12.013 20133167 · DOI · Paper page
  34. Schurger A. Sher S. (2008). Awareness, loss aversion, and post-decision wagering. Trends Cogn. Sci. 12 209– 210 10.1016/j.tics.2008.02.012 18482859 · DOI
  35. Scott R. B. Dienes Z. Barrett A. B. Bor D. Seth A. K. (2014). Blind insight: metacognitive discrimination despite chance task performance. Psychol. Sci. 10.1177/0956797614553944 [Epub ahead of print]. PMC4263819 25384551 · DOI
  36. Smith J. D. Shields W. E. Washburn D. A. (2003). The comparative psychology of uncertainty monitoring and metacognition. Behav. Brain Sci. 26 317– 339 10.1017/S0140525X03000086 14968691 · DOI
  37. Wickens T. D. (2002). Elementary Signal Detection Theory. New-York: Oxford University Press.

Authors
Massoni, S., Gajdos, T., & Vergnaud, J.-C.
Affiliations
Centre National de la Recherche Scientifique (FR)
ORCID
Venue
Frontiers in Psychology
Publisher
Frontiers Media SA
Published
2014-12-11
Volume
5
Issue
1455
Pages
1455-1455
ISSN
1664-1078 (ISSN-L)
DOI
10.3389/fpsyg.2014.01455
PMC
PMC4263084
OpenAlex
W2052082938
Cited by
58
References
45
Citations by year
2026: 4 · 2025: 3 · 2024: 1 · 2023: 5 · 2022: 5 · 2021: 6 · 2020: 5 · 2019: 8 · 2018: 11 · 2017: 4 · 2016: 3 · 2015: 2 · 2014: 1
FWCI
1.86
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Frontiers in Psychology (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • QUT ePrints (Queensland University of Technology) (acceptedVersion)
  • HAL AMU (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
Keywords
confidence measurement, signal detection theory, match probability, quadratic scoring rule, perceptual decision making, metacognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review