Individual consistency in the accuracy and distribution of confidence judgments
Open accessAbstract
We examine which aspects of the confidence distributions - its shape, its bias toward higher or lower values, and its ability to distinguish correct from erred trials - are idiosyncratic of the who (individual specificity), the when (variability across days) and the what (task specificity). Measuring confidence across different sessions of four different perceptual tasks we show that: (1) Confidence distributions are virtually identical when measured in different days for the same subject and the same task, constituting a subjective fingerprint, (2) The capacity of confidence reports to distinguish correct from incorrect responses is only modestly (but significantly) correlated when compared across tasks, (3) Confidence distributions are very similar for tasks that involve different sensory modalities but have similar structure, (4) Confidence accuracy is independent of the mean and width of the confidence distribution, (5) The mean of the confidence distribution (an individual's confidence bias) constitutes the most efficient indicator to infer a subject's identity from confidence reports and (6) Confidence bias measured in simple perceptual decisions correlates with an individual's optimism bias measured with standard questionnaire.
AI snapshot
Written from the paper’s full text.
TL;DR
Confidence distributions are highly stable within an individual for the same task, forming a “subjective fingerprint,” while the ability of confidence to distinguish correct from incorrect responses is only modestly correlated across tasks. The mean confidence bias is the most efficient single feature for identifying a subject and correlates with questionnaire-measured optimism.
Key findings
- Confidence distributions were almost identical across sessions of the same task for the same participant, but varied across tasks; hierarchical clustering grouped sessions by task, with partial report separated from the other tasks.
- Confidence accuracy (AUC) was significantly positively correlated within tasks (one-sided T-value = 7.24, df = 9, p < 5 × 10⁻⁵) and across tasks (one-sided T-value = 2.97, df = 44, p < 0.005), though correlations were higher for same-experiment sessions (one-sided T-value = 5.96, df = 53, p < 10⁻⁶).
- Mean confidence bias was highly consistent within and across tasks (same task: p < 10⁻⁶, T = 38, df = 9; different tasks: p < 10⁻⁶, T = 25.9, df = 44), and identity decoding accuracy was 16.2% for mean confidence, 10.3% for SD, 10.7% for multimodality, and 7.9% for AUC, all above chance (4.35%); combining all four features raised accuracy to 37.1%.
- Confidence bias correlated with LOT-R optimism score (Pearson’s r = 0.56, p = 0.007), and AUC was largely independent of the mean and width of the confidence distribution.
Methods
- Twenty-three participants (12 female; mean age 24 ± 1.7 years) completed eleven sessions across four perceptual tasks: auditory discrimination, contrast discrimination, luminance discrimination, and partial report.
- After each choice, participants reported confidence on a horizontal scale from “guessing” to “full certainty”; no feedback was provided.
- Analyses included Type-II ROC AUC for confidence accuracy, Kullback-Leibler divergence and hierarchical clustering for distribution similarity, mean/SD/Hartigan dip multimodality metrics, multinomial logistic regression for subject identity decoding, bootstrap comparisons, and the Life Orientation Test (LOT-R) for optimism/pessimism.
Limitations
- The paper states that subjects reported on a non-calibrated scale, so precise claims about whether participants were over- or under-confident cannot be made.
- The paper states that the parameter space across tasks is too large to infer which specific factors caused differences in confidence distributions, and notes an intrinsic methodological difficulty for universal claims because the partial report task may involve misleading or deceptive problems.
- Evident from the text: the sample was small (23 participants, with only 18 completing LOT-R), task sessions and trial counts differed across tasks, and the study used only four laboratory tasks.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 31 cited sources checked · 0 supported · 0 flagged
- Findingown data
AUC correlation coefficients were significantly higher when obtained from sessions that belonged to the same experiment.
“Correlations coefficients were significantly higher when obtained from sessions that belonged to the same experiment (one-sided T-value = 5.96, df = 53, p < 10 À6 ).”
Quote in the paperSubstantial claim - Findingown data
Confidence distributions are virtually identical when measured in different days for the same subject and the same task, constituting a subjective fingerprint.
“(1) Confidence distributions are virtually identical when measured in different days for the same subject and the same task, constituting a subjective fingerprint”
- Also stated in: § 3.2. Confidence distributions as individuals' fingerprints · boosted
“the shape of the confidence distributions was extremely consistent across sessions of the same task for the same participant.”
- Also stated in: § 4. Discussion · plain
“(1) Most aspects of confidence distribution (AUC, mean, shape estimators, variance) are highly reproducible across sessions for the same individual in the same context.”
Quote in the paperSubstantial claim - Also stated in: § 3.2. Confidence distributions as individuals' fingerprints · boosted
- Findingown data
Confidence bias measured in simple perceptual decisions correlates with an individual's optimism bias measured with standard questionnaire.
“(6) Confidence bias measured in simple perceptual decisions correlates with an individual's optimism bias measured with standard questionnaire”
- Also stated in: § 3.4. Confidence bias is correlated with an individual's bias toward optimism versus pessimism · boosted
“The bias toward optimism/ pessimism as measured with the LOT-R was strongly correlated with the average confidence computed from the psychophysics experiments $1 year earlier (Fig. 6).”
Quote in the paperSubstantial claim - Also stated in: § 3.4. Confidence bias is correlated with an individual's bias toward optimism versus pessimism · boosted
- Findingown data
The bias toward optimism/pessimism as measured with the LOT-R was strongly correlated with the average confidence computed from the psychophysics experiments ~1 year earlier.
“The bias toward optimism/ pessimism as measured with the LOT-R was strongly correlated with the average confidence computed from the psychophysics experiments $1 year earlier (Fig. 6).”
Quote in the paperSubstantial claim - Findingown data
The accuracy of the decoder increased significantly to 37.1% with a multinomial regression that includes all four features simultaneously.
“The classification capacity of these parameters was synergic which was revealed by the fact that the accuracy of the decoder increased significantly to 37.1% with a multinomial regression that includes all four features simultaneously.”
Quote in the paperSubstantial claim
- Findingown data
Hierarchical cluster analysis revealed four clear clusters, each grouping together the sessions that belonged to the same task.
“These observations were confirmed by a hierarchical cluster analysis, which revealed four clear clusters (coded in different colors in Fig. 3b, right), each grouping together the sessions that belonged to the same task.”
Quote in the paperSubstantial claim - Findingown data
AUC values were above 0.5 for every subject and experiment, indicating that all subjects were reliably above chance in assigning confidence judgments relative to correct and error trials.
“AUC values were above 0.5 for every subject and experiment, indicating that all subjects were reliably above chance in assigning confidence judgments relative to correct and error trials (Fig. 1b).”
Quote in the paperSubstantial claim - Findingown data
The classification accuracy for l was significantly higher than for AUC.
“A nonparametric bootstrap test (Efron & Tibshirani, 1994) revealed that the classification accuracy for l was significantly higher than for AUC (p < 0.02, N = 1,000 bootstrap samples).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Efron & Tibshirani, 1994
Not checked in the free run.
- Findingown data
The mean correlation coefficient of confidence bias was highly significant both within and across tasks.
“Across the population, the mean correlation coefficient was highly significant both when the session belonged to the same (p < 10 À6 , one sided T-value = 38, df = 9) and different (p < 10 À6 , one sided T-value = 25.9, df = 44) tasks (Fig. 4b).”
Quote in the paperSubstantial claim - Findingown data
Confidence accuracy is independent of the mean and width of the confidence distribution.
“(4) Confidence accuracy is independent of the mean and width of the confidence distribution”
- Also stated in: § 3.3. Confidence bias within and across tasks · plain
“subjects can report confidence within narrower regions of the confidence scale without affecting its ability to distinguish between correct and incorrect responses.”
Quote in the paperSubstantial claim - Also stated in: § 3.3. Confidence bias within and across tasks · plain
- Findingown data
Confidence bias (mean) is strongly correlated across sessions of the same task.
“We observed very strong correlations in the bias (l) across sessions of same task (Fig. 4a).”
Quote in the paperSubstantial claim - Findingown data
The capacity of confidence reports to distinguish correct from incorrect responses is only modestly (but significantly) correlated when compared across tasks.
“(2) The capacity of confidence reports to distinguish correct from incorrect responses is only modestly (but significantly) correlated when compared across tasks”
- Also stated in: § 3.1. Metacognitive ability within and across tasks · plain
“The AUC values obtained from different tasks typically resulted in weak but positive correlations (Fig. 2a, in blue).”
- Also stated in: § 4. Discussion · plain
“(2) The precision of confidence reports to distinguish correct and incorrect (AUC) trials is only modestly correlated when compared across tasks.”
Quote in the paperSubstantial claim - Also stated in: § 3.1. Metacognitive ability within and across tasks · plain
- Findingown data
Correlation coefficients of individual AUC measured in different sessions of the same task consistently showed high positive values.
“Correlation coefficients of the distribution of individual AUC measured in different sessions of the same task consistently showed high positive values (Fig. 2a, diagonal, red dots).”
Quote in the paperSubstantial claim - Findingown data
The four clusters further organize into two clusters, with the partial report task separated from the rest.
“The four clusters further organize into two clusters, with the partial report task separated from the rest.”
Quote in the paperSubstantial claim - Findingown data
Confidence distributions are very similar for tasks that involve different sensory modalities but have similar structure.
“(3) Confidence distributions are very similar for tasks that involve different sensory modalities but have similar structure”
- Also stated in: § 3.2. Confidence distributions as individuals' fingerprints · plain
“the shape of confidence distributions for the contrast and auditory discrimination task were very similar”
- Also stated in: § 4. Discussion · plain
“confidence distributions where very similar for the contrast and auditory discrimination tasks which are based on different modalities but have very similar task structures.”
Quote in the paperSubstantial claim - Also stated in: § 3.2. Confidence distributions as individuals' fingerprints · plain
- Findingown data
Across different tasks, confidence distributions showed larger variability than within the same task.
“Across different tasks, confidence distribution showed larger variability (Fig. 3a).”
Quote in the paperSubstantial claim - Findingown data
The average correlation coefficient of AUC was significantly positive both within and across tasks.
“The average correlation coefficient was significantly positive both within (one-sided T-value = 7.24, df = 9, p < 5 Â 10 À5 ) and across (one-sided T-value = 2.97, df = 44, p < 0.005) tasks (Fig. 2b).”
Quote in the paperSubstantial claim - Findingreasoning
The partial report task showed much higher scores of dissimilarity.
“The partial report task (as could be expected due to large differences in performance) shows instead much higher scores of dissimilarity.”
Quote in the paperNot a substantial claim - Findingown data
The shape of confidence distributions differed markedly across participants.
“The shape of confidence distributions differed markedly across participants: some distributions were bimodal, others were packed in a high confidence mode, others a single and wide bell-shaped distribution with close to zero mean.”
Quote in the paperSubstantial claim - Findingown data
The correlation coefficient of confidence bias between sessions of different experiments was significantly positive for every comparison.
“The correlation coefficient between sessions of different experiments was also significantly positive for every comparison (Fig. 4a).”
Quote in the paperSubstantial claim - Findingown data
All these classification accuracy values are above chance level of 4.35%.
“All these values are above chance level of 4.35%.”
Quote in the paperSubstantial claim - Findingown data
The mean of the confidence distribution (an individual's confidence bias) constitutes the most efficient indicator to infer a subject's identity from confidence reports.
“(5) The mean of the confidence distribution (an individual's confidence bias) constitutes the most efficient indicator to infer a subject's identity from confidence reports”
- Also stated in: § 3.3. Confidence bias within and across tasks · plain
“A nonparametric bootstrap test (Efron & Tibshirani, 1994) revealed that the classification accuracy for l was significantly higher than for AUC (p < 0.02, N = 1,000 bootstrap samples).”
- Also stated in: § 3.3. Confidence bias within and across tasks · plain
“Classification accuracy was 16.2%, 10.3%, 10.7%, and 7.9% for regressors l, r, mi, or AUC respectively.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Efron & Tibshirani, 1994
Not checked in the free run.
- Also stated in: § 3.3. Confidence bias within and across tasks · plain
- Findingown data
Every single correlation of average confidence across different sessions (different days, different tasks) was positive.
“In fact every single correlation of the average confidence across different sessions (measured in different days and performing different tasks) was positive.”
Quote in the paperSubstantial claim - Findingreasoning
Confidence distributions have a very high capacity to classify a subject's identity.
“Confidence distributions have a very high capacity to classify a subject's identity.”
Quote in the paperSubstantial claim - Findingown data
Average confidence (confidence bias) in perceptual decisions is significantly positively correlated with optimism measured by LOT-R (r = 0.56, p = 0.007).
“we observed a significantly positive correlation between confidence bias in the simple perceptual decisions we studied, and the general optimism bias measured with standard questionnaire (Scheier et al., 1994).”
- Also stated in: § 4. Discussion · plain
“The average confidence obtained after aggregating all trials is correlated with an index of optimism versus pessimism obtained from LOT-R.”
- Also stated in: § 4. Discussion · plain
“Pearson's correlation (q = 0.56) was significantly positive (p = 0.007).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Scheier et al., 1994 source
Not checked in the free run.
- Also stated in: § 4. Discussion · plain
- Findingnone given
Most aspects of confidence distribution (AUC, mean, shape estimators, variance) are highly reproducible across sessions for the same individual in the same context.
“most aspects of confidence distribution (AUC, mean, shape estimators, variance) are highly reproducible across sessions for the same individual in the same context.”
Quote in the paperSubstantial claim - Findingnone given
Our study shows that the ability of confidence reports to distinguish correct from incorrect decisions is largely independent from the confidence bias.
“Our study shows that, contrary to what intuition may have suggested, the ability of confidence reports to distinguish correct from incorrect decisions is largely independent from the confidence bias.”
Quote in the paperSubstantial claim - Findingown data
A regression analysis showed that average confidence was a more efficient indicator of a subject's identity than the capacity of confidence to distinguish correct from incorrect decisions.
“A regression analyses showed that the average confidence was a more efficient indicator of a subject's identity than the capacity of confidence to distinguish correct from incorrect decisions.”
Quote in the paperSubstantial claim - Findingown data
For each pair of sessions of different tasks, the correlation of accuracy of confidence is weak.
“For each pair of sessions of different tasks the correlation is weak.”
Quote in the paperSubstantial claim - Findingown data
When all possible pairs of sessions (for different tasks) are taken together, the distribution of r values is highly significantly shifted toward positive values.
“when all possible pairs of sessions (for different tasks) are taken together, the distribution of r values of the distribution is highly significantly shifted toward positive values”
Quote in the paperSubstantial claim - Findingown data
Despite large variations in the shape of confidence distributions across tasks, the mean of this distribution was highly indicative of a subject's identity.
“Despite the fact that there were large variations in the shape of confidence distributions across tasks, the mean of this distribution was highly indicative of a subject's identity.”
Quote in the paperSubstantial claim - Findingown data
In our study, the accuracy of confidence reports to distinguish correct and incorrect trials is only modestly correlated when compared across tasks.
“In our study the accuracy of confidence reports to distinguish correct and incorrect trials is only modestly correlated when compared across tasks.”
Quote in the paperSubstantial claim - Interpretationown data
The features that characterize the shape of the confidence distributions are a more robust fingerprint of a participant than the confidence accuracy as measured by the AUC.
“Together, these results indicate that the features that characterize the shape of the confidence distributions are a more robust fingerprint of a participant than the confidence accuracy as measured by the AUC.”
Quote in the paperSubstantial claim - Premisecitation
The capacity of confidence to distinguish correct from incorrect knowledge can vary markedly between subjects for a given task.
“research in behavioral economics and cognitive sciences has repeatedly shown that the capacity of confidence to distinguish correct from incorrect knowledge -henceforth referred as the accuracy of confidence -can vary markedly between subjects for a given task.”
Quote in the paperSubstantial claim0 of 2 sources checked - Interpretationown data
The mean of the confidence distribution (l) is the most reproducible feature of how an individual expresses confidence.
“The previous analysis indicates that the mean of the distribution (l) is the most reproducible feature of how an individual expresses confidence.”
Quote in the paperSubstantial claim - Premisecitation
This variability in confidence accuracy has been related to individual differences in brain structure and function.
“This variability has been related to individual differences in brain structure and function (Barttfeld et al., 2013;Fleming, Weil, Nagy, Dolan, & Rees, 2010).”
Quote in the paperNot a substantial claim0 of 2 sources checked - Interpretationown data
An individual's confidence bias is consistent across tasks.
“This indicates that an individual's confidence bias is consistent across tasks.”
Quote in the paperSubstantial claim - Premisecitation
Confidence bias is a highly reproducible trait of how confidence is reported.
“confidence bias is a highly reproducible trait of how confidence is reported (Stankov & Crawford, 1996)”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Stankov & Crawford, 1996
Not checked in the free run.
- Premisecitation
Hartigan's dip test statistic is consistent for testing any unimodal against any multimodal distribution.
“We used Hartigan's dip test statistic, as this non-parametric test is consistent for testing any unimodal against any multimodal distribution (Hartigan & Hartigan, 1985).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Hartigan & Hartigan, 1985
Not checked in the free run.
- Interpretationown data
Confidence is not the mere report of an internal probability but is expressed in an idiosyncratic manner.
“our work conveys the idea that confidence is not the mere report of an internal probability but that, instead, it is expressed in an idiosyncratic manner.”
Quote in the paperSubstantial claim - Interpretationreasoning
These findings are consistent with our observation that the confidence bias is the most stable individual trait of confidence distributions.
“These findings are consistent with our observation that the confidence bias is the most stable individual trait of confidence distributions.”
Quote in the paperSubstantial claim - Premisecitation
Individual persistence in the policy to express confidence has been reported.
“Individual persistence in the policy to express confidence has also been reported in a seminal study by Adams and Adams (Adams & Adams, 1961) who reported a grossly overconfident calibration curve of a schizophrenic who believed he was Jesus Christ (Lichtenstein et al., 1977).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Adams & Adams, 1961
Not checked in the free run.
- cites Lichtenstein et al., 1977
Not checked in the free run.
- Premisecitation
Spatial transposition of a given shape is a frequent unconscious operation that results in subjects being very confident about a response that is perfectly correct within a flawed scheme of reasoning, a sort of visual illusion.
“We demonstrated that spatial transposition of a given shape is a frequent unconscious operation which results in subjects being very confident about a response which is perfectly correct within a flawed scheme of reasoning, a sort of visual illusion (Graziano & Sigman, 2009).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Graziano & Sigman, 2009
Not checked in the free run.
- Interpretationreasoning
There does not seem to be a universal language to express probabilities.
“There does not seem to be a universal language to express probabilities.”
Quote in the paperSubstantial claim - Premisecitation
Budescu and colleagues have shown that they can reduce communication errors by a 'translation' device that standardizes an individual's linguistic idiosyncrasy to map uncertainty.
“Budescu and colleagues have shown that they can reduce communication errors by a ''translation”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Karelitz & Budescu, 2004
Not checked in the free run.
- Interpretationcitation
This presents an intrinsic methodological difficulty for the study of universal aspects of confidence judgments.
“This presents an intrinsic methodological difficulty for the study of universal aspects of confidence judgments (Klayman, Soll, González-Vallejo, & Barlas, 1999).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Klayman, Soll, González-Vallejo, & Barlas, 1999
Not checked in the free run.
- Interpretationown data
This reveals a more likely tendency toward positively correlated measures.
“revealing a more likely tendency toward positively correlated measures.”
Quote in the paperSubstantial claim - Interpretationreasoning
The distribution of confidence in a given task constitutes a subjective fingerprint.
“The distribution of confidence in a given task constitutes a subjective fingerprint.”
Quote in the paperSubstantial claim - Interpretationown data
Confidence in a perceptual choice indexes traits of optimism.
“Confidence in a perceptual choice indexes traits of optimism.”
Quote in the paperSubstantial claim
- May combine two claims. 13
- The quote only partly states this claim. 1
- Classified as probably not a substantial claim. 16
May combine two claims.
Findingown dataClassification accuracy was 16.2%, 10.3%, 10.7%, and 7.9% for regressors l, r, mi, or AUC respectively.
“Classification accuracy was 16.2%, 10.3%, 10.7%, and 7.9% for regressors l, r, mi, or AUC respectively.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataClassification power for the three parameters of the distribution was highly significant (p < 10^-3) while classification power for the AUC was only marginally significant (p = 0.053).
“Classification power for the three parameters of the distribution was highly significant (p < 10 À3 ) while classification power for the AUC was only marginally significant (p = 0.053).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe standard deviation and multimodality index of the confidence distribution were also highly significantly correlated both within and across tasks.
“In Figs.A.2 and A.3 we show the same analysis for the standard deviation and multimodality index of the confidence distribution, which were also highly significant both within and across tasks.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe shape of confidence distributions for the contrast and auditory discrimination tasks were very similar, less so to the luminance discrimination task.
“It also shows that the shape of confidence distributions for the contrast and auditory discrimination task were very similar, less so to the luminance discrimination task.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe relation between AUC and mean confidence had very low explanatory power and did not reach significance, indicating that the subjective choice of whether to report confidence in higher or lower average values has a very weak effect on the accuracy of the confidence system.
“However, the explanatory power of these fits was very low (the average R 2 for the data shown in Fig. 5a is 0.22) and did not reach significance indicating that the subjective choice of whether to report confidence in higher or lower average values (which is persistent and reliable across sessions and experiments, as described above) has a very weak effect on the accuracy of the confidence system.”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone givenFor some individuals, confidence is collapsed in a single mode; for others, it is expressed in two different categories.
“For some individuals collapsed in a single mode, for others being expressed in two different categories.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn the partial report task, accuracy was close to 35% (chance level 3%), whereas in the other tasks accuracy was held close to 75% (chance level 50%).
“accuracy, which is close to 35% percent on this task (where chance level is $3%) while in other tasks it was held close to 75% with chance levels of 50%.”
Quote in the paperSubstantial claimThe quote only partly states this claim.
FindingreasoningSpikes of very high (almost perfect) confidence were much more frequent in the low-accuracy partial report task than in the other tasks.
“spikes of very high (almost perfect) confidence where much more frequent”
- Also stated in: § 4. Discussion · plain
“in this low-accuracy task (Fig. 3)”
Quote in the paperSubstantial claim- Also stated in: § 4. Discussion · plain
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataSessions from the same task grouped together in hierarchical clustering of pairwise distances between confidence distributions.
“The matrix of pairwise distances was hierarchically clustered (right), revealing four clear clusters where sessions from the same task grouped together.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationA model based on a single process to compute confidence in different domains can explain what factors control the degree of correlation between tasks.
“Moreover, they show that a model based on a single process to compute confidence in different domains can explain what factors control the degree of correlation between tasks.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Pleskac & Busemeyer, 2010 source
Not checked in the free run.
- cites Yu, Pleskac, & Zeigenfuse, 2015
Not checked in the free run.
Classified as probably not a substantial claim.
PremisecitationA single process can explain confidence in choices made in perceptual and cognitive tasks.
“Pleskac and collaborators have shown that a single process can explain confidence in choices made in perceptual (line discrimination and random dot motion) and cognitive (city size inference) tasks (Pleskac & Busemeyer, 2010;Yu, Pleskac, & Zeigenfuse, 2015).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Pleskac & Busemeyer, 2010 source
Not checked in the free run.
- cites Yu, Pleskac, & Zeigenfuse, 2015
Not checked in the free run.
May combine two claims.
Interpretationown dataThe similarity in the usage of the confidence scale is supramodal, as no clear separation between auditory and visual tasks was observed.
“Hence, the clustering analysis reveals that the similarity in the usage of the confidence scale is supramodal, as we failed to observe a clear separation between auditory and visual tasks.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenEighteen of the twenty-three participants completed the LOT-R test.
“Eighteen of the twentythree participants completed the test.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationIndividual differences in policies to express confidence may be cultural, distinguishing groups who express continuous notions of probabilities from those who express it in a more categorical fashion.
“individual differences in policies to express confidence may be cultural, distinguishing groups of people who express (verbally and numerically) continuous notions of probabilities from those who express it in a more categorical (all-or-none) fashion (Phillips & Wright, 1977).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Phillips & Wright, 1977
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationA non-significant correlation between confidence accuracy for memory and for perception was found.
“Baird and colleagues (Baird, Smallwood, Gorgolewski, & Margulies, 2013) showed a non-significant correlation between confidence accuracy for memory and for perception and different connectivity patterns (with the prefrontal cortex as a shared hub) accounting for the variance in the accuracy of mnemonic or perceptual confidence judgments.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Baird, Smallwood, Gorgolewski, & Margulies, 2013
Not checked in the free run.
May combine two claims.
InterpretationreasoningThe flaw in the reasoning is unnoticed to subjects' introspection, and subjects are hence highly confident as they would also be in the wrong assessment of a visual illusion.
“The flaw is of course unnoticed to subject's introspection and subjects are hence highly confident as they would also be in the wrong assessment of a visual illusion.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductreasoningIn the present study, subjects report on a non-calibrated scale, and thus we cannot make precise claims about whether a participant is over or under confident.
“It is important to emphasize that in the present study subjects report in a non-calibrated scale, and thus we cannot make precise claims about whether a participant is over or under confident.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationDifferent connectivity patterns (with the prefrontal cortex as a shared hub) accounted for the variance in the accuracy of mnemonic or perceptual confidence judgments.
“Baird and colleagues (Baird, Smallwood, Gorgolewski, & Margulies, 2013) showed a non-significant correlation between confidence accuracy for memory and for perception and different connectivity patterns (with the prefrontal cortex as a shared hub) accounting for the variance in the accuracy of mnemonic or perceptual confidence judgments.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Baird, Smallwood, Gorgolewski, & Margulies, 2013
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationWeaker but significant correlations were found between the accuracy of confidence judgments based on mnemonic and perceptual decisions.
“McCurdy and colleagues found weaker (but significant) correlations between the accuracy of confidence judgments based on mnemonic and perceptual decisions (McCurdy et al., 2013).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites McCurdy et al., 2013 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationIn the optimism/pessimism classification, there seems to be a bias toward optimism, typically expressed in people believing they will live longer than average, underestimating divorce chances, health problems and risks such as car accidents.
“In the optimism/pessimism classification, there seems to be a bias toward optimism, typically expressed in people believing that they will live longer than average, underestimating divorce chances, health problems and risks such as car accidents (Weinstein, 1980).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Weinstein, 1980 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe calibration curves of each of Swets et al.'s observers were widely different.
“Swets, Tanner, Wilson, and Birdsall (1961) also observed that the calibration curves of each of their observers were widely different.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Swets, Tanner, Wilson, and Birdsall (1961)
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningThe fact that some subjects respond with very high scores (saturating and losing precision in a narrow range of confidence) leads to a re-scaling of confidence without losing precision.
“The fact that some subjects are responding with very high scores (and then seemingly saturating and loosing precision in a narrow range of confidence) leads to a re-scaling of confidence without losing precision.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationThere are cultural and individual differences in the way that people think and conceptualize probabilities and uncertainty.
“there are cultural and individual differences in the way that people think and conceptualize probabilities and uncertainty (Wright et al., 1978).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Wright et al., 1978 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPositive correlations in the accuracy of confidence were found for two tasks that required discrimination of orientation or contrast.
“Song and colleagues found positive correlations in the accuracy of confidence for two tasks that required discrimination of orientation or contrast (Song et al., 2011).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Song et al., 2011 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationPessimism is typical in patients with depressive symptoms, indicating that specific populations may have a bias toward a specific policy of confidence (causality not known).
“Pessimism is typical in patients with depressive symptoms indicating that specific populations may have a bias (the causality of this correlation is not known) toward a specific policy of confidence (Drevets et al., 1997).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Drevets et al., 1997 source
Not checked in the free run.
May combine two claims.
PremisecitationBeing over- or under-confident and generally optimistic or pessimistic is a stable individual trait.
“previous research has also identified that being over or under confident and generally optimistic or pessimist is a stable individual trait (Plomin et al., 1992).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Plomin et al., 1992 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationDistinct cerebral correlates of confidence accuracy for each task suggest the existence of functionally segregated confidence systems in the human brain.
“Moreover, they found distinct cerebral correlates of confidence accuracy for each task suggesting the existence of functionally segregated confidence systems in the human brain.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites McCurdy et al., 2013 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe Life Orientation Test (LOT-R) is a test devised to measure individual differences in generalized optimism/pessimism.
“Participants completed the Life Orientation Test (LOT-R), a test devised to measure individual differences in generalized optimism/pessimism (Scheier, Carver, & Bridges, 1994), in its Spanish version (Perczek, Carver, Price, & Pozo-Kaderman, 2000).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Scheier, Carver, & Bridges, 1994 source
Not checked in the free run.
- cites Perczek, Carver, Price, & Pozo-Kaderman, 2000
Not checked in the free run.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Individual differences in metacognitive efficiency are correlated across distinct task domains, after controlling for correlations in performance.
“Behaviorally, individual differences in metacognitive efficiency have been shown to be correlated across distinct task domains, after controlling for correlations in performance (Ais et al. 2016, Faivre et al. 2018, Mazancieux et al. 2020, Rouault et al. 2018b).”
From Metacognition and Confidence: A Review and Synthesis (2024) · cites this paper as Ais et al. 2016
Section 3.1 reports weak but significantly positive correlations in confidence accuracy across different tasks, supporting the cross-task part of the claim. The tasks were perceptual, and the reported analysis used Pearson correlations of AUC values; our copy does not report controlling those correlations for correlations in task performance. (LLM, conf=0.86)
Experimental — accuracy not yet measuredMetacognitive efficiency shows moderate test-retest reliability, both across different sessions of the same experiment and across different days.
“Metacognitive efficiency shows moderate test-retest reliability, both across different sessions of the same experiment (Ais et al. 2016, Fleming et al. 2010) and across different days (Wright et al. 2012).”
From Metacognition and Confidence: A Review and Synthesis (2024) · cites this paper as Ais et al. 2016
Section 3.1 reports consistently positive correlations in confidence-accuracy AUC across sessions of the same task, supporting within-experiment reliability. The paper measures confidence accuracy, however, rather than a measure it calls metacognitive efficiency. The different-days clause is attributed to Wright et al., not Ais et al. (LLM, conf=0.82)
Experimental — accuracy not yet measuredMetacognitive bias (calibration) shows stronger test-retest reliability, with stable confidence fingerprints seen across different tasks and testing sessions.
“Metacognitive bias (calibration) shows stronger test-retest reliability, with stable confidence fingerprints seen across different tasks and testing sessions (Ais et al. 2016).”
From Metacognition and Confidence: A Review and Synthesis (2024) · cites this paper as Ais et al. 2016
Section 3.3 reports very strong correlations in mean confidence bias across sessions of the same task and significant correlations across different tasks; the Abstract describes confidence distributions as subjective fingerprints. But Section 4.2 says participants used a non-calibrated scale, so the study does not establish the reliability of calibration to an objective norm. Its stable measure is mean confidence bias, not calibration. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe evaluation of additional evidence may explain why confidence ratings are more strongly correlated with accuracy when they are reported with less time pressure, and why some individuals appear to be better than others at discriminating correct from incorrect decisions.
“The evaluation of additional evidence may explain why confidence ratings are more strongly correlated with accuracy when they are reported with less time pressure (Yu et al., 2015), and why some individuals appear to be better than others at discriminating correct from incorrect decisions (e.g. Ais et al., 2016).”
From A common mechanism underlies changes of mind about decisions and confidence (2016) · cites this paper as Ais et al., 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Figure caption misidentifies the KL comparison as inter-subject
MinorThe results describe calculating KL divergence between sessions separately for each subject and then averaging across subjects. The figure caption instead calls the distance "inter-subject," which would mean comparing different subjects.
“Distance was defined as the average across subjects of the inter-subject KL divergence of the distribution of confidence reports (see Section 2).”
“For each subject, we computed the KLdiv for every pair of sessions (a total of 11 2 comparisons).After averaging across subjects, we obtained a matrix M(i,j) which measures the average distance between sessions i and j (Fig. 3b, left).”
The citation lists haven’t been fetched yet
References 60 · Cited by 174
Research works printed but not linked: 1
- Hastie, 2009, Vol. 2
- Authors
- Ais, J., Zylberberg, A., Barttfeld, P., & Sigman, M.
- Affiliations
- Consejo Nacional de Investigaciones Científicas y Técnicas (AR)
- Howard Hughes Medical Institute (US)
- Corresponding author
- Sigman, Mariano
- Venue
- Cognition
- Publisher
- Elsevier BV
- Published
- 2016-01
- Volume
- 146
- Pages
- 377-386
- ISSN
- 0010-0277 (ISSN-L)
- OpenAlex
- W2180932792
- Semantic Scholar
- 6f7c713f7d8a46c345de8087f5a000b01f84725b
- Cited by
- 174
- References
- 60
- Citations by year
- 2026: 21 · 2025: 18 · 2024: 13 · 2023: 32 · 2022: 17 · 2021: 11 · 2020: 15 · 2019: 14 · 2018: 13 · 2017: 13 · 2016: 7
- FWCI
- 6.61
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Cognition (publishedVersion)
- PubMed (publishedVersion)
- Conicet (submittedVersion, cc-by-nc-sa)
- Funders
- James S. McDonnell Foundation
- Secretaría de Ciencia y Técnica, Universidad de Buenos Aires
- Consejo Nacional de Investigaciones Científicas y Técnicas
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Keywords
- metacognition, confidence judgments, confidence distributions, individual differences, confidence bias, perceptual decision making, task specificity, sensory modalities, optimism bias
- MeSH terms
- Metacognition, Adult, Female, Humans, Judgment, Male, Perception, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
