The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index; 7 of the 21 sources its claims cite were checked for support.
Large Language Models lack essential metacognition for reliable medical reasoning
Open accessAbstract
Large Language Models have demonstrated expert-level accuracy on medical board examinations, suggesting potential for clinical decision support systems. However, their metacognitive abilities, crucial for medical decision-making, remain largely unexplored. To address this gap, we developed MetaMedQA, a benchmark incorporating confidence scores and metacognitive tasks into multiple-choice medical questions. We evaluated twelve models on dimensions including confidence-based accuracy, missing answer recall, and unknown recall. Despite high accuracy on multiple-choice questions, our study revealed significant metacognitive deficiencies across all tested models. Models consistently failed to recognize their knowledge limitations and provided confident answers even when correct options were absent. In this work, we show that current models exhibit a critical disconnect between perceived and actual capabilities in medical reasoning, posing significant risks in clinical settings. Our findings emphasize the need for more robust evaluation frameworks that incorporate metacognitive abilities, essential for developing reliable Large Language Model enhanced clinical decision support systems. Large Language Models demonstrate expert-level accuracy in medical exams, supporting their potential inclusion in healthcare settings. Here, authors reveal that their metacognitive abilities are underexplored, showing significant gaps in recognizing knowledge limitations, difficulties in modulating their confidence, and challenges in identifying when a problem cannot be answered due to insufficient information.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces MetaMedQA, a modified MedQA-USMLE benchmark with confidence scoring and unanswerable/missing-answer tasks, and evaluates 12 LLMs on medical multiple-choice questions. It finds that despite high accuracy, models largely fail to recognize knowledge limits, rarely use low confidence, and almost never identify unknown or unanswerable questions.
Key findings
- MetaMedQA contains 1,373 questions with six answer choices and one correct choice; it includes 100 fictional Glianorex questions, 55 manually identified malformed questions, and 125 modified questions.
- Overall accuracy tracked model size and recency: GPT-4o-2024-05-13 reached 73.3% (SEM = 1.2%), Qwen2 72B 64.3% (SEM = 1.3%), Qwen2 7B 43.9% (SEM = 1.3%), and Yi 1.5 9B 29.6% (SEM = 1.2%).
- Most models always assigned maximum confidence 5; only GPT-3.5-turbo-0125, GPT-4o-2024-05-13, and Qwen2-72B varied confidence. GPT-4o had high-confidence accuracy 83.2%, medium-confidence accuracy 45.9%, and low-confidence accuracy 16.7%.
- Missing-answer recall was low: GPT-4o scored 46.1% (SEM = 4.7%) versus GPT-3.5-turbo-0125 at 11.3% (SEM = 2.9%). Unknown recall was near zero for most models, with GPT-4o at 3.7%, Meerkat 7B at 1.2%, and Yi 1.5 34B at 0.6%; prompt engineering raised GPT-4o unknown recall to 51.8% with the “role-full” prompt.
Methods
- The authors created MetaMedQA from MedQA-USMLE by adding fictional questions, identifying malformed questions, and modifying selected questions, yielding 1,373 six-choice questions.
- They evaluated 12 proprietary and open-weight models, including GPT-4o-2024-05-13, GPT-3.5-turbo-0125, Mixtral-8x7B, Mistral-7B, Yi-1.5-9B/34B, Llama-3-8B/70B, Qwen2-7B/72B, Meerkat-7B, and Internist-7B, using temperature 0 and constrained decoding via Guidance.
- Main analyses covered overall accuracy, confidence-based accuracy, missing-answer recall, unknown recall, and prompt engineering on GPT-4o; statistical comparisons used one-way ANOVA with Tukey correction, and prompt comparisons used Fisher’s least significant difference test.
Limitations
- The paper states that MedQA, even modified, may not capture real-world clinical complexity; multiple-choice questions cannot replicate nuanced clinical decision-making, and manual modifications/audits may introduce human error or interpretation bias.
- The paper states that confidence scoring from 1 to 5 may not fully represent model certainty, the evaluated models were limited to those available as of June 2024, proprietary models limit insight into training data, and the reliance on Dual Process Theory may not fully represent clinical cognition; the work investigates System 1 thinking exclusively.
- Evident from the text: unknown-recall analysis was severely limited because 9 of 12 models scored 0, making regression/correlation results hard to interpret; confidence analysis was limited to only three models that varied confidence, and prompt engineering was tested only on GPT-4o.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
35 Own result · 22 Own interpretation · 9 No full text of the sources · 6 Own method · 6 No basis given · 4 Supported · 3 Partly supported · 2 Not checked yet · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While these methods allow for the swift assessment of model performance, they are primarily limited to evaluating pattern recognition and information recall2.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
What the sources say
- Partly supportedcites [2] Freiwald, T., Salimi, M., Khaljani, E. & Harendza, S. Pattern recognition as a concept for multiple-choice questions in a national licensing exam. BMC Med. Educ. 14, 232 (2014). sourceExperimental — accuracy not yet measured
The Methods classify questions as pattern-recognition, knowledge-recall, or related types, and the Results report that 42.2% were pattern-recognition questions and 44.3% were knowledge questions. But the study examined four disciplines in one German medical licensing exam, not multiple-choice evaluations generally or assessments of model performance. The Background also says well-constructed MCQs can test application of knowledge, so the claim's broad limitation goes beyond the paper. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“These findings reinforce previous research recommendations on mitigating healthcare data biases in machine learning30, identifying a probable training data bias that predisposes models to provide confident answers in most scenarios, even when a more cautious response is warranted.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Supportedcites [30] Ghassemi, M., Nsoesie, E. O. In medicine, how do we machine learn anything real? Patterns https://doi.org/10.1016/j.patter.2021.100392 (2024). source
Citation 30 is attached to the statement that prior research recommends mitigating healthcare data biases in machine learning. In “What role does machine learning have?” (page unavailable), the source explicitly offers recommendations for algorithm developers, including recognizing bias and examining who is excluded from data. The subsequent inference about models giving confident answers is not the part attributed to this source. (LLM, conf=0.91)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Recent studies on GPT-4V’s performance on medical multiple-choice questions demonstrated that despite the impressive results of models on multiple-choice questions, the rationale behind correct answers is flawed in a significant percentage of cases39.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Supportedcites [39] Jin, Q. et al. Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine. Npj Digit. Med. 7, 1–6 (2024). source
The Abstract reports that GPT-4V achieved 81.6% accuracy on medical multiple-choice questions, yet presented flawed rationales in 35.5% of cases where its final choice was correct. This directly supports the attributed claim. (LLM, conf=0.95)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Another analysis of GPT-4’s errors on the USMLE demonstrated that most errors are either caused by an anchoring bias or incorrect conclusions40.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Partly supportedcites [40] Roy, S. et al. Beyond accuracy: Investigating error types in GPT-4 responses to USMLE questions. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. (2024). sourceExperimental — accuracy not yet measured
The Introduction (p. 2) identifies “sticking with the wrong diagnosis” and “incorrect or vague conclusion” as the most frequent reasoning-error subclasses; the former resembles anchoring. But §5.2 (p. 8) reports them at 26.9% and 22.21%, respectively, and does not establish that most errors fall into either category. The paper also describes the pattern as observed in its annotated subset, not all GPT-4 USMLE errors. (LLM, conf=0.87)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Crucially, the ability to recognize knowledge limitations enables clinicians to dynamically shift between these cognitive strategies33.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Supportedcites [33] Zwaan, L., Hautz, W. E. Bridging the gap between uncertainty, confidence and diagnostic accuracy: calibration is key. BMJ Qual. Saf. 28, 352–355 (2019). source
In “Making decisions while acknowledging uncertainty,” the paper says metacognition helps clinicians recognise their confidence in a decision, slow down when needed, and override intuitive impressions with deliberate reasoning. This supports the paraphrase about recognising limitations and shifting between cognitive strategies. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the reliance on the Dual Process Theory (DPT)42 may not accurately represent the cognition processes involved in clinical decision-making.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Partly supportedcites [42] Bellini-Leite, S. C. Dual Process Theory: Embodied and Predictive; Symbolic and Classical. Front. Psychol. 13, https://doi.org/10.3389/fpsyg.2022.805386 (2022). sourceExperimental — accuracy not yet measured
The Features of Dual Process Theory section says DPT’s features have been criticized as poorly defined, and the Conclusion says the theory needs further development. Those passages support caution about relying on DPT, but the paper does not assess whether it accurately represents cognition in clinical decision-making specifically. (LLM, conf=0.79)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, only three models, with GPT-4o standing out, effectively vary their confidence levels.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“there was no statistically significant correlation (Pearson r = 0.574, p = 0.051).”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Unknown analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“most models exhibit poor performance in recognizing unanswerable questions and managing uncertainty”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4o stood out in this regard, showing a robust capacity to provide higher accuracy when highly confident compared to answers with lower confidence scores.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Qwen2 72B (M = 64.3%, SEM = 1.3%) is significantly more accurate (p < 0.0001) than Qwen2 7B (M = 43.9%, SEM = 1.3%) with a moderate effect size (Cohen’s d = 0.417).”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Overall accuracy
plainn = 1,373- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The other two models, however, only provided high or medium confidence scores, never utilizing low confidence ratings.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Impact of confidence
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“with GPT-4o demonstrating the best ability to assess its answers accurately.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Impact of confidence
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When examining other models, we found that larger and more recent models generally outperformed their smaller and older counterparts, mirroring the overall accuracy pattern.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yi 1.5 9B (M = 29.6%, SEM = 1.2%) is significantly less accurate than all other models (p < 0.0001).”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Overall accuracy
plainn = 1,373- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recall of “None of the above” answers revealed significant differences in how models handle uncertainty.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The regression yielded a statistically significant positive slope of 1.319 (95% CI: 1.136 - 1.502, p < 0.0001) as shown in Fig. 4.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
plaincorrelationaln = 115- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The “unknown recall” metric, assessing the ability to recognize unanswerable questions, showed poor performance across all models, highlighting a fundamental limitation in current LLMs’ metacognitive abilities.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After excluding the outliers Yi 1.5 9B and Meerkat 7B, which selected “None of the above” for 37.9% and 21.5% of questions respectively (greatly overestimating their performance in this category), we found a strong positive correlation between these two metrics by calculating the Pearson correlation coefficient with a two-tailed p-value (Pearson r = 0.947, p < 0.0001).”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we manually audited the benchmark and identified 55 questions that either relied on missing media or lacked necessary information.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Benchmark creation and preprocessing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For these models, higher confidence levels were correlated with higher accuracy”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Impact of confidence
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This proved to be the most challenging task for the models, with most scoring 0%.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Unknown analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results obtained by the different models correlate with their size and release date; larger and more recent models achieved higher accuracy than their smaller and older counterparts as shown in Fig. 2.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Overall accuracy
plaincorrelationaln = 1,373- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The notably low performance of Yi 1.5 9B, compared to similar-sized models stands out as an outlier.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Overall accuracy
plainn = 1,373- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only GPT-3.5-turbo-0125, GPT-4o-2024-05-13, and Qwen2-72B exhibited varying confidence levels, as shown in Table 3.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Impact of confidence
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the regression analysis yielded a statistically significant slope of 0.05232 (95% CI: 0.0231 - 0.0815, p < 0.001)”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Unknown analysis
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4o-2024-05-13 (M = 73.3%, SEM = 1.2%) is significantly more accurate than all other models (p < 0.0001)”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Overall accuracy
plainn = 1,373- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A significant improvement in accuracy, high confidence accuracy, and unknown recall appeared (p < 0.0001) once the prompt explicitly informs the model that it may not be able to answer some questions, as shown in Table 4.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Prompt engineering analysis
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This proved to be the most challenging task for the models, with most scoring 0%.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Unknown analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this work, we show that current LLMs demonstrate significant limitations in metacognitive abilities crucial for clinical decision-making.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4o-2024-05-13 (M = 46.1%, SEM = 4.7%) is significantly more accurate (p < 0.0001) than GPT-3.5-turbo-0125 (M = 11.3%, SEM = 2.9%) with a large effect size (d = 0.826).”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recall of missing answers when the correct response is “None of the above,” as shown in Fig. 3, indicates that models struggle more with this option compared to others.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The discrepancy between performance on standard questions and metacognitive tasks highlights a critical area for improvement in LLM development.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This can be attributed to the model selecting “None of the above” 520 times, or 37.9% of the questions, leading to an inflated score in this area but poor performance on other metrics.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, models that rarely chose this option might be overly confident, missing opportunities to acknowledge when none of the given answers are correct.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These findings emphasize the limits of multiple-choice to assess cognitive capabilities, especially in reasoning tasks.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“studies examining specific tasks, such as ICD (International Classification of Diseases) coding9, have revealed significant performance deficiencies”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
What the sources say
- Not checked — no full text of the sourcecites [9] Soroush, A. et al. Large language models are poor medical coders — benchmarking of medical code querying. NEJM AI 1, AIdbp2300040 (2024). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The limited use of low confidence scores by most models suggests a tendency toward overconfidence, which could pose risks if models are used in clinical practice without appropriate checks.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This trend suggests that advancements in model architecture and training techniques contribute significantly to improved accuracy.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indicates that models with higher overall accuracy generally performed better at identifying missing answers.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
hedgedcorrelationaln = 115- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Cognitive errors are an important source of diagnostic error35”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Not checked — no full text of the sourcecites [35] Croskerry, P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad. Med. 78, 775 (2003). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, such testing methodologies are inherently limited. They focus predominantly on accuracy in answering specific questions, without adequately addressing the critical aspects of model safety and the potential for generating erroneous or misleading information.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Current LLMs, despite their capabilities, exhibit overconfidence and deficiencies in recognizing their limitations, making them unlikely to appropriately employ these nuanced strategies.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the inclusion of both proprietary and open-weight models allows for a broader assessment, demonstrating that our findings are not confined to a single type of model but rather indicative of general trends in LLM performance and metacognitive abilities.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“these results suggest that current LLMs, despite high accuracy on certain tasks, lack essential capabilities for safe deployment in clinical settings.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“studies have shown that switching to System 2 may not always reduce reasoning errors in humans43.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Not checked — no full text of the sourcecites [43] Norman, G. R. et al. The Causes of Errors in Clinical Reasoning: Cognitiv. Acad. Med. 92, 23–30 (2017). source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 24
Every source the paper cites for this claim was checked and supports it.
In the paper
“LLMs also appear to suffer from a similar shortcoming and fail to self-correct when their reasoning is faulty44.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Supportedcites [44] Huang, J. et al. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations. (2024). source
The Conclusion and Discussion (p. 8) states that current LLMs struggle to recognize and rectify their reasoning mistakes without external feedback. That supports the citing sentence’s hedged claim about failure to self-correct faulty reasoning; the paper’s finding concerns unaided self-correction. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that most models consistently assigned a maximum confidence level of 5, rendering them unsuitable for the confidence analysis.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Impact of confidence
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, the Llama 3 8B model almost never selected this option, while the Mistral 7B and Internist 7B models never did.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The models either never used this answer choice or used it less than 10 times over the 1373 questions.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Unknown analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observed that the model fails to use mid and low confidence appropriately when given additional instructions in the system prompt, but the high confidence accuracy was either similar to or higher than baseline.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Prompt engineering analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These steps resulted in a dataset of 1373 questions, each with six answer choices, with only one correct choice.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Benchmark creation and preprocessing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While we observed some enhancements in metacognitive task performance through prompt engineering with GPT-4o, these improvements remain constrained.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing answer recall improved when the prompt explicitly informs the model that the correct answer might not be present in the choices, but it was not statistically significant (p = 0.07).”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Prompt engineering analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Exceptions were GPT-4o-2024-05-13, which achieved 3.7%, Yi 1.5 34B which scored 0.6%, and Meerkat 7B with 1.2%.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Unknown analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Prompts had to explicitly inform the LLM of potential biases and dangers, necessitating an exhaustive—and impractical—list of all potential pitfalls for real-world applications.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A similar but less pronounced trend was observed with the Meerkat 7B model, which chose “None of the above” 295 times.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Yi 1.5 9B model, which had the lowest overall accuracy, achieved the highest score on this specific metric.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, providing the complete benchmark design instructions did not improve the performance compared to baseline except for unknown recall but underperforms compared to explicit prompts.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Prompt engineering analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the tendency of models to avoid low confidence scores suggests a potential bias towards overconfidence.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“methods such as reflective medical practice36, which help clinicians enhance their ability to navigate complex cases37, or debiasing through feedback to identify and correct cognitive biases38 can help in reducing the number of cognitive errors.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Not checked — no full text of the sourcecites [36] Mamede, S. & Schmidt, H. G. The structure of reflective practice in medicine. Med. Educ. 38, 1302–1308 (2004). source
- Not checked — no full text of the sourcecites [37] Mamede, S., Schmidt, H. G. & Penaforte, J. C. Effects of reflective practice on the accuracy of medical diagnoses. Med. Educ. 42, 468–475 (2008). source
- Not matchedcites [38] Diagnosing Diagnosis Errors: Lessons from a Multi-institutional Collaborative Project. in Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology) - NCBI Bookshelf. (2024).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The predominance of zero scores severely limits the interpretability and practical significance of these statistical findings.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Unknown analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This inability to reliably indicate when they lack sufficient information or knowledge suggests a risk of generating misleading or incorrect information, which could have serious implications in clinical applications.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“the manual modifications and audits we performed, although thorough, are subject to human error and interpretation biases.”
- Also stated in: § Discussion · hedged
“The selection and modification of questions, as well as the auditing process, could have introduced subjective biases affecting the outcomes of our evaluations.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Also stated in: § Discussion · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, their fixed nature and the challenges associated with providing meaningful feedback leave minimal room for improvement.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Metacognition in AI systems can be split into two categories23: knowledge of cognition and regulation of cognition.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
What the sources say
- Not checked — no full text of the sourcecites [23] Gonullu, I. & Artar, M. Metacognition in medical education. Educ. Health 27, 225 (2014). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Clinicians adeptly employ these cognitive strategies concurrently, selecting the most appropriate approach based on their expertize32.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Not checked — no full text of the sourcecites [32] Bowen, J. L. Educational strategies to promote clinical diagnostic reasoning. N. Engl. J. Med. 355, 2217–2225 (2006). source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the MedQA benchmark, even with our modifications, may not fully capture the complexity and variability of real-world clinical scenarios.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The limited access to external tools, along with their imperfections raises concerns about relying solely on such tools to prevent errors stemming from metacognitive deficits.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that LLMs’ tendency towards overconfidence and inability to recognize knowledge gaps pose potential risks in clinical applications.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additional medical training demonstrated an improvement in accuracy for both models and the ability to detect missing answers for Meerkat 7b which could be explained by the inclusion of questions with 5 choices and a wider range of questions in the training dataset.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Current LLMs, despite their capabilities, exhibit overconfidence and deficiencies in recognizing their limitations, making them unlikely to appropriately employ these nuanced strategies.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the controlled nature of multiple-choice questions cannot replicate the nuanced decision-making processes required in clinical practice.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Beyond internal processes, clinicians may also leverage external resources, such as clinical guidelines or second opinions, to inform their decision-making34.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Not checked — no full text of the sourcecites [34] Elstein, A. S. Thinking about diagnostic thinking: a 30-year perspective. Adv. Health Sci. Educ. 14, 7–18 (2009). source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Comprehensive models of cognition, such as the transtheoretical model31 incorporate external factors, including interactions with team members or databases, which could be implemented for LLMs with Retrieval Augmented Generation31.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Not checked — no full text of the sourcecites [31] Parsons, A. S. et al. Beyond thinking fast and slow: Implications of a transtheoretical model of clinical reasoning and error on teaching, assessment, and research. Med. Teach. 1–12 https://doi.org/10.1080/0142159x.2024.2359963 (2024). source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“the systematic approach and open access to our modifications ensure that our findings remain reliable and reproducible”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Recent efforts have included the use of official board examinations to evaluate LLM performance across different medical specialties such as pediatrics3, oncology4, ophthalmology5, radiology6, or plastic surgery7, often demonstrating that these models can perform at a level comparable to medical professionals8.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
What the sources say
- Not checked — no full text of the sourcecites [8] Katz, U. et al. GPT versus resident physicians — A benchmark based on official board scores. NEJM AI 1, AIdbp2300192 (2024). source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the notably poor performance of certain models like Yi 1.5-9B, despite being relatively recent, indicates that model optimization and specific training datasets also play crucial roles.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgeddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While external factors might partially compensate for the lack of internal metacognition, this approach presents limitations.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“We used Python 3.12 and Guidance, a Python library designed to enforce model adherence to specific instructions through constrained decoding45, ensuring the models selected only from the allowed choices (A/B/C/D/E/F) and provided confidence scores (1/2/3/4/5)46.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Methods > Benchmark procedure
What the sources say
- Not matchedcites [45] guidance-ai/guidance: A guidance language for controlling large language models. https://github.com/guidance-ai/guidance (2023).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [46] Griot, M., Hemptinne, C., Vanderdonckt, J., Yuksel, D. MetaMedQA benchmark code. https://doi.org/10.5281/zenodo.14177940 (2024). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
In the paper
“All models were evaluated with a temperature setting of 0 to ensure reliability and reproducibility of results56.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Methods > Benchmark procedure
What the sources say
- Not checked — no full text of the sourcecites [56] Ronanki, K., Cabrero-Daniel, B., Horkoff, J. & Berger, C. Requirements engineering using generative AI: prompts and prompting patterns. In Generative AI for Effective Software Development. (Springer, Cham, 2024). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Considering LLMs have restricted access to external cognitive processes, we argue that internal cognition processes from the DPT must compensate.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: LLMs have restricted access to external cognitive processes.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This temporal limitation means the findings may not be fully generalizable to future models or those trained with different objectives and datasets.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The proprietary nature of some models, such as OpenAI’s GPT-4o, limits our insight into their training data and methodologies. This constraint could influence their performance and the interpretation of our results.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This gap raises concerns about a form of deceptive expertise, where systems appear knowledgeable but fail to recognize their own limitations.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The final metrics, including confidence accuracy, missing answer recall, and unknown recall, are designed to provide a comprehensive assessment but may not capture all aspects of model performance and safety.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“while we implemented a confidence scoring system to capture models’ confidence levels on a scale from 1 to 5, this may not fully represent the nuanced levels of certainty a model might have.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The metacognitive abilities of LLMs remain largely unexplored.
“However, their metacognitive abilities, crucial for medical decision-making, remain largely unexplored.”
“we propose investigating a crucial but underexplored area: the assessment of LLMs’ metacognition.”
The paper relies on this without support.
LLMs have restricted access to external cognitive processes.
“Considering LLMs have restricted access to external cognitive processes, we argue that internal cognition processes from the DPT must compensate.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This vulnerability is amplified by a structural property of LLMs, which may express high confidence in incorrect conclusions, producing an illusion of expertise that expert clinicians can often detect and novices cannot 30 .”
From AI-induced never-skilling in medical education (2026) · cites this paper as [30]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Griot et al. show that medical LLMs may perform well on benchmark assessments while lacking critical metacognitive reliability, such as recognizing when correct options are absent [19].”
From Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence (2026) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“Prior studies have noted the presence of malformed and incomplete questions in the MedQA benchmark,22 consistent with our observations during evaluation.”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“While acknowledging various advantages of LLMs (Choi and Ferrara, 2024; Zhao et al., 2024), including their support for information extraction and synthesis, facts checking, multilingual analysis, and decision-support assistants, recent work (Griot et al., 2025) has also highlighted their important limitations in uncertainty handling, self-assessment, and metacognitive reliability in high-stakes”
From Enhancement of modern defense capabilities by convergence of cognitive neuroscience and artificial intelligence (2026) · cites this paper as Griot et al., 2025
The cited source says what the claim attributes to it.
In the citing paper
“Benchmarks, such as MetaMedQA, further quantify these shortcomings by measuring metrics such as Confidence Accu-racy and Unknown Recall, which gauge a model's ability to recognize when it does not know the answer (Griot et al., 2025).”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Griot et al., 2025
In the citing paper
“a recent study revealed a substantial discrepancy between LLMs’ miscalibrated overconfidence in their outputs and their actual accuracy, underscoring the risks of overreliance on LLMs in the medical domain25.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [25]
In the citing paper
“These results align with recent findings suggesting current LLMs lack metacognition, demonstrating a mismatch between low uncertainty (or overconfidence) and their actual capabilities in medical reasoning tasks25.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [25]
In the citing paper
“Compounding these shortcomings are recently demonstrated LLM deficiencies in metacognition—specifically, the inability to recognize their own limitations—and overconfidence25.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [25]
In the citing paper
“Whereas conventional QA benchmarks are established at assessing human clinical reasoning, their reliability at evaluating LLM reasoning remains unclear25.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Prompt engineering analysis
“A significant improvement in accuracy, high confidence accuracy, and unknown recall appeared (p < 0.0001) once the prompt explicitly informs the model that it may not be able to answer some questions”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Prompt engineering analysis
“Accuracy: role-explicit-unknown, p = 0.013”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
“Pearson r = 0.947, p < 0.0001”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
“The regression yielded a statistically significant positive slope of 1.319 (95% CI: 1.136 - 1.502, p < 0.0001) as shown in Fig. 4.”
Citations
Cites
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Artificial intelligence in healthcare: transforming the practice of medicine
Bajwa, J., Munir, U., Nori, A., & Williams, B. (2021) · Future Healthcare Journal · doi:10.7861/fhj.2021-0095 · Cited by 1,856 · cited 1×
“AI offers immensepotentialtoaddresstheshortageofhealthcareworkers 16 and promises to reduce clerical work 17,18 , and has already demonstrated uses in precision diagnostics and therapeutics 19 .”
The Importance of Cognitive Errors in Diagnosis and Strategies to Minimize Them
Croskerry, P. (2003) · Academic Medicine · doi:10.1097/00001888-200308000-00003 · Cited by 1,675 · cited 1×
“Cognitive errors are an important source of diagnostic error 35 and methods such as re fl ective medical practice 36 , which help clinicians enhance their ability to navigate complex cases 37 , or debiasing through feedback to identify and correctcognitivebiases 38…”
Educational Strategies to Promote Clinical Diagnostic Reasoning
Bowen, J. L. (2006) · New England Journal of Medicine · doi:10.1056/nejmra054782 · Cited by 931 · cited 1×
“Clinicians adeptly employ these cognitive strategies concurrently, selecting the most appropriate approach based on their expertize 32 .”
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 712 · cited 1×
“LLMs also appear to suffer from a similar shortcoming and fail to self-correct when their reasoning is faulty 44 .”
Norman, G. R., Monteiro, S. D., Sherbino, J., Ilgen, J. S., Schmidt, H. G., & Mamede, S. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001421 · Cited by 594 · cited 1×
“While additional experiments involving System 2 should be conducted to improve our understanding of LLM cognition, it ’ s important to note that studies have shown that switching to System 2 may not always reduce reasoning errors in humans 43 .”
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Research works printed but not linked: 1
- American Medical Association. Physician sentiments around the use of AI in heath care: motivations, opportunities, risks, and use cases. https://www.ama-assn.org/system/files/physician-ai-sentiment-report.pdf (2023).
Cited by
Sorted by citations
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78
“These results align with recent findings highlighting a mismatch between LLM uncertainty and their actual capabilities in medical reasoning tasks[25].”
InfluentialEvaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 97
MedAgentBench: a virtual EHR environment to benchmark medical LLM agents
Jiang, Y., Black, K. C., Geng, G., Park, D., Zou, J., Ng, A. Y., & Chen, J. H. (2025) · NEJM AI · doi:10.1056/aidbp2500144 · Cited by 66
Take caution in using LLMs as human surrogates
Gao, Y., Lee, D., Burtch, G., & Fazelpour, S. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2501660122 · Cited by 52
A large language model improves clinicians’ diagnostic performance in complex critical illness cases
Wu, X. C., Huang, Y., & He, Q. (2025) · Critical Care · doi:10.1186/s13054-025-05468-7 · Cited by 51
“LLMs may confidently produce misinformation or exhibit cognitive biases, underscoring the risks of unsupervised reliance on these tools [11, 26, 27].”
Metadata
- Authors
- Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D.
- Affiliations
- UCLouvain (BE)
- Cliniques Universitaires Saint-Luc (BE)
- Corresponding author
- Griot, Maxime
- Venue
- Nature Communications
- Published
- 2025-01-14
- Volume
- 16
- Issue
- 1
- Pages
- 642-642
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC11733150
- OpenAlex
- W4406371336
- Semantic Scholar
- 1aaa4c92848a921714ee438ea7c2785888f6a49f
- Cited by
- 172
- References
- 49
- Citations by year
- 2026: 123 · 2025: 48
- FWCI
- 32.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Nature Communications (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Digital Access to Libraries (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Fondation Saint-Luc, Grant 467E Fond Spécial de Recherche of Université catholique de Louvain
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, metacognition, medical reasoning, confidence calibration, clinical decision support, limits of knowledge, medical licensing examinations
- MeSH terms
- Clinical Decision-Making, Metacognition, Clinical Reasoning, Large Language Models, Decision Making, Humans, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
