CoChat
Sign Up
Cited by 119Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareTopic Modeling

Large Language Models lack essential metacognition for reliable medical reasoning

Open access
Griot, Maxime, Hemptinne, Coralie, Vanderdonckt, Jean, Yuksel, Demet·2025·Nature Communications
DOI 10.1038/s41467-024-55628-6PMC PMC11733150
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models have demonstrated expert-level accuracy on medical board examinations, suggesting potential for clinical decision support systems. However, their metacognitive abilities, crucial for medical decision-making, remain largely unexplored. To address this gap, we developed MetaMedQA, a benchmark incorporating confidence scores and metacognitive tasks into multiple-choice medical questions. We evaluated twelve models on dimensions including confidence-based accuracy, missing answer recall, and unknown recall. Despite high accuracy on multiple-choice questions, our study revealed significant metacognitive deficiencies across all tested models. Models consistently failed to recognize their knowledge limitations and provided confident answers even when correct options were absent. In this work, we show that current models exhibit a critical disconnect between perceived and actual capabilities in medical reasoning, posing significant risks in clinical settings. Our findings emphasize the need for more robust evaluation frameworks that incorporate metacognitive abilities, essential for developing reliable Large Language Model enhanced clinical decision support systems. Large Language Models demonstrate expert-level accuracy in medical exams, supporting their potential inclusion in healthcare settings. Here, authors reveal that their metacognitive abilities are underexplored, showing significant gaps in recognizing knowledge limitations, difficulties in modulating their confidence, and challenges in identifying when a problem cannot be answered due to insufficient information.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces MetaMedQA, a modified MedQA-USMLE benchmark with confidence scoring and unanswerable/missing-answer tasks, and evaluates 12 LLMs on medical multiple-choice questions. It finds that despite high accuracy, models largely fail to recognize knowledge limits, rarely use low confidence, and almost never identify unknown or unanswerable questions.

Key findings

  • MetaMedQA contains 1,373 questions with six answer choices and one correct choice; it includes 100 fictional Glianorex questions, 55 manually identified malformed questions, and 125 modified questions.
  • Overall accuracy tracked model size and recency: GPT-4o-2024-05-13 reached 73.3% (SEM = 1.2%), Qwen2 72B 64.3% (SEM = 1.3%), Qwen2 7B 43.9% (SEM = 1.3%), and Yi 1.5 9B 29.6% (SEM = 1.2%).
  • Most models always assigned maximum confidence 5; only GPT-3.5-turbo-0125, GPT-4o-2024-05-13, and Qwen2-72B varied confidence. GPT-4o had high-confidence accuracy 83.2%, medium-confidence accuracy 45.9%, and low-confidence accuracy 16.7%.
  • Missing-answer recall was low: GPT-4o scored 46.1% (SEM = 4.7%) versus GPT-3.5-turbo-0125 at 11.3% (SEM = 2.9%). Unknown recall was near zero for most models, with GPT-4o at 3.7%, Meerkat 7B at 1.2%, and Yi 1.5 34B at 0.6%; prompt engineering raised GPT-4o unknown recall to 51.8% with the “role-full” prompt.

Methods

  • The authors created MetaMedQA from MedQA-USMLE by adding fictional questions, identifying malformed questions, and modifying selected questions, yielding 1,373 six-choice questions.
  • They evaluated 12 proprietary and open-weight models, including GPT-4o-2024-05-13, GPT-3.5-turbo-0125, Mixtral-8x7B, Mistral-7B, Yi-1.5-9B/34B, Llama-3-8B/70B, Qwen2-7B/72B, Meerkat-7B, and Internist-7B, using temperature 0 and constrained decoding via Guidance.
  • Main analyses covered overall accuracy, confidence-based accuracy, missing-answer recall, unknown recall, and prompt engineering on GPT-4o; statistical comparisons used one-way ANOVA with Tukey correction, and prompt comparisons used Fisher’s least significant difference test.

Limitations

  • The paper states that MedQA, even modified, may not capture real-world clinical complexity; multiple-choice questions cannot replicate nuanced clinical decision-making, and manual modifications/audits may introduce human error or interpretation bias.
  • The paper states that confidence scoring from 1 to 5 may not fully represent model certainty, the evaluated models were limited to those available as of June 2024, proprietary models limit insight into training data, and the reliance on Dual Process Theory may not fully represent clinical cognition; the work investigates System 1 thinking exclusively.
  • Evident from the text: unknown-recall analysis was severely limited because 9 of 12 models scored 0, making regression/correlation results hard to interpret; confidence analysis was limited to only three models that varied confidence, and prompt engineering was tested only on GPT-4o.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41467-024-55628-6

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

7 of 21 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Existing multiple-choice evaluations are primarily limited to evaluating pattern recognition and information recall.

    “While these methods allow for the swift assessment of model performance, they are primarily limited to evaluating pattern recognition and information recall2.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [2] Freiwald, T., Salimi, M., Khaljani, E. & Harendza, S. Pattern recognition as a concept for multiple-choice questions in a national licensing exam. BMC Med. Educ. 14, 232 (2014). source

      The Methods classify questions as pattern-recognition, knowledge-recall, or related types, and the Results report that 42.2% were pattern-recognition questions and 44.3% were knowledge questions. But the study examined four disciplines in one German medical licensing exam, not multiple-choice evaluations generally or assessments of model performance. The Background also says well-constructed MCQs can test application of knowledge, so the claim's broad limitation goes beyond the paper. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Interpretationreasoning

    These findings identify a probable training data bias that predisposes models to provide confident answers in most scenarios, even when a more cautious response is warranted.

    “These findings reinforce previous research recommendations on mitigating healthcare data biases in machine learning30, identifying a probable training data bias that predisposes models to provide confident answers in most scenarios, even when a more cautious response is warranted.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [30] Ghassemi, M., Nsoesie, E. O. In medicine, how do we machine learn anything real? Patterns https://doi.org/10.1016/j.patter.2021.100392 (2024). source
  • Premisecitation

    Recent studies on GPT-4V's performance on medical multiple-choice questions demonstrated that despite impressive results, the rationale behind correct answers is flawed in a significant percentage of cases.

    “Recent studies on GPT-4V’s performance on medical multiple-choice questions demonstrated that despite the impressive results of models on multiple-choice questions, the rationale behind correct answers is flawed in a significant percentage of cases39.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [39] Jin, Q. et al. Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine. Npj Digit. Med. 7, 1–6 (2024). source
  • Premisecitation

    An analysis of GPT-4's errors on the USMLE demonstrated that most errors are either caused by an anchoring bias or incorrect conclusions.

    “Another analysis of GPT-4’s errors on the USMLE demonstrated that most errors are either caused by an anchoring bias or incorrect conclusions40.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [40] Roy, S. et al. Beyond accuracy: Investigating error types in GPT-4 responses to USMLE questions. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. (2024). source

      The Introduction (p. 2) identifies “sticking with the wrong diagnosis” and “incorrect or vague conclusion” as the most frequent reasoning-error subclasses; the former resembles anchoring. But §5.2 (p. 8) reports them at 26.9% and 22.21%, respectively, and does not establish that most errors fall into either category. The paper also describes the pattern as observed in its annotated subset, not all GPT-4 USMLE errors. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Premisecitation

    The ability to recognize knowledge limitations enables clinicians to dynamically shift between System 1 and System 2 cognitive strategies.

    “Crucially, the ability to recognize knowledge limitations enables clinicians to dynamically shift between these cognitive strategies33.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [33] Zwaan, L., Hautz, W. E. Bridging the gap between uncertainty, confidence and diagnostic accuracy: calibration is key. BMJ Qual. Saf. 28, 352–355 (2019). source
  • May combine two claims. 25
  • Classified as probably not a substantial claim. 24

  • May combine two claims.

    Premisecitation

    LLMs appear to suffer from a similar shortcoming and fail to self-correct when their reasoning is faulty.

    “LLMs also appear to suffer from a similar shortcoming and fail to self-correct when their reasoning is faulty44.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [44] Huang, J. et al. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations. (2024). source

  • May combine two claims.

    Findingown data

    Most models consistently assigned a maximum confidence level of 5, rendering them unsuitable for confidence analysis.

    “We observed that most models consistently assigned a maximum confidence level of 5, rendering them unsuitable for the confidence analysis.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Llama 3 8B almost never selected 'None of the above', while Mistral 7B and Internist 7B never did.

    “Conversely, the Llama 3 8B model almost never selected this option, while the Mistral 7B and Internist 7B models never did.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Models either never used the unknown answer choice or used it less than 10 times over the 1373 questions.

    “The models either never used this answer choice or used it less than 10 times over the 1373 questions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The model fails to use mid and low confidence appropriately when given additional instructions in the system prompt, but high confidence accuracy was either similar to or higher than baseline.

    “We also observed that the model fails to use mid and low confidence appropriately when given additional instructions in the system prompt, but the high confidence accuracy was either similar to or higher than baseline.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The MetaMedQA dataset contains 1373 questions, each with six answer choices, with only one correct choice.

    “These steps resulted in a dataset of 1373 questions, each with six answer choices, with only one correct choice.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Prompt engineering with GPT-4o produced some enhancements in metacognitive task performance, but these improvements remain constrained.

    “While we observed some enhancements in metacognitive task performance through prompt engineering with GPT-4o, these improvements remain constrained.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Missing answer recall improved when the prompt explicitly informs the model that the correct answer might not be present in the choices, but it was not statistically significant (p = 0.07).

    “Missing answer recall improved when the prompt explicitly informs the model that the correct answer might not be present in the choices, but it was not statistically significant (p = 0.07).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Exceptions to 0% unknown recall were GPT-4o-2024-05-13 (3.7%), Yi 1.5 34B (0.6%), and Meerkat 7B (1.2%).

    “Exceptions were GPT-4o-2024-05-13, which achieved 3.7%, Yi 1.5 34B which scored 0.6%, and Meerkat 7B with 1.2%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Prompts had to explicitly inform the LLM of potential biases and dangers, necessitating an exhaustive and impractical list of all potential pitfalls for real-world applications.

    “Prompts had to explicitly inform the LLM of potential biases and dangers, necessitating an exhaustive—and impractical—list of all potential pitfalls for real-world applications.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Meerkat 7B chose 'None of the above' 295 times.

    “A similar but less pronounced trend was observed with the Meerkat 7B model, which chose “None of the above” 295 times.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Yi 1.5 9B achieved the highest score on missing answer recall despite having the lowest overall accuracy.

    “The Yi 1.5 9B model, which had the lowest overall accuracy, achieved the highest score on this specific metric.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Providing the complete benchmark design instructions did not improve performance compared to baseline except for unknown recall, but underperforms compared to explicit prompts.

    “Interestingly, providing the complete benchmark design instructions did not improve the performance compared to baseline except for unknown recall but underperforms compared to explicit prompts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Models tend to avoid low confidence scores, suggesting a potential bias towards overconfidence.

    “the tendency of models to avoid low confidence scores suggests a potential bias towards overconfidence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Methods such as reflective medical practice and debiasing through feedback can help reduce the number of cognitive errors.

    “methods such as reflective medical practice36, which help clinicians enhance their ability to navigate complex cases37, or debiasing through feedback to identify and correct cognitive biases38 can help in reducing the number of cognitive errors.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [36] Mamede, S. & Schmidt, H. G. The structure of reflective practice in medicine. Med. Educ. 38, 1302–1308 (2004). source
    • cites [37] Mamede, S., Schmidt, H. G. & Penaforte, J. C. Effects of reflective practice on the accuracy of medical diagnoses. Med. Educ. 42, 468–475 (2008). source
    • cites [38] Diagnosing Diagnosis Errors: Lessons from a Multi-institutional Collaborative Project. in Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology) - NCBI Bookshelf. (2024).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The predominance of zero scores severely limits the interpretability and practical significance of the statistical findings for unknown recall.

    “The predominance of zero scores severely limits the interpretability and practical significance of these statistical findings.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    This inability to reliably indicate when they lack sufficient information or knowledge suggests a risk of generating misleading or incorrect information, which could have serious implications in clinical applications.

    “This inability to reliably indicate when they lack sufficient information or knowledge suggests a risk of generating misleading or incorrect information, which could have serious implications in clinical applications.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The manual modifications and audits performed by the authors are subject to human error and interpretation biases, and could have introduced subjective biases affecting the outcomes of the evaluations.

    “the manual modifications and audits we performed, although thorough, are subject to human error and interpretation biases.”
    • Also stated in: § Discussion · hedged
      “The selection and modification of questions, as well as the auditing process, could have introduced subjective biases affecting the outcomes of our evaluations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    The fixed nature of current LLMs and the challenges associated with providing meaningful feedback leave minimal room for improvement.

    “Moreover, their fixed nature and the challenges associated with providing meaningful feedback leave minimal room for improvement.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Metacognition in AI systems can be split into knowledge of cognition and regulation of cognition.

    “Metacognition in AI systems can be split into two categories23: knowledge of cognition and regulation of cognition.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [23] Gonullu, I. & Artar, M. Metacognition in medical education. Educ. Health 27, 225 (2014). source

  • May combine two claims.

    Premisecitation

    Clinicians adeptly employ both pattern recognition (System 1) and deliberate analytical processes (System 2) concurrently, selecting the most appropriate approach based on their expertise.

    “Clinicians adeptly employ these cognitive strategies concurrently, selecting the most appropriate approach based on their expertize32.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [32] Bowen, J. L. Educational strategies to promote clinical diagnostic reasoning. N. Engl. J. Med. 355, 2217–2225 (2006). source

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The MedQA benchmark, even with the paper's modifications, may not fully capture the complexity and variability of real-world clinical scenarios.

    “the MedQA benchmark, even with our modifications, may not fully capture the complexity and variability of real-world clinical scenarios.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Limited access to external tools, along with their imperfections, raises concerns about relying solely on such tools to prevent errors stemming from metacognitive deficits.

    “The limited access to external tools, along with their imperfections raises concerns about relying solely on such tools to prevent errors stemming from metacognitive deficits.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    LLMs' tendency towards overconfidence and inability to recognize knowledge gaps pose potential risks in clinical applications.

    “We find that LLMs’ tendency towards overconfidence and inability to recognize knowledge gaps pose potential risks in clinical applications.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    Additional medical training demonstrated an improvement in accuracy for both models and the ability to detect missing answers for Meerkat 7b, which could be explained by the inclusion of questions with 5 choices and a wider range of questions in the training dataset.

    “Additional medical training demonstrated an improvement in accuracy for both models and the ability to detect missing answers for Meerkat 7b which could be explained by the inclusion of questions with 5 choices and a wider range of questions in the training dataset.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Current LLMs exhibit overconfidence and deficiencies in recognizing their limitations.

    “Current LLMs, despite their capabilities, exhibit overconfidence and deficiencies in recognizing their limitations, making them unlikely to appropriately employ these nuanced strategies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    The controlled nature of multiple-choice questions cannot replicate the nuanced decision-making processes required in clinical practice.

    “the controlled nature of multiple-choice questions cannot replicate the nuanced decision-making processes required in clinical practice.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Clinicians may leverage external resources, such as clinical guidelines or second opinions, to inform their decision-making.

    “Beyond internal processes, clinicians may also leverage external resources, such as clinical guidelines or second opinions, to inform their decision-making34.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [34] Elstein, A. S. Thinking about diagnostic thinking: a 30-year perspective. Adv. Health Sci. Educ. 14, 7–18 (2009). source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Comprehensive models of cognition, such as the transtheoretical model, incorporate external factors, including interactions with team members or databases, which could be implemented for LLMs with Retrieval Augmented Generation.

    “Comprehensive models of cognition, such as the transtheoretical model31 incorporate external factors, including interactions with team members or databases, which could be implemented for LLMs with Retrieval Augmented Generation31.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31] Parsons, A. S. et al. Beyond thinking fast and slow: Implications of a transtheoretical model of clinical reasoning and error on teaching, assessment, and research. Med. Teach. 1–12 https://doi.org/10.1080/0142159x.2024.2359963 (2024). source

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductreasoning

    The systematic approach and open access to the paper's modifications ensure that the findings remain reliable and reproducible.

    “the systematic approach and open access to our modifications ensure that our findings remain reliable and reproducible”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Large Language Models have demonstrated expert-level accuracy on medical board examinations, performing at a level comparable to medical professionals.

    “Recent efforts have included the use of official board examinations to evaluate LLM performance across different medical specialties such as pediatrics3, oncology4, ophthalmology5, radiology6, or plastic surgery7, often demonstrating that these models can perform at a level comparable to medical professionals8.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [8] Katz, U. et al. GPT versus resident physicians — A benchmark based on official board scores. NEJM AI 1, AIdbp2300192 (2024). source

  • May combine two claims.

    Interpretationown data

    Model optimization and specific training datasets also play crucial roles, as indicated by the poor performance of certain models like Yi 1.5-9B despite being relatively recent.

    “However, the notably poor performance of certain models like Yi 1.5-9B, despite being relatively recent, indicates that model optimization and specific training datasets also play crucial roles.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    While external factors might partially compensate for the lack of internal metacognition, this approach presents limitations.

    “While external factors might partially compensate for the lack of internal metacognition, this approach presents limitations.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    The use of Guidance with constrained decoding ensured that models selected only from allowed choices (A/B/C/D/E/F) and provided confidence scores (1/2/3/4/5).

    “We used Python 3.12 and Guidance, a Python library designed to enforce model adherence to specific instructions through constrained decoding45, ensuring the models selected only from the allowed choices (A/B/C/D/E/F) and provided confidence scores (1/2/3/4/5)46.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [45] guidance-ai/guidance: A guidance language for controlling large language models. https://github.com/guidance-ai/guidance (2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [46] Griot, M., Hemptinne, C., Vanderdonckt, J., Yuksel, D. MetaMedQA benchmark code. https://doi.org/10.5281/zenodo.14177940 (2024). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    All models were evaluated with a temperature setting of 0 to ensure reliability and reproducibility of results.

    “All models were evaluated with a temperature setting of 0 to ensure reliability and reproducibility of results56.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [56] Ronanki, K., Cabrero-Daniel, B., Horkoff, J. & Berger, C. Requirements engineering using generative AI: prompts and prompting patterns. In Generative AI for Effective Software Development. (Springer, Cham, 2024). source

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    LLMs have restricted access to external cognitive processes, so internal cognition processes from the DPT must compensate.

    “Considering LLMs have restricted access to external cognitive processes, we argue that internal cognition processes from the DPT must compensate.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: LLMs have restricted access to external cognitive processes.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Because this work focused on a limited set of LLMs available and popular as of June 2024, the findings may not be fully generalizable to future models or those trained with different objectives and datasets.

    “This temporal limitation means the findings may not be fully generalizable to future models or those trained with different objectives and datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The proprietary nature of some models, such as OpenAI's GPT-4o, limits insight into their training data and methodologies, which could influence their performance and the interpretation of results.

    “The proprietary nature of some models, such as OpenAI’s GPT-4o, limits our insight into their training data and methodologies. This constraint could influence their performance and the interpretation of our results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    This gap raises concerns about a form of deceptive expertise, where systems appear knowledgeable but fail to recognize their own limitations.

    “This gap raises concerns about a form of deceptive expertise, where systems appear knowledgeable but fail to recognize their own limitations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductreasoning

    The final metrics (confidence accuracy, missing answer recall, unknown recall) may not capture all aspects of model performance and safety.

    “The final metrics, including confidence accuracy, missing answer recall, and unknown recall, are designed to provide a comprehensive assessment but may not capture all aspects of model performance and safety.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductnone given

    The confidence scoring system implemented (1 to 5) may not fully represent the nuanced levels of certainty a model might have.

    “while we implemented a confidence scoring system to capture models’ confidence levels on a scale from 1 to 5, this may not fully represent the nuanced levels of certainty a model might have.”
    Quote in the paperNot a substantial claim
41 of 41

  • The paper relies on this without support.

    The metacognitive abilities of LLMs remain largely unexplored.

    “However, their metacognitive abilities, crucial for medical decision-making, remain largely unexplored.”
    “we propose investigating a crucial but underexplored area: the assessment of LLMs’ metacognition.”

  • The paper relies on this without support.

    LLMs have restricted access to external cognitive processes.

    “Considering LLMs have restricted access to external cognitive processes, we argue that internal cognition processes from the DPT must compensate.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs may express high confidence in incorrect conclusions, producing an illusion of expertise that expert clinicians can often detect and novices cannot.

    “This vulnerability is amplified by a structural property of LLMs, which may express high confidence in incorrect conclusions, producing an illusion of expertise that expert clinicians can often detect and novices cannot 30 .”

    From AI-induced never-skilling in medical education (2026) · cites this paper as [30]

    The Discussion (page not available in our copy) reports that many models tend toward overconfidence and describes a form of “deceptive expertise,” in which they appear knowledgeable while failing to recognize their limitations. It does not say that expert clinicians can often detect this illusion while novices cannot; that comparison is unsupported. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Griot et al. show that medical LLMs may perform well on benchmark assessments while lacking critical metacognitive reliability, such as recognizing when correct options are absent.

    “Griot et al. show that medical LLMs may perform well on benchmark assessments while lacking critical metacognitive reliability, such as recognizing when correct options are absent [19].”

    From Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence (2026) · cites this paper as [19]

  • Prior studies have noted the presence of malformed and incomplete questions in the MedQA benchmark.

    “Prior studies have noted the presence of malformed and incomplete questions in the MedQA benchmark,22 consistent with our observations during evaluation.”

    From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [22]

  • LLMs have advantages including support for information extraction and synthesis, fact checking, multilingual analysis, and decision-support assistants, but also important limitations in uncertainty handling, self-assessment, and metacognitive reliability in high-stakes settings.

    “While acknowledging various advantages of LLMs (Choi and Ferrara, 2024; Zhao et al., 2024), including their support for information extraction and synthesis, facts checking, multilingual analysis, and decision-support assistants, recent work (Griot et al., 2025) has also highlighted their important limitations in uncertainty handling, self-assessment, and metacognitive reliability in high-stakes”

    From Enhancement of modern defense capabilities by convergence of cognitive neuroscience and artificial intelligence (2026) · cites this paper as Griot et al., 2025

  • MetaMedQA quantifies calibration shortcomings by measuring Confidence Accuracy and Unknown Recall, which gauge a model's ability to recognize when it does not know the answer.

    “Benchmarks, such as MetaMedQA, further quantify these shortcomings by measuring metrics such as Confidence Accu-racy and Unknown Recall, which gauge a model's ability to recognize when it does not know the answer (Griot et al., 2025).”

    From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Griot et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Accuracy improvement is assigned the wrong significance level

    Minor

    The text says accuracy improved at p < 0.0001 when the prompt explicitly warned that questions might be unanswerable. For the role-explicit-unknown prompt, Table 4 instead reports p = 0.013 for accuracy.

  • Missing-answer regression interval is too narrow for the reported correlation

    Minor

    For a simple regression of the 10 models shown in Fig. 4, r = 0.947 and a slope of 1.319 imply an approximate 95% slope interval of 0.95-1.68. The reported interval, 1.136-1.502, is about half as wide.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. American Medical Association. Physician sentiments around the use of AI in heath care: motivations, opportunities, risks, and use cases. https://www.ama-assn.org/system/files/physician-ai-sentiment-report.pdf (2023).

Cited by

Sorted by citations

Related

Authors
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D.
Affiliations
  • UCLouvain (BE)
  • Cliniques Universitaires Saint-Luc (BE)
Corresponding author
Griot, Maxime
ORCID
Venue
Nature Communications
Publisher
Springer Science and Business Media LLC
Published
2025-01-14
Volume
16
Issue
1
Pages
642-642
ISSN
2041-1723 (ISSN-L)
DOI
10.1038/s41467-024-55628-6
PMC
PMC11733150
OpenAlex
W4406371336
Semantic Scholar
1aaa4c92848a921714ee438ea7c2785888f6a49f
Cited by
113
References
49
Citations by year
2026: 73 · 2025: 40
FWCI
24.28
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
  • RePEc: Research Papers in Economics (submittedVersion)
  • Digital Access to Libraries (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Fondation Saint-Luc, Grant 467E Fond Spécial de Recherche of Université catholique de Louvain
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, metacognition, medical reasoning, confidence calibration, clinical decision support, limits of knowledge, medical licensing examinations
MeSH terms
Clinical Decision-Making, Metacognition, Clinical Reasoning, Large Language Models, Decision Making, Humans

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review