The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index; 7 of the 21 sources its claims cite were checked for support.
Large Language Models lack essential metacognition for reliable medical reasoning
Open accessAbstract
Large Language Models have demonstrated expert-level accuracy on medical board examinations, suggesting potential for clinical decision support systems. However, their metacognitive abilities, crucial for medical decision-making, remain largely unexplored. To address this gap, we developed MetaMedQA, a benchmark incorporating confidence scores and metacognitive tasks into multiple-choice medical questions. We evaluated twelve models on dimensions including confidence-based accuracy, missing answer recall, and unknown recall. Despite high accuracy on multiple-choice questions, our study revealed significant metacognitive deficiencies across all tested models. Models consistently failed to recognize their knowledge limitations and provided confident answers even when correct options were absent. In this work, we show that current models exhibit a critical disconnect between perceived and actual capabilities in medical reasoning, posing significant risks in clinical settings. Our findings emphasize the need for more robust evaluation frameworks that incorporate metacognitive abilities, essential for developing reliable Large Language Model enhanced clinical decision support systems. Large Language Models demonstrate expert-level accuracy in medical exams, supporting their potential inclusion in healthcare settings. Here, authors reveal that their metacognitive abilities are underexplored, showing significant gaps in recognizing knowledge limitations, difficulties in modulating their confidence, and challenges in identifying when a problem cannot be answered due to insufficient information.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces MetaMedQA, a modified MedQA-USMLE benchmark with confidence scoring and unanswerable/missing-answer tasks, and evaluates 12 LLMs on medical multiple-choice questions. It finds that despite high accuracy, models largely fail to recognize knowledge limits, rarely use low confidence, and almost never identify unknown or unanswerable questions.
Key findings
- MetaMedQA contains 1,373 questions with six answer choices and one correct choice; it includes 100 fictional Glianorex questions, 55 manually identified malformed questions, and 125 modified questions.
- Overall accuracy tracked model size and recency: GPT-4o-2024-05-13 reached 73.3% (SEM = 1.2%), Qwen2 72B 64.3% (SEM = 1.3%), Qwen2 7B 43.9% (SEM = 1.3%), and Yi 1.5 9B 29.6% (SEM = 1.2%).
- Most models always assigned maximum confidence 5; only GPT-3.5-turbo-0125, GPT-4o-2024-05-13, and Qwen2-72B varied confidence. GPT-4o had high-confidence accuracy 83.2%, medium-confidence accuracy 45.9%, and low-confidence accuracy 16.7%.
- Missing-answer recall was low: GPT-4o scored 46.1% (SEM = 4.7%) versus GPT-3.5-turbo-0125 at 11.3% (SEM = 2.9%). Unknown recall was near zero for most models, with GPT-4o at 3.7%, Meerkat 7B at 1.2%, and Yi 1.5 34B at 0.6%; prompt engineering raised GPT-4o unknown recall to 51.8% with the “role-full” prompt.
Methods
- The authors created MetaMedQA from MedQA-USMLE by adding fictional questions, identifying malformed questions, and modifying selected questions, yielding 1,373 six-choice questions.
- They evaluated 12 proprietary and open-weight models, including GPT-4o-2024-05-13, GPT-3.5-turbo-0125, Mixtral-8x7B, Mistral-7B, Yi-1.5-9B/34B, Llama-3-8B/70B, Qwen2-7B/72B, Meerkat-7B, and Internist-7B, using temperature 0 and constrained decoding via Guidance.
- Main analyses covered overall accuracy, confidence-based accuracy, missing-answer recall, unknown recall, and prompt engineering on GPT-4o; statistical comparisons used one-way ANOVA with Tukey correction, and prompt comparisons used Fisher’s least significant difference test.
Limitations
- The paper states that MedQA, even modified, may not capture real-world clinical complexity; multiple-choice questions cannot replicate nuanced clinical decision-making, and manual modifications/audits may introduce human error or interpretation bias.
- The paper states that confidence scoring from 1 to 5 may not fully represent model certainty, the evaluated models were limited to those available as of June 2024, proprietary models limit insight into training data, and the reliance on Dual Process Theory may not fully represent clinical cognition; the work investigates System 1 thinking exclusively.
- Evident from the text: unknown-recall analysis was severely limited because 9 of 12 models scored 0, making regression/correlation results hard to interpret; confidence analysis was limited to only three models that varied confidence, and prompt engineering was tested only on GPT-4o.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
35 Own result · 22 Own interpretation · 9 No full text of the sources · 6 Own method · 6 No basis given · 4 Supported · 3 Partly supported · 2 Not checked yet · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While these methods allow for the swift assessment of model performance, they are primarily limited to evaluating pattern recognition and information recall2.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
What the sources say
- Partly supportedcites [2] Freiwald, T., Salimi, M., Khaljani, E. & Harendza, S. Pattern recognition as a concept for multiple-choice questions in a national licensing exam. BMC Med. Educ. 14, 232 (2014). sourceExperimental — accuracy not yet measured
The Methods classify questions as pattern-recognition, knowledge-recall, or related types, and the Results report that 42.2% were pattern-recognition questions and 44.3% were knowledge questions. But the study examined four disciplines in one German medical licensing exam, not multiple-choice evaluations generally or assessments of model performance. The Background also says well-constructed MCQs can test application of knowledge, so the claim's broad limitation goes beyond the paper. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“These findings reinforce previous research recommendations on mitigating healthcare data biases in machine learning30, identifying a probable training data bias that predisposes models to provide confident answers in most scenarios, even when a more cautious response is warranted.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Supportedcites [30] Ghassemi, M., Nsoesie, E. O. In medicine, how do we machine learn anything real? Patterns https://doi.org/10.1016/j.patter.2021.100392 (2024). source
Citation 30 is attached to the statement that prior research recommends mitigating healthcare data biases in machine learning. In “What role does machine learning have?” (page unavailable), the source explicitly offers recommendations for algorithm developers, including recognizing bias and examining who is excluded from data. The subsequent inference about models giving confident answers is not the part attributed to this source. (LLM, conf=0.91)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Recent studies on GPT-4V’s performance on medical multiple-choice questions demonstrated that despite the impressive results of models on multiple-choice questions, the rationale behind correct answers is flawed in a significant percentage of cases39.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Supportedcites [39] Jin, Q. et al. Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine. Npj Digit. Med. 7, 1–6 (2024). source
The Abstract reports that GPT-4V achieved 81.6% accuracy on medical multiple-choice questions, yet presented flawed rationales in 35.5% of cases where its final choice was correct. This directly supports the attributed claim. (LLM, conf=0.95)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Another analysis of GPT-4’s errors on the USMLE demonstrated that most errors are either caused by an anchoring bias or incorrect conclusions40.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Partly supportedcites [40] Roy, S. et al. Beyond accuracy: Investigating error types in GPT-4 responses to USMLE questions. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. (2024). sourceExperimental — accuracy not yet measured
The Introduction (p. 2) identifies “sticking with the wrong diagnosis” and “incorrect or vague conclusion” as the most frequent reasoning-error subclasses; the former resembles anchoring. But §5.2 (p. 8) reports them at 26.9% and 22.21%, respectively, and does not establish that most errors fall into either category. The paper also describes the pattern as observed in its annotated subset, not all GPT-4 USMLE errors. (LLM, conf=0.87)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Crucially, the ability to recognize knowledge limitations enables clinicians to dynamically shift between these cognitive strategies33.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Supportedcites [33] Zwaan, L., Hautz, W. E. Bridging the gap between uncertainty, confidence and diagnostic accuracy: calibration is key. BMJ Qual. Saf. 28, 352–355 (2019). source
In “Making decisions while acknowledging uncertainty,” the paper says metacognition helps clinicians recognise their confidence in a decision, slow down when needed, and override intuitive impressions with deliberate reasoning. This supports the paraphrase about recognising limitations and shifting between cognitive strategies. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the reliance on the Dual Process Theory (DPT)42 may not accurately represent the cognition processes involved in clinical decision-making.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Partly supportedcites [42] Bellini-Leite, S. C. Dual Process Theory: Embodied and Predictive; Symbolic and Classical. Front. Psychol. 13, https://doi.org/10.3389/fpsyg.2022.805386 (2022). sourceExperimental — accuracy not yet measured
The Features of Dual Process Theory section says DPT’s features have been criticized as poorly defined, and the Conclusion says the theory needs further development. Those passages support caution about relying on DPT, but the paper does not assess whether it accurately represents cognition in clinical decision-making specifically. (LLM, conf=0.79)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, only three models, with GPT-4o standing out, effectively vary their confidence levels.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“there was no statistically significant correlation (Pearson r = 0.574, p = 0.051).”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Unknown analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“most models exhibit poor performance in recognizing unanswerable questions and managing uncertainty”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4o stood out in this regard, showing a robust capacity to provide higher accuracy when highly confident compared to answers with lower confidence scores.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 24
Every source the paper cites for this claim was checked and supports it.
In the paper
“LLMs also appear to suffer from a similar shortcoming and fail to self-correct when their reasoning is faulty44.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Supportedcites [44] Huang, J. et al. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations. (2024). source
The Conclusion and Discussion (p. 8) states that current LLMs struggle to recognize and rectify their reasoning mistakes without external feedback. That supports the citing sentence’s hedged claim about failure to self-correct faulty reasoning; the paper’s finding concerns unaided self-correction. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed that most models consistently assigned a maximum confidence level of 5, rendering them unsuitable for the confidence analysis.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Impact of confidence
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, the Llama 3 8B model almost never selected this option, while the Mistral 7B and Internist 7B models never did.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The models either never used this answer choice or used it less than 10 times over the 1373 questions.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Unknown analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observed that the model fails to use mid and low confidence appropriately when given additional instructions in the system prompt, but the high confidence accuracy was either similar to or higher than baseline.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Prompt engineering analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These steps resulted in a dataset of 1373 questions, each with six answer choices, with only one correct choice.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Benchmark creation and preprocessing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While we observed some enhancements in metacognitive task performance through prompt engineering with GPT-4o, these improvements remain constrained.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing answer recall improved when the prompt explicitly informs the model that the correct answer might not be present in the choices, but it was not statistically significant (p = 0.07).”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Prompt engineering analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Exceptions were GPT-4o-2024-05-13, which achieved 3.7%, Yi 1.5 34B which scored 0.6%, and Meerkat 7B with 1.2%.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Unknown analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Prompts had to explicitly inform the LLM of potential biases and dangers, necessitating an exhaustive—and impractical—list of all potential pitfalls for real-world applications.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A similar but less pronounced trend was observed with the Meerkat 7B model, which chose “None of the above” 295 times.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Yi 1.5 9B model, which had the lowest overall accuracy, achieved the highest score on this specific metric.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, providing the complete benchmark design instructions did not improve the performance compared to baseline except for unknown recall but underperforms compared to explicit prompts.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Prompt engineering analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the tendency of models to avoid low confidence scores suggests a potential bias towards overconfidence.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“methods such as reflective medical practice36, which help clinicians enhance their ability to navigate complex cases37, or debiasing through feedback to identify and correct cognitive biases38 can help in reducing the number of cognitive errors.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Not checked — no full text of the sourcecites [36] Mamede, S. & Schmidt, H. G. The structure of reflective practice in medicine. Med. Educ. 38, 1302–1308 (2004). source
- Not checked — no full text of the sourcecites [37] Mamede, S., Schmidt, H. G. & Penaforte, J. C. Effects of reflective practice on the accuracy of medical diagnoses. Med. Educ. 42, 468–475 (2008). source
- Not matchedcites [38] Diagnosing Diagnosis Errors: Lessons from a Multi-institutional Collaborative Project. in Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology) - NCBI Bookshelf. (2024).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The predominance of zero scores severely limits the interpretability and practical significance of these statistical findings.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Unknown analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This inability to reliably indicate when they lack sufficient information or knowledge suggests a risk of generating misleading or incorrect information, which could have serious implications in clinical applications.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“the manual modifications and audits we performed, although thorough, are subject to human error and interpretation biases.”
- Also stated in: § Discussion · hedged
“The selection and modification of questions, as well as the auditing process, could have introduced subjective biases affecting the outcomes of our evaluations.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Also stated in: § Discussion · hedged
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, their fixed nature and the challenges associated with providing meaningful feedback leave minimal room for improvement.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Metacognition in AI systems can be split into two categories23: knowledge of cognition and regulation of cognition.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
What the sources say
- Not checked — no full text of the sourcecites [23] Gonullu, I. & Artar, M. Metacognition in medical education. Educ. Health 27, 225 (2014). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Clinicians adeptly employ these cognitive strategies concurrently, selecting the most appropriate approach based on their expertize32.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Not checked — no full text of the sourcecites [32] Bowen, J. L. Educational strategies to promote clinical diagnostic reasoning. N. Engl. J. Med. 355, 2217–2225 (2006). source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the MedQA benchmark, even with our modifications, may not fully capture the complexity and variability of real-world clinical scenarios.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The limited access to external tools, along with their imperfections raises concerns about relying solely on such tools to prevent errors stemming from metacognitive deficits.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that LLMs’ tendency towards overconfidence and inability to recognize knowledge gaps pose potential risks in clinical applications.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additional medical training demonstrated an improvement in accuracy for both models and the ability to detect missing answers for Meerkat 7b which could be explained by the inclusion of questions with 5 choices and a wider range of questions in the training dataset.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Current LLMs, despite their capabilities, exhibit overconfidence and deficiencies in recognizing their limitations, making them unlikely to appropriately employ these nuanced strategies.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the controlled nature of multiple-choice questions cannot replicate the nuanced decision-making processes required in clinical practice.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Beyond internal processes, clinicians may also leverage external resources, such as clinical guidelines or second opinions, to inform their decision-making34.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Not checked — no full text of the sourcecites [34] Elstein, A. S. Thinking about diagnostic thinking: a 30-year perspective. Adv. Health Sci. Educ. 14, 7–18 (2009). source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Comprehensive models of cognition, such as the transtheoretical model31 incorporate external factors, including interactions with team members or databases, which could be implemented for LLMs with Retrieval Augmented Generation31.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
What the sources say
- Not checked — no full text of the sourcecites [31] Parsons, A. S. et al. Beyond thinking fast and slow: Implications of a transtheoretical model of clinical reasoning and error on teaching, assessment, and research. Med. Teach. 1–12 https://doi.org/10.1080/0142159x.2024.2359963 (2024). source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“the systematic approach and open access to our modifications ensure that our findings remain reliable and reproducible”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Recent efforts have included the use of official board examinations to evaluate LLM performance across different medical specialties such as pediatrics3, oncology4, ophthalmology5, radiology6, or plastic surgery7, often demonstrating that these models can perform at a level comparable to medical professionals8.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Introduction
What the sources say
- Not checked — no full text of the sourcecites [8] Katz, U. et al. GPT versus resident physicians — A benchmark based on official board scores. NEJM AI 1, AIdbp2300192 (2024). source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the notably poor performance of certain models like Yi 1.5-9B, despite being relatively recent, indicates that model optimization and specific training datasets also play crucial roles.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgeddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While external factors might partially compensate for the lack of internal metacognition, this approach presents limitations.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“We used Python 3.12 and Guidance, a Python library designed to enforce model adherence to specific instructions through constrained decoding45, ensuring the models selected only from the allowed choices (A/B/C/D/E/F) and provided confidence scores (1/2/3/4/5)46.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Methods > Benchmark procedure
What the sources say
- Not matchedcites [45] guidance-ai/guidance: A guidance language for controlling large language models. https://github.com/guidance-ai/guidance (2023).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [46] Griot, M., Hemptinne, C., Vanderdonckt, J., Yuksel, D. MetaMedQA benchmark code. https://doi.org/10.5281/zenodo.14177940 (2024). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
In the paper
“All models were evaluated with a temperature setting of 0 to ensure reliability and reproducibility of results56.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Methods > Benchmark procedure
What the sources say
- Not checked — no full text of the sourcecites [56] Ronanki, K., Cabrero-Daniel, B., Horkoff, J. & Berger, C. Requirements engineering using generative AI: prompts and prompting patterns. In Generative AI for Effective Software Development. (Springer, Cham, 2024). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Considering LLMs have restricted access to external cognitive processes, we argue that internal cognition processes from the DPT must compensate.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: LLMs have restricted access to external cognitive processes.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This temporal limitation means the findings may not be fully generalizable to future models or those trained with different objectives and datasets.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The proprietary nature of some models, such as OpenAI’s GPT-4o, limits our insight into their training data and methodologies. This constraint could influence their performance and the interpretation of our results.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This gap raises concerns about a form of deceptive expertise, where systems appear knowledgeable but fail to recognize their own limitations.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The final metrics, including confidence accuracy, missing answer recall, and unknown recall, are designed to provide a comprehensive assessment but may not capture all aspects of model performance and safety.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“while we implemented a confidence scoring system to capture models’ confidence levels on a scale from 1 to 5, this may not fully represent the nuanced levels of certainty a model might have.”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The metacognitive abilities of LLMs remain largely unexplored.
“However, their metacognitive abilities, crucial for medical decision-making, remain largely unexplored.”
“we propose investigating a crucial but underexplored area: the assessment of LLMs’ metacognition.”
The paper relies on this without support.
LLMs have restricted access to external cognitive processes.
“Considering LLMs have restricted access to external cognitive processes, we argue that internal cognition processes from the DPT must compensate.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This vulnerability is amplified by a structural property of LLMs, which may express high confidence in incorrect conclusions, producing an illusion of expertise that expert clinicians can often detect and novices cannot 30 .”
From AI-induced never-skilling in medical education (2026) · cites this paper as [30]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Griot et al. show that medical LLMs may perform well on benchmark assessments while lacking critical metacognitive reliability, such as recognizing when correct options are absent [19].”
From Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence (2026) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“Prior studies have noted the presence of malformed and incomplete questions in the MedQA benchmark,22 consistent with our observations during evaluation.”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“While acknowledging various advantages of LLMs (Choi and Ferrara, 2024; Zhao et al., 2024), including their support for information extraction and synthesis, facts checking, multilingual analysis, and decision-support assistants, recent work (Griot et al., 2025) has also highlighted their important limitations in uncertainty handling, self-assessment, and metacognitive reliability in high-stakes”
From Enhancement of modern defense capabilities by convergence of cognitive neuroscience and artificial intelligence (2026) · cites this paper as Griot et al., 2025
The cited source says what the claim attributes to it.
In the citing paper
“Benchmarks, such as MetaMedQA, further quantify these shortcomings by measuring metrics such as Confidence Accu-racy and Unknown Recall, which gauge a model's ability to recognize when it does not know the answer (Griot et al., 2025).”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Griot et al., 2025
In the citing paper
“a recent study revealed a substantial discrepancy between LLMs’ miscalibrated overconfidence in their outputs and their actual accuracy, underscoring the risks of overreliance on LLMs in the medical domain25.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [25]
In the citing paper
“These results align with recent findings suggesting current LLMs lack metacognition, demonstrating a mismatch between low uncertainty (or overconfidence) and their actual capabilities in medical reasoning tasks25.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [25]
In the citing paper
“Compounding these shortcomings are recently demonstrated LLM deficiencies in metacognition—specifically, the inability to recognize their own limitations—and overconfidence25.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [25]
In the citing paper
“Whereas conventional QA benchmarks are established at assessing human clinical reasoning, their reliability at evaluating LLM reasoning remains unclear25.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Prompt engineering analysis
“A significant improvement in accuracy, high confidence accuracy, and unknown recall appeared (p < 0.0001) once the prompt explicitly informs the model that it may not be able to answer some questions”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Prompt engineering analysis
“Accuracy: role-explicit-unknown, p = 0.013”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
“Pearson r = 0.947, p < 0.0001”
§ Large Language Models lack essential metacognition for reliable medical reasoning > Results > Missing answer analysis
“The regression yielded a statistically significant positive slope of 1.319 (95% CI: 1.136 - 1.502, p < 0.0001) as shown in Fig. 4.”
Citations
Cites
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Artificial intelligence in healthcare: transforming the practice of medicine
Bajwa, J., Munir, U., Nori, A., & Williams, B. (2021) · Future Healthcare Journal · doi:10.7861/fhj.2021-0095 · Cited by 1,856 · cited 1×
“AI offers immensepotentialtoaddresstheshortageofhealthcareworkers 16 and promises to reduce clerical work 17,18 , and has already demonstrated uses in precision diagnostics and therapeutics 19 .”
The Importance of Cognitive Errors in Diagnosis and Strategies to Minimize Them
Croskerry, P. (2003) · Academic Medicine · doi:10.1097/00001888-200308000-00003 · Cited by 1,675 · cited 1×
“Cognitive errors are an important source of diagnostic error 35 and methods such as re fl ective medical practice 36 , which help clinicians enhance their ability to navigate complex cases 37 , or debiasing through feedback to identify and correctcognitivebiases 38…”
Educational Strategies to Promote Clinical Diagnostic Reasoning
Bowen, J. L. (2006) · New England Journal of Medicine · doi:10.1056/nejmra054782 · Cited by 931 · cited 1×
“Clinicians adeptly employ these cognitive strategies concurrently, selecting the most appropriate approach based on their expertize 32 .”
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 712 · cited 1×
“LLMs also appear to suffer from a similar shortcoming and fail to self-correct when their reasoning is faulty 44 .”
Norman, G. R., Monteiro, S. D., Sherbino, J., Ilgen, J. S., Schmidt, H. G., & Mamede, S. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001421 · Cited by 594 · cited 1×
“While additional experiments involving System 2 should be conducted to improve our understanding of LLM cognition, it ’ s important to note that studies have shown that switching to System 2 may not always reduce reasoning errors in humans 43 .”
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Research works printed but not linked: 1
- American Medical Association. Physician sentiments around the use of AI in heath care: motivations, opportunities, risks, and use cases. https://www.ama-assn.org/system/files/physician-ai-sentiment-report.pdf (2023).
Cited by
Sorted by citations
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78
“These results align with recent findings highlighting a mismatch between LLM uncertainty and their actual capabilities in medical reasoning tasks[25].”
InfluentialEvaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 97
MedAgentBench: a virtual EHR environment to benchmark medical LLM agents
Jiang, Y., Black, K. C., Geng, G., Park, D., Zou, J., Ng, A. Y., & Chen, J. H. (2025) · NEJM AI · doi:10.1056/aidbp2500144 · Cited by 66
Take caution in using LLMs as human surrogates
Gao, Y., Lee, D., Burtch, G., & Fazelpour, S. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2501660122 · Cited by 52
A large language model improves clinicians’ diagnostic performance in complex critical illness cases
Wu, X. C., Huang, Y., & He, Q. (2025) · Critical Care · doi:10.1186/s13054-025-05468-7 · Cited by 51
“LLMs may confidently produce misinformation or exhibit cognitive biases, underscoring the risks of unsupervised reliance on these tools [11, 26, 27].”
Large Language Model Performance and Clinical Reasoning Tasks
Rao, A. S., Esmail, K. P., Lee, R. S., Jiang, S., Carlo, B. A., Gill, J., Khanna, P., Kalmowitz, E., Montagnese, B., Heydari, K., et al. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.4003 · Cited by 66
Workum, J. D., Volkers, B. W. S., van de Sande, D., Arora, S., Goeijenbier, M., Gommers, D. A. M. P. J., & van Genderen, M. E. (2025) · Critical Care · doi:10.1186/s13054-025-05302-0 · Cited by 40
AI-induced never-skilling in medical education
Ke, Y., Jin, L., Ong, J. C. L., Thirunavukarasu, A. J., Car, J., Cheung, C. Y., Tham, Y. C., Ting, D. S. W., Ong, M. E. H., Compton, S., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04438-y · Cited by 71
“This vulnerability is amplified by a structural property of LLMs, which may express high confidence in incorrect conclusions, producing an illusion of expertise that expert clinicians can often detect and novices cannot 30 .”
Mavrych, V., Yousef, E. M., Yaqinuddin, A., & Bolgova, O. (2025) · Medical Education Online · doi:10.1080/10872981.2025.2534065 · Cited by 38
“Assessing LLMs’ performance in this field provides insight into their understanding of fundamental biological concepts and their ability to apply these concepts in identifying tissue structures and pathological changes [16].”
Large language models for clinical decision support in gastroenterology and hepatology
Wiest, I. C., Bhat, M., Clusmann, J., Schneider, C. V., Jiang, X., & Kather, J. N. (2025) · Nature Reviews Gastroenterology & Hepatology · doi:10.1038/s41575-025-01108-1 · Cited by 34
Uncertainty-aware large language models for explainable disease diagnosis
Zhou, S., Wang, J., Xu, Z., Wang, S., Brauer, D., Welton, L., Cogan, J., Chung, Y.-H., Tian, L., Zhan, Z., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02071-6 · Cited by 27
“First, LLM con fi dence often misaligns with factual accuracy 29 , as LLMs may exhibit overcon fi dence in erroneous diagnoses due to fl awed internal knowledge 30 .”
Implementation of large language models in electronic health records
Griot, M., Vanderdonckt, J., & Yuksel, D. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0001141 · Cited by 23
Self-citationAutomating expert-level medical reasoning evaluation of large language models
Zhou, S., Xie, W., Li, J., Zhan, Z., Song, M., Yang, H., Espinoza, C., Welton, L., Mai, X., Jin, Y., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02208-7 · Cited by 30
“This is because the credibility of such arti fi cial intelligence (AI)-generated rationales is uncertain 9,39,40 , and the alignment with human expert judgment remains unclear 41 .”
Metacognitive sensitivity: The key to calibrating trust and optimal decision making with AI
Lee, D., Pruitt, J., Zhou, T., Du, J., & Odegaard, B. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf133 · Cited by 33
“This difference is attributed to LLMs’ reliance on training data for confidence ratings, which are not adjusted based on task experience (109).”
InfluentialMetacognition and Uncertainty Communication in Humans and Large Language Models
Steyvers, M., & Peters, M. A. K. (2026) · Current Directions in Psychological Science · doi:10.1177/09637214251391158 · Cited by 18
“As LLMs are increasingly integrated into many highly different tasks and reasoning domains, attending to their domain-specific versus domain-general meta-cognitive capacities will become increasingly urgent (for LLMs’ metacognitive failures in medical reasoning, see, e.g., Griot et al., 2025).”
Generative AI and the augmentation of information practices in knowledge work
Jarrahi, M. H., Li, L., Robinson, A. P., & Meng, S. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2551570 · Cited by 33
A longitudinal analysis of declining medical safety messaging in generative AI models
Sharma, S., Alaa, A. M., & Daneshjou, R. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01943-1 · Cited by 21
“…encode more clinical knowledge and become more integrated into everyday health-seeking behavior, whether for understanding symptoms, interpreting diagnostic tests, or guiding treatment decisions, users may increasingly mistake fl uent, authoritative outputs for clinician-approved advice 14,15 .”
Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
Griot, M., Vanderdonckt, J., Yuksel, D., & Hemptinne, C. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.266 · Cited by 13
Self-citationFrom Evidence-based Endodontics to Generative AI: A Comparative Study of 11 Large Language Models
de Araújo, L. P., Moreno, L. B., de Araújo, B. C. C., et al. (2026) · Journal of Endodontics · doi:10.1016/j.joen.2026.01.009 · Cited by 10
Language models for drug–drug interactions: current applications, pitfalls, and future directions
Meslamani, A. Z. A., & Hajal, A. A. (2025) · Expert Opinion on Drug Metabolism & Toxicology · doi:10.1080/17425255.2025.2551724 · Cited by 13
“First, tune alert thresholds and surface calibrated confidence scores so only high-probability signals interrupt workflow; benchmarks such as MetaMedQA show that attaching well-calibrated probabilities improves clinical safety judgments [110].”
Fast, slow, and metacognitive thinking in AI
Ganapini, M. B., Campbell, M. S., Fabiano, F., Lior Horesh, C., Lenchner, J., Loreggia, A., Mattei, N., Rossi, F., Srivastava, B., & Venable, K. B. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00027-5 · Cited by 10
“However,itisgenerallyacknowledged thatAIstilllacksmanycapabilitieswhichwouldnaturallybeincludedina notionof(human)intelligence,suchasgeneralizability,adaptability, robustness,explainability,causalanalysis,abstraction,commonsenserea-soning,metacognition,andethicaljudgement 2,3 .”
Metacognitive mechanisms of color communication
Griber, Y. A. (2025) · Journal of Modern Foreign Psychology · doi:10.17759/jmfp.2025140302 · Cited by 6
Lightweight Deep Learning Approaches on Edge Devices for Fetal Movement Monitoring
Rattanasak, A., Jumphoo, T., Kokkhunthod, K., et al. (2025) · Biosensors · doi:10.3390/bios15100662 · Cited by 5
“Although such model compression is necessary to meet the resource constraints of embedded deployment, the resulting drop in classification performance may limit their suitability for clinical applications, where high reliability and diagnostic confidence are essential [41].”
Large Language Models in Cardiovascular Prevention: A Narrative Review and Governance Framework
Santos, J. F., & Dores, H. (2026) · Diagnostics · doi:10.3390/diagnostics16030390 · Cited by 8
“Evidence suggests that when presented with multimorbid scenarios, LLMs may default to aggressive, standard-of-care recommendations that fail to account for competing risks, frailty, or pill burden [57,59,60].”
ChatGPT-5 versus other mainstream large language models in core diabetic retinopathy patient queries
Cang, X., MengXia, N., Song, C., Zhao, J., Guo, Y., Zou, Y., Zhang, Z., & Jiang, L. (2026) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2026.1754221 · Cited by 6
“1, Wenxinyiyan 4.5 Turbo, and Doubao offered competitive professional coverage, indicating promising potential for Chinese LLMs in medical corpora and localized knowledge integration, but with room for improvement in patient-oriented accessibility and stability control (Griot et al., 2025).”
InfluentialJiang, L., Zhu, Y., Song, C., Hu, X., Fan, X., Yang, W., & Zhao, Y. (2026) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2026.1785731 · Cited by 6
MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM AgentsPreprint
Jiang, Y., Black, K. C., Park, D., Zou, J., Ng, A. Y., & Chen, J. H. (2025) · Qeios · doi:10.32388/vn3yh7 · Cited by 10
Semi-supervised graph convolutional community detection empowered by large language models
Zhang, S., Huang, X., Ji, S., Zhang, S., & Zhang, S. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114236 · Cited by 5
Mokhtari, K., & Ghimire, N. (2025) · Education and human development · doi:10.5772/intechopen.1012695 · Cited by 4
Paniza, A. C. D. J., & de Moraes, F. Y. (2025) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djaf358 · Cited by 5
Competing Biases underlie Overconfidence and Underconfidence in LLMs
Kumaran, D., Fleming, S. M., Markeeva, L., Heyward, J., Banino, A., Mathur, M., Pascanu, R., Osindero, S., Martino, B. D., Veličković, P., et al. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01217-9 · Cited by 29
Comparing the Performances of a 54-Year-Old Computer-Based Consultation to ChatGPT-4o
Verdi, E. B., & Akbilgiç, O. (2025) · Applied Clinical Informatics · doi:10.1055/a-2628-8408 · Cited by 4
Evaluation of DeepSeek-R1 for Ophthalmic Diagnosis and Reasoning: A Comparison with OpenAI o1 and o3
Ming, S., Yao, X., Guo, Q., Chen, D., Guo, X., Xie, K., & Lei, B. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02264-2 · Cited by 4
“Furthermore, conventional LLMs were shown to lack essential meta-cognitive faculties crucial for reliable medical reasoning—revealing a critical disconnect between perceived and actual capabilities that posed significant risks in clinical settings [ 8].”
Tini, P., Novi, F., Donnini, F., Perrella, A., Bagnacci, G., Mazzei, M. A., & Minniti, G. C. (2025) · Journal of Neuro-Oncology · doi:10.1007/s11060-025-05254-z · Cited by 4
Zero-shot learning for clinical phenotyping: Comparing LLMs and rule-based methods
Neves, B. D., Moreira, J.-M., Gonçalves, S., Cerejo, J., da Silva, N. A., Leite, F., & Silva, M. J. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110181 · Cited by 4
Evaluating Large Reasoning Model Performance on Complex Medical Scenarios In The MMLU-Pro BenchmarkPreprint
Hoyt, R. E., Knight, D. R. T., Haider, M., & Bajwa, M. (2025) · medRxiv · doi:10.1101/2025.04.07.25325385 · Cited by 3
“[8 ] Massive Multitask Language Understanding (MMLU) is another large question-and-answer dataset developed by a team of researchers in 2020.”
Jagarapu, J., Babata, K., Chamarthi, S., & Hoyt, R. (2025) · medRxiv · doi:10.64898/2025.11.29.25341091 · Cited by 3
Hills, T. T. (2026) · AI · doi:10.3390/ai7010033 · Cited by 2
Paik, J. E., Choung, H., & Yang, Q. (2026) · Health Communication · doi:10.1080/10410236.2026.2620497 · Cited by 4
Naderi, N., Safavi‐Naini, S. A. A., Savage, T., Khalafi, M. A., Lewis, P. R., Atf, Z., Nadkarni, G. N., & Soroush, A. (2026) · npj gut and liver. · doi:10.1038/s44355-026-00053-3 · Cited by 13
“Although previous studies have found evidence of poor calibration in general medical tasks 12,13 , it remains unclear whether this limitation is con fi ned to certain systems or represents a universal property of current LLMs.”
Wang, J., Zheng, C., Tang, Z., & Xie, X. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131704 · Cited by 2
Advancing medical AI through benchmarking and competition for specialty triage
Ding, C., Bian, M., Yuan, M., Jiang, L., Luo, K., Chen, P., JIANG, Y., & Xu, J. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02433-8 · Cited by 2
“However, challenges such as diagnostic reliability, integration with existing infrastructures, and clin-icianacceptancemustbeaddressedtofullyrealizethepotentialofAI-driven triage in improving healthcare accessibility and patient outcomes 16,17 .”
Blocking the Reflection: Milestones and Hurdles for Digital Twins in Mental Health
Verhees, F. G., Wiest, I. C., Kather, J. N., Kambeitz, J., & Mikoláš, P. (2026) · Pharmacopsychiatry · doi:10.1055/a-2816-2869 · Cited by 3
“The generative models themselves that could form the core of an 94 turn out to be of little use for these systems 95 , 96 and point to them being restricted in truly modelling a human counterpart as DT.”
Tutorial: guidance on the use of large language models for medical research
Jin, Q., Wan, N., Leaman, R., Tian, S., Wang, Z., Yang, Y., Wang, Z., Xiong, G., Lai, P.-T., Zhu, Q. L., et al. (2026) · Nature Protocols · doi:10.1038/s41596-026-01408-z · Cited by 15
Safety and security of large language models in healthcare
Clusmann, J., Freyer, O., Ostermann, M., Ferber, D., Laleh, N. G., Hilgers, L., Kolbinger, F. R., Schneider, C. V., Downing, A., Wekenborg, M., et al. (2026) · Nature · doi:10.1038/s41586-026-10687-1 · Cited by 5
Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making
Chappidi, S., Singh, J. P., & Krauze, A. V. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791428 · Cited by 5
What Large Language Models Know About Plant Molecular BiologyPreprint
Burda, M. F., Ferrero, L., Gaggion, N., Fonouni‐Farde, C., Crespi, M. D., Ariel, F., & Ferrante, E. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.31.672925 · Cited by 3
“Clinical workflows further demonstrate that, without careful oversight, LLMs can present over-confident hallucinations—such as fabricated drug interactions—or omit critical diagnostic details, reinforcing the necessity of human-in-the-loop validation 15 .”
Structured reasoning failures compromise LLM interpretation of clinical oncology notes
Kenaston, M. W., Ayub, U., Parmar, M., Kumar, P., Anjum, M. U., Naqvi, S. A. A., Chaudhuri, A. A., Zakharia, Y., Heath, E. I., Bekaii-Saab, T. S., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02951-5 · Cited by 2
Sun, X., Hong, Q., Zhang, M., Li, Y., Chen, T., Huang, Z., Liang, G., Tang, W., Xu, S., Ni, X., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102547 · Cited by 3
“Prior studies have noted the presence of malformed and incomplete questions in the MedQA benchmark, 22 consistent with our observations during evaluation.”
Cosic, K., Kuljanac, A., & Palermo, S. (2026) · Frontiers in psychology · doi:10.3389/fpsyg.2026.1821242 · Cited by 3
Jiang, Z., Peng, Q., Li, L., & Xiang, S. (2025) · MedComm – Future Medicine · doi:10.1002/mef2.70042 · Cited by 3
Monteith, S., Glenn, T., Geddes, J. R., Whybrow, P. C., Achtyes, E. D., Bauer, R., & Bauer, M. (2026) · Pharmacopsychiatry · doi:10.1055/a-2743-2564 · Cited by 1
Yuan, Q., Chen, J., & Shi, Z. (2026) · Displays · doi:10.1016/j.displa.2026.103359 · Cited by 1
Pan, Y., Duan, C., Du, J., Zhang, J., Du, K., Zhang, C., Liu, Z., Zhang, W., Wang, B., Ren, Y., et al. (2026) · Annals of Surgical Oncology · doi:10.1245/s10434-026-19176-1 · Cited by 2
Promises and challenges of applying large language models in the healthcare domain
Wang, Q., Gong, Z., Lai, Z., Bu, L., Dahlweid, F.-M., & Sun, H. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1772274 · Cited by 4
“Furthermore, current models remain constrained by hallucinations (37, 53, 57).”
Metacognition of ChatGPT in confidence judgements
YOSHIZAWA, S., Onzo, A., Nozawa, S., Takano, T., Ishikawa, T., & Mogi, K. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1694192 · Cited by 1
“Nevertheless, previous studies employing a range of LLMs under diverse experimental conditions have also reported that LLMs tend to be overconfident (Cash et al., 2025; Griot et al., 2025), suggesting that our finding with GPT-4 is consistent with this broader trend.”
InfluentialHoglund, Z. T., Wu, A. Q., Kathawate, V., Sollenberger, C. H., Englander, R., Rani, N., Saadoun, J., Massaad, E., Dagli, M. M., Malhotra, N., et al. (2026) · World Neurosurgery · doi:10.1016/j.wneu.2026.125023 · Cited by 1
Marini, N., Liang, Z., Rajaraman, S., Xue, Z., & Antani, S. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105055 · Cited by 1
Song, J., Feng, J., Zhang, Y., et al. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001474 · Cited by 1
Chaabene, N. E. H. B., Kahloul, L., Selmi, Z., et al. (2026) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-032-23311-0_30 · Cited by 1
How metacognitive architectures remember their own thoughts: a systematic scoping review
Nolte, R., Pomarlan, M., Janssen, A., Beßler, D., Javanmardi, K., Jongebloed, S., Porzel, R., Bateman, J. F., Beetz, M., & Malaka, R. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11669-5 · Cited by 5
Navigating Ethical Challenges of AI Chatbots for Sex Therapy
Fu, M., & Chan, W. (2025) · Oxford University Press eBooks · doi:10.1093/9780198945215.003.0075 · Cited by 1
Human versus artificial social cognition and metacognition: the normative difference
Tison, R., & Zawidzki, T. W. (2025) · AI and Ethics · doi:10.1007/s43681-025-00774-w · Cited by 1
“Improving the metacognitive abilities of LLMs is also a key goal of recent AI research (Toy et al. 2024 ; Griot et al. 2025 ).”
A hybrid deployment model for generative artificial intelligence in hospitals
Griot, M., Hemptinne, C., Vanderdonckt, J. M., & Yüksel, D. (2025) · Machine Learning Health · doi:10.1088/3049-477x/addb51 · Cited by 1
Self-citationChen, Y., Liu, J., Li, Z., Zhang, J., Wang, Y., Guo, L., Li, Y., Hu, B., Zhu, S., Bao, J., et al. (2025) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.70072
“…(e.g., aCCI and biliary malignancy) from mediators (e.g., preoperative hospital stay and prophylactic antibiotic use), thereby avoiding common pitfalls of data-driven methods, such as the “table 2 fallacy” introduced by stepwise regression or the “black box” nature of machine learning [29, 30].”
Wang, B., Schaefer, E., Aguirre, N. G., Huang, J., Li, X., Kolamuri, S. R., Ying, L., Li, X., Chao, G., Ang, S. S., et al. (2025) · medRxiv · doi:10.64898/2025.12.04.25341678
Multimorbidity measurement strategies for predicting hospital visits
Neves, B. D., Moreira, J. M., Gonçalves, S., Cerejo, J., Mota, I., Silva, N. A., Leite, F., & Silva, M. J. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13817-z
Benchmarking large language models for cardiovascular risk stratification using clinical vignettesPreprint
Santos, J. F., Duarte, R. D. B., Mota, I., Santos, R. C., Moreira, J.-M., Campos, J., Silva, N. A., Neves, B. D., Ladeiras‐Lopes, R., Leite, F., et al. (2025) · Research Square · doi:10.21203/rs.3.rs-8307079/v1
Zheng, T., Liu, J., Feng, S., & Jiang, Z. (2026) · Scientific Reports · doi:10.1038/s41598-026-36627-7
Li, P., Chen, Y., Li, Z., Liu, J., Zhang, J., 王玉柱, Guo, L., Li, Y., Hu, B., Zhu, S., et al. (2026) · Antimicrobial Resistance and Infection Control · doi:10.1186/s13756-026-01707-w · Cited by 1
“…we distinguished seven true confounders from mediator of antibiotic prophylaxis, thereby avoiding over-adjustment bias—a critical limitation of data-driven approaches like stepwise regression that risk “Table 2 Fallacy” [27] or the “black box” nature of machine learning in variable selection [28].”
Hoyt, R. E., & Bajwa, M. (2026) · Applied Clinical Informatics · doi:10.1055/a-2807-4256 · Cited by 2
Large Language Models Default to Population-Level Reasoning in Counterfactual Clinical Prediction
Rago, A. R. P., Weinreb, S. J., Ezekian, J. E., et al. (2026) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2026.03.016
Yang, J., Yang, S., Shao, Z., Chen, T., Shen, H., Zhou, P., Xia, B., Lei, X., Wang, L., Xue, D., et al. (2026) · Burns & Trauma · doi:10.1093/burnst/tkag026
“Collectively, these functions address the ECCM requirements for timely decision-making and comprehensive patient assessment [ 58].”
Evaluation of ChatGPT vs. DeepSeek from a Privacy Perspective
Alissa, K. A., & Min‐Allah, N. (2026) · Electronics · doi:10.3390/electronics15081644
Ganuthula, V. R. R., & Kumari, N. (2026) · Human Behavior and Emerging Technologies · doi:10.1155/hbe2/6725387
What is diagnosis in the age of artificial intelligence
Krishnan, L., & Cassel, C. K. (2026) · Diagnosis · doi:10.1515/dx-2025-0166
“Recent studies revealing concerning de fi ciencies in LLM meta-reasoning amplify this worry [14].”
Can Large Language Models Handle Numeric Constraints? A Comprehensive Study and Solutions
Wang, H., Binru, Z., Xu, Z., Zhou, H., Yang, L., Xu, W., Zhu, J., & Wang, H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7078-2_16
Modeling the joint impact of human and AI metacognitive sensitivity on human–AI collaboration
Li, Z., & Steyvers, M. (2026) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2026.102988
“…Wang, 2023) and metacognitive sensitivity (Stangel et al., 2025; Zhang et al., 2025) in large language models (Ganapini et al., 2025; Eliot, 2024; Griot et al., 2025; Steyvers et al., 2025; Wei et al., 2024), though whether such improvements translate to better human–AI collaboration has yet to…”
How can large language models answer questions about a device that does not exist?Preprint
Hedberg, A. (2026) · Research Square · doi:10.21203/rs.3.rs-9509092/v1
Do AI know what they know? Exploring metacognition in LLMs
Iqbal, S. (2026) · Intelligent Data Analysis · doi:10.1177/1088467x261436903 · Cited by 1
The Impact of Generative AI Interaction on Human Metacognition and Critical Thinking
Galib, A. A., Uddowla, M. A., Islam, M. A., Abbas, J., Bashir, A., & Seyam, R. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491521
Core Techniques for Explaining LLMs in Healthcare
Farhadi, A., & Siasifar, R. (2026) · Springer eBooks · doi:10.1007/978-3-032-15088-2_4
Application of Large Models in the Diagnosis of Lung Diseases
Jing, S., Liu, S., Chen, X., & Wang, C. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-8978-4_35
Kundurti, N., & Schweizer, J. (2026) · doi:10.2196/preprints.101964
A Robust Rag-Based Pipeline for Automated Medical Conversation Transcription and Validation
Balaji, A., Finch, J. D., & Breithaupt, A. (2025) · IEEE MIT Undergraduate Research Technology Conference (URTC) · doi:10.1109/urtc68753.2025.11533007
“Ensuring the trustworthiness, factual accuracy, and clinical relevance of AI-generated content is vital, and general-purpose LLMs often lack the inherent metacognition required for reliable medical reasoning [11].”
Can LLMs Acquire Human-Like World Models?
Teoharie, C. (2026) · Philosophies · doi:10.3390/philosophies11030094
“The results of the study conducted by Griot et al. suggest that the apparent competence of LLMs is limited to pattern recognition rather than true, introspective understanding of tasks—indicating a gap between task performance and deeper comprehension [33].”
Kara, S., & Karcıoğlu, A. A. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12279-6
Metacognition and consciousness
Peters, M. A. K. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-29258-3.00052-5
Clinical reasoning in large language models under ecologically valid conditionsPreprint
Hedberg, A. (2026) · Research Square · doi:10.21203/rs.3.rs-9708115/v1
The AI Education Void in Internal Medicine Clerkships: From Awareness to Action
Burk‐Rafel, J., Abdulnour, R.-E. E., & Boscardin, C. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-026-10595-0
Embracing Digital as a Paradigm Shift in Medical Affairs
Lalwani, A., Zdon, J., Addicks, M., Marzouki, Y., Seo, D., & Chisholm, O. (2026) · Pharmaceutical Medicine · doi:10.1007/s40290-026-00623-z
“The future use of AI in MA should therefore remain anchored in human judgement, validated source material, and clear accountability for outputs used in scientific exchange [30–32, 40, 41].”
Gómez-Gutiérrez, O. A., Imbrognio, F., la Torre, O. A. D., Vegas, D. H., & Fabiani, M. A. (2026) · Annals of Vascular Surgery · doi:10.1016/j.avsg.2026.07.004
Qin, X., Song, P., Yan, Z., Qian, H., Jiang, L., & Wang, C. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1874243 · Cited by 1
“Existing studies have primarily focused on the performance of LLMs in medical examination answering, clinical question answering, case discussion, patient education, and academic writing assistance (7–11).”
Can AI assist in reducing diagnostic error? A narrative review
Scott, I. A. (2026) · Diagnosis · doi:10.1515/dx-2026-0021
“First, LLMs can hallucinate [100], con fi dently generate erroneous outputs [101, 102], partly due to inaccurate and potentially biased training data [103], and lack meta-cognition (i.e. awareness of their own level of uncertainty) [104].”
Ferguson, J. C., & Tekely, M. (2026) · Legal Medicine · doi:10.1016/j.legalmed.2026.102936
Unified, Comprehensive Metacognition within the Common Model of Cognition
Laird, J. E., Lebiere, C., Rosenbloom, P. S., & Stocco, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33195-3_1
Probing conflict resolution between contextual and parametric knowledge in medical language models
Yu, Z., Ding, Y., & Liu, X. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133720
Shao, M., Shao, H., Zhang, H., & Zhang, H. (2026) · doi:10.2196/preprints.108814
Metadata
- Authors
- Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D.
- Affiliations
- UCLouvain (BE)
- Cliniques Universitaires Saint-Luc (BE)
- Corresponding author
- Griot, Maxime
- Venue
- Nature Communications
- Published
- 2025-01-14
- Volume
- 16
- Issue
- 1
- Pages
- 642-642
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC11733150
- OpenAlex
- W4406371336
- Semantic Scholar
- 1aaa4c92848a921714ee438ea7c2785888f6a49f
- Cited by
- 172
- References
- 49
- Citations by year
- 2026: 123 · 2025: 48
- FWCI
- 32.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Nature Communications (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Digital Access to Libraries (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Fondation Saint-Luc, Grant 467E Fond Spécial de Recherche of Université catholique de Louvain
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, metacognition, medical reasoning, confidence calibration, clinical decision support, limits of knowledge, medical licensing examinations
- MeSH terms
- Clinical Decision-Making, Metacognition, Clinical Reasoning, Large Language Models, Decision Making, Humans, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
