Evaluation and mitigation of the limitations of large language models in clinical decision-making
Open accessPublished version of a preprint
The DOI matches its Crossref record; 92 of the 92 references it cites were found in a registry or an index; 20 of the 24 sources its claims cite were checked for support.
Abstract
Clinical decision-making is one of the most impactful parts of a physician's responsibilities and stands to benefit greatly from artificial intelligence solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent performance on medical licensing exams, these tests fail to assess many skills necessary for deployment in a realistic clinical decision-making environment, including gathering information, adhering to guidelines, and integrating into clinical workflows. Here we have created a curated dataset based on the Medical Information Mart for Intensive Care database spanning 2,400 real patient cases and four common abdominal pathologies as well as a framework to simulate a realistic clinical setting. We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients. Furthermore, we move beyond diagnostic accuracy and demonstrate that they cannot be easily integrated into existing workflows because they often fail to follow instructions and are sensitive to both the quantity and order of information. Overall, our analysis reveals that LLMs are currently not ready for autonomous clinical decision-making while providing a dataset and framework to guide future studies.
Overview
CoChat Summary
TL;DR
This study evaluates open-access LLMs on a realistic clinical decision-making task using 2,400 real emergency department cases of acute abdominal pain from MIMIC-IV. The models performed significantly worse than physicians, failed to follow diagnostic/treatment guidelines, and were too fragile and instruction-sensitive for autonomous clinical use.
Key findings
- The curated dataset, MIMIC-CDM, contains 2,400 unique patients with appendicitis (957), cholecystitis (648), diverticulitis (257), and pancreatitis (538).
- On a reader study with 80 patients and 4 hospitalists, current LLMs performed significantly worse than clinicians on aggregate across all diseases (P < 0.001 for doctors versus Llama 2 Chat, OASST, WizardLM, Clinical Camel, and Meditron).
- Diagnostic performance dropped further when models had to gather information themselves: mean diagnostic accuracy fell to 45.5% for Llama 2 Chat, 54.9% for OASST, and 53.9% for WizardLM, versus 58.8%, 67.8%, and 65.1% on the full-information setup, respectively.
- Models also failed basic safety checks: only Llama 2 Chat consistently requested physical examination information (97.1% as first action; 98.1% at all), and laboratory interpretation was poor even with reference ranges (for example, low results: 26.5% Chat, 70.2% OASST, 45.8% WizardLM; high results: 50.1%, 77.2%, 24.1%).
Methods
- The authors built MIMIC-CDM from the MIMIC-IV EHR database by filtering to emergency department cases with acute abdominal pain and a primary diagnosis of one of four target pathologies, then extracting history of present illness, physical exams, labs, microbiology, imaging, and procedures.
- They evaluated open-access models derived from Llama 2, focusing mainly on Llama 2 Chat, OASST, and WizardLM, and compared them with clinicians on a 100-patient reader study (80 target cases plus 5 each of other conditions), with each LLM evaluated 20 times on the physician comparison subset using different seeds.
- Main analysis included per-class accuracy, guideline adherence for diagnostic tests and treatments, lab interpretation tasks, instruction-following, robustness to changes in wording, order, and amount of information, and comparisons using two-sided Student’s t-tests with Bonferroni correction.
Limitations
- The paper states that the dataset and models have a clear bias toward the USA, so generalizability to other languages or countries with different guidelines is unknown.
- The authors note that, because they used real-world clinical data, they had to deny requests for information not in the dataset; they argue this was mitigated because MIMIC-IV contains all data gathered during the hospital stay.
- Evident from the text: the study focuses only on four abdominal pathologies, so it does not cover the full breadth of clinical decision-making (the paper itself notes this as a constraint).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures














Verification
Claims the paper makes
6 Valid · 2 Not valid · 3 Not checked · 106 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“where LLMs score well above passing8–10,17–20”
§ Evaluation and mitigation of the limitations of large language models in clinical decision-making > Main
What the sources say
- Supportedcites [8] Singhal, K. et al. Towards expert-level medical question answering with large language models. Preprint at https://arxiv.org/abs/2305.09617 (2023). source
The Abstract (page not supplied) reports that Med-PaLM 2 scored 86.5% on MedQA, a dataset of USMLE-style questions, and that its predecessor had exceeded a passing score at 67.2%. This supports the claim about LLM performance on medical-licensing-exam-style questions. (LLM, conf=0.93)
- Partly supportedcites [9] Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023). sourceExperimental — accuracy not yet measured
The ‘State of the art on MedQA’ section reports that Flan-PaLM scored 67.6% on USMLE-style questions, supporting strong performance on a medical-exam benchmark. Our copy does not establish a passing threshold or show that LLMs scored well above it on actual licensing exams. (LLM, conf=0.86)
- Supportedcites [10] Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on medical challenge problems. Preprint at https://arxiv.org/abs/2303.13375 (2023). source
The Abstract (p. 1) reports that GPT-4 exceeded the USMLE passing score by over 20 points. The Introduction (p. 2) specifies that the results came from official sample exams and self-assessment materials, supporting this citation as an example of an LLM scoring well above passing on medical licensing exam materials. (LLM, conf=0.93)
- Disputedcites [17] Kung, T. H. et al. Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models. PLoS Digital Health 2, 0000198 (2023). sourceExperimental — accuracy not yet measured
The Discussion (‘The rising accuracy of ChatGPT’) says ChatGPT exceeded 60% accuracy in some analyses, while the USMLE passing threshold is approximately 60%; it concludes that ChatGPT ‘approaches the passing range.’ The paper describes performance as at or near passing, not well above it. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Disputedcites [18] Gilson, A. et al. How does chatgpt perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med. Educ. 9, 45312 (2023). sourceExperimental — accuracy not yet measured
The Results (Overall Performance; page unavailable) report that ChatGPT exceeded the paper’s approximately 60% passing benchmark on only one of four question sets, scoring 64.4%. It scored below that benchmark on the other three; InstructGPT and GPT-3 scored below it on all four. The paper therefore does not show LLMs scoring well above passing. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
- Partly supportedcites [19] Toma, A. et al. Clinical camel: an open-source expert-level medical language model with dialogue-based knowledge encoding. Preprint at https://arxiv.org/abs/2305.12031 (2023). sourceExperimental — accuracy not yet measured
Table 4 reports strong five-shot scores on the USMLE Sample Exam, including 86.6% for GPT-4 and 64.3% for Clinical Camel-70B. But the paper gives no passing threshold, and other models score substantially lower, so it does not establish that LLMs generally score well above passing. (LLM, conf=0.82)
- Supportedcites [20] Nori, H. et al. Can generalist foundation models outcompete special-purpose tuning? Case study in medicine. Preprint at https://arxiv.org/abs/2311.16452 (2023). source
Section 2.1 reports that GPT-4 exceeded the USMLE passing score by over 20 points. The Abstract also reports that GPT-4 with Medprompt surpassed 90% on MedQA, a USMLE-style exam-question dataset. These passages support the cited claim that LLMs can score well above passing on medical licensing exam questions. (LLM, conf=0.91)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We note that Llama 2, Clinical Camel and Meditron have been shown to match and even exceed Chat-GPT performance on medical licensing exams and biomedical question answering tests19,35.”
§ Evaluation and mitigation of the limitations of large language models in clinical decision-making > Results > Creating the MIMIC-CDM dataset and evaluation framework
What the sources say
- Partly supportedcites [19] Toma, A. et al. Clinical camel: an open-source expert-level medical language model with dialogue-based knowledge encoding. Preprint at https://arxiv.org/abs/2305.12031 (2023). sourceExperimental — accuracy not yet measured
Section 4 (Evaluation) and Table 4 report that Clinical Camel-70B exceeded GPT-3.5 on five-shot medical benchmarks, including MedQA, PubMedQA and the USMLE sample exam. The paper does not show that LLaMA 2 or Meditron achieved those results; Clinical Camel was fine-tuned from LLaMA 2. It also says Clinical Camel generally fell short of GPT-4, except on PubMedQA. Page numbers were not available in our copy. (LLM, conf=0.88)
- Partly supportedcites [35] Chen, Z. et al. MEDITRON-70B: scaling medical pretraining for large language models. Preprint at https://arxiv.org/abs/2311.16079 (2023). sourceExperimental — accuracy not yet measured
Section 6.2 reports that MEDITRON-70B outperformed GPT-3.5 on the medical benchmarks, including MedQA and PubMedQA. It evaluates Llama-2 and Clinical-Camel as baselines but does not establish that either matched or exceeded GPT-3.5 on those tests. The benchmark comparison also uses text-davinci-003 as GPT-3.5, not ChatGPT itself. (LLM, conf=0.87)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“All guidelines recommended physical examinations as an essential part of the diagnostic process, preferably as the first action.”
§ Evaluation and mitigation of the limitations of large language models in clinical decision-making > Results > Current LLMs are hasty and unsafe clinical decision-makers
What the sources say
- Partly supportedcites [36] Di Saverio, S. et al. Diagnosis and treatment of acute appendicitis: 2020 update of the WSES Jerusalem guidelines. World J. Emerg. Surg. 15, 27 (2020). sourceExperimental — accuracy not yet measured
In Results, Topic 1: Diagnosis, the WSES guideline includes physical examination and clinical signs in its diagnostic approach, and Recommendation 1.10 makes imaging conditional on clinical assessment. This supports an early role for examination, but this single guideline does not establish what all guidelines recommend or explicitly say physical examination should be the first action. (LLM, conf=0.85)
- Partly supportedcites [37] Pisano, M. et al. 2020 World Society of Emergency Surgery updated guidelines for the diagnosis and treatment of acute calculus cholecystitis. World J. Emerg. Surg. 15, 61 (2020). sourceExperimental — accuracy not yet measured
Section 1, Diagnosis of ACC (page not provided), suggests including a complete clinical examination alongside history, laboratory tests and imaging. It does not say examination should be the first action, and this single guideline does not establish what all guidelines recommend. (LLM, conf=0.87)
- Partly supportedcites [38] Hall, J. et al. The american society of colon and rectal surgeons clinical practice guidelines for the treatment of left-sided colonic diverticulitis. Dis. Colon Rectum 63, 728–747 (2020). sourceExperimental — accuracy not yet measured
In “Initial Evaluation of Acute Diverticulitis” (recommendation 1; page not supplied), this guideline recommends a physical examination as part of the initial evaluation. It does not establish what all guidelines recommend or say that the examination should preferably be the first action. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, we see that even minor changes in instructions can greatly change diagnostic accuracy such as asking for the ‘main diagnosis’ (up to +7.0% for Chat on diverticulitis, down to −10.6% for WizardLM on cholecystitis) or ‘primary diagnosis’ (up +8.7% for Chat on pancreatitis, down to −5.2% for WizardLM on cholecystitis) instead of ‘final diagnosis’.”
§ Evaluation and mitigation of the limitations of large language models in clinical decision-making > Results > Current LLMs require extensive clinician supervision
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians)”
§ Evaluation and mitigation of the limitations of large language models in clinical decision-making > Abstract
plainn = 2,400- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
59 more claims
Create a free account to see all 117 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 40
- Probably not a substantial claim. 20
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While medical licensing exams and diagnostic tasks are prioritized in 64% of LLM healthcare evaluations8, many administrative and operational tasks that represent significant human bottlenecks are often overlooked.”
From Holistic evaluation of large language models for medical tasks with MedHELM (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While these models offer powerful foundational capabilities, adapting them to specific tasks and datasets without losing robustness is a significant hurdle, often resulting in significant variability in accuracy across different settings.7,9,10”
From Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently (2025) · cites this paper as [9]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While LLM performance on medical QA benchmarks has been demonstrated to rival human-level performance, their capabilities in simulated real-world medical scenarios have been more limited12.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“Another study demonstrated that while LLMs show potential for clinical decision-making, they currently underperform compared to human clinicians in diagnosing common conditions such as appendicitis, cholecystitis, diverticulitis, and pancreatitis.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [30]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Given these constraints, we calculate and use the per-class accuracy throughout our work, which is the only metric that can be calculated without bias as it only requires the samples of a single class.”
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287 · cited 1×
“Future models could possibly soon reach or even outperform clinicians in residency and thus provide a low-cost, interactive second opinion to consult, as is already the case for AI models in other areas such as mammography screening 70 .”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1×
“To optimize performance on this specific clinical decision-making task, future studies could explore fine-tuning 40 or prompt-tuning 41 a base LLM as well as automated prompt engineering 42 .”
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 2×
“Future models could possibly soon reach or even outperform clinicians in residency and thus provide a low-cost, interactive second opinion to consult, as is already the case for AI models in other areas such as mammography screening 70 .”
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · cited 1×
“The sensitivity of LLMs to the order of presented information has been well documented on multiple-choice questions 45,46 and information retrieval 47 .”
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599
Diagnosis and treatment of acute appendicitis: 2020 update of the WSES Jerusalem guidelines
Di Saverio, S., Podda, M., De Simone, B., Ceresoli, M., Augustin, G., Gori, A., Boermeester, M., Sartelli, M., Coccolini, F., Tarasconi, A., et al. (2020) · World Journal of Emergency Surgery · doi:10.1186/s13017-020-00306-3 · Cited by 1,446 · cited 2×
“The current clinical guidelines used for this study are available in the literature for appendicitis 36 , cholecystitis 37 , diverticulitis 38 and pancreatitis 39 .”
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
Human-computer collaboration for skin cancer recognition
Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., Janda, M., Lallas, A., Longo, C., Malvehy, J., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0942-0 · Cited by 1,024 · cited 1×
“Importantly, studies as to the impact of automation bias 53,54 and human– AI interaction biases could also be explored in such a context 55–57 .”
2019 WSES guidelines for the management of severe acute pancreatitis
Leppäniemi, A., Tolonen, M., Tarasconi, A., Segovia-Lohse, H., Gamberini, E., Kirkpatrick, A. W., Ball, C. G., Parry, N., Sartelli, M., Wolbrink, D., et al. (2019) · World Journal of Emergency Surgery · doi:10.1186/s13017-019-0247-0 · Cited by 983
Clinical decision support in the era of artificial intelligence
Shortliffe, E. H., & Sepúlveda, M. J. (2018) · JAMA · doi:10.1001/jama.2018.17163 · Cited by 920 · cited 2×
“We believe that there exists great potential in using LLMs as clinical support systems with close collaboration between models and clinicians 21,27 ; however, we have primarily evaluated the capabilities of models as autonomous decision-makers in this study.”
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891 · cited 1×
“LLMs can summarize reports 3–5 , generate reports 2,4 , serve as diagnostic assistants 21,27 and could ultimately autonomously diagnose patients.”
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · cited 1×
“For each model and disease combination, we divided the number of correct diagnoses by the total number of patients with that disease.”
Pisano, M., Allievi, N., Gurusamy, K., Borzellino, G., Cimbanassi, S., Boerna, D., Coccolini, F., Tufo, A., Di Martino, M., Leung, J., et al. (2020) · World Journal of Emergency Surgery · doi:10.1186/s13017-020-00336-x · Cited by 581
ChatGPT and the future of medical writing
Biswas, S. (2023) · Radiology · doi:10.1148/radiol.223312 · Cited by 571
Transparency and reproducibility in artificial intelligence
Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., et al. (2020) · Nature · doi:10.1038/s41586-020-2766-y · Cited by 540
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509
Hall, J., Hardiman, K., Lee, S., Lightner, A., Stocchi, L., Paquette, I. M., Steele, S. R., Feingold, D. L., & Prepared on behalf of the Clinical Practice Guidelines Committee of the American Society of Colon and Rectal Surgeons (2020) · Diseases of the Colon & Rectum · doi:10.1097/dcr.0000000000001679 · Cited by 485 · cited 2×
“All models struggle to follow the provided instructions (Extended Data Fig.”
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2024) · NEJM AI · doi:10.1056/aioa2300138 · Cited by 483
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483 · cited 1×
“LLMs can summarize reports 3–5 , generate reports 2,4 , serve as diagnostic assistants 21,27 and could ultimately autonomously diagnose patients.”
Automation bias and verification complexity: a systematic review
Lyell, D., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw105 · Cited by 451 · cited 1×
“Importantly, studies as to the impact of automation bias 53,54 and human– AI interaction biases could also be explored in such a context 55–57 .”
Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge
Kanjee, Z., Crowe, B., & Rodman, A. (2023) · JAMA · doi:10.1001/jama.2023.8288 · Cited by 449
RedPajama: an Open Dataset for Training Large Language Models
Weber, M., Fu, D. Y., Anthony, Q. G., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., et al. (2024) · neural information processing systems · doi:10.52202/079017-3697 · Cited by 428
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023) · Journal of Medical Internet Research · doi:10.2196/48659 · Cited by 412 · cited 1×
“However, the only analysis that tested an LLM throughout the diagnostic clinical workflow used curated lists of possible answers and examined only 36 hypothetical clinical vignettes 28 .”
OpenAssistant Conversations - Democratizing Large Language Model Alignment
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D., Stanley, O., Nagyfi, R., et al. (2023) · neural information processing systems · doi:10.52202/075280-2064 · Cited by 402
Towards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338
Faith and Fate: Limits of Transformers on Compositionality
Dziri, N., Lu, X., Sclar, M., Li, X. (., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Bras, R. L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3081 · Cited by 335 · cited 1×
“The subset was evenly split between the four target pathologies with 20 patients randomly selected from each pathology and matching the makeup of the full dataset (Supplementary Section H).”
Impact of a deep learning assistant on the histopathologic classification of liver cancer
Kiani, A., Uyumazturk, B., Rajpurkar, P., Wang, A., Gao, R., Jones, E., Yu, Y., Langlotz, C. P., Ball, R. L., Montine, T. J., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0232-8 · Cited by 325
Cervellin, G., Mora, R., Ticinesi, A., Meschi, T., Comelli, I., Catena, F., & Lippi, G. (2016) · Annals of Translational Medicine · doi:10.21037/atm.2016.09.10 · Cited by 316 · cited 1×
“We chose these target pathologies as they represent clinically important diagnoses of a common chief complaint, abdominal pain, which accounts for 10% of all emergency department visits 30,31 .”
Towards Accurate Differential Diagnosis with Large Language Models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00164 · Cited by 311 · cited 2×
“LLMs can summarize reports 3–5 , generate reports 2,4 , serve as diagnostic assistants 21,27 and could ultimately autonomously diagnose patients.”
Towards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311 · cited 2×
“LLMs can summarize reports 3–5 , generate reports 2,4 , serve as diagnostic assistants 21,27 and could ultimately autonomously diagnose patients.”
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291
Machine learning in clinical decision making
Adlung, L., Cohen, Y., Mor, U., & Elinav, E. (2021) · Med · doi:10.1016/j.medj.2021.04.006 · Cited by 282
Automation bias in intelligent time critical decision support systems
Cummings, M. L. (2017) · Decision Making in Aviation · doi:10.4324/9781315095080-17 · Cited by 233
OpenAssistant Conversations - Democratizing Large Language Model Alignment
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07327 · Cited by 218 · cited 1×
“We test both generalist versions such as Llama 2 Chat (70B) 32 , Open Assistant (OASST) (70B) 33 and WizardLM (70B) 34 , as well as medical-domain aligned models such as Clinical Camel (70B) 19 and Meditron (70B) 35 .”
Use of gpt-4 to diagnose complex clinical cases
Eriksen, A. V., Möller, S., & Ryg, J. (2024) · NEJM AI · doi:10.1056/aip2300031 · Cited by 200
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022) · arXiv · doi:10.48550/arxiv.2210.17323 · Cited by 197
Thirunavukarasu, A. J., Hassan, R., Mahmood, S., Sanghera, R., Barzangi, K., El Mukashfi, M., & Shah, S. (2023) · JMIR Medical Education · doi:10.2196/46599 · Cited by 187
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16452 · Cited by 178
Enhancing Clinical Decision Making: Development of a Contiguous Definition and Conceptual Framework
Tiffen, J., Corbridge, S. J., & Slimmer, L. (2014) · Journal of Professional Nursing · doi:10.1016/j.profnurs.2014.01.006 · Cited by 175 · cited 1×
“In our simulated clinical environment, which uses the MIMIC-CDM dataset, the LLM must specify all information it wishes to gather to evaluating the facts to reach an evidence-based decision on a patient’s diagnosis and treatment 25,26 .”
Improving chest X-ray report generation by leveraging warm starting
Nicolson, A., Dowling, J., & Koopman, B. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102633 · Cited by 161
MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2022) · arXiv · doi:10.48550/arxiv.2203.14371 · Cited by 157
Large Language Models as Optimizers
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., & Chen, X. (2023) · arXiv · doi:10.48550/arxiv.2309.03409 · Cited by 151 · cited 1×
“A subset of 80 representative patients of the MIMIC-CDM-FI data-set was randomly selected to be used for comparison with the physicians.”
Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models
van Sonsbeek, T., Derakhshani, M. M., Najdenkoska, I., Snoek, C. G. M., & Worring, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43904-9_70 · Cited by 149
Large Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134 · cited 1×
“All four of the hospitalists are internal medicine physicians with emergency department experience.”
Multimodal LLMs for Health Grounded in Individual-Specific Data
Belyaeva, A., Cosentino, J., Hormozdiari, F., Eswaran, K., Shetty, S., Corrado, G. S., Carroll, A. W., McLean, C. Y., & Furlotte, N. A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47679-2_7 · Cited by 128
Faith and Fate: Limits of Transformers on Compositionality
Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., et al. (2023) · arXiv · Cited by 124 · cited 1×
“The subset was evenly split between the four target pathologies with 20 patients randomly selected from each pathology and matching the makeup of the full dataset (Supplementary Section H).”
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16079 · Cited by 122 · cited 2×
“We note that Llama 2, Clinical Camel and Meditron have been shown to match and even exceed Chat-GPT performance on medical licensing exams and biomedical question answering tests 19,35 .”
WizardLM: Empowering large pre-trained language models to follow complex instructions
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., & Jiang, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.12244 · Cited by 106 · cited 1×
“We test both generalist versions such as Llama 2 Chat (70B) 32 , Open Assistant (OASST) (70B) 33 and WizardLM (70B) 34 , as well as medical-domain aligned models such as Clinical Camel (70B) 19 and Meditron (70B) 35 .”
LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation
Lee, S., Kim, W. J., Chang, J., & Ye, J. C. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2305.11490 · Cited by 104 · cited 1×
“LLMs can summarize reports 3–5 , generate reports 2,4 , serve as diagnostic assistants 21,27 and could ultimately autonomously diagnose patients.”
Vision–Language Model for Visual Question Answering in Medical Imagery
Bazi, Y., Rahhal, M. M. A., Bashmal, L., & Zuair, M. (2023) · Bioengineering · doi:10.3390/bioengineering10030380 · Cited by 87
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
Pezeshkpour, P., & Hruschka, E. (2023) · arXiv · doi:10.48550/arxiv.2308.11483 · Cited by 86
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
Pezeshkpour, P., & Hruschka, E. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.130 · Cited by 85
(2021) · doi:10.18653/v1/2021.naacl-main.53 · Cited by 78 · cited 1×
“The difficulty LLMs have in interpreting numbers 48 and solving simple arithmetic 49 is an active research topic 50,51 .”
Mitigating bias in AI at the point of care
DeCamp, M., & Lindvall, C. (2023) · Science · doi:10.1126/science.adh2713 · Cited by 70
PRAG: Paninian Retrieval-Augmented Generation for Safety-Critical Medical Question Answering
Jin, D., Pan, E., Oufattole, N., Weng, W., Fang, H., & Peter Szolovits, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13081 · Cited by 63
EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers
Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., & Yang, Y. (2023) · arXiv · doi:10.48550/arxiv.2309.08532 · Cited by 50 · cited 1×
“If any other diagnosis was written in the discharge diagnosis before one of our targets, the patient’s case was removed from the dataset.”
Bridging the Gap Between Consumers’ Medication Questions and Trusted Answers
Ben, A. A., Yassine, M., Mark, S., Goodwin Travis, R., Shooshan Sonya, E., & Dina, D.-F. (2019) · Studies in Health Technology and Informatics · doi:10.3233/shti190176 · Cited by 41
Instruction-Following Evaluation for Large Language Models
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., & Hou, L. (2023) · arXiv · doi:10.48550/arxiv.2311.07911 · Cited by 39 · cited 1×
“To mitigate the risk of physicians recognizing the pattern of four distinct target pathologies, a further five patients presenting with gastritis, a urinary tract infection, esophageal reflux and a hernia were included.”
Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding
Toma, A., Lawler, P. R., Ba, J., Krishnan, R. G., Rubin, B. B., & Wang, B. (2023) · arXiv · doi:10.48550/arxiv.2305.12031 · Cited by 36
Generative AI could revolutionize health care — but not if control is ceded to big tech
Toma, A., Senkaiahliyan, S., Lawler, P. R., Rubin, B., & Wang, B. (2023) · Nature · doi:10.1038/d41586-023-03803-y · Cited by 29 · cited 2×
“We strongly agree with the current sentiment that open-source models must drive progress in the field of medical AI due to patient privacy and safety concerns, corporate lack of transparency and the danger of unreliable external providers 74 .”
RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models
Van Veen, D., Van Uden, C., Attias, M., Pareek, A., Bluethgen, C., Polacin, M., Chiu, W., Delbrouck, J.-B., Zambrano Chaves, J., Langlotz, C., et al. (2023) · The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks · doi:10.18653/v1/2023.bionlp-1.42 · Cited by 25
Testolin, A. (2024) · Applied Sciences · doi:10.3390/app14020744 · Cited by 24 · cited 1×
“The difficulty LLMs have in interpreting numbers 48 and solving simple arithmetic 49 is an active research topic 50,51 .”
TRAM: Benchmarking Temporal Reasoning for Large Language Models
Wang, Y., & Zhao, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.00835 · Cited by 24
TRAM: Benchmarking Temporal Reasoning for Large Language Models
Wang, Y., & Zhao, Y. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.382 · Cited by 24
Multimodal Foundation Models Exploit Text to Make Medical Image PredictionsPreprint
Buckley, T., Diao, J. A., Rajpurkar, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.05591 · Cited by 18
Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language ModelsPreprint
van Sonsbeek, T., Derakhshani, M. M., Najdenkoska, I., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.05977 · Cited by 11
Berman, S. (2011) · Elsevier eBooks · doi:10.1016/b978-0-323-05405-8.00010-3 · Cited by 10
Multimodal LLMs for health grounded in individual-specific dataPreprint
Belyaeva, A., Cosentino, J., Hormozdiari, F., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.09018 · Cited by 8
RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models
Veen, D. V., Uden, C. V., Attias, M., Pareek, A., Blüthgen, C., Polacin, M., Chiu, W. K., Delbrouck, J.-B., Chaves, J. M. Z., Langlotz, C. P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.01146 · Cited by 6 · cited 1×
“LLMs can summarize reports 3–5 , generate reports 2,4 , serve as diagnostic assistants 21,27 and could ultimately autonomously diagnose patients.”
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Research works printed but not linked: 4
- Di Saverio, S. et al. Diagnosis and treatment of acute appendicitis: 2020 update of the WSES Jerusalem guidelines. World J. Emerg. Surg. 15, 27 (2020). · DOI · Paper page
- Zheng, C., Zhou, H., Meng, F., Zhou, J. & Huang, M. On large language models’ selection bias in multi-choice questions. In 12th International Conference on Learning Representations (ICLR, 2024).
- Golkar, Siavash, et al. xVal: a continuous number encoding for large language models. In NeurIPS 2023 AI for Science Workshop (2023).
- Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019).
Cited by
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2024) · doi:10.1101/2024.04.15.24305869 · Cited by 738
Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2025) · JAMA · doi:10.1001/jama.2024.21700 · Cited by 662
A Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
“Moreover, more efforts from the government and large corporations are spent to regularize and monitor the use of AI in various fields, including healthcare and medicine.”
BackgroundBenchmark evaluation of DeepSeek large language models in clinical decision-making
Sandmann, S., Hegselmann, S., Fujarski, M., Bickmann, L., Wild, B., Eils, R., & Varghese, J. (2025) · Nature Medicine · doi:10.1038/s41591-025-03727-2 · Cited by 233
Takita, H., Kabata, D., Walston, S. L., Tatekawa, H., Saito, K., Tsujimoto, Y., Miki, Y., & Ueda, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01543-z · Cited by 203
“…physicians outperformed the model, whereas negative values (to the left) indicate that the model was more accurate than the physicians. that of physicians, recent research demonstrates that generative AI may perform signi fi cantly worse than physicians in more complex scenarios 80 .”
Metadata
- Authors
- Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., & Rueckert, D.
- Affiliations
- TUM Klinikum (DE)
- Imperial College London (GB)
- Christiana Care Health System (US)
- Ludwig-Maximilians-Universität München (DE)
- Corresponding author
- Hager, Paul
- ORCID
- Venue
- Nature Medicine
- Published
- 2024-07-04
- Volume
- 30
- Issue
- 9
- Pages
- 2613-2622
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11405275
- OpenAlex
- W4400324908
- Cited by
- 798
- References
- 74
- Citations by year
- 2026: 435 · 2025: 317 · 2024: 46
- FWCI
- 74.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Medicine (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PuSH - Publication Server of Helmholtz Zentrum München (submittedVersion, cc-by)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
- Funders
- Technische Universität München
- H2020 European Research Council (884622)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, clinical decision-making, artificial intelligence in healthcare, MIMIC-III, abdominal pathology, diagnostic accuracy, clinical guidelines adherence, laboratory test interpretation, clinical workflow integration, medical licensing examinations, LLM limitations
- MeSH terms
- Clinical Decision-Making, Artificial Intelligence, Humans, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.