Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 92 of the 92 references it cites were found in a registry or an index; 20 of the 24 sources its claims cite were checked for support.

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Open access

Published version of a preprint

Abstract

Clinical decision-making is one of the most impactful parts of a physician's responsibilities and stands to benefit greatly from artificial intelligence solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent performance on medical licensing exams, these tests fail to assess many skills necessary for deployment in a realistic clinical decision-making environment, including gathering information, adhering to guidelines, and integrating into clinical workflows. Here we have created a curated dataset based on the Medical Information Mart for Intensive Care database spanning 2,400 real patient cases and four common abdominal pathologies as well as a framework to simulate a realistic clinical setting. We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients. Furthermore, we move beyond diagnostic accuracy and demonstrate that they cannot be easily integrated into existing workflows because they often fail to follow instructions and are sensitive to both the quantity and order of information. Overall, our analysis reveals that LLMs are currently not ready for autonomous clinical decision-making while providing a dataset and framework to guide future studies.

Overview

CoChat Summary

TL;DR

This study evaluates open-access LLMs on a realistic clinical decision-making task using 2,400 real emergency department cases of acute abdominal pain from MIMIC-IV. The models performed significantly worse than physicians, failed to follow diagnostic/treatment guidelines, and were too fragile and instruction-sensitive for autonomous clinical use.

Key findings

  • The curated dataset, MIMIC-CDM, contains 2,400 unique patients with appendicitis (957), cholecystitis (648), diverticulitis (257), and pancreatitis (538).
  • On a reader study with 80 patients and 4 hospitalists, current LLMs performed significantly worse than clinicians on aggregate across all diseases (P < 0.001 for doctors versus Llama 2 Chat, OASST, WizardLM, Clinical Camel, and Meditron).
  • Diagnostic performance dropped further when models had to gather information themselves: mean diagnostic accuracy fell to 45.5% for Llama 2 Chat, 54.9% for OASST, and 53.9% for WizardLM, versus 58.8%, 67.8%, and 65.1% on the full-information setup, respectively.
  • Models also failed basic safety checks: only Llama 2 Chat consistently requested physical examination information (97.1% as first action; 98.1% at all), and laboratory interpretation was poor even with reference ranges (for example, low results: 26.5% Chat, 70.2% OASST, 45.8% WizardLM; high results: 50.1%, 77.2%, 24.1%).

Methods

  • The authors built MIMIC-CDM from the MIMIC-IV EHR database by filtering to emergency department cases with acute abdominal pain and a primary diagnosis of one of four target pathologies, then extracting history of present illness, physical exams, labs, microbiology, imaging, and procedures.
  • They evaluated open-access models derived from Llama 2, focusing mainly on Llama 2 Chat, OASST, and WizardLM, and compared them with clinicians on a 100-patient reader study (80 target cases plus 5 each of other conditions), with each LLM evaluated 20 times on the physician comparison subset using different seeds.
  • Main analysis included per-class accuracy, guideline adherence for diagnostic tests and treatments, lab interpretation tasks, instruction-following, robustness to changes in wording, order, and amount of information, and comparisons using two-sided Student’s t-tests with Bonferroni correction.

Limitations

  • The paper states that the dataset and models have a clear bias toward the USA, so generalizability to other languages or countries with different guidelines is unknown.
  • The authors note that, because they used real-world clinical data, they had to deny requests for information not in the dataset; they argue this was mitigated because MIMIC-IV contains all data gathered during the hospital stay.
  • Evident from the text: the study focuses only on four abdominal pathologies, so it does not cover the full breadth of clinical decision-making (the paper itself notes this as a constraint).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41591-024-03097-1

Cites

92 of 92 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Overview of dataset creation and evaluation framework. a, To properly evaluate LLMs for clinical decision-making in realistic conditions, we created a curated dataset from real-world cases derived from the MIMIC-IV database, which contains comprehensive electronic health record data recorded during hospital admissions. b, Our evaluation framework reflects a realistic clinical setting and thoroughly evaluates LLMs across multiple criteria, including diagnostic accuracy, adherence to diagnostic and treatment guidelines, consistency in following instructions, ability to interpret laboratory results, and robustness to changes in instruction, information quantity and information order. ICD, International Classification of Diseases; CT, computed tomography; US, ultrasound; MRCP, magnetic resonance cholangiopancreatography.
Fig. 1 Overview of dataset creation and evaluation framework. a, To properly evaluate LLMs for clinical decision-making in realistic conditions, we created a curated dataset from real-world cases derived from the MIMIC-IV database, which contains comprehensive electronic health record data recorded during hospital admissions. b, Our evaluation framework reflects a realistic clinical setting and thoroughly evaluates LLMs across multiple criteria, including diagnostic accuracy, adherence to diagnostic and treatment guidelines, consistency in following instructions, ability to interpret laboratory results, and robustness to changes in instruction, information quantity and information order. ICD, International Classification of Diseases; CT, computed tomography; US, ultrasound; MRCP, magnetic resonance cholangiopancreatography.CC BY · PubMed Central
Fig. 2 LLMs diagnose significantly worse than doctors when provided with all information. On a subset (n = 80) of MIMIC-CDM-FI, we compared the mean diagnostic accuracy of LLMs over multiple seeds (n = 20) with clinicians (n = 4) and found that LLMs perform significantly worse on average (P < 0.001) and especially on cholecystitis (P < 0.001) and diverticulitis (P < 0.001). The mean diagnostic accuracy is shown above each bar. Vertical lines indicate the standard deviation. The individual data points are shown.
Fig. 2 LLMs diagnose significantly worse than doctors when provided with all information. On a subset (n = 80) of MIMIC-CDM-FI, we compared the mean diagnostic accuracy of LLMs over multiple seeds (n = 20) with clinicians (n = 4) and found that LLMs perform significantly worse on average (P < 0.001) and especially on cholecystitis (P < 0.001) and diverticulitis (P < 0.001). The mean diagnostic accuracy is shown above each bar. Vertical lines indicate the standard deviation. The individual data points are shown.CC BY · PubMed Central
Fig. 3 Diagnostic accuracy of LLMs decreased in an autonomous clinical decision-making scenario. When tasked with gathering all information required for clinical decision-making themselves, LLMs perform best when diagnosing appendicitis but perform poorly on the other three pathologies of cholecystitis, diverticulitis and pancreatitis. In such a realistic clinical scenario, model performance decreased compared to the retrospective diagnosis with all information provided (MIMIC-CDM-FI). The exact diagnostic accuracy is shown above each bar.
Fig. 3 Diagnostic accuracy of LLMs decreased in an autonomous clinical decision-making scenario. When tasked with gathering all information required for clinical decision-making themselves, LLMs perform best when diagnosing appendicitis but perform poorly on the other three pathologies of cholecystitis, diverticulitis and pancreatitis. In such a realistic clinical scenario, model performance decreased compared to the retrospective diagnosis with all information provided (MIMIC-CDM-FI). The exact diagnostic accuracy is shown above each bar.CC BY · PubMed Central
Fig. 4 LLMs do not consistently recommend essential and patient-specific treatment. Expected treatments were determined based on clinical guidelines and actual treatments received by patients in the dataset. Models fail to recommend appropriate treatments especially for patients with more severe forms of the pathologies. We only scored models on the subset of patients that they correctly diagnosed and that actually received a specific treatment. For example, of the 957 patients with appendicitis, 808 received an appendectomy (indicated below the treatment name). Of those 808 patients, Llama 2 Chat correctly diagnosed 603 (indicated below the Llama 2 Chat bar). Of those 603 patients, Llama 2 Chat correctly recommended an appendectomy 97.5% of the time. ERCP, endoscopic retrograde cholangiopancreatography.
Fig. 4 LLMs do not consistently recommend essential and patient-specific treatment. Expected treatments were determined based on clinical guidelines and actual treatments received by patients in the dataset. Models fail to recommend appropriate treatments especially for patients with more severe forms of the pathologies. We only scored models on the subset of patients that they correctly diagnosed and that actually received a specific treatment. For example, of the 957 patients with appendicitis, 808 received an appendectomy (indicated below the treatment name). Of those 808 patients, Llama 2 Chat correctly diagnosed 603 (indicated below the Llama 2 Chat bar). Of those 603 patients, Llama 2 Chat correctly recommended an appendectomy 97.5% of the time. ERCP, endoscopic retrograde cholangiopancreatography.CC BY · PubMed Central
Fig. 5 LLMs are sensitive to the quantity of information provided. We compared the performance of each model using all diagnostic information to using only a single diagnostic exam in addition to the history of present illness. For almost all diseases, providing all information does not lead to the best performance on the MIMIC-CDM-FI dataset. This suggests that LLMs cannot focus on the key facts and degrade in performance when too much information is provided. This poses a problem in the clinic where an abundance of information is typically gathered to holistically understand the patient’s health and being able to focus on key facts is an essential skill. The gray theoretical best line shows the mean performance if a clinician were to select the best diagnostic test for each pathology. HPI, history of present illness.
Fig. 5 LLMs are sensitive to the quantity of information provided. We compared the performance of each model using all diagnostic information to using only a single diagnostic exam in addition to the history of present illness. For almost all diseases, providing all information does not lead to the best performance on the MIMIC-CDM-FI dataset. This suggests that LLMs cannot focus on the key facts and degrade in performance when too much information is provided. This poses a problem in the clinic where an abundance of information is typically gathered to holistically understand the patient’s health and being able to focus on key facts is an essential skill. The gray theoretical best line shows the mean performance if a clinician were to select the best diagnostic test for each pathology. HPI, history of present illness.CC BY · PubMed Central
Fig. 6 LLMs are sensitive to the order of information. By changing the order in which diagnostic information from MIMIC-CDM-FI is presented to LLMs, their diagnostic accuracy changes despite the information included staying the same. This places an unnecessary burden upon clinicians who would need to make preliminary diagnoses to decide the order in which they feed the models with information for best performance.
Fig. 6 LLMs are sensitive to the order of information. By changing the order in which diagnostic information from MIMIC-CDM-FI is presented to LLMs, their diagnostic accuracy changes despite the information included staying the same. This places an unnecessary burden upon clinicians who would need to make preliminary diagnoses to decide the order in which they feed the models with information for best performance.CC BY · PubMed Central
Extended Data Fig. 1 Model performance on the MIMIC-CDM-FI dataset. LLMs perform best on the MIMIC-CDM-FI dataset where all information required for a diagnosis is provided, especially on pathologies with strong indications such as appendicitis (dilated appendix described in radiologist report) and pancreatitis (elevated pancreatic enzymes listed in laboratory test results).
Extended Data Fig. 1 Model performance on the MIMIC-CDM-FI dataset. LLMs perform best on the MIMIC-CDM-FI dataset where all information required for a diagnosis is provided, especially on pathologies with strong indications such as appendicitis (dilated appendix described in radiologist report) and pancreatitis (elevated pancreatic enzymes listed in laboratory test results).CC BY · PubMed Central
Extended Data Fig. 2 LLMs fail to consistently ask for a physical examination. The diagnostic guidelines of each disease require a physical examination to be performed as the first action. Llama 2 Chat was the only LLM that consistently requested physical examinations. (Late) Physical Examinations counted the physical examination if it wasn’t the first information requested.
Extended Data Fig. 2 LLMs fail to consistently ask for a physical examination. The diagnostic guidelines of each disease require a physical examination to be performed as the first action. Llama 2 Chat was the only LLM that consistently requested physical examinations. (Late) Physical Examinations counted the physical examination if it wasn’t the first information requested.CC BY · PubMed Central
Extended Data Fig. 3 LLMs often do not order the necessary laboratory tests required to establish a diagnosis. The tests, defined by current diagnostic guidelines, help differentiate abdominal pathologies, as results can indicate which organ is currently pathologically afflicted or functioning normally. When testing the presence of the necessary tests in the MIMIC-CDM dataset, we find that the MIMIC Doctors requested all necessary tests in every patient case.
Extended Data Fig. 3 LLMs often do not order the necessary laboratory tests required to establish a diagnosis. The tests, defined by current diagnostic guidelines, help differentiate abdominal pathologies, as results can indicate which organ is currently pathologically afflicted or functioning normally. When testing the presence of the necessary tests in the MIMIC-CDM dataset, we find that the MIMIC Doctors requested all necessary tests in every patient case.CC BY · PubMed Central
Extended Data Fig. 4 LLMs are incapable of interpreting lab results. To test the ability of LLMs to interpret laboratory data, we provided each laboratory test result and its reference range and asked the model to classify the result as below the reference range (low), within the range (normal) or above the range (high). We found that LLMs are incapable of consistently interpreting the result as normal, low or high, despite being provided with all required information. The models performed especially poorly on abnormal results which are of particular importance to establishing a diagnosis.
Extended Data Fig. 4 LLMs are incapable of interpreting lab results. To test the ability of LLMs to interpret laboratory data, we provided each laboratory test result and its reference range and asked the model to classify the result as below the reference range (low), within the range (normal) or above the range (high). We found that LLMs are incapable of consistently interpreting the result as normal, low or high, despite being provided with all required information. The models performed especially poorly on abnormal results which are of particular importance to establishing a diagnosis.CC BY · PubMed Central
Extended Data Fig. 5 The first imaging modality requested by the LLMs and the doctors in the MIMIC dataset. LLMs sometimes follow diagnostic guidelines concerning imaging but often diagnose without requesting any imaging at all. As we show that imaging is the most useful diagnostic tool for all LLMs for each pathology except pancreatitis, this could be partly responsible for their low diagnostic accuracy. The legend specifies the colors of the imaging modalities and the patterns of the models.
Extended Data Fig. 5 The first imaging modality requested by the LLMs and the doctors in the MIMIC dataset. LLMs sometimes follow diagnostic guidelines concerning imaging but often diagnose without requesting any imaging at all. As we show that imaging is the most useful diagnostic tool for all LLMs for each pathology except pancreatitis, this could be partly responsible for their low diagnostic accuracy. The legend specifies the colors of the imaging modalities and the patterns of the models.CC BY · PubMed Central
Extended Data Fig. 6 LLMs struggle to follow instructions. During the autonomous clinical decision making task, LLMs often introduce errors when providing the next action to take and hallucinate non-existent tools, up to once every two patients. Formatting errors while providing the diagnosis also regularly occur. In the clinic, extensive manual supervision would be required to ensure proper performance.
Extended Data Fig. 6 LLMs struggle to follow instructions. During the autonomous clinical decision making task, LLMs often introduce errors when providing the next action to take and hallucinate non-existent tools, up to once every two patients. Formatting errors while providing the diagnosis also regularly occur. In the clinic, extensive manual supervision would be required to ensure proper performance.CC BY · PubMed Central
Extended Data Fig. 7 Small changes in instruction phrasing changes diagnostic accuracy. Often small changes in instructions, such as changing final diagnosis to main diagnosis or primary diagnosis, greatly affects the performance of the LLMs on the MIMIC-CDM-FI dataset. This would vary the quality of responses received depending on who is using the model.
Extended Data Fig. 7 Small changes in instruction phrasing changes diagnostic accuracy. Often small changes in instructions, such as changing final diagnosis to main diagnosis or primary diagnosis, greatly affects the performance of the LLMs on the MIMIC-CDM-FI dataset. This would vary the quality of responses received depending on who is using the model.CC BY · PubMed Central
Extended Data Fig. 8 Filtering for only abnormal results generally improves LLM performance. Filtering to include only abnormal laboratory results using the laboratory reference ranges provided in MIMIC-IV database generally improves model performance, especially for the cholecystitis pathology. This allows the model to focus more on abnormal, pathological signals. While this approach is appropriate for models as they function today, healthy laboratory test results are an important source of information for clinicians and should not degrade model performance.
Extended Data Fig. 8 Filtering for only abnormal results generally improves LLM performance. Filtering to include only abnormal laboratory results using the laboratory reference ranges provided in MIMIC-IV database generally improves model performance, especially for the cholecystitis pathology. This allows the model to focus more on abnormal, pathological signals. While this approach is appropriate for models as they function today, healthy laboratory test results are an important source of information for clinicians and should not degrade model performance.CC BY · PubMed Central

Verification

Claims the paper makes

6 Valid · 2 Not valid · 3 Not checked · 106 Own results and interpretations

59 more claims

Create a free account to see all 117 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 40
  • Probably not a substantial claim. 20
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 74 references

Research works printed but not linked: 4

  1. Di Saverio, S. et al. Diagnosis and treatment of acute appendicitis: 2020 update of the WSES Jerusalem guidelines. World J. Emerg. Surg. 15, 27 (2020). · DOI · Paper page
  2. Zheng, C., Zhou, H., Meng, F., Zhou, J. & Huang, M. On large language models’ selection bias in multi-choice questions. In 12th International Conference on Learning Representations (ICLR, 2024).
  3. Golkar, Siavash, et al. xVal: a continuous number encoding for large language models. In NeurIPS 2023 AI for Science Workshop (2023).
  4. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019).

Cited by

See all 798

Metadata

Authors
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., & Rueckert, D.
Affiliations
  • TUM Klinikum (DE)
  • Imperial College London (GB)
  • Christiana Care Health System (US)
  • Ludwig-Maximilians-Universität München (DE)
Corresponding author
Hager, Paul
Venue
Nature Medicine
Published
2024-07-04
Volume
30
Issue
9
Pages
2613-2622
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
OpenAlex
W4400324908
Cited by
798
References
74
Citations by year
2026: 435 · 2025: 317 · 2024: 46
FWCI
74.24
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion, cc-by)
  • medRxiv (acceptedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PuSH - Publication Server of Helmholtz Zentrum München (submittedVersion, cc-by)
  • mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
Funders
  • Technische Universität München
  • H2020 European Research Council (884622)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
large language models, clinical decision-making, artificial intelligence in healthcare, MIMIC-III, abdominal pathology, diagnostic accuracy, clinical guidelines adherence, laboratory test interpretation, clinical workflow integration, medical licensing examinations, LLM limitations
MeSH terms
Clinical Decision-Making, Artificial Intelligence, Humans, Decision Support Systems, Clinical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluation and mitigation of the limitations of large language models in clinical decision-making. https://app.cochat.ai/papers/10.1038/s41591-024-03097-1
The paper
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., & Rueckert, D. (2024). Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine. https://doi.org/10.1038/s41591-024-03097-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.