The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Real-Time Evaluation of a Large Language Model for Clinical Practice Guideline Development
Abstract
BACKGROUND: The purpose of this study was to evaluate the capability of a large language model (LLM) for performing each of the steps of clinical practice guideline development from framing a healthcare question to creating the evidence-to-decision framework. METHODS: The LLM tool used for this study was OpenAI Generative Pretrained Transformer (GPT)-4o. This evaluation of an LLM was conducted concomitantly with development of a clinical practice guideline on neuromuscular blockade in adults with acute respiratory distress syndrome for the Society of Critical Care Medicine. The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) handbook provided the steps of the process that served as the outline for this evaluation of GPT-4o. Each request for information posed to the LLM was performed during or soon after the same period as the respective step of the process being conducted by the guideline panel. The results follow the major sections of the GRADE process: framing the healthcare question and selecting and rating the importance of outcomes, summarizing the evidence and quality of evidence, and going from evidence to recommendations. RESULTS AND CONCLUSIONS: The LLM demonstrated the most usefulness for the initial step of the guideline development process that involved framing the healthcare question and selecting and rating outcomes. The limitations of the LLM became most apparent during the remaining steps of the development process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The limitations of LLMs in high-stakes critical care tasks were recently demonstrated in a real-time evaluation during Society of Critical Care Medicine guideline development, where GPT-4o showed significant gaps when applied to the full clinical evidence synthesis workflow (34).”
From Hallucination Rate of Peer-Reviewed Citations Generated by Large Language Models in Neurocritical Care (2026) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Accessing Artificial Intelligence for Clinical Decision-Making
Giordano, C., Brennan, M., Mohamed, B., Rashidi, P., Modave, F., & Tighe, P. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.645232 · Cited by 326
The Integration of Artificial Intelligence into Clinical Practice
Karalis, V. D. (2024) · Applied Biosciences · doi:10.3390/applbiosci3010002 · Cited by 255
Ovelman, C., Kugley, S., Gartlehner, G., & Viswanathan, M. (2024) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.12041 · Cited by 18
Research works printed but not linked: 5
- Alowais, 2023, Revolutionizing healthcare: The role of artificial intelligence in clinical practice., BMC Med Educ, 23, 689, 10.1186/s12909-023-04698-z · DOI · Paper page
- Šuster, 2023, Automating quality assessment of medical evidence in systematic reviews: Model development and validation study., J Med Internet Res, 25, e35568, 10.2196/35568 · DOI
- Sousa-Pinto, 2025, Guidelines International Network: Principles for use of artificial intelligence in the health guideline enterprise., Ann Intern Med, 178, 408, 10.7326/ANNALS-24-02338 · DOI
- Erstad, 2026, Society of Critical Care Medicine guidelines for the administration of neuromuscular blockade in adults with acute respiratory distress syndrome., Crit Care Med, 54, 634, 10.1097/CCM.0000000000007002 · DOI
- McNicholas, 2025, Natural language processing in critical care: Opportunities, challenges, and future directions., Intensive Care Med, 51, 585, 10.1007/s00134-024-07776-y · DOI
Cited by
Seifi, A., & Seyfi, A. (2026) · Critical Care Explorations · doi:10.1097/cce.0000000000001474 · Cited by 1
Metadata
- Authors
- Erstad, B. L.
- Affiliations
- Department of Pharmacy Practice and Science, University of Arizona R. Ken Coit College of Pharmacy, Tucson, AZ.
- ORCID
- Erstad, Brian L.
- Venue
- Critical Care Explorations
- Published
- 2026-04-24
- Volume
- 8
- Issue
- 5
- Pages
- e1409
- ISSN
- 2639-8028
- Cited by
- 0
- References
- 3
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Real-Time Evaluation of a Large Language Model for Clinical Practice Guideline Development. https://app.cochat.ai/papers/10.1097/cce.0000000000001409
- The paper
- Erstad, B. L. (2026). Real-Time Evaluation of a Large Language Model for Clinical Practice Guideline Development. Critical Care Explorations. https://doi.org/10.1097/cce.0000000000001409
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.