Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 1

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Real-Time Evaluation of a Large Language Model for Clinical Practice Guideline Development

Erstad, Brian L. · 2026 · Critical Care Explorations

Abstract

BACKGROUND: The purpose of this study was to evaluate the capability of a large language model (LLM) for performing each of the steps of clinical practice guideline development from framing a healthcare question to creating the evidence-to-decision framework. METHODS: The LLM tool used for this study was OpenAI Generative Pretrained Transformer (GPT)-4o. This evaluation of an LLM was conducted concomitantly with development of a clinical practice guideline on neuromuscular blockade in adults with acute respiratory distress syndrome for the Society of Critical Care Medicine. The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) handbook provided the steps of the process that served as the outline for this evaluation of GPT-4o. Each request for information posed to the LLM was performed during or soon after the same period as the respective step of the process being conducted by the guideline panel. The results follow the major sections of the GRADE process: framing the healthcare question and selecting and rating the importance of outcomes, summarizing the evidence and quality of evidence, and going from evidence to recommendations. RESULTS AND CONCLUSIONS: The LLM demonstrated the most usefulness for the initial step of the guideline development process that involved framing the healthcare question and selecting and rating outcomes. The limitations of the LLM became most apparent during the remaining steps of the development process.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1097/cce.0000000000001409

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 3 references

Research works printed but not linked: 5

  1. Alowais, 2023, Revolutionizing healthcare: The role of artificial intelligence in clinical practice., BMC Med Educ, 23, 689, 10.1186/s12909-023-04698-z · DOI · Paper page
  2. Šuster, 2023, Automating quality assessment of medical evidence in systematic reviews: Model development and validation study., J Med Internet Res, 25, e35568, 10.2196/35568 · DOI
  3. Sousa-Pinto, 2025, Guidelines International Network: Principles for use of artificial intelligence in the health guideline enterprise., Ann Intern Med, 178, 408, 10.7326/ANNALS-24-02338 · DOI
  4. Erstad, 2026, Society of Critical Care Medicine guidelines for the administration of neuromuscular blockade in adults with acute respiratory distress syndrome., Crit Care Med, 54, 634, 10.1097/CCM.0000000000007002 · DOI
  5. McNicholas, 2025, Natural language processing in critical care: Opportunities, challenges, and future directions., Intensive Care Med, 51, 585, 10.1007/s00134-024-07776-y · DOI

Metadata

Authors
Erstad, B. L.
Affiliations
Department of Pharmacy Practice and Science, University of Arizona R. Ken Coit College of Pharmacy, Tucson, AZ.
Venue
Critical Care Explorations
Published
2026-04-24
Volume
8
Issue
5
Pages
e1409
ISSN
2639-8028
Cited by
0
References
3
Language
en

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Real-Time Evaluation of a Large Language Model for Clinical Practice Guideline Development. https://app.cochat.ai/papers/10.1097/cce.0000000000001409
The paper
Erstad, B. L. (2026). Real-Time Evaluation of a Large Language Model for Clinical Practice Guideline Development. Critical Care Explorations. https://doi.org/10.1097/cce.0000000000001409

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.