Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 24 of the 32 sources its claims cite were checked for support.

A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population

Open access
Urquhart, Emma, Ryan, John, Hartigan, Sean, Nita, Ciprian, Hanley, Ciara, Moran, Peter, Bates, John, Jooste, Rachel, Judge, Conor, Laffey, John G., Madden, Michael G., McNicholas, Bairbre A.·2024·Intensive Care Medicine Experimental

Abstract

BACKGROUND: Artificial intelligence, through improved data management and automated summarisation, has the potential to enhance intensive care unit (ICU) care. Large language models (LLMs) can interrogate and summarise large volumes of medical notes to create succinct discharge summaries. In this study, we aim to investigate the potential of LLMs to accurately and concisely synthesise ICU discharge summaries. METHODS: Anonymised clinical notes from ICU admissions were used to train and validate a prompting structure in three separate LLMs (ChatGPT, GPT-4 API and Llama 2) to generate concise clinical summaries. Summaries were adjudicated by staff intensivists on ability to identify and appropriately order a pre-defined list of important clinical events as well as readability, organisation, succinctness, and overall rank. RESULTS: In the development phase, text from five ICU episodes was used to develop a series of prompts to best capture clinical summaries. In the testing phase, a summary produced by each LLM from an additional six ICU episodes was utilised for evaluation. Overall ability to identify a pre-defined list of important clinical events in the summary was 41.5 ± 15.2% for GPT-4 API, 19.2 ± 20.9% for ChatGPT and 16.5 ± 14.1% for Llama2 (p = 0.002). GPT-4 API followed by ChatGPT had the highest score to appropriately order a pre-defined list of important clinical events in the summary as well as readability, organisation, succinctness, and overall rank, whilst Llama2 scored lowest for all. GPT-4 API produced minor hallucinations, which were not present in the other models. CONCLUSION: Differences exist in large language model performance in readability, organisation, succinctness, and sequencing of clinical events compared to others. All encountered issues with narrative coherence and omitted key clinical data and only moderately captured all clinically meaningful data in the correct order. However, these technologies suggest future potential for creating succinct discharge summaries.

Overview

CoChat Summary

## TL;DR

This pilot feasibility study tested ChatGPT, GPT-4 API, and Llama 2 on anonymised ICU notes from an Irish hospital to see how well they could generate concise discharge summaries. GPT-4 API performed best, but all models missed many clinically important events and still showed problems with narrative coherence and completeness.

## Key findings

  • The study used clinical details from 11 ICU episodes in 9 patients (5 female, 4 male), with notes split into a development phase (5 episodes) and evaluation phase (6 episodes).
  • For recalling a pre-defined list of important clinical events, GPT-4 API scored 41.5 ± 15.2%, compared with 19.2 ± 20.9% for ChatGPT and 16.5 ± 14.1% for Llama 2 (p = 0.002).
  • For appropriate sequencing of important events, GPT-4 API scored 42.9 ± 18.9%, versus 22.1 ± 24.8% for ChatGPT and 17.3 ± 15.8% for Llama 2 (p = 0.009).
  • The paper states that none of the LLMs identified more than 40% of events considered important by intensivists, and GPT-4 API was rated best overall for readability, organisation, succinctness, and rank.

## Methods

  • Pilot feasibility study using anonymised ICU clinical notes from consecutive admissions at Galway University Hospital; patients with capacity were approached and consented, and ethical approval was obtained.
  • Three LLMs were tested: ChatGPT, GPT-4 API, and Llama 2. Notes from 5 episodes were used to develop prompts, and 6 additional episodes were used for evaluation.
  • Summaries were generated with zero-shot prompting and low/zero temperature; clinicians scored outputs against a pre-defined checklist of clinical events for inclusion and placement, plus Likert-scale ratings for readability, organisation, succinctness, and accuracy. Statistical comparisons used the Kruskal–Wallis test and kappa for interrater reliability.

## Limitations

  • The paper’s main stated limitation was small sample size, driven by GDPR-related consent requirements and the difficulty of large-scale patient data processing in Europe.
  • Other stated limitations: only healthcare-provider-generated text was included, laboratory and radiology data were excluded unless mentioned in notes, and GPT-4 API-generated prompts were reused for the other models rather than being independently developed for each model.
  • Evident from the text: the study evaluated summaries against a checklist of events rather than physician-written discharge summaries, so results may not reflect comparison with standard clinical summaries; the authors explicitly note this as a methodological limitation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s40635-024-00656-1

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Comparison of recall and correct sequencing of clinical details by LLMs in summarising intensive care unit clinical notes
Fig. 1 Comparison of recall and correct sequencing of clinical details by LLMs in summarising intensive care unit clinical notesCC BY · PubMed Central
Fig. 2 Comparison of readability, organisation, succinctness, and accuracy of LLMs in summarising intensive care unit clinical notes
Fig. 2 Comparison of readability, organisation, succinctness, and accuracy of LLMs in summarising intensive care unit clinical notesCC BY · PubMed Central

Verification

Claims the paper makes

9 Valid · 0 Not valid · 4 Not checked · 37 Own results and interpretations

12 more claims

Create a free account to see all 50 claims and to verify the 3 references still open.

See all claims — free account
  • Probably not a substantial claim. 17
  • It may state two separate claims as one. 21

Create a free account to see these claims.

Citations

Cites

Show all 19 references

Research works printed but not linked: 2

  1. O PA (2021) Shaking the foundations: delusions in sequence models for interaction and control. arXiv preprint. https://doi.org/10.4855/arXiv.2110.10819 · doi:10.4855/arxiv.2110.10819
  2. Wang J, Shi E, Yu S, Wu Z, Ma C, Dai H, Yang Q, Kang Y, Wu J, Hu H, Yue C (2023) Prompt engineering for healthcare: methodologies and applications. arXiv preprint. https://doi.org/10.4855/arXiv.2304.14670 · doi:10.4855/arxiv.2304.14670

Cited by

See all 18

Metadata

Authors
Urquhart, E., Ryan, J., Hartigan, S., Nita, C., Hanley, C., Moran, P., Bates, J., Jooste, R., Judge, C., Laffey, J. G., Madden, M. G., & McNicholas, B. A.
Affiliations
  • Ollscoil na Gaillimhe – University of Galway (IE)
  • University Hospital Galway (IE)
Corresponding author
McNicholas, Bairbre A.
Venue
Intensive Care Medicine Experimental
Published
2024-08-16
Volume
12
Issue
1
Pages
71-71
ISSN
2197-425X (ISSN-L)
OpenAlex
W4401653419
Cited by
18
References
19
Citations by year
2026: 7 · 2025: 10 · 2024: 1
FWCI
1.69
Type
journal article
Language
en
Open access
Licence
Locations
  • Intensive Care Medicine Experimental (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, ChatGPT, GPT-4, Llama 2, prompt engineering, hallucinations, readability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population. https://app.cochat.ai/papers/10.1186/s40635-024-00656-1
The paper
Urquhart, E., Ryan, J., Hartigan, S., Nita, C., Hanley, C., Moran, P., Bates, J., Jooste, R., Judge, C., Laffey, J. G., Madden, M. G., & McNicholas, B. A. (2024). A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population. Intensive Care Medicine Experimental. https://doi.org/10.1186/s40635-024-00656-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.