The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 24 of the 32 sources its claims cite were checked for support.
A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population
Open accessAbstract
BACKGROUND: Artificial intelligence, through improved data management and automated summarisation, has the potential to enhance intensive care unit (ICU) care. Large language models (LLMs) can interrogate and summarise large volumes of medical notes to create succinct discharge summaries. In this study, we aim to investigate the potential of LLMs to accurately and concisely synthesise ICU discharge summaries. METHODS: Anonymised clinical notes from ICU admissions were used to train and validate a prompting structure in three separate LLMs (ChatGPT, GPT-4 API and Llama 2) to generate concise clinical summaries. Summaries were adjudicated by staff intensivists on ability to identify and appropriately order a pre-defined list of important clinical events as well as readability, organisation, succinctness, and overall rank. RESULTS: In the development phase, text from five ICU episodes was used to develop a series of prompts to best capture clinical summaries. In the testing phase, a summary produced by each LLM from an additional six ICU episodes was utilised for evaluation. Overall ability to identify a pre-defined list of important clinical events in the summary was 41.5 ± 15.2% for GPT-4 API, 19.2 ± 20.9% for ChatGPT and 16.5 ± 14.1% for Llama2 (p = 0.002). GPT-4 API followed by ChatGPT had the highest score to appropriately order a pre-defined list of important clinical events in the summary as well as readability, organisation, succinctness, and overall rank, whilst Llama2 scored lowest for all. GPT-4 API produced minor hallucinations, which were not present in the other models. CONCLUSION: Differences exist in large language model performance in readability, organisation, succinctness, and sequencing of clinical events compared to others. All encountered issues with narrative coherence and omitted key clinical data and only moderately captured all clinically meaningful data in the correct order. However, these technologies suggest future potential for creating succinct discharge summaries.
Overview
CoChat Summary
## TL;DR
This pilot feasibility study tested ChatGPT, GPT-4 API, and Llama 2 on anonymised ICU notes from an Irish hospital to see how well they could generate concise discharge summaries. GPT-4 API performed best, but all models missed many clinically important events and still showed problems with narrative coherence and completeness.
## Key findings
- The study used clinical details from 11 ICU episodes in 9 patients (5 female, 4 male), with notes split into a development phase (5 episodes) and evaluation phase (6 episodes).
- For recalling a pre-defined list of important clinical events, GPT-4 API scored 41.5 ± 15.2%, compared with 19.2 ± 20.9% for ChatGPT and 16.5 ± 14.1% for Llama 2 (p = 0.002).
- For appropriate sequencing of important events, GPT-4 API scored 42.9 ± 18.9%, versus 22.1 ± 24.8% for ChatGPT and 17.3 ± 15.8% for Llama 2 (p = 0.009).
- The paper states that none of the LLMs identified more than 40% of events considered important by intensivists, and GPT-4 API was rated best overall for readability, organisation, succinctness, and rank.
## Methods
- Pilot feasibility study using anonymised ICU clinical notes from consecutive admissions at Galway University Hospital; patients with capacity were approached and consented, and ethical approval was obtained.
- Three LLMs were tested: ChatGPT, GPT-4 API, and Llama 2. Notes from 5 episodes were used to develop prompts, and 6 additional episodes were used for evaluation.
- Summaries were generated with zero-shot prompting and low/zero temperature; clinicians scored outputs against a pre-defined checklist of clinical events for inclusion and placement, plus Likert-scale ratings for readability, organisation, succinctness, and accuracy. Statistical comparisons used the Kruskal–Wallis test and kappa for interrater reliability.
## Limitations
- The paper’s main stated limitation was small sample size, driven by GDPR-related consent requirements and the difficulty of large-scale patient data processing in Europe.
- Other stated limitations: only healthcare-provider-generated text was included, laboratory and radiology data were excluded unless mentioned in notes, and GPT-4 API-generated prompts were reused for the other models rather than being independently developed for each model.
- Evident from the text: the study evaluated summaries against a checklist of events rather than physician-written discharge summaries, so results may not reflect comparison with standard clinical summaries; the authors explicitly note this as a methodological limitation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
9 Valid · 0 Not valid · 4 Not checked · 37 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models (LLM) such as ChatGPT, GPT-4 API and Llama2 can interrogate and summarise large volumes of medical notes to create succinct summaries of radiology reports, internal medical progress notes and patient dialogue [5–8].”
§ A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population > Background
What the sources say
- Supportedcites [5] Van Veen D, Van Uden C, Blankemeier L et al (2024) Adapted large language models can outperform medical experts in clinical text summarization. Nat Med 30(4):1134–1142 source
The Methods (LLMs) identify GPT-3.5, GPT-4 and Llama-2 among the models studied. The Methods (Data) describe summarizing radiology reports, internal-medicine progress notes and doctor–patient dialogue; the Results and Discussion report successful clinical text summarization. Page numbers are not available in our copy. (LLM, conf=0.88)
- Partly supportedcites [6] Guevara M, Chen S, Thomas S et al (2024) Large language models to identify social determinants of health in electronic health records. NPJ Digit Med 7(1):6 sourceExperimental — accuracy not yet measured
The Methods (ChatGPT-family model evaluation) and Discussion describe using ChatGPT and GPT-4 to classify clinical sentences and explore extraction of social determinants of health from EHR text. They do not establish that these models produce succinct summaries of radiology reports, progress notes or patient dialogue, and the cited evaluation does not include Llama2. (LLM, conf=0.84)
- Partly supportedcites [7] Schwartz IS, Link KE, Daneshjou R, Cortes-Penfield N (2023) Black box warning: large language models and the future of infectious diseases consultation. Clin Infect Dis. https://doi.org/10.1093/cid/ciad633 sourceExperimental — accuracy not yet measured
The ‘Use Cases for Large Language Models in Healthcare’ section says LLMs might synthesize text from electronic medical records into discharge summaries. It does not establish that ChatGPT, GPT-4 API, and Llama2 can each summarize large volumes of notes, or specifically radiology reports, progress notes, and patient dialogue. (LLM, conf=0.85)
- Not checked — no full text of the sourcecites [8] Patel SB, Lam K (2023) ChatGPT: the future of discharge summaries? Lancet Digit Health 5(3):e107–e108 source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Most LLM studies have depended on human lead semi-quantitative assessment of coherence, comprehensiveness, harmfulness and factual inconsistencies as well as comparing LLM with human based summaries [5, 13, 14].”
§ A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population > Discussion
What the sources say
- Partly supportedcites [5] Van Veen D, Van Uden C, Blankemeier L et al (2024) Adapted large language models can outperform medical experts in clinical text summarization. Nat Med 30(4):1134–1142 sourceExperimental — accuracy not yet measured
The Methods (Reader study; page unavailable) describe physician ratings of completeness and correctness in blinded comparisons of model and medical-expert summaries; a supplemental study assessed possible harm. This supports the paper as an example of human evaluation, but not the claim that most LLM studies depend on it or that this study assessed coherence. The Discussion (page unavailable) instead says many previous clinical NLP studies relied primarily on quantitative metrics. (LLM, conf=0.87)
- Partly supportedcites [13] Tang L, Sun Z, Idnay B et al (2023) Evaluating large language models on medical evidence summarization. NPJ Digit Med 6(1):158 sourceExperimental — accuracy not yet measured
The Results (Human evaluation) describe human ratings of coherence, factual consistency, comprehensiveness, and harmfulness. The Discussion also compares LLM-generated with human-generated summaries. This paper supports the described evaluation approach, but it does not establish that most LLM studies have depended on it. (LLM, conf=0.82)
- Partly supportedcites [14] Peng C, Yang X, Chen A et al (2023) A study of generative large language model for medical research and healthcare. NPJ Digit Med 6(1):210 sourceExperimental — accuracy not yet measured
The Methods (Turing test of text generation for healthcare settings; page unavailable) describe two physicians rating GatorTronGPT and human-written clinical paragraphs on 1–9 scales for readability and clinical relevance and consistency. This supports human-led semi-quantitative assessment and comparison with human text, but the paper does not establish that most LLM studies use this approach, assess harmfulness, or compare LLM output with human-written summaries. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Many of the studies on LLM have been conducted using large anonymised US datasets [5–7].”
§ A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population > Background
What the sources say
- Supportedcites [5] Van Veen D, Van Uden C, Blankemeier L et al (2024) Adapted large language models can outperform medical experts in clinical text summarization. Nat Med 30(4):1134–1142 source
The Methods > Data section describes this LLM study’s use of large US clinical datasets, including 128,000 MIMIC-CXR reports and 67,000 MIMIC-III reports. Methods > Ethics approval states that the study used only retrospective, de-identified data. This paper supports the claim as one example among the studies cited, rather than establishing how common the practice is across all LLM studies. (LLM, conf=0.88)
- Partly supportedcites [6] Guevara M, Chen S, Thomas S et al (2024) Large language models to identify social determinants of health in electronic health records. NPJ Digit Med 7(1):6 sourceExperimental — accuracy not yet measured
Methods > Data (page not available) confirms that this LLM study used clinical-note datasets from three Boston-area US hospitals, including MIMIC-III. It describes 800 notes in the primary dataset and 200 in each test dataset, but does not establish that these were large anonymised datasets. One study also cannot establish the claim that many LLM studies used such datasets. (LLM, conf=0.83)
- Could not verifycites [7] Schwartz IS, Link KE, Daneshjou R, Cortes-Penfield N (2023) Black box warning: large language models and the future of infectious diseases consultation. Clin Infect Dis. https://doi.org/10.1093/cid/ciad633 sourceExperimental — accuracy not yet measured
The 'How Large Language Models Work' section says LLMs are trained on large quantities of text, but does not say that many LLM studies used large anonymised US datasets. Searches for that phrase and for anonymized US patient datasets found no passage addressing the claim. (LLM, conf=0.74)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Appropriate sequencing of facts, an indicator of the LLMs ability to appropriately rank clinically significant events was highest for GPT-4 API (42.9 ± 18.9%) compared to 22.1 ± 24.8% for ChatGPT and 17.3 ± 15.8% for Llama (p = 0.009) (Fig. 1, Table 2).”
§ A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population > Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“No hallucinations were identified in our analyses of the outputs of ChatGPT or Llama 2.”
§ A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
12 more claims
Create a free account to see all 50 claims and to verify the 3 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 21
Citations
Cites
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 1×
“AI-based algorithms, capable of predicting deteriorating patient outcomes and mortality using extensive datasets, have gained traction [3, 4].”
BackgroundAdapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891
ChatGPT: the future of discharge summaries?
Patel, S. B., & Lam, K. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00021-3 · Cited by 728 · cited 1×
“Large language models (LLM) such as ChatGPT, GPT-4 API and Llama2 can interrogate and summarise large volumes of medical notes to create succinct summaries of radiology reports, internal medical progress notes and patient dialogue [5–8].”
MethodPrompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/50638 · Cited by 709 · cited 1×
“Recent literature has investigated the application of prompt engineering to the healthcare domain, as a means of exploiting the potential of LLMs to extract information from large volumes of medical data [6, 21–23].”
BackgroundZack, T., Lehman, E., Suzgun, M., Rodriguez, J. A., Celi, L. A., Gichoya, J., Jurafsky, D., Szolovits, P., Bates, D. W., Abdulnour, R.-E. E., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00225-x · Cited by 510 · cited 2×
“Recent literature has investigated the application of prompt engineering to the healthcare domain, as a means of exploiting the potential of LLMs to extract information from large volumes of medical data [6, 21–23].”
BackgroundEvaluating large language models on medical evidence summarization
Tang, L., Sun, Z., Idnay, B., Nestor, J. G., Soroush, A., Elias, P. A., Xu, Z., Ding, Y., Durrett, G., Rousseau, J. F., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00896-7 · Cited by 423
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358
Large language models to identify social determinants of health in electronic health records
Guevara, M., Chen, S., Thomas, S., Chaunzwa, T. L., Franco, I., Kann, B. H., Moningi, S., Qian, J. M., Goldstein, M., Harper, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00970-0 · Cited by 278 · cited 2×
“Recent literature has investigated the application of prompt engineering to the healthcare domain, as a means of exploiting the potential of LLMs to extract information from large volumes of medical data [6, 21–23].”
BackgroundMethodUse of a Large Language Model to Assess Clinical Acuity of Adults in the Emergency Department
Williams, C. Y. K., Zack, T., Miao, B. Y., Sushil, M., Wang, M., Kornblith, A. E., & Butte, A. J. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.8895 · Cited by 148
Research works printed but not linked: 2
- O PA (2021) Shaking the foundations: delusions in sequence models for interaction and control. arXiv preprint. https://doi.org/10.4855/arXiv.2110.10819 · doi:10.4855/arxiv.2110.10819
- Wang J, Shi E, Yu S, Wu Z, Ma C, Dai H, Yang Q, Kang Y, Wu J, Hu H, Yue C (2023) Prompt engineering for healthcare: methodologies and applications. arXiv preprint. https://doi.org/10.4855/arXiv.2304.14670 · doi:10.4855/arxiv.2304.14670
Cited by
The use of large language models in clinical documentation: A scoping review
Woo, B., Cato, K., Cho, H., You, S. B., & Song, J. (2025) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2025.105322 · Cited by 40
Large Language Models in Critical Care Medicine: Scoping Review
Shi, T., Ma, J., Yu, Z., Xu, H., Yang, R., Xiong, M., Xiao, M., Li, Y., Zhao, H., & Kong, G. (2025) · JMIR Medical Informatics · doi:10.2196/76326 · Cited by 21
“Therefore, improving the interpretability and transparency of LLMs remains a critical challenge in their integration into clinical workflows [123,124].”
Primer on large language models: an educational overview for intensivists
Idan, D., & Einav, S. (2025) · Critical Care · doi:10.1186/s13054-025-05479-4 · Cited by 21
“A pilot feasibility study also demonstrated the ability of LLMs to generate concise summaries of ICU admissions for discharge documentation [18].”
Natural language processing in critical care: opportunities, challenges, and future directions
McNicholas, B., Madden, M. G., & Laffey, J. G. (2025) · Intensive Care Medicine · doi:10.1007/s00134-024-07776-y · Cited by 11
“…and interpret machine-generated insights, with the goal of guiding clinical decision-making for the critically ill. Adoption and safe use of NLP, including LLMs, will require rigor and governmental regulation to ensure the information we glean from unstructured data is used to its best effect [2].”
Self-citationClinical applications of large language models in knee osteoarthritis: a systematic review
Ma, Z., Liu, Y., Zhang, Z., Chen, R., Fan, H., Cao, X., & Ni, L. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1670824 · Cited by 11
Metadata
- Authors
- Urquhart, E., Ryan, J., Hartigan, S., Nita, C., Hanley, C., Moran, P., Bates, J., Jooste, R., Judge, C., Laffey, J. G., Madden, M. G., & McNicholas, B. A.
- Affiliations
- Ollscoil na Gaillimhe – University of Galway (IE)
- University Hospital Galway (IE)
- Corresponding author
- McNicholas, Bairbre A.
- ORCID
- Venue
- Intensive Care Medicine Experimental
- Published
- 2024-08-16
- Volume
- 12
- Issue
- 1
- Pages
- 71-71
- ISSN
- 2197-425X (ISSN-L)
- PMC
- PMC11327225
- OpenAlex
- W4401653419
- Cited by
- 18
- References
- 19
- Citations by year
- 2026: 7 · 2025: 10 · 2024: 1
- FWCI
- 1.69
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Intensive Care Medicine Experimental (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, ChatGPT, GPT-4, Llama 2, prompt engineering, hallucinations, readability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population. https://app.cochat.ai/papers/10.1186/s40635-024-00656-1
- The paper
- Urquhart, E., Ryan, J., Hartigan, S., Nita, C., Hanley, C., Moran, P., Bates, J., Jooste, R., Judge, C., Laffey, J. G., Madden, M. G., & McNicholas, B. A. (2024). A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population. Intensive Care Medicine Experimental. https://doi.org/10.1186/s40635-024-00656-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.