Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index; 38 of the 51 sources its claims cite were checked for support.

A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation

Open access

Published version of a preprint

Abstract

Integrating large language models into healthcare can enhance workflow efficiency and patient care by automating tasks such as summarising consultations. Their framework evaluated 12,999 clinician-annotated sentences and observed a 1.47% hallucination rate and a 3.45% omission rate after prompt/workflow refinement.

Overview

CoChat Summary

## TL;DR

The paper proposes a clinician-in-the-loop framework, CREOLA, to evaluate hallucinations, omissions, and clinical safety in LLM-generated medical note summaries. Across 18 experiments on 450 primary care transcript–note pairs, the authors report 1.47% hallucinations and 3.45% omissions, and show that prompt/workflow refinements can reduce major errors.

## Key findings

  • Across 18 experiments on 450 consultation transcript–note pairs, 12,999 clinical note sentences were manually evaluated; 191 sentences had hallucinations (1.47%) and 1,712 transcript sentences were omitted (3.45%).
  • Of the hallucinations, 84/191 (44%) were major; of the omissions, 286/1,712 (16.7%) were major.
  • Hallucinations were most common in the Plan section (21%), followed by Assessment (10.5%) and Symptoms (5.2%); the most common hallucination type was fabrication (82/191, 43%).
  • Iterative prompting/workflow changes reduced major hallucinations by 75% (from 4 to 1), major omissions by 58% (from 24 to 10), and minor omissions by 35% (from 114 to 74) when comparing Experiments 3 to 8; in another set of function-call experiments, major omissions fell from 61 to 0 and minor omissions from 130 to 54, while total hallucinations dropped from 4 to 3.

## Methods

  • The study introduces a four-part framework: an error taxonomy, an experimental comparison structure, a clinical safety assessment, and the CREOLA graphical user interface for clinician annotation.
  • Data came from 18 experiments using 25 primary care consultation transcripts each from the PriMock dataset, yielding 450 transcript–note pairs and 49,590 transcript sentences plus 12,999 note sentences for manual review.
  • Two clinicians reviewed each note sentence for hallucinations and transcript sentences for omissions; disagreements were resolved by a senior clinician, and the authors compared prompt/workflow variants using GPT-4 (GPT-4-32k-0613) with seed 210, temperature 0, and top-p 0.95.

## Limitations

  • The sample size of medical transcripts was relatively small, chosen to balance annotation burden against the number of experiments; this is stated in the paper.
  • Only one LLM was evaluated (GPT-4), so the findings may not generalize to other models; this is stated in the paper.
  • The experiments used direct prompting only, not newer methods such as RAG, CoT, or knowledge graphs; this is stated in the paper as a scope limitation.
  • Evident from the text: annotations were manual and relied on clinician reviewers, which the paper itself notes is expensive and unsustainable at scale.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-025-01670-7

Cites

65 of 65 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Our workflow for the assessment of LLM output using CREOLA platform. This diagram illustrates the process we followed using the available dataset for various experiments, including the input from clinicians for labelling and consolidating reviews, followed by a safety analysis.
Fig. 1 Our workflow for the assessment of LLM output using CREOLA platform. This diagram illustrates the process we followed using the available dataset for various experiments, including the input from clinicians for labelling and consolidating reviews, followed by a safety analysis.CC BY-NC-ND · PubMed Central
Fig. 2 Incidence of hallucinations, their types, the section of the note they appear in, and clinical risk. The figure illustrates the occurrence of hallucinations in the generated sentences based on different prompts (a) and their corresponding percentages (b). It also highlights the type and clinical severity of hallucinations (c) and the specific sections of the note where they appeared (d).
Fig. 2 Incidence of hallucinations, their types, the section of the note they appear in, and clinical risk. The figure illustrates the occurrence of hallucinations in the generated sentences based on different prompts (a) and their corresponding percentages (b). It also highlights the type and clinical severity of hallucinations (c) and the specific sections of the note where they appeared (d).CC BY-NC-ND · PubMed Central
Fig. 3 Number of omitted sentences based on different prompts. The figure illustrates the number of omitted sentences using different prompts (a) and their respective percentage (b).
Fig. 3 Number of omitted sentences based on different prompts. The figure illustrates the number of omitted sentences using different prompts (a) and their respective percentage (b).CC BY-NC-ND · PubMed Central
Fig. 4 Severity of risk in major hallucinations. We assessed the clinical risk resulting from major hallucinations based on our suggested framework.
Fig. 4 Severity of risk in major hallucinations. We assessed the clinical risk resulting from major hallucinations based on our suggested framework.CC BY-NC-ND · PubMed Central
Fig. 5 Hallucination risk assessment. We have categorised the clinician severity of hallucination risks based on their type (a) and the section of the clinical notes where they occurred (b).
Fig. 5 Hallucination risk assessment. We have categorised the clinician severity of hallucination risks based on their type (a) and the section of the clinical notes where they occurred (b).CC BY-NC-ND · PubMed Central
Fig. 6 Severity of clinical risk for major omissions and the section of the clinical note where they most occurred. This figure illustrates the clinical risk severity due to major omissions (a) and indicates the sections of the notes where they are most likely to occur (b).
Fig. 6 Severity of clinical risk for major omissions and the section of the clinical note where they most occurred. This figure illustrates the clinical risk severity due to major omissions (a) and indicates the sections of the notes where they are most likely to occur (b).CC BY-NC-ND · PubMed Central
Fig. 7 A comparison of omissions and hallucinations between Experiment 1 and Experiment 8. Comparing omissions (a) and hallucinations (b) using the base prompt from Experiment 1 and the updated style prompt from Experiment 8. Each connected dot represents the change in hallucinations or omissions for a given document.
Fig. 7 A comparison of omissions and hallucinations between Experiment 1 and Experiment 8. Comparing omissions (a) and hallucinations (b) using the base prompt from Experiment 1 and the updated style prompt from Experiment 8. Each connected dot represents the change in hallucinations or omissions for a given document.CC BY-NC-ND · PubMed Central
Fig. 8 Comparison of hallucination and omission counts between two experiments, assessing differences in prompt engineering effect on the quality of outputs. The figure shows the changes in the counts of hallucinations (a) and omissions (b) between Experiments 3 and 8 which have resulted from the changes in the prompts.
Fig. 8 Comparison of hallucination and omission counts between two experiments, assessing differences in prompt engineering effect on the quality of outputs. The figure shows the changes in the counts of hallucinations (a) and omissions (b) between Experiments 3 and 8 which have resulted from the changes in the prompts.CC BY-NC-ND · PubMed Central
Fig. 9 A comparison of omissions and hallucinations between Experiment 1 and Experiment 5. Comparing omissions (a) and hallucinations (b) in our base experiment (Experiment 1) with Experiment 5 where the transcript was broken down into concise facts before being passed to the LLM for note generation. Each connected dot represents the change in hallucinations or omissions for a given document.
Fig. 9 A comparison of omissions and hallucinations between Experiment 1 and Experiment 5. Comparing omissions (a) and hallucinations (b) in our base experiment (Experiment 1) with Experiment 5 where the transcript was broken down into concise facts before being passed to the LLM for note generation. Each connected dot represents the change in hallucinations or omissions for a given document.CC BY-NC-ND · PubMed Central
Fig. 10 Comparison of hallucination and omission counts between two experiments, assessing the efficacy of a data-extraction intermediate step versus a normal note-generation step. The figure compares the number of hallucinations (a) and omissions (b) between Experiment 3, where the standard note generation step was used, and experiment 5, where data extraction was performed before note generation.
Fig. 10 Comparison of hallucination and omission counts between two experiments, assessing the efficacy of a data-extraction intermediate step versus a normal note-generation step. The figure compares the number of hallucinations (a) and omissions (b) between Experiment 3, where the standard note generation step was used, and experiment 5, where data extraction was performed before note generation.CC BY-NC-ND · PubMed Central
Fig. 11 Number of hallucinations and omissions in the function call experiments. This figure illustrates how the change in prompts within the function call experiments can change the number of major and minor hallucinations and omissions in the output.
Fig. 11 Number of hallucinations and omissions in the function call experiments. This figure illustrates how the change in prompts within the function call experiments can change the number of major and minor hallucinations and omissions in the output.CC BY-NC-ND · PubMed Central
Fig. 12 Clinical risk assessment for hallucinations in experiments 8 and 11. The figures illustrate the type and severity of clinical risk resulting from major hallucinations in Experiments 8 (a) and 11 (c) and the section of the notes they occur (b and d).
Fig. 12 Clinical risk assessment for hallucinations in experiments 8 and 11. The figures illustrate the type and severity of clinical risk resulting from major hallucinations in Experiments 8 (a) and 11 (c) and the section of the notes they occur (b and d).CC BY-NC-ND · PubMed Central
Fig. 13 Assessment of clinical risk resulting from major omissions. The figure illustrates the clinical risk from major omissions in Experiment 8 and the section of the clinical notes where it has occurred.
Fig. 13 Assessment of clinical risk resulting from major omissions. The figure illustrates the clinical risk from major omissions in Experiment 8 and the section of the clinical notes where it has occurred.CC BY-NC-ND · PubMed Central
Fig. 14 Risk estimation based on the likelihood and consequence of harm occurrence. This figure illustrates the scoring of clinical risk based on the likelihood of an incident occurring and the severity of harm it may cause.
Fig. 14 Risk estimation based on the likelihood and consequence of harm occurrence. This figure illustrates the scoring of clinical risk based on the likelihood of an incident occurring and the severity of harm it may cause.CC BY-NC-ND · PubMed Central
Fig. 15 The annotation user interface is used to identify hallucinations (a) and omissions (b) and categorise them into major and minor categories. To facilitate clinician review, the closest sentence matches (highlighted in yellow) for each portion of the text under review were extracted from the counterpart document. In the case of hallucinations, portions of text in the note were compared to the consultation (a), whereas for omissions, portions of the transcript were compared against the note (b).
Fig. 15 The annotation user interface is used to identify hallucinations (a) and omissions (b) and categorise them into major and minor categories. To facilitate clinician review, the closest sentence matches (highlighted in yellow) for each portion of the text under review were extracted from the counterpart document. In the case of hallucinations, portions of text in the note were compared to the consultation (a), whereas for omissions, portions of the transcript were compared against the note (b).CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

22 Valid · 5 Not valid · 3 Not checked · 58 Own results and interpretations

34 more claims · 6 with their cited sources checked

Create a free account to see all 88 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 36
  • It may state two separate claims as one. 17

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

8 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 54 references

Research works printed but not linked: 1

  1. Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, 74–81, Barcelona, Spain. Association for Computational Linguistics. https://aclanthology.org/W04-1013/ (2004).

Cited by

See all 423

Metadata

Authors
Asgari, E., Montaña-Brown, N., Dubois, M., Khalil, S., Balloch, J., Yeung, J. A., & Pimenta, D.
Affiliations
Guy's and St Thomas' NHS Foundation Trust (GB)
Corresponding author
Asgari, Elham
Venue
npj Digital Medicine
Published
2025-05-13
Volume
8
Issue
1
Pages
274-274
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4410343193
Cited by
423
References
54
Citations by year
2026: 340 · 2025: 77
FWCI
99.51
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
large language models, hallucinations, clinical safety, clinical note generation, error taxonomy, omission rate, prompt refinement, clinical documentation, patient safety

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. https://app.cochat.ai/papers/10.1038/s41746-025-01670-7
The paper
Asgari, E., Montaña-Brown, N., Dubois, M., Khalil, S., Balloch, J., Yeung, J. A., & Pimenta, D. (2025). A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digital Medicine. https://doi.org/10.1038/s41746-025-01670-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.