The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index; 38 of the 51 sources its claims cite were checked for support.
A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation
Open accessPublished version of a preprint
Abstract
Integrating large language models into healthcare can enhance workflow efficiency and patient care by automating tasks such as summarising consultations. Their framework evaluated 12,999 clinician-annotated sentences and observed a 1.47% hallucination rate and a 3.45% omission rate after prompt/workflow refinement.
Overview
CoChat Summary
## TL;DR
The paper proposes a clinician-in-the-loop framework, CREOLA, to evaluate hallucinations, omissions, and clinical safety in LLM-generated medical note summaries. Across 18 experiments on 450 primary care transcript–note pairs, the authors report 1.47% hallucinations and 3.45% omissions, and show that prompt/workflow refinements can reduce major errors.
## Key findings
- Across 18 experiments on 450 consultation transcript–note pairs, 12,999 clinical note sentences were manually evaluated; 191 sentences had hallucinations (1.47%) and 1,712 transcript sentences were omitted (3.45%).
- Of the hallucinations, 84/191 (44%) were major; of the omissions, 286/1,712 (16.7%) were major.
- Hallucinations were most common in the Plan section (21%), followed by Assessment (10.5%) and Symptoms (5.2%); the most common hallucination type was fabrication (82/191, 43%).
- Iterative prompting/workflow changes reduced major hallucinations by 75% (from 4 to 1), major omissions by 58% (from 24 to 10), and minor omissions by 35% (from 114 to 74) when comparing Experiments 3 to 8; in another set of function-call experiments, major omissions fell from 61 to 0 and minor omissions from 130 to 54, while total hallucinations dropped from 4 to 3.
## Methods
- The study introduces a four-part framework: an error taxonomy, an experimental comparison structure, a clinical safety assessment, and the CREOLA graphical user interface for clinician annotation.
- Data came from 18 experiments using 25 primary care consultation transcripts each from the PriMock dataset, yielding 450 transcript–note pairs and 49,590 transcript sentences plus 12,999 note sentences for manual review.
- Two clinicians reviewed each note sentence for hallucinations and transcript sentences for omissions; disagreements were resolved by a senior clinician, and the authors compared prompt/workflow variants using GPT-4 (GPT-4-32k-0613) with seed 210, temperature 0, and top-p 0.95.
## Limitations
- The sample size of medical transcripts was relatively small, chosen to balance annotation burden against the number of experiments; this is stated in the paper.
- Only one LLM was evaluated (GPT-4), so the findings may not generalize to other models; this is stated in the paper.
- The experiments used direct prompting only, not newer methods such as RAG, CoT, or knowledge graphs; this is stated in the paper as a scope limitation.
- Evident from the text: annotations were manual and relied on clinician reviewers, which the paper itself notes is expensive and unsustainable at scale.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures















Verification
Claims the paper makes
22 Valid · 5 Not valid · 3 Not checked · 58 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“However, the MedHALT benchmark is limited to assessing LLM’s reasoning capabilities over the medical domain in a QA format.”
§ A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation > Introduction
What the sources say
- Disputedcites [43] Pal, A., Umapathi, L. K. & Sankarasubbu, M. Med-HALT: Medical Domain Hallucination Test for Large Language Models. Computer Science: Computation and Language. https://doi.org/10.48550/arXiv.2307.15343 (2023). sourceExperimental — accuracy not yet measured
Section 3.2, “Memory Hallucination Tests,” describes tasks that assess factual recall, including generating a PubMed article link from its abstract or a title from its PMID. The Abstract also says Med-HALT includes both reasoning and memory-based tests. It is therefore not limited to assessing reasoning in a medical QA format. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Errors in clinical documentation generation can lead to inaccurate recording and communication of facts10,11.”
§ A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation > Introduction
What the sources say
- Not checked — no full text of the sourcecites [10] Kripalani, S. et al. Deficits in communication and information transfer between hospital-based and primary care physicians. JAMA 297, 831 (2007). source
- Supportedcites [11] Adane, K., Gizachew, M. & Kendie, S. The role of medical data in efficient patient care delivery: a review. Risk Manag Health. Policy 12, 67–73 (2019). source
The Background describes mistakes in recording or transcribing patient data and says accurate information is needed for clinical communication. The Discussion section ‘Healthcare service at a distance’ also reports errors during manual recording and failures to record data promptly. Together, these passages support the claim as a general statement about documentation errors. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Inaccuracies in the document summarisation task can introduce misleading details8 into transcribed conversations or summaries, potentially delaying diagnoses12 and causing unnecessary patient anxiety.”
§ A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation > Introduction
What the sources say
- Could not verifycites [8] Ahmad, M. A., Yaramis, I. & Roy, T. D. Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AI. Computer Science: Computation and Language. https://doi.org/10.48550/arXiv.2311.01463 (2023). sourceExperimental — accuracy not yet measured
Citation 12 is attached to the possibility of delayed diagnoses. The Conclusion says LLMs can generate misleading information and assist with diagnosis, but neither it nor the passages retrieved for delayed diagnosis link inaccurate summaries to diagnostic delays. The copy does not address that specific consequence. (LLM, conf=0.74)
- Not checked — no full text of the sourcecites [12] Schiff, G. D. Diagnostic error in medicine. Arch. Intern Med. 169, 1881 (2009). source
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, these relevant clinical categorisations have not assessed the implications of the mistakes for downstream harm.”
§ A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation > Introduction
What the sources say
- Not checked — no full text of the sourcecites [26] Tierney, A. A. et al. Ambient Artificial Intelligence Scribes to Alleviate the Burden of Clinical Documentation. NEJM Catal. 5 (2024). source
- Partly supportedcites [23] Abacha, A. Ben, Yim, W., Michalopoulos, G. & Lin, T. An Investigation of Evaluation Metrics for Automated Medical Note Generation. Computer Science: Computation and Language. https://doi.org/10.48550/arXiv.2305.17364 (2023). sourceExperimental — accuracy not yet measured
Section 3.1 (page unavailable) defines this paper’s error categories by whether facts are omitted, hallucinated, correct, or incorrect; Section 3 describes evaluating metrics against those annotations. These passages support the narrower observation that this paper does not assess downstream harm, but they do not establish that existing clinical categorisations generally have not done so. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, they did not assess the model’s impact or human errors on patient safety as part of their study.”
§ A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation > Introduction
What the sources say
- Partly supportedcites [21] Moramarco, F. et al. Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) 5739–5754 (Association for Computational Linguistics, Stroudsburg, PA, USA, 2022). https://doi.org/10.18653/v1/2022.acl-long.394. sourceExperimental — accuracy not yet measured
The Abstract describes an evaluation intended to help understand effects on patients’ clinical safety, and the evaluation procedure tags errors as critical when they have essential clinical importance. Section 5.2, “Human error,” also investigates errors in clinician-written notes. The copy does not report measured patient-safety outcomes, but the claim overlooks its assessment of safety-relevant model and human errors. (LLM, conf=0.80)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
34 more claims · 6 with their cited sources checked
Create a free account to see all 88 claims and every source verdict.
- Probably not a substantial claim. 36
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Nevertheless, these benchmarking efforts, like earlier studies, do not quantify the downstream impact on clinical workflows.”
From Implementation of large language models in electronic health records (2025) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Severity is classified on the 0-5 ordinal scale of Asgari et al. (2025), mapped to AHRQ Common Format harm categories (Agency for Healthcare Research and Quality, 2017).”
From Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework (2026) · cites this paper as Asgari et al., 2025
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Models fine-tuned to resist adversarial or paraphrased inputs (e.g., via MedFuzz-style perturbations) demonstrate lower hallucination rates, since they rely less on spurious patterns (Asgari et al., 2025).”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Asgari et al., 2025
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This inconsistency reflects a known and well-documented limitation of LLMs: their tendency to generate fluent, plausible, and contextually appropriate outputs that nonetheless do not correspond to information explicitly present in source materials.”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Despite comparable performance with clinician-written summaries (Veen et al., 2024), hallucinations manifest as fabricated or distorted clinical statements that are hard to detect (Asgari et al., 2025; Kim et al., 2025).”
From Hallucination Detection-Guided Preference Optimization for Clinical Summarization (2026) · cites this paper as Asgari et al., 2025
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“The occurrence of hallucinations has previously been attributed to the data qualityduring model training 14,15 , the type of model training methodology 16 and prompting strategies 17 .”
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Research works printed but not linked: 1
- Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, 74–81, Barcelona, Spain. Association for Computational Linguistics. https://aclanthology.org/W04-1013/ (2004).
Cited by
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
“…assessment framework currently exists, a step toward addressing this gap is the risk matrix-based evaluation framework introduced by Asgari et al [81], which provides a structured methodology for categorizing and quantifying errors, enabling a systematic assessment of their impact on patient…”
Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review
Neha, F., Bhati, D., & Shukla, D. K. (2025) · AI · doi:10.3390/ai6090226 · Cited by 117
“By anchoring answers in peer-reviewed literature or structured medical knowledge, these systems reduce the hallucinations often seen in standalone LLMs [64].”
Beyond human ears: navigating the uncharted risks of AI scribes in clinical practice
Topaz, M., Peltonen, L., & Zhang, Z. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01895-6 · Cited by 103
Prompt Engineering in Clinical Practice: Tutorial for Clinicians
Liu, J., Liu, F., Wang, C., & Liu, S. (2025) · Journal of Medical Internet Research · doi:10.2196/72644 · Cited by 76
Moëll, B., & Aronsson, F. S. (2025) · Journal of Medical Internet Research · doi:10.2196/75849 · Cited by 70
Metadata
- Authors
- Asgari, E., Montaña-Brown, N., Dubois, M., Khalil, S., Balloch, J., Yeung, J. A., & Pimenta, D.
- Affiliations
- Guy's and St Thomas' NHS Foundation Trust (GB)
- Corresponding author
- Asgari, Elham
- Venue
- npj Digital Medicine
- Published
- 2025-05-13
- Volume
- 8
- Issue
- 1
- Pages
- 274-274
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC12075489
- OpenAlex
- W4410343193
- Cited by
- 423
- References
- 54
- Citations by year
- 2026: 340 · 2025: 77
- FWCI
- 99.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by-nc-nd)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- large language models, hallucinations, clinical safety, clinical note generation, error taxonomy, omission rate, prompt refinement, clinical documentation, patient safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. https://app.cochat.ai/papers/10.1038/s41746-025-01670-7
- The paper
- Asgari, E., Montaña-Brown, N., Dubois, M., Khalil, S., Balloch, J., Yeung, J. A., & Pimenta, D. (2025). A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digital Medicine. https://doi.org/10.1038/s41746-025-01670-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.