CoChat
Sign Up
PreprintCited by 892Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareTopic Modeling

The DOI matches its Crossref record; 93 of the 93 references it cites were found in a registry or an index.

Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts

Open access
Veen, Dave Van, Uden, Cara Van, Blankemeier, Louis, Delbrouck, Jean-Benoit, Aali, Asad, Bluethgen, Christian, Pareek, Anuj, Polacin, Malgorzata, Reis, Eduardo Pontes, Seehofnerova, Anna, Rohatgi, Nidhi, Hosamani, Poonam, Collins, William, Ahuja, Neera, Langlotz, Curtis, Hom, Jason, Gatidis, Sergios, Pauly, John, Chaudhari, Akshay·2023·Nature Medicine
DOI 10.21203/rs.3.rs-3483777/v1PMC PMC10635391

Preprint — published as 10.1038/s41591-024-02855-5 · Paper page

Open this paper in CoChat — free accountOpen PDF

Abstract

Sifting through vast textual data and summarizing key information from electronic health records (EHR) imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown immense promise in natural language processing (NLP) tasks, their efficacy on a diverse range of clinical summarization tasks has not yet been rigorously demonstrated. In this work, we apply domain adaptation methods to eight LLMs, spanning six datasets and four distinct clinical summarization tasks: radiology reports, patient questions, progress notes, and doctor-patient dialogue. Our thorough quantitative assessment reveals trade-offs between models and adaptation methods in addition to instances where recent advances in LLMs may not improve results. Further, in a clinical reader study with ten physicians, we show that summaries from our best-adapted LLMs are preferable to human summaries in terms of completeness and correctness. Our ensuing qualitative analysis highlights challenges faced by both LLMs and human experts. Lastly, we correlate traditional quantitative NLP metrics with reader study scores to enhance our understanding of how these metrics align with physician preferences. Our research marks the first evidence of LLMs outperforming human experts in clinical text summarization across multiple tasks. This implies that integrating LLMs into clinical workflows could alleviate documentation burden, empowering clinicians to focus more on personalized patient care and the inherently human aspects of medicine.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.21203/rs.3.rs-3483777/v1

Cites

93 of 93 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure A1 | Comparing Llama-2 (7B) vs. Llama-2 (13B). The dashed line denotes equivalence, and each data point corresponds to the average score of s = 250 samples for a given experimental configuration, i.e. {dataset × m in-context examples}.
Figure A1 | Comparing Llama-2 (7B) vs. Llama-2 (13B). The dashed line denotes equivalence, and each data point corresponds to the average score of s = 250 samples for a given experimental configuration, i.e. {dataset × m in-context examples}.CC BY · PubMed Central
Figure A2 | Summarization performance comparing one in-context example (ICL) vs. QLoRA across all open-source models on patient health questions. Figure 3b contains similar results with the Open-i radiology report dataset.
Figure A2 | Summarization performance comparing one in-context example (ICL) vs. QLoRA across all open-source models on patient health questions. Figure 3b contains similar results with the Open-i radiology report dataset.CC BY · PubMed Central
Figure A3 | Metric scores vs. number of in-context examples across models and datasets. We also include the best model fine-tuned with QLoRA (FLAN-T5) as a horizontal dashed line.
Figure A3 | Metric scores vs. number of in-context examples across models and datasets. We also include the best model fine-tuned with QLoRA (FLAN-T5) as a horizontal dashed line.CC BY · PubMed Central
Figure A4 | Annotation of two patient health question examples from the reader study. The table (lower left) contains reader scores for these two examples and the task average across all samples.
Figure A4 | Annotation of two patient health question examples from the reader study. The table (lower left) contains reader scores for these two examples and the task average across all samples.CC BY · PubMed Central
Figure A5 | Annotation of a progress notes summarization example evaluated in the reader study. The table (lower right) contains reader scores for this example and the task average across all samples.
Figure A5 | Annotation of a progress notes summarization example evaluated in the reader study. The table (lower right) contains reader scores for this example and the task average across all samples.CC BY · PubMed Central
Figure A6 | Annotation of a progress notes summarization example evaluated in the reader study. The table (lower right) contains reader scores for this example and the task average across all samples.
Figure A6 | Annotation of a progress notes summarization example evaluated in the reader study. The table (lower right) contains reader scores for this example and the task average across all samples.CC BY · PubMed Central
Figure A7 | Example of the doctor-patient dialogue summarization task, including “assessment and plan” sections generated by both a human expert and GPT-4.
Figure A7 | Example of the doctor-patient dialogue summarization task, including “assessment and plan” sections generated by both a human expert and GPT-4.CC BY · PubMed Central
Figure 1 | Overview. First we quantitatively evaluate each valid combination (×) of LLM and adaptation method across four distinct summarization tasks comprising six datasets. We then conduct a clinical reader study in which ten physicians compare summaries of the best model/method against those of a human expert.
Figure 1 | Overview. First we quantitatively evaluate each valid combination (×) of LLM and adaptation method across four distinct summarization tasks comprising six datasets. We then conduct a clinical reader study in which ten physicians compare summaries of the best model/method against those of a human expert.CC BY · PubMed Central
Figure 2 | Prompt anatomy. Each summarization task uses a slightly different instruction, as depicted in Table A1.
Figure 2 | Prompt anatomy. Each summarization task uses a slightly different instruction, as depicted in Table A1.CC BY · PubMed Central
Figure 3 | Quantitative results. (a) Alpaca vs. Med-Alpaca. Each data point corresponds to one experimental configuration, and the dashed lines denote equal performance. (b) One in-context example (ICL) vs. QLoRA methods across all open-source models on the Open-i radiology report dataset. (c) MEDCON scores vs. number of in-context examples across models and datasets. We also include the best model fine-tuned with QLoRA as a horizontal dashed line for valid datasets. See Figure A3 for results across all four metrics.(d) Model win rate: a head-to-head winning percentage of each model combination, where red/blue intensities highlight the degree to which models on the vertical axis outperform models on the horizontal axis.
Figure 3 | Quantitative results. (a) Alpaca vs. Med-Alpaca. Each data point corresponds to one experimental configuration, and the dashed lines denote equal performance. (b) One in-context example (ICL) vs. QLoRA methods across all open-source models on the Open-i radiology report dataset. (c) MEDCON scores vs. number of in-context examples across models and datasets. We also include the best model fine-tuned with QLoRA as a horizontal dashed line for valid datasets. See Figure A3 for results across all four metrics.(d) Model win rate: a head-to-head winning percentage of each model combination, where red/blue intensities highlight the degree to which models on the vertical axis outperform models on the horizontal axis.CC BY · PubMed Central
Figure 4 | Clinical reader study. (a) Study design comparing the summarization of GPT-4 vs. that of human experts on three attributes: completeness, correctness, and conciseness. (b) Results. Highlight colors correspond to a value’s location on the color spectrum. Asterisks denote statistical significance by Wilcoxon signed-rank test, *p-value < 0.001. (c) Reader study user interface. (d) Distribution of reader scores for each summarization task across attributes. Horizontal axes denote reader preference as measured by a five-point Likert scale. Vertical axes denote frequency count, with 1,500 total reports for each plot.
Figure 4 | Clinical reader study. (a) Study design comparing the summarization of GPT-4 vs. that of human experts on three attributes: completeness, correctness, and conciseness. (b) Results. Highlight colors correspond to a value’s location on the color spectrum. Asterisks denote statistical significance by Wilcoxon signed-rank test, *p-value < 0.001. (c) Reader study user interface. (d) Distribution of reader scores for each summarization task across attributes. Horizontal axes denote reader preference as measured by a five-point Likert scale. Vertical axes denote frequency count, with 1,500 total reports for each plot.CC BY · PubMed Central
Figure 5 | Annotation of two radiologist report examples from the reader study. The table (lower left) contains reader scores for these two examples and the task average across all samples.
Figure 5 | Annotation of two radiologist report examples from the reader study. The table (lower left) contains reader scores for these two examples and the task average across all samples.CC BY · PubMed Central
Figure 6 | Spearman correlation coefficients between NLP metrics and reader preference assessing completeness, correctness, and conciseness.
Figure 6 | Spearman correlation coefficients between NLP metrics and reader preference assessing completeness, correctness, and conciseness.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 71 references

Research works printed but not linked: 5

  1. [15]Chung H. W., Hou L., Longpre S., Zoph B., Tay Y., Fedus W., Li E., Wang X., Dehghani M., Brahma S., Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022). · arXiv
  2. [18]Demner-Fushman D., Ananiadou S. & Cohen K. B. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks in The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks (2023).
  3. [27]Gao Y., Miller T., Afshar M. & Dligach D. BioNLP Workshop 2023 Shared Task 1A: Problem List Summarization in Proceedings of the 22nd Workshop on Biomedical Language Processing (2023).
  4. [61]Soldaini L. & Goharian N. Quickumls: a fast, unsupervised approach for medical concept extraction in MedIR workshop, sigir (2016), 1– 4.
  5. [65]Tay Y., Dehghani M., Tran V. Q., Garcia X., Wei J., Wang X., Chung H. W., Bahri D., Schuster T., Zheng S., Ul2: Unifying language learning paradigms in The Eleventh International Conference on Learning Representations (2022).

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Veen, D. V., Uden, C. V., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerova, A., Rohatgi, N., Hosamani, P., Collins, W., Ahuja, N., Langlotz, C., Hom, J., Gatidis, S., Pauly, J., & Chaudhari, A.
Affiliations
  • Stanford University (US)
  • The University of Texas at Austin (US)
  • University Hospital Zurich
  • Copenhagen University Hospital (DK)
  • University of Zurich (CH)
  • Hospital Israelita Albert Einstein (BR)
  • Stanford University
  • Stanford Medicine (US)
Corresponding author
Veen, Dave Van
ORCID
Venue
Nature Medicine
Publisher
Springer Science and Business Media LLC
Published
2023-10-30
Volume
30
Issue
4
Pages
1134-1142
ISSN
1078-8956 (ISSN-L)
DOI
10.21203/rs.3.rs-3483777/v1
PMC
PMC10635391
OpenAlex
W4392193048
Cited by
892
References
71
Citations by year
2026: 342 · 2025: 404 · 2024: 144 · 2023: 2
FWCI
81.57
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion)
  • Research Square (acceptedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • PubMed Central (submittedVersion, cc-by)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
  • Zurich Open Repository and Archive (University of Zurich)
Funders
  • Foundation for the National Institutes of Health (R01 AR077604, R01EB002524, P41 EB027060, R01 AR079431)
  • Gordon and Betty Moore Foundation
  • National Institutes of Health (75N92020C00021, 75N92020C00008)
  • Agency for Healthcare Research and Quality (R18 HS026886)
  • National Heart, Lung, and Blood Institute (75N92020C00008, 75N92020C00021, R01HL155410, R01HL169345, R01HL167974, R01HL157235)
  • National Institute of Arthritis and Musculoskeletal and Skin Diseases (R01AR077604, R01AR079431)
  • National Institute of Biomedical Imaging and Bioengineering (75N92020C00021, EB 002524, 75N92020C00008, P41EB027060, R01EB002524)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, clinical text summarization, LLM adaptation, electronic health records, radiology reports, progress notes, doctor-patient dialogue, patient questions, summary quality, safety analysis, clinical documentation burden

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review