CoChat
Sign Up
Cited by 46Machine Learning in HealthcareTopic ModelingArtificial Intelligence in Healthcare and Education

SPeC: A Soft Prompt-Based Calibration on Performance Variability of Large Language Model in Clinical Notes Summarization

Open access
Chuang, Yu-Neng, Tang, Ruixiang, Jiang, Xiaoqian, Hu, Xia·2024·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2024.104606PMC PMC11608453
Open this paper in CoChat — free account

Abstract

Electronic health records (EHRs) store an extensive array of patient information, encompassing medical histories, diagnoses, treatments, and test outcomes. These records are crucial for enabling healthcare providers to make well-informed decisions regarding patient care. Summarizing clinical notes further assists healthcare professionals in pinpointing potential health risks and making better-informed decisions. This process contributes to reducing errors and enhancing patient outcomes by ensuring providers have access to the most pertinent and current patient data. Recent research has shown that incorporating instruction prompts with large language models (LLMs) substantially boosts the efficacy of summarization tasks. However, we show that this approach also leads to increased performance variance, resulting in significantly distinct summaries even when instruction prompts share similar meanings. To tackle this challenge, we introduce a model-agnostic Soft Prompt-BasedCalibration (SPeC) pipeline that employs soft prompts to lower variance while preserving the advantages of prompt-based summarization. Experimental findings on multiple clinical note tasks and LLMs indicate that our method not only bolsters performance but also effectively regulates variance across different LLMs, providing a more consistent and reliable approach to summarizing critical medical information.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2024.104606

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most studies applying large language models on clinical notes have been conducted in the US without individual consent, using de-identified data, with only one study from France utilising retrospective MRI reports without identifiable data.

    “Most studies applying LLM on clinical notes have been conducted in the US without individual consent, using de-identified data, with only one study from France utilising retrospective MRI reports without identifiable data [5, 6, 13, 14, 17–20].”

    From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [19]

    The Dataset section (page unavailable) establishes that this study used radiology reports from a Boston hospital for clinical-note summarization, supporting it as a US example. Neither that section nor Limitations and ethics statements establishes that the data were de-identified or used without individual consent; the paper also does not survey where most such studies occurred or identify a sole French study using retrospective MRI reports. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 33 · Cited by 46

Research works printed but not linked: 27

  1. [1]Wagner T, Shweta F, Murugadoss K, Awasthi S, Venkatakrishnan A, Bade S, Puranik A, Kang M, Pickering BW, O’Horo JC,, Augmented curation of clinical notes from a massive EHR system reveals symptoms of impending COVID-19 diagnosis, Elife 9 (2020) e58227. 32633720 PMC7410498
  2. [2]Pivovarov R, Elhadad N, Automated methods for the summarization of electronic health records, J. Am. Med. Inform. Assoc 22 (5) (2015) 938– 947. 25882031 PMC4986665
  3. [3]Wang M, Wang M, Yu F, Yang Y, Walker J, Mostafa J, A systematic review of automatic text summarization for biomedical literature and EHRs, J. Am. Med. Inform. Assoc 28 (10) (2021) 2287– 2297. 34338801 PMC8449627
  4. [4]Gershanik EF, Lacson R, Khorasani R, Critical finding capture in the impression section of radiology reports, in: AMIA Annual Symposium Proceedings, vol. 2011, American Medical Informatics Association, 2011, p. 465. PMC3243237 22195100
  5. [6]Xiao C, Choi E, Sun J, Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review, J. Am. Med. Inform. Assoc 25 (10) (2018) 1419– 1428. 29893864 PMC6188527
  6. [7]Choi E, Xiao C, Stewart W, Sun J, Mime: Multilevel medical embedding of electronic health records for predictive healthcare, Adv. Neural Inf. Process. Syst 31 (2018).
  7. [8]Wei J, Tay Y, Bommasani R, Raffel C, Zoph B, Borgeaud S, Yogatama D, Bosma M, Zhou D, Metzler D,, Emergent abilities of large language models, 2022, arXiv preprint arXiv:2206.07682. · arXiv
  8. [10]Zhang Y, Zhou K, Liu Z, Neural prompt search, 2022, arXiv preprint arXiv:2206.04673. · arXiv
  9. [15]Lin H, Ng V, Abstractive summarization: A survey of the state of the art, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. No. 01, 2019, pp. 9815– 9822.
  10. [16]See A, Liu PJ, Manning CD, Get to the point: Summarization with pointer-generator networks, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017, pp. 1073– 1083.
  11. [18]Jain R, Jangra A, Saha S, Jatowt A, A survey on medical document summarization, 2022, arXiv preprint arXiv:2212.01669. · arXiv
  12. [19]Zhou Y, Muresanu AI, Han Z, Paster K, Pitis S, Chan H, Ba J, Large language models are human-level prompt engineers, 2022, arXiv preprint arXiv:2211.01910. · arXiv
  13. [20]Shin T, Razeghi Y, Logan IV RL, Wallace E, Singh S, Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020, arXiv preprint arXiv:2010.15980. · arXiv
  14. [22]Wang Z, Sun J, PromptEHR: Conditional electronic healthcare records generation with prompt learning, 2022, arXiv preprint arXiv:2211.01761. PMC11824924 39949499 · arXiv
  15. [23]Vu T, Lester B, Constant N, Al-Rfou R, Cer D, Spot: Better frozen model adaptation through soft prompt transfer, 2021, arXiv preprint arXiv:2110.07904. · arXiv
  16. [24]Kim M, Kim H-I, Ro YM, Prompt tuning of deep neural networks for speaker-adaptive visual speech recognition, 2023, arXiv preprint arXiv:2302.08102. 39437301 · arXiv
  17. [25]Zhang S, Li H, Tang R, Ding S, Rasmy L, Zhi D, Zou N, Hu X, PheME: A deep ensemble framework for improving phenotype prediction from multi-modal data, 2023, arXiv preprint arXiv:2303.10794. · arXiv
  18. [26]Yuan J, Tang R, Jiang X, Hu X, LLM for patient-trial matching: Privacy-aware data augmentation towards better performance and generalizability, 2023, arXiv preprint arXiv:2303.16756. · arXiv
  19. [27]Arora S, Narayan A, Chen MF, Orr LJ, Guha N, Bhatia K, Chami I, Sala F, Ré C, Ask me anything: A simple strategy for prompting language models, 2022, arXiv preprint arXiv:2210.02441. · arXiv
  20. [28]Wu T, Jiang E, Donsbach A, Gray J, Molina A, Terry M, Cai CJ, Promptchainer: Chaining large language model prompts through visual programming, in: CHI Conference on Human Factors in Computing Systems Extended Abstracts, 2022, pp. 1– 10.
  21. [29]Mishra S, Khashabi D, Baral C, Choi Y, Hajishirzi H, Reframing instructional prompts to GPTk’s language, 2021, arXiv preprint arXiv:2109.07830. · arXiv
  22. [31]Kahn CE Jr., Langlotz CP, Burnside ES, Carrino JA, Channin DS, Hovsepian DM, Rubin DL, Toward best practices in radiology reporting, Radiology 252 (3) (2009) 852– 856. 19717755
  23. [32]OpenAI, GPT-4 technical report, 2023, arXiv preprint arXiv:2303.08774. · arXiv
  24. [33]Zhang J, Zhao Y, Saleh M, Liu PJ, PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization, 2019, arXiv preprint arXiv:1912.08777. · arXiv
  25. [34]Lin C-Y, Rouge: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, 2004, pp. 74– 81.
  26. [35]Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M,, Huggingface’s transformers: State-of-the-art natural language processing, 2019, arXiv preprint arXiv:1910.03771. · arXiv
  27. [37]Lakshminarayanan B, Pritzel A, Blundell C, Simple and scalable predictive uncertainty estimation using deep ensembles, Adv. Neural Inf. Process. Syst 30 (2017).

Authors
Chuang, Y.-N., Tang, R., Jiang, X., & Hu, X.
Affiliations
  • Rice University (US)
  • The University of Texas Health Science Center at Houston (US)
Corresponding author
  • Chuang, Yu-Neng
  • Hu, Xia
ORCID
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2024-03
Volume
151
Pages
104606
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2024.104606
PMC
PMC11608453
OpenAlex
W4391826804
Cited by
46
References
33
Citations by year
2026: 12 · 2025: 17 · 2024: 15 · 2023: 2
FWCI
6.49
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Digital Commons-TMC (Texas Medical Center) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2124789)
  • Cancer Prevention and Research Institute of Texas (RR180012)
  • University of Texas Health Science Center at Houston (RR180012)
  • National Institutes of Health (U01TR002062, 1ot2od032581-01, 1OT2-OD032581, 1OT2OD032581-02-211, 1OT2OD032581-02-164, OT2OD032581)
  • National Institute on Aging (R01AG066749, R01AG082721, U01AG079847)
  • National Human Genome Research Institute (U54HG012510)
  • National Cancer Institute (U01CA274576)
  • U.S. National Library of Medicine (U24LM013755, R01LM013712, R01LM014520)
  • National Center for Advancing Translational Sciences
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
clinical note summarization, large language models, soft prompts, performance variability, electronic health records
MeSH terms
Calibration, Health Personnel, Humans, Language, Natural Language Processing, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review