SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarization
Open accessAbstract
Medical dialogue summarization is challenging due to the unstructured nature of medical conversations, the use of medical terminology in gold summaries, and the need to identify key information across multiple symptom sets. We present a novel system for the Dialogue2Note Medical Summarization tasks in the MEDIQA 2023 Shared Task. Our approach for section-wise summarization (Task A) is a two-stage process of selecting semantically similar dialogues and using the top-k similar dialogues as in-context examples for GPT-4. For full-note summarization (Task B), we use a similar solution with k=1. We achieved 3rd place in Task A (2nd among all teams), 4th place in Task B Division Wise Summarization (2nd among all teams), 15th place in Task A Section Header Classification (9th among all teams), and 8th place among all teams in Task B. Our results highlight the effectiveness of few-shot prompting for this task, though we also identify several weaknesses of prompting-based approaches. We compare GPT-4 performance with several finetuned baselines. We find that GPT-4 summaries are more abstractive and shorter. We make our code publicly available.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In-context learning provides the model with context, enhancing LLM performance for a particular task or domain.
“This simple approach provides the model with context, enhancing LLM performance for a particular task or domain27,28.”
From Adapted large language models can outperform medical experts in clinical text summarization (2024) · cites this paper as [28]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · see citing works
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2021) · arXiv · doi:10.48550/arxiv.2104.08786 · Cited by 615 · see citing works
Eliciting Knowledge from Language Models Using Automatically Generated PromptsPreprint
Shin, T., Razeghi, Y., Logan, R. L., IV, Wallace, E., & Singh, S. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2010.15980 · Cited by 584 · see citing works
Complementary Explanations for Effective In-Context LearningPreprint
Ye, X., Iyer, S., Celikyilmaz, A., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2211.13892 · Cited by 127 · see citing works
Flagged first, then by citations
Research works printed but not linked: 20
- Asma Ben Abacha, Wen-wai Yim, Griffin Adams, Neal Snider, and Meliha Yetisgen. 2023. Overview of the mediqa-chat 2023 shared tasks on the summarization and generation of doctor-patient conversations. In ACL-ClinicalNLP 2023.
- Asma Ben Abacha, Wen-wai Yim, Yadan Fan, and Thomas Lin. 2023. An empirical study of clinical note generation from doctor-patient encounters. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 2291-2302, Dubrovnik, Croatia. As- sociation for Computational Linguistics.
- Amanda Bertsch, Graham Neubig, and Matthew R. Gormley. 2022. He said, she said: Style transfer for shifting the perspective of dialogues. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 4823-4840, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
- Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering doc- uments and producing summaries. In Proceedings of the 21st annual international ACM SIGIR confer- ence on Research and development in information retrieval, pages 335-336.
- Bharath Chintagunta, Namit Katariya, Xavier Amatri- ain, and Anitha Kannan. 2021. Medically aware GPT-3 as a data generator for medical dialogue sum- marization. In Proceedings of the Second Workshop on Natural Language Processing for Medical Conver- sations, pages 66-76, Online. Association for Com- putational Linguistics. · DOI
- Yinpei Dai, Hangyu Li, Chengguang Tang, Yongbin Li, Jian Sun, and Xiaodan Zhu. 2020. Learning low- resource end-to-end goal-oriented dialog for fast and reliable system deployment. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 609-618, Online. Associ- ation for Computational Linguistics. · DOI
- Risha Gidwani, Cathina Nguyen, Alexis Kofoed, Cather- ine Carragee, Tracy Rydel, Ian Nelligan, Amelia Sat- tler, Megan Mahoney, and Steven Lin. 2017. Im- pact of scribes on physician satisfaction, patient sat- isfaction, and charting efficiency: A randomized con- trolled trial. Annals of family medicine, 15:427-433.
- Jade Goldstein and Jaime Carbonell. 1998. Summariza- tion: (1) using MMR for diversity-based reranking and (2) evaluating summaries. In TIPSTER TEXT PROGRAM PHASE III: Proceedings of a Workshop held at Baltimore, Maryland, October 13-15, 1998, pages 181-195, Baltimore, Maryland, USA. Associa- tion for Computational Linguistics. · DOI
- Anirudh Joshi, Namit Katariya, Xavier Amatriain, and Anitha Kannan. 2020b. Dr. summarize: Global sum- marization of medical dialogue by exploiting local structures. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 3755- 3763, Online. Association for Computational Lin- guistics. · DOI
- Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
- Jessica López Espejel. 2019. Automatic summariza- tion of medical conversations, a review. In Actes de la Conférence sur le Traitement Automatique des Langues Naturelles (TALN) PFIA 2019. Volume III : RECITAL, pages 487-498, Toulouse, France.
- Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics. · DOI · Paper page
- Yuning Mao, Yanru Qu, Yiqing Xie, Xiang Ren, and Jiawei Han. 2020. Multi-document summarization with maximal marginal relevance-guided reinforce- ment learning. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 1737-1751, Online. As- sociation for Computational Linguistics. · DOI
- George Michalopoulos, Kyle Williams, Gagandeep Singh, and Thomas Lin. 2022. MedicalSum: A guided clinical abstractive summarization model for generating medical reports from patient-doctor con- versations. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 4741- 4749, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- David Fraile Navarro, Mark Dras, and Shlomo Berkovsky. 2022. Few-shot fine-tuning SOTA sum- marization models for medical dialogues. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies: Student Research Workshop, pages 254-266, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics. · DOI
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-5551.
- Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2655-2671, Seattle, United States. Association for Computational Linguistics. · DOI · Paper page
- Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A. Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin Rousseau, Chunhua Weng, and Yifan Peng. 2023. Evaluating large language models on medical evidence summarization. medRxiv. · DOI · Paper page
- Kristen Ullman, Lauren McKenzie, Bradley Bart, Glen- non Park, Roderick MacDonald, Eric Linskens, and Timothy J. Wilt. 2021. The effect of medical scribes in emergency departments: A systematic review. The Journal of Emergency Medicine, 61:19-28.
- Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural In- formation Processing Systems, 33:5776-5788.
Cited by
Sorted by citations
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891 · see citing works
“Similarly, documentation responsibilities for nurses can consume up to 60% of their time and account for considerable work stress 11–13 .”
Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts
Veen, D. V., Uden, C. V., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerova, A., et al. (2023) · doi:10.21203/rs.3.rs-3483777/v1 · Cited by 892 · see citing works
“This simple approach provides the model with context, enhancing LLM performance for a particular task or domain [47, 72].”
BackgroundMethod
- Authors
- Mathur, Y., Rangreji, S., Kapoor, R., Palavalli, M., Bertsch, A., & Gormley, M. R.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 490-502
- arXiv
- 2306.17384
- OpenAlex
- W4385571552
- Cited by
- 18
- References
- 5
- Citations by year
- 2026: 7 · 2025: 4 · 2024: 6 · 2023: 1
- FWCI
- 3.19
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Clinical Natural Language Processing Workshop (ClinicalNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- GPT-4, in-context learning, few-shot prompting, medical terminology, abstractive summarization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
