Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts
Open accessPreprint — published as 10.1038/s41591-024-02855-5 · Paper page
Abstract
Sifting through vast textual data and summarizing key information from electronic health records (EHR) imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown immense promise in natural language processing (NLP) tasks, their efficacy on a diverse range of clinical summarization tasks has not yet been rigorously demonstrated. In this work, we apply domain adaptation methods to eight LLMs, spanning six datasets and four distinct clinical summarization tasks: radiology reports, patient questions, progress notes, and doctor-patient dialogue. Our thorough quantitative assessment reveals trade-offs between models and adaptation methods in addition to instances where recent advances in LLMs may not improve results. Further, in a clinical reader study with ten physicians, we show that summaries from our best-adapted LLMs are preferable to human summaries in terms of completeness and correctness. Our ensuing qualitative analysis highlights challenges faced by both LLMs and human experts. Lastly, we correlate traditional quantitative NLP metrics with reader study scores to enhance our understanding of how these metrics align with physician preferences. Our research marks the first evidence of LLMs outperforming human experts in clinical text summarization across multiple tasks. This implies that integrating LLMs into clinical workflows could alleviate documentation burden, empowering clinicians to focus more on personalized patient care and the inherently human aspects of medicine.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Domain-specific LLMs perform well in clinical text summarization, note generation, and medical question answering.
“Domain-specific LLMs, fine-tuned on medical corpora or using knowledge graph embeddings, perform well in clinical text summarization [10, 11], note generation [12, 13], and medical question answering [14].”
From Enhancing hepatopathy clinical trial efficiency: a secure, large language model-powered pre-screening pipeline (2025) · cites this paper as [12]
Most prior research in this field speculates on high-level concerns such as inaccuracy and real-world impacts, but rarely articulates concrete criteria for when and why LLMs should or should not provide professional advice to users.
“Most prior research in this field speculates on high-level concerns such as inaccuracy and real-world impacts [43, 44, 54, 86, 93]. However, these studies rarely articulate concrete criteria for when and why LLMs should or should not provide professional advice to users.”
From (A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice (2024) · cites this paper as [86]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“Large language models (LLMs) have demonstrated astounding performance, propelled by both the transformer architecture [73] and increasing scales of data and compute, resulting in widespread adoption of models such as ChatGPT [6].”
MethodA guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 29,639 · cited 1× · see citing works
“We estimate intra-reader correlation based on a mean-rating, fixed agreement, two-may mixed effects model [42] using the Pingouin package [80].”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 8× · see citing works
“Large language models (LLMs) have demonstrated astounding performance, propelled by both the transformer architecture [73] and increasing scales of data and compute, resulting in widespread adoption of models such as ChatGPT [6].”
BackgroundMethodInfluentialBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 8× · see citing works
“BLEU [52], the simplest metric, calculates the degree of overlap between the reference and generated texts by considering 1- to 4-gram sequences.”
BackgroundInfluentialKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1× · see citing works
“The original T5 “text-to-text transfer transformer” model [55] demonstrated excellent performance in transfer learning using the seq2seq architecture.”
MethodTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
International Journal of Environmental Research and Public Health
Casariego Vales, E., A.M., C.-C., J.L., L., R.M., M.-P., D., O.-B., M., O.-C., P., R.-S., & F.J., S. (2013) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph · Cited by 7,742 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 2× · see citing works
“Recent work in clinical natural language processing (NLP) has demonstrated potential on medical text [75, 86], adapting to the medical domain by either training a new model [68, 79], fine-tuning an existing model [76, 81], or supplying task-specific examples in the model prompt [53, 81].”
BackgroundMethodLarge Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · cited 2× · see citing works
“Recent work in clinical natural language processing (NLP) has demonstrated potential on medical text [75, 86], adapting to the medical domain by either training a new model [68, 79], fine-tuning an existing model [76, 81], or supplying task-specific examples in the model prompt [53, 81].”
BackgroundHuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · cited 2× · see citing works
“All open-source models are available on HuggingFace [85].”
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.03109 · Cited by 3,159 · cited 1× · see citing works
“Beyond the scope of our work, there’s further potential to reduce hallucinations through incorporating checks by a human, checks by another LLM, or using a model ensemble to create a “committee of experts” [11, 39].”
MethodBERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · cited 1× · see citing works
“In addition to these syntactic metrics, we employ BERTScore, which leverages contextual BERT embeddings to evaluate the semantic similarity of the generated and reference texts [82].”
MethodA Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · cited 1× · see citing works
“Beyond the scope of our work, there’s further potential to reduce hallucinations through incorporating checks by a human, checks by another LLM, or using a model ensemble to create a “committee of experts” [11, 39].”
MethodLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 3× · see citing works
“Descendants of this original model have taken additional fine-tuning approaches, such as fine-tuning via instruction following (Alpaca [66]), medical Q&A data (Med-Alpaca [30]), user-shared conversations (Vicuna [10]), and reinforcement learning from human feedback (Llama-2 [70]).”
BackgroundMethodPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1× · see citing works
“Although several of the more expansive models, such as GPT-4 [51] and PaLM [11], remain proprietary and provide access via “black-box” interfaces, there has been a pronounced shift towards open-sourced alternatives such as Llama-2 [70].”
MethodLoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · cited 2× · see citing works
“Low-rank adaptation (LoRA) [31] has emerged as an effective, lightweight approach for fine-tuning LLMs by altering a small subset of model weights—often < 0.”
BackgroundMethodPrefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 1× · see citing works
“To enable domain adaptation with limited computational resources, prefix tuning [45] and low-rank adaptation (LoRA) [35] have emerged as effective methods that require tuning less than 1% of total parameters over a small training set.”
MethodA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · cited 1× · see citing works
“Still other works have proposed generalist models to perform many tasks across the medical domain [54, 79].”
BackgroundJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · see citing works
Pingouin: statistics in Python
Vallat, R. (2018) · Journal of Open Source Software · doi:10.21105/joss.01026 · Cited by 1,780 · cited 1× · see citing works
“We estimate intra-reader correlation based on a mean-rating, fixed agreement, two-may mixed effects model [42] using the Pingouin package [80].”
MethodSparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 1× · see citing works
“In recent years, large language models (LLMs) have gained remarkable traction, leading to widespread adoption of models such as ChatGPT [8], which excel at information retrieval, nuanced understanding, and text generation [9, 93].”
BackgroundAllocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties
Sinsky, C., Colligan, L., Li, L., Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., & Blike, G. (2016) · Annals of Internal Medicine · doi:10.7326/m16-0961 · Cited by 1,494 · cited 1× · see citing works
“Recent data indicates that physicians can expend two hours on documentation for each hour of patient interaction [69].”
BackgroundScaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 1× · see citing works
“A derivative model, FLAN-T5 [16, 50], improved performance via instruction prompt tuning.”
MethodExploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y.-H. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02410 · Cited by 1,202 · see citing works
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 1× · see citing works
“One approach is customizing model weights via instruction tuning, a process where language models are trained to generate human-aligned responses given specific instructions [75].”
BackgroundShanafelt, T. D., Dyrbye, L. N., Sinsky, C., Hasan, O., Satele, D., Sloan, J., & West, C. P. (2016) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2016.05.007 · Cited by 1,086 · cited 1× · see citing works
“These tasks divert attention from patient care, leading to worse outcomes for patients as well as disillusionment and decreased job satisfaction for clinicians [2, 57, 59].”
BackgroundThe Flan Collection: Designing Data and Methods for Effective Instruction TuningPreprint
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2301.13688 · Cited by 1,029 · cited 1× · see citing works
“A derivative model, FLAN-T5 [16, 50], improved performance via instruction prompt tuning.”
MethodArndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W.-J., Sinsky, C. A., & Gilchrist, V. J. (2017) · The Annals of Family Medicine · doi:10.1370/afm.2121 · Cited by 972 · cited 2× · see citing works
“Currently, clinicians spend significant time summarizing vast amounts of textual information—whether it be compiling diagnostic reports, writing progress notes, or synthesizing a patient’s treatment history across different specialists [4, 27, 33].”
BackgroundIEEE Transactions on Visualization and Computer Graphics
(2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2945 · Cited by 719 · see citing works
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 507 · cited 3× · see citing works
“Recently, QLoRA [22] has been introduced as a more memory-efficient variant of LoRA, employing 4-bit quantization to enable the fine-tuning of larger LLMs given the same hardware constraints.”
BackgroundMethodInfluentialThe Best of Both Worlds: Combining Recent Advances in Neural Machine Translation
Chen, M., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G. F., Jones, L., Schuster, M., Shazeer, N., Parmar, N., et al. (2018) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p18-1008 · Cited by 484 · cited 1× · see citing works
“These models have shown strong performance in machine translation [12] and summarization [67].”
BackgroundTowards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2024) · NEJM AI · doi:10.1056/aioa2300138 · Cited by 483 · see citing works
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483 · cited 3× · see citing works
“Recent work in clinical natural language processing (NLP) has demonstrated potential on medical text [75, 86], adapting to the medical domain by either training a new model [68, 79], fine-tuning an existing model [76, 81], or supplying task-specific examples in the model prompt [53, 81].”
BackgroundHolistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 1× · see citing works
“While excellent LLM benchmarks for general NLP tasks exist [40, 84], they do not evaluate clinical performance.”
BackgroundEvaluating Large Language Models on Medical Evidence Summarization
Tang, L., Sun, Z., Idnay, B., Nestor, J. G., Soroush, A., Elias, P. A., Xu, Z., Ding, Y., Durrett, G., Rousseau, J., et al. (2023) · doi:10.1101/2023.04.22.23288967 · Cited by 423 · see citing works
Evaluating large language models on medical evidence summarization
Tang, L., Sun, Z., Idnay, B., Nestor, J. G., Soroush, A., Elias, P. A., Xu, Z., Ding, Y., Durrett, G., Rousseau, J. F., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00896-7 · Cited by 423 · see citing works
(2013) · Europe PMC (PubMed Central) · Cited by 396 · see citing works
Journal Of Trauma And Acute Care Surgery
Krakker, J. (2013) · Wilderness and Environmental Medicine · doi:10.1016/j.wem.2013.03.009 · Cited by 387 · see citing works
Khamisa, N., Peltzer, K., & Oldenburg, B. (2013) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph10062214 · Cited by 338 · cited 1× · see citing works
“Meanwhile, documentation responsibilities for nurses consume up to 60% of their time and account for significant work stress [8, 21, 36].”
BackgroundThe shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · cited 2× · see citing works
“Recent work has adapted LLMs for various medical tasks, demonstrating great potential for medical language understanding and generation [42, 68, 71, 77].”
BackgroundNeural Abstractive Text Summarization with Sequence-to-Sequence Models
Shi, T., Keneshloo, Y., Ramakrishnan, N., & Reddy, C. K. A. (2021) · ACM/IMS Transactions on Data Science · doi:10.1145/3419106 · Cited by 263 · see citing works
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255 · see citing works
Toward Best Practices in Radiology Reporting
Kahn, C. E., Langlotz, C. P., Burnside, E. S., Carrino, J. A., Channin, D. S., Hovsepian, D. M., & Rubin, D. L. (2009) · Radiology · doi:10.1148/radiol.2523081992 · Cited by 245 · cited 1× · see citing works
“One such task is the summarization of radiology reports, which aims to consolidate detailed findings from radiological studies into significant observations and conclusions drawn by the radiologist [35].”
BackgroundSelf-citationInteractive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209479 · Cited by 204 · cited 1× · see citing works
“Prompt phrasing and model hyperparameters can be very important for a LLM, as demonstrated in the literature [71, 83] and in Table 3.”
BackgroundGPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022) · arXiv · doi:10.48550/arxiv.2210.17323 · Cited by 197 · cited 1× · see citing works
“After selecting these s samples, we choose another s as a We leverage PyTorch, using the parameter-efficient fine-tuning [52] and the generative pre-trained transformers quantization [28] libraries to implement QLoRA.”
MethodA usability and safety analysis of electronic health records: a multi-center study
Ratwani, R. M., Savage, E., Will, A., Arnold, R., Khairat, S., Miller, K., Fairbanks, R. J., Hodgkins, M., & Hettinger, A. Z. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy088 · Cited by 180 · cited 1× · see citing works
“The widespread adoption of electronic health records (EHR) has expanded clinical documentation workload, directly contributing to increasing stress and clinician burnout [26, 32, 63].”
BackgroundDesign and Development of a Multimodal Biomedical Information Retrieval System
Demner-Fushman, D., Antani, S., Simpson, M., & Thoma, G. R. (2012) · Journal of Computing Science and Engineering · doi:10.5626/jcse.2012.6.2.168 · Cited by 161 · cited 3× · see citing works
“From the initial set of 4K studies, Demner-Fushman et al. [21] selected a final set of 3.4K reports based on the quality of imaging views and diagnostic content.”
BackgroundCombining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023) · doi:10.2139/ssrn.4505053 · Cited by 161 · cited 1× · see citing works
“Ultimately, such new tools may improve the clinical workflow [3], resulting in decreased clinician strain and improved patient care.”
BackgroundPrompt Engineering for Healthcare: Methodologies and Applications
Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.14670 · Cited by 150 · cited 1× · see citing works
“Prompt phrasing and model hyperparameters can be very important for a LLM, as demonstrated in the literature [71, 83] and in Table 3.”
BackgroundCan language models learn from explanations in context?
Lampinen, A., Dasgupta, I., Chan, S., Mathewson, K., Tessler, M., Creswell, A., McClelland, J., Wang, J., & Hill, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.38 · Cited by 142 · see citing works
MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training DataPreprint
Han, T., Adams, L. C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., Figueroa, A., Löser, A., Truhn, D., & Bressem, K. K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08247 · Cited by 117 · cited 2× · see citing works
“For example, Han et al. [34] released Med-Alpaca, a version of Alpaca [73] which was further instruction-tuned with medical Q&A text, consequently improving performance for the task of medical question-answering.”
BackgroundOn the Summarization of Consumer Health Questions
Abacha, A. B., & Demner-Fushman, D. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1215 · Cited by 114 · see citing works
Hyena Hierarchy: Towards Larger Convolutional Language Models
Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., & Ré, C. (2023) · arXiv · doi:10.48550/arxiv.2302.10866 · Cited by 96 · cited 1× · see citing works
“Addressing these cases demands two key advancements: (1) transcending GPT-4’s current context length of 32,768 tokens, potentially through multi-query aggregation or methods which increase context length [20, 53], and (2) introducing open-source datasets that encompass broader tasks and lengthier documents.”
BackgroundRobinson, K. E., & Kersey, J. A. (2018) · Medicine · doi:10.1097/md.0000000000012319 · Cited by 90 · cited 1× · see citing works
“These tasks divert attention from direct patient care, leading to worse outcomes for patients as well as disillusionment and decreased job satisfaction for clinicians [4, 64, 66, 77].”
BackgroundThe Burden and Burnout in Documenting Patient Care: An Integrative Literature Review
Gesner, E. J., Gazarian, P., & Dykes, P. (2019) · Medinfo · doi:10.3233/shti190415 · Cited by 72 · cited 1× · see citing works
“The proliferation of electronic health records (EHR) has drastically increased clinical documentation workload, directly contributing to increasing stress and clinician burnout [22, 28, 56].”
BackgroundMedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30205 · Cited by 66 · see citing works
Self-citationMedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E. P., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.14089 · Cited by 65 · cited 2× · see citing works
“To address this, there have been recent releases of expert evaluations for instruction following [27] and radiology report generation [90].”
BackgroundInfluentialSelf-citationYim, W.-W., Fu, Y., Ben Abacha, A., Snider, N., Lin, T., & Yetisgen, M. (2023) · Scientific Data · doi:10.1038/s41597-023-02487-3 · Cited by 64 · cited 4× · see citing works
“Lastly, there has been significant work on summarizing extended conversations between a doctor and patient into patient visit summaries [1, 53, 89].”
BackgroundMethodInfluentialSimple and Efficient Algorithm for Approximate Dictionary Matching
Okazaki, N., & Tsujii, J. (2010) · Cited by 62 · see citing works
Do We Still Need Clinical Language Models?
Lehman, E., Hernandez, E., Mahajan, D., Wulff, J., Smith, M. J., Ziegler, Z., Nadler, D., Szolovits, P., Johnson, A., & Alsentzer, E. (2023) · arXiv · doi:10.48550/arxiv.2302.08091 · Cited by 62 · cited 1× · see citing works
“This T5 model family has proven effective for various clinical NLP tasks [44, 81].”
BackgroundRadiology-GPT: A Large Language Model for Radiology
Liu, Z., Zhong, A., Li, Y., Yang, L., Ju, C., Wu, Z., Ma, C., Shu, P., Chen, C., Kim, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.08666 · Cited by 59 · cited 2× · see citing works
“Examples of clinical instruction-tuned models include Med-PALM [68] for medical question-answering or Radiology-GPT [49] and Radiology-Llama2 [49] for radiology tasks.”
MethodTechnology as friend or foe? Do electronic health records increase burnout?
Ehrenfeld, J. M., & Wanderer, J. P. (2018) · Current Opinion in Anaesthesiology · doi:10.1097/aco.0000000000000588 · Cited by 51 · see citing works
Unintended errors with EHR-based result management: a case series
Yackel, T. R., & Embi, P. J. (2010) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3294 · Cited by 48 · cited 1× · see citing works
“Even for experienced physicians with a high level of expertise, this intricate task naturally introduces the possibility for errors, which can be detrimental in a field where precision is paramount [7, 31, 87].”
Background(2011) · PubMed Central · Cited by 46 · see citing works
What’s in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarization
Adams, G., Alsentzer, E., Ketenci, M., Zucker, J. E., & Elhadad, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.382 · Cited by 45 · cited 1× · see citing works
“LLMs have shown promise on this task [81] and other tasks such as summarizing daily progress notes into a concise “problem list” of medical diagnoses [29] or summarization of clinical narratives [2].”
Leveraging Large Language Models for Generating Responses to Patient Messages
Liu, S., McCoy, A. B., Wright, A. P., Carew, B., Genkins, J. Z., Huang, S. S., Peterson, J. F., Steitz, B., & Wright, A. (2023) · doi:10.1101/2023.07.14.23292669 · Cited by 43 · cited 1× · see citing works
“Recent work has adapted LLMs for various medical tasks, demonstrating great potential for medical language understanding and generation [42, 68, 71, 77].”
BackgroundClinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding
Toma, A., Lawler, P. R., Ba, J., Krishnan, R. G., Rubin, B. B., & Wang, B. (2023) · arXiv · doi:10.48550/arxiv.2305.12031 · Cited by 36 · cited 2× · see citing works
“LoRA has been shown to work well for medical question-answering [69] and summarizing radiology reports [72].”
BackgroundPoint of Care Documentation Impact on the Nurse-Patient Interaction
Duffy, W. J., Kharasch, M. S., & Du, H. (2010) · Nursing Administration Quarterly · doi:10.1097/naq.0b013e3181c95ec4 · Cited by 34 · cited 1× · see citing works
“Meanwhile, documentation responsibilities for nurses consume up to 60% of their time and account for significant work stress [10, 25, 41].”
BackgroundLongNet: Scaling Transformers to 1, 000, 000, 000 Tokens
Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., Zheng, N., & Wei, F. (2023) · arXiv · doi:10.48550/arxiv.2307.02486 · Cited by 34 · cited 1× · see citing works
“…these cases demands two key advancements: (1) transcending GPT-4’s current context length of 32,768 tokens, potentially through multi-query aggregation or methods which increase context length [24, 60], and (2) introducing open-source datasets that encompass broader tasks and lengthier documents.”
BackgroundNurses’ Experiences of an Initial and Reimplemented Electronic Health Record Use
Chang, C.-P., Lee, T.-T., Liu, C.-H., & Mills, M. E. (2016) · CIN: Computers, Informatics, Nursing · doi:10.1097/cin.0000000000000222 · Cited by 32 · cited 1× · see citing works
“Meanwhile, documentation responsibilities for nurses consume up to 60% of their time and account for significant work stress [8, 21, 36].”
BackgroundThe painful truth: The documentation burden of a trauma surgeon
Golob, J. F., Como, J. J., & Claridge, J. A. (2016) · Journal of Trauma and Acute Care Surgery · doi:10.1097/ta.0000000000000986 · Cited by 32 · cited 1× · see citing works
“Currently, clinicians spend significant time summarizing vast amounts of textual information—whether it be compiling diagnostic reports, writing progress notes, or synthesizing a patient’s treatment history across different specialists [2, 23, 29].”
BackgroundGao, Y.-J., Dligach, D., Miller, T., et al. (2023) · Workshop on Biomedical Natural Language Processing · doi:10.48550/arxiv.2306.05270 · Cited by 29 · cited 3×
“For this we employ the ProbSum dataset [29], extracted from the MIMIC-III database of de-identified hospital intensive care unit (ICU) admissions.”
MethodRadiology-Llama2: Best-in-Class Large Language Model for RadiologyPreprint
Liu, Z., Li, Y., Shu, P., Zhong, A., Yang, L., Ju, C., Wu, Z., Ma, C., Jie, L., Chen, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.06419 · Cited by 28 · cited 1× · see citing works
“Examples of clinical instruction-tuned models include Med-PALM [68] for medical question-answering or Radiology-GPT [49] and Radiology-Llama2 [49] for radiology tasks.”
MethodRadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models
Van Veen, D., Van Uden, C., Attias, M., Pareek, A., Bluethgen, C., Polacin, M., Chiu, W., Delbrouck, J.-B., Zambrano Chaves, J., Langlotz, C., et al. (2023) · The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks · doi:10.18653/v1/2023.bionlp-1.42 · Cited by 25 · see citing works
Self-citationEvidence Extraction to Validate Medical Claims in Fake News Detection
Deka, P., Jurek, A., & Deepak, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20627-6_1 · Cited by 24 · see citing works
Ben Abacha, A., Yim, W.-W., Adams, G., Snider, N., & Yetisgen, M. (2023) · Proceedings of the 5th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2023.clinicalnlp-1.52 · Cited by 20 · see citing works
SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarization
Mathur, Y., Rangreji, S., Kapoor, R., Palavalli, M., Bertsch, A., & Gormley, M. R. (2023) · arXiv · doi:10.48550/arxiv.2306.17384 · Cited by 18 · cited 3× · see citing works
“This simple approach provides the model with context, enhancing LLM performance for a particular task or domain [47, 72].”
BackgroundMethodGao, Y., Dligach, D., Miller, T. A., & Afshar, M. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.43 · Cited by 17 · see citing works
Delbrouck, J.-B., Varma, M., Chambon, P., & Langlotz, C. P. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.45 · Cited by 16 · cited 4× · see citing works
“We utilize a preprocessed version via RadSum23 [15, 16].”
MethodInfluentialSelf-citationCan language models learn from explanations in context?Preprint
Lampinen, A. K., Dasgupta, I., Chan, S. C. Y., Matthewson, K., Tessler, M. H., Creswell, A., McClelland, J. L., Wang, J. X., & Hill, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.02329 · Cited by 16 · cited 3× · see citing works
“Because in-context learning does not alter model weights, it can be performed with black-box model access using only a few training examples [43].”
BackgroundMethodInfluentialToward Expanding the Scope of Radiology Report Summarization to Multiple Anatomies and Modalities
Chen, Z., Varma, M., Wan, X., Langlotz, C. P., & Delbrouck, J.-B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.41 · Cited by 11 · cited 2× · see citing works
“One such task is the summarization of radiology reports [13], which aims to consolidate detailed findings from radiological studies into significant observations and conclusions drawn by the radiologist [40].”
BackgroundMethodSelf-citationImproving Few-Shot Performance of Language Models via Nearest Neighbor Calibration
Nie, F., Chen, M., Zhang, Z., & Cheng, X. (2022) · arXiv · doi:10.48550/arxiv.2212.02216 · Cited by 9 · cited 1× · see citing works
“Note that choosing “relevant” in-context examples typically outperforms choosing examples at random [55].”
BackgroundRadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language ModelsPreprint
Veen, D. V., Uden, C. E. V., Attias, M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.01146 · Cited by 6 · cited 5× · see citing works
“LoRA has been shown to work well for medical question-answering [69] and summarizing radiology reports [72].”
BackgroundMethodInfluentialSelf-citationYim, W., Fu, Y., Abacha, A. B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.02022 · Cited by 3 · cited 4× · see citing works
“Lastly, there has been significant work on summarizing extended conversations between a doctor and patient into patient visit summaries [1, 53, 89].”
BackgroundMethodInfluentialDesign Considerations for High Impact, Automated Echocardiogram Analysis
Toussaint, W., Van Veen, D., Irwin, C., Nachmany, Y., Barreiro-Perez, M., Díaz-Peláez, E., de Sousa, S. G., Millán, L., Sánchez, P. L., Sánchez-Puente, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.06292 · Cited by 1 · cited 1× · see citing works
“These tasks divert attention from direct patient care, leading to worse outcomes for patients as well as disillusionment and decreased job satisfaction for clinicians [4, 64, 66, 77].”
BackgroundSelf-citationRaLEs: a Benchmark for Radiology Language Evaluations
Chaves, J. Z., Bhaskhar, N., Attias, M., Delbrouck, J.-B., Rubin, D., Loening, A., Langlotz, C., & Chaudhari, A. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3254 · cited 1×
“Recent work has adapted LLMs for various medical tasks, demonstrating great potential for medical language understanding and generation [48, 75, 79, 86, 91].”
BackgroundSelf-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 5
- [15]Chung H. W., Hou L., Longpre S., Zoph B., Tay Y., Fedus W., Li E., Wang X., Dehghani M., Brahma S., Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022). · arXiv
- [18]Demner-Fushman D., Ananiadou S. & Cohen K. B. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks in The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks (2023).
- [27]Gao Y., Miller T., Afshar M. & Dligach D. BioNLP Workshop 2023 Shared Task 1A: Problem List Summarization in Proceedings of the 22nd Workshop on Biomedical Language Processing (2023).
- [61]Soldaini L. & Goharian N. Quickumls: a fast, unsupervised approach for medical concept extraction in MedIR workshop, sigir (2016), 1– 4.
- [65]Tay Y., Dehghani M., Tran V. Q., Garcia X., Wei J., Wang X., Chung H. W., Bahri D., Schuster T., Zheng S., Ul2: Unifying language learning paradigms in The Eleventh International Conference on Learning Representations (2022).
Cited by
Sorted by citations
The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)
Zeng, S.-L., Zhang, J., He, P.-F., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.16893 · Cited by 280
“2.1 Retrieval-Augmented Generation (RAG) Retrieval-augmented generation (RAG), first introduced by Lewis et al. (2020), has emerged as one of the most popular approaches to enhance the generation ability of LLMs (Liu, 2022; Chase, 2022; Van Veen et al., 2023; Ram et al., 2023; Shi et al., 2023).”
BackgroundInfluentialA Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
“…though using LLMs for clinical report generation or summarization has been proven to be more complete and more accurate than the human counterpart [139], there are still concerns with hallucination, as well as a tendency to approach inputs with a literal view instead of an assumption based…”
BackgroundA Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray InterpretationPreprint
Chen, Z., Varma, M., Xu, J., Paschali, M., Veen, D. V., Johnston, A., Youssef, A., Blankemeier, L., Blüthgen, C., Altmayer, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.12208 · Cited by 26 · see citing works
“…CheXagent generates high-quality reports when compared to other medical FMs. • For findings summarization, as shown in Table 4, CheXagent outperforms LLMs of comparable size on the Rouge-L metric and achieves comparable performance to LLMs with more than twice the number of parameters [80, 81].”
Self-citationA Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models
Zhang, H., Yu, P. S., & Zhang, J. (2025) · ACM Computing Surveys · doi:10.1145/3731445 · Cited by 159 · see citing works
“…[121] Graph LexRank [55], TextRank [148], PACSUM [278], [49] Deep Learning (§3.2) CNN [254], [180], NN-SE [30], T-CONVS2S [157], AttSum [20] RNN/LSTM [40], SummaRuNNer [154], ABS [155], [271], [197], ExtSum-LG [235] RL [165], [245], REFRESH [158], [50] GNN [253], DiscoBert [240], [124], HSG…”
InfluentialA comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Ли, Ж., Liu, X., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 130 · see citing works
“These models offer promising solutions for medical report generation [19, 20], clinical diagnosis [17, 21], mental health services [22, 23], and various other clinical applications.”
Large language models for structured reporting in radiology: past, present, and future
Busch, F., Hoffmann, L., dos Santos, D. P., Makowski, M. R., Saba, L., Prucker, P., Hadamitzky, M., Navab, N., Kather, J. N., Truhn, D., et al. (2024) · European Radiology · doi:10.1007/s00330-024-11107-6 · Cited by 134 · see citing works
“In addition to language translation, LLMs could support language correction/editing, medical text summarization, or text simpli fi cation, e.g., when reporting examination results to patients, but also for streamlining imaging protocols [78, 79].”
Enhancing medical AI with retrieval-augmented generation: A mini narrative review
Gargari, O. K., & Habibi, G. (2025) · Digital Health · doi:10.1177/20552076251337177 · Cited by 105 · see citing works
Abstractive text summarization: State of the art, challenges, and improvements
Shakil, H., Farooq, A., & Kalita, J. K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128255 · Cited by 89 · see citing works
“They can adapt effectively to user preferences by analyzing interactions and feedback, enabling them to generate summaries that are increasingly tailored to individual user profiles over time [195].”
Li, L., Zhou, J., Gao, Z., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2405.03066 · Cited by 89
“BART and T5 have also been employed for summarizing data 149,152,154 .”
MethodCheong, I., Xia, K., Feng, K. J. K., Chen, Q. Z., & Zhang, A. X. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659048 · Cited by 91 · see citing works
“Researchers have facilitated expert discussions to evaluate the sociotechnical implications of LLMs [6, 61, 73, 76].”
BackgroundMitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data
Zeng, S.-L., Zhang, J., He, P.-F., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2406.14773 · Cited by 78
“Our experimental re-sults show that using our synthetic data as retrieval data can achieve comparable performance with us-ing original data while substantially reducing the associated privacy risks.”
LLMIF: Augmented Large Language Model for Fuzzing IoT Devices
Wang, J., Yu, L., & Luo, X. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00211 · Cited by 68 · see citing works
“[18] shows that LLMs can perform well in comprehending the given document, e.g., radiology reports and doctor-patient dialogue, and reason it with domain knowledge.”
BackgroundA Continued Pretrained LLM Approach for Automatic Medical Note Generation
Yuan, D., Rastogi, E., Naik, G., et al. (2024) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2403.09057 · Cited by 65
“They were trained by explanation tuning a LLaMA2 13B model (Touvron et al., 2023) using bigger models like GPT4 as a teacher.”
BackgroundMethodInfluentialLarge language models in biomedicine and healthcare
Zhou, J., Li, H., Chen, S., Chen, Z., Han, Z., & Gao, X. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00047-1 · Cited by 61 · see citing works
“A study 136 evaluating LLMs for clinical text summarization shows that models adapted to speci fi c tasks and domains produce higher-quality summaries.”
Introduction to large language models (LLMs) for dementia care and research
Treder, M. S., Lee, S., & Tsvetanov, K. A. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1385303 · Cited by 63 · see citing works
Yang, Y., Mishra, S., Chiang, J., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2403.07384 · Cited by 63
Li, J.-W., Yang, Y.-Z., Bai, Y., Zhou, X., Li, Y.-H., Sun, H., Liu, Y., Si, X., Ye, Y., Wu, Y., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2024.acl-long.599 · Cited by 63
Data-Centric Foundation Models in Computational Healthcare: A Survey
Zhang, Y., Gao, J. G., Tan, Z., Zhou, L., Ding, K., Zhou, M., Zhang, S., & Wang, D. (2026) · ACM Computing Surveys · doi:10.1145/3800677 · Cited by 42 · see citing works
“Complementary to masked modeling, autoregressive pre-training supports generative biomedical LMs. BioGPT [151] is pre-trained on PubMed 1 abstracts following GPT-2 [184] for generative language tasks.”
Fundamental Capabilities and Applications of Large Language Models: A Survey
Li, J.-W., Gao, Y., Yang, Y.-Z., Bai, Y., Zhou, X., Li, Y.-H., Sun, H., Liu, Y., Si, X., Ye, Y., et al. (2025) · ACM Computing Surveys · doi:10.1145/3735632 · Cited by 54 · see citing works
“And more importantly, LLMs have the capabilities to diagnose.”
Bias Patterns in the Application of LLMs for Clinical Decision Support
Poulain, R., Adiba, F. I., Fayyaz, H., & Beheshti, R. (2026) · Delaware Journal of Public Health · doi:10.32481/djph.2026.03.10 · Cited by 32 · see citing works
“The rapid adoption of LLMs in healthcare has had some forefront applications in areas where LLMs (with their NLP roots) shine, including summarizing medical (free-text) notes, answering patients’ questions, and generating patient discharge letters [54].”
BackgroundMarkowitz, D. M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae387 · Cited by 48 · see citing works
“The second aim is to evaluate if such lay summaries can be made even simpler with generative AI, which has demonstrated immense potential to summarize and simplify science writing in prior work (33, 34).”
BackgroundBest Practices for Large Language Models in Radiology
Blüthgen, C., Veen, D. V., Zakka, C., Link, K. E., Fanous, A. H., Daneshjou, R., Frauenfelder, T., Langlotz, C. P., Gatidis, S., & Chaudhari, A. S. (2025) · Radiology · doi:10.1148/radiol.240528 · Cited by 41 · see citing works
“52 Frequency and presence penalty are parameters preventing the LLM from repeating itself too often.”
Self-citationEconomics and Equity of Large Language Models: Health Care Perspective
Nagarajan, R., Kondo, M., Salas, F., Sezgın, E., Yao, Y., Klotzman, V., Godambe, S. A., Khan, N., Limon, A., Stephenson, G. S., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/64226 · Cited by 34 · see citing works
“3c , tasks critical for managing extensive medical documentation and improving clinical workflows [145] .”
AI solutions to the radiology workforce shortage
Jing, A. B., Garg, N., Zhang, J., & Brown, J. J. (2025) · npj Health Systems · doi:10.1038/s44401-025-00023-6 · Cited by 40 · see citing works
“These summaries can highlight critical fi ndings or longitudinal trends that help inform interpretation 50 .”
Durak, H. Y., Eğin, F., & Onan, A. (2025) · European Journal of Education · doi:10.1111/ejed.70014 · Cited by 24 · see citing works
Enhancing Medical Text Evaluation with GPT-4Preprint
Xie, Y., Zhang, S., Cheng, H., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2311.09581 · Cited by 4 · see citing works
Qin, R., Yang, K., Abbasi, A., Dobolyi, D. G., Seyedi, S. S., Griner, E. L., Kwon, H., Cotes, R. O., Jiang, Z., Clifford, G. D., et al. (2024) · ACM Transactions on Management Information Systems · doi:10.1145/3673906 · Cited by 18 · see citing works
“Although it can be used for sequence classification, Clinical Camel is primarily used for text summarization [107] and information extraction tasks involving clinical text data [28, 98].”
Multimodal Large Language Models for Medicine: A Comprehensive SurveyPreprint
Ye, J.-R., & Tang, H. (2025) · arXiv.org · doi:10.48550/arxiv.2504.21051 · Cited by 29 · see citing works
“Through fine-tuning on medical healthcare data, LLMs have been proven to have great ability and potential in processing medical text [27], [28], [29].”
Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective
Zeng, S.-L., Zhang, J., Li, B., et al. (2024) · Volume 1 · doi:10.48550/arxiv.2411.14572 · Cited by 26
“This approach has been applied to various domains and scenarios (Liu, 2022; Chase, 2022; Van Veen et al., 2023; Ram et al., 2023; Shi et al., 2023; Siriwardhana et al., 2023; Parvez et al., 2021; Panagoulias et al., 2024; Pipitone and Alami, 2024; Mozharovskii, 2024).”
Can Large Language Models abstract Medical Coded Language?
Lee, S. A., & Lindsey, T. (2024) · doi:10.48550/arxiv.2403.10822 · Cited by 25
“One of the key utilities of LLMs lies in their capability to efficiently summarize clinical texts, a feature that has been highlighted in studies by (Van Veen et al., 2023, 2024).”
BackgroundA Brief Review on Benchmarking for Large Language Models Evaluation in Healthcare
Cilar, L., Chen, H., Chen, A., Topaz, M., Tam, W., Bian, J., & Štiglic, G. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70010 · Cited by 24 · see citing works
“For instance, Veen et al. (Van Veen et al. 2023) demonstrated that domain-adapted LLMs could effectively summarize radiology reports, reducing clinician workload while enhancing the quality and accuracy of the summaries.”
InfluentialAlsanosi, S., & Padmanabhan, S. (2024) · Healthcare · doi:10.3390/healthcare12070788 · Cited by 31 · see citing works
“LLMs can summarise research findings, suggest medication alternatives based on the latest studies and even predict drug interactions by analysing extensive medical texts [19,22].”
BackgroundSavage, T., Ma, S. P., Boukil, A., Rangan, E. S., Patel, V., López, I., & Chen, J. H. (2025) · Journal of Medical Internet Research · doi:10.2196/76048 · Cited by 18 · see citing works
“Summarization helps clinicians condense lengthy documentation into concise, high-yield summaries to support faster chart review [17].”
AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization
Afzal, A., Chalumattu, R., Matthes, F., et al. (2024) · CUSTOMNLP4U · doi:10.48550/arxiv.2407.11591 · Cited by 21
“Van Veen et al. (2023) focus on clinical data and tackle the summarization of electronic health records.”
BackgroundTowards a Robust Retrieval-Based Summarization System
Liu, S., Wu, J., Bao, J., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2403.19889 · Cited by 18
A survey on biomedical automatic text summarization with large language models
Huang, Z., Chen, X., Wang, Y., Huang, J., & Zhao, X. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104216 · Cited by 23 · see citing works
SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMs
Hu, Y., Song, K.-Q., Cho, S., Wang, X.-Y., Foroosh, H., Yu, D., & Liu, F. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.10979 · Cited by 18
“…for both teams are indicated in dotted circles and are withheld from LLMs. text and structured data (Adams et al., 2021; Bard-han et al., 2022; Veen et al., 2023; Ben Abacha et al., 2023); LLM Assistants for online shopping need to process product catalogs, sales transactions, and customer…”
BackgroundLi, J., & Maiti, A. (2025) · Future Internet · doi:10.3390/fi17030100 · Cited by 12 · see citing works
“LLMs are excellent at summarizing text [47–50].”
The use of large language models to enhance cancer clinical trial educational materials
Gao, M., Varshney, A., Chen, S., Goddla, V., Gallifant, J., Doyle, P., Novack, C., Dillon-Martin, M., Perkins, T. A., Correia, X., et al. (2025) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkaf021 · Cited by 17 · see citing works
Text summarization with ChatGPT for drug labeling documents
Ying, L., Liu, Z., Fang, H., Kusko, R. L., Wu, L., Harris, S., & Tong, W. (2024) · Drug Discovery Today · doi:10.1016/j.drudis.2024.104018 · Cited by 18 · see citing works
Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation
Wang, H., Gao, C., Liu, B., Xu, Q., Hussein, G., Labban, M. E., Iheasirim, K., Korsapati, H. R., Outcalt, C., & Sun, J. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.626 · Cited by 1 · see citing works
“We used Gemini 1.0 Pro as the teacher model in this study. achieved using GPT-4 along with few-shot in-context learning (Giorgi et al., 2023; Van Veen et al., 2023).”
InfluentialWu, H., Boulenger, P., Faure, A., et al. (2024) · Workshop on Biomedical Natural Language Processing · doi:10.18653/v1/2024.bionlp-1.61 · Cited by 15
“The appearance of large language models (LLMs) has brought unprecedented changes (Achiam et al., 2023; Ouyang et al., 2022), and demonstrated potential strong capabilities in clinical text summarization (Van Veen et al., 2023).”
BackgroundPeng, C., Zhang, K., Lyu, M., Liu, H., Sun, L., & Wu, Y. (2025) · Journal of Biomedical Informatics · doi:10.48550/arxiv.2505.17436 · Cited by 15
“Large language models (LLMs) such as GPT-4[7], LlaMA[8], and Mistral[9] have been widely used to speed up new treatment development[10] and reduce the burden of healthcare providers through generative tasks such as clinical note summarization[11] and question answering[12].”
Susceptibility of Large Language Models to User-Driven Factors in Medical Queries
Lim, K. H., Kang, U., Li, X., Kim, J. S., Jung, Y., Park, S. J., & Kim, B.-H. (2025) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-025-00218-4 · Cited by 8 · see citing works
Chen, Y.-N., Li, J., Song, C.-W., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2409.06164 · Cited by 14
“For example, Van Veen et al. 37 explored the application of LLMs in clinical text summarization, covering various tasks such as summarizing radiology reports and doctor-patient dialogues.”
BackgroundExploring the challenges faced by Dutch truck drivers in the era of technological advancement
de Winter, J., Driessen, T., Dodou, D., & Cannoo, A. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1352979 · Cited by 12 · see citing works
“In summarization applications, ChatGPT’s capabilities have been shown in various fields (51), including clinical texts (52) and news items (53).”
BackgroundAligning (Medical) LLMs for (Counterfactual) Fairness
Poulain, R., Fayyaz, H., & Beheshti, R. (2024) · arXiv.org · doi:10.48550/arxiv.2408.12055 · Cited by 13
“With their capacity for natural language understanding, they have shown promise in applications like summarizing medical notes, answering patient questions, and generating discharge letters [2].”
BackgroundLyu, M., Peng, C., Paredes, D., et al. (2024) · Workshop on Biomedical Natural Language Processing · doi:10.48550/arxiv.2407.15359 · Cited by 13
“Additionally, excessive tokens can overwhelm LLM’s capacity to focus on important patient information, affecting both the quality and coherence of the generated summaries (Van Veen et al., 2023).”
BackgroundChen, Y., Li, K., Bao, W., Patel, D., Kong, Y., Min, M. R., & Metaxas, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73007-8_12 · Cited by 5 · see citing works
“Extracting Procedure Steps Using LLM. Recent studies demonstrate that LLMs possess a strong capability for text summarization [33, 43] and understanding task procedures [1,12,49].”
BackgroundMethodGrazhdanski, G., Vasilev, V., Vassileva, S., Taskov, D., Antova, I., Koychev, I., & Boytcheva, S. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104906 · Cited by 8 · see citing works
Campos, V. M. S., Prudente, T. P., Leão, L. L., Costa, M., Oliva, H. N. P., & Monteiro-Júnior, R. S. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00368-0 · Cited by 12 · see citing works
“These labor savings can translate into real economic gains [56, 57].”
Learning to summarize user information for personalized reinforcement learning from human feedbackPreprint
Nam, H., Wan, Y., Liu, M., Ahnn, P., Lian, J., & Jaques, N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.13579
“Finetuning LLMs for improved and domain-specific summarization has long been of interest [27, 35, 31, 28, 24, 29].”
Aly, W., Soliman, T., & AbdelAziz, A. (2025) · arXiv.org · doi:10.48550/arxiv.2507.05123 · Cited by 10
“The authors in [48] applied an adaptation of eight open-source and proprietary LLMs for four distinct summarization tasks comprising six datasets in the clinical domain.”
Revisiting the MIMIC-IV Benchmark: Experiments Using Language Models for Electronic Health Records
Lovón-Melgarejo, J., Ben-Haddi, T., Scala, J. D., et al. (2024) · CL4HEALTH · doi:10.48550/arxiv.2504.20547 · Cited by 10
“For example, electronic health records (EHR), composed of non-linguistic information such as laboratory measurements, procedures, and medication codes, are translated into linguistic reports using these models (Van Veen et al., 2023).”
Hegselmann, S., Shen, S. Z., Gierse, F., Agrawal, M., Sontag, D., & Jiang, X. (2024) · arXiv · doi:10.48550/arxiv.2402.15422 · Cited by 6 · see citing works
“Given the large burden of and repetition within clinical documentation, automated clinical summarization is of great interest across healthcare; generation targets include discharge summaries, radiology impression sections, patient-facing instructions, and problem lists (Van Veen et al., 2023b; Adams et al., 2022).”
BackgroundResultInfluentialSubramanian, A., Schlegel, V., Kashyap, A. R., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2406.03699 · Cited by 9
“Examples of tasks include summarization of clinical text (Veen et al., 2023), automatic note generation for physicians (Ben Abacha et al., 2023b), and condensation of doctor-patient dialogues (Ben Abacha et al., 2023a; Toma et al., 2023).”
BackgroundFIN2SUM: Advancing AI-Driven Financial Text Summarization with LLMs
Wilson, E., Saxena, A., Mahajan, J., et al. (2024) · International Conference on Trends in Quantum Computing and Emerging Business Technologies (TQCEBT) · doi:10.1109/tqcebt59414.2024.10545078 · Cited by 6 · see citing works
Error Detection in Medical Note through Multi Agent Debate
Maiga, A., Shah, A. D., & Yılmaz, E. (2025) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2025.bionlp-1.12 · Cited by 1 · see citing works
“In some task like medical summarization (Van Veen et al., 2023), adapted model can even surpass medical experts (Van Veen et al., 2024).”
Sousa, M. S. A., Peiris, S., Figueiró, M. F., Haby, M. M., Baraldi, A. C., Revéiz, L., & Souza, J. P. (2026) · Systematic Reviews · doi:10.1186/s13643-025-02842-y · Cited by 11 · see citing works
AI-driven report-generation tools in mental healthcare: A review of commercial tools
Bouguettaya, A., Victoria Team, Stuart, E. M., & Aboujaoude, E. N. (2025) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2025.02.018 · Cited by 10 · see citing works
Cl2sum: abstractive summarization via contrastive prompt constructed by LLMs hallucination
Huang, X., Nong, Q., Wang, X., Zhang, H., Du, K., Yin, C., Li, Y., Yan, B., & Zhang, X. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01795-y · Cited by 7 · see citing works
“In the medical domain, the work of Van et al. also demonstrated that cross-multitasking clinical text summarization based on LLMs outperforms human experts [6].”
Liu, D., Hu, X., Xiao, C., Bai, J., Barandouzi, Z. A., Lee, S., Webster, C., Brock, L.-U. B., Lee, L., Bold, D., et al. (2025) · JMIR Cancer · doi:10.2196/67914 · Cited by 8 · see citing works
“Study by Veen et al [53] assessed approaches for LLMs to summarize clinical texts.”
An Efficient Text Cleaning Pipeline for Clinical Text for Transformer Encoder Models
Ridoy, S. Z., Sultana, J., Ria, Z. F., et al. (2024) · IEEE International Conference on Intelligent Systems · doi:10.1109/is61756.2024.10705199 · Cited by 8
“The study by Veen et al. [13] highlights the methods and substantial efforts that are used in the field of clinical text summarization.”
BackgroundA Comprehensive Review of Large Language Models in Abstractive Summarization of News Articles
G, B., M, A., Yashaswini, G., et al. (2024) · 2024 Asia Pacific Conference on Innovation in Technology (APCIT) · doi:10.1109/apcit62007.2024.10673650 · Cited by 8
“Incorrelation to our study, [16] demonstrates the effectiveness of LLM’s in clinical text summarization tasks, surpassing human experts in completeness and correctness of summaries across various clinical domains.”
MethodSATS: simplification aware text summarization of scientific documents
Zaman, F., Kamiran, F., Shardlow, M., Hassan, S., Karim, A., & Aljohani, N. R. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1375419 · Cited by 5 · see citing works
Gui, X., Lv, H., Wang, X., Lv, L., Xiao, Y., & Wang, L. (2025) · BioData Mining · doi:10.1186/s13040-025-00458-5 · Cited by 7 · see citing works
“Domain-specific LLMs, fine-tuned on medical corpora or using knowledge graph embeddings, perform well in clinical text summarization [10, 11], note generation [12, 13], and medical question answering [14].”
Large language model performance versus human expert ratings in automated suicide risk assessment
Thomas, J., Elyoseph, Z., Kuchinke, L., & Meinlschmidt, G. (2025) · Scientific Reports · doi:10.1038/s41598-025-22402-7 · Cited by 7 · see citing works
Chu, B.-H., Li, M.-J., Frihat, S., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.43 · Cited by 7
“Unlike previous work (Van Veen et al., 2023), which assigns an overall score, our approach emphasizes identifying which specific salient information is retained or omitted.”
Balch, J. A., Desaraju, S., Nolan, V. J., Vellanki, D., Buchanan, T. R., Brinkley, L. M., Penev, Y. P., Bilgili, A., Patel, A., Chatham, C. E., et al. (2025) · JMIR Medical Informatics · doi:10.2196/71176 · Cited by 5 · see citing works
“They have been used in the well-studied NLP tasks of medical questioning and answering [25,39-41], summarization [16,23,42], named-entity recognition [30,43-46], and document classification [23,47-49].”
Cheese, E., Bichoo, R. A., Grover, K., Dumitru, D., Zenonos, A., Groark, J., Gibson, D. F. C., & Pope, R. A. (2025) · Journal of Medical Internet Research · doi:10.2196/70971 · Cited by 8 · see citing works
“…used the text-to-text transfer transformer (T5) model to generate layperson-friendly summaries of clinical radiology notes, while Van Veen et al [45] evaluated various LLMs across a range of clinical text summarization tasks and found that fine-tuned language net–T5 (FLAN-T5) and GPT-4…”
Hier, D. B., Carrithers, M. D., Platt, S. K., Nguyen, A. Q., Giannopoulos, I., & Obafemi-Ajayi, T. (2025) · Information · doi:10.3390/info16060446 · Cited by 12 · see citing works
PRESS: Defending Privacy in Retrieval-Augmented Generation via Embedding Space Shifting
He, J., Liu, C., Hou, G., Jiang, W., & Li, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887843 · Cited by 3 · see citing works
“Retrieval-augmented generation (RAG) [1]–[3] is an advanced technique that enhances large language models (LLMs) by dynamically retrieving information from external data relevant to input queries.”
Word Matters: What Influences Domain Adaptation in Summarization?
Li, Y.-H., Miao, S., Huang, H.-Y., & Gao, Y. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2406.14828 · Cited by 7
SPEER: Sentence-Level Planning of Long Clinical Summaries via Embedded Entity Retrieval
Adams, G., Zucker, J., & Elhadad, N. (2024) · arXiv.org · doi:10.48550/arxiv.2401.02369 · Cited by 7
Large Language Models in Spine Surgery: A Promising Technology
Shah, R., & Schwab, J. H. (2025) · HSS Journal®: The Musculoskeletal Journal of Hospital for Special Surgery · doi:10.1177/15563316251340696 · Cited by 5 · see citing works
He, Y., Yamagiwa, H., & Shimodaira, H. (2024) · Workshop on Biomedical Natural Language Processing · doi:10.48550/arxiv.2406.18094 · Cited by 6
“With this in mind, there has been active research in recent years into applying natural language processing (NLP) to EHR to reduce the burden on end-users (Dong et al., 2022; Houssein et al., 2023; Veen et al., 2023).”
BackgroundRepresentational Alignment Supports Effective Machine Teaching
Sucholutsky, I., Collins, K. M., Malaviya, M., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2406.04302 · Cited by 6
Extractive Text Summarization of Clinical Text Using Deep Learning Models
Chandra Shekar, G., Sai Teja, K., Nithin Datta, D., Geetha Sri Abhinay, P., & Mudigonda, K. S. P. (2024) · International Conference on Emerging Trends in Information Technology and Engineering (ICETITE) · doi:10.1109/ic-etite58242.2024.10493738 · Cited by 5 · see citing works
AUTOGEN and the Ethics of Co-Creation with Personalized LLMs—Reply to the Commentaries
Mann, S. P., Earp, B. D., Møller, N., Suren, V., & Savulescu, J. (2024) · The American Journal of Bioethics · doi:10.1080/15265161.2024.2308175 · Cited by 8 · see citing works
“Research on summarization tasks has found similar or greater performance for LLMs as compared to humans in both specialist and general domains (Van Veen et al. 2023; Zhang et al. 2023).”
BackgroundLi, S., Zhang, Y., Deng, C., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2024.findings-acl.59 · Cited by 6
“6 Van Veen et al. (2023b) select 126 samples containing an “assessment and plan” section from the dataset.”
BackgroundInfluentialAutomating evaluation of LLM-generated responses to patient questions about rare diseases
Zhao, M., Oh, I. Y., Gupta, A., Cohen‐Cutler, S., Harmoney, K. M., Lai, A. M., & Sisk, B. A. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag054 · Cited by 4 · see citing works
“To address this gap, we systematically assessed both traditional NLP metrics and LLM-based methods for evaluating LLM responses to patient questions about CLAs.”
ArgCMV: An Argument Summarization Benchmark for the LLM-era
Gurjar, O., Goyal, A., & Chandrasekharan, E. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2508.19580 · Cited by 5
“Beyond argument summarization, LLMs have been extensively used for summarizing news articles (Zhang (Tang et al., 2023; Van Veen et al., 2023), books (Chang et al., 2023), and dialogues (Tian et al., 2024; Yin et al., 2024; Zhu et al., 2024).”
SpecialtyScribe: Enhancing SOAP note Scribing for Medical Specialties using LLMs
Rastogi, E., Goyal, S., Zhao, F., & Yuan, D. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3706131 · Cited by 1 · see citing works
“With the emergence of LLMs, recent works (Van Veen et al., 2023; Biswas and Talukdar, 2024; Goyal et al., 2024) have focused on leveraging or prompting powerful private LLMs, such as GPT-4 and MedPaLM, to enhance medical note generation.”
Voigt, K. R., Sun, Y., Patandin, A., Hendriks, J., Goossens, R., Verhoef, C. K., Husson, O., Grünhagen, D. J., & Jung, J. (2025) · JMIR Cancer · doi:10.2196/58834 · Cited by 4 · see citing works
“As an example, a large language model, a pretrained model, can be used to interpret and summarize large quantity textual data [30].”
On the Role of LLM to Forecast the Next Pandemic
Ruga, T., Vocaturo, E., & Zumpano, E. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822835 · Cited by 3 · see citing works
Evaluation of Medical Large Language Models: Taxonomy, Review, and Directions
Lacerda, A. M., Pappa, G. L., Pereira, A. C. M., Meira, W., & Barros, A. G. D. A. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/1169
“Additionally, the authors in [Van Veen and others, 2023] show that LLM-generated reports tend to be less concise compared to human-written ones.”
Chrysafi, P., Lam, B. D., Carton, S., & Patell, R. D. (2024) · Hämostaseologie · doi:10.1055/a-2415-8408 · Cited by 8 · see citing works
“5 compared favorably to human experts at summarizing clinical text, 89,90 while Ge et al showed that GPT-4 was able to automatically extract key information from radiology reports.”
Kumar, B., Amar, J., Yang, E., et al. (2024) · Machine Learning in Health Care · doi:10.48550/arxiv.2405.06093 · Cited by 5
Çelikten, T., & Onan, A. (2025) · IEEE Access · doi:10.1109/access.2025.3604351 · Cited by 6 · see citing works
“…summaries produced by LLMs should not be of inferior quality to summaries produced by experienced clinicians; this is particularly important when making sensitive clinical decisions where accuracy, clarity, and reliability are crucial for ensuring patient safety and effective medical practice [8].”
Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation (RAG)
Qi, Z.-S., Sahu, U., Ma, L., et al. (2026) · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3770855.3817524 · Cited by 4
OralGPT: A Two-Stage Vision-Language Model for Oral Mucosal Disease Diagnosis and Description
Zhang, J., Du, B.-D., Miao, Y., et al. (2025) · doi:10.48550/arxiv.2510.13911 · Cited by 4
“Similar designs have shown cross-task advantages in medical visual question answering (Med-VQA [49]) and clinical text summarization (ClinicalDocSum [50]).”
Investigating Symbolic Triggers of Hallucination in Gemma Models Across HaluEval and TruthfulQA
Lamba, N., Tiwari, S., & Gaur, M. (2025) · International Conference on Semantic Systems · doi:10.48550/arxiv.2509.09715 · Cited by 4
Organ-Agents: Virtual Human Physiology Simulator via LLMs
Chang, R., Jiao, H., Nie, W., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2508.14357 · Cited by 4
“Summarization-focused studies, such as those by Hom et al., demonstrate LLMs outperforming clinicians in note condensation [21], and models like ChatGPT have been preferred over physicians in patient-facing QA due to perceived empathy [22].”
Goldberg, E. M., Macis, M., Bounds, M., Picazo, J. G., & Nicholas, L. H. (2025) · JMIR Aging · doi:10.2196/76335 · Cited by 5 · see citing works
“In particular, our methods were shaped by studies by Nashwan and Abukhadijah [13] and Van Veen et al [14], who provided prompt examples.”
Clinical Text Summarization using NLP Pretrained Language Models: A Case Study of MIMIC-IV-Notes
Arokodare, O., Wimmer, H., & Du, J. (2025) · Journal of Information Systems Applied Research and Analytics · doi:10.62273/naka3054 · Cited by 2 · see citing works
Weerasinghe, J., Seepersaud, O., Smothers, G., et al. (2025) · Applied Sciences · doi:10.3390/app15052467 · Cited by 3 · see citing works
Evaluating LLMs for Diagnosis Summarization
Santos, J., dos Santos, H. D. P., Ulbrich, A. H. D. P. S., Faccio, D., Tabalipa, F. D. O., Nogueira, R. F., & Costa, M. M. (2024) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc53108.2024.10782231 · Cited by 5 · see citing works
“In the summarization of clinical notes, LLMs have demonstrated the capability to outperform human experts [19], showcasing their potential to reduce documentation burdens and enable clinicians to prioritize personalized patient care.”
DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries
Boll, H. O., Boll, A. O., Boll, L. P., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2506.16777 · Cited by 1
“Large language models (LLMs) have shown increasing promise for automating clinical note summarization, with recent studies indicating they perform at least as well as, and often better than, human experts (Veen et al., 2023; Schoonbeek et al., 2024).”
Natural Language Processing in Healthcare: From Unstructured Data to Clinical Intelligence
Kumar, D., Choudhary, N., Gupta, V., Tanwar, R., Bahmani, K., Antal, S., Jarora, S., Gupta, S., Shaikh, M. S., Webster, T. J., et al. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200698 · Cited by 2 · see citing works
Discriminative Representation Learning for Clinical Prediction
Zhang, Y., Fan, L., Lawrence, S., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.20921 · Cited by 3
“Continued pretraining on domain-specific corpora [109, 110, 111, 112, 113, 114] enables improved semantic reasoning over clinical narratives.”
- Authors
- Veen, D. V., Uden, C. V., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerova, A., Rohatgi, N., Hosamani, P., Collins, W., Ahuja, N., Langlotz, C., Hom, J., Gatidis, S., Pauly, J., & Chaudhari, A.
- Affiliations
- Stanford University (US)
- The University of Texas at Austin (US)
- University Hospital Zurich
- Copenhagen University Hospital (DK)
- University of Zurich (CH)
- Hospital Israelita Albert Einstein (BR)
- Stanford University
- Stanford Medicine (US)
- Corresponding author
- Veen, Dave Van
- ORCID
- Venue
- Nature Medicine
- Published
- 2023-10-30
- Volume
- 30
- Issue
- 4
- Pages
- 1134-1142
- ISSN
- 1078-8956 (ISSN-L)
- PMC
- PMC10635391
- OpenAlex
- W4392193048
- Cited by
- 892
- References
- 71
- Citations by year
- 2026: 342 · 2025: 404 · 2024: 144 · 2023: 2
- FWCI
- 81.57
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Nature Medicine (publishedVersion)
- Research Square (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Zurich Open Repository and Archive (University of Zurich)
- Funders
- Foundation for the National Institutes of Health (R01 AR077604, R01EB002524, P41 EB027060, R01 AR079431)
- Gordon and Betty Moore Foundation
- National Institutes of Health (75N92020C00021, 75N92020C00008)
- Agency for Healthcare Research and Quality (R18 HS026886)
- National Heart, Lung, and Blood Institute (75N92020C00008, 75N92020C00021, R01HL155410, R01HL169345, R01HL167974, R01HL157235)
- National Institute of Arthritis and Musculoskeletal and Skin Diseases (R01AR077604, R01AR079431)
- National Institute of Biomedical Imaging and Bioengineering (75N92020C00021, EB 002524, 75N92020C00008, P41EB027060, R01EB002524)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, clinical text summarization, LLM adaptation, electronic health records, radiology reports, progress notes, doctor-patient dialogue, patient questions, summary quality, safety analysis, clinical documentation burden
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
