A dataset and benchmark for hospital course summarization with adapted large language models
Open accessAbstract
Objective: Brief hospital course (BHC) summaries are clinical documents that summarize a patient’s hospital stay. While large language models (LLMs) depict remarkable capabilities in automating real-world tasks, their capabilities for healthcare applications such as synthesizing BHCs from clinical notes have not been shown. We introduce a novel preprocessed dataset, the MIMIC-IV-BHC, encapsulating clinical note and BHC pairs to adapt LLMs for BHC synthesis. Furthermore, we introduce a benchmark of the summarization performance of 2 general-purpose LLMs and 3 healthcare-adapted LLMs. Materials and Methods: Using clinical notes as input, we apply prompting-based (using in-context learning) and fine-tuning-based adaptation strategies to 3 open-source LLMs (Clinical-T5-Large, Llama2-13B, and FLAN-UL2) and 2 proprietary LLMs (Generative Pre-trained Transformer [GPT]-3.5 and GPT-4). We evaluate these LLMs across multiple context-length inputs using natural language similarity metrics. We further conduct a clinical study with 5 clinicians, comparing clinician-written and LLM-generated BHCs across 30 samples, focusing on their potential to enhance clinical decision-making through improved summary quality. We compare reader preferences for the original and LLM-generated summary using Wilcoxon signed-rank tests. We further request optional qualitative feedback from clinicians to gain deeper insights into their preferences, and we present the frequency of common themes arising from these comments. Results: The Llama2-13B fine-tuned LLM outperforms other domain-adapted models given quantitative evaluation metrics of Bilingual Evaluation Understudy (BLEU) and Bidirectional Encoder Representations from Transformers (BERT)-Score. GPT-4 with in-context learning shows more robustness to increasing context lengths of clinical note inputs than fine-tuned Llama2-13B. Despite comparable quantitative metrics, the reader study depicts a significant preference for summaries generated by GPT-4 with in-context learning compared to both Llama2-13B fine-tuned summaries and the original summaries (P<.001), highlighting the need for qualitative clinical evaluation. Discussion and Conclusion: We release a foundational clinically relevant dataset, the MIMIC-IV-BHC, and present an open-source benchmark of LLM performance in BHC synthesis from clinical notes. We observe high-quality summarization performance for both in-context proprietary and fine-tuned open-source LLMs using both quantitative metrics and a qualitative clinical reader study. Our research effectively integrates elements from the data assimilation pipeline: our methods use (1) clinical data sources to integrate, (2) data translation, and (3) knowledge creation, while our evaluation strategy paves the way for (4) deployment.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
On MIMIC-IV-Ext-BHC, instruction_ep5 has first rank 26.3%.
“| instruction_ep5 | 26.3% | | | |”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Aali et al., 2024
The Dataset section describes MIMIC-IV-BHC, and the Model performance analysis section reports other performance comparisons, but neither gives a first-rank percentage for instruction_ep5 on MIMIC-IV-Ext-BHC. Searches for “instruction_ep5,” “26.3,” and the dataset and ranking terms found no passage addressing that result. (LLM, conf=0.76)
Experimental — accuracy not yet measuredIn the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between instruction_ep5 and lm_ep2 is 1.712586e-07.
“| instruction_ep5 | 0.872852 | 1.0 | 1.712586e-07 | 0.000039 |”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Aali et al., 2024
The Dataset section describes MIMIC-IV-BHC, while Statistical analysis describes Wilcoxon signed-rank tests for clinician preferences; neither reports the claimed Friedman posthoc comparison. Searches for “1.712586e-07,” “instruction_ep5,” “lm_ep2,” and “MIMIC-IV-Ext-BHC” found no matching result in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredIn the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between lm_ep2 and lm_ep5 is 0.723773.
“| lm_ep2 | 0.000010 | 1.712586e-07 | 1.0 | 0.723773 |”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Aali et al., 2024
The Statistical analysis section describes Wilcoxon signed-rank tests, not a Friedman posthoc comparison. Searches for 0.723773, lm_ep2, lm_ep5, and Friedman found no passage reporting the claimed p-value; the paper’s Dataset section describes MIMIC-IV-BHC, but does not establish this comparison. (LLM, conf=0.78)
Experimental — accuracy not yet measuredIn the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between instruction_ep2 and lm_ep2 is 9.882236e-06.
“| instruction_ep2 | 1.0 | 8.728525e-01 | 9.882236e-06 | 0.001026 |”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Aali et al., 2024
The Dataset section describes MIMIC-IV-BHC, but does not report the claimed model comparison or p-value. The Statistical analysis section describes Wilcoxon signed-rank tests; searches for the exact p-value, model labels, and Friedman posthoc test found no passage reporting this result. Page numbers were unavailable in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredOn MIMIC-IV-Ext-BHC, lm_ep5 has first rank 12.6%.
“| lm_ep5 | 12.6% | | | |”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Aali et al., 2024
The Dataset and Model performance analysis sections describe MIMIC-IV-BHC and benchmarked models, but neither reports a first-rank result for lm_ep5. Searches for “lm_ep5,” “12.6%,” and first-rank results found no passage supporting the stated percentage. (LLM, conf=0.72)
Experimental — accuracy not yet measured
On MIMIC-IV-Ext-BHC, lm_ep2 has first rank 37.4%.
“| lm_ep2 | 37.4% | | | |”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Aali et al., 2024
The Dataset section describes MIMIC-IV-BHC, but the passages from our copy do not report a result for lm_ep2 on MIMIC-IV-Ext-BHC. Searches for the model name, dataset name, first-rank measure, and 37.4% found no matching result. (LLM, conf=0.72)
Experimental — accuracy not yet measuredOn MIMIC-IV-Ext-BHC, instruction_ep2 has first rank 23.7%.
“| instruction_ep2 | 23.7% | | | |”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Aali et al., 2024
The Dataset section describes MIMIC-IV-BHC, and the Model performance analysis section reports results for adaptation strategies, but neither identifies MIMIC-IV-Ext-BHC or reports a first-rank percentage for instruction_ep2. Searches for “instruction_ep2” and “23.7% first rank” found no passage supporting the attributed result. (LLM, conf=0.76)
Experimental — accuracy not yet measuredIn the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between instruction_ep2 and lm_ep5 is 0.001026.
“| instruction_ep2 | 1.0 | 8.728525e-01 | 9.882236e-06 | 0.001026 |”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Aali et al., 2024
The Dataset section identifies MIMIC-IV-BHC, while the Statistical analysis section describes Wilcoxon signed-rank tests, not the stated Friedman posthoc comparison. Searches for 0.001026, instruction_ep2, lm_ep5, and MIMIC-IV-Ext-BHC found no passage reporting the claimed p-value. (LLM, conf=0.76)
Experimental — accuracy not yet measuredIn the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between instruction_ep2 and instruction_ep5 is 8.728525e-01.
“| instruction_ep2 | 1.0 | 8.728525e-01 | 9.882236e-06 | 0.001026 |”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Aali et al., 2024
The Dataset section describes MIMIC-IV-BHC, not MIMIC-IV-Ext-BHC. The Statistical analysis section describes Wilcoxon signed-rank tests, but our copy provides no Friedman posthoc comparison between instruction_ep2 and instruction_ep5 or the stated p-value (searched for the model names, test, dataset, and value). Neither section has page numbers in our copy. (LLM, conf=0.75)
Experimental — accuracy not yet measuredIn the Friedman posthoc test on MIMIC-IV-Ext-BHC, the p-value for the comparison between instruction_ep5 and lm_ep5 is 0.000039.
“| instruction_ep5 | 0.872852 | 1.0 | 1.712586e-07 | 0.000039 |”
From AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment (2025) · cites this paper as Aali et al., 2024
The Dataset and Adaptation strategies sections describe MIMIC-IV-BHC and the models’ adaptation methods, but do not report a Friedman posthoc comparison. Searches for “instruction_ep5,” “lm_ep5,” “0.000039,” and “MIMIC-IV-Ext-BHC” found no passage establishing the claimed p-value. Page numbers were not available in our copy. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Kripalani, S., LeFevre, F., Phillips, C. O., Williams, M. V., Basaviah, P., & Baker, D. W. (2007) · JAMA · doi:10.1001/jama.297.8.831 · Cited by 2,103 · see citing works
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891 · see citing works
Self-citationQLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 507 · see citing works
Lyu, Q., Tan, J., Zapadka, M. E., Ponnatapura, J., Niu, C., Myers, K. J., Wang, G., & Whitlow, C. T. (2023) · Visual Computing for Industry, Biomedicine, and Art · doi:10.1186/s42492-023-00136-5 · Cited by 367 · see citing works
Moy, A. J., Schwartz, J. M., Chen, R., Sadri, S., Lucas, E., Cato, K. D., & Rossetti, S. C. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa325 · Cited by 284 · cited 1× · see citing works
“Clinicians spend significant time on clinical documentation [1–3], a crucial component of data assimilation [4] beginning with data sources such as EHRs and progressing through data translation and knowledge creation where these records can be processed for clinically-relevant machine learning…”
DRG-LLaMA : tuning LLaMA model to predict diagnosis-related group for hospitalized patients
Wang, H., Gao, C., Dantona, C., Hull, B., & Sun, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00989-3 · Cited by 99 · cited 1× · see citing works
“Other works [40, 41] further explore domain adaptation of LLMs in the clinical domain through fine-tuning for improved performance in medical tasks.”
Flagged first, then by citations
Research works printed but not linked: 33
- Chaiyachati KH, Shea JA, Asch DA, et al. Assessment of inpatient time allocation among first-year internal medicine residents using time-motion observations. JAMA Intern Med. 2019;179:760-767.
- Mamykina L, Vawdrey DK, Hripcsak G. How do residents spend their shift time? A time and motion study with a particular focus on the use of computers. Acad Med. 2016;91:827-832.
- Albers D, Behn CD, Hripcsak G. Data assimilation in medicine. SIAM News. 2020. Accessed August 12, 2024. https://dsweb.siam. org/The-Magazine/All-Issues/data-assimilation-in-medicine 05. Clough RAJ, Sparkes WA, Clough OT, et al. Transforming health- care documentation: harnessing the potential of AI to generate dis- charge summaries. BJGP Open. 2024;8:BJGPO.2023.0116.
- Patel SB, Lam K. ChatGPT: the future of discharge summaries? Lancet Digit Health. 2023;5:e107-e108.
- Warraich HJ, Tazbaz T, Califf RM. FDA perspective on the regu- lation of artificial intelligence in health care and biomedicine. JAMA. 2024;331:1443-1450.
- Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Adv Neural Inf Process Syst. 2017;30:5998-6008.
- Radford A, Wu J, Child R, et al. Language models are unsuper- vised multitask learners. OpenAI Blog. 2019;1:9.
- Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877-1901.
- Zhang S, Dong L, Li X, et al. Instruction tuning for large language models: a survey. arXiv. 2024;arXiv:2308.10792. · arXiv
- Wang B, Xie Q, Pei J, et al. Pre-trained language models in bio- medical domain: a systematic survey. ACM Comput Surv. 2023;56:1-52.
- Adams G, Alsentzer E, Ketenci M, et al. What's in a summary? Laying the groundwork for advances in hospital-course summari- zation. Proc Conf North Am Chapter Assoc Comput Linguist. 2021;2021:4794.
- Searle T, Ibrahim Z, Teo J, et al. Discharge summary hospital course summarisation of in patient electronic health record text with clinical concept guided deep pre-trained transformer models. J Biomed Inform. 2023;141:104358.
- Hartman VC, Bapat SS, Weiner MG, et al. A method to automate the discharge summary hospital course for neurology patients. J Am Med Inform Assoc. 2023;30:1995-2003.
- Jung H, Kim Y, Choi H, et al. Enhancing clinical efficiency through LLM: discharge note generation for cardiac patients. arXiv. 2024; arXiv:2404.05144. · arXiv
- Aali A, Van Veen D, Arefeen YI, et al. MIMIC-IV-Ext-BHC: labeled clinical notes dataset for hospital course summarization. PhysioNet. 2024;1.1.0. https://doi.org/10.13026/fh2q-4148. · DOI
- Johnson AEW, Pollard TJ, Horng S, et al. MIMIC-IV-Note: dei- dentified free-text clinical notes. PhysioNet. 2023;2.2. https://doi. org/10.13026/7qgp-kc16. · DOI
- Lehman E, Johnson A. Clinical-t5: large language models built using mimic clinical text. PhysioNet. 2023;1.0.0. https://doi.org/ 10.13026/rj8x-v335. · DOI
- Lampinen AK, Dasgupta I, Chan SC, et al. Can language models learn from explanations in context? Find Assoc Comput Linguist EMNLP. 2022;537-563.
- Touvron H, Lavril T, Izacard G, et al. Llama: open and efficient foundation language models. arXiv. 2023;arXiv:2302.13971. · arXiv
- OpenAI AJ, Adler S, Agarwal S, et al. GPT-4 technical report. arXiv. 2024;arXiv:2303.08774. · arXiv
- Tay Y, Dehghani M, Tran VQ, et al. Ul2: unifying language learn- ing paradigms. In: Proceedings of the International Conference on Learning Representations. 2023.
- Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst. 2022;35:24824-24837.
- Hu EJ, Shen Y, Wallis P, et al. LoRA: low-rank adaptation of large language models. In: Proceedings of the International Conference on Learning Representations. 2022.
- Ding N, Qin Y, Yang G, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nat Mach Intell. 2023;5:220-235.
- Papineni K, Roukos S, Ward T, et al. BLEU: a method for auto- matic evaluation of machine translation. Assoc Comput Linguist. 2002;40:311-318.
- Lin C-Y. ROUGE: a package for automatic evaluation of summa- ries. Assoc Comput Linguist. 2004;1:74-81.
- Liu F, Liu Y. Exploring correlation between ROUGE and human evaluation on meeting summaries. IEEE Trans Audio Speech Lang Process. 2009;18:187-196.
- Zhang T, Kishore V, Wu F, et al. BERTScore: evaluating text gen- eration with BERT. In: Proceedings of the International Confer- ence on Learning Representations. 2020.
- Chen Z, Varma M, Wan X, et al. Toward expanding the scope of radiology report summarization to multiple anatomies and modal- ities. Assoc Comput Linguist. 2023;2:469-484.
- Singh S, Djalilian A, Ali MJ. ChatGPT and ophthalmology: exploring its potential with discharge summaries and operative notes. Semin Ophthalmol. 2023;38:503-507.
- Koh HY, Ju J, Zhang H, et al. How far are we from robust long abstractive summarization? Find Assoc Comput Linguist EMNLP. 2022:2682-2698.
- Fleming SL, Lozano A, Haberkorn WJ, et al. MedAlign: a clinician-generated dataset for instruction following with elec- tronic medical records. AAAI. 2024;38:22021-22030.
- For commercial -use, please for on the article page on our site-for further information @oup.com. Journal of the American Medical Informatics Association, 2025, 32, 470-479 https://doi.org/10.1093/jamia/ocae312 Research and Applications · DOI
Cited by
Sorted by citations
Holistic evaluation of large language models for medical tasks with MedHELM
Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04151-2 · Cited by 59 · see citing works
Self-citation
- Authors
- Aali, A., Van Veen, D., Arefeen, Y. I., Hom, J., Bluethgen, C., Reis, E. P., Gatidis, S., Clifford, N., Daws, J., Tehrani, A. S., Kim, J., & Chaudhari, A. S.
- Affiliations
- Stanford Medicine (US)
- Center for Artificial Intelligence in Medicine and Imaging, Stanford University , Palo Alto, CA 94304,
- The University of Texas at Austin (US)
- Department of Medicine, Stanford University , Stanford, CA 94304,
- Hospital Israelita Albert Einstein (BR)
- One Medical , San Francisco, CA 94111,
- Amazon (United States) (US)
- Corresponding author
- Aali, Asad
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-12-30
- Volume
- 32
- Issue
- 3
- Pages
- 470-479
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC11833472
- OpenAlex
- W4392736665
- Cited by
- 32
- References
- 10
- Citations by year
- 2026: 18 · 2025: 14
- FWCI
- 4.71
- Type
- journal article
- Language
- en
- Open access
- green
- pmc.ncbi.nlm.nih.gov
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NIH (R01 HL167974, R01 HL169345, R01 AR077604, R01 EB002524, R01 AR079431, P41 EB027060, 75N92020C00008, 75N92020C00021)
- Stanford Center for Artificial Intelligence and Medicine
- Stanford Institute for Human Centered AI
- Stanford Center for Digital Health
- Stanford Cardiovascular Institute
- Stanford Center for Precision Health and Integrated Diagnostics
- GE Healthcare, Philips and Amazon
- ProMedica Foundation
- One Medical and Stanford University
- Accelerate Foundation Models Academic Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- clinical note summarization, large language models, LLM fine-tuning, in-context learning, GPT-4, clinical decision support, clinical text generation
- MeSH terms
- Large Language Models, Humans, Natural Language Processing, Benchmarking, Electronic Health Records, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
