LongHealth: A Question Answering Benchmark with Long Clinical Documents
Open accessAbstract
Background: Recent advancements in large language models (LLMs) offer potential benefits in healthcare, particularly in processing extensive patient records. However, existing benchmarks do not fully assess LLMs' capability in handling real-world, lengthy clinical data. Methods: We present the LongHealth benchmark, comprising 20 detailed fictional patient cases across various diseases, with each case containing 5,090 to 6,754 words. The benchmark challenges LLMs with 400 multiple-choice questions in three categories: information extraction, negation, and sorting, challenging LLMs to extract and interpret information from large clinical documents. Results: We evaluated nine open-source LLMs with a minimum of 16,000 tokens and also included OpenAI's proprietary and cost-efficient GPT-3.5 Turbo for comparison. The highest accuracy was observed for Mixtral-8x7B-Instruct-v0.1, particularly in tasks focused on information retrieval from single and multiple patient documents. However, all models struggled significantly in tasks requiring the identification of missing information, highlighting a critical area for improvement in clinical data interpretation. Conclusion: While LLMs show considerable potential for processing long clinical documents, their current accuracy levels are insufficient for reliable clinical use, especially in scenarios requiring the identification of missing information. The LongHealth benchmark provides a more realistic assessment of LLMs in a healthcare setting and highlights the need for further model refinement for safe and effective clinical application. We make the benchmark and evaluation code publicly available.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent datasets by Zakka et al. target open-ended responses and LongHealth focuses on patient-related content.
“Recent advancements have been made with the introduction of new datasets by Zakka et al.16, targeting the enhancement of open-ended responses, and LongHealth27, focusing on patient-related content.”
From Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology (2025) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · cited 1× · see citing works
“Given that our benchmark consists of 400 questions, the latter approach would require a significant amount of work, making it impractical for the scope of our study.”
The future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045 · cited 1× · see citing works
“The integration of LLMs into medical settings is increasingly recognized as a promising approach to improve the efficiency of healthcare professionals [1].”
Self-citationJin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · cited 1× · see citing works
“PubMedQA’s yes/no/maybe format, while useful, does not provide the nuance required for complex clinical decision-making [7].”
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358 · cited 1× · see citing works
“LLMs offer a promising solution by enabling faster navigation through large volumes of data, assisting in the extraction of critical information [4], and efficiently summarizing lengthy clinical documents [5].”
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358 · see citing works
Self-citationMa, Y., Cao, Y., Hong, Y., & Sun, A. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.findings-emnlp.710 · Cited by 247 · cited 1× · see citing works
“Given that our benchmark consists of 400 questions, the latter approach would require a significant amount of work, making it impractical for the scope of our study.”
Konrad, T. R., Link, C. L., Shackelton, R., Marceau, L. D., Knesebeck, O. V. D., Siegrist, J., Arber, S., Adams, A., & McKinlay, J. B. (2010) · Medical Care · doi:10.1097/mlr.0b013e3181c12e6a · Cited by 167 · cited 1× · see citing works
“For example, the average consultation time in the UK and Germany is approximately 11 and 16 minutes per patient, respectively [2].”
Proceedings of the Conference on Health, Inference, and Learning
(2021) · doi:10.1145/3450439 · Cited by 93 · see citing works
A frame semantic overview of NLP-based information extraction for cancer-related EHR notes
Datta, S., Bernstam, E. V., & Roberts, K. E. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103301 · Cited by 92 · cited 1× · see citing works
“LLMs offer a promising solution by enabling faster navigation through large volumes of data, assisting in the extraction of critical information [4], and efficiently summarizing lengthy clinical documents [5].”
Length and redundancy of outpatient progress notes across a decade at an academic medical center
Rule, A., Bedrick, S., Chiang, M. F., & Hribar, M. R. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.15334 · Cited by 90 · cited 1× · see citing works
“Rule et al. [3] reported a significant 60.1% increase in the length of outpatient notes from 2018 to 2019, partly due to the use of templated notes and redundant documentation.”
InfluentialZephyr: Direct Distillation of LM AlignmentPreprint
Tunstall, L., Beeching, E., Lambert, N., Rajani, N. F., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.16944 · Cited by 80 · see citing works
Moon, S., Pakhomov, S., Liu, N., Ryan, J. O., & Melton, G. B. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001506 · Cited by 72 · cited 1× · see citing works
“Their work used the Clinical Acronym Sense Inventory (CASI) and MIMIC-III datasets, to which they provided additional annotations for their analysis [24, 25, 26].”
Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models
Shyr, C., Hu, Y., Bastarache, L., Cheng, A. C., Hamid, R., Harris, P. A., & Xu, H. (2023) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-023-00155-0 · Cited by 63 · see citing works
Chuang, Y.-N., Tang, R., Jiang, X., & Hu, X. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104606 · Cited by 46 · see citing works
Assertion Detection in Clinical Notes: Medical Language Models to the Rescue?
van Aken, B., Trajanovska, I., Siu, A., Mayrdorfer, M., Budde, K., & Loeser, A. (2021) · Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations · doi:10.18653/v1/2021.nlpmc-1.5 · Cited by 25 · see citing works
Flagged first, then by citations
3 of the 21 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
- Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. Large language models are few-shot clinical information extractors. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1998-2022, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
- Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
- Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, Jean Steiner, Itay Laish, and Amir Feder. Llms accelerate annotation for medical information extraction. In Stefan Hegselmann, Antonio Parziale, Divya Shanmugam, Shengpu Tang, Mercy Nyamewaa Asiedu, Serina Chang, Tom Hartvigsen, and Harvineet Singh, editors, Proceedings of the 3rd Machine Learning for Health Symposium, volume 225 of Proceedings of Machine Learning Research, pages 82-100. PMLR, 10 Dec 2023.
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Cited by
Sorted by citations
Ferber, D., El Nahhas, O. S. M., Wölflein, G., Wiest, I. C., Clusmann, J., Leßmann, M.-E., Foersch, S., Lammert, J., Tschochohei, M., Jäger, D., et al. (2025) · Nature Cancer · doi:10.1038/s43018-025-00991-6 · Cited by 175 · see citing works
“Recent advancements have been made with the introduction of new datasets by Zakka et al. 16 , targeting the enhancement of open-ended responses, and LongHealth 27 , focusing on patient-related content.”
Self-citationLarge Language Models Are Poor Clinical Decision-Makers: A Comprehensive BenchmarkPreprint
Liu, F., Li, Z., Zhou, H., Yin, Q., Yang, J., Tang, X., Luo, C., Zeng, M. Y., Jiang, H., Gao, Y., et al. (2024) · medRxiv · doi:10.1101/2024.04.24.24306315 · Cited by 16 · see citing works
A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry
Huang, Y., Tang, K., Chen, M., & Wang, B. (2024) · arXiv · doi:10.48550/arxiv.2404.15777 · Cited by 14 · see citing works
Aldea, M. D., Zullo, L., Levrat, V., Bennouna, J., Schneider, S., Mercier, O. M., Mougenot, E., Bergot, E., Dujon, C., Cloarec, N., et al. (2025) · Annals of Oncology · doi:10.1016/j.annonc.2025.12.006 · Cited by 6 · see citing works
A Conversational Agent for Handling Health Report Inquiries
Carchiolo, V., Malgeri, M., & Sapari, L. S. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-93598-5_15 · Cited by 2 · see citing works
Sugihara, Y., Milosavljević, A. M., Jankovskaja, S., & Falk, M. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1778786 · Cited by 1 · see citing works
“For example, in a recent benchmark, projected expenses reached ≈ $8,515 for a single evaluation configuration at 16k-token inputs, leading to the exclusion of a frontier model (90).”
A Dataset for Addressing Patient's Information Needs related to Clinical Course of Hospitalization.Preprint
Soni, S., & Demner‐Fushman, D. (2025) · PubMed · doi:10.1038/s41597-026-06639-z
“Finally, the grounded QA task itself can also be extended to full-length clinical notes 57 , whose document identifiers are preserved and released with the dataset.”
Keerthana, S. M., Gayathri, S., Jothi, B., Saranya, V., Sudhahar, T. N., & Gunasekaran, K. (2026) · Biomedical Engineering Applications Basis and Communications · doi:10.4015/s1016237226500092
Zazueta, L., Covarrubias, B. L. L., Cota, C. X. N., Vázquez-Briseño, M., Hipólito, J. I. N., Romo-Cárdenas, G. S., & Avilés-Rodríguez, G. J. (2026) · Healthcare · doi:10.3390/healthcare14131865
Eryılmaz, B., Bahn, M., Damm, H., Chu, B., Bender, N., Kleesiek, J., & Nensa, F. (2026) · doi:10.2196/preprints.109219
Islamaj, R., Leaman, R., Chan, J., Wan, N., Jin, Q., Xie, N., Wilbur, J., Tian, S., Yeganova, L. E., Lai, P.-T., et al. (2026) · Database · doi:10.1093/database/baag057
Cartridges: Lightweight and general-purpose long context representations via self-study
Eyuboglu, S., Ehrlich, R., Arora, S., Guha, N., Zinsley, D., Liu, E., Tennien, W., Rudra, A., Zou, J., Mirhoseini, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.06266
- Authors
- Adams, L., Busch, F., Han, T., Excoffier, J.-B., Ortala, M., Löser, A., Aerts, H. J., Kather, J. N., Truhn, D., & Bressem, K.
- Affiliations
- TUM Klinikum (DE)
- Universitätsklinikum Aachen (DE)
- Berliner Hochschule für Technik (DE)
- Harvard University (US)
- University Hospital Heidelberg (DE)
- Corresponding author
- Busch, Felix
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Volume
- 9
- Issue
- 3
- Pages
- 280-296
- ISSN
- 2509-4971 (ISSN-L)
- arXiv
- 2401.14490
- PMC
- PMC12290132
- OpenAlex
- W4411306282
- Semantic Scholar
- 8a608d30bd3a467efdd1caa07e3e6d025ec567c5
- Cited by
- 10
- References
- 21
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 1
- FWCI
- 2.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Healthcare Informatics Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion)
- RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
- arXiv (Cornell University) (cc-by)
- Funders
- Technische Universität München
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- question answering, large language models, clinical information extraction, negation detection, medical records, GPT-3.5-turbo
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
