LiveMedBench: A Contamination-Limited Medical Benchmark for LLMs with Automated Rubric Evaluation
Open accessAbstract
The deployment of Large Language Models (LLMs) in high-stakes clinical settings demands rigorous and reliable evaluation. However, existing medical benchmarks remain static, suffering from two critical limitations: (1) data contamination, where test sets inadvertently leak into training corpora, leading to inflated performance estimates; and (2) temporal misalignment, failing to capture the rapid evolution of medical knowledge. Furthermore, current evaluation metrics for open-ended clinical reasoning often rely on either shallow lexical overlap (e.g., ROUGE) or subjective LLM-as-a-Judge scoring, both inadequate for verifying clinical correctness. % To bridge these gaps, we introduce LiveMedBench, a continuously updated, contamination-limited, and rubric-based benchmark that weekly harvests real-world clinical cases from online medical communities, ensuring strict temporal separation from model training data. We propose a Multi-Agent Clinical Curation Framework that filters raw data noise and validates clinical integrity against evidence-based medical principles. For evaluation, we develop an Automated Rubric-based Evaluation Framework that decomposes physician responses into granular, case-specific criteria, achieving substantially stronger alignment with expert physicians than LLM-as-a-Judge. % To date, LiveMedBench comprises 2,756 real-world cases spanning 38 medical specialties and two languages, paired with 16,702 unique evaluation criteria. Extensive evaluation of 38 LLMs reveals that even the best-performing model achieves only 39.2%, and 84% of models exhibit performance degradation on post-cutoff cases, confirming pervasive data contamination risks. Error analysis further identifies contextual application---not factual knowledge---as the dominant bottleneck, with 35-48% of failures stemming from the inability to tailor medical knowledge to patient-specific constraints. The code and data are available at https://github.com/ZhilingYan/LiveMedBench/ LiveMedBench.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167 · cited 2× · see citing works
“A detailed breakdown is provided in Appendix A.4.”
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
Self-citationJin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
RAGAs: Automated Evaluation of Retrieval Augmented Generation
Es, S., James, J., Espinosa Anke, L., & Schockaert, S. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2024.eacl-demo.16 · Cited by 651 · see citing works
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · neural information processing systems · doi:10.52202/075280-1240 · Cited by 590 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
Data Contamination: From Memorization to Exploitation
Magar, I., & Schwartz, R. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2203.08242 · Cited by 325
A taxonomy of generic clinical questions: classification study
Ely, J. (2000) · BMJ · doi:10.1136/bmj.321.7258.429 · Cited by 317 · cited 1× · see citing works
“Concretely, the Screener performs: (i) Narrative aggregation ( (i) Query validity ( 𝑆 𝑐𝑐 ): a binary score indicating whether 𝑄 is a clinically meaningful chief complaint (CC) as defined by Ely’s taxonomy [5].”
Towards accurate differential diagnosis with large language models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2025) · Nature · doi:10.1038/s41586-025-08869-4 · Cited by 311 · see citing works
DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.19437 · Cited by 294 · see citing works
Self-citationJin, Q., Kim, W., Chen, Q., Comeau, D. C., Yeganova, L., Wilbur, W. J., & Lu, Z. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad651 · Cited by 175 · see citing works
GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI
Chen, P., Ye, J., Wang, G.-A., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2408.03361 · Cited by 134
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16079 · Cited by 122 · see citing works
Zhao, X., Liu, S., Yang, S.-Y., & Miao, C. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714782 · Cited by 116 · cited 3× · see citing works
“A candidate case 𝐷 is retained if and only if 𝐾𝑒𝑒𝑝 ( 𝐷 ) ⇐⇒ ( 𝑆 𝑐𝑐 = 1 ) ∧ ( 𝑆 𝑖𝑛𝑓 𝑖𝑛𝑓 ) ∧ ( 𝑆 𝑎𝑙𝑖𝑔𝑛 𝑎𝑙𝑖𝑔𝑛 ) , where 𝜏 𝑖𝑛𝑓 = 0 .”
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 111 · see citing works
Data Contamination: From Memorization to Exploitation
Magar, I., & Schwartz, R. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.18 · Cited by 88 · see citing works
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.10671 · Cited by 63 · see citing works
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
Cai, Y., Wang, L., Wang, Y., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2312.12806 · Cited by 57
Sellergren, A., Kazemzadeh, S., Jaroensri, T., Kiraly, A., Traverse, M., Kohlberger, T., Xu, S., Jamil, F., Hughes, C., Lau, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.05201 · Cited by 49 · see citing works
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
Cai, Y., Wang, L., Wang, Y., De Melo, G., Zhang, Y., Wang, Y., & He, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29723 · Cited by 42 · see citing works
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., & Stoica, I. (2024) · arXiv · doi:10.48550/arxiv.2403.07974 · Cited by 42 · see citing works
CMB: A Comprehensive Medical Benchmark in Chinese
Wang, X., Chen, G., Dingjie, S., Zhang, Z., Chen, Z., Xiao, Q., Chen, J., Jiang, F., Li, J., Wan, X., et al. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.343 · Cited by 38 · see citing works
HealthBench: Evaluating Large Language Models Towards Improved Human Health
Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.08775 · Cited by 28 · see citing works
Singh, A., Fry, A., Perelman, A., Tart, A., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., Ananthram, A., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.03267 · Cited by 26 · see citing works
GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI
Chen, P., Ye, J., Wang, G., et al. (2024) · neural information processing systems · doi:10.52202/079017-2992 · Cited by 25 · see citing works
LiveBench: A Challenging, Contamination-Limited LLM Benchmark
White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Dey, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.19314 · Cited by 20 · see citing works
Ouyang, Z., Qiu, Y., Wang, L., De Melo, G., Zhang, Y., Wang, Y., & He, L. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.480 · Cited by 11 · see citing works
Leong, C. E., Kallis, L., Kuhn, I. L., Martin, G. P., & Fritz, Z. B. M. (2025) · Clinical Medicine · doi:10.1016/j.clinme.2025.100518 · Cited by 4 · cited 1× · see citing works
“Clinical guidelines and health policies are frequently updated, and new diseases and public health emergencies can emerge unexpectedly [14].”
MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey
Wu, X., Zhao, Y., Zhang, Y., et al. (2024) · neural information processing systems · doi:10.52202/079017-2782 · Cited by 4 · see citing works
Baichuan-M2: Scaling Medical Capability with Large Verifier SystemPreprint
M2 Team, Dou, C., Liu, C., Yang, F., Li, F., Jia, J., Chen, M., Ju, Q., Wang, S., Dang, S., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.02208 · Cited by 2 · see citing works
Self-citationGourabathina, A., Hao, Y., Gerych, W., & Ghassemi, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.17163 · Cited by 2 · see citing works
From Crisis to Coordination: How AI Transformed Public Health Policies during COVID-19
Cheshmehzangi, A., & Su, Z. (2025) · Rural and regional development · doi:10.70322/rrd.2025.10016 · Cited by 1 · see citing works
Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLM
Song, D., Lai, S., Wang, M., Chen, S., Sun, L., & Wang, B. (2024) · arXiv · doi:10.48550/arxiv.2411.03823
Self-citationOuyang, Z., Qiu, Y., Wang, L., de Melo, G., Zhang, Y., Wang, Y., & He, L. (2024) · arXiv · doi:10.48550/arxiv.2410.03502
Zhang, X., Li, L., Zhou, Y., Zhou, X., Zhang, Y., & Wu, X. (2025) · arXiv · doi:10.48550/arxiv.2510.09275
Flagged first, then by citations
Research works printed but not linked: 4
- Patrick Lewis Ethan Perez Aleksandra Piktus Fabio Petroni Vladimir Karpukhin Naman Goyal Heinrich Küttler Mike Lewis Wen-tau Yih Tim Rocktäschel et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems Vol. 33 (2020) 9459-9474.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74-81.
- Larry L. Weed et al., [n.d.]. SOAP (Subjective, Objective, Assessment, Plan) documentation. Standard clinical documentation framework; see e.g. StatPearls NBK482263..
- Yuxin Zuo, Shang Qu, Yifei Li, Zhang-Ren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR, 80961-80990. https://proceedings.mlr.press/v267/zuo25a.html
Cited by
Sorted by citations
Wang, J., Zheng, Y., Bao, L., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.16742 · Cited by 3
“Submissions received after that deadline will be evaluated in the Fall Open.”
LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment
Li, L., Li, D., Chen, C., Ma, R., Yu, R., Lin, M., Yin, R., Fan, L., Shyr, C., Ma, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.25273 · Cited by 1 · see citing works
“Many studies further apply rubric-based or Likert-scale evaluations across clinically relevant dimensions, as in LiveMedBench [155] and CounselBench [81].”
InfluentialAPTER: Adaptive Post-Training with Expert-Grounded Rubrics
Wang, X.-K., Li, L., Xu, Z.-Y., et al. (2026) · doi:10.48550/arxiv.2608.14212 · Cited by 1
“The main medical training set contains 14,713 queries from LiveMedBench [35], SpeechMedDataset [36], and II-Medical queries [37] distributed through RubricHub [38].”
MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters
Yang, Y.-M., Sun, X., Zou, Y.-W., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2607.08257 · Cited by 1
RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty
Zhang, Z., Hu, X., Huang, Y., et al. (2026) · doi:10.48550/arxiv.2602.12424 · Cited by 1
“…(2024d) assess the abilities of LLMs in science domains, and recent benchmarks also extend evaluation to medicine with contamination-aware protocols and automated rubrics (Yan et al., 2026) and to structure-based drug design with systematic trade-offs across representations (Zheng et al., 2026).”
Self-citationMedSonor: Learning to Use Clinical Audio in Medical LLMs
Yu, G., Chen, J.-Y., & Wang, B.-Y. (2026) · doi:10.48550/arxiv.2610.22405
“MedSonor ranks first among the evaluated open-weight systems on LiveMedBench Clean-TTS, MMedFD-Lite, MedVoice-AID, and MedVoice-Dial (Table 3).”
Self-Evolving Agent HarnessPreprint
Wei, C., Guan, H., Zhu, M., Zhou, K., He, J., Zhou, X., Chen, H., & Shen, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.18813
Agha, A., Jalil, M. N., Anwar, E., & Bakoush, O. (2026) · JMIR Research Protocols · doi:10.2196/103220
“All vignettes are newly authored, unpublished cases written by clinicians; they are not drawn from public question banks or datasets, and the 100-case phase 2 bank is withheld until data collection is complete, which limits the risk that a model has encountered the cases during training [27].”
A primer on evaluation methods for large language models in healthcarePreprint
McKinney, S. E., Vu, P., Justice, S. A., Humphries, C., Pradhan, A., Keyes, T., Black, K. C., Havlik, J. L., Wu, D. J., O’Sullivan, J. W., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.14819
“8 Benchmarks, which are discussed more below, aim to avoid this contamination although may themselves become part of training corpora in due course.”
Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks
Dixit, P., & Hosseini, P. (2026) · doi:10.48550/arxiv.2609.16023
“LiveMed-Bench [3] is a larger, contamination-controlled benchmark of 2,756 cases and 16,702 criteria, scored as the mean over cases of clip( i p i · m i /P + ) for points p i , met-decisions m i , and total positive points P + .”
Dense Clinical Contrasts Enhance Medical Knowledge Updating in Large Language Models
Huang, Y., Quan, S., Geng, H., et al. (2026) · doi:10.48550/arxiv.2608.30405
“Newer benchmarks improve task realism or contamination control (Arora et al., 2025; Yan et al., 2026; Quan et al., 2026), but do not directly target clinical update events; temporal QA studies further show that LLMs can rely on stale pretraining facts (Vu et al., 2024; Mousavi et al., 2024).”
MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
Shi, R., Zhou, Q., Xu, Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2607.09142
“The closest realistic settings include HealthBench Professional [30] and LiveMedBench [31].”
InfluentialWhen Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical QueriesPreprint
Li, Y.-G., Sun, J., Zhao, W., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.28332
“…and safeguard robustness across broad risk categories (Zhang et al., 2024a; Wang et al., 2024a), while medical LLM benchmarks mainly assess clinical knowledge, reasoning, diagnosis, or decision support (Wu et al., 2024; Zhang et al., 2025; Arora et al., 2025; Yan et al., 2026; Liu et al., 2024a).”
- Authors
- Yan, Z., Song, D., Fang, Z., Ji, Y., Li, X., Li, Q., & Sun, L.
- Affiliations
- Lehigh University (US)
- Harvard University (US)
- Imperial College London (GB)
- Venue
- Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
- Publisher
- ACM
- Published
- 2026-08-08
- Pages
- 10162-10173
- OpenAlex
- W7196993840
- Cited by
- 0
- References
- 15
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, cc-by)
- Funders
- National Science Foundation (CRII-2246067, ATD-2427915, NSF POSE-2346158, NSF POSE-2449280)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, contaminated data, temporal misalignment, clinical reasoning, LLM-as-a-judge, error analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
