MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models
Open accessAbstract
Ensuring the general efficacy and benefit for human beings from medical Large Language Models (LLM) before real-world deployment is crucial. However, a widely accepted and accessible evaluation process for medical LLM, especially in the Chinese context, remains to be established. In this work, we introduce “MedBench”, a comprehensive, standardized, and reliable benchmarking system for Chinese medical LLM. First, MedBench assembles the currently largest evaluation dataset (300 901 questions) to cover 43 clinical specialties, and performs multi-faceted evaluation on medical LLM. Second, MedBench provides a standardized and fully automatic cloud-based evaluation infrastructure, with physical separations between question and ground truth. Third, MedBench implements dynamic evaluation mechanisms to prevent shortcut learning and answer memorization. Applying MedBench to popular general and medical LLMs, we observe unbiased, reproducible evaluation results largely aligning with medical professionals' perspectives. This study establishes a significant foundation for preparing the practical applications of Chinese medical LLMs. MedBench is publicly accessible at https://medbench.opencompass.org.cn.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Broad-spectrum evaluations like Med-Bench recognize hallucination as an error category but do not provide a dedicated mechanism for systematically inducing and diagnosing it.
“Similarly, broad-spectrum evaluations like Med-Bench (Liu et al. 2024) recognize hallucination as an error category but do not provide a dedicated mechanism for systematically inducing and diagnosing it.”
From MHB: Medical Hallucination Benchmark for Large Language Models in Complex Clinical Tasks (2026) · cites this paper as Liu et al. 2024
The Introduction (p. 2) and Conclusion (p. 11) describe MedBench as a broad-spectrum evaluation across 43 clinical specialties. The Dynamic Evaluation Mechanisms section (p. 5) describes choice shuffling and random prompt matching to address shortcut learning and response-pattern bias, not hallucination induction or diagnosis. Searches for hallucination and a dedicated mechanism found no passage in our copy that recognizes hallucination as an error category; that part of the claim is unsupported. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Part of this paper’s reference list could not be split into single entries.
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · see citing works
Self-citationPublisher Correction: Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06455-0 · Cited by 101 · see citing works
A new paradigm for applying deep learning to protein–ligand interaction prediction
Wang, Z., Wang, S., Li, Y., Guo, J., Wei, Y., Mu, Y., Zheng, L., & Li, W. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae145 · Cited by 41 · see citing works
Self-citationDon't Make Your LLM an Evaluation Benchmark Cheater
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., & Han, J. (2023) · arXiv · doi:10.48550/arxiv.2311.01964 · Cited by 25 · see citing works
Flagged first, then by citations
Research works printed but not linked: 23
- [1]1. Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023). Large language models in medicine. Nature medicine , 29 (8), 1930-1940.
- [3]3. Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., ... & Natarajan, V. (2022). Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138. · arXiv
- [4]4. Li, J., Zhong, S., & Chen, K. (2021, November). MLEC-QA: A Chinese multi-choice biomedical question answering dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 8862-8874).
- [5]5. Liu, J., Zhou, P., Hua, Y., Chong, D., Tian, Z., Liu, A., ... & Li, M. L. (2023). Benchmarking Large Language Models on CMExam--A Comprehensive Chinese Medical Exam Dataset. arXiv preprint arXiv:2306.03030. · arXiv
- [6]6. Wang X, Chen G H, Song D, et al. Cmb: A comprehensive medical benchmark in chinese[J]. arXiv preprint arXiv:2308.08833, 2023. · arXiv
- [7]7. Zhang N, Chen M, Bi Z, et al. Cblue: A chinese biomedical language understanding evaluation benchmark[J]. arXiv preprint arXiv:2106.08087, 2021. · arXiv
- [10]10. Wang, Y., Teng, Y., Huang, K., Lyu, C., Zhang, S., Zhang, W., ... & Wang, Y. (2023). Fake Alignment: Are LLMs Really Aligned Well?. arXiv preprint arXiv:2311.05915. · arXiv
- [12]12. Sohrabi, C., Mathew, G., Maria, N., Kerwan, A., Franchi, T., & Agha, R. A. (2023). The SCARE 2023 guideline: updating consensus Surgical CAse REport (SCARE) guidelines. International Journal of Surgery, 109(5), 1136-1140.
- [13]13. Wang, X., Zhang, X., Wang, G., He, J., Li, Z., Zhu, W., ... & Zhang, S. (2024). OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models in Medicine. arXiv preprint arXiv:2402.18028. · arXiv
- [14]14. Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., ... & Tang, J. (2022). GLM-130B: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. · arXiv
- [15]15. Wang, H., Liu, C., Xi, N., Qiang, Z., Zhao, S., Qin, B., & Liu, T. (2023). Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975. · arXiv
- [16]16. Chen, Y., Wang, Z., Xing, X., Xu, Z., Fang, K., Wang, J., ... & Xu, X. (2023). Bianque: Balancing the questioning and suggestion ability of health llms with multi-turn health conversations polished by chatgpt. arXiv preprint arXiv:2310.15896. · arXiv
- [17]17. Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002, July). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (pp. 311-318).
- [18]18. Lin, C. Y., & Och, F. J. (2004, July). Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL) (pp. 605-612).
- [21]21. Abdou, I., & Eude, T. (2023, July). Open-ended questions automated evaluation: proposal of a new generation. In Proceedings of the 2023 International Joint Conference on Robotics and Artificial Intelligence (pp. 143-147).
- [10]10. multi-modal model an all-around player?. arXiv preprint arXiv:2307.06281. · arXiv
- [24]24. Wang, W., Feng, C., Han, R., Wang, Z., Ye, L., Du, Z., ... & Yang, J. (2023). trRosettaRNA: automated prediction of RNA 3D structure with transformer network. Nature Communications, 14(1), 7266.
- [26]26. Wang, Z., Liu, C., Zhang, S., & Dou, Q. (2023, October). Foundation model for endoscopy video analysis via large-scale self-supervised pre-train. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 101-111). Cham: Springer Nature Switzerland.
- [27]27. Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T., Wang, H., ... & Qiao, Y . (2023). SAM-Med2D. arXiv preprint arXiv:2308.16184. · arXiv
- [29]29. Zhang, Y., Gao, J., Tan, Z., Zhou, L., Ding, K., Zhou, M., ... & Wang, D. (2024). Data-centric foundation models in computational healthcare: A survey. arXiv preprint arXiv:2401.02458. · arXiv
- [31]31. Saab, K., Tu, T., Weng, W. H., Tanno, R., Stutz, - D., Wulczyn, E., ... & Natarajan, V. (2024). Capabilities of Gemini Models in Medicine. arXiv preprint arXiv:2404.18416. · arXiv
- [32]32. Wang, D., Wang, X., Wang, L., Li, M., Da, Q., Liu, X., ... & Zhang, S. (2023). A real-world dataset and benchmark for foundation model adaptation in medical image classification. Scientific Data, 10(1), 574.
- [33]33. Hu, Y., Li, T., Lu, Q., Shao, W., He, J., Qiao, Y ., & Luo, P. (2024). OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM. arXiv preprint arXiv:2402.09181. · arXiv
Cited by
Sorted by citations
MHB: Medical Hallucination Benchmark for Large Language Models in Complex Clinical Tasks
Lu, J., Liu, J., Zheng, X., Yang, M., Wang, J., Wang, P., & Zhang, Y. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i45.41243 · Cited by 1 · see citing works
- Authors
- Liu, M., Hu, W., Ding, J., Xu, J., Li, X., Zhu, L., Bai, Z., Shi, X., Wang, B., Song, H., Liu, P., Zhang, X., Wang, S., Li, K., Wang, H., Ruan, T., Huang, X., Sun, X., & Zhang, S.
- Affiliations
- Shanghai Artificial Intelligence Laboratory (CN)
- Shanghai Jiao Tong University (CN)
- Chinese University of Hong Kong, Shenzhen (CN)
- Chinese Academy of Sciences (CN)
- Sichuan University (CN)
- Tongji University (CN)
- East China University of Science and Technology (CN)
- Fudan University (CN)
- ORCID
- Venue
- Big Data Mining and Analytics
- Publisher
- Tsinghua University Press
- Published
- 2024-12
- Volume
- 7
- Issue
- 4
- Pages
- 1116-1128
- ISSN
- 2096-0654 (ISSN-L)
- 2097-406X
- OpenAlex
- W4405022053
- Cited by
- 41
- References
- 5
- Citations by year
- 2026: 21 · 2025: 19 · 2024: 1
- FWCI
- 3.49
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Big Data Mining and Analytics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- National Key R&D Program of China (2022ZD0160705,2022ZD0160704)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- medical large language models, benchmarking, evaluation dataset, medical specialties, shortcut learning, medical question answering, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
