MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
Open accessAbstract
The emergence of various medical large language models (LLMs) in the medical domain has highlighted the need for unified evaluation standards, as manual evaluation of LLMs proves to be time-consuming and labor-intensive. To address this issue, we introduce MedBench, a comprehensive benchmark for the Chinese medical domain, comprising 40,041 questions sourced from authentic examination exercises and medical reports of diverse branches of medicine. In particular, this benchmark is composed of four key components: the Chinese Medical Licensing Examination, the Resident Standardization Training Examination, the Doctor In-Charge Qualification Examination, and real-world clinic cases encompassing examinations, diagnoses, and treatments. MedBench replicates the educational progression and clinical practice experiences of doctors in Mainland China, thereby establish- ing itself as a credible benchmark for assessing the mastery of knowledge and reasoning abilities in medical language learning models. We perform extensive experiments and conduct an in-depth analysis from diverse perspectives, which culminate in the following findings: (1) Chinese medical LLMs underperform on this benchmark, highlighting the need for significant advances in clinical knowledge and diagnostic precision. (2) Several general-domain LLMs surprisingly possess considerable medical knowledge. These findings elucidate both the capabilities and limitations of LLMs within the context of MedBench, with the ultimate goal of aiding the medical research community.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large-scale medical benchmarks often comprise tens of thousands of multiple-choice questions, leading to substantial token consumption and API costs, particularly for proprietary models.
“Large-scale medical benchmarks often comprise tens of thousands of multiple-choice questions (MCQs) (e.g., MedQA: 12,723 items2; MedMCQA: 6,150 items9; MedBench: 40,041 items4; Med-HALT: 59,254 items10), leading to substantial token consumption and API costs—particularly for proprietary models.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [4]
The Proposed Benchmark (page unavailable) identifies MedBench as containing 40,041 exercises, including a large multiple-choice examination component, but 1,025 of the exercises are clinical-case question–answer pairs rather than MCQs. Quantitative Analysis (page unavailable) reports high computational cost when using an API for a benchmark of this size. The paper does not specifically discuss token consumption or say that the cost is particularly high for proprietary models. (LLM, conf=0.86)
Experimental — accuracy not yet measuredCurrent practice for evaluating LLMs in healthcare relies heavily on static, fixed-length benchmarks such as MedQA, MMLU-Med, and MedBench.
“Current practice relies heavily on static, fixed-length benchmarks such as, MedQA2, MMLU-Med3, and MedBench4.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [4]
The Proposed Benchmark section identifies MedBench as a defined set of 40,041 medical questions, supporting its use as an example of a medical LLM benchmark. But the Quantitative Analysis section proposes evaluating models on a smaller set of questions selected by difficulty, so the paper does not establish that MedBench evaluation is necessarily fixed-length. (LLM, conf=0.78)
Experimental — accuracy not yet measuredThese frameworks typically focus on narrow clinical scenarios (1-28 datasets), rely on limited data sources or synthetic data, and assess primarily accuracy without comprehensive taxonomy validation or deployment considerations.
“However, these frameworks typically focus on narrow clinical scenarios (1–28 datasets), rely on limited data sources or synthetic data, and assess primarily accuracy without comprehensive taxonomy validation or deployment considerations.”
From Holistic evaluation of large language models for medical tasks with MedHELM (2026) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2004) · Cited by 9,718 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,682 · see citing works
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.03109 · Cited by 3,159 · see citing works
Self-citationA Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Self-citationJin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,320 · see citing works
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,028 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S. B., & Zhang, Y. (2023) · Cureus · doi:10.7759/cureus.40895 · Cited by 616 · see citing works
Self-citationCapabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
Crosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z. X., Schoelkopf, H., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.891 · Cited by 361 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2022) · arXiv · doi:10.48550/arxiv.2207.08143 · Cited by 343 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · see citing works
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · see citing works
Self-citationGLM-130B: An Open Bilingual Pre-trained Model
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02414 · Cited by 281 · see citing works
Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data
Xu, C., Guo, D., Duan, N., & McAuley, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.385 · Cited by 191 · see citing works
MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2022) · arXiv · doi:10.48550/arxiv.2203.14371 · Cited by 157 · see citing works
MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training DataPreprint
Han, T., Adams, L. C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., Figueroa, A., Löser, A., Truhn, D., & Bressem, K. K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08247 · Cited by 117 · see citing works
HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge
Wang, H., Liu, C., Xi, N., Qiang, Z., Zhao, S., Qin, B., & Liu, T. (2023) · arXiv · doi:10.48550/arxiv.2304.06975 · Cited by 115 · see citing works
Towards Conversational Diagnostic AIPreprint
Tu, T., Palepu, A. K., Schaekermann, M., Saab, K. K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Tomašev, N., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.05654 · Cited by 114 · see citing works
DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task
Xiong, H., Wang, S., Zhu, Y., Zhao, Z., Liu, Y., Huang, L., Wang, Q., & Shen, D. (2023) · arXiv · doi:10.48550/arxiv.2304.01097 · Cited by 81 · see citing works
Statistical theory for logistic mental test models with a prior distribution of ability
Birnbaum, A. (1969) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(69)90005-4 · Cited by 77 · see citing works
PMC-LLaMA: Towards Building Open-source Language Models for MedicinePreprint
Wu, C., Lin, W., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.14454 · Cited by 48 · see citing works
Self-citationChen, Y., Wang, Z., Xing, X., zheng, H., Xu, Z., Fang, K., Wang, J., Li, S., Wu, J., Liu, Q., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.15896 · Cited by 36 · see citing works
Benchmarking Large Language Models on CMExam - A Comprehensive Chinese Medical Exam Dataset
Liu, J., Zhou, P., Hua, Y., Chong, D., Tian, Z., Liu, A., Wang, H., You, C., Guo, Z., Zhu, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.03030 · Cited by 36 · see citing works
Crosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T. L., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.01786 · Cited by 29 · see citing works
MLEC-QA: A Chinese Multi-Choice Biomedical Question Answering Dataset
Li, J., Zhong, S., & Chen, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.698 · Cited by 26 · see citing works
STATISTICAL THEORY FOR LOGISTIC MENTAL TEST MODELS WITH A PRIOR DISTRIBUTION OF ABILITY
Birnbaum, A. (1967) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1967.tb00363.x · Cited by 16 · see citing works
MedGPTEval: A Dataset and Benchmark to Evaluate Responses of Large Language Models in MedicinePreprint
Xu, J., Lu, L., Yang, S., Liang, B., Peng, X., Pang, J., Ding, J., Shi, X., Yang, L., Song, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.07340 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Evaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 87 · see citing works
A Survey of Conversational Search
Mo, F., Mao, K., Zhao, Z., Qian, H., Chen, H., Cheng, Y., Li, X., Zhu, Y., Dou, Z., & Nie, J. (2025) · ACM Transactions on Information Systems · doi:10.1145/3759453 · Cited by 51 · see citing works
Holistic evaluation of large language models for medical tasks with MedHELM
Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04151-2 · Cited by 59 · see citing works
RareBench: Can LLMs Serve as Rare Diseases Specialists?
Chen, X., Mao, X., Guo, Q., Wang, L., Zhang, S., & Chen, T. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671576 · Cited by 41 · see citing works
Self-citationGong, E. J., Bang, C. S., Lee, J. J., & Baik, G. H. (2025) · Journal of Medical Internet Research · doi:10.2196/84120 · Cited by 49 · see citing works
The pitfalls of multiple-choice questions in generative AI and medical education
Singh, S., Alyakin, A., Alber, D. A., Stryker, J., Tong, A. P., Sangwon, K. L., Goff, N. K., Paz, M. D. L., Hernandez‐Rovira, M. A., Park, K. Y., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-26036-7 · Cited by 20 · see citing works
Ji, H., Wang, X., Sia, C., Yap, J. J. L., Lim, S. T., Djohan, A. H., Chang, Y., Zhang, N., Guo, M., Li, F., et al. (2025) · Communications Medicine · doi:10.1038/s43856-025-00802-0 · Cited by 18 · see citing works
Self-citationBenchmarking Medical LLMs on Anesthesiology: A Comprehensive Dataset in Chinese
Zhou, B., Zhan, Y., Wang, Z., Li, Y., Zhang, C., Yu, B., Ding, L., Jin, H., Liu, W., Wang, X., et al. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3502465 · Cited by 9 · see citing works
Large Language Models for Information Retrieval: Challenges and Chances
Breuer, T., Frihat, S., Fuhr, N., Lewandowski, D., Schaer, P., & Schenkel, R. (2025) · Datenbank-Spektrum · doi:10.1007/s13222-025-00503-x · Cited by 13 · see citing works
Wang, Y., Cao, P., Fang, H., & Ye, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107250 · Cited by 7 · see citing works
Self-citationXia, B., Lu, Q., Zhu, L., Xing, Z., Zhao, D., & Zhang, H. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5775317 · Cited by 6 · see citing works
ElecBench: A large language model benchmark in electric power domain
Zhang, S., Huang, Q., Zhang, Q., Liang, X. M., Liu, W., Gao, K., Zhou, F., & Shi, C. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112310 · Cited by 6 · see citing works
Roeschl, T., Hoffmann, M., Hashemi, D., Rarreck, F., Hinrichs, N., Trippel, T. D., Gröschel, M. I., Unbehaun, A., Klein, C., Kempfert, J., et al. (2024) · medRxiv · doi:10.1101/2024.11.20.24313385 · Cited by 2 · see citing works
Zheng, T., Jiang, Z., Liu, J., & Feng, S. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02671-w · Cited by 1 · see citing works
Zhang, J., Li, X., Fang, J., Wei, W., & Cui, M. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4298-4_3 · Cited by 1 · see citing works
Benchmarking international construction contract knowledge of large language models
Xia, H., Chen, Y., Qi, X., & Wang, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131754 · Cited by 1 · see citing works
Self-citationLee, J., Shin, J., & Cho, B. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02567-9 · Cited by 1 · see citing works
Benchmarking International Construction Contract Knowledge of Large Language ModelsPreprint
Xia, H., Chen, Y., Qi, X., & Wang, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5247570 · Cited by 1 · see citing works
Self-citationProbing the Surgical Competence of LLMs: A global health study leveraging AfriMedQA benchmarksPreprint
Olatunji, T., Omofoye, F., Aka, E. C., Itzikowitz, G., Macaulay, D., Adaramola, O. G., Adewale, B. A., Asuzu, C., Popoola, S., Kinara, W., et al. (2025) · medRxiv · doi:10.1101/2025.10.05.25337350 · Cited by 1 · see citing works
MedMisBench: Measuring Epistemic Resilience of LLMs Under Misleading Medical ContextPreprint
Zhou, H.-J., Zou, X.-Y., Wu, J.-G., Wu, S., Yu, J.-C., Segal, B., Niebuhr, T. E., Amro, S., Petrus, M., Momin, S., et al. (2026) · bioRxiv · doi:10.64898/2026.05.25.727671 · Cited by 3 · see citing works
LiveMedBench: A Contamination-Limited Medical Benchmark for LLMs with Automated Rubric Evaluation
Yan, Z., Song, D., Fang, Z., Ji, Y., Li, X., Li, Q., & Sun, L. (2026) · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3770855.3817579
Improving clinical expertise in large language models using electronic medical recordsPreprint
Zhu, L., Liu, J., Wang, J., Zhang, W., Jiang, S., Yang, H., Wang, C., Ye, Q., Ruan, T., Rui, X., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-5285540/v1
Peng, S., Zhu, Y. L., Wang, B., Zhang, M., Wang, Z., Yao, K., Hao, M., & Wang, J. (2024) · doi:10.2196/preprints.68152
Pan, Z., Peng, Y., Jian, Z., Chen, Y., Qiu, W., Ma, H., Yao, J., Wang, M., & Wu, Q. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890213
Fang, Q., Wang, Y., Pan, Y., Zhang, Z., & Peng, X. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-1809-5_35
Self-citationBuilding Healthcare Educational Scenarios by Tagging Clinical Narratives Using Large Language Models
Buendía, F., Gayoso-Cabada, J., & Sierra, J. L. (2025) · Lecture notes in educational technology · doi:10.1007/978-981-96-5658-5_25
Dong, F., Zhang, H., Ren, Y., Li, W., Li, Y., Cui, G., & Liu, P. (2025) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.70437
Machines flunking an exam: Evaluating large language models on course-related open questionsPreprint
Huang, J., Wei, Y., Zhang, L., Lai, R., Lai, F., & Zheng, Y. (2025) · Research Square · doi:10.21203/rs.3.rs-6679662/v1
FinLBench: A Benchmark for Evaluating Large Language Models on Long-Text Financial Documents
Hu, S., Yu, D., Zhang, Y., Yang, Z., Zhou, L., Jiang, X., Li, Y., Fei, B., & Liu, J. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4091-4_17
Self-citationBenchmarking International Construction Contract Knowledge of Large Language ModelsPreprint
Xia, H., Chen, Y., Qi, X., & Wang, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5775312
Self-citationTaxBen: Benchmarking the Chinese Tax Knowledge of Large Language Models
Chen, Y., Lv, S., Xia, P., Wang, Z., Qin, Y., Wang, Q., & Hu, G. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3343-5_24
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
Association for Artificial Intelligence 2024, Cai, Y., de Melo, G., He, L., Wang, Y., Wang, L., Wang, Y., Zhang, Y., & Zhang, Y. (2024) · Underline Science Inc. · doi:10.48448/q3qj-3742
Self-citationMedKRG: A Knowledge and Retrieval-Guided Framework for Electronic Health Record Synthesis
Lu, F., & Duan, J. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356281
LAGA: A graph adapter for long-tail text classification via semantic space refinement
Wang, S., Zhou, G., Lan, M., Lu, J., Li, Z., & Xia, Y. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00656-z
Liu, X., Lin, W., Hu, M., Xu, C., & Li, H. (2026) · Research Square · doi:10.21203/rs.3.rs-9084876/v1
A parsimonious q-rung orthopair fuzzy decision-making method for ship design concept evaluation
Yu, L., Chen, Z., & Qin, Z. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3923
Zhang, M., Wang, X., Wang, Y., Xing, H., Zhuang, P., & Ji, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7251-9_15
Self-citationLitBench: A Graph-Centric Large Language Model Benchmarking Tool For Literature Tasks
Varvarigos, A., Maatouk, A., Zhang, J., Bui, N., Chen, J., Tassiulas, L., & Ying, R. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770854.3785679
MetSEval-1k: A Comprehensive Benchmark for Evaluating Large Language Models in Meteorology
Yu, T., Wang, K., Li, Z. M., Wang, M., Chen, Y., Chen, H., Zhao, R., Xu, Y., Song, Y., & Xu, B. (2026) · Applied Sciences · doi:10.3390/app16168032
MedEthicsBench: Evaluating Medical Ethics Reasoning Across Jurisdictions
Kang, Y., Lee, S.-Y., Chu, L., Qu, W., Liu, W., Hou, C., Wu, S., Cao, L., & Chu, W. C. (2026) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh72779.2026.00031
Lin, Y., Shao, Y., Mou, W., Zheng, Z., Zhu, R., Zhuang, Y., Tao, Y., Song, J., & Li, D. (2026) · doi:10.2196/preprints.110729
Diagnostic Accuracy Evolution in DeepSeek Large Language Models: A Pediatric Telehealth Evaluation
Wang, L., & Michel, P. C. (2026) · Asia Conference on Machine Learning and Computing (ACMLC) · doi:10.1109/acmlc70381.2026.11700496
Self-citation
- Authors
- Cai, Y., Wang, L., Wang, Y., De Melo, G., Zhang, Y., Wang, Y., & He, L.
- Affiliations
- East China Normal University (CN)
- Shanghai Artificial Intelligence Laboratory (CN)
- Shanghai Jiao Tong University (CN)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2024-03-24
- Volume
- 38
- Issue
- 16
- Pages
- 17709-17717
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4393156546
- Cited by
- 42
- References
- 28
- Citations by year
- 2026: 15 · 2025: 22 · 2024: 5
- FWCI
- 11.29
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- other-oa
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion, cc-by)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Natural Science Foundation of China (12105105)
- East China Normal University
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- medical large language models, benchmark, clinical reasoning, diagnostic accuracy, medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
