MTCMB: A multi-task benchmark framework for evaluating LLMs on knowledge, reasoning, and safety in traditional Chinese medicine
Open accessAbstract
Traditional Chinese Medicine (TCM) is a holistic medical system with millennia of accumulated clinical experience, playing a vital role in global healthcare-particularly across East Asia. However, the implicit reasoning, diverse textual forms, and lack of standardization in TCM pose major challenges for computational modeling and evaluation. Large Language Models (LLMs) have demonstrated remarkable potential in processing natural language across diverse domains, including general medicine. Yet, their systematic evaluation in the TCM domain remains underdeveloped. Existing benchmarks either focus narrowly on factual question answering or lack domain-specific tasks and clinical realism. To fill this gap, we introduce MTCMB-a Multi-Task Benchmark for Evaluating LLMs on TCM Knowledge, Reasoning, and Safety. Developed in collaboration with certified TCM experts, MTCMB comprises 12 sub-datasets spanning five major categories: knowledge QA, language understanding, diagnostic reasoning, prescription generation, and safety evaluation. The benchmark integrates real-world case records, national licensing exams, and classical texts, providing an authentic and comprehensive testbed for TCM-capable models. Preliminary results indicate that current LLMs perform well on foundational knowledge but fall short in clinical reasoning, prescription planning, and safety compliance. These findings highlight the urgent need for domain-aligned benchmarks like MTCMB to guide the development of more competent and trustworthy medical AI systems. All datasets, code, and evaluation tools are publicly available at: https://github.com/Wayyuanyuan/MTCMB.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ROUGE measures, namely ROUGE-1, ROUGE-2, and ROUGE-L, quantify overlap between generated text and actual clinical reports, and elevated ROUGE scores signify superior correspondence with expert-written descriptions, making them appropriate for evaluating LLM-based dermatology systems.
“Performance for multimodal or report-generation models is frequently assessed using ROUGE measures, namely ROUGE-1, ROUGE-2, and ROUGE-L, which quantify the overlap between the generated text and actual clinical reports (25). Elevated ROUGE scores signify superior correspondence with expert-written descriptions, enabling these measures appropriate for evaluating LLM-based dermatology systems (33).”
From Revolutionizing dermatopathology using AI in skin diagnostics: scoping review (2026) · cites this paper as [33]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(1954) · doi:10.5694/j.1326-5377.1954.tb66294.x · Cited by 105,830 · cited 1×
“Traditional Chinese Medicine (TCM) is a holistic healthcare system with millennia of clinical history, serving as a cornerstone of medical practice in China and East Asia while gaining global prominence [6, 27].”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1× · see citing works
“Domain-specific benchmarks have emerged to assess medical LLMs. Early efforts like MedQA [7] and MedMCQA [15] focused on Western medical knowledge through multiple-choice questions, later expanded by MultiMedQA [18] to include diverse clinical formats.”
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · cited 1× · see citing works
“Large Language Models (LLMs) demonstrate transformative potential in medicine through corpus-scale pattern recognition, achieving state-of-the-art performance on medical QA tasks [21].”
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · cited 1× · see citing works
“Benchmarks like GLUE and SuperGLUE [22] established standardized tasks for evaluating general linguistic competence, while MMLU [5] extended this paradigm to 57 academic disciplines.”
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718 · cited 2× · see citing works
“Domain-specific benchmarks have emerged to assess medical LLMs. Early efforts like MedQA [7] and MedMCQA [15] focused on Western medical knowledge through multiple-choice questions, later expanded by MultiMedQA [18] to include diverse clinical formats.”
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · cited 1× · see citing works
“Current resources either target general medicine (MedQA [7], PubMedQA [8], MedMCQA [15]) or focus narrowly on Chinese licensing exams (TCMBench [29], TCMD [28]).”
DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.19437 · Cited by 294 · see citing works
Self-citationMeasuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 1× · see citing works
“Benchmarks like GLUE and SuperGLUE [22] established standardized tasks for evaluating general linguistic competence, while MMLU [5] extended this paradigm to 57 academic disciplines.”
MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2022) · arXiv · doi:10.48550/arxiv.2203.14371 · Cited by 157 · cited 2× · see citing works
“Domain-specific benchmarks have emerged to assess medical LLMs. Early efforts like MedQA [7] and MedMCQA [15] focused on Western medical knowledge through multiple-choice questions, later expanded by MultiMedQA [18] to include diverse clinical formats.”
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.15115 · Cited by 128 · cited 1× · see citing works
“5-Max [20], Doubao-1.”
Self-citationHuatuoGPT-o1, Towards Medical Complex Reasoning with LLMsPreprint
Chen, J., Cai, Z., Ji, K., Wang, X., Liu, W., Wang, R., Hou, J., & Wang, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.18925 · Cited by 123 · cited 1× · see citing works
“• Medical-specialized LLMs : Baichuan-14B-M1 [23], HuatuoGPT-o1-72B [3], DISC-MedLLM [2], Taiyi-LLM [11], and WiNGPT2-14B-Chat [26].”
Taiyi: a bilingual fine-tuned large language model for diverse biomedical tasks
Luo, L., Ning, J., Zhao, Y., Wang, Z., Ding, Z., Chen, P., Fu, W., Han, Q., Xu, G., Qiu, Y., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae037 · Cited by 89 · cited 1× · see citing works
“• Medical-specialized LLMs : Baichuan-14B-M1 [23], HuatuoGPT-o1-72B [3], DISC-MedLLM [2], Taiyi-LLM [11], and WiNGPT2-14B-Chat [26].”
Self-citationSG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types
Mou, Y., Zhang, S.-K., & Ye, W. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2410.21965 · Cited by 87
Traditional Chinese medicine for prevention and treatment of hepatocarcinoma: From bench to bedside.
Hu, B., Wang, S.-S., & Du, Q. (2015) · World Journal of Hepatology · doi:10.4254/wjh.v7.i9.1209 · Cited by 71 · cited 1×
“Traditional Chinese Medicine (TCM) is a holistic healthcare system with millennia of clinical history, serving as a cornerstone of medical practice in China and East Asia while gaining global prominence [6, 27].”
Qibo: A Large Language Model for Traditional Chinese Medicine
Jia, Y., Ji, X.-Y., Wang, X., et al. (2024) · Expert systems with applications · doi:10.48550/arxiv.2403.16056 · Cited by 52 · cited 1×
“Qibo-Benchmark [30], accompanying a LLaMA-based TCM model, introduces entity recognition and syndrome differentiation tasks but lacks multi-turn clinical dialogues.”
Self-citationBaichuan-M1: Pushing the Medical Capability of Large Language ModelsPreprint
Wang, B., Zhao, H., Zhou, H., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2502.12671 · Cited by 49 · cited 1× · see citing works
“• Medical-specialized LLMs : Baichuan-14B-M1 [23], HuatuoGPT-o1-72B [3], DISC-MedLLM [2], Taiyi-LLM [11], and WiNGPT2-14B-Chat [26].”
Self-citationBenchmarking Large Language Models on CMExam - A Comprehensive Chinese Medical Exam Dataset
Liu, J., Zhou, P., Hua, Y., Chong, D., Tian, Z., Liu, A., Wang, H., You, C., Guo, Z., Zhu, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.03030 · Cited by 36 · cited 1× · see citing works
“For Chinese medical AI, CMExam [10] aggregates 60,000+ licensing exam questions, and CMB [24] integrates TCM queries within a broader clinical dataset.”
MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models
Pandit, S., Xu, J., Hong, J., Wang, Z., Chen, T., Xu, K., & Ding, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.14302 · Cited by 31 · cited 1× · see citing works
“LLM safety research has produced benchmarks like MedHallu [16] for medical hallucinations and BeHonest [4] for output veracity.”
Self-citationYue, W., Wang, X., Zhu, W., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2406.01126 · Cited by 30 · cited 3×
“These factors—symbolic inference, linguistic complexity, and terminological ambiguity—hinder conventional AI systems from faithfully representing TCM principles [29].”
DISC-MedLLM: Bridging General Large Language Models and Real-World Medical ConsultationPreprint
Bao, Z., Chen, W., Xiao, S., Ren, K., Wu, J., Zhong, C., Peng, J., Huang, X., & Wei, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.14346 · Cited by 26 · cited 1× · see citing works
“• Medical-specialized LLMs : Baichuan-14B-M1 [23], HuatuoGPT-o1-72B [3], DISC-MedLLM [2], Taiyi-LLM [11], and WiNGPT2-14B-Chat [26].”
Wang, Z., Hao, M., Peng, S., Huang, Y., Lu, Y., Yao, K., Yang, X., & Zhu, Y. (2025) · Scientific Data · doi:10.1038/s41597-025-04772-9 · Cited by 23 · cited 1× · see citing works
“TCMEval-SDT [25] provides 300 annotated cases for syndrome differentiation, yet its narrow scope excludes prescription generation and safety evaluation.”
Self-citationSafetyBench: Evaluating the Safety of Large Language Models
Zhang, Z., Lei, L., Wu, L., Sun, R., Huang, Y., Long, C., Liu, X., Lei, X., Tang, J., & Huang, M. (2023) · arXiv · doi:10.48550/arxiv.2309.07045 · Cited by 19 · see citing works
TCMD: A Traditional Chinese Medicine QA Dataset for Evaluating Large Language Models
Yu, P., Song, K.-T., He, F., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2406.04941 · Cited by 16 · cited 1×
“Current resources either target general medicine (MedQA [7], PubMedQA [8], MedMCQA [15]) or focus narrowly on Chinese licensing exams (TCMBench [29], TCMD [28]).”
BeHonest: Benchmarking Honesty of Large Language Models
Chern, S., Hu, Z., Yang, Y., Chern, E., Guo, Y., Jin, J., Wang, B.-J., & Liu, P.-F. (2024) · arXiv.org · doi:10.48550/arxiv.2406.13261 · cited 1×
“LLM safety research has produced benchmarks like MedHallu [16] for medical hallucinations and BeHonest [4] for output veracity.”
CMB: A Comprehensive Medical Benchmark in Chinese
Wang, X., Chen, G. H., Song, D., Zhang, Z., Chen, Z., Xiao, Q., Jiang, F., Li, J., Wan, X., Wang, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08833 · cited 1×
“For Chinese medical AI, CMExam [10] aggregates 60,000+ licensing exam questions, and CMB [24] integrates TCM queries within a broader clinical dataset.”
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Revolutionizing dermatopathology using AI in skin diagnostics: scoping review
Rammal, R., Mohy U Din, A., & Alam, T. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1614681 · Cited by 1 · see citing works
“Elevated ROUGE scores signify superior correspondence with expert-written descriptions, enabling these measures appropriate for evaluating LLM-based dermatology systems (33).”
- Authors
- Kong, S., Yang, X., Wei, Y., Wang, Z., Tang, H., Qin, J., Lan, S., Wang, Y., Bai, J., Chen, Z., Zheng, Z., Liu, C., & Liang, H.
- Affiliations
- Sun Yat-sen University (CN)
- Hunan University of Traditional Chinese Medicine (CN)
- Cornell University (US)
- Guilin University of Electronic Technology (CN)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2506.01252
- OpenAlex
- W4414895665
- Cited by
- 3
- References
- 25
- Citations by year
- 2026: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Ministry of Education of the People's Republic of China (21YJC870009)
- Sun Yat-sen University
- Humanities and Social Sciences Youth Foundation, Ministry of Education of the People's Republic of China (21YJC870009)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- traditional Chinese medicine, large language models, multi-task benchmark, diagnostic reasoning, prescription generation, LLM safety evaluation, artificial intelligence in healthcare, clinical reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
