CoChat
Sign Up
PreprintCited by 23Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareArtificial Intelligence in Healthcare

A novel evaluation benchmark for medical LLMs: Illuminating safety and effectiveness in clinical domains

Open access
Wang, Shirui, Tang, Zhihui, Yang, Huaxia, Gong, Qiuhong, Gu, Tiantian, Ma, Hongyang, Wang, Yongxin, Sun, Wubin +30 more·2025·arXiv
DOI 10.48550/arxiv.2507.23486arXiv 2507.23486PMC PMC12855988
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) hold promise in clinical decision support but face major challenges in safety evaluation and effectiveness validation. We developed the Clinical Safety-Effectiveness Dual-Track Benchmark (CSEDB), a multidimensional framework built on clinical expert consensus, encompassing 30 criteria covering critical areas like critical illness recognition, guideline adherence, and medication safety, with weighted consequence measures. Thirty-two specialist physicians developed and reviewed 2,069 open-ended Q&A items aligned with these criteria, spanning 26 clinical departments to simulate real-world scenarios. Benchmark testing of six LLMs revealed moderate overall performance (average total score 57.2%, safety 54.7%, effectiveness 62.3%), with a significant 13.3% performance drop in high-risk scenarios (p < 0.0001). Domain-specific medical LLMs showed consistent performance advantages over general-purpose models, with relatively higher top scores in safety (0.912) and effectiveness (0.861). The findings of this study not only provide a standardized metric for evaluating the clinical application of medical LLMs, facilitating comparative analyses, risk exposure identification, and improvement directions across different scenarios, but also hold the potential to promote safer and more effective deployment of large language models in healthcare environments.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.23486

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The CSEDB study demonstrated an average 13.3% performance drop in high-risk scenarios, with general-purpose models underperforming domain-specific medical LLMs in both safety and effectiveness metrics.

    “Their study demonstrated an average 13.3% performance drop in high-risk scenarios, with general-purpose models underperforming domain-specific medical LLMs in both safety and effectiveness metrics.”

    From Artificial intelligence-powered prediction of diabetic complications: from clinical data to molecular omics (2026) · cites this paper as [146]

  • Wang et al. proposed the Clinical Safety-Effectiveness Dual-Track Benchmark (CSEDB), designed by 32 specialist physicians, covering 26 clinical departments and 30 key indicators.

    “Wang et al. [146] proposed the Clinical Safety-Effectiveness Dual-Track Benchmark (CSEDB), designed by 32 specialist physicians, covering 26 clinical departments and 30 key indicators, including critical illness recognition, medication safety, and guideline adherence.”

    From Artificial intelligence-powered prediction of diabetic complications: from clinical data to molecular omics (2026) · cites this paper as [146]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 32

  1. [1][1] Omiye, J. A., Gui, H., Rezaei, S. J., Zou, J. &amp; Daneshjou, R. Large Language Models in Medicine: The Potentials and Pitfalls : A Narrative Review. Ann Intern Med 177 , 210-220, doi:10.7326/m23-2772 (2024). - · DOI
  2. [2][2] McDuff, D. et al. Towards accurate differential diagnosis with large language models. Nature 642 , 451-457, doi:10.1038/s41586-025-08869-4 (2025). - · DOI · Paper page
  3. [3][3] Bedi, S. et al. Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. Jama 333 , 319-328, doi:10.1001/jama.2024.21700 (2025). - · DOI · Paper page
  4. [4][4] Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616 , 259-265, doi:10.1038/s41586-023-05881-4 (2023). - · DOI · Paper page
  5. [5][5] Tordjman, M. et al. Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning. Nat Med , doi:10.1038/s41591-025-03726-3 (2025). - · DOI
  6. [6][6] Dada, A. et al. MeDiSumQA: Patient-Oriented Question-Answer Generation from Discharge Letters. arXiv e-prints , doi:10.48550/arXiv.2502.03298 (2025). - · DOI
  7. [7][7] Van Veen, D. et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat Med 30 , 1134-1142, doi:10.1038/s41591-024-02855-5 (2024). - · DOI · Paper page
  8. [8][8] Ive, J. et al. Clean &amp; Clear: Feasibility of Safe LLM Clinical Guidance. arXiv:2503.20953 (2025). &lt; https://ui.adsabs.harvard.edu/abs/2025arXiv250320953I &gt; . - · arXiv
  9. [9][9] de Hond, A. et al. From text to treatment: the crucial role of validation for generative large language models in health care. Lancet Digit Health 6 , e441-e443, doi:10.1016/s25897500(24)00111-0 (2024). - · DOI
  10. [10][10] Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med 30 , 2613-2622, doi:10.1038/s41591-024-03097-1 (2024). - · DOI · Paper page
  11. [11][11] Lee, J., Park, S., Shin, J. &amp; Cho, B. Analyzing evaluation methods for large language models in the medical field: a scoping review. BMC Med Inform Decis Mak 24 , 366, doi:10.1186/s12911-024-02709-7 (2024). - · DOI · Paper page
  12. [12][12] Liu, M. et al. MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models. arXiv:2407.10990 (2024). &lt; https://ui.adsabs.harvard.edu/abs/2024arXiv240710990L &gt; . - · arXiv
  13. [13][13] Ying, Z. et al. SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models. arXiv:2410.18927 (2024). &lt; https://ui.adsabs.harvard.edu/abs/ 2024arXiv241018927Y &gt; . - · arXiv
  14. [14][14] Zhang, Z. et al. Agent-SafetyBench: Evaluating the Safety of LLM Agents. arXiv:2412.14470 (2024). &lt; https://ui.adsabs.harvard.edu/abs/2024arXiv241214470Z &gt; . - · arXiv
  15. [15][15] Deniz, F. et al. aiXamine: Simplified LLM Safety and Security. arXiv:2504.14985 (2025). &lt; https://ui.adsabs.harvard.edu/abs/2025arXiv250414985D &gt; . - · arXiv
  16. [16][16] Gaber, F. et al. Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis. NPJ Digit Med 8 , 263, doi:10.1038/s41746-025-01684-1 (2025). - · DOI · Paper page
  17. [17][17] Arora, R. K. et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv:2505.08775 (2025). &lt; https://ui.adsabs.harvard.edu/ abs/2025arXiv250508775A &gt; . - · arXiv
  18. [18][18] Liu, L. et al. Towards Automatic Evaluation for LLMs ' Clinical Capabilities: Metric, Data, and Algorithm. arXiv:2403.16446 (2024). &lt; https://ui.adsabs.harvard.edu/abs/ 2024arXiv240316446L &gt; . - · arXiv
  19. [19][19] Singhal, K. et al. Towards Expert-Level Medical Question Answering with Large Language Models. arXiv:2305.09617 (2023). &lt; https://ui.adsabs.harvard.edu/abs/ 2023arXiv230509617S &gt; . - · arXiv
  20. [20][20] Johri, S. et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat Med 31 , 77-86, doi:10.1038/s41591-024-03328-5 (2025). - · DOI · Paper page
  21. [21][21] Tu, T. et al. Towards Conversational Diagnostic AI. arXiv:2401.05654 (2024). &lt; https: //ui.adsabs.harvard.edu/abs/2024arXiv240105654T &gt; . - · arXiv
  22. [22][22] Schmidgall, S. et al. AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments. arXiv:2405.07960 (2024). &lt; https://ui.adsabs.harvard.edu/abs/ 2024arXiv240507960S &gt; . - · arXiv
  23. [23][23] Liao, Y., Meng, Y., Liu, H., Wang, Y. &amp; Wang, Y. An Automatic Evaluation Framework for Multi-turn Medical Consultations Capabilities of Large Language Models. arXiv:2309.02077 (2023). &lt; https://ui.adsabs.harvard.edu/abs/2023arXiv230902077L &gt; . - · arXiv
  24. [24][24] Shi, X. et al. LLM-Mini-CEX: Automatic Evaluation of Large Language Model for Diagnostic Conversation. arXiv:2308.07635 (2023). &lt; https://ui.adsabs.harvard.edu/abs/ 2023arXiv230807635S &gt; . - · arXiv
  25. [25][25] Fast, D. et al. Autonomous medical evaluation for guideline adherence of large language models. NPJ Digit Med 7 , 358, doi:10.1038/s41746-024-01356-6 (2024). - · DOI
  26. [26][26] Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 (2023). &lt; https://ui.adsabs.harvard.edu/abs/ 2023arXiv230605685Z &gt; . - · arXiv
  27. [27][27] Croxford, E. et al. Automating Evaluation of AI Text Generation in Healthcare with a Large Language Model (LLM)-as-a-Judge. medRxiv , doi:10.1101/2025.04.22.25326219 (2025). - · DOI · Paper page
  28. [28][28] Wu, Y. et al. Effectiveness of various general large language models in clinical consensus and case analysis in dental implantology: a comparative study. BMC Med Inform Decis Mak 25 , 147, doi:10.1186/s12911-025-02972-2 (2025). - · DOI
  29. [29][29] Verlingue, L. et al. Artificial intelligence in oncology: ensuring safe and effective integration of language models in clinical practice. Lancet Reg Health Eur 46 , 101064, doi:10.1016/j.lanepe.2024.101064 (2024). - · DOI
  30. [30][30] Bedi, S. et al. MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks. arXiv:2505.23802 (2025). &lt; https://ui.adsabs.harvard.edu/abs/ 2025arXiv250523802B &gt; . - · arXiv
  31. [31][31] Zhang, Y. et al. Aligning Large Language Models with Humans: A Comprehensive Survey of ChatGPT ' s Aptitude in Pharmacology. Drugs 85 , 231-254, doi:10.1007/s40265-024-02124-2 (2025). - · DOI
  32. [32][32] Dinc, M. T., Bardak, A. E., Bahar, F. &amp; Noronha, C. Comparative analysis of large language models in clinical diagnosis: performance evaluation across common and complex medical cases. JAMIA Open 8 , ooaf055, doi:10.1093/jamiaopen/ooaf055 (2025). · DOI

Cited by

Sorted by citations

Authors
Wang, S., Tang, Z., Yang, H., Gong, Q., Gu, T., Ma, H., Wang, Y., Sun, W., Lian, Z., Mao, K., Jiang, Y., Huang, Z., Ma, L., Shen, W., Ji, Y., Tan, Y., Wang, C., Gao, Y., Ye, Q., Lin, R., Chen, M., Niu, L., Wang, Z., Yu, P., Lang, M., Liu, Y., Zhang, H., Shen, H., Chen, L., Zhao, Q., Liu, S.-X., Zhou, L., Gao, H., Ye, D., Meng, L., Yu, Y., Liang, N., & Wu, J.
Affiliations
  • Peking University (CN)
  • Chinese Academy of Medical Sciences & Peking Union Medical College (CN)
  • Chinese PLA General Hospital (CN)
  • Shanghai University of Traditional Chinese Medicine (CN)
  • Harbin Medical University (CN)
  • Tongji University (CN)
  • Shanghai Medical College of Fudan University (CN)
  • Xinjiang Medical University (CN)
  • Shanxi Medical University (CN)
  • Jinan University (CN)
  • Capital Medical University (CN)
Corresponding author
  • Yu, Youtao
  • Liang, Naixin
  • Wu, Jianxiong
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Volume
9
Issue
1
Pages
91-91
ISSN
2398-6352 (ISSN-L)
DOI
10.48550/arxiv.2507.23486
arXiv
2507.23486
PMC
PMC12855988
OpenAlex
W7117318448
Cited by
23
References
0
Citations by year
2026: 23
FWCI
5.54
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
medical large language models, benchmark evaluation, clinical safety, clinical effectiveness, clinical decision support, guideline adherence, drug safety, expert consensus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review