Quantifying the reasoning abilities of LLMs on clinical cases
Open accessAbstract
Recent advances in reasoning-enhanced large language models (LLMs) show promise, yet their application in professional medicine, especially the evaluation of their reasoning process, remains underexplored. We present MedR-Bench, a benchmark of 1453 structured patient cases with reference reasoning derived from clinical case reports, spanning 13 body systems and 10 specialties across common and rare diseases. Our evaluation framework covers three stages of care: examination recommendation, diagnostic decision-making, and treatment planning. To assess reasoning quality, we develop the Reasoning Evaluator, an automated scorer of written reasoning along efficiency, factual accuracy, and completeness. We evaluate seven state-of-the-art reasoning LLMs. Here we show that current models exceed 85% accuracy on simple diagnostic tasks when sufficient examination results are available, but performance drops on examination recommendation and treatment planning. Reasoning is generally factual, yet critical steps are often missing. Open-source models are closing the gap with proprietary systems, highlighting potential for more accessible, equitable clinical AI. Evaluation of medical LLMs’ reasoning process in professional medicine remains underexplored. Here, the authors present MedR Bench, which evaluates LLMs’ medical reasoning across exam recommendation, diagnosis, and treatment. They find that models excel at diagnosis but struggle with exams and treatment.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MedR-Bench revealed that models achieve 81.34% accuracy in static knowledge retrieval, but success rate plummets to 22.34% when managing dynamic, patient-specific clinical workflows.
“MedR-Bench revealed a performance gap [15]: while models achieve 81.34% accuracy in static knowledge retrieval, their success rate plummets to just 22.34% when tasked with managing dynamic, patient-specific clinical workflows.”
From Multi-Agent Dynamic Refinement Outperforms Static RAG in Clinical Reasoning for Complex Nephrology Cases (2026) · cites this paper as [15]
The Introduction (page unavailable) reports over 85% accuracy on relatively simple diagnostic tasks but difficulty with examination recommendation and treatment planning. The Results in treatment planning (page unavailable) report a substantially lower best model accuracy of 30.65%. This supports a performance gap, but our copy does not report 81.34% for static knowledge retrieval or 22.34% for dynamic clinical workflows. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. (2025) · Nature · doi:10.1038/s41586-025-09422-z · Cited by 1,701 · cited 1× · see citing works
“These models have excel-led in structured problem-solving and logical inference, achieving notable success in fi elds like mathematics and programming 2 – 4 .”
Self-citationGemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
Self-citationEvaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · cited 2× · see citing works
“Similar to previous works 14,26,27 , we initialize an LLM-powered agent to play the role of the patient.”
ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing
Neumann, M., King, D., Beltagy, I., & Ammar, W. (2019) · BioNLP Workshop and Shared Task (BioNLP) · doi:10.18653/v1/w19-5034 · Cited by 772 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · cited 3× · see citing works
“First, these benchmarks predominantly focus on exam-style 19,20 questions or sourcing from online conversational cases 18 , these sour-cestypicallydonotfullycapturethedepth,complexity,andstructured reasoning present in real-world clinical practice.”
PMC-LLaMA: toward building open-source language models for medicine
Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae045 · Cited by 300 · see citing works
Self-citationTowards generalist foundation model for radiology by leveraging web-scale 2D&3D medical data
Wu, C., Zhang, X., Zhang, Y., Hui, H., Wang, Y., & Xie, W. (2025) · Nature Communications · doi:10.1038/s41467-025-62385-7 · Cited by 273 · see citing works
Self-citationBenchmark evaluation of DeepSeek large language models in clinical decision-making
Sandmann, S., Hegselmann, S., Fujarski, M., Bickmann, L., Wild, B., Eils, R., & Varghese, J. (2025) · Nature Medicine · doi:10.1038/s41591-025-03727-2 · Cited by 233 · see citing works
Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks
Sandmann, S., Riepenhausen, S., Plagwitz, L., & Varghese, J. (2024) · Nature Communications · doi:10.1038/s41467-024-46411-8 · Cited by 219 · see citing works
An evaluation framework for clinical use of large language models in patient interaction tasks
Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03328-5 · Cited by 201 · see citing works
Tordjman, M., Liu, Z., Yuce, M., Fauveau, V., Mei, Y., Hadjadj, J., Bolger, I., Almansour, H., Horst, C., Parihar, A. S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03726-3 · Cited by 177 · see citing works
Towards building multilingual language model for medicine
Qiu, P., Wu, C., Zhang, X., Lin, W., Wang, H., Zhang, Y., Wang, Y., & Xie, W. (2024) · Nature Communications · doi:10.1038/s41467-024-52417-z · Cited by 147 · see citing works
Self-citation[Orphanet: a European database for rare diseases].
Weinreich, S. S., Mangon, R., Sikkens, J. J., et al. (2008) · PubMed · Cited by 138 · see citing works
A large-scale dataset of patient summaries for retrieval-based clinical decision support systems
Zhao, Z., Jin, Q., Chen, F., Peng, T., & Yu, S. (2023) · Scientific Data · doi:10.1038/s41597-023-02814-8 · Cited by 63 · see citing works
Towards evaluating and building versatile large language models for medicine
Wu, C., Qiu, P., Liu, J., Gu, H., Li, N., Zhang, Y., Wang, Y., & Xie, W. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01390-4 · Cited by 52 · see citing works
Self-citationYi: Open Foundation Models by 01.AI
AI, 0., :, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Wang, G., Li, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.04652 · Cited by 50 · see citing works
Self-citationOpenAI, :, Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., et al. (2024) · arXiv · Cited by 50 · see citing works
Self-citationBaichuan-M1: Pushing the Medical Capability of Large Language ModelsPreprint
Wang, B., Zhao, H., Zhou, H., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2502.12671 · Cited by 49 · see citing works
RaTEScore: A Metric for Radiology Report Generation
Zhao, W., Wu, C., Zhang, X., Zhang, Y. P., Wang, Y., & Xie, W. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.836 · Cited by 32 · see citing works
Self-citationEvaluation of OpenAI o1: Opportunities and Challenges of AGIPreprint
Zhong, T., Liu, Z., Pan, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.18486 · Cited by 22 · see citing works
Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
Liao, Y., Meng, Y., Wang, Y., Liu, H., Wang, Y., & Wang, Y. (2024) · arXiv · doi:10.48550/arxiv.2403.08495 · Cited by 15 · see citing works
Self-citationA Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?Preprint
Xie, Y., Wu, J., Tu, H., Yang, S., Zhao, B., Zong, Y., Jin, Q., Xie, C., & Zhou, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.15277 · Cited by 15 · see citing works
CoD, towards an interpretable medical agent using chain of diagnosis
Chen, J., Gui, C., Gao, A., Ji, K., Wang, X., Wan, X., & Wang, B. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.740 · Cited by 13 · see citing works
Radiology-Aware Model-Based Evaluation Metric for Report GenerationPreprint
Calamida, A., Nooralahzadeh, F., Rohanian, M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.16764 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- [4]Phan, L. et al. Humanity’s last exam. Preprint at arXiv10.48550/arXiv.2501.14249 (2025).
- [7]Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proc.Conference on Health, Inference, and Learning 248–260 (PMLR, 2022).
- [15]Lamparth, M. et al. Moving beyond medical exam questions: a clinician-annotated dataset of real-world tasks and ambiguity in mental healthcare. Preprint at 10.48550/arXiv.2502.16051 (2025). · DOI
- [20]Zuo, Y. et al. MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding. In Forty-second International Conference on Machine Learning (ICML 2025).
- [24]National Library of Medicine. PMC open access subset. https://pmc.ncbi.nlm.nih.gov/tools/openftlist/ (2003).
- [40]Qiu, P. et al. Quantifying the reasoning abilities of LLMs on clinical cases. 10.5281/zenodo.17046070 (2025). PMC12592457 41198657 · DOI
Cited by
Sorted by citations
Teng, D., Tan, L., Cao, Q., Xia, Y., Zhang, N., Li, J., & Zhao, D. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02547-z · Cited by 18 · see citing works
Structured reasoning failures compromise LLM interpretation of clinical oncology notes
Kenaston, M. W., Ayub, U., Parmar, M., Kumar, P., Anjum, M. U., Naqvi, S. A. A., Chaudhuri, A. A., Zakharia, Y., Heath, E. I., Bekaii-Saab, T. S., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02951-5 · Cited by 2 · see citing works
Yano, Y., Kakizaki, H., Nagasu, H., Kishi, S., Koshida, T., Nihei, Y., Hirano, A., Sugawara, Y., Imaizumi, T., Osakabe, Y., et al. (2026) · doi:10.64898/2026.07.15.26358121
- Authors
- Qiu, P., Wu, C., Liu, S., Fan, Y., Zhao, W., Chen, Z., Gu, H., Peng, C., Zhang, Y., Wang, Y., & Xie, W.
- Affiliations
- Shanghai Jiao Tong University (CN)
- XinHua Hospital (CN)
- China Mobile (China) (CN)
- Corresponding author
- Wang, Yanfeng
- Xie, Weidi
- ORCID
- Venue
- Nature Communications
- Published
- 2025-11-06
- Volume
- 16
- Issue
- 1
- Pages
- 9799-9799
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC12592457
- OpenAlex
- W4415956903
- Cited by
- 53
- References
- 29
- Citations by year
- 2026: 50 · 2025: 3
- FWCI
- 12.67
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Nature Communications (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Scientific Research Innovation Capability Support Project for Young Faculty
- National Key R&D Program of China
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- large language models, medical reasoning, diagnostic decision-making, treatment planning, open-source vs closed-source models
- MeSH terms
- Clinical Decision-Making, Clinical Reasoning, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
