MedS³: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process Supervision
Open accessAbstract
Medical language models face critical barriers to real-world clinical reasoning applications. However, mainstream efforts, which fall short in task coverage, lack fine-grained supervision for intermediate reasoning steps, and rely on proprietary systems, are still far from a versatile, credible and efficient language model for clinical reasoning usage. To this end, we propose MedS3, a self-evolving framework that imparts robust reasoning capabilities to small, deployable models. Starting with 8,000 curated instances sampled via a curriculum strategy across five medical domains and 16 datasets, we use a small base policy model to conduct Monte Carlo Tree Search (MCTS) for constructing rule-verifiable reasoning trajectories. Self-explored reasoning trajectories ranked by node values are used to bootstrap the policy model via reinforcement fine-tuning and preference learning. Moreover, we introduce a soft dual process reward model that incorporates value dynamics: steps that degrade node value are penalized, enabling fine-grained identification of reasoning errors even when the final answer is correct. Experiments on eleven benchmarks show that MedS3 outperforms the previous state-of-the-art medical model by +6.45 accuracy points and surpasses 32B-scale general-purpose reasoning models by +8.57 points. Additional empirical analysis further demonstrates that MedS3 achieves robust and faithful reasoning behavior.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent studies demonstrate that the use of reasoning models and test-time scaling significantly boosts performance on medical benchmark datasets.
“Recent studies demonstrate that the use of reasoning models and test-time scaling significantly boosts performance on medical benchmark datasets [30].”
From Model and Task-Aware Test-Time Scaling Strategies for Large Language and Vision-Language Models in Medicine: Evaluation Study (2026) · cites this paper as [30]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Lessons of Developing Process Reward Models in Mathematical ReasoningPreprint
Zhang, Z.-R., Zheng, C., Wu, Y., Zhang, B.-C., Lin, R.-J., Yu, B.-W., Liu, D., Zhou, J.-R., & Lin, J.-Y. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2501.07301 · Cited by 528 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
Hallucination is Inevitable: An Innate Limitation of Large Language Models
Xu, Z., Jain, S., & Kankanhalli, M. (2024) · arXiv · doi:10.48550/arxiv.2401.11817 · Cited by 180 · see citing works
Med42-v2: A Suite of Clinical LLMs
Christophe, C., Kanithi, P., Raha, T., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2408.06142 · Cited by 125
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., & Cobbe, K. (2023) · arXiv · doi:10.48550/arxiv.2305.20050 · Cited by 65 · see citing works
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.10671 · Cited by 63 · see citing works
Self-citationOpenAI, :, Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., et al. (2024) · arXiv · Cited by 50 · see citing works
Self-citationO1 Replication Journey - Part 3: Inference-time Scaling for Medical Reasoning
Huang, Z.-Z., Geng, G., Hua, S., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2501.06458 · Cited by 40
A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?Preprint
Xie, Y., Wu, J., Tu, H., Yang, S., Zhao, B., Zong, Y., Jin, Q., Xie, C., & Zhou, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.15277 · Cited by 15 · see citing works
Yu, H., Cheng, T., Wang, Y.-W., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.09213 · Cited by 14 · see citing works
Self-citationFrom Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond
Nori, H., Usuyama, N., King, N., McKinney, S. M., Fernandes, X., Zhang, S., & Horvitz, E. (2024) · arXiv · doi:10.48550/arxiv.2411.03590 · Cited by 9 · see citing works
Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac PatientsPreprint
Jung, H., Kim, Y., Choi, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.05144 · Cited by 8 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 28
- Biswas, A.; and Talukdar, W. 2024. Intelligent Clinical Doc- umentation: Harnessing Generative AI for Patient-Centric Clinical Note Generation. arXiv preprint arXiv:2405.18346. · arXiv
- Cabral, S.; Restrepo, D.; Kanjee, Z.; Wilson, P.; Crowe, B.; Abdulnour, R.-E.; and Rodman, A. 2024. Clinical reasoning of a generative artificial intelligence model compared with physicians. JAMA internal medicine, 184(5): 581-583.
- Chen, J.; Cai, Z.; Ji, K.; Wang, X.; Liu, W.; Wang, R.; and Wang, B. 2025. Towards Medical Complex Reasoning with LLMs through Medical Verifiable Problems. In Che, W.; Nabende, J.; Shutova, E.; and Pilehvar, M. T., eds., Findings of the Association for Computational Linguistics: ACL 2025, 14552-14573. Vienna, Austria: Association for Computa- tional Linguistics.
- Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv e-prints, arXiv-2407.
- Guan, X.; Zhang, L. L.; Liu, Y.; Shang, N.; Sun, Y.; Zhu, Y.; Yang, F.; and Yang, M. 2025. rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking. In Forty-second International Conference on Machine Learning.
- Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu, R.; Zhang, R.; Ma, S.; Bi, X.; et al. 2025. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081): 633-638.
- Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021. Measuring Massive Multi- task Language Understanding. In International Conference on Learning Representations.
- Huang, X.; Wu, J.; Liu, H.; Tang, X.; and Zhou, Y. 2025a. m1: Unleash the Potential of Test-Time Scaling for Medical Rea- soning in Large Language Models. In The 2nd Workshop on GenAI for Health: Potential, Trust, and Policy Compliance.
- Matsumoto, Y.; and Prasad, R., eds., Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, 877-887. Osaka, Japan: The COLING 2016 Organizing Committee.
- Jin, D.; Pan, E.; Oufattole, N.; Weng, W.-H.; Fang, H.; and Szolovits, P. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14): 6421.
- Jin, Q.; Dhingra, B.; Liu, Z.; Cohen, W.; and Lu, X. 2019. PubMedQA: A Dataset for Biomedical Research Question Answering. In Inui, K.; Jiang, J.; Ng, V.; and Wan, X., eds., Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), 2567-2577. Hong Kong, China: Association for Computational Linguistics.
- Khandekar, N.; Jin, Q.; Xiong, G.; Dunn, S.; Applebaum, S.; Anwar, Z.; Sarfo-Gyamfi, M.; Safranek, C.; Anwar, A.; Zhang, A.; et al. 2024. Medcalc-bench: Evaluating large lan- guage models for medical calculations. Advances in Neural Information Processing Systems, 37: 84730-84745.
- Kotonya, N.; and Toni, F. 2020. Explainable Automated Fact- Checking for Public Health Claims. In Webber, B.; Cohn, T.; He, Y.; and Liu, Y., eds., Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), 7740-7754. Online: Association for Computa- tional Linguistics.
- Liao, Y.; Jiang, S.; Chen, Z.; Wang, Y.; and Wang, Y. 2024. MedCare: advancing medical LLMs through decoupling clin- ical alignment and knowledge aggregation. In Findings of the Association for Computational Linguistics: EMNLP 2024, 10562-10581.
- Pal, A.; Umapathi, L. K.; and Sankarasubbu, M. 2022. Medm- cqa: A large-scale multi-subject multi-choice dataset for med- ical domain question answering. In Conference on health, inference, and learning, 248-260.
- PMLR.
- Cohen, K. B.; Ananiadou, S.; and Tsujii, J., eds., Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, 140-149. Online: Association for Computational Linguistics.
- Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Er- mon, S.; and Finn, C. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 53728-53741.
- Singhal, K.; Azizi, S.; Tu, T.; Mahdavi, S. S.; Wei, J.; Chung, H. W.; Scales, N.; Tanwani, A.; Cole-Lewis, H.; Pfohl, S.; et al. 2023. Large language models encode clinical knowl- edge. Nature, 620(7972): 172-180.
- Tsatsaronis, G.; Schroeder, M.; Paliouras, G.; Almirantis, Y.; Androutsopoulos, I.; Gaussier, E.; Gallinari, P.; Artieres, T.; Alvers, M. R.; Zschunke, M.; et al. 2012. Bioasq: A challenge on large-scale biomedical semantic indexing and question answering. In 2012 AAAI Fall Symposium Series.
- Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2024. Math-Shepherd: Verify and Rein- force LLMs Step-by-step without Human Annotations. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9426-9439. Bangkok, Thailand: Association for Computational Linguistics.
- Wang, T.; Jiang, Z.; He, Z.; Yang, W.; Zheng, Y.; Li, Z.; He, Z.; Tong, S.; and Gong, H. 2025. Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models. arXiv preprint arXiv:2503.13551. · arXiv
- Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824-24837.
- Winands, M. H.; Björnsson, Y.; and Saito, J.-T. 2008. Monte- Carlo tree search solver. In Computers and Games: 6th In- ternational Conference, CG 2008, Beijing, China, September 29-October 1, 2008. Proceedings 6, 25-36. Springer.
- Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 11809-11822. Yonsei. 2024. MedLLAMA3-v20. website.
- Zhang, D.; Zhoubian, S.; Hu, Z.; Yue, Y.; Dong, Y.; and Tang, J. 2024a. Rest-mcts*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37: 64735-64772.
- Zhang, K.; Zeng, S.; Hua, E.; Ding, N.; Chen, Z.-R.; Ma, Z.; Li, H.; Cui, G.; Qi, B.; Zhu, X.; Lv, X.; Jinfang, H.; Liu, Z.; and Zhou, B. 2024b. UltraMedical: Building Special- ized Generalists in Biomedicine. In The Thirty-eight Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track.
- Zuo, Y.; Qu, S.; Li, Y.; Chen, Z.-R.; Zhu, X.; Hua, E.; Zhang, K.; Ding, N.; and Zhou, B. 2025. MedXpertQA: Benchmark- ing Expert-Level Medical Reasoning and Understanding. In Forty-second International Conference on Machine Learning.
Cited by
Sorted by citations
Oh, G., Kim, S., Park, S., & Kim, B.-H. (2026) · Journal of Medical Internet Research · doi:10.2196/90693 · Cited by 1 · see citing works
- Authors
- Jiang, S., Liao, Y., Chen, Z., Zhang, Y., Wang, Y., & Wang, Y.
- Affiliations
- Fudan University (CN)
- Shanghai Jiao Tong University (CN)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2026-03-14
- Volume
- 40
- Issue
- 37
- Pages
- 31319-31327
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W7137832862
- Cited by
- 4
- References
- 13
- Citations by year
- 2026: 4
- FWCI
- 37.42
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- National Natural Science Foundation of China
- Science and Technology Commission of Shanghai Municipality
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- clinical language models, clinical reasoning, reasoning paths, Monte Carlo tree search, reinforcement learning from human feedback, preference learning, curriculum learning, small language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
