Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions
Open accessAbstract
Hanjie Chen, Zhouxiang Fang, Yash Singla, Mark Dredze. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Extensive prior work has benchmarked LLMs against medical board exams and sophisticated research-based questions.
“Extensive prior work has focused on advanced medical question answering, in which LLMs are benchmarked against medical board exams and sophisticated research-based questions13–17.”
From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [16]
The Related Work section says many LLMs have been evaluated on medical QA benchmarks, mostly board-exam or general medical questions. It also notes attempts to test challenging clinical cases, but describes those efforts as limited in scope or scale. Thus, the source supports extensive prior benchmarking on board exams, not extensive prior work on sophisticated research-based questions. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 19
- Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman. 2017. Overview of the medical question answering task at trec 2017 liveqa. In TREC, pages 1-12.
- Asma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis R Goodwin, Sonya E Shooshan, and Dina Demner-Fushman. 2019. Bridging the gap between consumers' medication questions and trusted an- swers. In MedInfo, pages 25-29.
- Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay, Quentin Malartic, et al. 2023. Falcon-40b: an open large language model with state- of-the-art performance. Findings of the Association for Computational Linguistics: ACL, 2023:10755- 10773.
- Asma Ben Abacha and Dina Demner-Fushman. 2019. A question-entailment approach to question answering. BMC bioinformatics, 20(1):1-23.
- Edward M Bennett, Renee Alpert, and AC Goldstein. 1954. Communications through limited-response questioning. Public Opinion Quarterly, 18(3):303- 308.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
- Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
- Fenglin Liu, Hongjian Zhou, Yining Hua, Omid Ro- hanian, Anshul Thakur, Lei Clifton, and David A Clifton. 2024. Large language models in the clinic: A comprehensive benchmark. medRxiv, pages 2024- 04.
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
- Ankit Pal, Logesh Kumar Umapathi, and Malaikan- nan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical do- main question answering. In Conference on Health, Inference, and Learning, pages 248-260.
- PMLR.
- Cecilia Panigutti, Andrea Beretta, Fosca Giannotti, and Dino Pedreschi. 2022. Understanding the impact of explanations on advice-taking: a user study for ai-based clinical decision support systems. In Pro- ceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-9.
- Yat-Fung Shea, Cynthia Min Yao Lee, Whitney Chin Tung Ip, Dik Wai Anderson Luk, and Stephanie Sze Wing Wong. 2023. Use of gpt-4 to analyze medical records of patients with extensive investi- gations and delayed diagnosis. JAMA Network Open, 6(8):e2325000-e2325000.
- Sarvesh Soni, Meghana Gudala, Atieh Pajouhi, and Kirk Roberts. 2022. Radqa: A question answering dataset to improve comprehension of radiology reports. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6250-6259.
- Eric Strong, Alicia DiGiammarino, Yingjie Weng, An- dre Kumar, Poonam Hosamani, Jason Hom, and Jonathan H Chen. 2023. Chatbot vs medical stu- dent performance on free-response clinical reasoning examinations. JAMA internal medicine, 183(9):1028- 1030.
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
- Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text gener- ation. Advances in Neural Information Processing Systems, 34:27263-27277.
- Xiang Yue, Xinliang Frederick Zhang, Ziyu Yao, Simon Lin, and Huan Sun. 2021. Cliniqg4qa: Generating diverse questions for domain adaptation of clinical question answering. In 2021 IEEE International Con- ference on Bioinformatics and Biomedicine (BIBM), pages 580-587. IEEE.
Cited by
Sorted by citations
Пантели, Д., Adib, K., Buttigieg, S., Goiana-da-Silva, F., Ladewig, K., Muscat, N. A., Figueras, J., Novillo-Ortiz, D., & McKee, M. (2025) · The Lancet Public Health · doi:10.1016/s2468-2667(25)00036-2 · Cited by 193 · see citing works
Holistic evaluation of large language models for medical tasks with MedHELM
Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04151-2 · Cited by 59 · see citing works
Large language models provide unsafe answers to patient-posed medical questions
Draelos, R. L., Afreen, S., Blasko, B., Brazile, T. L., Chase, N., Desai, D. P., Evert, J., Gardner, H. L., Herrmann, L., House, A. V., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02428-5 · Cited by 42 · see citing works
Gong, E. J., Bang, C. S., Lee, J. J., & Baik, G. H. (2025) · Journal of Medical Internet Research · doi:10.2196/84120 · Cited by 49 · see citing works
From System 1 to System 2: A Survey of Reasoning Large Language Models
Zhang, D., Li, Z., Zhang, M.-L., Zhang, J., Liu, Z., Yao, Y., Xu, H., Zheng, J., Chen, X., Zhang, Y., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3637037 · Cited by 172 · see citing works
Automating expert-level medical reasoning evaluation of large language models
Zhou, S., Xie, W., Li, J., Zhan, Z., Song, M., Yang, H., Espinoza, C., Welton, L., Mai, X., Jin, Y., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02208-7 · Cited by 30 · see citing works
Benchmarking large language model-based agent systems for clinical decision tasks
Liu, Y., Carrero, Z. I., Jiang, X., Ferber, D., Wölflein, G., Zhang, L., Jayabalan, S., Lenz, T., Hui, Z., & Kather, J. N. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02443-6 · Cited by 31 · see citing works
Ogdu, C. U., Arslanoğlu, K., & Karaköse, M. (2025) · IEEE Access · doi:10.1109/access.2025.3613340 · Cited by 18 · see citing works
McCaffrey, P., Jackups, R., Seheult, J. N., Zaydman, M. A., Balis, U. J., Thaker, H. M., Rashidi, H. H., & Gullapalli, R. R. (2024) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0208-ra · Cited by 13 · see citing works
MedCoAct: Confidence-Aware Multi-Agent Collaboration for Complete Clinical Decision
Zheng, H., Shi, Z., & Yi, P. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356820 · Cited by 6 · see citing works
Yao, M. S., Chae, A., Saraiya, P. V., Kahn, C. E., Witschey, W. R. T., Gee, J. C., Sagreiya, H., & Bastani, O. (2025) · Communications Medicine · doi:10.1038/s43856-025-01061-9 · Cited by 7 · see citing works
Susceptibility of Large Language Models to User-Driven Factors in Medical Queries
Lim, K. H., Kang, U., Li, X., Kim, J. S., Jung, Y., Park, S. J., & Kim, B.-H. (2025) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-025-00218-4 · Cited by 8 · see citing works
MedMobile: a mobile-sized language model with clinical capabilities
Vishwanath, K., Stryker, J., Alyakin, A., Alber, D., & Oermann, E. (2024) · BMJ digital health & AI · doi:10.1136/bmjdhai-2025-000068 · Cited by 7 · see citing works
Wen, A., Lu, Q., Chuang, Y.-N., Wang, G., Yuan, J., Zhang, J., Wang, L. H., Fu, S., Miller, K., Jia, H., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02253-2 · Cited by 2 · see citing works
Oh, G., Kim, S., Park, S., & Kim, B.-H. (2026) · Journal of Medical Internet Research · doi:10.2196/90693 · Cited by 1 · see citing works
Revolutionizing Medical Inquiry Responses Harnessing Advanced Language Models for Enhanced Precision
Keerthana, S. M., Kirthidevi, S., & Nandhana, V. (2024) · International Conference on Emerging Research in Computational Science (ICERCS) · doi:10.1109/icercs63125.2024.10895183 · Cited by 1 · see citing works
Hierarchical agent reflection for aligning LLM reasoning with clinical diagnostic processes
Wang, X., Li, X., Zhao, D., Feng, K., Liang, L., Zhang, Z., Ding, K., Chen, H., Wan, B., & Zhang, Q. (2026) · Health Information Science and Systems · doi:10.1007/s13755-025-00410-1 · Cited by 1 · see citing works
Self-citationHe, N., Yang, L., Hu, X., & He, Y. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1785443 · Cited by 1 · see citing works
Large Language Models for Clinical Narrative Processing: Methods, Applications, and Challenges
Livieratos, A., Lin, J., Chasani, P., Gaga, M., Fousekis, F. S., Gogos, C. A., Akinosoglou, K., Katsanos, K. H., & Gamalo, M. (2026) · Methods and Protocols · doi:10.3390/mps9030069 · Cited by 1 · see citing works
Foundation model embeddings for quantitative tumor imaging biomarkersPreprint
Pai, S., Hadžić, I., Fedorov, A., Mak, R. H., & Aerts, H. J. W. L. (2025) · Research Square · doi:10.21203/rs.3.rs-6630446/v1 · Cited by 1 · see citing works
LatentEvolve: Self-Evolving Test-Time Scaling in Latent Space
Zhang, G., Meng, F., Wan, G., Li, Z., Wang, K., Yin, Z., Bai, L., & Yan, S. (2025) · arXiv · doi:10.48550/arxiv.2509.24771
Wang, Y., Zhang, X., Yao, K., Zeng, Z., Song, K., Lin, J., & Shen, Z. (2026) · arXiv · doi:10.48550/arxiv.2606.05241
KG-Infused LLM for Virtual Health Assistant: Accelerated Inference and Enhanced Performance
Katta, S. K., Ray, A., Firouzi, F., & Chakrabarty, K. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00099
Psych-R1: A New LLM for Psychiatric Diseases Based on Group Relative Policy Optimization
Luo, Z. F., Xu, Z., & Yang, Q. (2025) · ICME International Conference on Complex Medical Engineering (CME) · doi:10.1109/cme67420.2025.11239382
promises, challenges, and an agenda for policy makers and public health institutions
Panteli, D., Adib, K., Buttigieg, S., Goiana-da-Silva, F., Ladewig, K., Azzopardi-Muscat, N., Figueras, J., Novillo-Ortiz, D., & McKee, M. (2025) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
ALIAS: An Open-Ended Adaptively Learnable Intelligent AI System for Efficient Domain Adaptation
Vetcha, N., & Ganesan, S. (2026) · Australasian Computer Science Week Multiconference (ACSW) · doi:10.1145/3793811.3793815
Epistemologically Guided LLM Reasoning for Differential Diagnosis
Rancati, S., Bergomi, L., Parimbelli, E., Nicora, G., & Bellazzi, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30710-1_14
A Cloud Native Data and Intelligence Framework for Medical Reasoning and Digital Health Analysis
Arya, P., M, D. B., & Khanna, M. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.06.356
Niu, Z., Du, C., Wang, B., Song, Y., Xiong, J., & Li, L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3426-4_36
Shao, M., Shao, H., Zhang, H., & Zhang, H. (2026) · doi:10.2196/preprints.108814
Zhang, J., Li, Y., Feng, X., Zhang, L., Huang, Z., & Chen, C. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818853
Decision models can offload structured clinical tasks from large language modelsPreprint
Ariel, D., Laban, G., Endelman, S., Balicer, R., Dagan, N., & Cohen, S. (2026) · medRxiv · doi:10.64898/2026.10.09.26365130
Elicit and Enhance: Advancing Multimodal Reasoning in Medical Scenarios
Huang, Z., Mu, L., Gu, Y., Hu, K., Hua, S., & Zhang, X. (2026) · doi:10.1109/cvprf71909.2026.00539
- Authors
- Chen, H., Fang, Z., Singla, Y., & Dredze, M.
- Affiliations
- Rice University (US)
- Johns Hopkins University (US)
- Venue
- Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
- Published
- 2025
- Pages
- 3563-3599
- OpenAlex
- W4411119840
- Cited by
- 31
- References
- 0
- Citations by year
- 2026: 15 · 2025: 13 · 2024: 3
- FWCI
- 26.59
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, benchmarking, medical question answering, medical explanation, model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
