Toward expert-level medical question answering with large language models
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a 'passing' score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These models were chosen as they represent current leading LLMs3,36,40,41 as of February 2024.”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To account for the inherent stochasticity of LLMs, we report results from 10 independent runs per benchmark as mean accuracy with 95% confidence intervals (CIs), moving beyond the single-run point estimates common in prior works.2,3,4”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“medical domain-specific LLMs such as Medical Pathways Language Model (Med-PaLM) [2] and Biomedical Generative Pretrained Transformer (BioGPT) [3] have shown improved performance across various clinical natural language processing (NLP) tasks, including question answering, summarization, entity recognition, relation extraction, document classification, and literature mining.”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs have demonstrated strong performance across a wide range of biomedical knowledge and reasoning tasks, accelerating their adoption in health care workflows [1-6].”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models (LLMs) like OpenAI’s GPT-5 and knowledge-specific systems, such as OpenEvidence, have reported the ability to generate patient-specific, evidence-based recommendations.10”
From Large Language Model–Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations (2026) · cites this paper as [10]
Experimental — accuracy not yet measured
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“The advent of transformers 4 and LLMs 5,6 has renewed interest in the possibilities of AI for medical question-answering tasks—a long-standing ‘grand challenge’ 7–9 .”
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
New creatinine- and cystatin C-based equations to estimate GFR without race
Inker, L. A., Eneanya, N. D., Coresh, J., Tighiouart, H., Wang, D., Sang, Y., Crews, D. C., Doria, A., Estrella, M. M., Froissart, M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2102953 · Cited by 4,695
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“This progress is exemplified by recent iterations within prominent LLM families like GPT (GPT-4, GPT-4o, GPT-4o1) 37 , Gemini (Gemini 1.0, Gemini 1.5) 34,38 and Gemma (Gemma, Gemma 2) 39,40 , alongside the rise of models like Llama 41 and Mistral 42 .”
InfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“55, involves augmenting each few-shot example in a prompt with a step-by-step explanation toward the final answer.”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citation
Research works printed but not linked: 4
- 5. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT Vol. 1 (eds Burstein, J. et al.) 4171-4186 (Association for Computational Linguistics, 2019).
- 10. Bobrow, D. G.) 167-180 (MIT Press, 1994).
- 38. Achiam, J. et al. GPT-4 technical report. Preprint at https://arxiv. org/abs/2303.08774 (2023).
- 63. Randolph, Justus J. 2005 "Free-Marginal Multirater Kappa (multirater K [free]): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa." Presented at the Joensuu Learning and Instruction Symposium, vol. 2005 https://eric.ed.gov/?id=ED490661
Cited by
Items 301–400 of 1,031
Chimirri, L., Caufield, J. H., Bridges, Y., Matentzoglu, N., Gargano, M. A., Cazalla, M., Chen, S., Daniš, D., Dingemans, A. J. M., Gehle, K., et al. (2025) · EBioMedicine · doi:10.1016/j.ebiom.2025.105957 · Cited by 30
“…proficient in supporting differential diagnosis and related domains such as bedside consultation question answering and addressing questions from the general public, 38 there is a great potential to improve care for people in LMICs by supplementing existing systems with LLM-driven services.”
Large language models and global health equity: a roadmap for equitable adoption in LMICs
Chen, H., Zeng, D., Qin, Y., Fan, Z., Ci, F. N. Y., Klonoff, D. C., Ji, J. S., Zhang, S., Amissah-Arthur, K. N., Jiménez, M., et al. (2025) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2025.101707 · Cited by 47
“In addition, the cutting-edge well-known medical LLMs are primarily developed by a few leading countries, including the United States (AIME, 24 Medpalm 25 ), and China (Huatuo, 26 MedFound, 12 Baichuan-M1 27 ).”
Yin, C., Du, K., Nong, Q., Zhang, H., Yang, L., Yan, B., Huang, X., Wang, X., & Zhang, X. (2023) · Expert Systems · doi:10.1111/exsy.13513 · Cited by 26
LLM-based Solutions for Healthcare Chatbots: a Comparative Analysis
Montagna, S., Aguzzi, G., Ferretti, S., Pengo, M. F., Klopfenstein, L. C., Ungolo, M., & Magnini, M. (2024) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops59983.2024.10503257 · Cited by 29
Medical Dialogue System: A Survey of Categories, Methods, Evaluation and Challenges
Shi, X., Liu, Z., Du, L., Wang, Y., Wang, H., Guo, Y., Ruan, T., Xu, J., Zhang, X., & Zhang, S. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.167 · Cited by 51
Gaps in the Safety Evaluation of Generative AI
Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Comanescu, R., Akbulut, C., Stepleton, T., Mateos-Garcia, J. C., Bergman, S., Kay, J., et al. (2024) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v7i1.31717 · Cited by 71
Carchiolo, V., & Malgeri, M. (2025) · Future Internet · doi:10.3390/fi17020076 · Cited by 26
Roșca, C.-M., Bold, R.-A., & Gerea, A.-E. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-97-5703-9_13 · Cited by 23
Development of a Preliminary Patient Safety Classification System for Generative AI
Hose, B., Handley, J. L., Biro, J., Reddy, S., Krevat, S. A., Hettinger, A. Z., & Ratwani, R. M. (2025) · BMJ Quality & Safety · doi:10.1136/bmjqs-2024-017918 · Cited by 24
Xu, T., Chen, L., Hu, Z., & Li, B. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127582 · Cited by 30
Integrating LLMs and Knowledge Graphs for Medical AI: Advances, Challenges, and Future Directions
Murali, L., Gopakumar, G., Viswanathan, D. M., Raman, R., & Nedungadi, P. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3622058 · Cited by 21
“Additionally, LLMs augmented with medical textbooks (LLM-AMT) have demonstrated substantial improvements in response quality for medical question answering (QA) tasks [14], outperforming specialized models such as Med-PaLM 2 [15].”
Bailicai: A Domain-Optimized Retrieval-Augmented Generation Framework for Medical Applications
Cui, L., Liu, Y., Ouyang, C., Yu, Y., Zhang, G., Wan, Y., & Yang, F. (2026) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020097 · Cited by 28
Self-citationPotential merits and flaws of large language models in epilepsy care: A critical review
Diessen, E. G. A. L. V., van Amerongen, R. A., Zijlmans, M., & Otte, W. M. (2024) · Epilepsia · doi:10.1111/epi.17907 · Cited by 26
Domain-Adapted LLMs for VLSI Design and Verification: A Case Study on Formal Verification
Liu, M., Kang, M., Hamad, G. B., Suhaib, S. M., & Ren, H. M. (2024) · IEEE VLSI Test Symposium (VTS) · doi:10.1109/vts60656.2024.10538589 · Cited by 34
Khalpey, Z. I., Kumar, U. A., King, N. C., Abraham, A., & Khalpey, A. (2024) · Cureus · doi:10.7759/cureus.65083 · Cited by 22
Teng, D., Tan, L., Cao, Q., Xia, Y., Zhang, N., Li, J., & Zhao, D. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02547-z · Cited by 18
Liu, Y., Li, X., Luo, Y., Du, J., Zhang, Y., Lv, T., Yin, H., Tang, X., & Liu, H. (2025) · Engineering · doi:10.1016/j.eng.2025.02.010 · Cited by 22
Self-citationEfficient and Reproducible Biomedical Question Answering Using Retrieval Augmented Generation
Stuhlmann, L., Saxer, M. A., & Fürst, J. (2025) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds66131.2025.00029 · Cited by 25
Artificial intelligence in primary care: innovation at a crossroads
Laranjo, L., Car, L. T., Payne, R. E., Neves, A. L., Kidd, M. R., & Miranda, J. J. (2025) · The Lancet Primary Care · doi:10.1016/j.lanprc.2025.100078 · Cited by 36
Wen, P., Ai, Q., Fan, X., Gu, Z., 陈波, & Qian, H. (2026) · Cancer Letters · doi:10.1016/j.canlet.2026.218257 · Cited by 21
ChatGPT Yields a Passing Score on a Pediatric Board Preparatory Exam but Raises Red Flags
Le, M., & Davis, M. K. (2024) · Global Pediatric Health · doi:10.1177/2333794x241240327 · Cited by 26
Text summarization with ChatGPT for drug labeling documents
Ying, L., Liu, Z., Fang, H., Kusko, R. L., Wu, L., Harris, S., & Tong, W. (2024) · Drug Discovery Today · doi:10.1016/j.drudis.2024.104018 · Cited by 23
Kambhamettu, H., Metaxa, D., Johnson, K. B., & Head, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642573 · Cited by 28
Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions
Roy, S., Khatua, A., Ghoochani, F., Hadler, U., Nejdl, W., & Ganguly, N. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657882 · Cited by 17
Pugh, S. L., Chandler, C., Cohen, A. S., Diaz-Asper, C. M., Elvevåg, B., & Foltz, P. W. (2024) · Psychiatry Research · doi:10.1016/j.psychres.2024.116119 · Cited by 21
A Survey of Large Language Model for Drug Research and Development
Guo, H., Xing, X., Zhou, Y., Jiang, W., Chen, X., Wang, T., Jiang, Z., Wang, Y., Hou, J., Jiang, Y., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3552256 · Cited by 24
Akbasli, I. T., Bırbılen, A. Z., & Tekşam, Ö. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02871-6 · Cited by 22
Trials for LLM-supported clinical decisions in African primary healthcare
Mateen, B. A., Menon, V., Agweyu, A., Korom, R., Omoluabi, E., McAfee, D., Shimelash, N., Rutunda, S., Rugege, C., Williams, G., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03815-3 · Cited by 22
Elzayyat, M., Mohammad, J. N., & Zaqout, S. (2025) · Medical Education Online · doi:10.1080/10872981.2025.2554678 · Cited by 22
“These trends align with existing literature comparing LLM performance: GPT-4 (ChatGPT) often outperforms Gemini on medical knowledge tasks [2,6,22], and studies like Rossettini et al. support this model-specific variance [23].”
Bajramagic, M., Battelino, T., Cos, X., Coté, M., Cui, N., Forbes, A., Galderisi, A., Heinemann, L., Hussain, S., Imbert, J., et al. (2025) · Diabetologia · doi:10.1007/s00125-025-06601-5 · Cited by 23
Principles to guide clinical AI readiness and move from benchmarks to real-world evaluation
Azad, T. D., Krumholz, H. M., & Saria, S. (2026) · Nature Medicine · doi:10.1038/s41591-025-04198-1 · Cited by 28
Rollwage, M., McFadyen, J., Juechems, K., Balogh, A., Pisupati, S., Mircea, M.-T., Hauser, T. U., Prichard, G., & Harper, R. J. (2026) · Nature Medicine · doi:10.1038/s41591-026-04278-w · Cited by 32
“Unlike other AI systems that rely heavily on prompt engineering, instruction optimization or fine-tuned foundational models 9,10,33 , this scaffolding architecture of our system is model agnostic and can thus act as a versatile controller for a wide range of foundation models.”
InfluentialKarusala, N., Yan, S. D., Rajkumar, N., Victoria, G., & Anderson, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610210 · Cited by 21
An overview of diagnostics and therapeutics using large language models
Malgaroli, M., & McDuff, D. (2024) · Journal of Traumatic Stress · doi:10.1002/jts.23082 · Cited by 23
GPT-4's performance in supporting physician decision-making in nephrology multiple-choice questions
Noda, R., Tanabe, K., Ichikawa, D., & Shibagaki, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-99774-3 · Cited by 24
Mese, I., D’Antonoli, T. A., Bluethgen, C., Bressem, K., Cuocolo, R., Chaudhari, A., Tejani, A. S., Isaac, A., Ponsiglione, A., Meddeb, A., et al. (2026) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2026.263812 · Cited by 17
Jiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72130-7 · Cited by 18
Yang, R., Zeng, Q., You, K., Qiao, Y., Huang, L., Hsieh, C.-C., Rosand, B., Goldwasser, J., Dave, A., Keenan, T., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/60601 · Cited by 16
Recommended antibiotic treatment agreement between infectious diseases specialists and ChatGPT®
Montiel-Romero, S., Rajme‐López, S., Román-Montes, C. M., López-Íñiguez, Á., Rivera-Villegas, H., Ochoa‐Hein, E., González-Lara, M. F., Ponce‐de‐León, A., Tamez-Torres, K. M., & Martínez-Guerra, B. A. (2025) · BMC Infectious Diseases · doi:10.1186/s12879-024-10426-9 · Cited by 16
From Transformers to ChatGPT: An Analysis of Large Language Models Research
Cotfas, L., Sandu, A., Delcea, C., Diaconu, P., Frăsineanu, C., & Stănescu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3600739 · Cited by 23
“Some of the papers discuss about medical question answering [8], medical decision support [9], pediatric diagnosis in rural environments [10], traditional medicine [11], radiology assessment [12], chronic disease [13], biomedical tasks [14] and plastic surgery [15].”
Liu, E., Yang, H., Sharma, S., van Leerdam, M., Niu, P., VandeHaar, M. J., & Hostens, M. (2025) · Journal of Dairy Science · doi:10.3168/jds.2025-26775 · Cited by 18
Artificial intelligence for nanomedicine
Song, X., Gao, X., Wang, H., Yu, F., Qin, M., Li, Y., Liu, Y., Feng, W., Zhou, C., Chukavin, N. N., et al. (2025) · Science China Chemistry · doi:10.1007/s11426-025-2942-5 · Cited by 21
Self-citationDual-process theory and decision-making in large language models
Brady, O., Nulty, P., Zhang, L., Ward, T. E., & McGovern, D. P. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-025-00506-1 · Cited by 33
“This access facilitates medical question-and-answer tools 165 , automated tutoring 166 and decision-support systems in financial and legal settings, for example.”
Large language models and the future of rheumatology: assessing impact and emerging opportunities
Mannstadt, I., & Mehta, B. Y. (2023) · Current Opinion in Rheumatology · doi:10.1097/bor.0000000000000981 · Cited by 21
Sarvari, P., Al-Fagih, Z., Ghuwel, A., & Al-Fagih, O. S. (2024) · Health care science · doi:10.1002/hcs2.79 · Cited by 19
Li, H., Li, Y., Wang, H., Zeng, J., Xu, H., Cai, P., Chen, L., Yan, J., Feng, X., Xiong, L., et al. (2024) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0313 · Cited by 33
Steurer, B., Vanhaelen, Q., & Zhavoronkov, A. A. (2024) · The Journals of Gerontology Series A · doi:10.1093/gerona/glae006 · Cited by 18
Applying Large Language Models for Causal Structure Learning in Non Small Cell Lung Cancer
Naik, N., Khandelwal, A., Joshi, M., Atre, M., Wright, H., Kannan, K. K., Hill, S., Mamidipudi, G., Srinivasa, G., Bifulco, C. B., et al. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00110 · Cited by 30
An historical overview of artificial intelligence for diagnosis of major depressive disorder
Liu, H., Wu, H., Yang, Z., Ren, Z., Dong, Y., Zhang, G., & Li, M. D. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1417253 · Cited by 17
LLM Questionnaire Completion for Automatic Psychiatric Assessment
Rosenman, G., Hendler, T., & Wolf, L. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.23 · Cited by 40
Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive BenchmarkPreprint
Liu, F., Li, Z., Zhou, H., Yin, Q., Yang, J., Tang, X., Luo, C., Zeng, M. Y., Jiang, H., Gao, Y., et al. (2024) · medRxiv · doi:10.1101/2024.04.24.24306315 · Cited by 17
Tomita, K., Nishida, T., Kitaguchi, Y., Kitazawa, K., & Miyake, M. (2025) · Clinical ophthalmology · doi:10.2147/opth.s494480 · Cited by 23
GLOSS: Group of LLMs for Open-ended Sensemaking of Passive Sensing Data for Health and Wellbeing
Choube, A., Le, H., Li, J., Ji, K., Swain, V. D., & Mishra, V. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3749474 · Cited by 30
Hanai, A., Ishikawa, T., Kawauchi, S., Iida, Y., & Kawakami, E. (2024) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100924 · Cited by 19
What does artificial intelligence mean in rheumatology?
Chandwar, K., & Misra, D. P. (2024) · Archives of Rheumatology · doi:10.46497/archrheumatol.2024.10664 · Cited by 21
Ho, T.-C., Kharrat, F., Abid, A., Karray, F., & Koubâa, A. (2024) · IEEE International Symposium on Medical Measurements and Applications (MeMeA) · doi:10.1109/memea60663.2024.10596778 · Cited by 22
CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches
Wu, S., Khasahmadi, A. H., Katz, M., Jayaraman, P. K., Pu, Y., Willis, K. D. D., & Liu, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72897-6_21 · Cited by 65
Assessing Large Language Models for Oncology Data Inference From Radiology Reports
Chen, L.-C., Zack, T. I., Demirci, A., Sushil, M., Miao, B. Y., Kasap, C., Butte, A. J., Collisson, E. A., & Hong, J. C. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci.24.00126 · Cited by 22
R2GenCSR: Mining Contextual and Residual Information for LLMs-Based Radiology Report Generation
Wang, X., Li, Y., Wang, F., Wang, S., Li, C., & Jiang, B. (2026) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2026.3669539 · Cited by 38
“VMamba [33] introduces a Cross-Scan Module (CSM) to ensure global receptive fields with linear complexity.”
Huang, X., Wang, X., Zhang, H., Zhu, Y., Xi, J., An, J., Wang, H., Liang, H., & Pan, C. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i4.32396 · Cited by 14
Medical multimodal large language models: A systematic review
Hu, Y., Xu, C., Lin, B., Yang, W., & Tang, Y. Y. (2025) · Intelligent Oncology · doi:10.1016/j.intonc.2025.09.005 · Cited by 14
Towards Medical Complex Reasoning with LLMs through Medical Verifiable Problems
Chen, J., Cai, Z., Ji, K., Wang, X., Liu, W., Wang, R., & Wang, B. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.751 · Cited by 14
Self-citationThe Development Landscape of Large Language Models for Biomedical Applications
Cao, Z., Keloth, V. K., Xie, Q., Qian, L., Liu, Y., Wang, Y., Shi, R., Zhou, W., Yang, G. F., Zhang, J., et al. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-102224-074736 · Cited by 21
Self-citationMultimodal Medical Image Analysis: Integrating LLM and RAG Deep Learning Strategies
Yan, H., & Shao, D. (2025) · Journal of Advances in Information Technology · doi:10.12720/jait.16.4.568-581 · Cited by 20
Understanding pre-training data effects in retinal foundation models using two large fundus cohorts
Zhou, Y., Wang, Z., Wu, Y., Ong, A. Y., Wagner, S. K., Ruffell, E., Chia, M. A., Guan, Z., Ju, L., Engelmann, J., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-70077-z · Cited by 22
“Details of the downstream task datasets are listed in Supplementary Data 1 and 2.”
InfluentialBRIDGE: benchmarking large language models for understanding real-world clinical practice texts
Wu, J., Gu, B., Zhou, R., Xie, K., Snyder, D., Jiang, Y., Carducci, V., Wyss, R., Desai, R. J., Alsentzer, E., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01719-2 · Cited by 51
Self-citationArtificial intelligence across oncology specialties: current applications and emerging tools
Kang, J., Lafata, K. J., Kim, E., Yao, C. M. K. L., Lin, F. P. Y., Rattay, T., Nori, H., Katsoulakis, E., & Lee, C. I. (2024) · BMJ Oncology · doi:10.1136/bmjonc-2023-000134 · Cited by 18
Kharitonova, K., Pérez‐Fernández, D., Gutiérrez-Hernando, J., Gutiérrez-Fandiño, A., Callejas, Z., & Griol, D. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2321959 · Cited by 22
Deep Learning in Hematology: From Molecules to Patients
Wang, J. (2024) · Clinical Hematology International · doi:10.46989/001c.124131 · Cited by 21
Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark
Liu, F., Li, Z., Zhou, H., Yin, Q., Yang, J., Tang, X., Luo, C., Zeng, M. Y., Jiang, H., Gao, Y., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.759 · Cited by 47
Saxena, S., Chang, O. L. B., Suppan, M., Meço, B. C., Vacas, S., Radtke, F. M., Matot, I., Devos, A., Maze, M., Gisselbaek, M., et al. (2025) · British Journal of Anaesthesia · doi:10.1016/j.bja.2025.01.001 · Cited by 14
Yang, E.-W., Waldrup, B., & Velazquez-Villarreal, E. I. (2025) · AI · doi:10.3390/ai6070137 · Cited by 14
Med-R 2 : Crafting Trustworthy LLM Physicians via Retrieval and Reasoning of Evidence-Based Medicine
Lu, K., Zheng, L., Pan, D., Zhang, S., Dong, G., Leng, H., Cui, B., wu, Z., & Zhang, W. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792076 · Cited by 26
Lessons learned from RadiologyNET foundation models for transfer learning in medical radiology
Napravnik, M., Hržić, F., Urschler, M., Miletić, D., & Štajduhar, I. (2025) · Scientific Reports · doi:10.1038/s41598-025-05009-w · Cited by 15
Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
Chen, J., Yoon, J., Ebrahimi, S., Arik, S., Pfister, T., & Jha, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.345 · Cited by 13
Self-citationLiao, L., Mo, W., Wen, Y., Liu, S., Zou, Y., Mo, L., Chengke, W., & Fan, C. (2025) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-6315 · Cited by 14
Generative Artificial Intelligence: Applications and Future Prospects in Dentistry
Ma, Z., Du, C., Lao, Q., & Xie, X. (2025) · International Dental Journal · doi:10.1016/j.identj.2025.108276 · Cited by 17
Human evaluators vs. LLM-as-a-Judge: toward scalable evaluation of GenAI in global health
Williams, G., Rutunda, S., Nzabakira, F., & Mateen, B. A. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02992-w · Cited by 25
Foundation models and the privatization of public knowledge
Ferrari, F., van Dijck, J., & van den Bosch, A. P. J. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00695-5 · Cited by 19
ShennongMGS: An LLM-based Chinese Medication Guidance System
Dou, Y., Huang, Y., Zhao, X., Zou, H., Shang, J., Lu, Y., Yang, X., Xiao, J., & Peng, S. (2024) · ACM Transactions on Management Information Systems · doi:10.1145/3658451 · Cited by 19
GPT-4 as a biomedical simulator
Schaefer, M., Reichl, S., ter Horst, R., Nicolas, A. M., Krausgruber, T., Piras, F., Stepper, P., Bock, C., & Samwald, M. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108796 · Cited by 17
Yang, H., Chang, J., He, W., Wee, C. F., Soong, J. T. Y., & Feng, M. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3430226 · Cited by 16
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models
Diao, S., Pan, R. N., Dong, H., Shum, K., Zhang, J., Xiong, W., & Zhang, T. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-demo.12 · Cited by 29
Hammane, Z., Ben-Bouazza, F.-E., & Fennan, A. (2024) · 2024 International Conference on Intelligent Systems and Computer Vision (ISCV) · doi:10.1109/iscv60512.2024.10620139 · Cited by 11
Hasan, M. J., Rahman, F., & Mohammed, N. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105764 · Cited by 16
Hypnos: A domain-specific large language model for anesthesiology
Wang, Z., Jiang, J., Zhan, Y., Zhou, B., Li, Y., Zhang, C., Yu, B., Ding, L., Jin, H., Peng, J., et al. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129389 · Cited by 16
A Survey on the Impact of Pre-Trained Language Models in Sentiment Classification Task
Gautam, H., Gaur, A., & Yadav, D. K. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00805-z · Cited by 26
Huang, T., Lu, L., Chen, J., Liu, L., He, J., Zhao, Y., Tang, W., & Xu, J. (2026) · Communications Medicine · doi:10.1038/s43856-026-01631-5 · Cited by 24
“Forexample,Med-PaLM2,developedbyGoogle, focusesonprovidinghigh-qualityanswerstomedicalquestionsand generatingcomprehensivesummariesformedicalprofessionals 9 .”
Self-citationHow does DeepSeek-R1 perform on USMLE?Preprint
de Paiva, L. F., Luijten, G., Puladi, B., & Egger, J. (2025) · medRxiv · doi:10.1101/2025.02.06.25321749 · Cited by 34
Zuo, K., Zhong, Z., Huang, P., Tang, S., Chen, Y., & Jiang, Y. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.03.657521 · Cited by 20
Wang, Y., Wan, Y., Lei, X., Chen, Q., & Hu, H. (2025) · Array · doi:10.1016/j.array.2025.100504 · Cited by 18
Idan, D., Shitrit, I. B., Volevich, M., Binyamin, Y., Nassar, R., Nassar, M., Abelson, N., Zlotnik, A. A., & Einav, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-21689-w · Cited by 17
Jafer, M. A., Ibraheem, W., Dawood, T., Abbas, A., Hakami, K., Khurayzi, T. A., Hakami, A. J., Alqahtani, S., Aldosari, M., Ageely, K., et al. (2025) · Diagnostics · doi:10.3390/diagnostics15243247 · Cited by 13
“These include large language models adapted for healthcare, such as GPT-4-based medical systems and Med-PaLM [66].”
Large language models for clinical artificial intelligence in healthcare a systematic review
Ghnemat, R., & Saleh, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00784-x · Cited by 18
The development, application, and future of LLM similar to ChatGPT
Yan, H., Liu, Y., Jin, L., & Bai, X. (2023) · Journal of Image and Graphics · doi:10.11834/jig.230536 · Cited by 14
Self-citationA review of big data technology and its application in cancer care
Xiao, T., Kong, S., Zhang, Z., Hua, D., & Liu, F. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108577 · Cited by 12
Kadoya, N., Arai, K., Tanaka, S., Kimura, Y., Tozuka, R., Yasui, K., Hayashi, N., Katsuta, Y., Takahashi, H., Inoue, K., et al. (2024) · Radiological Physics and Technology · doi:10.1007/s12194-024-00838-2 · Cited by 15
Barbosa, R. R., Santos, R., & Nováis, P. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-73058-0_17 · Cited by 20
Thieme, A., Rajamohan, A. M., Cooper, B., Groombridge, H., Simister, R. J., Wong, B., Woznitza, N., Pinnock, M. A., Wetscherek, M. T., Morrison, C. P., et al. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3716500 · Cited by 20
Metadata
- Authors
- Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., Neal, D., Rashid, Q. M., Schaekermann, M., Wang, A., Dash, D., Chen, J. H., Shah, N. H., Lachgar, S., Mansfield, P. A., Prakash, S., Green, B., Dominowska, E., Agüera y Arcas, B., Tomašev, N., Liu, Y., Wong, R., Semturs, C., Mahdavi, S. S., Barral, J. K., Webster, D. R., Corrado, G. S., Matias, Y., Azizi, S., Karthikesalingam, A., & Natarajan, V.
- Affiliations
- Google (United States) (US)
- Stanford Medicine (US)
- Stanford Health Care (US)
- Corresponding author
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- ORCID
- Singhal, Karan
- Tu, Tao
- Gottweis, Juraj
- Sayres, Rory
- Amin, Mohamed
- Clark, Kevin
- Pfohl, Stephen R.
- Cole-Lewis, Heather
- Rashid, Qazi Mamunur
- Schaekermann, Mike
- Wang, Amy
- Dash, Dev
- Chen, Jonathan H.
- Shah, Nigam H.
- Lachgar, Sami
- Mansfield, Philip Andrew
- Prakash, Sushant
- Green, Bradley
- Dominowska, Ewa
- Tomašev, Nenad
- Liu, Yun
- Wong, Renee
- Semturs, Christopher
- Mahdavi, S. Sara
- Barral, Joelle K.
- Webster, Dale R.
- Corrado, Greg S.
- Matias, Yossi
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- Venue
- Nature Medicine
- Published
- 2025-01-08
- Volume
- 31
- Issue
- 3
- Pages
- 943-950
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11922739
- OpenAlex
- W4406152279
- Semantic Scholar
- b34c0b0169925a6c7f14e3de9764ed9505f30de3
- Cited by
- 1,734
- References
- 57
- Citations by year
- 2026: 771 · 2025: 591
- FWCI
- 277.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Nature Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Gordon and Betty Moore Foundation (#12409)
- American Heart Association
- National Institutes of Health (UM1TR004921)
- Stanford Bio-X
- National Institute on Drug Abuse (UG1DA015815)
- National Institute of Allergy and Infectious Diseases (R01AI178121)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- medical question answering, large language models, ensemble refinement, clinical reasoning, grounding, USMLE, MedQA, physician preferences
- MeSH terms
- Large Language Models, Humans, Physicians, Pilot Projects, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.