Toward expert-level medical question answering with large language models
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a 'passing' score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These models were chosen as they represent current leading LLMs3,36,40,41 as of February 2024.”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To account for the inherent stochasticity of LLMs, we report results from 10 independent runs per benchmark as mean accuracy with 95% confidence intervals (CIs), moving beyond the single-run point estimates common in prior works.2,3,4”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“medical domain-specific LLMs such as Medical Pathways Language Model (Med-PaLM) [2] and Biomedical Generative Pretrained Transformer (BioGPT) [3] have shown improved performance across various clinical natural language processing (NLP) tasks, including question answering, summarization, entity recognition, relation extraction, document classification, and literature mining.”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs have demonstrated strong performance across a wide range of biomedical knowledge and reasoning tasks, accelerating their adoption in health care workflows [1-6].”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models (LLMs) like OpenAI’s GPT-5 and knowledge-specific systems, such as OpenEvidence, have reported the ability to generate patient-specific, evidence-based recommendations.10”
From Large Language Model–Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations (2026) · cites this paper as [10]
Experimental — accuracy not yet measured
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“The advent of transformers 4 and LLMs 5,6 has renewed interest in the possibilities of AI for medical question-answering tasks—a long-standing ‘grand challenge’ 7–9 .”
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
New creatinine- and cystatin C-based equations to estimate GFR without race
Inker, L. A., Eneanya, N. D., Coresh, J., Tighiouart, H., Wang, D., Sang, Y., Crews, D. C., Doria, A., Estrella, M. M., Froissart, M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2102953 · Cited by 4,695
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“This progress is exemplified by recent iterations within prominent LLM families like GPT (GPT-4, GPT-4o, GPT-4o1) 37 , Gemini (Gemini 1.0, Gemini 1.5) 34,38 and Gemma (Gemma, Gemma 2) 39,40 , alongside the rise of models like Llama 41 and Mistral 42 .”
InfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“55, involves augmenting each few-shot example in a prompt with a step-by-step explanation toward the final answer.”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citation
Research works printed but not linked: 4
- 5. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT Vol. 1 (eds Burstein, J. et al.) 4171-4186 (Association for Computational Linguistics, 2019).
- 10. Bobrow, D. G.) 167-180 (MIT Press, 1994).
- 38. Achiam, J. et al. GPT-4 technical report. Preprint at https://arxiv. org/abs/2303.08774 (2023).
- 63. Randolph, Justus J. 2005 "Free-Marginal Multirater Kappa (multirater K [free]): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa." Presented at the Joensuu Learning and Instruction Symposium, vol. 2005 https://eric.ed.gov/?id=ED490661
Cited by
Items 1,001–1,032 of 1,032
Leveraging Online Data to Enhance Medical Knowledge in a Small Persian Language Model
Ghassabi, M., Rostami, P., Kashani, H. B., Poursina, A., Kazemi, Z., & Tavakoli, M. (2025) · National and International Iranian Conference on Biomedical Engineering (ICBME) · doi:10.1109/icbme68496.2025.11392505 · Cited by 4
“Despite the vast amount of research and development dedicated to English medical language models, such as Med-Palm [2] [3] and others, there remains a significant disparity in resources available for non-English languages, particularly Persian.”
Hao, P., Zhang, J., Zhang, S., Wu, F., & Tian, S. (2026) · Scientific Reports · doi:10.1038/s41598-026-41439-w · Cited by 2
“…by multitask instruction tuning, resulting in many notable models that analyze user inputs related to medical queries and generate professional explanations or provide relevant medical knowledge as responses, such as PubMedGPT 25 , Med-PaLM 26 , Med-PaLM2 27 , MedGemma 28 and II-Medical 29 .”
Generative AI and the Ethics of Innovation: Towards Inclusive and Sustainable Futures
AbuEid, A. I., & Jazzar, M. M. (2025) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca66035.2025.11430745 · Cited by 1
A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLP
Ono, S., Sukeda, I., Fujii, T., Buma, K., & Sasaki, S. (2025) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2025.ijcnlp-long.72 · Cited by 4
Yin, C., Sun, X., Dai, A., Ye, X., Lu, Y., Wang, W., Chen, Y., Jiang, H., Yu, J., Chong, S., et al. (2026) · APL Bioengineering · doi:10.1063/5.0299327 · Cited by 2
Çelik, Y. S., Özer, N., Öksüzoğlu, M. E., Macit, Ş. S., Okumuş, H. G., Kaşak, M., Efe, A., & Öztürk, Y. (2026) · Psychiatric Quarterly · doi:10.1007/s11126-026-10275-6 · Cited by 2
Pinhanez, C. S., Cavalin, P., Sanctos, C. S., & Grave, M. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i39.40550 · Cited by 1
MedXplain-VQA: Multi-Component Explainable Medical Visual Question Answering
Nguyen, H.-D., Dang, M.-A., Le, M.-T., & Le, M.-T. (2025) · International Conference on Health Science and Technology (ICHST) · doi:10.1109/ichst66555.2025.11428774 · Cited by 2
“Explaining how models arrive at answers is crucial in healthcare, where traditional VQA systems produce direct answers without rationale, hindering trust.”
Yang, Z., Wang, L., Zhou, Z., Jiang, S., Li, Y., & Chen, C. (2026) · Array · doi:10.1016/j.array.2026.100767 · Cited by 1
Could ChatGPT and co. replace forensic experts? A comparative study on medical liability expertise
Bérar, A., Allain, J., & Bouvet, R. (2026) · International Journal of Legal Medicine · doi:10.1007/s00414-026-03777-2 · Cited by 1
When Biology Meets Medicine: A Perspective on Foundation Models
Niu, K., Wang, Y., Zou, L., & Huang, J. (2026) · Advanced Intelligent Discovery · doi:10.1002/aidi.202500245 · Cited by 1
Zhang, H., Wang, W., Shao, S., Zhang, L., Wang, Q., Li, J., & Hu, J. (2026) · Chinese Journal of Mechanical Engineering · doi:10.1016/j.cjme.2026.100284 · Cited by 2
Med.ai ASK: an agentic system for biomedical question answering
Nguyen, N. T. H., Lituiev, D., Liu, Z., Kashyap, A., Jenkinson, G., Kuhl, K., Corrado, C. T., Patel, N. M., Snigdha, K., Saeedi, S., et al. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag038 · Cited by 2
“In MedQA, the system performed competitively with Med-PaLM2, 23 although its performance did not match that of Med-Gemini.”
Virtual medicine: medical AI in human health and diseases
Zhang, C., Wang, F.-X., Tang, X., Li, J.-L., Ding, N., Hong, Y., Song, P., Bai, L., & Su, J. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100012 · Cited by 4
Modern Advancements in AI for Healthcare Diagnostics: Transforming Medical Care
Khan, N. A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/407372 · Cited by 1
Ren, K., Weng, Q., Chen, Q., Li, H., Xie, D., Zeng, C., Wei, J., Lei, G., & Wang, Y. (2026) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-026-06844-x · Cited by 1
“More advanced medical-specific LLMs, such as Google’s Med-PaLM 2, have often been preferred over physicians’ answers for scientific accuracy, reasoning, and precision in pair-wise evaluations [18].”
Wang, Y., & Ge, Y. (2026) · Information Systems Research · doi:10.1287/isre.2024.1518 · Cited by 3
Zitianellis, J. (2026) · IntechOpen eBooks · doi:10.5772/intechopen.1014718 · Cited by 1
Zang, Z., Gu, W., Yang, D., Ma, S., Zhang, Z., Fan, G., Ling, W.-K., Shen, Y., & YANG, F. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464831 · Cited by 4
Liao, Y., Franco, Z., Lizarraga Mazaba, J. E., & Chen, K. (2026) · Proceedings of the 17th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics · doi:10.1145/3807503.3819453 · Cited by 1
Suzgen, E. E., Şahin, M. E., Bakir, M., Arisoy, Ç., Ulutaş, E. G., Günay, R. B., Yildiz, G., & Ulutaş, H. (2026) · 2026 2nd International Symposium on AI-Driven Engineering Systems (ISADES) · doi:10.1109/isades69945.2026.11608117
“Singhal et al. demonstrated that LLMs encode substantial clinical knowledge [11], a development culminating in Med-PaLM 2, which achieved near-expert performance on USMLE-style questions [12].”
Explainability of decoder-only clinical large language models: A scoping review
Mishra, N., Abu-Hanna, A., & Calixto, I. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103501
Lewis, M., Thio, S., Roberts, A., Siju, C., Mukit, W., Kuruvilla, R., Jiang, Z. J., Möller-Grell, N., Borakati, A., Dobson, R. J., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.02967
“Google has been at the forefront of training LLMs specifically for the medical domain with their Med-PaLM (Singhal et al., 2022, 2025), Med-Gemini (Saab et al., 2024) and MedGemma (Sellergren et al., 2025) models.”
Stephenson, C. R., Varayil, J. E., Salonen, B. R., Wight, E. C., Sang, P. D., Ghosh, A. K., Shung, D. L., Croghan, I. T., Schroeder, D. R., & Hurt, R. T. (2026) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2026.100404
Liang, Z., Sheffield, C., Butera, G., & Antani, S. (2026) · JMIR AI · doi:10.2196/93761
HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench
Flathers, M., Nguyen, P. A., Noorily, J., Herpertz, J., Chen, M., Multani, J., Powell, S., Granof, M., Kalinch, M., & Torous, J. (2026) · arXiv · doi:10.48550/arxiv.2608.25071
Trustworthy Medical Question Answering: An Evaluation-Centric Survey
Wang, Y., Wang, B., Mercer, R. E., Rudzicz, F., Roy, S. S., Ren, P., Chen, Z., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2506.03659
A Curated Dataset for Question Answering on Laboratory Test Reference Ranges and Interpretation
Bhasuran, B., Jin, Q., Deville, A., Wu, Y., Hanna, K., Lu, Z., & He, Z. (2026) · Scientific Data · doi:10.1038/s41597-026-07554-z
“5 29 , LLaMA 4 30 , Gemini 2.0 Pro 31 , and PaLM 32 , along with their medically adapted versions (e.g., MedPaLM 33 , MedPaLM 2 34 , MedGemma 35 ), has driven rapid advancements in medical QA performance.”
InfluentialMed-KGMA: A novel AI-driven medical support system leveraging knowledge graphs and medical advisors
Varshney, S., Jain, B., Singh, P., Rani, D., & Mehra, S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110929
Phasook, P., Pitijaroonpong, R., Kinchagawat, J., Chinkamol, A., Saengja, T., Udomlapsakul, K., Sawatphol, J., & Ittichaiwong, P. (2025) · Proceedings of the 24th Workshop on Biomedical Language Processing (Shared Tasks) · doi:10.18653/v1/2025.bionlp-share.34
Retrieval-Augmented Agentic Rubric Generation for Reliable Medical Response Evaluation
Chen, Y., Maiga, A., Rahmani, H. A., & Yilmaz, E. (2026) · arXiv · doi:10.48550/arxiv.2601.15161
Metadata
- Authors
- Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., Neal, D., Rashid, Q. M., Schaekermann, M., Wang, A., Dash, D., Chen, J. H., Shah, N. H., Lachgar, S., Mansfield, P. A., Prakash, S., Green, B., Dominowska, E., Agüera y Arcas, B., Tomašev, N., Liu, Y., Wong, R., Semturs, C., Mahdavi, S. S., Barral, J. K., Webster, D. R., Corrado, G. S., Matias, Y., Azizi, S., Karthikesalingam, A., & Natarajan, V.
- Affiliations
- Google (United States) (US)
- Stanford Medicine (US)
- Stanford Health Care (US)
- Corresponding author
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- ORCID
- Singhal, Karan
- Tu, Tao
- Gottweis, Juraj
- Sayres, Rory
- Amin, Mohamed
- Clark, Kevin
- Pfohl, Stephen R.
- Cole-Lewis, Heather
- Rashid, Qazi Mamunur
- Schaekermann, Mike
- Wang, Amy
- Dash, Dev
- Chen, Jonathan H.
- Shah, Nigam H.
- Lachgar, Sami
- Mansfield, Philip Andrew
- Prakash, Sushant
- Green, Bradley
- Dominowska, Ewa
- Tomašev, Nenad
- Liu, Yun
- Wong, Renee
- Semturs, Christopher
- Mahdavi, S. Sara
- Barral, Joelle K.
- Webster, Dale R.
- Corrado, Greg S.
- Matias, Yossi
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- Venue
- Nature Medicine
- Published
- 2025-01-08
- Volume
- 31
- Issue
- 3
- Pages
- 943-950
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11922739
- OpenAlex
- W4406152279
- Semantic Scholar
- b34c0b0169925a6c7f14e3de9764ed9505f30de3
- Cited by
- 1,734
- References
- 57
- Citations by year
- 2026: 771 · 2025: 591
- FWCI
- 277.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Nature Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Gordon and Betty Moore Foundation (#12409)
- American Heart Association
- National Institutes of Health (UM1TR004921)
- Stanford Bio-X
- National Institute on Drug Abuse (UG1DA015815)
- National Institute of Allergy and Infectious Diseases (R01AI178121)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- medical question answering, large language models, ensemble refinement, clinical reasoning, grounding, USMLE, MedQA, physician preferences
- MeSH terms
- Large Language Models, Humans, Physicians, Pilot Projects, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.