Toward expert-level medical question answering with large language models
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a 'passing' score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These models were chosen as they represent current leading LLMs3,36,40,41 as of February 2024.”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To account for the inherent stochasticity of LLMs, we report results from 10 independent runs per benchmark as mean accuracy with 95% confidence intervals (CIs), moving beyond the single-run point estimates common in prior works.2,3,4”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“medical domain-specific LLMs such as Medical Pathways Language Model (Med-PaLM) [2] and Biomedical Generative Pretrained Transformer (BioGPT) [3] have shown improved performance across various clinical natural language processing (NLP) tasks, including question answering, summarization, entity recognition, relation extraction, document classification, and literature mining.”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs have demonstrated strong performance across a wide range of biomedical knowledge and reasoning tasks, accelerating their adoption in health care workflows [1-6].”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models (LLMs) like OpenAI’s GPT-5 and knowledge-specific systems, such as OpenEvidence, have reported the ability to generate patient-specific, evidence-based recommendations.10”
From Large Language Model–Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations (2026) · cites this paper as [10]
Experimental — accuracy not yet measured
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“The advent of transformers 4 and LLMs 5,6 has renewed interest in the possibilities of AI for medical question-answering tasks—a long-standing ‘grand challenge’ 7–9 .”
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
New creatinine- and cystatin C-based equations to estimate GFR without race
Inker, L. A., Eneanya, N. D., Coresh, J., Tighiouart, H., Wang, D., Sang, Y., Crews, D. C., Doria, A., Estrella, M. M., Froissart, M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2102953 · Cited by 4,695
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“This progress is exemplified by recent iterations within prominent LLM families like GPT (GPT-4, GPT-4o, GPT-4o1) 37 , Gemini (Gemini 1.0, Gemini 1.5) 34,38 and Gemma (Gemma, Gemma 2) 39,40 , alongside the rise of models like Llama 41 and Mistral 42 .”
InfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“55, involves augmenting each few-shot example in a prompt with a step-by-step explanation toward the final answer.”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citation
Research works printed but not linked: 4
- 5. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT Vol. 1 (eds Burstein, J. et al.) 4171-4186 (Association for Computational Linguistics, 2019).
- 10. Bobrow, D. G.) 167-180 (MIT Press, 1994).
- 38. Achiam, J. et al. GPT-4 technical report. Preprint at https://arxiv. org/abs/2303.08774 (2023).
- 63. Randolph, Justus J. 2005 "Free-Marginal Multirater Kappa (multirater K [free]): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa." Presented at the Joensuu Learning and Instruction Symposium, vol. 2005 https://eric.ed.gov/?id=ED490661
Cited by
Items 801–900 of 1,031
The Enterprising and Elusive Prospects of Human-AI Collaboration
Gadiraju, U., & Balayn, A. (2025) · Springer eBooks · doi:10.1007/978-3-032-01940-0_7 · Cited by 5
Ruoyan, Z., Liu, L., Zhao, Q., Haodi, L., Chunkai, M., Feifei, T., & Xu, L. (2025) · BMC Oral Health · doi:10.1186/s12903-025-07236-z · Cited by 2
“For instance, medical-specific models like Med-PaLM have preliminarily validated the feasibility of generating clinical recommendations [6].”
Sesen, M. B., Yeung, J. A., & Asgari, E. (2026) · BMJ Digital Health & AI · doi:10.1101/2025.11.25.25341010 · Cited by 2
“Various approaches have been used to address knowledge limitations, including prompt engineering 15 , medical domain fine-tuning 16 and the use of tools such as web search 17 with varying degrees of success.”
Liu, C., Wang, W. Y. C., & Khan, G. F. (2025) · Industrial Management & Data Systems · doi:10.1108/imds-05-2025-0609 · Cited by 3
Evaluation of the accuracy of large language models in answering bone cancer-related questions
Pan, Q., Huang, L., Liu, N., Lin, F., & Ye, S. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1719047 · Cited by 3
Ophtimus-V2-Tx: a compact domain-specific LLM for ophthalmic diagnosis and treatment planning
Kwon, M., Jang, K. J., Baek, S. J., Han, Y. S., Choi, H., Lee, I., & Kim, J. H. (2025) · Scientific Reports · doi:10.1038/s41598-025-27410-1 · Cited by 2
Lei, Y.-H., Chen, C., & Shen, C. (2025) · Medical Education Online · doi:10.1080/10872981.2025.2602788 · Cited by 2
“Previous studies [9–11, 20 ] In contrast, our token-splitting approach is designed to be accessible and practical for educators and clinicians.”
The Infectious Diseases Orchestrator: Embracing AI Literacy in the Agentic Era
Hanna, J. J., & Medford, R. J. (2025) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofaf794 · Cited by 5
“Unlike earlier NLP, advanced LLMs can reason across complex contexts [49, 50], and already power ambient documentation [51], automated chart summarization [52], and AI-assisted patient communication [53–55].”
Towards Safe and Trustworthy Healthcare AI: Risk Assessment of Medical Dialogue Using LLMs
Tanioka, H. (2025) · Human-Centric Intelligent Systems · doi:10.1007/s44230-025-00131-4 · Cited by 2
“Systems such as Med-PaLM 2 achieved 86.5% accuracy on USMLE-style medical QA [4], showing that biomedical knowledge can be effectively encoded.”
InfluentialGuessoum, A., Berkani, L., Ameur, M. S. H., & Aouichat, A. (2026) · Springer eBooks · doi:10.1007/978-3-031-99068-7_15 · Cited by 2
Roxas, D. Q., & Emekci, H. (2026) · Journal of Polytechnic · doi:10.2339/politeknik.1719005 · Cited by 2
Gupta, G., Nimbalkar, S., Kumar, R., Singh, P., & Mondal, H. (2026) · The Indian Journal of Pediatrics · doi:10.1007/s12098-025-05884-1 · Cited by 3
Carou‐Senra, P., Rodríguez‐Pombo, L., Alvarez‐Lorenzo, C., & Goyanes, Á. (2026) · International Journal of Pharmaceutics · doi:10.1016/j.ijpharm.2026.126587 · Cited by 3
Ekingen, E., & Üçdal, M. T. (2026) · Diagnostics · doi:10.3390/diagnostics16020323 · Cited by 3
“Accuracy rates exceeding 85% on diagnostic imaging tasks have been reported [5,6].”
Sallam, M., Stanley, A., Snygg, J., Al-Shakerchi, H., Atragchi, O. A., Abusamra, R., & Sallam, M. (2026) · Recent Progress in Sciences · doi:10.70462/rps.2026.3.001 · Cited by 4
“Emerging evidence suggests that large language models (LLMs) can rival, and at times surpass, students and physicians in standardized knowledge assessments [58-60].”
Evaluating reasoning large language models with human-like thinking in ophthalmic question answering
Wang, Z., Xu, C., Wang, L., Qiang, W., Li, Y., Li, D., Xu, F., Zhang, Y., Jiang, J., & Li, Z. (2026) · BMJ Open Ophthalmology · doi:10.1136/bmjophth-2025-002615 · Cited by 2
“10 However, AI with ‘black box’ inherency is hardly to be trusted, especially when interacting with patients, unless it can adequately simulate the thinking processes of human experts.”
Modi, M., Krull, J. E., Gauntner, T. D., Li, M., Johnson, D., Wang, X., Cheng, H., Zhao, J., Ma, Q., Zhang, P., et al. (2026) · npj Digital Medicine · doi:10.64898/2026.01.26.26344845 · Cited by 4
Duwe, G., Moench, K., Kauth, V., Angeloni, M., Eckhoff, J., Görtz, M., Hoefert, S., Kocar, T. D., Kollitsch, L., Mehralivand, S., et al. (2026) · Die Urologie · doi:10.1007/s00120-026-02778-8 · Cited by 3
Lin, S., shao, S., Liu, X., & Su, H. (2026) · Information · doi:10.3390/info17020138 · Cited by 2
Genetic Diagnosis and Discovery Enabled by Large Language Models
Tu, T., Saab, K. K., Liu, W., Fang, Z., Cheng, Z., Spasić, S., Djurisic, M. R., Mohri, H., Ren, W., Palepu, A. K., et al. (2026) · Advanced Science · doi:10.1002/advs.202518656 · Cited by 2
Self-citationPoser, P. L., Klimas, R., Luerweg, J., Reuter, E., Hanefeld, C., Gold, R., Salmen, A., & Motte, J. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1658575 · Cited by 4
“For reasons of feasibility, we have, for example, refrained from analyzing specialized medical LLMs such as Med-PaLM 2 or others (Singhal et al., 2025).”
Cheema, M. N., Nazir, A., Harmanci, A. O., Harmancı, A. S., Cheema, Y., Masood, S., & Khokhar, F. A. (2026) · The Visual Computer · doi:10.1007/s00371-026-04385-2 · Cited by 2
“Sources like MIMIC-III [115], MedQA [136], and Med-PaLM’s curated datasets [137] are commonly used.”
Lyu, S., Lim, K. Y. H., Chen, C., Youcef‐Toumi, K., & Yang, X. J. (2026) · Computers in Industry · doi:10.1016/j.compind.2026.104463 · Cited by 3
Lueji: A Swahili-Language Medical Chatbot for Low-Resource Specialties in Sub-Saharan Africa
Kabeya, D. K., Kambale, W. V., Kubakisa, C. K., Kayembe, I. L., Hamed, M., & Kyamakya, K. (2026) · WSEAS TRANSACTIONS ON BIOLOGY AND BIOMEDICINE · doi:10.37394/23208.2026.23.10 · Cited by 2
Liu, Y., Zhang, G., Luan, R., & Yu, X. (2026) · Sensors · doi:10.3390/s26051553 · Cited by 3
Self-citationA data-centric approach to detecting and mitigating demographic bias in pediatric mental health text
Ive, J., Bondaronek, P., Yadav, V., Santel, D., Glauser, T. A., Strawn, J. R., Agasthya, G. A., Tschida, J., Choo, S., Chandrashekar, M., et al. (2026) · Communications Medicine · doi:10.1038/s43856-026-01480-2 · Cited by 4
Kristjánsson, T. Ó., Henriksen, A. F., Hansen, M. L., Kovacs, D. G., & Bjerrum, A. (2026) · ESMO Real World Data and Digital Oncology · doi:10.1016/j.esmorw.2026.100689 · Cited by 5
BioFuse: an embedding fusion framework for biomedical foundation models
Hossain, M. N., & Harris‐Birtill, D. (2026) · PLoS ONE · doi:10.1371/journal.pone.0320989 · Cited by 2
Serrano, J. D. A., Caño-Marín, E., & Sicilia, M. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115882 · Cited by 2
Towards a clinical large language model: an ethico-legal case study analysis
Bernasconi, L., Stamm-Pfister, C., Großmann, R., & Rätsch, G. (2026) · AI and Ethics · doi:10.1007/s43681-026-01099-y · Cited by 2
Ferdowsi, M., Moustafa, N., Keshk, M., & Turnbull, B. (2026) · Electronics · doi:10.3390/electronics15071519 · Cited by 3
MAPC: Multi-Agent Planning-Collaboration Framework for Medical Question Answering
Elshaer, Z., A.Attia, N., Selim, S., & Al‐Atabany, W. (2026) · International Conference on Software and Information Engineering (ICSIE) · doi:10.1145/3789595.3789618 · Cited by 3
Xu, Y., Zhu, Y., Wang, H., Wu, Y. G., Ouyang, Y., Li, H., Zhou, W., Liu, X., Jiang, C., & Li, Q. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790835 · Cited by 5
CARE-ECG: Causal Agent-based Reasoning for Explainable and Counterfactual ECG Interpretation
Khatibi, E., Wang, Z., Sharma, A., Chakrabarty, K., Moosavi, S. R., Firouzi, F., & Rahmani, A. M. (2026) · IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies · doi:10.1109/chase69719.2026.00043 · Cited by 5
“LLMs have rapidly advanced medical question answering and decision support, enabling systems that can respond to clinician queries with structured explanations and multi-step reasoning.”
Wang, Z., Ma, T., & Wang, H. (2026) · Die Radiologie · doi:10.1007/s00117-026-01608-4 · Cited by 2
Consumer Understanding of Skin Concerns With an AI-Powered Informational Tool
Sayres, R., Jain, A., Venkatraman, M., Singh, P. K., Liu, Y., Winter, S. R., Schaekermann, M., Loh, A., Verma, S., Matias, Y., et al. (2026) · JAMA Dermatology · doi:10.1001/jamadermatol.2026.0597 · Cited by 2
Self-citationChen, X., Wang, G., Cao, J., & Lu, Y. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.110290 · Cited by 2
Wang, Z., Fang, Y., & Li, Q. (2026) · Journal of Translational Medicine · doi:10.1186/s12967-026-08211-0 · Cited by 4
“Although large language models can provide treatment suggestions based on existing guidelines, such as Med-PaLM [36], it is not a personalized treatment protocol for a specific case.”
When Chatbots Become Agents: The Next Phase of Healthcare AI
Karunanayake, N. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02402-4 · Cited by 4
“Even early LLM chatbots were performing sophisticated pattern retrieval against training data [ 5–7].”
Exploring the architectures of large language models and impact across multiple domains: a review
Waqas, M., Rajab, R., Laghari, A. A., Kryvinska, N., Hejaili, A. A., Khan, N., Khuhro, M. A., & Zhang, F. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11609-3 · Cited by 2
Jiang, L., Chen, J., Lu, L., Peng, X., Liu, L., He, J., & Xu, J. (2026) · JMIR AI · doi:10.2196/86864 · Cited by 2
Self-citationKaplan, A., Kaplan, B. A., Aydilek, A., Ege, T., Şentürk, S., & Solmaz, İ. (2026) · European Spine Journal · doi:10.1007/s00586-026-10094-0 · Cited by 2
Evaluation of ChatGPT-4o’s and DeepSeek R1’s responses to urological problems: a comparative study
Lu, H., Zhang, Y., Wang, Z., Zhao, Y., Liu, J., Qiu, D., & Zhang, Y. (2026) · BMC Urology · doi:10.1186/s12894-026-02228-9 · Cited by 2
“With the rapid development of AI, LLMs such as Ope-nAI’s ChatGPT have shown promise in medical education [7], diagnostic reasoning, and decision support [8].”
InfluentialMohammed, S. I., Abdulameer, K. A., salih, M., & Al-Hussaniy, H. A. (2026) · Journal of Umm Al-Qura University for Medical Sciences · doi:10.1007/s44361-026-00045-1 · Cited by 2
“Differential diagnosis gen - eration tasks yielded correct-diagnosis inclusion rates of 85–92% on curated test sets [21 , 22 ].”
Generative Artificial Intelligence and Large Language Models in Clinical Oncology
Yu, Y., Zhao, Z., Wang, Z., Lin, R., Tan, Y., Chen, Y., Li, T., Baptista‐Hon, D. T., Zhang, X., Wu, C., et al. (2026) · MedComm · doi:10.1002/mco2.70833 · Cited by 4
A roadmap for medical large language models: a review of foundations, applications, and challenges
Sha, Y.-Y., Yu, L., Lin, Z.-H., Kaur, A., Lou, Y., Gornale, S. S., Zhang, T., Wong, L. S., Wang, Z., Yan, Y., et al. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100050 · Cited by 4
“Owing to their capacity for instruction following and in-context adaptation, decoder-only models provide a practical basis for conversational assistance, medical question answering, and clinical documentation support [19,66,67].”
InfluentialNatural Language Processing in Healthcare: From Unstructured Data to Clinical Intelligence
Kumar, D., Choudhary, N., Gupta, V., Tanwar, R., Bahmani, K., Antal, S., Jarora, S., Gupta, S., Shaikh, M. S., Webster, T. J., et al. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200698 · Cited by 3
Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing
Meimandi, K. J., Rust, P. U. N., Eddy, D., Fraser, R., Vasan, N., Djordjevic, D., Dadlani, A., Lamparth, M., Kim, E., & Kochenderfer, M. J. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812332 · Cited by 11
“We reviewed evaluation approaches from Med-PaLM 2 [59], CRADLE Bench [11], CounselBench [36], the EPITOME framework [56], Sanjeewa et al. [54], clinical communication scales including the Motivational Interviewing Skill Code [41], and expanded on prior work [26].”
Shi, P., Li, J., Yang, Z., Cao, Y., Ju, L., Bao, H., Xu, H., Fan, Y., Ren, T., Xiao, Y., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02942-6 · Cited by 4
Self-citationVision-language models for human motion understanding: Lessons from stroke rehabilitation
Li, V., Kamalakannan, N., Parnandi, A., Schambra, H. M., & Fernandez‐Granda, C. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001506 · Cited by 3
“…have recently achieved remarkable performance across a wide range of computer vision tasks, including image and video captioning, visual question answering, optical character recognition, document understanding, open-vocabulary object detection [1–13], and medical image understanding [14,15].”
Governing Clinical Readiness Claims Derived from Medical AI Benchmark Results
Dong, Y., Cheng, J., Ding, C., & Lu, R. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02445-7 · Cited by 2
Zhao, Y., Miao, Y., Guo, R., Luo, Y., Wang, H., & Wu, Y. X. (2026) · Journal of Medical Internet Research · doi:10.2196/90046 · Cited by 2
“However, study designs, evaluation end points, and reporting practices remain heterogeneous, and deployment in high-stakes medical settings is constrained by the tendency of these models to produce factually incorrect, internally inconsistent, or insufficiently supported statements [7,8].”
Qi, X. S., Wang, Y., Yang, X., Ren, L., Liu, W., Benedict, S., Xiao, Y., Xing, L., & Naqa, I. M. E. (2026) · Medical Physics · doi:10.1002/mp.70616 · Cited by 3
The consensus-based CINEX guideline for reporting clinical information extraction studies
Reichenpfader, D., Zaghir, J., Cécilia‐Joseph, E., Delange, B., Shaitarova, A., Sylvestre, E., Dennstädt, F., Zhang, B., Teodoro, D., Meystre, S. M., et al. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag136 · Cited by 3
“8 These models have demonstrated improved performance for various healthcare applications, including medical question answering, 9,10 and clinical text summarization.”
Zhang, L., Huai, P., XuRui, Sun, J., Jin, R., & Lv, H. (2026) · Journal of Medical Internet Research · doi:10.2196/90744 · Cited by 2
“The research indicates RAG enhances clinicians’ perceived usefulness by improving accuracy and source verifiability [86-88].”
Evaluating large language model responses to unsafe clinical instructions
Omar, M., Agbareia, R., McGreevy, J., Gorenshtein, A., Charney, A. W., Sakhuja, A., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2026) · Communications Medicine · doi:10.1038/s43856-026-01933-8 · Cited by 2
Large Language Models Provide Human-Level Medical Text Snippet Labeling
Amara, I., Yu, H., Zhang, F., Liu, Y., Li, B., Liu, C. X., Kartha, R., & Goel, A. (2024) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2024.clinicalnlp-1.15 · Cited by 4
Self-citationBeyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs.
Christophe, C., Raha, T., Maslenkova, S., Salman, M., Kanithi, P., Pimentel, M. A., & Khan, S. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.618 · Cited by 7
Mir, A. E., Luoga, L. T., Chen, B., Hanif, M. A., & Shafique, M. (2024) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi62660.2024.10913565 · Cited by 6
A Large-Language Model Framework for Relative Timeline Extraction from PubMed Case ReportsPreprint
Wang, J., & Weiss, J. C. (2025) · Europe PMC (PubMed Central) · doi:10.48550/arxiv.2504.12350 · Cited by 22
Detecting Prefix Bias in LLM-based Reward Models
Kumar, A., He, Y., Markosyan, A. H., Chern, B., & Arrieta-Ibarra, I. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732204 · Cited by 20
Leveraging Large Language Models in Healthcare: From Speech Documentation to Conversational Agents
Winkler, C. (2026) · Springer eBooks · doi:10.1007/978-3-032-16837-5_16 · Cited by 2
Human-in-the-Loop Chest X-Ray Diagnosis: Enhancing Large Multimodal Models with Eye Fixation Inputs
Kim, Y., Wu, J., Abdulle, Y., Gao, Y., & Wu, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-67751-9_6 · Cited by 3
AI for Science in the Era of Large Language Models
Bi, Z., Xu, M., Tang, J., & Wang, X. (2024) · doi:10.18653/v1/2024.emnlp-tutorials.5 · Cited by 3
Understanding generative AI to harness its potentials and mini- mize risks: A perspective
Caselli, T. (2025) · European Journal of Radiology · doi:10.1016/j.ejrad.2025.111951 · Cited by 2
LLM-Driven Chest X-Ray Report Generation With a Modular, Reduced-Size Architecture
Vargas, T. V., Pedrini, H., & Santanchè, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-79032-4_14 · Cited by 2
Generative Artificial Intelligence
Carnahan, D. (2025) · Springer eBooks · doi:10.1007/978-3-031-83526-1_5 · Cited by 2
Abrantes, J. (2025) · Cureus · doi:10.7759/cureus.84165 · Cited by 3
“…prompt engineering techniques, such as chain-of-thought (CoT) prompting, which encourages step-by-step reasoning [15], can influence LLM outputs [16,17], their effectiveness varies significantly depending on the model, task complexity, and specific domain, including medicine [18]. standardized…”
PatientDx: Merging Large Language Models for Protecting Data-Privacy in Healthcare
Moreno, J. G., Lovón-Melgarejo, J., Robin-Charlet, M., Damase‐Michel, C., & Tamine, L. (2025) · Workshop on Patient-Oriented Language Processing (CL4Health) · doi:10.18653/v1/2025.cl4health-1.1 · Cited by 2
SFMSS: Service Flow aware Medical Scenario Simulation for Conversational Data Generation
Bao, Z., Liu, Q., Huang, X., & Wei, Z. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.259 · Cited by 12
Current State of Artificial Intelligence and Trauma Video Review: Insights for Trauma Resuscitation
Villarreal, J. A., Suh, E., Forrester, J. D., Jopling, J. K., & Dumas, R. P. (2025) · Current Trauma Reports · doi:10.1007/s40719-025-00293-3 · Cited by 3
“Emerging research on AI mechanistic inter-pretability aims to reverse-engineer network circuits into human-meaningful components, offering a path to inherently self-explainable models, though practical implementations remain largely experimental[17, 79].”
Chen, M., Zhang, Z., Chen, J., Rebibo, D., Wu, H., Fung, S.-H., Cheng, A. S., Tsui, S. K., Sinha, S., Cao, Q., et al. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.07.042 · Cited by 4
“…through diverse, specialized approaches; pioneering efforts in agent-based systems (e.g., Crow from FutureHouse), multi-LLM frameworks [34], and domain-specific models [35,36] are collectively paving the way toward the reliable automation of biomedical knowledgebase curation (Supplementary Text).”
Self-citationYu, F., An, W., Wang, H., & Yin, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05397-w · Cited by 4
Incentivizing inclusive contributions in model sharing markets
Zhang, E., Chai, J., Ye, R., Wang, Y., & Chen, S. (2025) · Nature Communications · doi:10.1038/s41467-025-62959-5 · Cited by 6
Tran, H. H.-V., Thu, A., Twayana, A. R., Fuertes, A., Gonzalez, M., Basta, M., James, M., Frishman, W. H., & Aronow, W. S. (2025) · Cardiology in Review · doi:10.1097/crd.0000000000001042 · Cited by 5
Tel, A., Bolognesi, F., Michelutti, L., Biglioli, F., & Robiony, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-19669-1 · Cited by 2
Synth-SBDH: A Synthetic Dataset of Social and Behavioral Determinants of Health for Clinical Text
Mitra, A., Yang, Z., Druhl, E., Goodwin, R., & Yu, H. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1418 · Cited by 6
Building a Chinese Medical Dialogue System: Integrating Large-scale Corpora and Novel Models
Wang, X., Li, H., Zheng, D., & Peng, Q. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227217 · Cited by 9
KT-LLM: an evidence-grounded and sequence text framework for auditable kidney transplant modeling
Zheng, H., Luo, Z., He, K., Zhou, W., Kong, Z., Dong, J., Dai, Q., & Sun, Q. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02323-5 · Cited by 5
Bharti, P. K., Panchal, M., Dalal, V., Agarwal, M., & Ekbal, A. (2026) · Scientometrics · doi:10.1007/s11192-025-05435-7 · Cited by 2
Developing and Evaluating a Model-Based Metric for Legal Question Answering Systems
Bakır, D., Yildiz, B., & Aktaş, M. S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386689 · Cited by 2
Primary care clinicians should proactively take up latest AI-based technology: Yes
Holt‐Quick, C. (2024) · Journal of Primary Health Care · doi:10.1071/hc24035 · Cited by 2
Unifying Corroborative and Contributive Attributions in Large Language Models
Worledge, T., Shen, J. H., Meister, N., Winston, C., & Guestrin, C. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00039 · Cited by 28
Yu, L., Li, H., Chen, K., & Shou, L. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671963 · Cited by 3
Natural Language Processing Can Automate Extraction of Barrett’s Esophagus Endoscopy Quality MetricsPreprint
Soroush, A., Diamond, C., Zylberberg, H. M., May, B. L., Tatonetti, N. P., Abrams, J. A., & Weng, C. (2023) · medRxiv · doi:10.1101/2023.07.11.23292529 · Cited by 1
Iterative improvements from feedback for language modelsPreprint
Li, Y. (2023) · doi:10.14293/pr2199.000220.v1 · Cited by 5
Empowering Transformers for Evidence-Based MedicinePreprint
Mohammed, S., Fiaidhi, J., & Shaik, H. (2023) · medRxiv · doi:10.1101/2023.12.25.23300520 · Cited by 3
A Comparative Survey on Large Language Models for Biological DataPreprint
Mousa, R., Sarabadani, A., Taami, T., Bengari, A. A., Eslamifar, O., Shalmani, M. A., & Shahmarvandi, E. K. (2025) · Preprints.org · doi:10.20944/preprints202504.2464.v1 · Cited by 5
Yılmaz, A., Yuceyalcin, F., Varol, R., Gökyayla, E., Erdem, O., Choi, D., Demirçalı, A. A., Gençoğlan, G., Posma, J. M., & Temelkuran, B. (2025) · medRxiv · doi:10.1101/2025.05.17.25327785 · Cited by 8
“Large language models (LLMs) are increasingly recognized as transformative tools in healthcare, enabling the creation of explainable analysis, structured diagnoses, and personalized treatment plans [1–4].”
OphthaBERT: Automated Glaucoma Diagnosis from Clinical NotesPreprint
Shah, R., Moradi, M., Eslami, S., Fujita, A., Aziz, K., Bineshfar, N., Elze, T., Eslami, M., Kazeminasab, S., Liebman, D. L., et al. (2025) · medRxiv · doi:10.1101/2025.06.08.25329151 · Cited by 6
“Language models optimized for medical domains, such as BioClinicalBERT and Med-PaLM, have proven to capture complex context and relationships in text [4–6].”
Pham, M.-K., Mai, T. T., Dao, T. T. P., Tran, M., Crane, M., & Bezbradica, M. (2025) · doi:10.36227/techrxiv.175021755.55836808/v1 · Cited by 3
A Survey of Generalization and Adaptation in Medical Imaging Foundation ModelsPreprint
Ying, H., Lia, Y., & Fu, Z. (2025) · Preprints.org · doi:10.20944/preprints202507.0942.v1 · Cited by 5
Elevating Management Reasoning to Preserve Professional Identity in the AI Era
Jones, R., Rodman, A. M., & Parsons, A. S. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-026-10182-3 · Cited by 2
WITHDRAWN: MedOS: AI-XR-Cobot World Model for Clinical Perception and ActionPreprint
Wu, Y. C., Yin, M., Shi, B., Zhang, Z., Yin, D., Wang, X., Wang, Y., Fan, J., Jin, R., Wang, H., et al. (2026) · medRxiv · doi:10.64898/2026.02.18.26345936 · Cited by 2
ClinicalAgents: Multi-Agent Orchestration for Clinical Decision Making with Dual-Memory
Ge, Z., Li, H., Wang, Y., Hu, N., Zhang, C. J., & Li, Q. (2026) · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3770855.3818931 · Cited by 2
Sun, X., Hong, Q., Zhang, M., Li, Y., Chen, T., Huang, Z., Liang, G., Tang, W., Xu, S., Ni, X., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102547 · Cited by 3
Medical students' use of large language models: a national survey
Barr, A. A., Rozman, R. C., Liu, K., Pham, M., Klarenbach, Z., Chinna-Meyyappan, A., Hassan, A. Y., Zarychta, M., Ferri, O. E., Al-Khaz’Aly, A., et al. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106589 · Cited by 1
Kong, S., Yang, X., Wei, Y., Wang, Z., Tang, H., Qin, J., Lan, S., Wang, Y., Bai, J., Chen, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01252 · Cited by 3
Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?
Yuan, G. C., Zhang, X., Kim, S. E., & Rajpurkar, P. (2026) · Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026) · doi:10.18653/v1/2026.healing-1.1 · Cited by 4
Metadata
- Authors
- Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., Neal, D., Rashid, Q. M., Schaekermann, M., Wang, A., Dash, D., Chen, J. H., Shah, N. H., Lachgar, S., Mansfield, P. A., Prakash, S., Green, B., Dominowska, E., Agüera y Arcas, B., Tomašev, N., Liu, Y., Wong, R., Semturs, C., Mahdavi, S. S., Barral, J. K., Webster, D. R., Corrado, G. S., Matias, Y., Azizi, S., Karthikesalingam, A., & Natarajan, V.
- Affiliations
- Google (United States) (US)
- Stanford Medicine (US)
- Stanford Health Care (US)
- Corresponding author
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- ORCID
- Singhal, Karan
- Tu, Tao
- Gottweis, Juraj
- Sayres, Rory
- Amin, Mohamed
- Clark, Kevin
- Pfohl, Stephen R.
- Cole-Lewis, Heather
- Rashid, Qazi Mamunur
- Schaekermann, Mike
- Wang, Amy
- Dash, Dev
- Chen, Jonathan H.
- Shah, Nigam H.
- Lachgar, Sami
- Mansfield, Philip Andrew
- Prakash, Sushant
- Green, Bradley
- Dominowska, Ewa
- Tomašev, Nenad
- Liu, Yun
- Wong, Renee
- Semturs, Christopher
- Mahdavi, S. Sara
- Barral, Joelle K.
- Webster, Dale R.
- Corrado, Greg S.
- Matias, Yossi
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- Venue
- Nature Medicine
- Published
- 2025-01-08
- Volume
- 31
- Issue
- 3
- Pages
- 943-950
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11922739
- OpenAlex
- W4406152279
- Semantic Scholar
- b34c0b0169925a6c7f14e3de9764ed9505f30de3
- Cited by
- 1,734
- References
- 57
- Citations by year
- 2026: 771 · 2025: 591
- FWCI
- 277.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Nature Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Gordon and Betty Moore Foundation (#12409)
- American Heart Association
- National Institutes of Health (UM1TR004921)
- Stanford Bio-X
- National Institute on Drug Abuse (UG1DA015815)
- National Institute of Allergy and Infectious Diseases (R01AI178121)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- medical question answering, large language models, ensemble refinement, clinical reasoning, grounding, USMLE, MedQA, physician preferences
- MeSH terms
- Large Language Models, Humans, Physicians, Pilot Projects, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.