Toward expert-level medical question answering with large language models
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a 'passing' score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These models were chosen as they represent current leading LLMs3,36,40,41 as of February 2024.”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To account for the inherent stochasticity of LLMs, we report results from 10 independent runs per benchmark as mean accuracy with 95% confidence intervals (CIs), moving beyond the single-run point estimates common in prior works.2,3,4”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“medical domain-specific LLMs such as Medical Pathways Language Model (Med-PaLM) [2] and Biomedical Generative Pretrained Transformer (BioGPT) [3] have shown improved performance across various clinical natural language processing (NLP) tasks, including question answering, summarization, entity recognition, relation extraction, document classification, and literature mining.”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs have demonstrated strong performance across a wide range of biomedical knowledge and reasoning tasks, accelerating their adoption in health care workflows [1-6].”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models (LLMs) like OpenAI’s GPT-5 and knowledge-specific systems, such as OpenEvidence, have reported the ability to generate patient-specific, evidence-based recommendations.10”
From Large Language Model–Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations (2026) · cites this paper as [10]
Experimental — accuracy not yet measured
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“The advent of transformers 4 and LLMs 5,6 has renewed interest in the possibilities of AI for medical question-answering tasks—a long-standing ‘grand challenge’ 7–9 .”
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
New creatinine- and cystatin C-based equations to estimate GFR without race
Inker, L. A., Eneanya, N. D., Coresh, J., Tighiouart, H., Wang, D., Sang, Y., Crews, D. C., Doria, A., Estrella, M. M., Froissart, M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2102953 · Cited by 4,695
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“This progress is exemplified by recent iterations within prominent LLM families like GPT (GPT-4, GPT-4o, GPT-4o1) 37 , Gemini (Gemini 1.0, Gemini 1.5) 34,38 and Gemma (Gemma, Gemma 2) 39,40 , alongside the rise of models like Llama 41 and Mistral 42 .”
InfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“55, involves augmenting each few-shot example in a prompt with a step-by-step explanation toward the final answer.”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citation
Research works printed but not linked: 4
- 5. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT Vol. 1 (eds Burstein, J. et al.) 4171-4186 (Association for Computational Linguistics, 2019).
- 10. Bobrow, D. G.) 167-180 (MIT Press, 1994).
- 38. Achiam, J. et al. GPT-4 technical report. Preprint at https://arxiv. org/abs/2303.08774 (2023).
- 63. Randolph, Justus J. 2005 "Free-Marginal Multirater Kappa (multirater K [free]): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa." Presented at the Joensuu Learning and Instruction Symposium, vol. 2005 https://eric.ed.gov/?id=ED490661
Cited by
Items 501–600 of 1,031
Asiedu, M., Haykel, I., Dieng, A., Kauer, K., Ahmed, T., Ofori, F., Chan, C., Pfohl, S., Rostamzadeh, N., & Heller, K. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732160 · Cited by 10
Self-citationGao, H., Wang, K., Yuan, Y., Wang, Y., Liu, Q., Wang, Y., Sun, J. Q., Wang, W., Wang, H., Zhou, S., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-07649-4 · Cited by 12
Liu, B., Nie, Y., Song, H., Lin, Y., Li, J., Weng, X., Su, Z., Suo, Y., Lv, T., Zhao, X., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103557 · Cited by 12
Quality of patient information on interstitial cystitis from artificial intelligence chatbots
Santucci, J., Stapleton, P., Ibrahim, J., Putra, L. J., Elmer, S., & Sathianathen, N. J. (2025) · British Journal of Urology · doi:10.1111/bju.70035 · Cited by 9
“Moreover, in a pilot study using real-world medical questions, specialists preferred Med-PaLM-2 responses over physician responses 65% of the time and the chatbot ’ s responses were rated as safe as physician answers [10].”
Zhong, J., Li, P., Luo, R., Yin, J., Ding, Y., Bai, J., Hong, C., Xiang, D., Ma, X., & Lu, S. (2025) · Energies · doi:10.3390/en18225921 · Cited by 9
How AI Is Transforming Medical Education: Bibliometric Analysis
Wang, Y., Chang, C., Shi, W., Liu, H., Huang, X., & Jiao, Y. (2025) · JMIR Medical Education · doi:10.2196/75911 · Cited by 9
“Notably, large language models have permeated all levels of medical education, suggesting their transformative potential across undergraduate, graduate, and continuing training contexts [17,28].”
The empirical structure of psychopathology is represented in large language models
Kambeitz, J. P., Schiffman, J. E., Kambeitz‐Ilankovic, L., Mittal, V. A., Ettinger, U., & Vogeley, K. T. (2025) · Nature Mental Health · doi:10.1038/s44220-025-00527-y · Cited by 17
Clinical applications of large language models in knee osteoarthritis: a systematic review
Ma, Z., Liu, Y., Zhang, Z., Chen, R., Fan, H., Cao, X., & Ni, L. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1670824 · Cited by 11
“Given the absence of a standardized framework for assessing the output quality of LLMs in clinical applications (9), we synthesized components from existing instruments to construct the clinical language model assessment (CliMA-10).”
Self-citationIncorporating information retrieval into AI chatbots for patient education on thyroid eye disease
Khabaz, K., Parekh, Z., Saeed, S., Krakauer, M., Silas, M. R., Farooq, A. V., & Shah, H. A. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106213 · Cited by 8
Susceptibility of Large Language Models to User-Driven Factors in Medical Queries
Lim, K. H., Kang, U., Li, X., Kim, J. S., Jung, Y., Park, S. J., & Kim, B.-H. (2025) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-025-00218-4 · Cited by 17
A conversational multi-agent AI system for automated plant phenotyping
Chen, F., Stogiannidis, I., Wood, A., Bueno, D., Williams, D., Macfarlane, F., Grieve, B., Wells, D. M., Atkinson, J. A., Hawkesford, M. J., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-71090-y · Cited by 19
Large Language Models in Cardiovascular Prevention: A Narrative Review and Governance Framework
Santos, J. F., & Dores, H. (2026) · Diagnostics · doi:10.3390/diagnostics16030390 · Cited by 9
“Information Synthesis: Consolidating fragmented patient histories into concise summaries [18,38].”
InfluentialA Multi-Agent System for Complex Reasoning in Radiology Visual Question Answering
Yi, Z., Liu, J., Albert, M. V., & Xiao, T. (2025) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl67857.2025.00025 · Cited by 10
“In the healthcare domain, MLLMs such as Med-PaLM 2 [13] and LLaVA-Med [14] have made notable progress in pharmaceutical research [15] and clinical support [16].”
Naderi, N., Safavi‐Naini, S. A. A., Savage, T., Khalafi, M. A., Lewis, P. R., Atf, Z., Nadkarni, G. N., & Soroush, A. (2026) · npj gut and liver. · doi:10.1038/s44355-026-00053-3 · Cited by 14
“5-Sonnet) exhibited substantial overcon fi dence (Brier scores 0.15 – 0.2, AUROC ~ 0.6).”
InfluentialHamamoto, R., Koyama, T., Takahashi, S., Yasuda, T., Kobayashi, K., Akagi, Y., Kouno, N., Sudo, K., Hirata, M., Sunami, K., et al. (2026) · Journal of Hematology & Oncology · doi:10.1186/s13045-026-01781-y · Cited by 8
“…structured human evaluation, physician raters preferred its long-form responses over physician-written answers across multiple clinical utility and safety-related axes; nevertheless, specialist-level consultation remains a challenging setting requiring careful deployment constraints (Table 2) [67].”
Lin, T., Lau, R. Y. K., & Hu, S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11565-y · Cited by 11
“Given the mature deployment of LLMs in finance (e.g., BloombergGPT Wu et al. 2023 ) and healthcare (e.g., Med-PaLM Singhal et al. 2025 ), the need for interpretability and corresponding governance mechanisms is likely to become increasingly prominent, whereas such emphasis and systematic…”
OpenAI ChatGPT: A Potential Medical Application
Chaddad, A., Jiang, Y., & He, C. (2023) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom56612.2023.10472387 · Cited by 13
Inductive reasoning with large language models: A simulated randomized controlled trial for epilepsy
Goldenholz, D. M., Goldenholz, S. R., Habib, S., & Westover, M. B. (2024) · Epilepsy Research · doi:10.1016/j.eplepsyres.2025.107532 · Cited by 12
Ali, S., Shahab, O., Shabeeb, R. Q. A., Ladak, F., Yang, J. O., Nadkarni, G. N., Echavarria, J., Babar, S., Shaukat, A., Soroush, A., et al. (2023) · medRxiv · doi:10.1101/2023.09.21.23295918 · Cited by 11
Prompt Engineering in Large Language Models for Patient Education: A Systematic ReviewPreprint
Mudrik, A., Nadkarni, G. N., Efros, O., Soffer, S., & Klang, E. (2025) · medRxiv · doi:10.1101/2025.03.28.25324834 · Cited by 15
Incorporating large language models as clinical decision support in oncology: the Woollie model
Heydari, K., Enichen, E. J., Li, B., & Kvedar, J. C. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01941-3 · Cited by 9
Agentic AI in Healthcare: Opportunities, Challenges, and Future Directions
Gridach, M., Nanavati, J., Zine El Abidine, K., Yacoubian, C., & Mack, C. (2026) · ACM Computing Surveys · doi:10.1145/3809164 · Cited by 8
Gui, X., Lv, H., Wang, X., Lv, L., Xiao, Y., & Wang, L. (2025) · BioData Mining · doi:10.1186/s13040-025-00458-5 · Cited by 10
Gautam, S., Chandra, M., De, A., Chakravorti, T., Malik, G., & De Choudhury, M. (2025) · arXiv · doi:10.48550/arxiv.2508.06849 · Cited by 7
Taveekitworachai, P., Chanmas, G., Paliyawan, P., Thawonmas, R., Nukoolkit, C., Dajpratham, P., & Thawonmas, R. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e32930 · Cited by 8
Maximizing discrimination masking for faithful question answering with machine reading
Dong, L., Tang, J., Wang, P., Li, S., & Wang, T. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103915 · Cited by 9
MEDSQ: Towards personalized medical education via multi-form interaction guidance
Ouyang, Y., Gao, W., Wang, H., Chen, L., Wang, J., & Zeng, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126138 · Cited by 9
Toward collaborative artificial intelligence development for animal well-being
Sun, J. (2025) · Journal of the American Veterinary Medical Association · doi:10.2460/javma.24.10.0650 · Cited by 9
Infusing Multi-Hop Medical Knowledge Into Smaller Language Models for Biomedical Question Answering
Chen, J., Wei, Z., Shen, W., & Shang, R. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3547444 · Cited by 10
Self-citationAfzaal, U., Su, Z., Sajjad, U., Stack, T., Lü, H., Niu, S., Akbar, A., Gürcan, M. N., & Niazi, M. K. K. (2025) · Patterns · doi:10.1016/j.patter.2025.101284 · Cited by 15
Zhong, W., Sun, M.-Y., Yao, S., Liu, Y., Peng, D., Liu, Y., Yang, K., Gao, H., Yan, H., Hao, W., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/73233 · Cited by 8
“With technological advancements, models like ChatGPT have improved their ability to generate high-quality responses comparable to those of experienced medical professionals [17,27-31].”
Self-citationYu, Z., Fang, L., Ding, Y., Yan, S., Xu, L., Cai, Y., & Liu, X. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104844 · Cited by 8
Zayed, A. M., Frans, G., & Delvaux, N. (2025) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2025-0647 · Cited by 8
Relational & culture-sensitive AI innovation
Herzog, C. (2025) · Journal of Responsible Technology · doi:10.1016/j.jrt.2025.100135 · Cited by 7
Sng, G. G. R., Xiang, Y., Lim, D. Y. Z., Tung, J. Y. M., Tan, J. H., & Chng, C. L. (2025) · JMIR Formative Research · doi:10.2196/70863 · Cited by 11
Prompt design for medical question answering with Large Language Models
Kuligin, L., Lammert, J., Ostapenko, A., Bressem, K. K., Boeker, M., & Tschochohei, M. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100758 · Cited by 12
Large language models in healthcare: a systematic evaluation on medical Q/A datasets
Tofeeq, K., Naseer, A., & Wali, A. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00397-9 · Cited by 9
Education Research: Can Large Language Models Match MS Specialist Training?
Inojosa, H., Ramezanzadeh, A., Gasparovic-Curtini, I., Wiest, I. C., Kather, J. N., Gilbert, S., & Ziemssen, T. (2025) · Neurology Education · doi:10.1212/ne9.0000000000200260 · Cited by 8
“4,5 6-13 However, performance varies depending on task complexity and alignment between training data and speci fi c medical context and less on speci fi c prompting methods or inclusion of media elements.”
How is language intelligence evolving? A multi-dimensional survey of large language models
Chen, X., Zhou, L., Chen, J., Wang, G., & Li, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130637 · Cited by 11
Self-citationYalcın, B. Y., Mutlu, Ü., Ok, A. M., Çaklılı, Ö. T., Hacışahinoğulları, H., Yalın, G. Y., Selçukbiricik, Ö. S., Üzüm, A. K., & Karşıdağ, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-27235-y · Cited by 10
“Previous literature suggests that LLMs adapted specifically for the medical field, especially those trained on medically focused data, may offer performance advantages over general-purpose counterparts 36 .”
Large language models in healthcare and biomedical informatics: A comprehensive review
Hornback, A. M., Sathu, H., Kim, K., Wang, Y., Zhu, Y., Isgut, M., Avula, P., Khimani, A., & Wang, M. D. (2026) · Innovation and Emerging Technologies · doi:10.1142/s2737599426300011 · Cited by 9
Improving TCM question answering through tree-organized self-reflective retrieval with LLMs
Liu, C., Chang, Y., Li, J., Qu, Y., Li, Y., Cao, L., & Lin, S. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1752778 · Cited by 7
“Med-PaLM and Med-PaLM2 achieve accuracy of 67.2 and 86.5%, respectively, surpassing the “pass” score on the USMLE (17, 18).”
InfluentialGutierrez, K. L. T., & Viacrusis, P. M. (2023) · Journal of Medicine University of Santo Tomas · doi:10.35460/2546-1621.2023-0081 · Cited by 9
Advancing Faithfulness of Large Language Models in Goal-Oriented Dialogue Question Answering
Sticha, A., Braunschweiler, N., Doddipatla, R., & Knill, K. (2024) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3640794.3665573 · Cited by 11
Toslali, M., Snible, E., Chen, J., Cha, A., Singh, S. P., Kalantar, M., & Parthasarathy, S. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663829 · Cited by 12
Self-citationLarge Language Models for Healthcare: Applications, Models, Datasets, and Challenges
Rezgui, K. (2024) · International Conference on Control, Decision and Information Technologies (CoDIT) · doi:10.1109/codit62066.2024.10708253 · Cited by 9
Srivastava, A. K., & Dames, P. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-3519-1_15 · Cited by 7
Li, Y., Yan, Y., Tong, Z., Wang, Y., Yang, Y., Bai, M., Pu, D., Xie, J., Liu, C., Li, B., et al. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113084 · Cited by 12
Withdrawn: Large Language Models in Healthcare Decision Support: A ReviewPreprint
Vavekanand, R., Karttunen, P., Xu, Y., Milani, S., & Li, H. (2024) · Preprints.org · doi:10.20944/preprints202407.1842.v1 · Cited by 14
Review of Artificial Intelligence for Clinical Use in Alzheimer's Disease and Related Dementias
Breithaupt, A. G., Tang, A., Paolillo, E. W., Bibars, M., Johnson, E. C. B., Saloner, R., Possin, K. L., Windon, C. C., Hill-Jarrett, T. G., Giorgio, J., et al. (2026) · Seminars in Neurology · doi:10.1055/a-2744-9871 · Cited by 6
Wang, A., Ruparel, R., Iurchenko, A., Jhun, P., Séguin, J. A., Strachan, P., Wong, R., Karthikesalingam, A., Matias, Y., Hassidim, A., et al. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3721208 · Cited by 6
Self-citationSugan, M., Nadkarni, S., Sanaeipoor, S., Patel, S., Natarajamoorthy, S. Y., Rashidi, E., & Miller, A. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3721270 · Cited by 6
Do Large Language Models Align with Core Mental Health Counseling Competencies?
Nguyen, V. C., Taher, M., Hong, D., Possobom, V. K., Gopalakrishnan, V. T., Raj, E., Li, Z., Soled, H. J., Birnbaum, M. L., Kumar, S., et al. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.418 · Cited by 32
Leveraging Large Language Models for Accurate AO Fracture Classification from CT Text Reports
Mergen, M., Spitzl, D., Ketzer, C., Strenzke, M., Marka, A. W., Makowski, M. R., Bressem, K. K., Adams, L. C., & Gassert, F. T. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01603-6 · Cited by 9
“General-purpose LLMs like ChatGPT-4o (OpenAI) and Gemini 2.0 Flash (Google) demonstrate remarkable text-processing capabilities but lack medical specialization [16].”
InfluentialIntegrating statistical design and inference: A roadmap for robust and trustworthy medical AI
Wei, Q., Cui, M., Liu, Z., Liu, Z., Feng, G., Li, Y., Christiani, D. C., Li, L., Wang, J., Hao, Y., et al. (2025) · The Innovation Medicine · doi:10.59717/j.xinn-med.2025.100145 · Cited by 8
Application of Large Language Models in Automated Interpretation of Urodynamic Parameters
Wang, Z., Xu, Z., Shi, Y., Xi, J., & Zhang, Y. (2025) · Neurourology and Urodynamics · doi:10.1002/nau.70160 · Cited by 6
Lu, S., Fang, C., Zeng, H., Hu, R., Wei, C., Miao, R., Gan, X., Guo, B., Yao, M., & He, S. (2025) · Science China Life Sciences · doi:10.1007/s11427-025-3074-8 · Cited by 10
Yin, L., Xue, X., Pan, S., Liu, X., Zhou, Y., Nahar, L., Chen, Y., Zhao, J., Zeng, S., & Xu, T. (2025) · Acta Materia Medica · doi:10.15212/amm-2025-0045 · Cited by 11
“Recent advancements in multimodal diagnostic systems have achieved paradigm-shifting innovations [167, 168].”
Moradi, M., Shah, R., Fujita, A., Bineshfar, N., Vu, D. M., Aziz, K., Liebman, D. L., Hashemabad, S. K., Wang, M., Elze, T., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02267-w · Cited by 10
Application of chatbots in chronic disease management: A scoping review
Hou, J., Lin, S., Teng, P., Han, Y., Luo, Y., & He, G. (2026) · Digital Health · doi:10.1177/20552076251411287 · Cited by 8
“15,16 Nevertheless, current research exhibits substantial heterogeneity in chatbot design elements, including interaction modalities, functional capabilities, and evaluation indicators, which compromises the comparability of study outcomes and hinders the development of standardized practice…”
Rogoz, A.-C., Ionescu, R. T., Anghel, A.-V., Antone-Iordache, I.-L., Coniac, S., & Ionescu, A.-I. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02465-0 · Cited by 10
ChatGPT-5 versus other mainstream large language models in core diabetic retinopathy patient queries
Cang, X., MengXia, N., Song, C., Zhao, J., Guo, Y., Zou, Y., Zhang, Z., & Jiang, L. (2026) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2026.1754221 · Cited by 6
Jiang, L., Zhu, Y., Song, C., Hu, X., Fan, X., Yang, W., & Zhao, Y. (2026) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2026.1785731 · Cited by 6
“Prior research on medical LLMs has shown that LLMs can approach clinician-level performance on multidisciplinary medical benchmarks in terms of knowledge expression and reasoning (Singhal et al., 2023; Khatri and Gupta, 2025; Singhal et al., 2025).”
Schwarzkopf, S.-C., Distler, M., Welsch, T., Krause-Jüttler, G., Weitz, J., & Kolbinger, F. R. (2023) · JMIR Serious Games · doi:10.2196/44708 · Cited by 7
Patients Facing Large Language Models in Oncology: A Narrative Review
Raynaud, C., Wu, D., Levy, J., Marengo, M., & Bibault, J. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci-24-00149 · Cited by 8
Verifiable, Debuggable, and Repairable Commonsense Logical Reasoning via LLM-based Theory Resolution
Toroghi, A., Guo, W., Pesaranghader, A., & Sanner, S. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.379 · Cited by 28
Iranfar, A., Soleymannejad, M., Moshiri, B., & Taghirad, H. D. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108686 · Cited by 9
Role of artificial intelligence in ICU therapeutic decision-making for severe infections
Giacobbe, D. R., Vena, A., & Bassetti, M. (2025) · Current Opinion in Critical Care · doi:10.1097/mcc.0000000000001304 · Cited by 8
Avnat, E., Levy, M., Herstain, D., Yanko, E., Joya, D. B., Katz, M. T., Eshel, D., Laros, S., Dagan, Y., Barami, S., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/64452 · Cited by 13
“First, although medically tuned LLMs have shown promising results [34], they are not publicly available and hence were not included in this study.”
WiseMind: a knowledge-guided multi-agent framework for accurate and empathetic psychiatric diagnosis
Wu, Y., Wan, G., Li, J., Zhao, S., Ma, L., Ye, T., Zhang, M., Pop, I., Zhang, Y., & Chen, J. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02559-9 · Cited by 22
“The action intent from the RA is passed to the EA to generate patient-facing communication (step 5).”
Self-citationMAP: evaluation and multi-agent enhancement of large language models for inpatient pathways
Chen, Z., Peng, Z., Liang, X., Wang, C., Liang, P., Zeng, L., Ju, M., & Yuan, Y. (2026) · npj Health Systems · doi:10.1038/s44401-026-00085-0 · Cited by 18
Jin, Z., Liu, S., Li, H., Zhao, X., & Qu, H. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3557568 · Cited by 14
Large Language Models in Spine Surgery: A Promising Technology
Shah, R., & Schwab, J. H. (2025) · HSS Journal®: The Musculoskeletal Journal of Hospital for Special Surgery · doi:10.1177/15563316251340696 · Cited by 5
Faria, F. T. J., Baniata, L. H., Choi, A., & Kang, S. (2025) · Mathematics · doi:10.3390/math13142322 · Cited by 9
Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening
Dogra, S., Arabshahi, S., Wei, J., Hu, E., Saidenberg, L., Sharma, S., Gu, Z., Siriruchatanon, M., & Kang, S. K. (2025) · Academic Radiology · doi:10.1016/j.acra.2025.08.014 · Cited by 5
ChatGPT’s role in the rapidly evolving hematologic cancer landscape
Nong, T., Britton, S., Bhanderi, V. K., & Taylor, J. (2025) · Future Science OA · doi:10.1080/20565623.2025.2546259 · Cited by 6
Ghosh, A., Jain, R., Jhangra, A., Saha, S., & Jatowt, A. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70045 · Cited by 8
Ruan, J., Tang, Y., Wei, Z., Xing, W., & Hu, Y. (2025) · Journal of Medical Internet Research · doi:10.2196/80289 · Cited by 6
“At the technical level, multiple approaches have been used to improve the accuracy of LLMs, such as prompt engineering, retrieval-augmented generation, and fine-tuning [27].”
Li, X., Li, G., Zhao, Y., Liang, Y., Dong, Y., & Zhang, J. (2025) · Clinical Interventions in Aging · doi:10.2147/cia.s551572 · Cited by 6
Groza, T., Marcello, A., Carlisle, T., Lim, W. K., Haendel, M. A., Karnani, N., Robinson, P. N., Graeßner, H., Chong, J. X., Baynam, G. S., et al. (2025) · Human Genetics and Genomics Advances · doi:10.1016/j.xhgg.2025.100558 · Cited by 10
Benchmarking GPT-5 in radiation oncology: measurable gains, but persistent need for expert oversight
Dinç, U., Sarkar, J., Schubert, P., Semrau, S., Weißmann, T., Karius, A., Brand, J., Axer, B., Gomaa, A., Stephan, P., et al. (2025) · Frontiers in Oncology · doi:10.3389/fonc.2025.1695468 · Cited by 8
“Retrieval-augmented systems such as Med-PaLM 2 and AMIE combine generative reasoning with evidence retrieval and self-critique, achieving strong performance but relying on curated sources at inference time (33, 34).”
Liu, D., Long, Y., Zuoqiu, S., Liu, D., Li, K., Lin, Y., Liu, H., Yin, R., & Tang, T. (2026) · Journal of Medical Internet Research · doi:10.2196/85206 · Cited by 5
“Furthermore, its responses were preferred over those of generalist physicians in 65% of expert evaluations [13].”
The rise of large language models in wearable health sensing
Farrahi, V. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.12.003 · Cited by 5
Advances, Evaluation, and Explainability of Large Language Models in Healthcare: A Systematic Review
Amin, S. U., Guizani, M., & Hossain, M. S. (2026) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3786334 · Cited by 10
Chen, Y., Gao, Y., Yu, S., Zhao, C., & Lu, Y. (2026) · Journal of Industrial Integration and Management · doi:10.1142/s2424862225300042 · Cited by 7
Advancing Healthcare with Large Language Models: Techniques and Application
Hu, Z., Peng, Z., Bi, Z., Shen, Q., Liu, Z., Lou, J., & Luo, X. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125540 · Cited by 12
AI agents in Alzheimer’s disease management: challenges and future directions
Grammenos, G., Vrahatis, A. G., Lazaros, K., Exarchos, T. P., Vlamos, P., & Krokidis, M. G. (2026) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2025.1735892 · Cited by 8
May, P., Nokodian, S., Nuernbergk, C., Knauer, M., Hefter, M., von Rose, A. B., Bassermann, F., & Jung, J. (2026) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00194 · Cited by 6
LLM-driven collaborative framework for knowledge-enhanced cancer pain assessment and management
Liu, H., Hu, Y., Li, D., Shi, B., Niu, Y., Zhang, X., Zheng, G., Li, C., Wang, L., & Bao, Y. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02362-6 · Cited by 8
Jiang, Z., Chen, H., Wu, Y., Qin, Y., Pei, C., Zeng, D., Sheng, B., & Wong, T. Y. (2026) · Intelligent Medicine · doi:10.1016/j.imed.2026.01.001 · Cited by 9
Liu, J., & Hu, X. (2026) · Frontiers in Blockchain · doi:10.3389/fbloc.2026.1766092 · Cited by 7
Li, Z., Liu, H., Tan, W., Tang, D., Duan, S., Zhou, B., Tang, L., Hu, X., Huang, L., Zhao, P., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02645-y · Cited by 9
Yeh, Y., Yang, H.-Y., Chiu, C., Chao, A., Chuang, Y.-C., & Chan, W.-S. (2026) · Intensive Care Medicine Experimental · doi:10.1186/s40635-026-00900-w · Cited by 6
Human–AI collaboration in clinical reasoning: a UK replication and interaction analysis
Healy, J., Kossoff, J., Lee, M., & Hasford, C. (2026) · Diagnosis · doi:10.1515/dx-2025-0176 · Cited by 7
“Initial strong performance on medical exams [1] has prompted investigation into their abilities in a broad range of medical tasks, with triage [2] physician queries [3] and direct patient interactions [4] all promising areas.”
Generative AI for healthcare: fundamentals, challenges, and perspectives
Chen, G., Liu, C., Anne, O. G., Tan, M., Xie, Z., Yin, J., Yip, J. W. L., Zhang, W., ZHU, J., & Ooi, B. C. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4927-2 · Cited by 10
“Med-PaLM [2] and its successor Med-PaLM 2 [3] extend Google’s PaLM [34, 35] architecture with large-scale instruction tuning, retrieval-augmented reasoning, and ensemble refinement, achieving state-of-the-art performance on MedQA [36] and the United States Medical Licensing Examination (USMLE).”
Michalowski, M., Wilk, S., Bauer, J. M., Carrier, M., Delluc, A., Gal, G. L., Wang, T., Siegal, D. M., & Michalowski, W. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-66535-6_33 · Cited by 8
Cao, Y., Wang, X., Wang, Q., Yuan, Z., Shi, Y., & Peng, D. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112373 · Cited by 5
Adopting Generative AI with Precaution in Dentistry: A Review and Reflection
Xu, M., Ye, C., Zeng, Z., Chang, C., Qi, S., Wu, Y., Yang, H., Chen, Y., Huang, H., Liu, L., et al. (2024) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh62654.2024.00047 · Cited by 7
Large Language Models in the Medical Field: Principles and Applications
Chen, X. (2024) · International Journal of Computer Science and Information Technology · doi:10.62051/ijcsit.v2n3.24 · Cited by 6
CurieLM: Enhancing Large Language Models for Nuclear Domain Applications
Bouhoun, Z., Allali, A., Cocci, R., Assaad, M. A., Plancon, A., Godest, F., Kondratenko, K., Rodriguez, J., Vitillo, F., Malhomme, O., et al. (2024) · EPJ Web of Conferences · doi:10.1051/epjconf/202430217006 · Cited by 6
Metadata
- Authors
- Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., Neal, D., Rashid, Q. M., Schaekermann, M., Wang, A., Dash, D., Chen, J. H., Shah, N. H., Lachgar, S., Mansfield, P. A., Prakash, S., Green, B., Dominowska, E., Agüera y Arcas, B., Tomašev, N., Liu, Y., Wong, R., Semturs, C., Mahdavi, S. S., Barral, J. K., Webster, D. R., Corrado, G. S., Matias, Y., Azizi, S., Karthikesalingam, A., & Natarajan, V.
- Affiliations
- Google (United States) (US)
- Stanford Medicine (US)
- Stanford Health Care (US)
- Corresponding author
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- ORCID
- Singhal, Karan
- Tu, Tao
- Gottweis, Juraj
- Sayres, Rory
- Amin, Mohamed
- Clark, Kevin
- Pfohl, Stephen R.
- Cole-Lewis, Heather
- Rashid, Qazi Mamunur
- Schaekermann, Mike
- Wang, Amy
- Dash, Dev
- Chen, Jonathan H.
- Shah, Nigam H.
- Lachgar, Sami
- Mansfield, Philip Andrew
- Prakash, Sushant
- Green, Bradley
- Dominowska, Ewa
- Tomašev, Nenad
- Liu, Yun
- Wong, Renee
- Semturs, Christopher
- Mahdavi, S. Sara
- Barral, Joelle K.
- Webster, Dale R.
- Corrado, Greg S.
- Matias, Yossi
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- Venue
- Nature Medicine
- Published
- 2025-01-08
- Volume
- 31
- Issue
- 3
- Pages
- 943-950
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11922739
- OpenAlex
- W4406152279
- Semantic Scholar
- b34c0b0169925a6c7f14e3de9764ed9505f30de3
- Cited by
- 1,734
- References
- 57
- Citations by year
- 2026: 771 · 2025: 591
- FWCI
- 277.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Nature Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Gordon and Betty Moore Foundation (#12409)
- American Heart Association
- National Institutes of Health (UM1TR004921)
- Stanford Bio-X
- National Institute on Drug Abuse (UG1DA015815)
- National Institute of Allergy and Infectious Diseases (R01AI178121)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- medical question answering, large language models, ensemble refinement, clinical reasoning, grounding, USMLE, MedQA, physician preferences
- MeSH terms
- Large Language Models, Humans, Physicians, Pilot Projects, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.