Toward expert-level medical question answering with large language models
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a 'passing' score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These models were chosen as they represent current leading LLMs3,36,40,41 as of February 2024.”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To account for the inherent stochasticity of LLMs, we report results from 10 independent runs per benchmark as mean accuracy with 95% confidence intervals (CIs), moving beyond the single-run point estimates common in prior works.2,3,4”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“medical domain-specific LLMs such as Medical Pathways Language Model (Med-PaLM) [2] and Biomedical Generative Pretrained Transformer (BioGPT) [3] have shown improved performance across various clinical natural language processing (NLP) tasks, including question answering, summarization, entity recognition, relation extraction, document classification, and literature mining.”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs have demonstrated strong performance across a wide range of biomedical knowledge and reasoning tasks, accelerating their adoption in health care workflows [1-6].”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models (LLMs) like OpenAI’s GPT-5 and knowledge-specific systems, such as OpenEvidence, have reported the ability to generate patient-specific, evidence-based recommendations.10”
From Large Language Model–Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations (2026) · cites this paper as [10]
Experimental — accuracy not yet measured
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“The advent of transformers 4 and LLMs 5,6 has renewed interest in the possibilities of AI for medical question-answering tasks—a long-standing ‘grand challenge’ 7–9 .”
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
New creatinine- and cystatin C-based equations to estimate GFR without race
Inker, L. A., Eneanya, N. D., Coresh, J., Tighiouart, H., Wang, D., Sang, Y., Crews, D. C., Doria, A., Estrella, M. M., Froissart, M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2102953 · Cited by 4,695
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“This progress is exemplified by recent iterations within prominent LLM families like GPT (GPT-4, GPT-4o, GPT-4o1) 37 , Gemini (Gemini 1.0, Gemini 1.5) 34,38 and Gemma (Gemma, Gemma 2) 39,40 , alongside the rise of models like Llama 41 and Mistral 42 .”
InfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“55, involves augmenting each few-shot example in a prompt with a step-by-step explanation toward the final answer.”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citation
Research works printed but not linked: 4
- 5. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT Vol. 1 (eds Burstein, J. et al.) 4171-4186 (Association for Computational Linguistics, 2019).
- 10. Bobrow, D. G.) 167-180 (MIT Press, 1994).
- 38. Achiam, J. et al. GPT-4 technical report. Preprint at https://arxiv. org/abs/2303.08774 (2023).
- 63. Randolph, Justus J. 2005 "Free-Marginal Multirater Kappa (multirater K [free]): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa." Presented at the Joensuu Learning and Instruction Symposium, vol. 2005 https://eric.ed.gov/?id=ED490661
Cited by
Items 101–200 of 1,031
TelecomGPT: A Framework to Build Telecom-Specific Large Language Models
Zou, H., Zhao, Q., Tian, Y., Bariah, L., Bader, F., Lestable, T., & Debbah, M. A. (2025) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2025.3593184 · Cited by 144
Performance of ChatGPT on Nephrology Test Questions
Miao, J., Thongprayoon, C., Garcia Valencia, O. A., Krisanapan, P., Sheikh, M. S., Davis, P. W., Mekraksakit, P., Suarez, M. G., Craici, I. M., & Cheungpasitporn, W. (2024) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.0000000000000330 · Cited by 68
A review of LLMs and their applications in the architecture, engineering and construction industry
Kampelopoulos, D., Tsanousa, A., Vrochidis, S., & Kompatsiaris, I. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11241-7 · Cited by 69
Potential applications and implications of large language models in primary care
Andrew, A. (2024) · Family Medicine and Community Health · doi:10.1136/fmch-2023-002602 · Cited by 85
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
Evaluation and mitigation of cognitive biases in medical language models
Schmidgall, S., Harris, C., Essien, I., Olshvang, D., Rahman, T., Kim, J. W., Ziaei, R., Eshraghian, J., Abadir, P., & Chellappa, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01283-6 · Cited by 68
Hadid, A., Chakraborty, T., & Busby, D. (2024) · Expert Systems · doi:10.1111/exsy.13654 · Cited by 91
CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation
Wang, J., Bhalerao, A. H., Yin, T., See, S. C. W., & He, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3354712 · Cited by 103
Englhardt, Z., Ma, C., Morris, M. E., Chang, C.-C., Xu, X., Qin, L., McDuff, D., Liu, X., Patel, S., & Iyer, V. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3659604 · Cited by 108
Augmenting Black-box LLMs with Medical Textbooks for Biomedical Question Answering
Wang, Y., Ma, X., & Chen, W. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.95 · Cited by 42
Large Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis
Su, H., Sun, Y., Li, R., Zhang, A., Yang, Y., Xiao, F., Duan, Z., Chen, J., Hu, Q., Yang, T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72062 · Cited by 79
“Here, we listed the QA accuracy of LLM performance in Figure 4 and Multimedia Appendix 3 [20,131-138].”
Self-citationLarge language models in psychiatry: Opportunities and challenges
Volkmer, S., Meyer-Lindenberg, A., & Schwarz, E. (2024) · Psychiatry Research · doi:10.1016/j.psychres.2024.116026 · Cited by 59
Increasing acceptance of AI ‐generated digital twins through clinical trial applications
Vidovszky, A. A., Fisher, C. K., Loukianov, A. D., Smith, A. M., Tramel, E. W., Walsh, J. R., & Ross, J. L. (2024) · Clinical and Translational Science · doi:10.1111/cts.13897 · Cited by 77
Zhang, Y., Chen, X., Jin, B., Wang, S., Ji, S., Wang, W., & Han, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.498 · Cited by 44
Yurdakurban, E., Topsakal, K. G., & Duran, G. S. (2023) · Journal of Stomatology Oral and Maxillofacial Surgery · doi:10.1016/j.jormas.2023.101757 · Cited by 74
MedExpQA: Multilingual benchmarking of Large Language Models for Medical Question Answering
Alonso, I., Oronoz, M., & Agerri, R. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102938 · Cited by 62
Utility of artificial intelligence‐based large language models in ophthalmic care
Biswas, S., Davies, L. N., Sheppard, A. L., Logan, N. S., & Wolffsohn, J. S. (2024) · Ophthalmic and Physiological Optics · doi:10.1111/opo.13284 · Cited by 73
Nie, J., Shao, H., Fan, Y., Shao, Q., You, H., Preindl, M., & Jiang, X. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3712299 · Cited by 88
Qibo: A Large Language Model for traditional Chinese medicine
Jia, Y., Ji, X., Wang, X., Zhang, H., Meng, Z., Zhang, J., Chen, Z., Zhuang, P., Xu, D., Guo, W., et al. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127672 · Cited by 75
An eyecare foundation model for clinical assistance: a randomized controlled trial
Wu, Y.-L., Qian, B., Li, T., Qin, Y., Guan, Z., Chen, T., Jia, Y., Zhang, P., Zeng, D., Moroi, S. E., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03900-7 · Cited by 91
“Among 334 cases, ophthalmologists made 25 diagnoses that differed from the EyeFM copilot’s results (Supplementary Table 27).”
A survey on deep learning in medical ultrasound imaging
Song, K., Feng, J., & Chen, D. (2024) · Frontiers in Physics · doi:10.3389/fphy.2024.1398393 · Cited by 59
Large language models can help boost food production, but be mindful of their risks
Clercq, D. D., Nehring, E., Mayne, H., & Mahdi, A. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1326153 · Cited by 76
On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study
Van, M.-H., Verma, P., & Wu, X. (2024) · IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies · doi:10.1109/chase60773.2024.00029 · Cited by 101
Transparency of medical artificial intelligence systems
Kim, C., Gadgil, S. U., & Lee, S. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00363-w · Cited by 83
“LLMs can also be fine-tuned on medical data, such as clinical notes 171 , articles indexed in PubMed 172 and question–answer pairs from professional medical licensing exams 173,174 .”
AI Agents in Clinical Medicine: A Systematic ReviewPreprint
Gorenshtein, A., Omar, M., Glicksberg, B. S., Nadkarni, G. N., & Klang, E. (2025) · medRxiv · doi:10.1101/2025.08.22.25334232 · Cited by 97
A multi-center study on the adaptability of a shared foundation model for electronic health records
Guo, L., Fries, J., Steinberg, E., Fleming, S. L., Morse, K., Aftandilian, C. C., Posada, J., Shah, N. H., & Sung, L. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01166-w · Cited by 120
Self-citationMedBLIP: Bootstrapping Language-Image Pretraining from 3D Medical Images and Texts
Chen, Q., & Hong, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0908-6_6 · Cited by 99
PhysioLLM: Supporting Personalized Health Insights with Wearables and Large Language Models
Fang, C. M., Danry, V., Whitmore, N. W., Bao, A., Hutchison, A., Pierce, C., & Maes, P. (2024) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi62660.2024.10913781 · Cited by 100
LLM-Based Framework for Administrative Task Automation in Healthcare
Gebreab, S. A., Salah, K., Jayaraman, R., Habib ur Rehman, M., & Ellaham, S. (2024) · 2024 12th International Symposium on Digital Forensics and Security (ISDFS) · doi:10.1109/isdfs60797.2024.10527275 · Cited by 53
Tian, Y., Gan, R., Song, Y., Zhang, J., & Zhang, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.386 · Cited by 41
Foundation models in ophthalmology
Chia, M. A., Antaki, F., Zhou, Y., Turner, A. W., Lee, A., & Keane, P. A. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2024-325459 · Cited by 71
Quantifying the reasoning abilities of LLMs on clinical cases
Qiu, P., Wu, C., Liu, S., Fan, Y., Zhao, W., Chen, Z., Gu, H., Peng, C., Zhang, Y., Wang, Y., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64769-1 · Cited by 53
A generative AI teaching assistant for personalized learning in medical education
Thesen, T., & Park, S. H. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02022-1 · Cited by 76
Transforming wearable data into personal health insights using large language model agents
Merrill, M. A., Paruchuri, A., Rezaei, N., Kovacs, G., Perez, J., Liu, Y., Schenck, E., Hammerquist, N., Sunshine, J., Tailor, S. A., et al. (2026) · Nature Communications · doi:10.1038/s41467-025-67922-y · Cited by 106
“Fifth, we emphasize that the aim of this research is not to build an LLM agent capable of addressing highly specialized or complex medical questions requiring expert knowledge 16,18 beyond the scope of wearable data.”
Self-citationTowards evaluating and building versatile large language models for medicine
Wu, C., Qiu, P., Liu, J., Gu, H., Li, N., Zhang, Y., Wang, Y., & Xie, W. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01390-4 · Cited by 52
Zhang, Y., Pan, Y., Zhong, T., Dong, P., Xie, K., Liu, Y., Jiang, H., Wu, Z., Liu, Z., Zhao, W., et al. (2024) · Meta-Radiology · doi:10.1016/j.metrad.2024.100103 · Cited by 52
Self-citationA survey on privacy risks and protection in large language models
Chen, K., Zhou, X., Lin, Y., Feng, S., Shen, L., & Wu, P. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00177-1 · Cited by 101
“However, when LLMs are applied to specialized fields, they inevitably encounter issues such as hallu-cination [30, 53], insufficient professional knowledge in specific areas [31], and a failure to integrate the latest knowledge into constantly evolving industry scenarios [4].”
Neuro-GPT: Towards A Foundation Model For EEG
Cui, W., Jeong, W., Thölke, P., Medani, T., Jerbi, K., Joshi, A. A., & Leahy, R. M. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635453 · Cited by 183
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 157
Self-citationHeart-to-heart with ChatGPT: the impact of patients consulting AI for cardiovascular health advice
Lautrup, A. D., Hyrup, T., Schneider-Kamp, A., Dahl, M., Lindholt, J. S., & Schneider-Kamp, P. (2023) · Open Heart · doi:10.1136/openhrt-2023-002455 · Cited by 48
Verlingue, L., Boyer, C., Olgiati, L., Brutti Mairesse, C., Morel, D., & Blay, J.-Y. (2024) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2024.101064 · Cited by 44
Zhou, S., Liu, K., Li, D., Fu, C., Ning, Y., Ji, W., Liu, X., Xiao, B., & Wei, R. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103142 · Cited by 51
Healthcare agent: eliciting the power of large language models for medical consultation
Ren, Z., Zhan, Y., Yu, B., Ding, L., Xu, P., & Tao, D. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00021-x · Cited by 86
Gong, E. J., Bang, C. S., Lee, J. J., & Baik, G. H. (2025) · Journal of Medical Internet Research · doi:10.2196/84120 · Cited by 96
“The multiple-choice format inherently simplifies complex medical scenarios, potentially overestimating actual clinical competence [11].”
AirGPT: Pioneering the convergence of conversational AI with atmospheric science
Song, J., Ma, C., & Ran, M. (2025) · npj Climate and Atmospheric Science · doi:10.1038/s41612-025-01070-4 · Cited by 48
“In the realm of personalized communication, LLMs have played a pivotal role in customized learning by facilitating adaptive assessments and content recommendations 13 – 16 .”
Large language model agents can use tools to perform clinical calculations
Goodell, A. J., Chu, S. N., Rouholiman, D., & Chu, L. F. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01475-8 · Cited by 48
Maitland, A., Fowkes, R., & Maitland, S. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-080558 · Cited by 47
Kozaily, E., Geagea, M., Akdogan, R. E., Atkins, J., Elshazly, M. B., Guglin, M. E., Tedford, R. J., & Wehbe, R. M. (2024) · International Journal of Cardiology · doi:10.1016/j.ijcard.2024.132115 · Cited by 61
Vulnerability of Large Language Models to Prompt Injection When Providing Medical Advice
Lee, R. W., Jun, T. J., Lee, J.-M., Cho, S. I., Park, H., & Suh, J. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.49963 · Cited by 72
The impact of ChatGPT and LLMs on medical imaging stakeholders: Perspectives and use cases
Yang, J., Li, H. B., & Wei, D. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100007 · Cited by 46
Gupta, N., Khatri, K., Malik, Y., Lakhani, A., Kanwal, A., Aggarwal, S., & Dahuja, A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06592-8 · Cited by 58
Alba, C., Xue, B., Abraham, J., Kannampallil, T., & Lu, C. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01489-2 · Cited by 67
ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions
Buhr, C. R., Smith, H., Huppertz, T., Bahr-Hamm, K., Matthias, C., Blaikie, A., Kelsey, T., Kuhn, S., & Eckrich, J. (2023) · JMIR Medical Education · doi:10.2196/49183 · Cited by 45
KoSEL: Knowledge subgraph enhanced large language model for medical question answering
Zeng, Z., Cheng, Q., Hu, X., Zhuang, Y., Liu, X., He, K., & Liu, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112837 · Cited by 60
Li, H., Fu, J., & Python, A. (2025) · Journal of Medical Internet Research · doi:10.2196/71916 · Cited by 69
“Med-PaLM 2 [110], which is trained using a lower amount of but higher-quality domain data, has performed close to or exceeded the state-of-the-art level on several clinical datasets.”
Chandra, M., Naik, S., Ford, D., Okoli, E., De Choudhury, M., Ershadi, M., Ramos, G., Hernandez, J., Bhattacharjee, A., Warreth, S., et al. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732063 · Cited by 44
LLM4SecHW: Leveraging Domain-Specific Large Language Model for Hardware Debugging
Fu, W., Yang, K., Dutta, R. G., Guo, X., & Qu, G. (2023) · Asian Hardware Oriented Security and Trust Symposium (AsianHOST) · doi:10.1109/asianhost59942.2023.10409307 · Cited by 78
Large Language Models and Their Applications in Drug Discovery and Development: A Primer
Lu, J. J.-Q., Choi, K., Eremeev, M., Gobburu, J. V. S., Goswami, S., Liu, Q., Mo, G., Musante, C. J., & Shahin, M. H. (2025) · Clinical and Translational Science · doi:10.1111/cts.70205 · Cited by 56
Mansoor, I., Abdullah, M., Rizwan, M. D., & Fraz, M. M. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00403-0 · Cited by 55
“Our review shows that LLMs when enhanced with self-consistency, ensemble voting, or iterative refinement, can substantially improve performance on complex benchmarks like MedQA [74, 78].”
Kim, S. H., Wihl, J., Schramm, S., Berberich, C., Rosenkranz, E., Schmitzer, L., Serguen, K., Klenk, C., Lenhart, N., Zimmer, C., et al. (2025) · European Radiology · doi:10.1007/s00330-025-11484-6 · Cited by 38
Khosravi, B., Purkayastha, S., Erickson, B. J., Trivedi, H., & Gichoya, J. W. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100890 · Cited by 62
“2,3 6 Although these models primarily process text as input, with some using images for conditioning purposes, their primary focus is on generating high-quality images.”
Large language models provide unsafe answers to patient-posed medical questions
Draelos, R. L., Afreen, S., Blasko, B., Brazile, T. L., Chase, N., Desai, D. P., Evert, J., Gardner, H. L., Herrmann, L., House, A. V., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02428-5 · Cited by 87
Park, J., Fang, Y., Ta, C., Zhang, G., Idnay, B., Chen, F., Feng, D. D., Shyu, R., Gordon, E. R., Spotnitz, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104649 · Cited by 66
ClinicalAgent: Clinical Trial Multi-Agent System with Large Language Model-based Reasoning
Yue, L., Xing, S., Chen, J., & Fu, T. (2024) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3698587.3701359 · Cited by 113
Self-citationMitigating the risk of health inequity exacerbated by large language models
Ji, Y., Ma, W., Sivarajkumar, S., Zhang, H., Sadhu, E. M., Li, Z., Wu, X., Visweswaran, S., & Wang, Y. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01576-4 · Cited by 83
A foundational architecture for AI agents in healthcare
Liu, F., Niu, Y., Zhang, Q., Wang, K., Dong, Z., Wong, I. N., Cheng, L., Li, T., Duan, L., Li, K., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102374 · Cited by 41
“59 For instance, in robotic surgery, AI agents use their action systems to control robotic arms, executing delicate and highly precise movements based on real-time sensory feedback.”
Han, C., Kim, D. W., Kim, S., You, S. C., Park, J. Y., Bae, S., & Yoon, D. (2024) · iScience · doi:10.1016/j.isci.2024.109022 · Cited by 62
Enhancing Postmarketing Surveillance of Medical Products With Large Language Models
Matheny, M. E., Yang, J., Smith, J. C., Walsh, C. G., Al-Garadi, M. A., Davis, S. E., Marsolo, K. A., Fabbri, D., Reeves, R. R., Johnson, K. B., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.28276 · Cited by 38
Large Language Models in Gastroenterology: Systematic Review
Gong, E. J., Bang, C. S., Lee, J. J., Park, J., Kim, E., Kim, S., Kimm, M., & Choi, S.-H. (2024) · Journal of Medical Internet Research · doi:10.2196/66648 · Cited by 45
Best Practices for Large Language Models in Radiology
Blüthgen, C., Veen, D. V., Zakka, C., Link, K. E., Fanous, A. H., Daneshjou, R., Frauenfelder, T., Langlotz, C. P., Gatidis, S., & Chaudhari, A. S. (2025) · Radiology · doi:10.1148/radiol.240528 · Cited by 56
Human–large language model collaboration in clinical medicine: a systematic review and meta-analysis
Wang, G., Zhang, K., Jiang, J., Wang, C., Bi, H., Liang, H., Qi, Z., Huang, Y., Li, Y., & Yang, X. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02382-2 · Cited by 67
Factors for Patient Trust and Acceptance of Medical Artificial Intelligence
Bracic, A., Spector‐Bagdady, K., Towle, S., Zhang, R., James, C. A., & Price, W. N., II (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.0815 · Cited by 45
Generative AI models should include detection mechanisms as a condition for public release
Knott, A., Pedreschi, D., Chatila, R., Chakraborti, T., Leavy, S., Baeza-Yates, R., Eyers, D., Trotman, A., Teal, P. D., Biecek, P., et al. (2023) · Ethics and Information Technology · doi:10.1007/s10676-023-09728-4 · Cited by 34
Performance of ChatGPT and Bard in self-assessment questions for nephrology board renewal
Noda, R., Izaki, Y., Kitano, F., Komatsu, J., Ichikawa, D., & Shibagaki, Y. (2024) · Clinical and Experimental Nephrology · doi:10.1007/s10157-023-02451-w · Cited by 38
Sevgi, M., Antaki, F., & Keane, P. A. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-325046 · Cited by 47
Graph Artificial Intelligence in Medicine
Johnson, R., Li, M. M., Noori, A., Queen, O., & Žitnik, M. (2024) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-110723-024625 · Cited by 62
The AI revolution in glaucoma: Bridging challenges with opportunities
Li, F., Wang, D., Yang, Z., Zhang, Y., Jiang, J., Liu, X., Kong, K., Zhou, F., Tham, C. C. Y., Medeiros, F. A., et al. (2024) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2024.101291 · Cited by 47
Lai, X., Chen, J., Lai, Y., Huang, S., Cai, Y., Sun, Z., Wang, X. P., Pan, K., Gao, Q., & Huang, C. (2025) · JMIR Medical Informatics · doi:10.2196/59309 · Cited by 50
“Research indicates that using self-supervision or continuing pretraining on domain-specific corpora enhances model performance in downstream tasks [18-20].”
Self-citationBhattacharya, M., Pal, S., Chatterjee, S., Lee, S., & Chakraborty, C. (2024) · Molecular Therapy — Nucleic Acids · doi:10.1016/j.omtn.2024.102255 · Cited by 54
A Comprehensive Overview and Analysis of Large Language Models: Trends and Challenges
Mohammed, A., & Kora, R. (2025) · IEEE Access · doi:10.1109/access.2025.3573955 · Cited by 53
Saibene, A. M., Allevi, F., Calvo‐Henríquez, C., Maniaci, A., Mayo‐Yáñez, M., Paderno, A., Vaira, L. A., Felisati, G., & Craig, J. R. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08372-4 · Cited by 39
Enabling Demonstrated Consent for Biobanking with Blockchain and Generative AI
Barnes, C., Aboy, M., Minssen, T., Allen, J. W., Earp, B. D., Savulescu, J., & Mann, S. P. (2024) · The American Journal of Bioethics · doi:10.1080/15265161.2024.2416117 · Cited by 41
Generating dermatopathology reports from gigapixel whole slide images with HistoGPT
Tran, M., Schmidle, P., Guo, R. R., Wagner, S. J., Koch, V., Lupperger, V., Novotny, B., Murphree, D. H., Hardway, H. D., D’Amato, M., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-60014-x · Cited by 36
AI in microbiome‐related healthcare
Probul, N., Huang, Z., Saak, C. C., Baumbach, J., & List, M. (2024) · Microbial Biotechnology · doi:10.1111/1751-7915.70027 · Cited by 36
BURKE, H. B., Hoang, A., Lopreiato, J. O., King, H. B., Hemmer, P. A., Montgomery, M., & Gagarin, V. (2024) · JMIR Medical Education · doi:10.2196/56342 · Cited by 47
Liu, R., Zhang, Q., Han, T., Yang, B., Zhang, W., Yin, S., & Zhou, D. (2024) · IEEE Transactions on Industrial Cyber-Physical Systems · doi:10.1109/ticps.2024.3425326 · Cited by 44
Xu, Y., Tang, J., Men, A., & Chen, Q. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3482175 · Cited by 55
Generative AI in Medicine and Healthcare: Moving Beyond the ‘Peak of Inflated Expectations’
Zhang, P., Shi, J., & Boulos, M. N. K. (2024) · Future Internet · doi:10.3390/fi16120462 · Cited by 44
Nguyen, T. P., Carvalho, B., Sukhdeo, H., Joudi, K., Guo, N., Chen, M. C., Wolpaw, J. T., Kiefer, J., Byrne, M., Jamroz, T., et al. (2024) · BJA Open · doi:10.1016/j.bjao.2024.100280 · Cited by 44
Can ChatGPT Plan Your Retirement?: Generative AI and Financial Advice
Lo, A. W., & Ross, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.ec74a002 · Cited by 67
Evaluating large language models in echocardiography reporting: opportunities and challenges
Chao, C., Banerjee, I., Arsanjani, R., Ayoub, C., Tseng, A. S., Delbrouck, J.-B., Kane, G. C., López-Jiménez, F. A., Attia, Z. I., Oh, J. K., et al. (2025) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztae086 · Cited by 45
Rasool, Z., Kurniawan, S., Balugo, S., Barnett, S., Vasa, R., Chesser, C., Hampstead, B. M., Belleville, S., Mouzakis, K., & Bahar-Fuchs, A. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100083 · Cited by 34
AI approaches for the discovery and validation of drug targets
Wenteler, A., Cabrera, C., Wei, W., Neduva, V., & Barnes, M. R. (2024) · Cambridge Prisms Precision Medicine · doi:10.1017/pcm.2024.4 · Cited by 53
Pan, G., & Ni, J. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05871-8 · Cited by 37
Enabling GPTs for Expert-Level Environmental Engineering Question Answering
Zhu, J., Yang, M., Jiang, J., Bai, Y., Chen, D., & Ren, Z. J. (2024) · Environmental Science & Technology Letters · doi:10.1021/acs.estlett.4c00665 · Cited by 38
Generative Artificial Intelligence in Medicine
Shanmugam, D., Agrawal, M., Movva, R., Chen, I. Y., Ghassemi, M., Jacobs, M. L., & Pierson, E. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095332 · Cited by 42
Kollitsch, L., Eredics, K., Marszalek, M., Rauchenwald, M., Brookman-May, S. D., Burger, M., Körner-Riffard, K., & May, M. (2024) · World Journal of Urology · doi:10.1007/s00345-023-04749-6 · Cited by 38
Fan, M., Jin, C., Li, D., Deng, Y., Lin, Y., Chen, Y., Ma, Y., & Wang, T. (2023) · Frontiers in Pharmacology · doi:10.3389/fphar.2023.1289901 · Cited by 37
Fatemi, S., Hu, Y., & Mousavi, M. (2025) · ACM Transactions on Management Information Systems · doi:10.1145/3706119 · Cited by 50
A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Qiu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2025) · ACM Computing Surveys · doi:10.1145/3729218 · Cited by 113
Self-citation
Metadata
- Authors
- Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., Neal, D., Rashid, Q. M., Schaekermann, M., Wang, A., Dash, D., Chen, J. H., Shah, N. H., Lachgar, S., Mansfield, P. A., Prakash, S., Green, B., Dominowska, E., Agüera y Arcas, B., Tomašev, N., Liu, Y., Wong, R., Semturs, C., Mahdavi, S. S., Barral, J. K., Webster, D. R., Corrado, G. S., Matias, Y., Azizi, S., Karthikesalingam, A., & Natarajan, V.
- Affiliations
- Google (United States) (US)
- Stanford Medicine (US)
- Stanford Health Care (US)
- Corresponding author
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- ORCID
- Singhal, Karan
- Tu, Tao
- Gottweis, Juraj
- Sayres, Rory
- Amin, Mohamed
- Clark, Kevin
- Pfohl, Stephen R.
- Cole-Lewis, Heather
- Rashid, Qazi Mamunur
- Schaekermann, Mike
- Wang, Amy
- Dash, Dev
- Chen, Jonathan H.
- Shah, Nigam H.
- Lachgar, Sami
- Mansfield, Philip Andrew
- Prakash, Sushant
- Green, Bradley
- Dominowska, Ewa
- Tomašev, Nenad
- Liu, Yun
- Wong, Renee
- Semturs, Christopher
- Mahdavi, S. Sara
- Barral, Joelle K.
- Webster, Dale R.
- Corrado, Greg S.
- Matias, Yossi
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- Venue
- Nature Medicine
- Published
- 2025-01-08
- Volume
- 31
- Issue
- 3
- Pages
- 943-950
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11922739
- OpenAlex
- W4406152279
- Semantic Scholar
- b34c0b0169925a6c7f14e3de9764ed9505f30de3
- Cited by
- 1,734
- References
- 57
- Citations by year
- 2026: 771 · 2025: 591
- FWCI
- 277.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Nature Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Gordon and Betty Moore Foundation (#12409)
- American Heart Association
- National Institutes of Health (UM1TR004921)
- Stanford Bio-X
- National Institute on Drug Abuse (UG1DA015815)
- National Institute of Allergy and Infectious Diseases (R01AI178121)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- medical question answering, large language models, ensemble refinement, clinical reasoning, grounding, USMLE, MedQA, physician preferences
- MeSH terms
- Large Language Models, Humans, Physicians, Pilot Projects, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.