Toward expert-level medical question answering with large language models
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a 'passing' score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These models were chosen as they represent current leading LLMs3,36,40,41 as of February 2024.”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To account for the inherent stochasticity of LLMs, we report results from 10 independent runs per benchmark as mean accuracy with 95% confidence intervals (CIs), moving beyond the single-run point estimates common in prior works.2,3,4”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“medical domain-specific LLMs such as Medical Pathways Language Model (Med-PaLM) [2] and Biomedical Generative Pretrained Transformer (BioGPT) [3] have shown improved performance across various clinical natural language processing (NLP) tasks, including question answering, summarization, entity recognition, relation extraction, document classification, and literature mining.”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs have demonstrated strong performance across a wide range of biomedical knowledge and reasoning tasks, accelerating their adoption in health care workflows [1-6].”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models (LLMs) like OpenAI’s GPT-5 and knowledge-specific systems, such as OpenEvidence, have reported the ability to generate patient-specific, evidence-based recommendations.10”
From Large Language Model–Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations (2026) · cites this paper as [10]
Experimental — accuracy not yet measured
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“The advent of transformers 4 and LLMs 5,6 has renewed interest in the possibilities of AI for medical question-answering tasks—a long-standing ‘grand challenge’ 7–9 .”
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
New creatinine- and cystatin C-based equations to estimate GFR without race
Inker, L. A., Eneanya, N. D., Coresh, J., Tighiouart, H., Wang, D., Sang, Y., Crews, D. C., Doria, A., Estrella, M. M., Froissart, M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2102953 · Cited by 4,695
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“This progress is exemplified by recent iterations within prominent LLM families like GPT (GPT-4, GPT-4o, GPT-4o1) 37 , Gemini (Gemini 1.0, Gemini 1.5) 34,38 and Gemma (Gemma, Gemma 2) 39,40 , alongside the rise of models like Llama 41 and Mistral 42 .”
InfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“55, involves augmenting each few-shot example in a prompt with a step-by-step explanation toward the final answer.”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citation
Research works printed but not linked: 4
- 5. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT Vol. 1 (eds Burstein, J. et al.) 4171-4186 (Association for Computational Linguistics, 2019).
- 10. Bobrow, D. G.) 167-180 (MIT Press, 1994).
- 38. Achiam, J. et al. GPT-4 technical report. Preprint at https://arxiv. org/abs/2303.08774 (2023).
- 63. Randolph, Justus J. 2005 "Free-Marginal Multirater Kappa (multirater K [free]): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa." Presented at the Joensuu Learning and Instruction Symposium, vol. 2005 https://eric.ed.gov/?id=ED490661
Cited by
Items 201–300 of 1,031
Wang, S., Tang, Z., Yang, H., Gong, Q., Gu, T., Ma, H., Wang, Y., Sun, W., Lian, Z., Mao, K., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02277-8 · Cited by 80
Self-citationA large language model for complex cardiology care
O’Sullivan, J. W., Palepu, A. K., Saab, K. K., Weng, W., Amponsah, D., Cheng, E., Cheng, Y., Chu, E. L., Desai, Y. B., Elezaby, A., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04190-9 · Cited by 50
Self-citationZsidai, B., Kaarre, J., Narup, E., Senorski, E. H., Pareek, A., Grassi, A., Ley, C., Longo, U. G., Herbst, E., Hirschmann, M. T., et al. (2024) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.12025 · Cited by 35
Applying Generative Artificial Intelligence to Critiquing Science Assessments
Nguyen, H., & Hayward, J. (2024) · Journal of Science Education and Technology · doi:10.1007/s10956-024-10177-x · Cited by 42
Evaluating Large Language Models in extracting cognitive exam dates and scores
Zhang, H., Jethani, N., Jones, S., Genes, N. G., Major, V. J., Jaffe, I. S., Cardillo, A. B., Heilenbach, N., Ali, N. F., Bonanni, L. J., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000685 · Cited by 39
Federated Learning in Practice: Reflections and Projections
Daly, K., Eichner, H., Kairouz, P., McMahan, H. B., Ramage, D., & Xu, Z. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00026 · Cited by 70
A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection
Yang, S., Sun, R., & Wan, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.256 · Cited by 84
Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions
Chen, H., Fang, Z., Singla, Y., & Dredze, M. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.182 · Cited by 31
Li, R., Wang, X., & Yu, H. Q. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.474 · Cited by 73
Machine learning for healthcare that matters: Reorienting from technical novelty to equitable impact
Balagopalan, A., Baldini, I., Celi, L. A., Gichoya, J. W., McCoy, L. G., Naumann, T., Shalit, U., van der Schaar, M., & Wagstaff, K. L. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000474 · Cited by 42
Exploring and Characterizing Large Language Models for Embedded System Development and Debugging
Englhardt, Z., Li, R. C., Nissanka, D., Zhang, Z., Narayanswamy, G., Breda, J., Liu, X., Patel, S., & Iyer, V. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650764 · Cited by 67
Citation-Enhanced Generation for LLM-based Chatbots
Li, W., Li, J., Ma, W., & Liu, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.79 · Cited by 20
Self-citationMulti-modal large language models in radiology: principles, applications, and potential
Shen, Y., Xu, Y., Ma, J., Rui, W., Zhao, C., Heacock, L. C., & Huang, C. (2024) · Abdominal Radiology · doi:10.1007/s00261-024-04708-8 · Cited by 41
EYE-Llama, an in-domain large language model for ophthalmology
Haghighi, T., Gholami, S., Sokol, J. T., Kishnani, E., Ahsaniyan, A., Rahmanian, H., Hedayati, F., Leng, T., & Alam, M. N. (2025) · iScience · doi:10.1016/j.isci.2025.112984 · Cited by 46
The impact of artificial intelligence on biomarker discovery
Javaid, H., Petrescu, C. C., Schmunk, L. J., Monahan, J., O’Reilly, P. G., Garg, M., McGirr, L., Khasawneh, M., Lail, M. A., Ganta, D., et al. (2025) · Emerging Topics in Life Sciences · doi:10.1042/etls20243003 · Cited by 36
Benchmarking large language model-based agent systems for clinical decision tasks
Liu, Y., Carrero, Z. I., Jiang, X., Ferber, D., Wölflein, G., Zhang, L., Jayabalan, S., Lenz, T., Hui, Z., & Kather, J. N. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02443-6 · Cited by 37
Self-citationLarge language model trained on clinical oncology data predicts cancer progression
Zhu, M., Lin, H., Jiang, J., Jinia, A. J., Jee, J., Pichotta, K., Waters, M., Rose, D., Schultz, N., Chalise, S., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01780-2 · Cited by 30
Automating expert-level medical reasoning evaluation of large language models
Zhou, S., Xie, W., Li, J., Zhan, Z., Song, M., Yang, H., Espinoza, C., Welton, L., Mai, X., Jin, Y., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02208-7 · Cited by 30
“…compare LLM-generated rationales with reference rationales 26,27 ; (2) humanexpert evaluation, which relies on domain experts ’ manual efforts to assess the reasoning quality 15,25,28,29 ; and (3) LLM-as-a-Judge, where a separate LLM is used to assess the quality of the reasoning process 30,31 .”
Scott, I. A., & Zuccon, G. (2024) · Internal Medicine Journal · doi:10.1111/imj.16393 · Cited by 39
Yang, X., Li, T., Wang, H., Zhang, R., Ni, Z., Liu, N., Zhai, H., Zhao, J., Meng, F., Zhou, Z., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01486-5 · Cited by 35
Self-citationLarge Language Models in radiology: A technical and clinical perspective
Kao, J.-P., & Kao, H.-T. (2025) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2025.100021 · Cited by 41
Giacobbe, D. R., Marelli, C., Manna, B. L., Padua, D., Malva, A., Guastavino, S., Signori, A., Mora, S., Rosso, N., Campi, C., et al. (2025) · npj Antimicrobials and Resistance · doi:10.1038/s44259-025-00084-5 · Cited by 33
LLM4WM: Adapting LLM for Wireless Multi-Tasking
Liu, X., Gao, S., Liu, B., Cheng, X., & Yang, L. (2025) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2025.3585845 · Cited by 82
A Systematic Literature Review of Hallucinations in Large Language Models
Woesle, C., Fischer-Brandies, L., & Buettner, R. (2025) · IEEE Access · doi:10.1109/access.2025.3601206 · Cited by 40
“This enables outcomes aligned with Scenarios 3– 7, as each output undergoes structured verification or correction in real time.”
InfluentialVavekanand, R., Laghari, A. A., & Kumar, T. (2026) · Iran Journal of Computer Science · doi:10.1007/s42044-025-00360-7 · Cited by 34
A Comprehensive Review of Multimodal Large Language Models for Medical Imaging and Omics Data
Vavekanand, R. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10504-y · Cited by 34
Exploring the Boundaries of GPT-4 in Radiology
Liu, Q., Hyland, S. L., Bannur, S., Bouzid, K., Castro, D. C., Wetscherek, M. T., Tinn, R., Sharma, H., Pérez‐García, F., Schwaighofer, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.891 · Cited by 59
HealthBench: Evaluating Large Language Models Towards Improved Human Health
Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.08775 · Cited by 654
“Some works have focused on specialized medical models (Luo et al., 2022; Singhal et al., 2023; Li et al., 2023; Singhal et al., 2025; Saab et al., 2024), and others on general models (Nori et al., 2023; Gilson et al., 2023; Takagi et al., 2023).”
Self-citationA Continued Pretrained LLM Approach for Automatic Medical Note Generation
Yuan, D., Rastogi, E., Naik, G., Rajagopal, S. P., Goyal, S., Zhao, F., Chintagunta, B., & Ward, J. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) · doi:10.18653/v1/2024.naacl-short.47 · Cited by 28
Toward Clinical-Grade Evaluation of Large Language Models
Moreno, A. C., & Bitterman, D. S. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2023.11.012 · Cited by 28
Latest developments of generative artificial intelligence and applications in ophthalmology
Feng, X., Xu, K., Luo, M., Chen, H., Yang, Y., He, Q., Song, C., Li, R., Wu, Y., Wang, H., et al. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100090 · Cited by 39
A case study on using a large language model to analyze continuous glucose monitoring data
Healey, E., Tan, A. L. M., Flint, K. L., Ruiz, J., & Kohane, I. S. (2025) · Scientific Reports · doi:10.1038/s41598-024-84003-0 · Cited by 36
Enhancing scientific table understanding with type-guided chain-of-thought
Yin, Z., & Wang, S. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104159 · Cited by 38
The Use of Artificial Intelligence for Cancer Therapeutic Decision-Making
Elemento, O., Khozin, S., & Sternberg, C. N. (2025) · NEJM AI · doi:10.1056/aira2401164 · Cited by 34
Siam, K., Varela, A., Faruk, M. J. H., Cheng, J. Q., Gu, H., Maruf, A. A., & Aung, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-31010-4 · Cited by 40
Centering health equity in large language model deployment
Singh, N., Lawrence, K., Richardson, S., & Mann, D. M. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000367 · Cited by 25
A Comprehensive Study of GPT-4V’s Multimodal Capabilities in Medical ImagingPreprint
Li, Y., Liu, Y., Wang, Z., Liang, X., Liu, L., Wang, L., Cui, L., Tu, Z., Wang, L., & Zhou, L. (2023) · medRxiv · doi:10.1101/2023.11.03.23298067 · Cited by 57
Self-citationPromise and Perils of Large Language Models for Cancer Survivorship and Supportive Care
Bitterman, D. S., Downing, A., Maués, J., & Lustberg, M. B. (2024) · Journal of Clinical Oncology · doi:10.1200/jco.23.02439 · Cited by 31
Large language models as a diagnostic support tool in neuropathology
Hewitt, K. J., Wiest, I. C., Carrero, Z. I., Bejan, L., Millner, T. O., Brandner, S., & Kather, J. N. (2024) · The Journal of Pathology: Clinical Research · doi:10.1002/2056-4538.70009 · Cited by 27
Large Language Models are Complex Table Parsers
Zhao, B., Ji, C., Zhang, Y., He, W., Wang, Y., Wang, Q., Feng, R., & Zhang, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.914 · Cited by 65
Mondillo, G., Frattolillo, V., Colosimo, S., Perrotta, A., Sessa, A. D., Guarino, S., del Giudice, E. M., & Marzuillo, P. (2024) · Pediatric Nephrology · doi:10.1007/s00467-024-06486-3 · Cited by 30
Gu, Z., Jia, W., Piccardi, M., & Yu, P. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103078 · Cited by 32
Towards conversational artificial intelligence for disease management
Liévin, V., Palepu, A. K., Weng, W., Saab, K. K., Stutz, D., Cheng, Y., Kulkarni, K., Mahdavi, S. S., Barral, J. K., Webster, D. R., et al. (2026) · Nature · doi:10.1038/s41586-026-10764-5 · Cited by 74
“Plan drafting and refinement are analogous to ‘ensemble refinement’ 46 and allow diverse management strategies to be explored concurrently.”
Self-citationIntegrating language into medical visual recognition and reasoning: A survey
Lu, Y., & Wang, A. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103514 · Cited by 48
Self-citationWatermarking techniques for large language models: a survey
Liang, Y., Xiao, J., Gan, W., & Yu, P. S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11474-6 · Cited by 65
Digital Forms for All: A Holistic Multimodal Large Language Model Agent for Health Data Entry
Cuadra, A., Breuch, J., Estrada, S., Ihim, D., Hung, I., Askaryar, D., Hassanien, M., Fessele, K. L., & Landay, J. A. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3659624 · Cited by 35
Kim, J., Lee, S., Jeon, H., Lee, K., Bae, H., Kim, B., & Seo, J. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456215 · Cited by 36
Leveraging foundation and large language models in medical artificial intelligence
Wong, I. N., Monteiro, O., Baptista‐Hon, D. T., Wang, K., Lu, W.-Y., Sun, Z., Nie, S., & Yin, Y. (2024) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003302 · Cited by 31
Joffe, I., Felobes, G., Elgouhari, Y., Talebi-Kalaleh, M., Mei, Q., & Chui, Y. H. (2025) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-6037 · Cited by 34
Opportunities and Challenges of Foundation Models in Industrial Manufacturing
Ayyat, M., Osman, M., & Nadeem, T. (2025) · IEEE Access · doi:10.1109/access.2025.3589777 · Cited by 31
A collaborative large language model for drug analysis
Zhou, H., Liu, F., Wu, J., Zhang, W., Huang, G., Clifton, L., Eyre, D. W., Luo, H., Liu, F., Branson, K., et al. (2025) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01471-z · Cited by 44
Safavi‐Naini, S. A. A., Ali, S., Shahab, O., Shahhoseini, Z., Savage, T., Rafiee, S., Samaan, J. S., Shabeeb, R. Q. A., Ladak, F., Yang, J. O., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02174-0 · Cited by 34
Artificial intelligence agents in cancer research and oncology
Truhn, D., Azizi, S., Zou, J., Cerdá-Alberich, L., Mahmood, F., & Kather, J. N. (2026) · Nature reviews. Cancer · doi:10.1038/s41568-025-00900-0 · Cited by 46
Self-citationA systematic evaluation of GPT-4V's multimodal capability for chest X-ray image analysis
Liu, Y., Li, Y., Wang, Z., Liang, X., Liu, L., Wang, L., Cui, L., Tu, Z., Wang, L., & Zhou, L. (2024) · Meta-Radiology · doi:10.1016/j.metrad.2024.100099 · Cited by 31
Self-citationGenerative AI from Theory to Practice: A Case Study of Financial Advice
Lo, A. W., & Ross, J. (2024) · doi:10.21428/e4baedd9.a1f6a281 · Cited by 30
Haoyu, T., Kuo, Y., Xin, D., Chenxi, Z., Mingwei, Y., Hongyan, W., Yiming, L., Minjie, H., Qiang, Z., Yu, J., et al. (2024) · Digital Chinese Medicine · doi:10.1016/j.dcmed.2025.01.007 · Cited by 29
A Brief Review on Benchmarking for Large Language Models Evaluation in Healthcare
Cilar, L., Chen, H., Chen, A., Topaz, M., Tam, W., Bian, J., & Štiglic, G. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70010 · Cited by 39
Bai, E., Luo, X., Zhang, Z., Adelgais, K., Ali, H., Finkelstein, J., & Kutzin, J. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02197-w · Cited by 27
“Literature [ 52, 53] highlights the importance of assessing the performance of systems using LLMs by comparing them to human assessments, particularly in the medical field.”
A review of ChatGPT in medical education: exploring advantages and limitations
Cheng, Y., & Zhu, L. (2025) · International Journal of Surgery · doi:10.1097/js9.0000000000002505 · Cited by 29
Livingston, L., Featherstone-Uwague, A., Barry, A., Barretto, K., Morey, T., Herrmannová, D., & Avula, V. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf054 · Cited by 29
Transforming agency: On the mode of existence of large language models
Barandiaran, X. E., & Almendros, L. S. (2025) · Phenomenology and the Cognitive Sciences · doi:10.1007/s11097-025-10094-3 · Cited by 41
“In concrete knowledge domains (like medicine) frontier models since 2023 often outperform average specialists at specific tasks (Guo et al., 2023; Singhal et al., 2025; Van Veen et al., 2024) and, perhaps not so surprisingly, can imitate philosophers with hardly distinguishable snippets…”
Artificial intelligence in bioinformatics: a survey
Jiang, J., Li, Y., Cao, S., Shan, Y., Liu, Y., Fei, T., Yu, Y., Feng, Y., Li, Y., Li, Y., et al. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf576 · Cited by 33
“On the other hand, Med-PaLM [103] and Med-PaLM2 [104] refine clinical dialogue through large-scale instruction tuning, while multimodal solutions like GMAI [102], ProtST [105], and Med-SA [106] embed text, imaging, and genomic data into a unified framework.”
Self-citationArtificial Intelligence for Risk Assessment on Primary Prevention of Coronary Artery Disease
Chen, S., Loguercio, S., Chen, K., Lee, S. E., Park, J., Liu, S., Sadaei, H. J., & Torkamani, A. (2023) · Current Cardiovascular Risk Reports · doi:10.1007/s12170-023-00731-4 · Cited by 27
Developing ChatGPT for biology and medicine: a complete review of biomedical question answering
Li, Q., Li, L., & Li, Y. (2024) · Biophysics Reports · doi:10.52601/bpr.2024.240004 · Cited by 36
Moglia, A., Georgiou, K. E., Cerveri, P., Mainardi, L. T., Satava, R. M., & Cuschieri, A. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10849-5 · Cited by 30
Large Language Models Can Be Contextual Privacy Protection Learners
Xiao, Y., Jin, Y., Bai, Y., Wu, Y., Yang, X., Luo, X., Yu, W., Zhao, X., Liu, Y., Gu, Q., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.785 · Cited by 23
Self-citationData Authenticity, Consent, and Provenance for AI Are All Broken: What Will It Take to Fix Them?
Longpre, S., Mahari, R., Obeng-Marnu, N., Brannon, W., South, T., Kabbara, J., & Pentland, A. (2024) · doi:10.21428/e4baedd9.a650f77d · Cited by 41
HALO: an ontology for representing and categorizing hallucinations in large language models
Nananukul, N., & Kejriwal, M. (2023) · Defense + Commercial Sensing · doi:10.1117/12.3014048 · Cited by 7
Ritoré, Á., Jiménez, C. M., González-Caballero, J. L., Rejón-Parrilla, J. C., Hervás, P., Toro, E., Parra-Calderón, C. L., Celi, L. A., Túnez, I., & de la Hoz, M. Á. A. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000599 · Cited by 27
Yang, K., Yu, Z., Su, X., Zhang, F., He, X., Wang, N., Zheng, Q., Yu, F., Wen, T., & Zhou, X. (2024) · Chinese Medicine · doi:10.1186/s13020-024-01005-w · Cited by 26
Neupane, S., Mitra, S., Mittal, S., Gaur, M., Golilarz, N. A., Rahimi, S., & Amirlatifi, A. (2025) · ACM Transactions on Computing for Healthcare · doi:10.1145/3709365 · Cited by 30
A Survey of Sustainability in Large Language Models: Applications, Economics, and Challenges
Singh, A., Patel, N. P., Ehtesham, A., Kumar, S., & Khoei, T. T. (2025) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc62904.2025.10903774 · Cited by 60
AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset
Nimo, C., Olatunji, T., Owodunni, A. T., Abdullahi, T., Ayodele, E., Sanni, M., Aka, E. C., Omofoye, F., Yuehgoh, F., Faniran, T., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.96 · Cited by 72
LLMs: A Promising New Tool for Improving Healthcare in Low-Resource Nations
Gangavarapu, A. (2023) · IEEE Global Humanitarian Technology Conference (GHTC) · doi:10.1109/ghtc56179.2023.10354650 · Cited by 27
Evaluating the quality of medical content on YouTube using large language models
Khalil, M., Mohamed, F., & Shoufan, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-94208-6 · Cited by 29
Patient agency and large language models in worldwide encoding of equity
Armoundas, A. A., & Loscalzo, J. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01598-y · Cited by 27
A survey on biomedical automatic text summarization with large language models
Huang, Z., Chen, X., Wang, Y., Huang, J., & Zhao, X. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104216 · Cited by 27
Yang, T., Xiao, Y., Bao, Z., Hao, J., & Peng, J. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf601 · Cited by 34
Agweyu, A., Mwaniki, P., Musau, W., Korom, R., Isaaka, L., Wanyama, C. W., Kiptinness, S., Adan, N., Emmanuel-Fabula, M., & Mateen, B. A. (2026) · Nature Health · doi:10.1038/s44360-026-00082-5 · Cited by 32
“Advanced LLMs have shown impressive capabilities on medical examinations 21,22 and standardized case vignettes 23 , often matching or surpassing human clinicians on structured clinical problem-solving tasks.”
Teaching multimodal LLMs to comprehend 12-lead electrocardiographic images
Liu, R., Bai, Y., Yue, X., & Zhang, P. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02551-3 · Cited by 59
Guthrie, E., Levy, D., & del Carmen, G. (2024) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2024.02.016 · Cited by 22
A Medical Diagnostic Assistant Based on LLM
Wu, C., Lin, Z., Fang, W., & Huang, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-1717-0_12 · Cited by 39
Opportunities and challenges in the application of large artificial intelligence models in radiology
Pan, L., Zhao, Z., Lu, Y., Tang, K., Fu, L., Liang, Q., & Peng, S. (2024) · Meta-Radiology · doi:10.1016/j.metrad.2024.100080 · Cited by 27
HumanELY: Human evaluation of LLM yield, using a novel web-based evaluation toolPreprint
Awasthi, R., Mishra, S. V., Mahapatra, D., Khanna, A. K., Maheshwari, K. K., Cywiński, J. B., Papay, F., & Mathur, P. (2023) · medRxiv · doi:10.1101/2023.12.22.23300458 · Cited by 37
Das, A., Tariq, A., Batalini, F., Dhara, B., & Banerjee, I. (2024) · medRxiv · doi:10.1101/2024.03.20.24304627 · Cited by 42
Rethinking artificial intelligence in medicine: from tools to agents
Lee, Y. C. (2025) · Clinical and Experimental Emergency Medicine · doi:10.15441/ceem.25.125 · Cited by 28
“For example, Med-PaLM 2 (Google) demonstrated expert-level performance on US Medical Licensing Examination (USMLE)-style questions [6], and GPT-4 (OpenAI) can generate reasonably accurate diagnostic suggestions [7].”
ArgMed-Agents: Explainable Clinical Decision Reasoning with LLM Disscusion via Argumentation Schemes
Hong, S., Xiao, L., Zhang, X., & Chen, J. (2024) · 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822109 · Cited by 21
Self-citationEvaluating Visual Explanations of Attention Maps for Transformer-Based Medical Imaging
Chung, M., Won, J. B., Kim, G., Kim, Y., & Özbulak, U. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-77610-6_11 · Cited by 33
Roshani, M. A., Zhou, X., Qiang, Y., Suresh, S., Hicks, S. D., Sethuraman, U., & Zhu, D. (2025) · JMIR AI · doi:10.2196/67363 · Cited by 28
“Despite the remarkable performance of proprietary black-box LLMs like GPT-4 and MedPaLM-2 [20], there is growing interest in deploying white-box models in health care and other high-stakes domains.”
Biomedical Natural Language Processing in the Era of Large Language Models
Usuyama, N., Wong, C., Zhang, S., Naumann, T., & Poon, H. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095406 · Cited by 27
Application of Large Language Models in Stroke Rehabilitation Health Education: 2-Phase Study
Qiang, S., Liao, Y., Gu, Y., Gu, Y., Zhang, Y., Wang, Y., Xu, Z., Xu, Z., Han, N., Zhang, H., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/73226 · Cited by 26
“For instance, some studies have found that Med-PaLM 2 has made significant progress in medical question answering, particularly across multiple medical benchmarks and real-world problem-solving [38].”
Li, M., Zhan, Z., Yang, H., Xiao, Y., Zhou, H., Huang, J., & Zhang, R. (2025) · Science Advances · doi:10.1126/sciadv.adr1443 · Cited by 42
Vatsal, S., Dubey, H., & Singh, A. (2026) · IEEE Access · doi:10.1109/access.2026.3651218 · Cited by 31
“For example, medical question answering systems refined with domain specific supervision and evaluation protocols achieve ground breaking reliability on USMLE style questions [42].”
Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges
Ahsan, H., McInerney, D. J., Kim, J., Potter, C., Young, G. S., Amir, S., & Wallace, B. (2023) · PubMed · doi:10.48550/arxiv.2309.04550 · Cited by 54
Li, B., Wang, A., Strachan, P., Séguin, J. A., Lachgar, S., Schroeder, K. C., Fleck, M. S., Wong, R., Karthikesalingam, A., Natarajan, V., et al. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651891 · Cited by 25
Self-citationTowards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm
Liu, L., Yang, X., Li, F., Chi, C., Shen, Y., Lyu, S., Zhang, M., Ma, X., Lv, X., Ma, L., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671575 · Cited by 38
Advancing EHR analysis: Predictive medication modeling using LLMs
Alghamdi, H. M., & Mostafa, A. (2025) · Information Systems · doi:10.1016/j.is.2025.102528 · Cited by 26
Grounding Foundation Models through Federated Transfer Learning: A General Framework
Kang, Y., Fan, T., Gu, H., Zhang, X., Fan, L., & Yang, Q. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3742788 · Cited by 47
Explainable differential diagnosis with dual-inference large language models
Zhou, S., Lin, M., Ding, S., Wang, J., Chen, C., Melton, G. B., Zou, J., & Zhang, R. (2025) · npj Health Systems · doi:10.1038/s44401-025-00015-6 · Cited by 20
Streamlining evidence based clinical recommendations with large language models
Li, D., Jiang, N., Huang, K., Tu, R., Ouyang, S., Yu, H., Lin, Q., Chen, Y., Zhou, T., Tong, D., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02273-y · Cited by 29
Metadata
- Authors
- Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., Neal, D., Rashid, Q. M., Schaekermann, M., Wang, A., Dash, D., Chen, J. H., Shah, N. H., Lachgar, S., Mansfield, P. A., Prakash, S., Green, B., Dominowska, E., Agüera y Arcas, B., Tomašev, N., Liu, Y., Wong, R., Semturs, C., Mahdavi, S. S., Barral, J. K., Webster, D. R., Corrado, G. S., Matias, Y., Azizi, S., Karthikesalingam, A., & Natarajan, V.
- Affiliations
- Google (United States) (US)
- Stanford Medicine (US)
- Stanford Health Care (US)
- Corresponding author
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- ORCID
- Singhal, Karan
- Tu, Tao
- Gottweis, Juraj
- Sayres, Rory
- Amin, Mohamed
- Clark, Kevin
- Pfohl, Stephen R.
- Cole-Lewis, Heather
- Rashid, Qazi Mamunur
- Schaekermann, Mike
- Wang, Amy
- Dash, Dev
- Chen, Jonathan H.
- Shah, Nigam H.
- Lachgar, Sami
- Mansfield, Philip Andrew
- Prakash, Sushant
- Green, Bradley
- Dominowska, Ewa
- Tomašev, Nenad
- Liu, Yun
- Wong, Renee
- Semturs, Christopher
- Mahdavi, S. Sara
- Barral, Joelle K.
- Webster, Dale R.
- Corrado, Greg S.
- Matias, Yossi
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- Venue
- Nature Medicine
- Published
- 2025-01-08
- Volume
- 31
- Issue
- 3
- Pages
- 943-950
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11922739
- OpenAlex
- W4406152279
- Semantic Scholar
- b34c0b0169925a6c7f14e3de9764ed9505f30de3
- Cited by
- 1,734
- References
- 57
- Citations by year
- 2026: 771 · 2025: 591
- FWCI
- 277.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Nature Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Gordon and Betty Moore Foundation (#12409)
- American Heart Association
- National Institutes of Health (UM1TR004921)
- Stanford Bio-X
- National Institute on Drug Abuse (UG1DA015815)
- National Institute of Allergy and Infectious Diseases (R01AI178121)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- medical question answering, large language models, ensemble refinement, clinical reasoning, grounding, USMLE, MedQA, physician preferences
- MeSH terms
- Large Language Models, Humans, Physicians, Pilot Projects, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.