Toward expert-level medical question answering with large language models
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have shown promise in medical question answering, with Med-PaLM being the first to exceed a 'passing' score in United States Medical Licensing Examination style questions. However, challenges remain in long-form medical question answering and handling real-world workflows. Here, we present Med-PaLM 2, which bridges these gaps with a combination of base LLM improvements, medical domain fine-tuning and new strategies for improving reasoning and grounding through ensemble refinement and chain of retrieval. Med-PaLM 2 scores up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19%, and demonstrates dramatic performance increases across MedMCQA, PubMedQA and MMLU clinical topics datasets. Our detailed human evaluations framework shows that physicians prefer Med-PaLM 2 answers to those from other physicians on eight of nine clinical axes. Med-PaLM 2 also demonstrates significant improvements over its predecessor across all evaluation metrics, particularly on new adversarial datasets designed to probe LLM limitations (P < 0.001). In a pilot study using real-world medical questions, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time. While specialist answers were still preferred overall, both specialists and generalists rated Med-PaLM 2 to be as safe as physician answers, demonstrating its growing potential in real-world medical applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These models were chosen as they represent current leading LLMs3,36,40,41 as of February 2024.”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To account for the inherent stochasticity of LLMs, we report results from 10 independent runs per benchmark as mean accuracy with 95% confidence intervals (CIs), moving beyond the single-run point estimates common in prior works.2,3,4”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“medical domain-specific LLMs such as Medical Pathways Language Model (Med-PaLM) [2] and Biomedical Generative Pretrained Transformer (BioGPT) [3] have shown improved performance across various clinical natural language processing (NLP) tasks, including question answering, summarization, entity recognition, relation extraction, document classification, and literature mining.”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs have demonstrated strong performance across a wide range of biomedical knowledge and reasoning tasks, accelerating their adoption in health care workflows [1-6].”
From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models (LLMs) like OpenAI’s GPT-5 and knowledge-specific systems, such as OpenEvidence, have reported the ability to generate patient-specific, evidence-based recommendations.10”
From Large Language Model–Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations (2026) · cites this paper as [10]
Experimental — accuracy not yet measured
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“The advent of transformers 4 and LLMs 5,6 has renewed interest in the possibilities of AI for medical question-answering tasks—a long-standing ‘grand challenge’ 7–9 .”
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
New creatinine- and cystatin C-based equations to estimate GFR without race
Inker, L. A., Eneanya, N. D., Coresh, J., Tighiouart, H., Wang, D., Sang, Y., Crews, D. C., Doria, A., Estrella, M. M., Froissart, M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2102953 · Cited by 4,695
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“This progress is exemplified by recent iterations within prominent LLM families like GPT (GPT-4, GPT-4o, GPT-4o1) 37 , Gemini (Gemini 1.0, Gemini 1.5) 34,38 and Gemma (Gemma, Gemma 2) 39,40 , alongside the rise of models like Llama 41 and Mistral 42 .”
InfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“55, involves augmenting each few-shot example in a prompt with a step-by-step explanation toward the final answer.”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citation
Research works printed but not linked: 4
- 5. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT Vol. 1 (eds Burstein, J. et al.) 4171-4186 (Association for Computational Linguistics, 2019).
- 10. Bobrow, D. G.) 167-180 (MIT Press, 1994).
- 38. Achiam, J. et al. GPT-4 technical report. Preprint at https://arxiv. org/abs/2303.08774 (2023).
- 63. Randolph, Justus J. 2005 "Free-Marginal Multirater Kappa (multirater K [free]): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa." Presented at the Joensuu Learning and Instruction Symposium, vol. 2005 https://eric.ed.gov/?id=ED490661
Cited by
Items 601–700 of 1,031
AI-Empowered Genome Decoding: Applications of Large Language Models in Genomics
Li, S., Fan, W., & Zhou, Y. (2025) · Frontiers of digital education. · doi:10.1007/s44366-025-0051-1 · Cited by 4
Augmenting Medical Diagnostics with AI: A Dual Approach Using RAG-Based Chatbots and Nano GPT Models
Bande, S. P., Vinutha, N., & Krishnamurthy, P. K. (2025) · International Conference on Recent Advances in Information Technology · doi:10.1109/rait65068.2025.11088917 · Cited by 5
Kim, H. B., Tan, H. Q., Nei, W. L., Tan, Y., Cai, Y., & Wang, F. (2025) · BMC Medical Imaging · doi:10.1186/s12880-025-01844-5 · Cited by 6
Zhou, Y., Zhao, L., Mi, L., Liu, S., Yang, Y., Wang, Q., He, H., & Long, Y. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07825-0 · Cited by 8
Xu, T., Wang, N., Jin, Y., Chen, Y., & Ma, K. (2025) · Neural Regeneration Research · doi:10.4103/nrr.nrr-d-25-00121 · Cited by 7
Synthetic patient and interview transcript creator: an essential tool for LLMs in mental health
Warner, A., LeDue, J. M., Cao, Y., Tham, J. C. W., & Murphy, T. H. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1625444 · Cited by 7
Mitigating Ethical Issues for Large Language Models in Oncology: A Systematic Review
Zhou, S., Liu, X., Xu, Z., Zhan, Z., Song, M., Wang, J., Liu, S., Xu, H., & Zhang, R. (2025) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00076 · Cited by 8
Chen, G., Lin, C., Zhang, L., Luo, Z., Shin, Y. S., & Li, X. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07872-7 · Cited by 8
Dani, M., Prakash, M. J., Rosa, F., Akata, Z., & Liebe, S. (2024) · Communications Medicine · doi:10.1038/s43856-026-01653-z · Cited by 4
Pan, J., Jian, B., Hager, P., Zhang, Y., Liu, C., Jungmann, F., Li, H., Canisius, J., You, C., Wu, J., et al. (2026) · Nature Health · doi:10.1038/s44360-026-00152-8 · Cited by 29
Self-citationYoung, A. T., Lane, B. N., Ozog, D., & Matthews, N. H. (2023) · JAAD International · doi:10.1016/j.jdin.2023.12.004 · Cited by 5
A Comparative Review of GPT-4’s Applications in Medicine and High Decision Making
Bitri, R., & Ali, M. (2023) · International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications (CoNTESA) · doi:10.1109/contesa61248.2023.10384948 · Cited by 6
Location-enhanced syntactic knowledge for biomedical relation extraction
Zhang, Y., Yang, Z., Yang, Y., Lin, H., & Wang, J. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104676 · Cited by 9
Michalowski, M., Wilk, S., Michalowski, W., Rao, M., & Carrier, M. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104681 · Cited by 6
Debiased Noise Editing on Foundation Models for Fair Medical Image Classification
Jin, R., Deng, W., Chen, M., & Li, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72117-5_16 · Cited by 13
Foundation Models for Translational Cancer Biology
Tsang, K. K., Kivelson, S., Cortina, J. M. A., Kuchi, A., Berkowitz, J. S., Liu, H., Srinivasan, A., Friedrich, N. A., Fatapour, Y., & Tatonetti, N. P. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095633 · Cited by 6
CancerBot: A Retrieval-Augmented Generation based Cancer Chatbot Using Large Language Models
Azam, A., Naz, Z., & Khan, M. U. G. (2024) · International Conference on Open Source Systems and Technologies (ICOSST) · doi:10.1109/icosst64562.2024.10871155 · Cited by 6
Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Models
Ye, R., Ge, R., Yuchi, F., Chai, J., Wang, Y., & Chen, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-82240-7_9 · Cited by 13
European regulation of digital respiratory healthcare
Gilbert, S. (2023) · European Respiratory Society eBooks · doi:10.1183/2312508x.10000923 · Cited by 7
Email subjects generation with large language models: GPT-3.5, PaLM 2, and BERT
Loukili, S., Fennan, A., & Elaachak, L. (2024) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v14i4.pp4655-4663 · Cited by 8
OmicsNavigator : An auditable scientific partner for scalable hypothesis validation in spatial omicsPreprint
Yiyao, L., Vakharia, N., Liang, W., Mayer, A. T., Luo, R., Trevino, A. E., & Wu, Z. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.07.21.665821 · Cited by 5
Dobreva, J., Karasmanakis, I., Ivanišević, F., Horvat, T., Kitanovski, D., Gams, M., Mishev, K., & Simjanoska, M. (2025) · Data in Brief · doi:10.1016/j.dib.2025.112146 · Cited by 6
“The integration of artificial intelligence (AI) in medical education has gained significant momentum, with RAG systems showing particular promise for knowledge assessment and educational content delivery [ 1 , 2 ].”
Reasoning LLMs in the Medical Domain: A Literature Survey
Berger, A., Khanna, S., Sparrenberg, L., Deußer, T., Berghaus, D., & Sifa, R. (2025) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa65442.2025.11247922 · Cited by 9
“Current evaluation frameworks are increasingly incorporating assessments for potential harm and bias [22], [33], [34].”
InfluentialShaukat, F., Parwez, K., Ashraf, Z., Alhabeeb, A., Alotaibi, F. A., & Alnfiai, M. M. (2026) · IEEE Access · doi:10.1109/access.2026.3665786 · Cited by 4
“Singhal et al. [18] introduced Med-PaLM, achieving SOTA on medical QA but with prohibitive compute for edge.”
InfluentialPromises and challenges of applying large language models in the healthcare domain
Wang, Q., Gong, Z., Lai, Z., Bu, L., Dahlweid, F.-M., & Sun, H. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1772274 · Cited by 4
“Despite the fact that their real-world performance still falls short of experienced clinicians and requires rigorous oversight (2, 3), analyses of GPT-series models on medical licensure exams have validated their utility (4, 5).”
InfluentialMultifacetEval: Multifaceted Evaluation to Probe LLMs in Mastering Medical Knowledge
Zhou, Y., Liu, X., Ning, C., & Wu, J. Y. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/737 · Cited by 14
Lee, W., Lee, J., & Kim, H. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2585 · Cited by 11
Pawar, D., & Phansalkar, S. (2024) · International Conference on Automation, Computing and Renewable Systems (ICACRS) · doi:10.1109/icacrs62842.2024.10841739 · Cited by 4
EHealth: A Chinese Biomedical Language Model Built via Multi-Level Text Discrimination
Wang, Q., Dai, S., Xu, B., Lyu, Y., Wu, H. J., & Wang, H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3536177 · Cited by 4
DINOSim : Training‐Free Semantic Segmentation on Microscopy Images
González-Marfil, A., Gómez‐de‐Mariscal, E., & Arganda‐Carreras, I. (2026) · Microscopy Research and Technique · doi:10.1101/2025.03.09.642092 · Cited by 8
Park, J.-H., Cheong, C., Kang, S., Lee, I., Lee, S., Yoon, K. S., & Lee�, H. (2024) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi62660.2024.10913702 · Cited by 9
Benchmarking Large-Language Models for Resource-Efficient Medical AI for Edge Deployment
Fime, A. A., Hossain, M. Z., Zaman, S., Shahid, A. R., & Imteaj, A. (2025) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v5i1.35580 · Cited by 6
Stepanov, A. V., & Shichkina, Y. A. (2025) · International Conference on Neural Networks and Neurotechnologies (NeuroNT) · doi:10.1109/neuront66873.2025.11049975 · Cited by 5
Arkangel AI: A conversational agent for real-time, evidence-based medical question-answering
Villa, M. C., Castano-Villegas, N., Llano, I., Martinez, J., Guevara, M. F., Zea, J., & Velásquez, L. (2025) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2025.100274 · Cited by 6
Pinna, S., Massa, S. M., Fenu, M., Casti, G., & Riboni, D. (2025) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3733155.3733192 · Cited by 4
Mitigating Clinician Information Overload: Generative AI for Integrated EHR and RPM Data Analysis
Shetgaonkar, A., Pradhan, D., Arora, L., Girija, S. S., Raj, A., & Kapoor, S. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00284 · Cited by 5
“Techniques like Chain-of-Thought (CoT) prompting [51] encourage step-by-step reasoning, improving performance on complex queries.”
InfluentialBiswas, R., Mukhopadhyay, A., & Mukhopadhyay, S. (2025) · Journal of Yeungnam Medical Science · doi:10.12701/jyms.2025.42.53 · Cited by 6
Harnessing Large Language Models for Digital Building Logbook Implementation
Urlainis, A., Giat, Y., & Mitelman, A. (2025) · Buildings · doi:10.3390/buildings15183399 · Cited by 4
“Singhal et al. (2025) introduced Med-PaLM 2, a domain-adapted LLM capable of handling diverse clinical tasks, such as answering complex medical questions and generating patient-friendly explanations [42].”
Tarhini, A. A., Dave, P., & Naqa, I. E. (2025) · BJR|Artificial Intelligence · doi:10.1093/bjrai/ubaf015 · Cited by 7
“48-50 Med-PaLM M is another example of how foundation models can impact precision medicine with general Gen AI.”
Rappaport, N., Nogal, B., Perrott, K. M., Domina, V., Hood, L., & Price, N. D. (2025) · The Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-062124-013423 · Cited by 9
A framework to synthetically generate fine-grained hallucinated data
Saxena, A., Aggarwal, D., Badathala, N., & Bhattacharyya, P. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09864-x · Cited by 7
Virtual Health Assist: An LLM-Powered AI Platform for Symptom Diagnosis and Healthcare Assistance
Jithesh Kumar, A., Sachin, G. P., Touseef Ahamed, K., Shivlinga, & Chittaragi, N. B. (2025) · International Conference on Networks, Multimedia and Information Technology (NMITCON) · doi:10.1109/nmitcon65824.2025.11188258 · Cited by 9
“…on PaLM2 and fine-tuned further on medical question-answering datasets, reached expert-level performance, with up to 86.5% accuracy on USMLE-style MedQA questions, an increase of 19% over the previous model, with similarly robust performance on MedMCQA, PubMedQA, and MMLU clinical topics [15].”
InfluentialBlašković, L., Tanković, N., Lorencin, I., & Šegota, S. B. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9100256 · Cited by 8
Benson, R., Kenny, C., Ashraf‐Ganjouei, A., Zhao, M., Darawsheh, R., Qian, A. S., & Hong, J. C. (2025) · JCO Clinical Cancer Informatics · doi:10.1200/cci-25-00112 · Cited by 8
OAAgent: Multimodal LLM Agent for Predicting Knee Osteoarthritis Progression
Ahadian, P., Yang, M., Powlison, E., Li, X., Xu, W. F., & Guan, Q. (2025) · IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies · doi:10.1145/3721201.3721397 · Cited by 10
Yigamawano, F. K., Odom-Mabey, A. R., Xue, C., Pandey, H. K., & Kolachalama, V. B. (2025) · The Journal of Prevention of Alzheimer s Disease · doi:10.1016/j.tjpad.2025.100396 · Cited by 7
Structural chain of thoughts for radiology education
Awasthi, A., Chung, B., Vu, A. M., Khan, S. B., Le, N., Deng, Z., Agrawal, R., Wu, C. C., & Nguyen, H. V. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114433 · Cited by 4
“Traditionally, these models have been explored extensively for tasks related to clinical decision support systems, including radiology report generation [12,13], image interpretation [14,15], and visual question answering (VQA) [16,17] to enhance diagnostic accuracy and workflow efficiency.”
Lamberti, G., Panzuto, F., Massironi, S., Cives, M., Salvia, A. L., Spada, F., Faggiano, A., Pusceddu, S., Albertelli, M., Tafuto, S., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02274-x · Cited by 4
Srinivasan, S., Ai, X., Lo, T. W. S., Gilson, A., Zou, M., Zou, K., Kim, H., Yang, M., Pushpanathan, K., Yew, S. M. E., et al. (2025) · Ophthalmology Science · doi:10.1016/j.xops.2025.101050 · Cited by 6
Optimizing RAG-based LLMs for healthcare question answering tasks
Bedwa, M., Hooda, N., & Kumar, V. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02638-5 · Cited by 6
Large language models in 6G from standard to on-device networks
Zou, H., Zhao, Q., Lasaulce, S., Zhang, C., Tian, Y., Bariah, L., Bader, F., & Debbah, M. A. (2026) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-025-00239-6 · Cited by 13
Retrieval augmented large language model system for comprehensive drug contraindications
Bang, B., Yoon, J. C., Chang, D. J., Park, S., & Lee, Y. O. (2026) · Health Information Science and Systems · doi:10.1007/s13755-025-00420-z · Cited by 8
“Med-PaLM was the first LLM to achieve a passing score on United States Medical Licensing Examination (USMLE)-style questions, marking a milestone for AI-assisted clinical reasoning [13].”
Artificial intelligence-based tools for optimizing surgical research publications
Coronel, B. V., Marescaux, J., & Giménez, M. (2026) · Artificial Intelligence Surgery · doi:10.20517/ais.2025.112 · Cited by 4
Liu, C., Lei, Y., Xie, H., Wang, J., Yu, Y., & Lo, D. (2026) · ACM Computing Surveys · doi:10.1145/3787202 · Cited by 7
More, V., Lu, L., Ding, Z., Ding, Z., Xi, Z., Mizia, S., & Guo, N. (2026) · npj Precision Oncology · doi:10.1038/s41698-025-01265-1 · Cited by 14
Han, G. I., Choi, H., & Son, Y.-J. (2026) · BMC Nursing · doi:10.1186/s12912-026-04518-x · Cited by 8
“Studies frequently raised concerns regarding data privacy, ethical issues, and the reliability of AI-generated outputs [61, 65], indicating that successful implementation will require clear governance structures, human verification processes, and competency-based training.”
Audrin, B., & Audrin, C. (2026) · Technology in Society · doi:10.1016/j.techsoc.2026.103289 · Cited by 4
Mashkoor, M., Zhang, S., & Stensballe, A. (2026) · Biomedicines · doi:10.3390/biomedicines14030668 · Cited by 7
“Recent years have also witnessed a transformative rise in the use of large language models (LLMs) as tools for rheumatic research [69–71].”
Artificial intelligence in orthopedics: current applications, challenges, and future directions
Kim, S. Y., Choi, B. S., Han, H., & Ro, D. H. (2026) · Knee Surgery and Related Research · doi:10.1186/s43019-026-00317-5 · Cited by 5
AI aversion? Effects of author disclosure on young people’s perceptions of mental health advice
Brandtzæg, P. B., Skjuve, M. B., & Følstad, A. (2026) · Cyberpsychology Journal of Psychosocial Research on Cyberspace · doi:10.5817/cp2026-2-1 · Cited by 4
“In a related study, Singhal et al. (2025) observed that while AI-generated responses to medical inquiries were favored over those from general practitioners, they did not surpass the quality of answers provided by specialists.”
DxDirector: an agentic large language model driving the full-process clinical diagnosis
Xu, S., Huang, X., Wei, Z., Pang, L., Shen, H., & Cheng, X. (2026) · Nature Communications · doi:10.1038/s41467-026-71928-5 · Cited by 10
“…primarily because current LLMs excel in diagnosingcaseswithcomprehensiveclinicaldata — suchassymptoms, medical history, diagnostic test results, and clear instructions 7,15,17 — whereas real-world full-process clinical diagnosis often begins with only a patient ’ s vague chief complaint 2,4,20 .”
Savage, C. H., Sulam, J., Huang, C., & Yi, P. H. (2026) · American Journal of Roentgenology · doi:10.2214/ajr.26.34829 · Cited by 6
Rouhi, A. D., Menon, S. V., Ghanem, Y. K., & Han, J. J. (2026) · Cardiology and Therapy · doi:10.1007/s40119-026-00453-9 · Cited by 4
“5 is a multimodal “thinking” model with long-context and tool-use capabilities [24], while domain-specific Med-PaLM 2 and Med-Gemini models have shown strong performance on US Medical Licensing Examination (USMLE)-style and broader medical benchmarks [27].”
Interpretable agentic AI system with localized reasoning for radiology
Chen, W., Dong, Y., Ding, Z., Shi, Y., Zeng, F., Zhou, Y., Zhao, H., Luo, Y., Lin, T., Su, Y., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02994-8 · Cited by 10
Safety and security of large language models in healthcare
Clusmann, J., Freyer, O., Ostermann, M., Ferber, D., Laleh, N. G., Hilgers, L., Kolbinger, F. R., Schneider, C. V., Downing, A., Wekenborg, M., et al. (2026) · Nature · doi:10.1038/s41586-026-10687-1 · Cited by 8
Ding, T., Zou, Y., Chen, C., Shah, M., & Tian, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37038-9_12 · Cited by 17
“While modern AI systems achieve strong performance in static outcome prediction [10,11,21,29–31,37 ,39 ,40 ,44 ,52 ,53 ], World Models (WMs) offer a promising paradigm for such tasks.”
ChatGPT: a promising AI technology for psychoradiology research and practice
Pan, N., Qin, K., Shekara, A., & DelBello, M. P. (2023) · Psychoradiology · doi:10.1093/psyrad/kkad018 · Cited by 5
Zadeh, P. M., & Sattler, D. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10386018 · Cited by 4
AutoRAG: Grounding text and symbols
Schulz, T., Luttermann, M., & Möller, R. (2024) · KI - Künstliche Intelligenz · doi:10.1007/s13218-024-00850-z · Cited by 4
Efficient Medical Question Answering with Knowledge-Augmented Question Generation
Khlaut, J., Dancette, C., Ferreres, É., Benani, A., Herent, H., & Manceron, P. (2024) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2024.clinicalnlp-1.2 · Cited by 10
Yilanli, M., McKay, I., Jackson, D. I., & Sezgın, E. (2026) · Early Intervention in Psychiatry · doi:10.1111/eip.70230 · Cited by 4
Towards Citizen-Centric Multiagent Systems Based on Large Language Models
Li, Z., Yazdanpanah, V., Sarkadi, Ş., He, Y., Shafipour, E., & Stein, S. (2024) · International Conference on Information Technology for Social Good (GoodIT) · doi:10.1145/3677525.3678636 · Cited by 7
The two dimensions of pharmacy artificial intelligence tools
Smoke, S. M. (2024) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxae293 · Cited by 4
Jacinto, M. V. G., Oliveira, L. H. L. D., Rodrigues, T. C., de Medeiros, G. C., Medeiros, D. R., Silva, M. A., de Montalvão, L. C., González, M., & de Almeida, R. V. (2024) · Abu Dhabi International Petroleum Exhibition & Conference (ADIPEC) · doi:10.2118/221864-ms · Cited by 4
Mir, A. E., Luoga, L. T., Chen, B., Hanif, M. A., & Shafique, M. (2024) · IEEE International Conference on Image Processing Workshops (ICIPW) · doi:10.1109/icipcw64161.2024.10769172 · Cited by 13
On the reliability of Large Language Models to misinformed and demographically informed prompts
Aremu, T., Akinwehinmi, O., Nwagu, C., Ahmed, S. I., Orji, R., Amo, P. A. D., & Saddik, A. E. (2025) · AI Magazine · doi:10.1002/aaai.12208 · Cited by 10
Weicken, E., Mittermaier, M., Hoeren, T., Kliesch, J., Wiegand, T., Witzenrath, M., Ballhausen, M., Karagiannidis, C., Sander, L. E., & Gröschel, M. I. (2025) · Die Innere Medizin · doi:10.1007/s00108-025-01861-0 · Cited by 4
Zero-shot learning for clinical phenotyping: Comparing LLMs and rule-based methods
Neves, B. D., Moreira, J.-M., Gonçalves, S., Cerejo, J., da Silva, N. A., Leite, F., & Silva, M. J. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110181 · Cited by 13
Wu, J., Mei, X., Mao, R., He, K., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103620 · Cited by 12
Stress detection through prompt engineering with a general-purpose LLM
Esmi, N., Shahbahrami, A., Nabati, Y., Rezaei, B., Gaydadjiev, G. N., & de Jonge, P. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105462 · Cited by 7
Osobase, K. O., Akadiri, O. O., Dirisu, C. D., Asolo, E., Adewara, D. E., & Ekechi, C. C. (2025) · Asian Journal of Basic Science & Research · doi:10.38177/ajbsr.2025.7312 · Cited by 8
Murphy, K. B., Schilder, B. M., & Skene, N. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1794934 · Cited by 5
Volokhovskyi, V. (2024) · Innovative technologies and scientific solutions for industries · doi:10.30837/2522-9818.2024.2.048 · Cited by 5
Large language models: game-changers in the healthcare industry
Dong, B., Zhang, L., Yuan, J., Chen, Y., Li, Q., & Shen, L. (2024) · 中国科学通报:英文版 · doi:10.1016/j.scib.2024.11.031 · Cited by 7
Şimşek, C., de Madaria, E., Ebigbo, A., Vaněk, P., Elshaarawy, O., Voiosu, A. M., Antonelli, G., Turró, R., Gisbert, J. P., Nyssen, O. P., et al. (2025) · Endoscopy International Open · doi:10.1055/a-2637-2163 · Cited by 8
Exploration of ChatGPT application in diabetes education: a multi-dataset, multi-reviewer studyPreprint
Ying, Z., Fan, Y., Lu, J., Wang, P., Zou, L., Tang, Q., Chen, Y., Li, X., & Chen, Y. (2023) · medRxiv · doi:10.1101/2023.09.27.23296144 · Cited by 6
Artificial General Intelligence (AGI) for Medical Education and TrainingPreprint
Lema, K. G. (2023) · AfricArXiv (OSF Preprints) · doi:10.31730/osf.io/xytzm · Cited by 6
AI4Research: a survey of artificial intelligence for scientific research
Chen, Q., Yang, M., Qin, L., Liu, J., Yan, Z., Guan, J., Peng, D., Ji, Y., Li, H., Hu, M., et al. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-5000-y · Cited by 4
Zhu, J. F., Feng, P., Lu, J., Fang, B., & Yang, H. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad9667 · Cited by 6
Fine-Tuning a Personalized OpenBioLLM Using Offline Reinforcement Learning
Shi, J., Yuan, Y., Wang, A., & Nie, M. (2025) · Applied Sciences · doi:10.3390/app15052486 · Cited by 4
“OpenBioLLM is fine-tuned using different medical guidance datasets and uses direct preference optimization technology to align the model’s preferences with the desired results, enabling it to surpass other large language models such as GPT-4 and Med-PaLM-2 [21] on various biomedical benchmarks.”
Self-citationWilson, B., Natali, C., Roach, M., Scott, D., Rahat, A., Rawlinson, D. R., & Cabitza, F. (2025) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-025-09514-4 · Cited by 12
Evaluation of Vector Databases and LLMs in RAG-Based Multi-document Question Answering
Filipovska, E., Mladenovska, A., Dobreva, J., Kitanovski, D., Mitrov, G., Lameski, P., & Zdravevski, E. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-86162-8_1 · Cited by 7
Cao, Y., Li, Z., Cui, L., & Miao, C. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719852 · Cited by 9
Cha, S., Park, J., Choi, H., Ryu, H., & Seo, K. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3720111 · Cited by 3
Methodological Advances in Expert Systems with Artificial Intelligence
dos Santos, J. K. S., Siqueira, L. S., de Faria, E. M. B., Bispo, K. A., da Silva, G. J. F., & Soares, M. S. (2025) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.5753/sbsi.2025.246618 · Cited by 4
Walking Backward to Ensure Risk Management of Large Language Models in Medicine
Onitiu, D., Wachter, S., & Mittelstadt, B. (2025) · The Journal of Law Medicine & Ethics · doi:10.1017/jme.2025.10132 · Cited by 7
“31 Other approaches may also incorporate a reward model trained on human feedback (i.e., RLHF).”
PILs of Knowledge: A Synthetic Benchmark for Evaluating Question Answering Systems in Healthcare
Lunardi, R., Soprano, M., Coppola, P., Mea, V. D., Mizzaro, S., & Roitero, K. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730283 · Cited by 6
“However, while medical QA benchmarks exist, they tend to focus on scientific literature, clinical records, or physician-facing applications [36, 46, 59].”
Wang, Z., Li, K., Peng, S., Liu, L., Yang, X., Yao, K., Heinrich Herre, B., & Zhu, Y. (2025) · JMIR Medical Informatics · doi:10.2196/69286 · Cited by 3
“For instance, Google’s Med-PaLM2, which was fine-tuned from PaLM2 using data from the medical domain, achieved 86.5% accuracy on the MedQA dataset [15], which is close to the response level of clinical doctors [16].”
InfluentialEvaluating List Construction and Temporal Understanding capabilities of Large Language Models
Dumitru, A., Venktesh, V., Jatowt, A., & Anand, A. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744606 · Cited by 5
Shin, C., Eom, D., Lee, S. M., Park, J. E., Kim, K., & Lee, K. H. (2025) · Scientific Reports · doi:10.1038/s41598-025-16213-z · Cited by 9
Metadata
- Authors
- Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., Neal, D., Rashid, Q. M., Schaekermann, M., Wang, A., Dash, D., Chen, J. H., Shah, N. H., Lachgar, S., Mansfield, P. A., Prakash, S., Green, B., Dominowska, E., Agüera y Arcas, B., Tomašev, N., Liu, Y., Wong, R., Semturs, C., Mahdavi, S. S., Barral, J. K., Webster, D. R., Corrado, G. S., Matias, Y., Azizi, S., Karthikesalingam, A., & Natarajan, V.
- Affiliations
- Google (United States) (US)
- Stanford Medicine (US)
- Stanford Health Care (US)
- Corresponding author
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- ORCID
- Singhal, Karan
- Tu, Tao
- Gottweis, Juraj
- Sayres, Rory
- Amin, Mohamed
- Clark, Kevin
- Pfohl, Stephen R.
- Cole-Lewis, Heather
- Rashid, Qazi Mamunur
- Schaekermann, Mike
- Wang, Amy
- Dash, Dev
- Chen, Jonathan H.
- Shah, Nigam H.
- Lachgar, Sami
- Mansfield, Philip Andrew
- Prakash, Sushant
- Green, Bradley
- Dominowska, Ewa
- Tomašev, Nenad
- Liu, Yun
- Wong, Renee
- Semturs, Christopher
- Mahdavi, S. Sara
- Barral, Joelle K.
- Webster, Dale R.
- Corrado, Greg S.
- Matias, Yossi
- Azizi, Shekoofeh
- Karthikesalingam, Alan
- Natarajan, Vivek
- Venue
- Nature Medicine
- Published
- 2025-01-08
- Volume
- 31
- Issue
- 3
- Pages
- 943-950
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC11922739
- OpenAlex
- W4406152279
- Semantic Scholar
- b34c0b0169925a6c7f14e3de9764ed9505f30de3
- Cited by
- 1,734
- References
- 57
- Citations by year
- 2026: 771 · 2025: 591
- FWCI
- 277.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Nature Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Gordon and Betty Moore Foundation (#12409)
- American Heart Association
- National Institutes of Health (UM1TR004921)
- Stanford Bio-X
- National Institute on Drug Abuse (UG1DA015815)
- National Institute of Allergy and Infectious Diseases (R01AI178121)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- medical question answering, large language models, ensemble refinement, clinical reasoning, grounding, USMLE, MedQA, physician preferences
- MeSH terms
- Large Language Models, Humans, Physicians, Pilot Projects, United States
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.