Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Open accessAbstract
Generalist foundation models such as GPT-4 have displayed surprising capabilities in a wide variety of domains and tasks. Yet, there is a prevalent assumption that they cannot match specialist capabilities of fine-tuned models. For example, most explorations to date on medical competency benchmarks have leveraged domain-specific training, as exemplified by efforts on BioGPT and Med-PaLM. We build on a prior study of GPT-4's capabilities on medical challenge benchmarks in the absence of special training. Rather than using simple prompting to highlight the model's out-of-the-box capabilities, we perform a systematic exploration of prompt engineering. We find that prompting innovation can unlock deeper specialist capabilities and show that GPT-4 easily tops prior leading results for medical benchmarks. The prompting methods we explore are general purpose, and make no specific use of domain expertise, removing the need for expert-curated content. Our experimental design carefully controls for overfitting during the prompt engineering process. We introduce Medprompt, based on a composition of several prompting strategies. With Medprompt, GPT-4 achieves state-of-the-art results on all nine of the benchmark datasets in the MultiMedQA suite. The method outperforms leading specialist models such as Med-PaLM 2 by a significant margin with an order of magnitude fewer calls to the model. Steering GPT-4 with Medprompt achieves a 27% reduction in error rate on the MedQA dataset over the best methods to date achieved with specialist models and surpasses a score of 90% for the first time. Beyond medical problems, we show the power of Medprompt to generalize to other domains and provide evidence for the broad applicability of the approach via studies of the strategy on exams in electrical engineering, machine learning, philosophy, accounting, law, nursing, and clinical psychology.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have surpassed the average human score on the USMLE, rising from 38.1% in September 2021 to 90.2% in November 2023, where the human passing score is 60% and human expert score is 87%.
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [3]
Table 1 reports 90.2% for GPT-4 with Medprompt on the US four-option MedQA benchmark. Section 3.1 says those questions are in the style of the USMLE, so the paper does not establish a 90.2% score on the licensing exam itself. The earlier score and human benchmarks have separate citations in the citing sentence and are not judged here. (LLM, conf=0.86)
Experimental — accuracy not yet measuredTo account for the inherent stochasticity of LLMs, results are reported from 10 independent runs per benchmark as mean accuracy with 95% confidence intervals.
“To account for the inherent stochasticity of LLMs, we report results from 10 independent runs per benchmark as mean accuracy with 95% confidence intervals (CIs), moving beyond the single-run point estimates common in prior works.2,3,4”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [4]
The citation is attached to the comparison with prior work, not to the citing authors’ own 10-run procedure. Table 1 (Results; page unavailable) reports benchmark accuracy as point estimates without confidence intervals, but the paper does not establish that those estimates came from single independent runs. Its five parallel API calls are an inference-time ensemble, not 10 benchmark runs. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe study included 6 cases that could feasibly be completed within a single study session while remaining comparable to standard practices in national licensing and objective structured clinical examinations.
“Our study included 6 cases that could feasibly be completed within a single study session while remaining comparable to standard practices in national licensing and objective structured clinical examinations to use a small, but broad sample of clinical cases.6,46,47,48,49”
From Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial (2024) · cites this paper as [47]
Section 3.1 (page unavailable) describes questions modeled on national medical licensing examinations, supporting the licensing-exam comparison. It does not address objective structured clinical examinations or whether six cases can be completed in one study session; the six cases are described as the citing authors’ own study design. (LLM, conf=0.84)
Experimental — accuracy not yet measuredLLMs have surpassed human experts on medical exams, with LLMs scoring 90.2% and human experts 87% against a passing score of 60%.
“More recently, LLMs have surpassed human experts on medical exams (LLMs: 90.2% (Nori et al. 2023), human experts: 87% (Li´ evin et al. 2022), passing score: 60%).”
From Human-Clinical AI Agent Collaboration (2025) · cites this paper as Nori et al. 2023
Medprompt, which does not use debate, performs best overall on MedQA and has lower costs associated with it.
“However, the recently introduced Medprompt strategy (Nori et al., 2023), which does not use debate, performs the best overall.”
From Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (2023) · cites this paper as Nori et al., 2023
LLMs have shown the ability to perform accurate single-turn medical question-answering.
“Recently, LLMs have shown the ability to encode clinical knowledge7,8, retrieve relevant medical texts9, and perform accurate single-turn medical question-answering3,10–12.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [3]
Reported multiple choice evaluations typically show specialist QAs have lower performance.
“reported multiple choice evaluations where specialist QAs typically have lower performance3”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [3]
GPT-4 shows superior performance compared to GPT-3.5 in knowledge assessments.
“with GPT-4 showing superior performance compared to GPT-3.5 in knowledge assessments3,43.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [3]
LLMs have advanced text generation capability and extensive domain-specific knowledge.
“LLMs have advanced text generation capability and extensive domain-specific knowledge [8,9].”
From Mitigating cognitive biases in clinical decision-making through multi-agent conversations using large language models: simulation study (2024) · cites this paper as [8]
LLMs tailored for healthcare have achieved state-of-the-art (SOTA) performance on a range of medical benchmarks.
“In recent years, large language models (LLMs) tailored for healthcare have achieved state-of-the-art (SOTA) performance on a range of medical benchmarks.2,3,4”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [4]
The LLM alone's better performance may be explained by sensitivity of LLM output to prompt formulation.
“This may be explained by the sensitivity of LLM output to prompt formulation.32”
From Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial (2024) · cites this paper as [32]
LLMs have achieved excellent performance on medical licensing exams.
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 17
- Niels J. Blunch. Position bias in multiple-choice questions. Journal of Marketing Research, 21(2):216-220, 1984.
- Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes. Ensemble selection from libraries of models. In Proceedings of the twenty-first international conference on Machine learning, page 18, 2004.
- Matthias Feurer and Frank Hutter. Hyperparameter optimization. Automated machine learning: Methods, systems, challenges, pages 3-33, 2019.
- Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao, and Hoifung Poon. Self-verification improves few-shot clinical information extraction. In ICML 3rd Workshop on Interpretable Machine Learning in Healthcare (IMLH), 2023.
- Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Nau- mann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomed- ical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1-23, 2021.
- Ayanna M. Howard, Cha Zhang, and Eric Horvitz. Addressing bias in machine learning algo- rithms: A pilot study on emotion recognition for intelligent systems. 2017 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO), pages 1-7, 2017.
- Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
- Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, and Jaewoo Kang. Look at the first sentence: Position bias in question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1109-1121, Online, November 2020. Association for Computational Linguistics.
- Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. Per- formance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLoS digital health, 2(2):e0000198, 2023.
- Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, and Hanna Wallach. Assessing the fairness of ai systems: AI practitioners' processes, challenges, and needs for support. In 25th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2022), February 2022.
- John McCarthy, Marvin L Minsky, Nathaniel Rochester, and Claude E Shannon. A proposal for the Dartmouth summer research project on artificial intelligence, August 31, 1955. AI magazine, 27(4):12-12, 2006.
- Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, pages 248-260. PMLR, 2022.
- Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
- Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. Survey Certification.
- Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy Liang, and Jure Leskovec. Deep bidirectional language-knowledge graph pretraining. In Neural Information Processing Systems (NeurIPS), 2022.
- Michihiro Yasunaga, Jure Leskovec, and Percy Liang. Linkbert: Pretraining language models with document links. In Association for Computational Linguistics (ACL), 2022.
- Travis Zack, Eric Lehman, Mirac Suzgun, Jorge A. Rodriguez, Leo Anthony Celi, Judy Gichoya, Dan Jurafsky, Peter Szolovits, David W. Bates, Raja-Elie E. Abdulnour, Atul J. Butte, and Emily Alsentzer. Coding inequity: Assessing gpt-4's potential for perpetuating racial and gender biases in healthcare. medRxiv, 2023.
Cited by
Sorted by citations
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · see citing works
Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial
Goh, E., Gallo, R., Hom, J., Strong, E., Weng, Y., Kerman, H., Cool, J. A., Kanjee, Z., Parsons, A. S., Ahuja, N., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.40969 · Cited by 724 · see citing works
Self-citationTowards conversational diagnostic artificial intelligence
Tu, T., Schaekermann, M., Palepu, A., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Cheng, Y., et al. (2025) · Nature · doi:10.1038/s41586-025-08866-7 · Cited by 366 · see citing works
“The scenario packs from the United Kingdom used in the OSCE study are also available for download from https://www. thefederation.uk/sites/default/files/documents/Station%202%20 Scenario%20Pack%20%2816%29.pdf.”
InfluentialEmpowering biomedical discovery with AI agents
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.1016/j.cell.2024.09.022 · Cited by 315 · see citing works
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.372 · Cited by 310 · see citing works
Matching patients to clinical trials with large language models
Jin, Q., Wang, Z., Floudas, C. S., Chen, F., Gong, C., Bracken-Clarke, D., Xue, E., Yang, Y., Sun, J., & Lu, Z. (2024) · Nature Communications · doi:10.1038/s41467-024-53081-z · Cited by 287 · see citing works
Chen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M. B., Ai, X., Lai, P.-T., Wang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-56989-2 · Cited by 219 · see citing works
“5, which may not provide a full spectrum of representative LLMs from different categories.”
A guide to artificial intelligence for cancer researchers
Pérez-López, R., Laleh, N. G., Mahmood, F., & Kather, J. N. (2024) · Nature reviews. Cancer · doi:10.1038/s41568-024-00694-7 · Cited by 220 · see citing works
Goh, E., Gallo, R. J., Strong, E., Weng, Y., Kerman, H. M., Freed, J. A., Cool, J. A., Kanjee, Z., Lane, K. P., Parsons, A. S., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03456-y · Cited by 179 · see citing works
Self-citationGaber, F., Shaik, M., Allega, F., Bilecz, A. J., Busch, F., Goon, K., Franke, V., & Akalin, A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01684-1 · Cited by 165 · see citing works
Use of a Large Language Model to Assess Clinical Acuity of Adults in the Emergency Department
Williams, C. Y. K., Zack, T., Miao, B. Y., Sushil, M., Wang, M., Kornblith, A. E., & Butte, A. J. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.8895 · Cited by 148 · see citing works
Towards building multilingual language model for medicine
Qiu, P., Wu, C., Zhang, X., Lin, W., Wang, H., Zhang, Y., Wang, Y., & Xie, W. (2024) · Nature Communications · doi:10.1038/s41467-024-52417-z · Cited by 147 · see citing works
Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine
Jin, Q., Chen, F., Zhou, Y., Xu, Z., Cheung, J. M., Chen, R., Summers, R. M., Rousseau, J. F., Ni, P., Landsman, M. J., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01185-7 · Cited by 146 · see citing works
Self-citationPrompt engineering as a new 21st century skill
Federiakin, D. A., Molerov, D., Zlatkin‐Troitschanskaia, O., & Maur, A. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1366434 · Cited by 185 · see citing works
Prompt Engineering Paradigms for Medical Applications: Scoping Review
Zaghir, J., Naguib, M., Bjelogrlic, M., Névéol, A., Tannier, X., & Lovis, C. (2024) · Journal of Medical Internet Research · doi:10.2196/60501 · Cited by 143 · see citing works
Physician- and Large Language Model–Generated Hospital Discharge Summaries
Williams, C. Y. K., Subramanian, C. R., Ali, S. S., Apolinario, M., Askin, E., Barish, P. N., Cheng, M., Deardorff, W. J., Donthi, N., Ganeshan, S., et al. (2025) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2025.0821 · Cited by 123 · see citing works
Capabilities of Gemini Models in Medicine
Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.18416 · Cited by 117 · see citing works
Application of large language models in medicine
Liu, F., Zhou, H., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Hua, Y., Zhou, P., et al. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00279-5 · Cited by 94 · see citing works
Self-citationLarge language models for preventing medication direction errors in online pharmacies
Pais, C., Liu, J., Voigt, R., Gupta, V., Wade, E., & Bayati, M. (2024) · Nature Medicine · doi:10.1038/s41591-024-02933-8 · Cited by 94 · see citing works
Exploring scalable medical image encoders beyond text supervision
Pérez-García, F., Sharma, H., Bond-Taylor, S., Bouzid, K., Salvatelli, V., Ilse, M., Bannur, S., Castro, D. C., Schwaighofer, A., Lungren, M. P., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00965-w · Cited by 115 · see citing works
Triola, M. M., & Rodman, A. (2025) · Academic Medicine · doi:10.1097/acm.0000000000005963 · Cited by 92 · see citing works
Designing Heterogeneous LLM Agents for Financial Sentiment Analysis
Xing, F. (2025) · ACM Transactions on Management Information Systems · doi:10.1145/3688399 · Cited by 88 · see citing works
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 82 · see citing works
Evaluating the ChatGPT family of models for biomedical reasoning and classification
Chen, S., Li, Y., Lu, S., Van, H., Aerts, H. J. W. L., Savova, G. K., & Bitterman, D. S. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad256 · Cited by 91 · see citing works
Self-citationLanguage models for biological research: a primer
Simon, E. P., Swanson, K., & Zou, J. (2024) · Nature Methods · doi:10.1038/s41592-024-02354-y · Cited by 76 · see citing works
Image and data mining in reticular chemistry powered by GPT-4V
Zheng, Z., He, Z., Khattab, O., Rampal, N., Zaharia, M., Borgs, C., Chayes, J., & Yaghi, O. M. (2024) · Digital Discovery · doi:10.1039/d3dd00239j · Cited by 69 · see citing works
Du, X., Novoa-Laurentiev, J., Plasek, J. M., Chuang, Y.-W., Wang, L., Marshall, G. A., Mueller, S. K., Chang, F., Datta, S., Paek, H., et al. (2024) · eBioMedicine · doi:10.1016/j.ebiom.2024.105401 · Cited by 69 · see citing works
Evaluation and mitigation of cognitive biases in medical language models
Schmidgall, S., Harris, C., Essien, I., Olshvang, D., Rahman, T., Kim, J. W., Ziaei, R., Eshraghian, J., Abadir, P., & Chellappa, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01283-6 · Cited by 68 · see citing works
Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
Cao, C., Sang, J., Arora, R., Chen, D., Kloosterman, R., Cecere, M., Gorla, J., Saleh, R., Drennan, I., Teja, B., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-24-02189 · Cited by 67 · see citing works
“MedPrompt (Microsoft) (11), a collection of prompting techniques to optimize GPT4 performance on medical benchmarks, has shown that well-prompted generalist models can surpass fi ne-tuning methods.”
Ke, Y., Yang, R., Lie, S. A., Lim, T. X. Y., Ning, Y., Li, I., Abdullah, H. R., Ting, D. S. W., & Liu, N. (2024) · Journal of Medical Internet Research · doi:10.2196/59439 · Cited by 64 · see citing works
“LLMs have advanced text generation capability and extensive domain-specific knowledge [8,9].”
Small language models learn enhanced reasoning skills from medical textbooks
Kim, H., Hwang, H., Lee, J.-W., Park, S., Kim, D. R., Lee, T., Yoon, C., Sohn, J., Park, J., Reykhart, O., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01653-8 · Cited by 82 · see citing works
A personal health large language model for sleep and fitness coaching
Khasentino, J., Belyaeva, A., Liu, X., Yang, Z., Furlotte, N. A., Lee, C., Schenck, E., Patel, Y., Cui, J. H., Schneider, L. D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03888-0 · Cited by 104 · see citing works
Ong, J. C. L., Jin, L., Elangovan, K., Lim, G. Y. S., Lim, D. Y. Z., Sng, G. G. R., Ke, Y. H., Tung, J. Y. M., Zhong, R. J., Koh, C. M. Y., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102323 · Cited by 58 · see citing works
The Persuasive Power of Large Language Models
Breum, S. M., Egdal, D. V., Gram Mortensen, V., Møller, A. G., & Aiello, L. M. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31304 · Cited by 57 · see citing works
HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?
Ji, S., Zheng, X., & Wu, C. (2024) · 2024 IEEE International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys) · doi:10.1109/fmsys62467.2024.00011 · Cited by 56 · see citing works
Systematic review: The use of large language models as medical chatbots in digestive diseases
Giuffrè, M., Kresevic, S., You, K., Dupont, J., Huebner, J., Grimshaw, A. A., & Shung, D. L. (2024) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.18058 · Cited by 59 · see citing works
From Intention To Implementation: Automating Biomedical Research via LLMs
Luo, Y., Shi, L., Li, Y., Zhuang, A., Gong, Y., Liu, L., & Lin, C. (2024) · arXiv · doi:10.48550/arxiv.2412.09429 · Cited by 53
Self-citationFrom intention to implementation: automating biomedical research via LLMs
Luo, Y., Shi, L., Li, Y., Zhuang, A., Gong, Y., Liu, L., & Chen, L. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4485-0 · Cited by 56 · see citing works
Self-citationGupta, S., Basu, A., Nievas, M., Thomas, J., Wolfrath, N., Ramamurthi, A., Taylor, B., Kothari, A. N., Schwind, R., Miller, T. M., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01274-7 · Cited by 52 · see citing works
Gwon, Y. N., Kim, J. H., Chung, H. S., Jung, E., Chun, M. G., Lee, S., & Shim, S. R. (2024) · JMIR Medical Informatics · doi:10.2196/51187 · Cited by 61 · see citing works
Large language model agents can use tools to perform clinical calculations
Goodell, A. J., Chu, S. N., Rouholiman, D., & Chu, L. F. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01475-8 · Cited by 48 · see citing works
Verlingue, L., Boyer, C., Olgiati, L., Brutti-Mairesse, C., Morel, D., & Blay, J. (2024) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2024.101064 · Cited by 50 · see citing works
McCoy, L. G., Ng, F. Y. C., Sauer, C. M., Legaspi, K. E. Y., Jain, B., Gallifant, J., McClurkin, M., Hammond, A., Goode, D., Gichoya, J. W., et al. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06048-z · Cited by 48 · see citing works
Sevgi, M., Antaki, F., & Keane, P. A. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-325046 · Cited by 40 · see citing works
Influence of a Large Language Model on Diagnostic Reasoning: A Randomized Clinical Vignette Study
Goh, E., Gallo, R., Hom, J., Strong, E., Weng, Y., Kerman, H., Cool, J., Kanjee, Z., Parsons, A. S., Ahuja, N., et al. (2024) · doi:10.1101/2024.03.12.24303785 · Cited by 38 · see citing works
Self-citationGenerating dermatopathology reports from gigapixel whole slide images with HistoGPT
Tran, M., Schmidle, P., Guo, R. R., Wagner, S. J., Koch, V., Lupperger, V., Novotny, B., Murphree, D. H., Hardway, H. D., D’Amato, M., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-60014-x · Cited by 36 · see citing works
“The fi xed-size outputs of dimension 640×1536 are then used as keys and values in the tanh-gated cross-attention block (XATTN).”
Payne, D. L., Purohit, K., Borrero, W. M., Chung, K., Hao, M., Mpoy, M., Jin, M. X., Prasanna, P., & Hill, V. B. (2024) · Academic Radiology · doi:10.1016/j.acra.2024.04.006 · Cited by 33 · see citing works
Jeon, S., & Kim, H. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110614 · Cited by 34 · see citing works
Promise and Perils of Large Language Models for Cancer Survivorship and Supportive Care
Bitterman, D. S., Downing, A., Maués, J., & Lustberg, M. B. (2024) · Journal of Clinical Oncology · doi:10.1200/jco.23.02439 · Cited by 28 · see citing works
Leveraging large language models for structured information extraction from pathology reports
Balasubramanian, J. B., Adams, D., Roxanis, I., de González, A. B., Coulson, P., Almeida, J. S., & García‐Closas, M. (2025) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2025.100521 · Cited by 31 · see citing works
Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions
Chen, H., Fang, Z., Singla, Y., & Dredze, M. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.182 · Cited by 31 · see citing works
Gu, Z., Jia, W., Piccardi, M., & Yu, P. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103078 · Cited by 27 · see citing works
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · arXiv · doi:10.48550/arxiv.2402.13178 · Cited by 312 · see citing works
Combining Multiple Large Language Models Improves Diagnostic Accuracy
Barabucci, G., Shia, V., Chu, E., Harack, B., Laskowski, K., & Fu, N. (2024) · NEJM AI · doi:10.1056/aics2400502 · Cited by 24 · see citing works
Maximising Large Language Model Utility in Cardiovascular Care: A Practical Guide
Nolin-Lapalme, A., Thériault-Lauzier, P., Corbin, D., Tastet, O., Sharma, A., Hussin, J., Kadoury, S., Jiang, R., Krahn, A. D., Gallo, R., et al. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.05.024 · Cited by 28 · see citing works
Sorich, M. J., Mangoni, A. A., Bacchi, S., Menz, B. D., & Hopkins, A. M. (2024) · Journal of Medical Internet Research · doi:10.2196/67409 · Cited by 26 · see citing works
Biomedical Natural Language Processing in the Era of Large Language Models
Usuyama, N., Wong, C., Zhang, S., Naumann, T., & Poon, H. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095406 · Cited by 21 · see citing works
Self-citationArgMed-Agents: Explainable Clinical Decision Reasoning with LLM Disscusion via Argumentation Schemes
Hong, S., Xiao, L., Zhang, X., & Chen, J. (2024) · 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822109 · Cited by 21 · see citing works
Taming Unleashed Large Language Models With Blockchain for Massive Personalized Reliable Healthcare
Sun, L., Liu, D., Wang, M., Han, Y., Zhang, Y., Zhou, B., Ren, Y. C., & zhu, P. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3528526 · Cited by 22 · see citing works
Large Language Models in Critical Care Medicine: Scoping Review
Shi, T., Ma, J., Yu, Z., Xu, H., Yang, R., Xiong, M., Xiao, M., Li, Y., Zhao, H., & Kong, G. (2025) · JMIR Medical Informatics · doi:10.2196/76326 · Cited by 21 · see citing works
Self-citationRetrieving Evidence from EHRs with LLMs: Possibilities and Challenges
Ahsan, H., McInerney, D. J., Kim, J., Potter, C., Young, G. S., Amir, S., & Wallace, B. (2023) · PubMed · doi:10.48550/arxiv.2309.04550 · Cited by 21 · see citing works
Crimson: Empowering Strategic Reasoning in Cybersecurity through Large Language Models
Jin, J., Tang, B., Ma, M., Liu, X., Wang, Y., Lai, Q., Yang, J., & Zhou, C. (2024) · International Conference on Computer, Big Data and Artificial Intelligence (ICCBD+AI) · doi:10.1109/iccbd-ai65562.2024.00011 · Cited by 18 · see citing works
Easing the cognitive load of general practitioners: AI design principles for future-ready healthcare
Hor, T., Fong, L., Wynne, K., & Verhoeven, B. (2025) · Technovation · doi:10.1016/j.technovation.2025.103208 · Cited by 17 · see citing works
A perspective for adapting generalist AI to specialized medical AI applications and their challenges
Wang, Z., Wang, H., Danek, B. P., Li, Y., Mack, C. D., Arbuckle, L., Biswal, D., Poon, H., Wang, Y., Rajpurkar, P., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01789-7 · Cited by 25 · see citing works
Self-citationLarge Language Models Are Poor Clinical Decision-Makers: A Comprehensive BenchmarkPreprint
Liu, F., Li, Z., Zhou, H., Yin, Q., Yang, J., Tang, X., Luo, C., Zeng, M. Y., Jiang, H., Gao, Y., et al. (2024) · medRxiv · doi:10.1101/2024.04.24.24306315 · Cited by 16 · see citing works
Han, T., Jeong, W. K., & Shin, J. (2025) · ULTRASONOGRAPHY · doi:10.14366/usg.25012 · Cited by 16 · see citing works
Kufel, W. D., Hanrahan, K. D., Seabury, R. W., Parsels, K. A., Gallagher, J. C., MacDougall, C., Covington, E. W., Chahine, E. B., Britt, R. S., & Steele, J. M. (2024) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofae641 · Cited by 16 · see citing works
Towards Medical Complex Reasoning with LLMs through Medical Verifiable Problems
Chen, J., Cai, Z., Ji, K., Wang, X., Liu, W., Wang, R., & Wang, B. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.751 · Cited by 14 · see citing works
Self-citationAI chatbots in higher education: Students’ beliefs and concerns
Fošner, A., & Aver, B. (2025) · Sustainable Futures · doi:10.1016/j.sftr.2025.100734 · Cited by 15 · see citing works
Large Language Model Symptom Identification From Clinical Text: Multicenter Study
McMurry, A., Phelan, D., Dixon, B. E., Geva, A., Gottlieb, D., Jones, J. R., Terry, M., Taylor, D. E., Callaway, H., Manoharan, S., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72984 · Cited by 20 · see citing works
Scaling medical AI across clinical contexts
Li, M. M., Reis, B. Y., Rodman, A. M., Cai, T., Dagan, N., Balicer, R. D., Loscalzo, J., Kohane, I. S., & Žitnik, M. (2026) · Nature Medicine · doi:10.1038/s41591-025-04184-7 · Cited by 17 · see citing works
Pugh, S. L., Chandler, C., Cohen, A. S., Diaz-Asper, C. M., Elvevåg, B., & Foltz, P. W. (2024) · Psychiatry Research · doi:10.1016/j.psychres.2024.116119 · Cited by 17 · see citing works
Altermatt, F. R., Neyem, A., Sumonte, N. I., Mendoza, M., Villagrán, I., & Lacassie, H. J. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07250-3 · Cited by 15 · see citing works
Hersh, W., & Hollis, K. F. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01251-0 · Cited by 14 · see citing works
McCaffrey, P., Jackups, R., Seheult, J. N., Zaydman, M. A., Balis, U. J., Thaker, H. M., Rashidi, H. H., & Gullapalli, R. R. (2024) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0208-ra · Cited by 13 · see citing works
Sushil, M., Zack, T., Mandair, D., Zheng, Z., Wali, A., Yu, Y.-N., Quan, Y., & Butte, A. (2024) · Research Square · doi:10.21203/rs.3.rs-3914899/v1 · Cited by 10 · see citing works
Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial
Goh, E., Gallo, R., Strong, E., Weng, Y., Kerman, H., Freed, J., Cool, J. A., Kanjee, Z., Lane, K. P., Parsons, A. S., et al. (2024) · medRxiv · doi:10.1101/2024.08.05.24311485 · Cited by 13 · see citing works
“22,29 This study found that the addition of LLM AI assistance improves physician management reasoning compared to conventional resources.”
BackgroundSelf-citationTextGrad: Automatic "Differentiation" via Text
Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., & Zou, J. (2024) · arXiv · doi:10.48550/arxiv.2406.07496 · Cited by 12 · see citing works
Hamane, Z., Ben-Bouazza, F.-E., & Fennan, A. (2024) · International Conference on Intelligent Systems and Computer Vision (ISCV) · doi:10.1109/iscv60512.2024.10620139 · Cited by 13 · see citing works
Wang, X., Ye, S., Feng, J., Feng, K., Yang, H., & Li, H. (2024) · Digital Health · doi:10.1177/20552076241297127 · Cited by 12 · see citing works
Do Multimodal Large Language Models Understand Welding?
Khvatskii, G., Lee, Y.-S., Angst, C., Gibbs, M., Landers, R. K., & Chawla, N. V. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103121 · Cited by 10 · see citing works
Self-citationAgentClinic: a multimodal benchmark for tool-using clinical AI agents
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02674-7 · Cited by 11 · see citing works
“Additionally, specialist cases sourced from MedMCQA exhibited higher average accuracy compared to non-specialist casesfromMedQA,whichdiffersfromreportedmultiplechoiceevaluations wherespecialistQAstypicallyhavelowerperformance 3 .”
InfluentialMensah, P. B., Quao, N. S. A., Dagadu, S., Mensah, J. K., Darkwah, J. D., & Clinician, P. E. G. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00093 · Cited by 11 · see citing works
Kornblith, A. E., Singh, C., Innes, J. C., Chang, T. P., Adelgais, K., Holsti, M., Kim, J., McClain, B., Nishijima, D. K., Rodgers, S., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-89996-w · Cited by 11 · see citing works
Sarma, K. V., Hanss, K. E., Halls, A. J. M., Krystal, A. D., Becker, D. F., Glowinski, A. L., & Butte, A. J. (2025) · Psychiatry Research · doi:10.1016/j.psychres.2025.116844 · Cited by 10 · see citing works
Investigating the Predictive Potential of Large Language Models in Student Dropout Prediction
Aboukacem, A., Berrada, I., Bergou, E. H., Iraqi, Y., & Mekouar, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64299-9_34 · Cited by 10 · see citing works
Effects of Different Prompts on the Quality of GPT-4 Responses to Dementia Care Questions
Li, Z., Xie, B., Hilsabeck, R. C., Aguirre, A., Zou, N., Luo, Z., & He, D. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00059 · Cited by 10 · see citing works
Evaluating the effectiveness of prompt engineering for knowledge graph question answering
Kosten, C., Nooralahzadeh, F., & Stockinger, K. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1454258 · Cited by 10 · see citing works
Advancing medical question answering with a knowledge embedding transformer
Zhu, X., Khan, M., Ahmed, A. T., & Othmani, A. (2025) · PLoS ONE · doi:10.1371/journal.pone.0329606 · Cited by 10 · see citing works
Multimodal Document Analytics for Banking Process Automation
Gerling, C., & Lessmann, S. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102973 · Cited by 21 · see citing works
Evaluating General Vision-Language Models for Clinical MedicinePreprint
Jiang, Y., Omiye, J. A., Zakka, C., Moor, M., Gui, H., Alipour, S., Mousavi, S. S., Chen, J. H., Rajpurkar, P., & Daneshjou, R. (2024) · medRxiv · doi:10.1101/2024.04.12.24305744 · Cited by 11 · see citing works
Kämmer, J. E., Hautz, W. E., Krummrey, G., Sauter, T. C., Penders, D., Birrenbach, T., & Bienefeld, N. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-087469 · Cited by 9 · see citing works
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 9 · see citing works
Evaluation of a novel large language model (LLM)-powered chatbot for oral boards scenarios
Silvestri, C. J., Roshal, J. A., Shah, M., Widmann, W. D., Townsend, C. M., Brian, R., L’Huillier, J. C., Navarro, S. M., Lund, S. B., & Sathe, T. S. (2024) · Global Surgical Education - Journal of the Association for Surgical Education · doi:10.1007/s44186-024-00303-z · Cited by 13 · see citing works
Gemini-the most powerful LLM: Myth or TruthPreprint
Ahmed, I., & Islam, R. (2024) · doi:10.36227/techrxiv.171177477.70151414/v1 · Cited by 9 · see citing works
Liu, B., Nie, Y., Song, H., Lin, Y., Li, J., Weng, X., Su, Z., Suo, Y., Lv, T., Zhao, X., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103557 · Cited by 8 · see citing works
AFlow: Automating Agentic Workflow Generation
Zhang, J., Xiang, J., Yu, Z., Teng, F., Chen, X., Chen, J., Zhuge, M., Cheng, X., Hong, S., Wang, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.10762 · Cited by 7 · see citing works
“…problems, such as code generation (Hong et al., 2024b; Ridnik et al., 2024; Zhong et al., 2024a), data analysis (Xie et al., 2024b; Ye et al., 2024; Li et al., 2024a; Zhou et al., 2023), mathematics (Zhong et al., 2024b; Xu et al., 2024), question answering (Nori et al., 2023; Zhou et al., 2024a).”
Subramaniam, S., Rizvi, S., Ramesh, R., Sehgal, V., Gurusamy, B., Arif, H., Tran, J. S., Thamman, R., Anyanwu, E. C., Mastouri, R. A., et al. (2025) · Scientific Reports · doi:10.1038/s41598-024-83540-y · Cited by 9 · see citing works
Xiao, X., Chen, P., Qi, B., Liang, J., Tong, J., & Wang, H. (2025) · Progress in Nuclear Energy · doi:10.1016/j.pnucene.2025.105908 · Cited by 7 · see citing works
Medication information extraction using local large language models
Richter-Pechanski, P., Seiferling, M., Kiriakou, C., Schwab, D. M., Geis, N. A., Dieterich, C., & Frank, A. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104898 · Cited by 9 · see citing works
- Authors
- Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., Luo, R., McKinney, S. M., Ness, R. O., Poon, H., Qin, T., Usuyama, N., White, C., & Horvitz, E.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2311.16452
- OpenAlex
- W4389156617
- Cited by
- 178
- References
- 0
- Citations by year
- 2026: 16 · 2025: 67 · 2024: 93 · 2023: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- GPT-4, prompt engineering, medical question answering, fine-tuned models, overfitting, domain generalization, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
