Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Open accessPublished version of a preprint
The DOI matches its Crossref record, with a small difference in title or year; 28 of the 28 references it cites were found in a registry or an index.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Kung et al. (2023) evaluated the performance of ChatGPT on the United States Medical Licensing Exam and demonstrated that the chatbot did not reach the passing threshold and achieved an overall accuracy of only 50% (Kung et al., 2023).”
From A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing (2025) · cites this paper as Kung et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [17]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI's capabilities, such as passing the United States Medical Licensing Exam [107], completing exams at the University of Minnesota Law School [108], and solving queries from Wharton School of Business tests [109], underscore its potential.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [107]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The HCI community has explored the application of LLMs for various healthcare practices, such as mental health support [66,68], health information seeking [81,117,128], and public health interventions [58]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
From DIALOGUE: A Generative AI-Based Pre–Post Simulation Study to Enhance Diagnostic Communication in Medical Students Through Virtual Type 2 Diabetes Scenarios (2025) · cites this paper as Kung et al., 2023
Experimental — accuracy not yet measured
52 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background
Cited by
Items 1,001–1,100 of 1,129
Khan, N. A., Osmonaliev, K., & Sarwar, M. Z. (2023) · Nepal Journal of Epidemiology · doi:10.3126/nje.v13i1.53721 · Cited by 37
Doo, F. X., Cook, T. S., Siegel, E. L., Joshi, A., Parekh, V. S., Elahi, A., & Yi, P. H. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.07.007 · Cited by 36
ChatGPT: Its Applications and Limitations
Chowdhury, M. N.-U.-R., & Haque, A. (2023) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit59222.2023.10205621 · Cited by 41
“Doctors can input symptoms into the model, and it can provide a list of potential diagnoses based on the input [11].”
BackgroundMethodMannam, S. S., Subtirelu, R. C., Chauhan, D., Ahmad, H. S., Matache, I., Bryan, K., Chitta, S., Bathula, S. C., Turlip, R., Wathen, C. A., et al. (2023) · World Neurosurgery · doi:10.1016/j.wneu.2023.10.043 · Cited by 32
Ono, D., Dickson, D. W., & Koga, S. (2024) · Neuropathology and Applied Neurobiology · doi:10.1111/nan.12997 · Cited by 39
“It has not only passed the United States Medical Licensing Examination but has also efficiently addressed subspecialty board exam questions and generated comprehensive differential diagnosis lists from clinical descriptions [2 – 5].”
Fear, K., & Gleber, C. D. (2023) · JMIR Aging · doi:10.2196/51776 · Cited by 32
Ravi, A., Neinstein, A., & Murray, S. G. (2023) · ATS Scholar · doi:10.34197/ats-scholar.2023-0036ps · Cited by 31
Morjaria, L., Burns, L., Bracken, K., Ngo, Q. N., Lee, M., Levinson, A. J., Smith, J., Thompson, P., & Sibbald, M. (2023) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205231204178 · Cited by 31
“8,9 While licensing examinations in Canada and the United States are in the multiple-choice format, assessment throughout the medical curriculum includes other formats.”
BackgroundIncorporating AI tools into medical education: harnessing the benefits of ChatGPT and Dall-E
Miftahul Amri, M., & Khairatun Hisan, U. (2023) · Journal of Novel Engineering Science and Technology · doi:10.56741/jnest.v2i02.315 · Cited by 31
“Currently, ChatGPT has been reported to be able to answer medical questions of Korean Parasitology Exam [18], US Medical Licensing Exam [7], and Indonesian”
BackgroundInfluentialLin, Y., Luo, Z., Ye, Z., Zhong, N., Zhao, L., Zhang, L., Li, X., Chen, Z., & Chen, Y. (2025) · JMIR Medical Education · doi:10.2196/71125 · Cited by 58
“In addition, the cutoff date for the training data means that GAI may lack knowledge of the latest research, leading to outdated or inaccurate recommendations [26,32,41,66,67,74,80,89,92,94,109, 129,136,141,144].”
Assessment of Large Language Models (LLMs) in Decision-Making Support for Gynecologic Oncology
Gumilar, K. E., Indraprasta, B. R., Faridzi, A. S., Wibowo, B. M., Herlambang, A., Rahestyningtyas, E., Irawan, B., Tambunan, Z., Bustomi, A. F., Brahmantara, B. N., et al. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.10.050 · Cited by 31
“Prior studies have shown that LLMs can effectively comprehend and reply to certain medical queries [17,18], and their performance varies greatly depending on the intricacy of the questions and the degree of medical expertise required [19 – 21].”
Performance of ChatGPT in Board Examinations for Specialists in the Japanese Ophthalmology Society
Sakai, D., Maeda, T., Ozaki, A., Kanda, G. N., Kurimoto, Y., & Takahashi, M. (2023) · Cureus · doi:10.7759/cureus.49903 · Cited by 41
Large language model trained on clinical oncology data predicts cancer progression
Zhu, M., Lin, H., Jiang, J., Jinia, A. J., Jee, J., Pichotta, K., Waters, M., Rose, D., Schultz, N., Chalise, S., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01780-2 · Cited by 30
Comparative analysis of NCLEX-RN questions: a duel between ChatGPT and human expertise
Cox, R. L., Hunt, K. L., & Hill, R. R. (2023) · Journal of Nursing Education · doi:10.3928/01484834-20231006-07 · Cited by 30
Rethinking assessment in response to generative artificial intelligence
Pearce, J., & Chiavaroli, N. (2023) · Medical Education · doi:10.1111/medu.15092 · Cited by 30
Language models for materials discovery and sustainability: Progress, challenges, and opportunities
Pei, Z., Yin, J., & Zhang, J. (2025) · Progress in Materials Science · doi:10.1016/j.pmatsci.2025.101495 · Cited by 30
Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review
Hua, Y., Xia, W., Bates, D., Hartstein, G. L., Kim, H. T., Li, M., Nelson, B. W., Stromeyer IV, C., King, D., Suh, J., et al. (2025) · JMIR AI · doi:10.2196/69006 · Cited by 30
Ajalo, E., Mukunya, D., Nantale, R., Kayemba, F., Pangholi, K., Babuya, J., Langoya Akuu, S., Namiiro, A. M., Nsubuga, Y. B., Mpagi, J. L., et al. (2025) · PLOS ONE · doi:10.1371/journal.pone.0313776 · Cited by 29
“Similarly, a study by Kung et al . showed that ChatGPT passes the United States Medical Licensing Exam (USMLE) with a score in the 60th percentile [6].”
Mediboina, A., Badam, R. K., & Chodavarapu, S. (2024) · Cureus · doi:10.7759/cureus.51544 · Cited by 29
“…variant as ChatGPT 3.5, has been observed to approach or achieve the passing threshold for the United States Medical Licensing Exam (USMLE) without specialized training, thereby emphasizing the potential applications of these chatbots in medical education and clinical decision making [3].”
BackgroundInfluentialGalido, P. V., Butala, S., Chakerian, M., & Agustines, D. (2023) · Cureus · doi:10.7759/cureus.38166 · Cited by 29
“Not only that, but it can also provide concise treatment recommendations, offer pharmacologic and nonpharmacologic treatments, and identify side effects of medications based on current standards of care and published literature [2,3,8,17].”
BackgroundChatGPT Is Moderately Accurate in Providing a General Overview of Orthopaedic Conditions
Sparks, C. A., Fasulo, S. M., Windsor, J. T., Bankauskas, V., Contrada, E. V., Kraeutler, M. J., & Scillia, A. J. (2024) · JBJS Open Access · doi:10.2106/jbjs.oa.23.00129 · Cited by 29
Huo, B., Calabrese, E., Sylla, P., Kumar, S., Ignacio, R. C., Oviedo, R., Hassan, I., Slater, B. J., Kaiser, A., Walsh, D. S., et al. (2024) · Surgical Endoscopy · doi:10.1007/s00464-024-10807-w · Cited by 28
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28
Chen, J., Cadiente, A., Kasselman, L. J., & Pilkington, B. (2024) · Journal of Medical Ethics · doi:10.1136/jme-2023-109366 · Cited by 28
Assessing Racial and Ethnic Bias in Text Generation for Healthcare-Related Tasks by ChatGPT 1
Hanna, J. J., Wakene, A. D., Lehmann, C. U., & Medford, R. J. (2023) · doi:10.1101/2023.08.28.23294730 · Cited by 28
The feasibility of using generative artificial intelligence for history taking in virtual patients
Yi, Y., & Kim, K.-J. (2025) · BMC Research Notes · doi:10.1186/s13104-025-07157-8 · Cited by 28
Cao, J. J., Kwon, D. H., Ghaziani, T. T., Kwo, P., Tse, G., Kesselman, A., Kamaya, A., & Tse, J. R. (2024) · Abdominal Radiology · doi:10.1007/s00261-024-04501-7 · Cited by 28
“Interestingly, Kung et al. found ChatGPT to approach passing performance when responding to USMLE questions in an open-ended format, akin to the natural user queries used in this study and other patient education studies [15].”
Meddeb, A., Ebert, P., Bressem, K. K., Desser, D., Dell'Orco, A., Bohner, G., Kleine, J. F., Siebert, E., Grauhan, N., Brockmann, M. A., et al. (2025) · Journal of NeuroInterventional Surgery · doi:10.1136/jnis-2024-022078 · Cited by 27
“3 These models, trained on vast datasets, have shown remarkable clinical reasoning capabilities 4–6 in passing medical licensing examinations 7 8 and generating prevention and treatment recommendations for various conditions including cardiovascular disease 9 10 and breast cancer.”
Performance of ChatGPT on factual knowledge questions regarding clinical pharmacy
van Nuland, M., Erdogan, A., Aςar, C., Contrucci, R., Hilbrants, S., Maanach, L., Egberts, T., & van der Linden, P. D. (2024) · The Journal of Clinical Pharmacology · doi:10.1002/jcph.2443 · Cited by 27
Yadava, O. P. (2023) · Indian Journal of Thoracic and Cardiovascular Surgery · doi:10.1007/s12055-023-01507-6 · Cited by 27
Transforming Maritime Health with ChatGPT-Powered Healthcare Services for Mariners
Sharma, M., & Sharma, S. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03195-0 · Cited by 27
“ChatGPT is empowered by an LLM (large language model) that enables it to comprehend human language and deliver solutions based on substantial corpora of data [4, 10].”
BackgroundHarnessing the power of ChatGPT in medical education
Guo, A. A., & Li, J. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2198094 · Cited by 27
Gandhi Periaysamy, A., Satapathy, P., Neyazi, A., & Padhi, B. K. (2023) · Annals of Medicine & Surgery · doi:10.1097/ms9.0000000000000371 · Cited by 27
“Research articles have been published as peer-reviewed articles [4] and preprints [2] with ChatGPT as one of the co-authors.”
BackgroundMethodWhat is the student experience of remote proctoring? A pragmatic scoping review
Marano, E., Newton, P. M., Birch, Z., Croombs, M., Gilbert, C., & Draper, M. J. (2024) · Higher Education Quarterly · doi:10.1111/hequ.12506 · Cited by 27
Large language models as a diagnostic support tool in neuropathology
Hewitt, K. J., Wiest, I. C., Carrero, Z. I., Bejan, L., Millner, T. O., Brandner, S., & Kather, J. N. (2024) · The Journal of Pathology: Clinical Research · doi:10.1002/2056-4538.70009 · Cited by 27
“Large language models (LLMs) have shown promising performance on several healthcare-related tasks including medical education [ 1,2], and administrative tasks such as writing hospital discharge summaries [3] and clinic letters [4] for cancer patients in a variety of scenarios.”
BackgroundChatgpt in developing economies
Kshetri, N. (2023) · IT Professional · doi:10.1109/mitp.2023.3254639 · Cited by 27
AI chatbots: A disguised enemy for academic integrity?
Niloy, A. C., Hafiz, R., Hossain, B. M., Gulmeher, F., Sultana, N., Islam, K. F., Bushra, F., Islam, S., Hoque, S. I., Rahman, M., et al. (2024) · International Journal of Educational Research Open · doi:10.1016/j.ijedro.2024.100396 · Cited by 26
Zhang, M., & Crosthwaite, P. (2026) · International Review of Applied Linguistics in Language Teaching · doi:10.1515/iral-2024-0196 · Cited by 26
“Due to its extensive training data, ChatGPT can provide instant feedback on grammar, lexis and coherence, easily extract key concepts, and even o ff er relevant citations (Kung et al. 2023).”
Flaharty, K. A., Hu, P., Hanchard, S. L., Ripper, M. E., Duong, D., Waikel, R. L., & Solomon, B. D. (2024) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2024.07.011 · Cited by 26
Sriwastwa, A., Ravi, P., Emmert, A., Chokshi, S., Kondor, S., Dhal, K., Patel, P., Chepelev, L. L., Rybicki, F. J., & Gupta, R. (2023) · 3D Printing in Medicine · doi:10.1186/s41205-023-00186-8 · Cited by 28
“For example, ChatGPT has been applied to the United States Medical Licensing Examination (USMLE) [30] and has been proven to pass the American Board of Radiology (ABR) specialty examinations [31].”
MethodPublic attitudes toward chatgpt on twitter: sentiments, topics, and occupations
Koonchanok, R., Pan, Y., & Jang, H. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01260-7 · Cited by 25
Mind + Machine: ChatGPT as a Basic Clinical Decisions Support Tool
Ayoub, M., Ballout, A. A., Zayek, R. A., & Ayoub, N. F. (2023) · Cureus · doi:10.7759/cureus.43690 · Cited by 25
“[1] Recent studies have shown GPT’s potential in answering United States Medical Licensing Exam questions, responding to preventive medicine questions, and outperforming physicians in empathy [2-5].”
BackgroundAlbogami, Y., Alfakhri, A., Alaqil, A., Alkoraishi, A., Alshammari, H., Elsharawy, Y., Alhammad, A., & Alhossan, A. (2024) · DIGITAL HEALTH · doi:10.1177/20552076241253523 · Cited by 25
Danish university policies on generative AI: Problems, assumptions and sustainability blind spots
Driessens, O., & Pischetola, M. (2024) · MedieKultur: Journal of media and communication research · doi:10.7146/mk.v40i76.143595 · Cited by 25
Yan, Z., Fan, K.-Q., Zhang, Q., Wu, X., Chen, Y., Wu, X., Yu, T., Su, N., Zou, Y., Chi, H., et al. (2025) · World Journal of Urology · doi:10.1007/s00345-025-05757-4 · Cited by 24
Luke, W. A. N. V., Seow Chong, L., Ban, K. H., Wong, A. H., Zhi Xiong, C., Shuh Shing, L., Taneja, R., Samarasekera, D. D., & Yap, C. T. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2308779 · Cited by 24
Exploring the role of ChatGPT in oncology: providing information and support for cancer patients
Cè, M., Chiarpenello, V., Bubba, A., Felisaz, P. F., Oliva, G., Irmici, G., & Cellina, M. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020049 · Cited by 24
“LLMs are already active in different areas of clinical practice and can generate differential diagnosis lists for typical clinical scenarios with good diagnostic accuracy [12].”
BackgroundTing, Y.-T., Hsieh, T.-C., Wang, Y.-F., Kuo, Y.-C., Chen, Y.-J., Chan, P.-K., & Kao, C.-H. (2024) · DIGITAL HEALTH · doi:10.1177/20552076231224074 · Cited by 24
Performance of ChatGPT on Chinese master’s degree entrance examination in clinical medicine
Li, K.-C., Bu, Z.-J., Shahjalal, M., He, B.-X., Zhuang, Z.-F., Li, C., Liu, J.-P., Wang, B., & Liu, Z.-L. (2024) · PLOS ONE · doi:10.1371/journal.pone.0301702 · Cited by 23
Educating patients on osteoporosis and bone health: Can “ChatGPT” provide high-quality content?
Ghanem, D., Shu, H., Bergstein, V., Marrache, M., Love, A., Hughes, A., Sotsky, R., & Shafiq, B. (2024) · European Journal of Orthopaedic Surgery & Traumatology · doi:10.1007/s00590-024-03990-y · Cited by 23
The Two Word Test: A Semantic Benchmark for Large Language Models
Riccardi, N., & Desai, R. H. (2023) · arXiv · doi:10.48550/arxiv.2306.04610 · Cited by 23
Perceptions and Use of Generative Artificial Intelligence in Medical Students: A Multicenter Survey
Tran, C., Hryciw, B. N., Moore, S. W., Chaput, A., & Seely, A. J. E. (2025) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205251391969 · Cited by 22
Guthrie, E., Levy, D., & del Carmen, G. (2024) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2024.02.016 · Cited by 22
Wang, J., Hu, H., Wang, Z., Yan, S., Sheng, Y., & He, D. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641917 · Cited by 21
How Can ChatGPT Benefit Pharmacy: A Case Report on Review Writing
Zhu, Y., Han, D., Chen, S., Zeng, F., & Wang, C. (2023) · doi:10.20944/preprints202302.0324.v1 · Cited by 21
Arbel, Y., Gimmon, Y., & Shmueli, L. (2025) · Physical Therapy · doi:10.1093/ptj/pzaf010 · Cited by 21
Unleashing ChatGPT's impact in higher education: Student and faculty perspectives
Rajabi, P., Taghipour, P., Cukierman, D., & Doleck, T. (2024) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2024.100090 · Cited by 21
Strasser, L. M., Anschuetz, W., Dennstädt, F., & Hastings, J. (2026) · JMIR Medical Education · doi:10.2196/81399 · Cited by 20
“Although recent work has shown substantial performance gains in vision-enabled large multimodal models, incorporating visual items introduces additional variance unrelated to linguistic processing [13,21-24].”
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
Balloccu, S., Schmidtová, P., Lango, M., & Dušek, O. (2024) · arXiv · doi:10.48550/arxiv.2402.03927 · Cited by 20
ChatGPT for low- and middle-income countries: a Greek gift?
Lam, K. (2023) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2023.100906 · Cited by 18
Teng, D., Tan, L., Cao, Q., Xia, Y., Zhang, N., Li, J., & Zhao, D. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02547-z · Cited by 18
Cherif, H., Moussa, C., Missaoui, A. M., Salouage, I., Mokaddem, S., & Dhahri, B. (2024) · JMIR Medical Education · doi:10.2196/52818 · Cited by 18
“This is different from past research where ChatGPT consistently demonstrates strong performance in English-language medical assessments like the United States Medical Licensing Examination, CBME evaluations, and the European Examination in Core Cardiology [17,25,28].”
ResultSocrates, V., Wright, D. S., Huang, T., Fereydooni, S., Dien, C., Chi, L., Albano, J., Patterson, B., Sasidhar Kanaparthy, N., Wright, C. X., et al. (2025) · JMIR Aging · doi:10.2196/69504 · Cited by 17
LAB-Bench: Measuring Capabilities of Language Models for Biology Research
Laurent, J. M., Janizek, J. D., Ruzo, M., Hinks, M. M., Hammerling, M. J., Narayanan, S., Ponnapati, M., White, A. D., & Rodriques, S. G. (2024) · arXiv · doi:10.48550/arxiv.2407.10362 · Cited by 21
“…data (i.e. a huge fraction of the internet), and the scale of such training has allowed even general models to demonstrate emergent abilities to solve complex problems they were not explicitly trained for, including in medicine [25, 48], law [22], writing code [43], or other broad capabilities [7].”
BackgroundYoung, R. A., Martin, C. M., Sturmberg, J. P., Hall, S., Bazemore, A., Kakadiaris, I. A., & Lin, S. (2024) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2023.230219r1 · Cited by 17
“9 In health care, ChatGPT has successfully answered medical license test questions, 10 but ChatGPT has been found to produce errors in both stating facts and synthesizing data from the medical literature, 11 and there is now evidence emerging that the mass use of ChatGPT actually worsens its…”
van Nuland, M., Lobbezoo, A.-F. H., van de Garde, E. M. W., Herbrink, M., van Heijl, I., Bognàr, T., Houwen, J. P. A., Dekens, M., Wannet, D., Egberts, T., et al. (2024) · Exploratory Research in Clinical and Social Pharmacy · doi:10.1016/j.rcsop.2024.100464 · Cited by 17
Abu-Jeyyab, M., Alrosan, S., & Alkhawaldeh, I. (2023) · High Yield Medical Reviews · doi:10.59707/hymrfbya5348 · Cited by 17
Hueber, A. J., & Kleyer, A. (2023) · RMD Open · doi:10.1136/rmdopen-2023-003248 · Cited by 16
International AI Safety Report
Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Fox, P., Garfinkel, B., Goldfarb, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.17805 · Cited by 16
The role of ChatGPT in enhancing ENT surgical training – a trainees’ perspective
Brennan, L., Balakumar, R., & Bennett, W. (2024) · The Journal of Laryngology & Otology · doi:10.1017/s0022215123001354 · Cited by 16
Variation in Large Language Model Recommendations in Challenging Inpatient Management Scenarios
Landon, S., Savage, T., Greysen, S. R., & Bressman, E. (2026) · Journal of General Internal Medicine · doi:10.1007/s11606-025-09888-7 · Cited by 16
“Early evaluations of these models, such as performance on multiple-choice United States Medical Licensing Examination (USMLE) questions, 1 have given way to more complex testing scenarios that probe the ability of LLMs to apply clinical reasoning to open-ended case vignettes.”
Accuracy of large language models in answering ophthalmology board-style questions: A meta-analysis
Wu, J.-H., Nishida, T., & Liu, T. Y. A. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100106 · Cited by 16
Naing, S. Z. S., & Udomwong, P. (2024) · Data Intelligence · doi:10.1162/dint_a_00250 · Cited by 15
“Kung et al. [6] conducted an interesting study that presented new and surprising evidence of ChatGPT’s ability to perform on the United States Medical Licensing Examinations (USMLE) with high accuracy.”
BackgroundCarl, N., Nguyen, L., Haggenmüller, S., Joachim Hetz, M., Theres Winterstein, J., Otto Hartung, F., Gruene, B., Nikolas Kather, J., Holland-Letz, T., Stephan Michel, M., et al. (2024) · European Urology Open Science · doi:10.1016/j.euros.2024.10.009 · Cited by 15
Natural language processing to adjudicate heart failure hospitalizations in global clinical trials
Marti-Castellote, P. M., Reeder, C., Claggett, B., Singh, P., Lau, E. S., Khurshid, S., Batra, P., Lubitz, S. A., Maddah, M., Vardeny, O., et al. (2025) · Circulation: Heart Failure · doi:10.1161/circheartfailure.124.012514 · Cited by 14
Unveiling the risks of ChatGPT in diagnostic surgical pathology
Guastafierro, V., Corbitt, D. N., Bressan, A., Fernandes, B., Mintemur, Ö., Magnoli, F., Ronchi, S., La Rosa, S., Uccella, S., & Renne, S. L. (2025) · Virchows Archiv · doi:10.1007/s00428-024-03918-1 · Cited by 14
When Autonomy Breaks: The Hidden Existential Risk of AI
Krook, J. (2025) · AI & SOCIETY · doi:10.1007/s00146-025-02397-5 · Cited by 14
Can ChatGPT outperform a neurosurgical trainee? a prospective comparative study.
Williams, S. C., Starup-Hansen, J., Funnell, J. P., Hanrahan, J. G., Valetopoulou, A., Singh, N., Sinha, S., Muirhead, W. R., & Marcus, H. J. (2025) · British Journal of Neurosurgery · doi:10.1080/02688697.2024.2308222 · Cited by 13
ChatGPT -Boon or Bane: A Study from Students Perspective
Wagholikar, S., Chandani, A., Atiq, R., Pathak, M., & Wagholikar, O. (2023) · 2023 International Conference on Advancement in Computation & Computer Technologies (InCACCT) · doi:10.1109/incacct57535.2023.10141820 · Cited by 13
“The ChatGPT showed a remarkable performance, indicating that it could be used to assist medical students in their education and ascertaining clinical decisions [11].”
BackgroundYanagita, Y., Shikino, K., Ishizuka, K., Uchida, S., Li, Y., Yokokawa, D., Tsukamoto, T., Noda, K., Uehara, T., & Ikusaka, M. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04370-6 · Cited by 13
Artificial Intelligence: Impact and Challenges to Authors, Journals and Medical Publishing
WCG, P., & A, S. (2023) · Malaysian Orthopaedic Journal · doi:10.5704/moj.2311.001 · Cited by 12
Suárez-García, R. X., Chavez-Castañeda, Q., Orrico-Pérez, R., Valencia-Marin, S., Castañeda-Ramírez, A. E., Quiñones-Lara, E., Ramos-Cortés, C. A., Gaytán-Gómez, A. M., Cortés-Rodríguez, J., Jarquín-Ramírez, J., et al. (2025) · European Journal of Investigation in Health, Psychology and Education · doi:10.3390/ejihpe15080152 · Cited by 18
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
Ruan, M., Fan, J., Liu, M., Meng, Z., Zhang, X., & Zhang, C. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02644-9 · Cited by 12
“For example, a prompt such as “Please design the best prompt for me based on this prompt: …” was used in this step [26].”
Leon, M., Ruaengsri, C., Pelletier, G., Bethencourt, D., Shibata, M., Flores, M. Q., & Shudo, Y. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13216543 · Cited by 12
“These findings highlight ChatGPT’s potential as both an educational tool and a decision-support system in clinical settings [13].”
BackgroundQuantum leap in medical mentorship: exploring ChatGPT’s transition from textbooks to terabytes
Chokkakula, S., Chong, S., Yang, B., Jiang, H., Yu, J., Han, R., Attitalla, I. H., Yin, C., & Zhang, S. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1517981 · Cited by 12
Tong, L., Wang, J., Rapaka, S., & Garg, P. S. (2025) · Medical Teacher · doi:10.1080/0142159x.2024.2363486 · Cited by 12
“This surpasses previously reported performance in standardized medical examinations [6,9], likely reflecting technological advancements and differences between standardized and faculty-generated questions.”
Goodings, A. J., Kajitani, S., Chhor, A., Albakri, A., Pastrak, M., Kodancha, M., Ives, R., Lee, Y. B., & Kajitani, K. (2024) · JMIR Medical Education · doi:10.2196/56128 · Cited by 12
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17
“[35] 2023 ✓ ✗ Education Assess ChatGPT’s performance on the United States Medical Licensing Exam (USMLE), where it impressively achieved scores near the passing threshold without any dedicated specialised training.”
BackgroundThe application of ChatGPT in nursing: a bibliometric and visualized analysis
Wang, P., Zhang, Q., Zhang, W., & Sun, J. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1521712 · Cited by 12
Mackey, B. P., Garabet, R., Maule, L., Tadesse, A., Cross, J., & Weingarten, M. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00135-2 · Cited by 11
“Initial studies have demonstrated that ChatGPT-4 can process board-level exam questions and provide useful clinical insights [7–12].”
ChatGPT in Research and Education: A SWOT Analysis of Its Academic Impact
Miah, A. S. M., Tusher, M. M. R., Hossain, M. M., Hossain, M. M., Rahim, M. A., Hamid, M. E., Islam, M. S., & Shin, J. (2025) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2025.064168 · Cited by 11
“Several studies have shown that while ChatGPT can provide basic medical information, it often fails in areas requiring diagnostic reasoning, clinical decision-making, and up-to-date medical knowledge [53,61].”
Jaworski, A., Jasiński, D., Jaworski, W., Hop, A., Janek, A., Sławińska, B., Konieczniak, L., Rzepka, M., Jung, M., Sysło, O., et al. (2024) · Cureus · doi:10.7759/cureus.66011 · Cited by 11
“The study results were surprising, as the model achieved a score of 60%, which resulted in passing the exam [6].”
ResultScherr, R., Spina, A., Dao, A., Andalib, S., Halaseh, F. F., Blair, S., Wiechmann, W., & Rivera, R. (2025) · JMIR Formative Research · doi:10.2196/66478 · Cited by 10
“Such studies indicated that LLMs can pass medical exams and possess solid foundations in medical reasoning [3,6-8].”
Can We Trust AI Chatbots’ Answers about Disease Diagnosis and Patient Care?
Huh, S. (2023) · Journal of the Korean Medical Association · doi:10.5124/jkma.2023.66.4.218 · Cited by 10
Meyer, B., Kfuri‐Rubens, R., Schmidt, G., Tariq, M., Riedel, C., Recker, F., Riedel, F., Kiechle, M., & Riedel, M. (2025) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.70251 · Cited by 10
Jain, A., Nimonkar, P., & Jadhav, P. (2025) · Journal of Cranio-Maxillofacial Surgery · doi:10.1016/j.jcms.2025.08.004 · Cited by 10
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T. H. (2023) · doi:10.1101/2023.05.13.23289943 · Cited by 10
“ChatGPT is a specific type of large language model developed by OpenAI (AI, 2023).”
BackgroundGandhi, A. P., Joesph, F. K., Rajagopal, V., Aparnavi, P., Katkuri, S., Dayama, S., Satapathy, P., Khatib, M. N., Gaidhane, S., Zahiruddin, Q. S., et al. (2024) · JMIR Formative Research · doi:10.2196/49964 · Cited by 9
“ChatGPT has been shown to complete exams from varied domains such as business administration [17], the medical licensing exam for medical graduates [18], and the parasitology exam at the undergraduate medical level [19], with mixed results.”
BackgroundTerwilliger, E., Bcharah, G., Bcharah, H., Bcharah, E., Richardson, C., & Scheffler, P. (2024) · Cureus · doi:10.7759/cureus.64204 · Cited by 9
“In recent studies, AI tools achieved above the median score for the Medical College Admission Test (MCAT) and passing scores on the United States Medical Licensing Exam (USMLE) step 1 exam along with medical board exams, including cardiology and neurosurgery [15-18].”
MethodCoughlan, T., Iniesto, F., & Carr, J. E. (2024) · Journal of Interactive Media in Education · doi:10.5334/jime.836 · Cited by 8
Metadata
- Authors
- Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-02-09
- Volume
- 2
- Issue
- 2
- Pages
- e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.