Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Open accessPublished version of a preprint
The DOI matches its Crossref record, with a small difference in title or year; 28 of the 28 references it cites were found in a registry or an index.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Kung et al. (2023) evaluated the performance of ChatGPT on the United States Medical Licensing Exam and demonstrated that the chatbot did not reach the passing threshold and achieved an overall accuracy of only 50% (Kung et al., 2023).”
From A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing (2025) · cites this paper as Kung et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [17]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI's capabilities, such as passing the United States Medical Licensing Exam [107], completing exams at the University of Minnesota Law School [108], and solving queries from Wharton School of Business tests [109], underscore its potential.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [107]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The HCI community has explored the application of LLMs for various healthcare practices, such as mental health support [66,68], health information seeking [81,117,128], and public health interventions [58]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
From DIALOGUE: A Generative AI-Based Pre–Post Simulation Study to Enhance Diagnostic Communication in Medical Students Through Virtual Type 2 Diabetes Scenarios (2025) · cites this paper as Kung et al., 2023
Experimental — accuracy not yet measured
52 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background
Cited by
Items 401–500 of 1,129
Yuan, M., Bao, P., Yuan, J., Shen, Y., Chen, Z., Xie, Y., Zhao, J., Li, Q., Chen, Y., Zhang, L., et al. (2024) · Medicine Plus · doi:10.1016/j.medp.2024.100030 · Cited by 70
Ethical Implications of Chatbot Utilization in Nephrology
Valencia, O. A. G., Suppadungsuk, S., Thongprayoon, C., Miao, J., Tangpanithandee, S., Craici, I. M., & Cheungpasitporn, W. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13091363 · Cited by 103
“These chatbots have found applications in a wide range of industries, including customer service, education, and healthcare [2,5,7,8].”
BackgroundKıyak, Y. S., Çoşkun, Ö., Budakoğlu, I. İ., & Uluoğlu, C. (2024) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-024-03649-x · Cited by 100
Performance of ChatGPT in Diagnosis of Corneal Eye Diseases
Delsoz, M., Madadi, Y., Raja, H., Munir, W. M., Tamm, B., Mehravaran, S., Soleimani, M., Djalilian, A., & Yousefi, S. (2024) · Cornea · doi:10.1097/ico.0000000000003492 · Cited by 86
“21 Another study based on StatPearls ophthalmology questions showed that ChatGPT-3.”
BackgroundGenerative artificial intelligence in mental health care: potential benefits and current challenges
Torous, J., & Blease, C. (2024) · World Psychiatry · doi:10.1002/wps.21148 · Cited by 90
Proposing a Principle-Based Approach for Teaching AI Ethics in Medical Education
Weidener, L., & Fischer, M. (2024) · JMIR Medical Education · doi:10.2196/55368 · Cited by 90
“Although becoming broadly available rather recently, AI-based chat applications such as ChatGPT have rapidly emerged as significant tools with the potential to revolutionize various aspects of medicine, including the education and training of future physicians [27,28].”
BackgroundThe Impact of ChatGPT on Learning Motivation: A Study Based on Self-Determination Theory
Zhou, L., & Li, J. (2023) · Education Science and Management · doi:10.56578/esm010103 · Cited by 117
“Applications of ChatGPT in various fields have been reported, such as medical exams (Gilson et al., 2022; Gilson et al., 2023), improving students' academic writing ability (Stokel-Walker, 2022), and insights into educational reforms (Kung et al., 2023; Kasneci et al., 2023).”
BackgroundPerformance of ChatGPT in medical examinations: A systematic review and a meta‐analysis
Levin, G., Horesh, N., Brezinov, Y., & Meyer, R. (2024) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.17641 · Cited by 90
Yanagita, Y., Yokokawa, D., Uchida, S., Tawara, J., & Ikusaka, M. (2023) · JMIR Formative Research · doi:10.2196/48023 · Cited by 92
“5 to answer law school examination questions [2] and the USMLE (United States Medical Licensing Examination) [3].”
BackgroundResultApplication of deep learning to fault diagnosis of rotating machineries
Su, H., Xiang, L., & Hu, A. (2024) · Measurement Science and Technology · doi:10.1088/1361-6501/ad1e20 · Cited by 97
Assessing the Value of ChatGPT for Clinical Decision Support Optimization
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · doi:10.1101/2023.02.21.23286254 · Cited by 90
“[25,26] As a novel AI algorithm, the goal of the large language model is to predict the next sequences of words based on the previous context.”
BackgroundE, K., S, P., R, G., R, K. L., A, B., M, G., T, O., S, R., V, R., H, M., et al. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04752-w · Cited by 89
“It can be utilized for automated scoring of student papers, easing the burden of grading for teachers (11,12).”
BackgroundLarge Language Models (GPT) Struggle to Answer Multiple-Choice Questions About Code
Savelka, J., Agarwal, A., Bogart, C., & Sakr, M. (2023) · Proceedings of the 15th International Conference on Computer Supported Education · doi:10.5220/0011996900003470 · Cited by 56
Jing, Y., Wang, H., Chen, X., & Wang, C. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-02751-w · Cited by 73
“…(Surameery and Shakor, 2023; Frieder et al., 2023), as well as its impact on fi nancial education (Dowling and Lucey, 2023), medical education (Kung et al., 2023; Gilson et al., 2022), second-language education (Yan, 2023), engineering education (Qadir, 2022), and programming education…”
Yau, J. Y.-S., Saadat, S., Hsu, E., Murphy, L. S.-L., Roh, J., Suchard, J. R., Tapia, A., Wiechmann, W. F., & Langdorf, M. I. (2024) · Journal of Medical Internet Research · doi:10.2196/60291 · Cited by 108
“In health care, LLMs have demonstrated the potential to revolutionize the field, with ChatGPT able to achieve a passing score on the US Medical Licensing Examination and write basic medical reports [7-9].”
Busch, F., Adams, L. C., & Bressem, K. K. (2023) · Medical Science Educator · doi:10.1007/s40670-023-01815-x · Cited by 88
“While LLMs can explain medical terminology to students (and patients) or simulate anamnestic interviews, they could also be used to cheat on exams, papers, and assignments [14].”
BackgroundChatGPT as a Virtual Dietitian: Exploring Its Potential as a Tool for Improving Nutrition Knowledge
Garcia, M. B. (2023) · Applied System Innovation · doi:10.3390/asi6050096 · Cited by 99
“A separate assessment conducted by Kung et al. [46] involved subjecting ChatGPT to the United States Medical Licensing Exam (USMLE).”
Song, H., Xia, Y., Luo, Z., Liu, H., Song, Y., Zeng, X., Li, T., Zhong, G., Li, J., Chen, M., et al. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-02021-3 · Cited by 81
“5 also demonstrated the potential to improve health education by providing consultation to healthcare providers and offering accessible and understandable medical knowledge to the general public [16, 17].”
Artificial General Intelligence for Medical Imaging Analysis
Li, X., Zhao, L., Zhang, L., Wu, Z., Liu, Z., Jiang, H., Cao, C., Xu, S., Li, Y., Dai, H., et al. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3493775 · Cited by 116
“While LLMs like Chat-GPT and its successors have demonstrated proficiency in general medical knowledge [41], they might encounter challenges when performing domain-specific tasks due to a deficiency in specialized knowledge [42], [43] and generate erroneous content.”
Benefits, limits, and risks of ChatGPT in medicine
Tangsrivimol, J. A., Darzidehkalani, E., Virk, H. U. H., Wang, Z., Egger, J., Wang, M., Hacking, S., Glicksberg, B. S., Strauss, M., & Krittanawong, C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1518049 · Cited by 76
“ChatGPT has demonstrated its capabilities in diverse applications, including achieving success in all three print-ahead sections of the USMLE test (Vaswani et al., 2017; Kung et al., 2023).”
Saban, M., & Dubovi, I. (2025) · Journal of Advanced Nursing · doi:10.1111/jan.16101 · Cited by 87
“Another study by Kung et al. ( 2023 )”
Chervenak, J., Lieman, H., Blanco-Breindel, M., & Jindal, S. (2023) · Fertility and Sterility · doi:10.1016/j.fertnstert.2023.05.151 · Cited by 87
Hopkins, B. S., Nguyen, V. N., Dallas, J., Texakalidis, P., Yang, M., Renn, A., Guerra, G., Kashif, Z., Cheok, S., Zada, G., et al. (2023) · Journal of Neurosurgery · doi:10.3171/2023.2.jns23419 · Cited by 84
Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations
Quah, B., Zheng, L., Sng, T. J. H., Yong, C. W., & Islam, I. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05881-6 · Cited by 70
“In addition, it is also capable of aiding in both lesson planning and student assessments [4, 5].”
BackgroundLarge Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis
Zong, H., Wu, R., Cha, J., Wang, J., Wu, E., Li, J., Zhou, Y. K., Zhang, C., Feng, W.-Z., & Shen, B. (2024) · Journal of Medical Internet Research · doi:10.2196/66114 · Cited by 101
“For example, ChatGPT has shown promising results for the United States Medical Licensing Exam, which consists of 3 steps: Step 1, Step 2CK, and Step 3 [22-24].”
InfluentialThe grass is not always greener: Teacher vs. GPT-assisted written corrective feedback
Lin, S., & Crosthwaite, P. (2024) · System · doi:10.1016/j.system.2024.103529 · Cited by 86
Raimondi, R., Tzoumas, N., Salisbury, T., Di Simplicio, S., Romano, M. R., North East Trainee Research in Ophthalmology Network (NETRiON), Bommireddy, T., Chawla, H., Chen, Y., Connolly, S., et al. (2023) · Eye · doi:10.1038/s41433-023-02563-3 · Cited by 85
“Previous reports have shown that LLMs can achieve accuracies of up to 67.6% in generalist medical examinations with the use of different training data and instruction prompt tuning [7, 12].”
BackgroundInfluentialChatGPT-4 Assistance in Optimizing Emergency Department Radiology Referrals and Imaging Selection
Barash, Y., Klang, E., Konen, E., & Sorin, V. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.06.009 · Cited by 85
Daraqel, B., Wafaie, K., Mohammed, H., Cao, L., Mheissen, S., Liu, Y., & Zheng, L. (2024) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2024.01.012 · Cited by 99
Seeing ChatGPT Through Students’ Eyes: An Analysis of TikTok Data
Haensch, A.-C., Ball, S., Herklotz, M., & Kreuter, F. (2023) · 2023 Big Data Meets Survey Science (BigSurv) · doi:10.1109/bigsurv59479.2023.10486710 · Cited by 62
“ChatGPT represents a significant advancement in the usability of natural language processing technology, which has important implications for a variety of fields such as computer science, law, finance, education, and medicine – to only name a few (Choi et al., 2023; Dowling and Lucey, 2023; Kung et al., 2023; Haque et…”
BackgroundDuan, S., Liu, C., Rong, T., Zhao, Y., & Liu, B. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07177-9 · Cited by 85
“Generative AI models have recently garnered attention for their capacity to assess intricate medical and clinical information, rendering them important assets for education and research [20–24].”
Jin, H. K., Lee, H. E., & Kim, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05944-8 · Cited by 85
“Among the studies, two were deemed to be at high risk [33, 34], and 18 at low risk [11, 13, 21, 27, 29–31 ,”
BackgroundInfluential(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.2.26 · Cited by 65
Mo, Z., & Crosthwaite, P. R. (2025) · Journal of English for Academic Purposes · doi:10.1016/j.jeap.2025.101499 · Cited by 87
Park, H., & Ahn, D. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642785 · Cited by 84
Giannos, P. (2023) · BMJ Neurology Open · doi:10.1136/bmjno-2023-000451 · Cited by 89
Krishna, S., Bhambra, N., Bleakney, R. R., & Bhayana, R. (2024) · Radiology · doi:10.1148/radiol.232715 · Cited by 91
Demystifying ChatGPT: An In-depth Survey of OpenAI’s Robust Large Language Models
Bhattacharya, P., Prasad, V. K., Verma, A., Gupta, D. K., Sapsomboon, A., Viriyasitavat, W., & Dhiman, G. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10115-5 · Cited by 118
Beaulieu-Jones, B. R., Berrigan, M. T., Shah, S., Marwaha, J. S., Lai, S.-L., & Brat, G. A. (2024) · Surgery · doi:10.1016/j.surg.2023.12.014 · Cited by 83
“19,23 The current findings are consistent with a study by Hopkins et al. , in which ChatGPT was tested on and achieved near human-level performance on a subset of questions from the Congress of Neurological Surgeons (CNS) Self-Assessment Neurosurgery (SANS).”
Performance of ChatGPT on the Peruvian National Licensing Medical Examination: Cross-Sectional Study
Flores-Cohaila, J. A., García-Vicente, A., Vizcarra-Jiménez, S. F., De la Cruz-Galán, J. P., Gutiérrez-Arratia, J. D., Quiroga Torres, B. G., & Taype-Rondan, A. (2023) · JMIR Medical Education · doi:10.2196/48039 · Cited by 82
Naik, K., Goyal, R. K., Foschini, L., Chak, C. M., Thielscher, C., Zhu, H., Lu, J. J.-Q., Lehár, J., Pacanoswki, M. A., Terranova, N., et al. (2024) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.3152 · Cited by 105
“39 These tools are being rapidly adopted by the community.”
BackgroundLarge Language Models: A Guide for Radiologists
Kim, S., Lee, C., & Kim, S. (2024) · Korean Journal of Radiology · doi:10.3348/kjr.2023.0997 · Cited by 95
“ChatGPT also performed at or near the passing thresholds for all three steps of the United States Medical Licensing Examination, thus demonstrating its potential for in-depth medical assistance [17].”
BackgroundInformation Technology College, U. O. F., & Qawqzeh, Y. (2024) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2024.14.4.2082 · Cited by 54
“Additionally, research delves into ChatGPT’s efficacy as an independent self-learning tool, particularly in answering multiple-choice questions and providing reasoned explanations [19].”
BackgroundLarge language models "ad referendum": How good are they at machine translation in the legal domain?
Brivá-Iglesias, V., Doğru, G., & Camargo, J. L. C. (2024) · MonTi Monografías de Traducción e Interpretación · doi:10.6035/monti.2024.16.02 · Cited by 96
Artificial Intelligence in Medicine & ChatGPT: De-Tether the Physician
DiGiorgio, A. M., & Ehrenfeld, J. M. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01926-3 · Cited by 82
Here's to the future: Conversational agents in higher education- a scoping review
Pereira, D. S. M., Falcão, F., Costa, L., Lunn, B. S., Pêgo, J. M., & Costa, P. (2023) · International Journal of Educational Research · doi:10.1016/j.ijer.2023.102233 · Cited by 95
Pushpanathan, K., Lim, Z. W., Er Yew, S. M., Chen, D. Z., Hui'En Lin, H. A., Lin Goh, J. H., Wong, W. M., Wang, X., Jin Tan, M. C., Chang Koh, V. T., et al. (2023) · iScience · doi:10.1016/j.isci.2023.108163 · Cited by 77
Pręgowska, A., Osial, M. E., & Gajda, A. (2023) · Metaverse Basic and Applied Research · doi:10.56294/mr202457 · Cited by 101
Evaluating ChatGPT Responses on Thyroid Nodules for Patient Education
Campbell, D. J., Estephan, L. E., Sina, E. M., Mastrolonardo, E. V., Alapati, R., Amin, D. R., & Cottrill, E. E. (2024) · Thyroid® · doi:10.1089/thy.2023.0491 · Cited by 81
Unveiling the Dark Side of ChatGPT: Exploring Cyberattacks and Enhancing User Awareness
Alawida, M., Abu Shawar, B., Abiodun, O. I., Mehmood, A., Omolara, A. E., & Al Hwaitat, A. K. (2024) · Information · doi:10.3390/info15010027 · Cited by 64
Large Language Models in Education: A Systematic Review
Dong, B., Bai, J., Xu, T., & Zhou, Y. (2024) · 2024 6th International Conference on Computer Science and Technologies in Education (CSTE) · doi:10.1109/cste62025.2024.00031 · Cited by 61
“The examination contents include the performance of medical school and law school examinations [25], the China National Medical Licensing Examination [26], written assignments [27], biological tests [28], and MCQ tests [29].”
MethodComparing Large Language Models and Human Programmers for Generating Programming Code
Hou, W., & Ji, Z. (2025) · Advanced Science · doi:10.1002/advs.202412279 · Cited by 48
“The emergence and growth of large language models (LLMs), such as GPT-4, Gemini, Claude, and Llama, are poised to revolutionize various sectors, including computer programming 1–8 , education 9,10 , and biomedical research 11–15 .”
Performance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81
Shimizu, I., Kasai, H., Shikino, K., Araki, N., Takahashi, Z., Onodera, M., Kimura, Y., Tsukamoto, T., Yamauchi, K., Asahina, M., et al. (2023) · JMIR Medical Education · doi:10.2196/53466 · Cited by 81
“For example, GAI is now capable of passing various certification examinations, including those for medical licensure in at least questions without images [11,12].”
BackgroundMadrid-García, A., Rosales-Rosado, Z., Freites-Nuñez, D., Pérez-Sancristóbal, I., Pato-Cour, E., Plasencia-Rodríguez, C., Cabeza-Osorio, L., Abasolo-Alcázar, L., León-Mateos, L., Fernández-Gutiérrez, B., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-49483-6 · Cited by 79
Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22.
Cecconi, M., Greco, M., Shickel, B., Angus, D. C., Bailey, H., Bignami, E., Calandra, T., Celi, L. A., Einav, S., Elbers, P., et al. (2025) · Critical Care · doi:10.1186/s13054-025-05532-2 · Cited by 80
“In addition to undergraduate education, it is essential to train experienced physicians, nurses, and other allied health professional [43].”
Artificial Intelligence in Health Professions Education assessment: AMEE Guide No. 178
Masters, K., MacNeil, H., Benjamin, J., Carver, T., Nemethy, K., Valanci-Aroesty, S., Taylor, D. C. M., Thoma, B., & Thesen, T. (2025) · Medical Teacher · doi:10.1080/0142159x.2024.2445037 · Cited by 80
“A good example is Kung et al. [63], who showed that their ChatGPT model could ask and answer questions set at the level of USMLE Steps 1, 2 and 3, and crucially, give reasonable explanations for the answers it chose.”
Maxwell, D., Oyarzun, B., Kim, S., & Bong, J. Y. (2025) · TechTrends · doi:10.1007/s11528-025-01109-6 · Cited by 80
“However, challenges in GenAI accuracy and reliability (Joseph, 2023; Imran et al., 2023; Kung et al., 2023) and data security concerns (Zorz, 2023) can influence student concerns and intention to use GenAI.”
Beyond Traditional Assessment: Exploring the Impact of Large Language Models on Grading Practices
Fagbohun, O., Iduwe, N. P., Abdullahi, M., Ifaturoti, A., & Nwanna, M. (2024) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/19 · Cited by 107
“37 Developed an automated rating system using key phrases and machine learning, laying a foundation for future improvements despite its limitations in application-based responses and lack of training data.”
Attributions toward artificial agents in a modified Moral Turing Test
Aharoni, E., Fernandes, S., Brady, D. J., Alexander, C., Criner, M., Queen, K., Rando, J., Nahmias, E., & Crespo, V. (2024) · Scientific Reports · doi:10.1038/s41598-024-58087-7 · Cited by 46
Guo, Y., Ovadje, A., Al-Garadi, M. A., & Sarker, A. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae210 · Cited by 111
“Kung et al 22 showed that ChatGPT performed at or near the passing threshold for the United States Medical Licensing Exam without requiring supervision with human-annotated data.”
BackgroundAnalysis of LLMs for educational question classification and generation
Faraby, S. A., Romadhony, A., & Adiwijaya, A. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100298 · Cited by 116
Koga, S., Martin, N. B., & Dickson, D. W. (2024) · Brain Pathology · doi:10.1111/bpa.13207 · Cited by 78
“A recent study showed that ChatGPT achieved scores at or near the passing threshold across all three stages of the United States Medical Licensing Exam (USMLE) without specialized training [1].”
BackgroundLarge Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis
Yu, H., Fan, L., Li, L., Zhou, J., Ma, Z., Xian, L., Hua, W., He, S., Jin, M., Zhang, Y., et al. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-024-00171-8 · Cited by 101
Appraising the performance of ChatGPT in psychiatry using 100 clinical case vignettes
Franco D’Souza, R., Amanullah, S., Mathew, M., & Surapaneni, K. M. (2023) · Asian Journal of Psychiatry · doi:10.1016/j.ajp.2023.103770 · Cited by 77
Uribe, S. E., Maldupa, I., Kavadella, A., El Tantawi, M., Chaurasia, A., Fontana, M., Marino, R., Innes, N., & Schwendicke, F. (2024) · European Journal of Dental Education · doi:10.1111/eje.13009 · Cited by 77
Taira, K., Itaya, T., & Hanada, A. (2023) · JMIR Nursing · doi:10.2196/47305 · Cited by 76
Medical education and physician training in the era of artificial intelligence
Xu, Y., Jiang, Z., Ting, D. S. W., Kow, A. W. C., Bello, F., Car, J., Tham, Y.-C., & Wong, T. Y. (2024) · Singapore Medical Journal · doi:10.4103/singaporemedj.smj-2023-203 · Cited by 76
“[32] In this regard, LLMs can be a useful learning tool to aid in revision.”
ChatGPT as a Complementary Mental Health Resource: A Boon or a Bane
Farhat, F. (2023) · doi:10.20944/preprints202307.1479.v1 · Cited by 76
“It has been utilized in a variety of fields, including education [10], scientific research [3, 4], and computer programming [9].”
Hua, H.-U., Kaakour, A.-H., Rachitskaya, A., Srivastava, S., Sharma, S., & Mammo, D. A. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.3119 · Cited by 76
Abdullahi, T., Singh, R., & Eickhoff, C. (2024) · JMIR Medical Education · doi:10.2196/51391 · Cited by 76
“Consequently, we surmise that LLMs are not yet ready to be used as stand-alone tools, which aligns with the findings of previous studies [5,17,18].”
BackgroundResultInfluentialCappellani, F., Card, K. R., Shields, C. L., Pulido, J. S., & Haller, J. A. (2024) · Eye · doi:10.1038/s41433-023-02906-0 · Cited by 85
Application of large language models in disease diagnosis and treatment
Yang, X., Li, T., Su, Q., Liu, Y., Kang, C., Lyu, Y., Zhao, L., Nie, Y., & Pan, Y. (2024) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003456 · Cited by 100
Harnessing the Power of Generative AI for Clinical Summaries: Perspectives From Emergency Physicians
Barak-Corren, Y., Wolf, R., Rozenblum, R., Creedon, J. K., Lipsett, S. C., Lyons, T. W., Michelson, K. A., Miller, K. A., Shapiro, D. J., Reis, B. Y., et al. (2024) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2024.01.039 · Cited by 75
Exploring the Usage of ChatGPT Among Medical Students in the United States
Zhang, J. S., Yoon, C., Williams, D. K. A., & Pinkas, A. (2024) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205241264695 · Cited by 75
Evaluating the performance of ChatGPT in medical physiology university examination of phase I MBBS
Mahila, S., Jaleel, I., & Surapaneni, K. M. (2023) · AJP Advances in Physiology Education · doi:10.1152/advan.00036.2023 · Cited by 82
Cortez, P., Ong, A. K. S., Diaz, J. F. T., German, J. D., & Jagdeep, S. J. S. S. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25896 · Cited by 85
“Compared to the study of Kung et al. [3], it could be posited that this was significant only during the review process of their respondents.”
BackgroundMethodResultInfluentialArtificial Intelligence in Cardiovascular Care—Part 2: Applications
Jain, S. S., Elias, P., Poterucha, T., Randazzo, M., Lopez Jimenez, F., Khera, R., Perez, M., Ouyang, D., Pirruccello, J., Salerno, M., et al. (2024) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2024.03.401 · Cited by 74
(2023) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets45212 · Cited by 74
ChatGPT for Clinical Vignette Generation, Revision, and Evaluation
Benoit, J. R. A. (2023) · doi:10.1101/2023.02.04.23285478 · Cited by 74
“These early results are corroborated by ChatGPT offering near-passing performance on all sections of the USMLE (United States Medical Licensing Exam).(44) However, providing appropriate triage advice of these models can and should still be improved.”
ResultAl-Worafi, Y. M., Goh, K. W., Hermansyah, A., Tan, C. S., & Ming, L. C. (2024) · JMIR Medical Education · doi:10.2196/47339 · Cited by 74
“As a precursor to future integration into clinical decision-making, Kung et al [11] indicate that LLMs, such as ChatGPT, performed at or near the qualifying accuracy threshold of 60% in the United States Medical Licensing Examination.”
MethodHulman, A., Dollerup, O. L., Mortensen, J. F., Fenech, M. E., Norman, K., Støvring, H., & Hansen, T. K. (2023) · PLOS ONE · doi:10.1371/journal.pone.0290773 · Cited by 70
“More studies and the scientific discourse have mostly focused on ChatGPT’s role in medical research (e.g. scientific writing 6,19,20 ), formal testing of large language models’ medical knowledge e.g. by taking medical licensing exams, 7-9 or answering questions in highly specialized topics.”
Danesh, A., Pazouki, H., Danesh, K., Danesh, F., & Danesh, A. (2023) · The Journal of the American Dental Association · doi:10.1016/j.adaj.2023.07.016 · Cited by 75
Beyond ChatGPT: What does GPT-4 add to healthcare? The dawn of a new era
Wójcik, S., Rulkiewicz, A., Pruszczyk, P., Lisik, W., Poboży, M., & Domienik-Karłowicz, J. (2023) · Cardiology Journal · doi:10.5603/cj.97515 · Cited by 68
“As detailed in PLoS Digital Health, ChatGPT’s performance was commendable, securing between 52.4% and 75% accurate responses [5, 6].”
BackgroundInfluentialA systematic review and meta-analysis on ChatGPT and its utilization in medical and dental research
Bagde, H., Dhopte, A., Alam, M. K., & Basri, R. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e23050 · Cited by 66
“By making accessible and understandable health information available to the general population, ChatGPT in health care settings also has the potential to increase health literacy and advance personalized treatment [20,22,39,47,48].”
BackgroundThe role of artificial intelligence in generating original scientific research
Elbadawi, M., Li, H., Basit, A. W., & Gaisford, S. (2024) · International Journal of Pharmaceutics · doi:10.1016/j.ijpharm.2023.123741 · Cited by 88
Zhang, J., Sun, K., Jagadeesh, A., Falakaflaki, P., Kayayan, E., Tao, G., Haghighat Ghahfarokhi, M., Gupta, D., Gupta, A., Gupta, V., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae184 · Cited by 70
Yilmaz, B. E., Yılmaz, B., & Özbey, F. (2025) · BMC Oral Health · doi:10.1186/s12903-025-05926-2 · Cited by 86
“The results obtained are consistent with similar studies conducted in various fields in the literature, demonstrating that ChatGPT achieves high accuracy rates in answering medical questions among the LLMs [16, 17].”
Yan, C., Grabowska, M. E., Dickson, A. L., Li, B., Wen, Z., Roden, D. M., Michael Stein, C., Embí, P. J., Peterson, J. F., Feng, Q., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01038-3 · Cited by 73
Ganjavi, C., Eppler, M., O’Brien, D., Ramacciotti, L. S., Ghauri, M. S., Anderson, I., Choi, J., Dwyer, D., Stephens, C., Shi, V., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000596 · Cited by 73
“…the accuracy and reliability of using ChatGPT for a variety of academic and medical related tasks, such as performance on the United States Medical Licensing Exams [5], ability to write accurate patient letters [6], automate clinical notes [25], and disseminate health related inquiries [26].”
Dr ChatGPT tell me what I want to hear: How different prompts impact health answer correctness
Koopman, B., & Zuccon, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.928 · Cited by 73
Liu, M., Okuhara, T., Dai, Z., Huang, W., Gu, L., Okada, H., Furukawa, E., & Kiuchi, T. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105673 · Cited by 73
Mooney, P., Cui, W., Guan, B., & Juhász, L. (2023) · ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery (GeoAI) · doi:10.1145/3615886.3627745 · Cited by 101
Günay, S., Öztürk, A., Özerol, H., Yiğit, Y., & Erenler, A. K. (2024) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2024.03.017 · Cited by 82
Künzle, P., & Paris, S. (2024) · Clinical Oral Investigations · doi:10.1007/s00784-024-05968-w · Cited by 84
Magalhães Araujo, S., & Cruz-Correia, R. (2024) · JMIR Medical Education · doi:10.2196/51151 · Cited by 72
“…AI into medical programs to prepare students for their future use of such tools in professional contexts [5,8,28,29], the implementation of ChatGPT has emerged as a potentially transformative force in medical education [30,31], offering support to students in their learning journey [30,32].”
BackgroundHuang, Y., Wu, R., He, J., & Xiang, Y. (2024) · Journal of Global Health · doi:10.7189/jogh.14.04070 · Cited by 72
“Consequently, it has been widely applied and researched across various domains [3–5], including academic research [6–8], health care analytics [9–11], and education [12,13].”
BackgroundCan we trust the evaluation on ChatGPT?
Aiyappa, R., An, J., Kwak, H., & Ahn, Y.-Y. (2023) · Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) · doi:10.18653/v1/2023.trustnlp-1.5 · Cited by 72
Scholich, T., Barr, M., Wiltsey Stirman, S., & Raj, S. (2025) · JMIR Mental Health · doi:10.2196/69709 · Cited by 72
“LLM responses have been evaluated across a wide range of questions, including questions on the United States Medical Licensing Examination [25], questions related to certain specialties [26], and responses to curated clinical vignettes [27].”
Agarwal, M., Sharma, P., & Goswami, A. (2023) · Cureus · doi:10.7759/cureus.40977 · Cited by 72
“ChatGPT not only exhibited the capacity to pass the USMLE independently without any human assistance but also showcased comprehensible reasoning and provided valid clinical insights in its responses [11,12].”
BackgroundResult
Metadata
- Authors
- Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-02-09
- Volume
- 2
- Issue
- 2
- Pages
- e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.