Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Open accessPublished version of a preprint
The DOI matches its Crossref record, with a small difference in title or year; 28 of the 28 references it cites were found in a registry or an index.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Kung et al. (2023) evaluated the performance of ChatGPT on the United States Medical Licensing Exam and demonstrated that the chatbot did not reach the passing threshold and achieved an overall accuracy of only 50% (Kung et al., 2023).”
From A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing (2025) · cites this paper as Kung et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [17]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI's capabilities, such as passing the United States Medical Licensing Exam [107], completing exams at the University of Minnesota Law School [108], and solving queries from Wharton School of Business tests [109], underscore its potential.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [107]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The HCI community has explored the application of LLMs for various healthcare practices, such as mental health support [66,68], health information seeking [81,117,128], and public health interventions [58]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
From DIALOGUE: A Generative AI-Based Pre–Post Simulation Study to Enhance Diagnostic Communication in Medical Students Through Virtual Type 2 Diabetes Scenarios (2025) · cites this paper as Kung et al., 2023
Experimental — accuracy not yet measured
52 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background
Cited by
Items 801–900 of 1,129
Pagano, S., Holzapfel, S., Kappenschneider, T., Meyer, M., Maderbacher, G., Grifka, J., & Holzapfel, D. E. (2023) · Journal of Orthopaedics and Traumatology · doi:10.1186/s10195-023-00740-4 · Cited by 42
Rodman, A. M., Buckley, T. A., Manrai, A. K., & Morgan, D. J. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.47075 · Cited by 52
Cheong, R. C. T., Pang, K. P., Unadkat, S. N., Mcneillis, V., Williamson, A., Joseph, J., Randhawa, P. S., Andrews, P. J., & Paleri, V. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08381-3 · Cited by 51
Performance of a large language model on practice questions for the neonatal board examination.
Beam, K., Sharma, P., Kumar, B., Wang, C., Brodsky, D., Martin, C. R., & Beam, A. (2023) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2023.2373 · Cited by 43
Patino, G. A., Amiel, J. M., Brown, M., Lypson, M. L., & Chan, T. M. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005636 · Cited by 43
“29 In addition, educators should make it clear when students are not allowed to use LLMs or other types of outside help (e.g., for re fl ective writing activities).”
BackgroundThe AI Future of Emergency Medicine
Petrella, R. J. (2024) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2024.01.031 · Cited by 43
Agapiou, A., & Lysandrou, V. (2023) · Heritage · doi:10.3390/heritage6050214 · Cited by 39
Borchert, R. J., Hickman, C. R., Pepys, J., & Sadler, T. J. (2023) · JMIR Medical Education · doi:10.2196/48978 · Cited by 39
“With regard to the United States Medical Licensing Examination (USMLE), ChatGPT scored at, or near, the pass mark for each step of the examination [4].”
BackgroundMethodInfluentialPassmore, J., & Tee, D. (2023) · Journal of Work-Applied Management · doi:10.1108/jwam-06-2023-0057 · Cited by 50
Potential Use of ChatGPT for Patient Information in Periodontology: A Descriptive Pilot Study
Babayiğit, O., Tastan Eroglu, Z., Ozkan Sen, D., & Ucan Yarkac, F. (2023) · Cureus · doi:10.7759/cureus.48518 · Cited by 42
“In a study by Kung et al., the performance of ChatGPT, a language-based AI, in the United States Medical Licensing Examination (USMLE) was evaluated [13].”
BackgroundMethodDurairaj, K. K., Baker, O., Bertossi, D., Dayan, S. H., Karimi, K., Kim, R. Y., Most, S. P., Robotti, E., & Rosengaus, F. (2023) · Facial Plastic Surgery & Aesthetic Medicine · doi:10.1089/fpsam.2023.0224 · Cited by 49
Kedia, N., Sanjeev, S., Ong, J., & Chhablani, J. K. (2024) · Eye · doi:10.1038/s41433-023-02915-z · Cited by 50
“5 has performed well on several knowledge-based tests including the United States Medical Licensing Exam (USMLE), where it was able to perform at or near the passing threshold [27].”
Assessment of Pathology Domain-Specific Knowledge of ChatGPT and Comparison to Human Performance
Wang, A. Y., Lin, S., Tran, C., Homer, R. J., Wilsdon, D., Walsh, J. C., Goebel, E. A., Sansano, I., Sonawane, S., Cockenpot, V., et al. (2024) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2023-0296-oa · Cited by 43
Performance of ChatGPT and Bard in self-assessment questions for nephrology board renewal
Noda, R., Izaki, Y., Kitano, F., Komatsu, J., Ichikawa, D., & Shibagaki, Y. (2024) · Clinical and Experimental Nephrology · doi:10.1007/s10157-023-02451-w · Cited by 38
“Prior research has reported on the performance of LLMs regarding the mandatory skills and general knowledge of medical doctors, such as the Japanese national medical exam and USMLE [2, 3, 18].”
Performance of large language models in oral and maxillofacial surgery examinations
Quah, B., Yong, C. W., Lai, C. W. M., & Islam, I. (2024) · International Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.ijom.2024.06.003 · Cited by 49
Ethical Application of Generative Artificial Intelligence in Medicine
Hasan, S. S., Fury, M. S., Woo, J. J., Kunze, K. N., & Ramkumar, P. N. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.12.011 · Cited by 51
The promise and peril of ChatGPT in geriatric nursing education: What We know and do not know
Qi, X., Zhu, Z., & Wu, B. (2023) · Aging and Health Research · doi:10.1016/j.ahr.2023.100136 · Cited by 47
Leveraging Large Language Models for Generating Responses to Patient Messages
Liu, S., McCoy, A. B., Wright, A. P., Carew, B., Genkins, J. Z., Huang, S. S., Peterson, J. F., Steitz, B., & Wright, A. (2023) · doi:10.1101/2023.07.14.23292669 · Cited by 43
“[16–18] In particular, a recent study used ChatGPT to generate responses to 195 patient questions from social media forums.”
MethodMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
Cai, Y., Wang, L., Wang, Y., De Melo, G., Zhang, Y., Wang, Y., & He, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29723 · Cited by 42
Artificial Intelligence in Ecology: A Commentary on a Chatbot's Perspective
Haghighi, S. R., Saqalaksari, M. P., & Johnson, S. N. (2023) · Bulletin of the Ecological Society of America · doi:10.1002/bes2.2097 · Cited by 47
AI and machine learning in resuscitation: Ongoing research, new concepts, and key challenges
Okada, Y., Mertens, M., Liu, N., Lam, S. S. W., & Ong, M. E. H. (2023) · Resuscitation Plus · doi:10.1016/j.resplu.2023.100435 · Cited by 47
“) One representative example of using LLM is that the LLM can pass the medical licensing examination without any additional training data.(66,67) Further, some research indicated that LLM can provide quality and empathetic responses to patient questions.”
MethodGobira, M., Nakayama, L. F., Moreira, R., Andrade, E., Regatieri, C. V. S., & Belfort Jr., R. (2023) · Revista da Associação Médica Brasileira · doi:10.1590/1806-9282.20230848 · Cited by 42
Choo, J. M., Ryu, H. S., Kim, J., Cheong, J. Y., Baek, S. J., Kwak, J., & Kim, J. (2023) · ANZ Journal of Surgery · doi:10.1111/ans.18749 · Cited by 43
Ülkü, A. (2023) · Tourism & Management Studies · doi:10.18089/tms.2023.190402 · Cited by 48
Marchi, F., Bellini, E., Iandelli, A., Sampieri, C., & Peretti, G. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-024-08525-z · Cited by 45
Bharatha, A., Ojeh, N. O., Rabbi, A. M. F., Campbell, M., Krishnamurthy, K., Layne-Yarde, R. N. A., Kumar, A., Springer, D., Connell, K., & Majumder, M. A. A. (2024) · Advances in Medical Education and Practice · doi:10.2147/amep.s457408 · Cited by 48
“21 Another study tasked ChatGPT to respond to complex pathology questions necessitating advanced reasoning.”
BackgroundCanagasuriam, D., & Lukacik, E. (2024) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12491 · Cited by 48
Assessing GPT-4’s Performance in Delivering Medical Advice: Comparative Analysis With Human Experts
Jo, E., Song, S., Kim, J.-H., Lim, S., Kim, J. H., Cha, J.-J., Kim, Y.-M., & Joo, H. J. (2024) · JMIR Medical Education · doi:10.2196/51282 · Cited by 39
“5 is its success in meeting the passing threshold for the United States Medical Licensing Examination [8], demonstrating its ability to offer medical advice comparable to that of trained professionals [9].”
BackgroundChatGPT Goes to College: Exploring Student Perspectives on Artificial Intelligence in the Classroom
Cavazos, J. T., Hauck, K. A., Baskin, H. M., & Bain, C. M. (2024) · Teaching of Psychology · doi:10.1177/00986283241268829 · Cited by 46
“These issues currently prevent users from having complete con fi dence in the accuracy of AI-generated content (Kung et al., 2023; Kusunose et al., 2023; Vaira et al., 2023).”
The AI revolution in glaucoma: Bridging challenges with opportunities
Li, F., Wang, D., Yang, Z., Zhang, Y., Jiang, J., Liu, X., Kong, K., Zhou, F., Tham, C. C. Y., Medeiros, F. A., et al. (2024) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2024.101291 · Cited by 47
Guo, J., Wang, M., Yin, H., Song, B., Chi, Y., Yu, F. R., & Yuen, C. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3496491 · Cited by 51
“Moreover, extensive explorations have been undertaken to apply LLMs and AIGC technologies in diverse application fields, including medical [19], [20], finance [21], [22], education [23], [24], software engineering [25], [26], and so on.”
BackgroundUsing ChatGPT to navigate ambivalent and contradictory research findings on artificial intelligence
Sohail, S. S., Madsen, D. Ø., Himeur, Y., & Ashraf, M. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1195797 · Cited by 35
“The research literature on the effects of ChatGPT is still in its infancy, and while some studies have shown promising results in the chatbot’s ability to answer exams on topics such as medicine (Fijačko et al., 2023; Kung et al., 2023; Mogali, 2023), law (Choi et al.”
Luan, L., Lin, X. Y., & Li, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.04818 · Cited by 24
Wu, J., Wu, X., Qiu, Z., Li, M., Lin, S., Zhang, Y., Zheng, Y., Yuan, C., & Yang, J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae079 · Cited by 41
“20,21,22 Despite significant advancements in LLMs, their development and validation predominantly rely on English-centric, general-domain datasets.”
BackgroundSauder, M., Tritsch, T., Rajput, V., Schwartz, G., & Shoja, M. M. (2024) · Cureus · doi:10.7759/cureus.51961 · Cited by 41
“Kung et al. evaluated the performance of ChatGPT on USMLE Step 1, Step 2, and Step 3 exams [22].”
MethodHenson, J. B., Brown, J. R. G., Lee, J. P., Patel, A. C., & Leiman, D. A. (2023) · The American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002397 · Cited by 44
“It has demonstrated su ffi cient medical knowledge to successfully pass the United States Medical Licensing Examination (4).”
BackgroundProfessional Certification Benchmark Dataset: The First 500 Jobs for Large Language Models
Noever, D. A., & Ciolino, M. (2023) · doi:10.5121/csit.2023.131211 · Cited by 41
Vrontis, D., Chaudhuri, R., & Chatterjee, S. (2023) · Sustainability · doi:10.3390/su151612196 · Cited by 44
“ChatGPT can also be useful in research activities [27,28].”
BackgroundBrainstorming Will Never Be the Same Again—A Human Group Supported by Artificial Intelligence
Lavrič, F., & Škraba, A. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5040065 · Cited by 50
Evaluation of information from artificial intelligence on rotator cuff repair surgery
Warren, E., Hurley, E. T., Park, C. N., Crook, B. S., Lorentz, S. G., Levin, J. M., Anakwenze, O. A., MacDonald, P. B., & Klifto, C. S. (2023) · JSES International · doi:10.1016/j.jseint.2023.09.009 · Cited by 42
“21 AI has already shown the potential for impacts in or-thopedics beyond writing such as the creation of outcome prediction algorithms in trauma, imaging interpretation, and extraction of data from electronic health records.”
BackgroundOon, M. L., Syn, N. L., Tan, C. L., Tan, K., & Ng, S. (2024) · Histopathology · doi:10.1111/his.15100 · Cited by 40
“3,5,6 7 – 9 For example, a recent study reported that ChatGPT was capable of providing factually accurate and contextually relevant answers in several knowledge domains, and out-scored human candidates in a virtual specialist Obstetrics and Gynaecology Objective Structured Clinical Examination…”
BackgroundOfem, U. J., ARIKPO, I. M., Ovat, S. V., Nwogwugwu, C. E., Anake, P. M., Udeh, M. I., & Diwa, O. B. (2024) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2024.7.1.9 · Cited by 62
Generative artificial intelligence and scientific publishing: urgent questions, difficult answers
Bagenal, J. D. (2024) · The Lancet · doi:10.1016/s0140-6736(24)00416-1 · Cited by 44
Exploring the competence of ChatGPT for customer and patient service management
Haleem, A., Javaid, M., & Singh, R. P. (2024) · Intelligent Pharmacy · doi:10.1016/j.ipha.2024.03.002 · Cited by 47
Kunze, K. N., Varady, N. H., Mazzucco, M. R., Lu, A. Z., Chahla, J. A., Martin, R. K., Ranawat, A. S., Pearle, A. D., & Williams, R. J. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.06.021 · Cited by 42
Apriani, E., Daulay, S. H., Aprilia, F., Marzuki, A. G., Warsah, I., Supardan, D., & Muthmainnah, M. (2025) · Journal of language and Education · doi:10.17323/jle.2025.17913 · Cited by 51
“They point out the infrequent acknowledgment of ChatGPT in academic literature, raising concerns about its scholarly acceptance (Kung et al., 2023; Manohar & Prasad, 2023; Stokel-Walker, 2022; Thorp, 2023).”
Exploring the ethical use of LLM chatbots in higher education
Mvondo, G. F. N., Niu, B., & Eivazinezhad, S. (2025) · Journal of Business Research · doi:10.1016/j.jbusres.2025.115575 · Cited by 45
Chlorogiannis, D.-D., Apostolos, A., Chlorogiannis, A., Palaiodimos, L., Giannakoulas, G., Pargaonkar, S., Xesfingi, S., & Kokkinidis, D. G. (2023) · Healthcare · doi:10.3390/healthcare11212906 · Cited by 37
“In a study by Kung et al., the performance of ChatGPT was evaluated on the United States Medical Licensing Examination, and the language model performed at or near the passing threshold [32].”
MethodChatGPT in Education - A Global and Vietnamese Research OverviewPreprint
Trương, H. (2023) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/r4uhd · Cited by 47
Generative Artificial Intelligence and Large Language Models in Primary Care Medical Education
Parente, D. J. (2024) · Family Medicine · doi:10.22454/fammed.2024.775525 · Cited by 40
“16 ,8–12,17– 19 There is risk here.”
Diagnostic and Management Performance of ChatGPT in Obstetrics and Gynecology
Allahqoli, L., Ghiasvand, M. M., Mazidimoradi, A., Salehiniya, H., & Alkatout, I. (2023) · Gynecologic and Obstetric Investigation · doi:10.1159/000533177 · Cited by 40
“In other studies, the accuracy of prep questions for medical licensing exams by ChatGPT was evaluated, and the results showed high levels of accuracy in its performance [12, 13].”
BackgroundMartinengo, L., Lin, X., Jabir, A. I., Kowatsch, T., Atun, R. A., Car, J., & Car, L. T. (2023) · Journal of Medical Internet Research · doi:10.2196/50767 · Cited by 45
“ChatGPT has recently been credited with passing the USMLE (United States Medical Licensing Exam), a standardized set of 3 exams required to obtain medical licensure in the United States [51].”
Ignjatović, A., & Stevanović, L. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.28 · Cited by 38
“To date, ChatGPT passed all 3 exams on the United States Medical Licensing Examination [4,5].”
Al-Sharif, E. M., Penteado, R. C., Dib El Jalbout, N., Topilow, N. J., Shoji, M. K., Kikkawa, D. O., Liu, C. Y., & Korn, B. S. (2024) · Ophthalmic Plastic & Reconstructive Surgery · doi:10.1097/iop.0000000000002567 · Cited by 40
“6,7 It is also able to compile detailed discharge summaries and even operative reports for advanced ophthalmic surgery, such as phacoemulsification and pars plana vitrectomy.”
BackgroundLee, Y., Tessier, L., Brar, K., Malone, S., Jin, D., McKechnie, T., Jung, J. J., Kroh, M., & Dang, J. T. (2024) · Surgery for Obesity and Related Diseases · doi:10.1016/j.soard.2024.04.014 · Cited by 42
Bhattacharya, M., Pal, S., Chatterjee, S., Lee, S., & Chakraborty, C. (2024) · Molecular Therapy — Nucleic Acids · doi:10.1016/j.omtn.2024.102255 · Cited by 54
“80,81,83 LLMs ’ medical reasoning and concept understanding can be improved even further by providing focused instruction that includes questions akin to those on a medical test and expertly chosen sample answers.”
BackgroundMaaß, L., Grab‐Kroll, C., Koerner, J., Öchsner, W., Schön, M., Messerer, D. A. C., Boeckers, T. M., & Böckers, A. (2024) · Journal of CME · doi:10.1080/28338073.2024.2437293 · Cited by 46
Quality assurance and validity of AI-generated single best answer questions
Ahmed, A., Kerr, E., & O’Malley, A. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06881-w · Cited by 44
“So far, this model has already been able to successfully pass the USMLE, so it is reasonable to hypothesize that the LLM could potentially write the exam itself [20].”
InfluentialLuo, D., Liu, M., Yu, R., Liu, Y., Jiang, W., Fan, Q., Kuang, N., Gao, Q., Yin, T., & Zheng, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-98949-2 · Cited by 53
“5 has passed the United States Medical Licensing Examination (USMLE) 16,17 , it has not succeeded in the NMLE 20–22 .”
Sharma, P., Thapa, K., Thapa, D., Dhakal, P., Upadhaya, M. D., Adhikari, S., & Khanal, S. R. (2023) · arXiv · doi:10.48550/arxiv.2307.00112 · Cited by 40
“Likewise, Kung (Kung et al., 2022) evaluated the ChatGPT performance on the USMLE, where they achieved scores at or near the passing threshold for all three exams without specialized training.”
MethodGPT detectors are biased against non-native English writers
Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2304.02819 · Cited by 40
Below average ChatGPT performance in medical microbiology exam compared to university students
Sallam, M., & Al-Salahat, K. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1333415 · Cited by 39
Tong, W., Guan, Y., Chen, J., Huang, X., Zhong, Y., Zhang, C., & Zhang, H. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1237432 · Cited by 41
ChatGPT: oportunidades y riesgos en la asistencia, docencia e investigación médica
Gutiérrez‐Cirlos, C., Carrillo‐Pérez, D. L., Bermudez‐Gonzalez, J. L., Hidrogo-Montemayor, I., Carrillo-Esper, R., & Mendiola, M. S. (2023) · Gaceta Médica de México · doi:10.24875/gmm.230001671 · Cited by 58
Rojas, M., Rojas, M., Burgess, V., Toro-Pérez, J., & Salehi, S. (2024) · JMIR Medical Education · doi:10.2196/55048 · Cited by 42
“In the United States, different studies have reported an accuracy of 80%-90% on the United States Medical Licensing Examination [4].”
BackgroundSiepmann, R. M., Huppertz, M., Rastkhiz, A., Reen, M., Corban, E., Schmidt, C., Wilke, S., Schad, P., Yüksel, C., Kuhl, C. K., et al. (2024) · European Radiology · doi:10.1007/s00330-024-10727-2 · Cited by 43
AI in Dental Radiology—Improving the Efficiency of Reporting With ChatGPT: Comparative Study
Stephan, D., Bertsch, A., Burwinkel, M., Vinayahalingam, S., Al‐Nawas, B., Kämmerer, P. W., & Thiem, D. G. E. (2024) · Journal of Medical Internet Research · doi:10.2196/60684 · Cited by 46
“[8-10].”
A review of ophthalmology education in the era of generative artificial intelligence
Heinke, A., Radgoudarzi, N., Huang, B. B., & Baxter, S. L. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100089 · Cited by 49
Sallam, M., Al‐Salahat, K., Eid, H., Egger, J., & Puladi, B. (2024) · Advances in Medical Education and Practice · doi:10.2147/amep.s479801 · Cited by 50
Levra, A. G., Gatti, M., Menè, R., Shiffer, D., Costantino, G., Solbiati, M., Furlan, R., & Dipaola, F. (2024) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2024.09.017 · Cited by 41
Artificial Intelligence Use in Medical Education: Best Practices and Future Directions
Thompson, R. A. M., Shah, Y., Aguirre, F., Stewart, C., Lallas, C. D., & Shah, M. S. (2025) · Current Urology Reports · doi:10.1007/s11934-025-01277-1 · Cited by 46
“The increased popularity of ChatGPT and other AI chat-bots among medical students suggests that medical school administrators should find ways to embrace their utility in better preparing students to write for standardized exams and clinical rotations [15].”
Yang, Z., Yao, Z., Tasmin, M., Vashisht, P., Jang, W. S., Ouyang, F., Wang, B., Berlowitz, D., & Yu, H. (2023) · medRxiv · doi:10.1101/2023.10.26.23297629 · Cited by 36
“17 ChatGPT has achieved commendable results in the United States Medical Licensing Examinations, leading to discussions about the readiness of LLM applications for integration into clinical 18–20 , and educational 21–23 environments.”
BackgroundEvaluation of ChatGPT and Google Bard using prompt engineering in cancer screening algorithms
Nguyen, D., Swanson, D., Newbury, A., & Kim, Y. H. (2024) · Academic Radiology · doi:10.1016/j.acra.2023.11.002 · Cited by 38
Mishra, A., Begley, S. L., Chen, A., Rob, M., Pelcher, I., Ward, M., & Schulder, M. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002598 · Cited by 41
“7,8 This is a potent and widely applicable technology that can synthesize and process large amounts of information for the user and modify responses based on user input.”
BackgroundExploring the Latest Applications of OpenAI and ChatGPT: An In-Depth Survey
Hong, Z., & Shao, H. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.030649 · Cited by 47
Role of ChatGPT in health science and research: A correspondence addressing potential application
Islam, M. R., Urmi, T. J., Mosharrafa, R. A., Rahman, M. S., & Kadir, M. F. (2023) · Health Science Reports · doi:10.1002/hsr2.1625 · Cited by 43
“12 the HIM workforce.”
Adopting artificial intelligence in cardiovascular medicine: a scoping review
Makimoto, H., & Kohro, T. (2023) · Hypertension Research · doi:10.1038/s41440-023-01469-7 · Cited by 47
“In the medicine fi eld, trials have also commenced, revealing that these models possess capabilities equivalent to passing national medical licensing examinations [151].”
ChatGPT in Language Education: Perceptions of Teachers - A Beneficial Tool or Potential Threat?
Widianingtyas, N., Mukti, T. W. P., & Silalahi, R. M. P. (2023) · Voices of English Language Education Society · doi:10.29408/veles.v7i2.20326 · Cited by 40
“Trained on vast amounts of text data, it has demonstrated impressive proficiency in understanding and generating coherent responses (Kung et al., 2023; Murphy, 2019; Lund and Wang, 2023).”
BackgroundRazdan, S., Siegal, A. R., Brewer, Y., Sljivich, M., & Valenzuela, R. J. (2023) · International Journal of Impotence Research · doi:10.1038/s41443-023-00797-z · Cited by 44
Evaluating the Current Ability of ChatGPT to Assist in Professional Otolaryngology Education
Zalzal, H. G., Cheng, J., & Shah, R. K. (2023) · OTO Open · doi:10.1002/oto2.94 · Cited by 39
The Basics of Artificial Intelligence in Nursing: Fundamentals and Recommendations for Educators
Schneidereith, T. A., & Thibault, J. (2023) · Journal of Nursing Education · doi:10.3928/01484834-20231006-03 · Cited by 37
Christy, M. N., Morris, M. T., Goldfarb, C. A., & Dy, C. J. (2023) · The Journal Of Hand Surgery · doi:10.1016/j.jhsa.2023.10.019 · Cited by 40
Chat bots to Virtual Tutors: An Overview of Chat GPT's Role in the Future of Education
Graefen, B., & Fazal, N. (2024) · Archives of Pharmacy Practice · doi:10.51847/touppjedsx · Cited by 28
Parmanto, B., Aryoyudanta, B., Soekinto, T. W., Setiawan, I. M. A., Wang, Y., Hu, H., Saptono, A., & Choi, Y. K. (2024) · JMIR Formative Research · doi:10.2196/54633 · Cited by 42
“…artificial intelligence (AI) [15-19] have resulted in the popularity and widespread use of large language models (LLMs), such as ChatGPT, captivating global interest with their ability to generate intelligent and context-aware responses to a wide spectrum of users’questions or prompts [18,20-22].”
BackgroundEvaluation of the performance of ChatGPT‐4 and ChatGPT‐4o as a learning tool in endodontics
Öztürk, E. A., Gökduman, C. T., & Çanakçı, B. C. (2025) · International Endodontic Journal · doi:10.1111/iej.14217 · Cited by 50
“The use of ChatGPT for education in medicine and dentistry has been widely evaluated (Ahmad et al., 2024; Antaki et al., 2023; Kung et al., 2023; Lian, 2024; Rokhshad et al., 2024; Suárez et al., 2024).”
InfluentialApplications of Artificial Intelligence in Psychiatry and Psychology Education: Scoping Review
Prégent, J., Chung, V.-H.-A., Adib, I. E., Désilets, M., & Hudon, A. (2025) · JMIR Medical Education · doi:10.2196/75238 · Cited by 44
“Recent research has switched to looking at how tools such as ChatGPT, GPT-4, and Claude might scaffold learning in medical education, whereas the majority of earlier applications were on patient-facing educational interventions (eg, mental health apps) [41].”
Dao, X.-Q., Le, N.-B., Phan, X.-D., & Ngo, B. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4473369 · Cited by 40
Performance of ChatGPT on the Chinese postgraduate examination for clinical medicine: survey study
Yu, P., Fang, C., Liu, X., Fu, W., Ling, J., Yan, Z., Jiang, Y., Cao, Z., Wu, M., Chen, Z., et al. (2024) · JMIR Medical Education · doi:10.2196/48514 · Cited by 37
Morreel, S., Verhoeven, V., & Mathysen, D. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000349 · Cited by 37
“Soon after the first bots became publicly available, higher medical education institutes started to report on their performance in medical exam simulations [3].”
BackgroundInnovating Personalized Nephrology Care: Exploring the Potential Utilization of ChatGPT
Miao, J., Thongprayoon, C., Suppadungsuk, S., Garcia Valencia, O. A., Qureshi, F., & Cheungpasitporn, W. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13121681 · Cited by 36
“%) for the United States Medical Licensing Examination (USMLE) [12,13]. been demonstrated to approach or even exceed the passing threshold (approximately 60”
BackgroundSahin, M. C., Sozer, A., Kuzucu, P., Turkmen, T., Sahin, M. B., Sozer, E., Tufek, O. Y., Nernekli, K., Emmez, H., & Celtikci, E. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107807 · Cited by 36
ChatGPT en Educación Superior: explorando sus potencialidades y sus limitaciones
Galli, M. G., & Kanobel, M. C. (2023) · Revista Educación Superior y Sociedad (ESS) · doi:10.54674/ess.v35i2.815 · Cited by 48
Performance of ChatGPT on Stage 1 of the Taiwanese medical licensing exam
Huang, C.-H., Hsiao, H.-J., Yeh, P.-C., Wu, K.-C., & Kao, C.-H. (2024) · DIGITAL HEALTH · doi:10.1177/20552076241233144 · Cited by 32
“In the work of Kung and colleagues, 9 where the questions from the United States Medical Licensing Examination (USMLE) in 2022 were used, ChatGPT yielded passing performance with moderate accuracy.”
MethodKhan, A. A., Yunus, R. A., Sohail, M., Rehman, T. A. Z. U., Saeed, S., Bu, Y., Jackson, C. D., Sharkey, A. M., Mahmood, F. U., & Matyal, R. (2024) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2024.01.032 · Cited by 42
Studying Self-Care with Generative AI Tools: Lessons for Design
Capel, T., Ploderer, B., Birčanin, F., Hanmer, S., Yates, J. P., Wang, J., Khor, K. L., Leong, T. W., Wadley, G., & Newcomb, M. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661614 · Cited by 44
The Roles of Social Perception and AI Anxiety in Individuals’ Attitudes Toward ChatGPT in Education
Wang, C., Li, X., Liang, Z., Sheng, Y., Zhao, Q., & Chen, S. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2365453 · Cited by 44
“Researchers across various domains consider ChatGPT a transformative technological advancement (Chow et al., 2023; Gilson et al., 2023; Ivanov & Soliman, 2023; Kung et al., 2023).”
Zhang, D., & Zhao, X. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108359 · Cited by 40
Understanding natural language: Potential application of large language models to ophthalmology
Yang, Z., Wang, D., Zhou, F., Song, D., Zhang, Y., Jiang, J., Kong, K., Liu, X., Qiao, Y., Chang, R. T., et al. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100085 · Cited by 50
Jiang, Q., Zhang, Y., Wei, W., & Gu, C. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100287 · Cited by 43
Using ChatGPT in Nursing: Scoping Review of Current Opinions
Zhou, Y. J., Li, S., Tang, X., He, Y.-C., Ma, H., Wang, A., Pei, R., & Piao, M. (2024) · JMIR Medical Education · doi:10.2196/54297 · Cited by 41
“…algorithms and natural language processing techniques, and trained with massive amounts of data from the internet, books, and articles, ChatGPT can automatically identify users’ inputs and generate appropriate responses to simulate the interactive dialogue and feedback process between humans [8].”
Metadata
- Authors
- Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-02-09
- Volume
- 2
- Issue
- 2
- Pages
- e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.