Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Open accessPublished version of a preprint
The DOI matches its Crossref record, with a small difference in title or year; 28 of the 28 references it cites were found in a registry or an index.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Kung et al. (2023) evaluated the performance of ChatGPT on the United States Medical Licensing Exam and demonstrated that the chatbot did not reach the passing threshold and achieved an overall accuracy of only 50% (Kung et al., 2023).”
From A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing (2025) · cites this paper as Kung et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [17]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI's capabilities, such as passing the United States Medical Licensing Exam [107], completing exams at the University of Minnesota Law School [108], and solving queries from Wharton School of Business tests [109], underscore its potential.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [107]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The HCI community has explored the application of LLMs for various healthcare practices, such as mental health support [66,68], health information seeking [81,117,128], and public health interventions [58]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
From DIALOGUE: A Generative AI-Based Pre–Post Simulation Study to Enhance Diagnostic Communication in Medical Students Through Virtual Type 2 Diabetes Scenarios (2025) · cites this paper as Kung et al., 2023
Experimental — accuracy not yet measured
52 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background
Cited by
Items 701–800 of 1,129
Feasibility and acceptability of ChatGPT generated radiology report summaries for cancer patients
Chung, E. M., Zhang, S. C., Nguyen, A. T., Atkins, K. M., Sandler, H. M., & Kamrava, M. (2023) · DIGITAL HEALTH · doi:10.1177/20552076231221620 · Cited by 45
Knoedler, L., Alfertshofer, M., Knoedler, S., Hoch, C. C., Funk, P. F., Cotofana, S., Maheta, B., Frank, K., Brébant, V., Prantl, L., et al. (2024) · JMIR Medical Education · doi:10.2196/51148 · Cited by 49
“For instance, Hoch et al [11] reported that ChatGPT answered 57% of facial surgery board certification test questions correctly, while Kung et al [12] used a limited set of USMLE test questions (USMLE Step 1: 119; USMLE Step 2 CK: 102; USMLE Step 3: 122) and found that ChatGPT achieved performance…”
BackgroundMaitland, A., Fowkes, R., & Maitland, S. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-080558 · Cited by 47
McCoy, L. G., Ng, F. Y. C., Sauer, C. M., Legaspi, K. E. Y., Jain, B., Gallifant, J., McClurkin, M., Hammond, A., Goode, D., Gichoya, J. W., et al. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06048-z · Cited by 63
“Even general-purpose models such as OpenAI’s ChatGPT have shown significant promise in these applications, as perhaps best captured in a highly publicized paper wherein the model successfully passed the United States Medical Licensing Examinations (USMLE) [1].”
BackgroundMethodZada, T., Tam, N., Barnard, F., Sittert, M. V., Bhat, V., & Rambhatla, S. (2025) · JMIR Formative Research · doi:10.2196/66207 · Cited by 60
“These methods have used the multiple-choice questions from the United States Medical Licensing Exam (USMLE) [51] to evaluate the capability of LLMs in attaining scores near the passing threshold of 60% [20,52].”
InfluentialChakraborty, C., Bhattacharya, M., Pal, S., Chatterjee, S., Das, A., & Lee, S. (2025) · Journal of Advanced Research · doi:10.1016/j.jare.2025.02.011 · Cited by 60
Riedel, M., Kaefinger, K., Stuehrenberg, A., Ritter, V., Amann, N., Graf, A., Recker, F., Klein, E., Kiechle, M., Riedel, F., et al. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1296615 · Cited by 49
ChatGPT for healthcare providers and patients: Practical implications within dermatology
Jin, J. Q., & Dobry, A. S. (2023) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2023.05.081 · Cited by 49
Impacts of generative artificial intelligence on the future of labor market: a systematic review
Salari, N., Beiromvand, M., Hosseinian-Far, A., Habibi, J., Babajani, F., & Mohammadi, M. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100652 · Cited by 49
“It is based on a versatile LLM trained with approximately 45 terabytes of human interaction data, and it can generate novel, meaningful sequences of text that did not previously exist (Kung et al., 2023).”
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30205 · Cited by 66
Ilgaz, H. B., & Çelik, Z. (2023) · Cureus · doi:10.7759/cureus.45301 · Cited by 49
“LLMs have the capacity to process and understand natural language data with extraordinary skill and are already used in a variety of applications such as chatbots, machine translation, and question-answering [1-3].”
Safaei, M., & Longo, J. (2024) · Digital Government Research and Practice · doi:10.1145/3604570 · Cited by 62
“The ChatGPT tool [115] was recently evaluated on the United States Medical Licensing Exam (USMLE) and demonstrated exceptional performance in passing all three medical exams, providing insightful explanations that showcased its advanced understanding of the material [81].”
BackgroundMethodEvaluating ChatGPT’s Decimal Skills and Feedback Generation in a Digital Learning Game
Nguyen, H. A., Stec, H., Hou, X., Di, S., & McLaren, B. M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42682-7_19 · Cited by 74
Daher, M., Koa, J., Boufadel, P., Singh, J., Fares, M. Y., & Abboud, J. A. (2023) · JSES International · doi:10.1016/j.jseint.2023.07.018 · Cited by 49
“1,4,6,15,27 However, this is not without limitations, such as the possibility of bias and transparency, healthcare cybersecurity, and data privacy, which would raise ethical questions.”
BackgroundneuroGPT-X: toward a clinic-ready large language model
Guo, E., Gupta, M., Sinha, S., Rössler, K., Tatagiba, M. S., Akagami, R., Al‐Mefty, O., Sugiyama, T., Stieg, P. E., Pickett, G. E., et al. (2023) · Journal of neurosurgery · doi:10.3171/2023.7.jns23573 · Cited by 54
Bushuven, S., Bentele, M., Bentele, S., Gerber, B., Bansbach, J., Ganter, J., Trifunovic-Koenig, M., & Ranisch, R. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-02019-x · Cited by 47
“Studies have investigated whether GPT-based models can pass medical licensing examinations [16, 17] or life support exams [18], with most finding that their performance falls below profes - sional benchmarks.”
BackgroundOn the Calibration of Large Language Models and Alignment
Zhu, C., Xu, B., Wang, Q., Zhang, Y., & Mao, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.654 · Cited by 111
Unraveling the landscape of large language models: a systematic review and future perspectives
Ding, Q., Ding, D., Wang, Y., Guan, C., & Ding, B. (2023) · Journal of Electronic Business & Digital Economics · doi:10.1108/jebde-08-2023-0015 · Cited by 58
Readability analysis of ChatGPT's responses on lung cancer
Gencer, A. (2024) · Scientific Reports · doi:10.1038/s41598-024-67293-2 · Cited by 52
“Many studies have been published on how successful ChatGPT can be in exams for medical students, physicians, and health professionals 10,39–41 .”
BackgroundChen, X., Wang, L., You, M., Liu, W., Fu, Y., Xu, J., Zhang, S., Chen, G., Li, K., & Li, J. (2024) · Journal of Medical Internet Research · doi:10.2196/58158 · Cited by 48
“Although benchmarks targeting general medical knowledge have been previously developed [22-24], recent research has suggested that these benchmarks are only preliminary indicators of medical knowledge.”
Interplay of rationality and morality in using ChatGPT for academic misconduct
Zhang, L., Amos, C., & Pentina, I. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2325023 · Cited by 48
“In addition to the demonstrated good performance on standard tests, such as the U.S. Medical Licensing Exam, bar exam, GRE, and psychology tests (Kung et al. 2023; Uludağ, Jiao, and Zhao 2023; Weiss 2023), ChatGPT attracts students and educators by its productivity and efficiency (Malik et al.…”
Generative Artificial Intelligence in Anatomic Pathology
Brodsky, V., Ullah, E., Bychkov, A., Song, A. H., Walk, E. E., Louis, P., Rasool, G., Singh, R. S., Mahmood, F., Bui, M. M., et al. (2025) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0215-ra · Cited by 48
Performance of ChatGPT on registered nurse license exam in Taiwan: a descriptive study
Huang, H. (2023) · Healthcare · doi:10.3390/healthcare11212855 · Cited by 48
“It achieved a score of 60.2% in the US Medical Licensing Examination (USMLE), 57.7% in Multiple-choice medical exam questions (MedMCQA), and 78.2% in the medical reading comprehension dataset (PubMedQA) [27,29].”
BackgroundInfluentialLarge language model agents can use tools to perform clinical calculations
Goodell, A. J., Chu, S. N., Rouholiman, D., & Chu, L. F. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01475-8 · Cited by 48
Capacity of ChatGPT to Identify Guideline-Based Treatments for Advanced Solid Tumors
Schulte, B. (2023) · Cureus · doi:10.7759/cureus.37938 · Cited by 50
“Recent literature has indicated that ChatGPT may have the ability to pass modern licensing examinations, while other models have conquered complex communication-oriented strategy games [6, 13].”
BackgroundYou, Y., Chen, Y., You, Y., Zhang, Q., & Cao, Q. (2023) · Sustainability · doi:10.3390/su15129355 · Cited by 53
“On th other hand, these AI chatbots can help students to complete homework faster, allowing them to focus on being creative and innovative [22].”
BackgroundAssessing ChatGPT's ability to pass the FRCS orthopaedic part A exam: A critical analysis
Saad, A., Iyengar, K. P., Kurisunkal, V., & Botchu, R. (2023) · The Surgeon · doi:10.1016/j.surge.2023.07.001 · Cited by 49
Hung, Y.-C., Chaker, S. C., Sigel, M., Saad, M., & Slater, E. D. (2023) · Annals of Plastic Surgery · doi:10.1097/sap.0000000000003634 · Cited by 43
Dianova, V. G., & Schultz, M. D. (2023) · Industry and Higher Education · doi:10.1177/09504222231199989 · Cited by 36
“5 technology, generated signi fi cant academic interest in possible applications of the new technology in policymaking and a variety of industries such as manufacturing, construction, and healthcare (Baldini et al., 2023; Biswas, 2023; Kung et al., 2023; Prieto et al., 2023; Rathore, 2023).”
BackgroundPark, J. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.29 · Cited by 39
Could ChatGPT Pass the UK Radiology Fellowship Examinations?
Ariyaratne, S., Jenko, N., Davies, A. M., Iyengar, K. P., & Botchu, R. (2023) · Academic Radiology · doi:10.1016/j.acra.2023.11.026 · Cited by 50
Generative AI as Virtual Healthcare Assistant for Enhancing Patient Care Quality
Samala, A. D., & Rawas, S. (2024) · International Journal of Online and Biomedical Engineering (iJOE) · doi:10.3991/ijoe.v20i05.45937 · Cited by 42
Zhu, L., Mou, W., Lai, Y., Chen, J., Lin, S., Xu, L., Lin, J., Guo, Z., Yang, T., Lin, A., et al. (2024) · International Journal of Surgery · doi:10.1097/js9.0000000000001359 · Cited by 52
Hirano, Y., Hanaoka, S., Nakao, T., Miki, S., Kikuchi, T., Nakamura, Y., Nomura, Y., Yoshikawa, T., & Abe, O. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-024-01561-z · Cited by 50
“5 model, attained scores above or close to passing thresholds in the United States Medical Licensing Examination (USMLE) [4].”
BackgroundPerformance of ChatGPT 3.5 and 4 on U.S. dental examinations: the INBDE, ADAT, and DAT
Dashti, M., Ghasemi, S., Ghadimi, N., Hefzi, D., Karimian, A., Zare, N., Fahimipour, A., Khurshid, Z., Chafjiri, M. M., & Ghaedsharaf, S. (2024) · Imaging Science in Dentistry · doi:10.5624/isd.20240037 · Cited by 53
Enlarged Education – Exploring the Use of Generative AI to Support Lecturing in Higher Education
Hennekeuser, D., Vaziri, D. D., Golchinfar, D., Schreiber, D., & Stevens, G. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00424-y · Cited by 72
“Additionally, the potential of LLMs in medical education is explored by Kung et al. ( 2023 ).”
Evaluation of Rhinoplasty Information from ChatGPT, Gemini, and Claude for Readability and Accuracy
Rossi-Meyer, M. K., Kandathil, C. K., Davis, S. J., Durairaj, K. K., Patel, P. N., Pepper, J., Spataro, E. A., & Most, S. P. (2024) · Aesthetic Plastic Surgery · doi:10.1007/s00266-024-04343-0 · Cited by 53
Large Language Models in Dental Licensing Examinations: Systematic Review and Meta-Analysis
Liu, M., Okuhara, T., Huang, W., Ogihara, A., Nagao, H. S., Okada, H., & Kiuchi, T. (2024) · International Dental Journal · doi:10.1016/j.identj.2024.10.014 · Cited by 62
Gupta, N., Khatri, K., Malik, Y., Lakhani, A., Kanwal, A., Aggarwal, S., & Dahuja, A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06592-8 · Cited by 58
“5 and GPT-4 obtained approximately 60% and 86.1%, respectively [45, 46].”
InfluentialKortemeyer, G., & Nöhl, J. (2025) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.21.010136 · Cited by 64
“Consequently, AI is increasingly integrated into physics education [5–7], including aiding educational research [8], solving problems [9–13], constructing new problems [14], and promising preliminary results for grading problem solutions [15–19].”
Adiyono, A., Suwartono, T., Nurhayati, S., Dalimarta, F. F., & Wijayanti, O. (2025) · International Journal of Learning Teaching and Educational Research · doi:10.26803/ijlter.24.3.22 · Cited by 59
MEDCO: Medical Education Copilots Based on a Multi-agent Framework
Wei, H., Qiu, J., Yu, H., & Yuan, W. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91813-1_8 · Cited by 77
Human–large language model collaboration in clinical medicine: a systematic review and meta-analysis
Wang, G., Zhang, K., Jiang, J., Wang, C., Bi, H., Liang, H., Qi, Z., Huang, Y., Li, Y., & Yang, X. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02382-2 · Cited by 67
Mastering medical terminology with ChatGPT and Termbot
Hsu, M.-H. (2024) · Health Education Journal · doi:10.1177/00178969231197371 · Cited by 47
Kumah-Crystal, Y., Mankowitz, S., Embi, P., & Lehmann, C. U. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad104 · Cited by 44
ChatGPT in Nuclear Medicine Education
Currie, G., & Barry, K. (2023) · Journal of Nuclear Medicine Technology · doi:10.2967/jnmt.123.265844 · Cited by 47
“Our findings are consistent with those reported for medical examinations, for which ChatGPT scored 43%–68% in open-ended questions and 40%–65% in multiple-choice questions (7,8).”
ResultBPPV Information on Google Versus AI (ChatGPT)
Bellinger, J. R., De La Chapa, J. S., Kwak, M. W., Ramos, G. A., Morrison, D., & Kesser, B. W. (2024) · Otolaryngology–Head and Neck Surgery · doi:10.1002/ohn.506 · Cited by 44
Kufel, J., Paszkiewicz, I., Bielówka, M., Bartnikowska, W., Janik, M., Stencel, M., Czogalik, Ł., Gruszczyńska, K., & Mielcarska, S. (2023) · Polish Journal of Radiology · doi:10.5114/pjr.2023.131215 · Cited by 50
“The model achieved a score of 60%, equivalent to passing the exam [5].”
BackgroundAnswering head and neck cancer questions: An assessment of ChatGPT responses
Wei, K., Fritz, C., & Rajasekaran, K. (2024) · American Journal of Otolaryngology · doi:10.1016/j.amjoto.2023.104085 · Cited by 46
Kunitsu, Y. (2023) · JMIR Medical Education · doi:10.2196/48452 · Cited by 43
“Furthermore, it has been reported that ChatGPT and GPT-4 could correctly answer questions on the United States Medical Licensing Examination (USMLE) [7,8] and the Japanese National Medical Licensing Examination [9] at a passing level, despite not having specialized in a particular field of study.”
BackgroundGenerative Language Models and Open Notes: Exploring the Promise and Limitations
Blease, C., Torous, J., McMillan, B., Hägglund, M., & Mandl, K. D. (2024) · JMIR Medical Education · doi:10.2196/51183 · Cited by 42
“Thus, while a study in March 2023 showed that ChatGPT (version 3) Could pass the United States Medical Licensing Examination [52], the authors of the study noted that to truly assess the potential of such LLMs, there is a need for “controlled and real-world learning scenarios with students across…”
BackgroundChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions
Buhr, C. R., Smith, H., Huppertz, T., Bahr-Hamm, K., Matthias, C., Blaikie, A., Kelsey, T., Kuhn, S., & Eckrich, J. (2023) · JMIR Medical Education · doi:10.2196/49183 · Cited by 45
“The fact that ChatGPT has been used as a tool to pass the United States Medical Licensing Examination (USMLE) demonstrates that LLMs can accurately answer medical questions [21,22].”
MethodZandi, R., Fahey, J. D., Drakopoulos, M., Bryan, J. M., Dong, S., Bryar, P. J., Bidwell, A. E., Bowen, R. C., Lavine, J. A., & Mirza, R. G. (2024) · Bioengineering · doi:10.3390/bioengineering11020120 · Cited by 52
“5 performed at or near passing for all three exams in the United States Medical Licensing Exam (USMLE) series [7].”
BackgroundChatGPT: An Automated Teacher's Guide to Learning
Kazi, K. S. L. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-4268-8.ch001 · Cited by 52
Schwieger, A., Angst, K., de Bardeci, M., Burrer, A., Cathomas, F., Ferrea, S., Grätz, F., Knorr, M. S., Kronenberg, G., Spiller, T. R., et al. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105654 · Cited by 56
Safety analysis in the era of large language models: A case study of STPA using ChatGPT
Qi, Y., Zhao, X., Khastgir, S., & Huang, X. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100622 · Cited by 69
Can Artificial Intelligence Language Models Effectively Address Dental Trauma Questions?
Kuru, E., Aşık, A., & Demir, D. M. (2025) · Dental Traumatology · doi:10.1111/edt.13063 · Cited by 52
“The existing literature has seen a limited number of studies that evaluate the performance of these LLMs in response to different medical examinations [13, 14] and specific questions related to the field of dentistry [5].”
Performance of ChatGPT on free-response, clinical reasoning exams
Strong, E., DiGiammarino, A., Weng, Y., Basaviah, P., Hosamani, P., Kumar, A., Nevins, A., Kugler, J., Hom, J., & Chen, J. H. (2023) · medRxiv · doi:10.1101/2023.03.24.23287731 · Cited by 48
ChatGPT, Large Language Models, and Generative AI as Future Augments of Surgical Cancer Care
Kothari, A. N. (2023) · Annals of Surgical Oncology · doi:10.1245/s10434-023-13442-2 · Cited by 51
Academic Writing with GPT-3.5 (ChatGPT): Reflections on Practices, Efficacy and Transparency
Buruk, O. '. (2023) · 26th International Academic Mindtrek Conference · doi:10.1145/3616961.3616992 · Cited by 46
Generative AI in Undergraduate Medical Education: A Rapid Review
Hale, J., Alexander, S., Wright, S. T., & Gilliland, K. (2024) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205241266697 · Cited by 46
Can ChatGPT improve communication in hospitals?
Santandreu-Calonge, D., Medina-Aguerrebere, P., Hultberg, P., & Shah, M.-A. (2023) · El Profesional de la información · doi:10.3145/epi.2023.mar.19 · Cited by 39
Levartovsky, A., Ben-Horin, S., Kopylov, U., Klang, E., & Barash, Y. (2023) · American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002483 · Cited by 43
Liu, H. Y., Bonetti, M. A., Arellano, J. A., & Egro, F. M. (2023) · Aesthetic Plastic Surgery · doi:10.1007/s00266-023-03709-0 · Cited by 48
Evaluating GPT-4-based ChatGPT's clinical potential on the NEJM quiz
Ueda, D., Walston, S. L., Matsumoto, T., Deguchi, R., Tatekawa, H., & Miki, Y. (2024) · BMC Digital Health · doi:10.1186/s44247-023-00058-5 · Cited by 44
“A previous study applied GPT-3-based ChatGPT to the United States Medical Licensing Examination and found that it achieved 60% accuracy [11].”
MethodHan, C., Kim, D. W., Kim, S., You, S. C., Park, J. Y., Bae, S., & Yoon, D. (2024) · iScience · doi:10.1016/j.isci.2024.109022 · Cited by 62
“5-turbo 8,24 ; GPT provided largely accurate responses to 284 medical questions posed by physicians across 17 specialties, as evaluated by academic physician specialists, and also provided largely appropriate responses to CVD prevention questions as evaluated by cardiology clinicians 10,25 ; when…”
BackgroundDetecting the corruption of online questionnaires by artificial intelligence
Lebrun, B., Temtsin, S., Vonasch, A. J., & Bartneck, C. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1277635 · Cited by 54
“Some LLMs have been able to pass academic examinations in various areas, such as computer science (Bordt and von Luxburg, 2023) or medicine (Gilson et al., 2022; Kung et al., 2023).”
ChatGPT: Literate or intelligent about UN sustainable development goals?
Raman, R., Lathabai, H. H., Mandal, S., Das, P., Kaur, T., & Nedungadi, P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0297521 · Cited by 52
“Consequently, Kung et al. [34] analyzed ChatGPT’s competence on the United States Medical Licensing Exam (USMLE) in comparison to human clinical experts.”
BackgroundHow to mitigate the risks of deployment of artificial intelligence in medicine?
İlikhan, S. U., Özer, M., Tanberkan, H., & Bozkurt, V. (2024) · TURKISH JOURNAL OF MEDICAL SCIENCES · doi:10.55730/1300-0144.5814 · Cited by 54
“On the other hand, especially with the development and continuous updates of ChatGPT, AI systems now offer significant opportunities in preparing for medical education and competency exams [22,24].”
BackgroundLong, C., Subburam, D., Lowe, K., dos Santos, A., Zhang, J., Hwang, S., Saduka, N., Horev, Y., Su, T., Côté, D. W. J., et al. (2024) · Otolaryngology–Head and Neck Surgery · doi:10.1002/ohn.864 · Cited by 46
Development of AI-generated medical responses using the ChatGPT for cancer patients
Lee, J.-W., Yoo, I.-S., Kim, J.-H., Kim, W. T., Jeon, H. J., Yoo, H.-S., Shin, J. G., Kim, G.-H., Hwang, S., Park, S., et al. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108302 · Cited by 38
AI Adoption for Collaboration: Factors Influencing Inclusive Learning Adoption in Higher Education
Alyoussef, I. Y., Drwish, A. M., Albakheet, F. A., Alqahtani, R. H. A., & Technology, A. A. A. E. (2025) · IEEE Access · doi:10.1109/access.2025.3567656 · Cited by 58
“Kung et al. [161] noted that while AI-driven engagement improves motivation, its impact on actual learning outcomes may vary by context and implementation design.”
Angel, M. C., Rinehart, J. B., Canneson, M. P., & Baldi, P. (2024) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000006892 · Cited by 50
“5,15 These initial results suggest that LLMs have the potential to transform both medical education and practice.”
Abedi, M., Alshybani, I., Shahadat, M., & Murillo, M. S. (2023) · Qeios · doi:10.32388/md04b0 · Cited by 69
Evaluating GPT-4-based ChatGPT’s Clinical Potential on the NEJM Quiz
Ueda, D., Walston, S. L., Matsumoto, T., Deguchi, R., Tatekawa, H., & Miki, Y. (2023) · doi:10.1101/2023.05.04.23289493 · Cited by 45
Dubinski, D., Won, S.-Y., Trnovec, S., Behmanesh, B., Baumgarten, P., Dinc, N., Konczalla, J., Chan, A., Bernstock, J. D., Freiman, T. M., et al. (2024) · Acta Neurochirurgica · doi:10.1007/s00701-024-05908-3 · Cited by 45
Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians
Hadi, A., Tran, E., Nagarajan, B., & Kirpalani, A. (2024) · PLOS ONE · doi:10.1371/journal.pone.0307383 · Cited by 45
Assessing the Performance of GPT-3.5 and GPT-4 on the 2023 Japanese Nursing Examination
Kaneda, Y., Takahashi, R., Kaneda, U., Akashima, S., Okita, H., Misaki, S., Yamashiro, A., Ozaki, A., & Tanimoto, T. (2023) · Cureus · doi:10.7759/cureus.42924 · Cited by 45
Solving the Self-regulated Learning Problem: Exploring the Performance of ChatGPT in Mathematics
Li, P., Lee, H.-Y., Cheng, Y., Starčič, A. I., & Huang, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40113-8_8 · Cited by 59
Aljindan, F. K., Al Qurashi, A. A., Albalawi, I. A. S., Alanazi, A. M. M., Aljuhani, H. A. M., Falah Almutairi, F., Aldamigh, O. A., Halawani, I. R., & K. Zino Alarki, S. M. (2023) · Cureus · doi:10.7759/cureus.45043 · Cited by 47
“Our observations are consistent with the findings of Kung et al. [11], who conducted a similar evaluation of ChatGPT’s performance but focused on the United States Medical Licensing Examination (USMLE).”
ResultGray, M., Baird, A., Sawyer, T., James, J., DeBroux, T., Bartlett, M., Krick, J., & Umoren, R. (2024) · JMIR Medical Education · doi:10.2196/50705 · Cited by 36
Garabet, R., Mackey, B. P., Cross, J., & Weingarten, M. S. (2023) · Medical Science Educator · doi:10.1007/s40670-023-01956-z · Cited by 48
Lv, X., Zhang, X., Li, Y., Ding, X., Lai, H., & Shi, J. (2024) · Journal of Medical Internet Research · doi:10.2196/55847 · Cited by 52
“5 performed at or near the passing threshold for the United States Medical Licensing Examination [6].”
Zhu, L., Mou, W., Lai, Y., Lin, J., & Luo, P. (2024) · Journal of Translational Medicine · doi:10.1186/s12967-024-05128-4 · Cited by 48
“ChatGPT, a well-known example of LLMs developed by OpenAI, has demonstrated powerful capabilities across many domains, including passing various standardized medical qualification exams such as the United States Medical Licensing Examination (USMLE) [1].”
BackgroundChatGPT, the perfect virtual teaching assistant? Ideological bias in learner-chatbot interactions
Poucke, M. V. (2024) · Computers & composition/Computers and composition · doi:10.1016/j.compcom.2024.102871 · Cited by 48
Analysis of ChatGPT Responses to Ophthalmic Cases: Can ChatGPT Think like an Ophthalmologist?
Chen, J., Reddy, A. J., Al-Sharif, E. M., Shoji, M. K., Kalaw, F. G. P., Eslani, M., Lang, P. Z., Arya, M., Koretz, Z. A., Bolo, K. A., et al. (2024) · Ophthalmology Science · doi:10.1016/j.xops.2024.100600 · Cited by 53
Strzalkowski, P., Strzalkowska, A., Chhablani, J. K., Pfau, K., Errera, M., Roth, M., Schaub, F., Bechrakis, N. E., Hoerauf, H., Reiter, C., et al. (2024) · International Journal of Retina and Vitreous · doi:10.1186/s40942-024-00579-9 · Cited by 50
“The ability of ChatGPT-4 to respond to questions about medical examinations, including those related to ophthalmology [20, 21], has been the subject of great interest and has been investigated in several studies [22, 23].”
BackgroundThe impact of generative AI on management innovation
Zhang, C., & Zhang, H. (2024) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2024.100767 · Cited by 60
Esmaeilzadeh, P., Maddah, M., & Mirzaei, T. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100127 · Cited by 60
Shan, G., & Qiu, L. (2025) · Information Systems Research · doi:10.1287/isre.2023.0332 · Cited by 79
Comprehensive Review of Natural Language Processing (NLP) in Vascular Surgery
Lareyre, F., Nasr, B., Chaudhuri, A., Lorenzo, G. D., Carlier, M., & Raffort, J. (2023) · EJVES Vascular Forum · doi:10.1016/j.ejvsvf.2023.09.002 · Cited by 50
Application of ChatGPT for Orthopedic Surgeries and Patient Care
Morya, V. K., Lee, H.-W., Shahid, H., Magar, A. G., Lee, J.-H., Kim, J.-H., Jun, L., & Noh, K.-C. (2024) · Clinics in Orthopedic Surgery · doi:10.4055/cios23181 · Cited by 44
Use of Large Language Models to Predict Neuroimaging
Nazario-Johnson, L., Zaki, H. A., & Tung, G. A. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.06.008 · Cited by 40
Artificial intelligence in the neonatal intensive care unit: the time is now
Beam, K., Sharma, P., Levy, P. T., & Beam, A. L. (2023) · Journal of Perinatology · doi:10.1038/s41372-023-01719-z · Cited by 61
“…learning techniques known as large language models (LLMs), which includes OpenAI ’ s ChatGPT [55, 56] and Google ’ s Bard [57], have demonstrated the capacity for complex medical reasoning [56, 58], providing compelling evidence of their potential to pass various medical licensing exams [59, 60].”
Can ChatGPT pass the thoracic surgery exam?
Gencer, A., & Aydin, S. (2023) · The American Journal of the Medical Sciences · doi:10.1016/j.amjms.2023.08.001 · Cited by 46
A descriptive study based on the comparison of ChatGPT and evidence-based neurosurgeons
Liu, J., Zheng, J., Cai, X., Wu, D., & Yin, C. (2023) · iScience · doi:10.1016/j.isci.2023.107590 · Cited by 44
“T. H. Kung et al.20 found that ChatGPT performed well in the United StatesMedical Licensing Exam (USMLE) without specialized training or reinforcement.”
AI Chatbots in Clinical Laboratory Medicine: Foundations and Trends
Yang, H. S., Wang, F., Greenblatt, M. B., Huang, X., & Zhang, Y. (2023) · Clinical Chemistry · doi:10.1093/clinchem/hvad106 · Cited by 56
A historical perspective of biomedical explainable AI research
Malinverno, L., Barros, V., Ghisoni, F., Visonà, G., Kern, R., Nickel, P. J., Ventura, B. E., Šimić, I., Stryeck, S., Manni, F., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100830 · Cited by 56
“62 Hence, it might become a tool to explain AI-based approaches and support AI-based decisions.”
MethodGonzález-Alday, R., García-Cuesta, E., Maojo, & Kulikowski, C. (2023) · Applied Sciences · doi:10.3390/app131910778 · Cited by 54
Understanding the brain with attention: A survey of transformers in brain sciences
Chen, C., Wang, H., Chen, Y., Yin, Z., Yang, X., Ning, H., Zhang, Q., Li, W., Xiao, R., & Zhao, J. (2023) · Brain‐X · doi:10.1002/brx2.29 · Cited by 47
Metadata
- Authors
- Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-02-09
- Volume
- 2
- Issue
- 2
- Pages
- e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.