Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Open accessPublished version of a preprint
The DOI matches its Crossref record, with a small difference in title or year; 28 of the 28 references it cites were found in a registry or an index.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Kung et al. (2023) evaluated the performance of ChatGPT on the United States Medical Licensing Exam and demonstrated that the chatbot did not reach the passing threshold and achieved an overall accuracy of only 50% (Kung et al., 2023).”
From A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing (2025) · cites this paper as Kung et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [17]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI's capabilities, such as passing the United States Medical Licensing Exam [107], completing exams at the University of Minnesota Law School [108], and solving queries from Wharton School of Business tests [109], underscore its potential.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [107]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The HCI community has explored the application of LLMs for various healthcare practices, such as mental health support [66,68], health information seeking [81,117,128], and public health interventions [58]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
From DIALOGUE: A Generative AI-Based Pre–Post Simulation Study to Enhance Diagnostic Communication in Medical Students Through Virtual Type 2 Diabetes Scenarios (2025) · cites this paper as Kung et al., 2023
Experimental — accuracy not yet measured
52 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background
Cited by
Items 501–600 of 1,129
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.28 · Cited by 50
The Impact of AI Chatbot-Based Learning on Students’ Motivation in English Writing Classroom
Silitonga, L. M., Hawanti, S., Aziez, F., Furqon, M., Zain, D. S. M., Anjarani, S., & Wu, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40113-8_53 · Cited by 82
Performance of ChatGPT on Nephrology Test Questions
Miao, J., Thongprayoon, C., Garcia Valencia, O. A., Krisanapan, P., Sheikh, M. S., Davis, P. W., Mekraksakit, P., Suarez, M. G., Craici, I. M., & Cheungpasitporn, W. (2024) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.0000000000000330 · Cited by 68
“7,8,11,13,14”
Artificial Intelligence in Endodontic Education
Aminoshariae, A., Nosrat, A., Nagendrababu, V., Dianat, O., Mohammad‐Rahimi, H., O'Keefe, A. W., & Setzer, F. C. (2024) · Journal of Endodontics · doi:10.1016/j.joen.2024.02.011 · Cited by 98
A Survey on Human–AI Collaboration with Large Foundation Models
Vats, V., Nizam, M. B., Liu, M., Wang, Z., Ho, R. C., Prasad, M. S., Titterton, V., Malreddy, S. V., Aggarwal, R., Xu, Y., et al. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3841472 · Cited by 74
Truhn, D., Weber, C. D., Braun, B. J., Bressem, K., Kather, J. N., Kuhl, C., & Nebelung, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-47500-2 · Cited by 71
Students' reflections on their experience with ChatGPT
Šedlbauer, J., Činčera, J., Slavík, M., & Hartlová, A. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12967 · Cited by 71
Sabri, H., Saleh, M. H. A., Hazrati, P., Merchant, K., Misch, J., Kumar, P. S., Wang, H., & Barootchi, S. (2025) · Journal of Periodontal Research · doi:10.1111/jre.13323 · Cited by 71
Analyzing the User's Sentiments of ChatGPT Using Twitter Data
KORKMAZ, A., AKTÜRK, C., & TALAN, T. (2023) · Iraqi Journal for Computer Science and Mathematics · doi:10.52866/ijcsm.2023.02.02.018 · Cited by 63
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
Balloccu, S., Schmidtová, P., Lango, M., & Dusek, O. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.5 · Cited by 70
Charting the Future of AI in Project-Based Learning: A Co-Design Exploration with Students
Zheng, C., Yuan, K., Guo, B., Hadi Mogavi, R., Peng, Z., Ma, S., & Ma, X. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642807 · Cited by 70
ChatGPT performance in the medical specialty exam: An observational study
Oztermeli, A. D., & Oztermeli, A. (2023) · Medicine · doi:10.1097/md.0000000000034673 · Cited by 70
“This situation has raised questions about the reliability of ChatGPT.”
MethodResultInfluentialGPT-aided diagnosis on agricultural image based on a new light YOLOPC
Qing, J., Deng, X., Lan, Y., & Li, Z. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108168 · Cited by 77
Chu, C. P. (2024) · Frontiers in Veterinary Science · doi:10.3389/fvets.2024.1395934 · Cited by 83
Kuang, E., Li, M., Fan, M., & Shinohara, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642168 · Cited by 86
Examining the Role of Large Language Models in Orthopedics: Systematic Review
Zhang, C., Liu, S., Zhou, X., Zhou, S., Tian, Y., Wang, S., Xu, N., & Li, W. (2024) · Journal of Medical Internet Research · doi:10.2196/59607 · Cited by 73
“When answering the 2022 United States Medical Licensing Examination questions, without further training or reinforcement, ChatGPT reached or approached a passing level in all 3 examinations [5].”
Şişmanoğlu, S., & Çapan, B. Ş. (2025) · BMC Medical Education · doi:10.1186/s12909-024-06389-9 · Cited by 83
ChatGPT in ophthalmology: the dawn of a new era?
Ting, D. S. J., Tan, T. F., & Ting, D. S. W. (2023) · Eye · doi:10.1038/s41433-023-02619-4 · Cited by 76
“However, its readiness to be deployed as an automatic decision-making tool in medical fi eld remains to be elucidated [5, 6, 27].”
Charting the Future of AI in Project-Based Learning: A Co-Design Exploration with Students
Zheng, C., Yuan, K., Guo, B., Mogavi, R. H., Peng, Z., Ma, S., & Ma, X. (2024) · arXiv · doi:10.48550/arxiv.2401.14915 · Cited by 70
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
The Emerging Role of Generative Artificial Intelligence in Medical Education, Research, and Practice
Shoja, M. M., Van de Ridder, J. M. M., & Rajput, V. (2023) · Cureus · doi:10.7759/cureus.40883 · Cited by 69
“The ChatGPT can generate meaningful, credible, and novel word sequences that the model has never encountered before [8].”
ChatGPT’s Response Consistency: A Study on Repeated Queries of Medical Examination Questions
Funk, P. F., Hoch, C. C., Knoedler, S., Knoedler, L., Cotofana, S., Sofo, G., Bashiri Dezfouli, A., Wollenberg, B., Guntinas-Lichius, O., & Alfertshofer, M. (2024) · European Journal of Investigation in Health, Psychology and Education · doi:10.3390/ejihpe14030043 · Cited by 69
“More specifically, Kung et al. demonstrated that ChatGPT 3.5 was unable to achieve a passing score for the Step 1 and Step 2CK examinations, scoring 55.8% and 59.1%, respectively, while it managed a passing score of 61.3% for the Step 3 examination [14].”
MethodInfluentialExploring the Possible Use of AI Chatbots in Public Health Education: Feasibility Study
Baglivo, F., De Angelis, L., Casigliani, V., Arzilli, G., Privitera, G. P., & Rizzo, C. (2023) · JMIR Medical Education · doi:10.2196/51421 · Cited by 69
“The authors suggested that, based on this result, ChatGPT may potentially support students in the medical education field [8].”
BackgroundA bibliometric analysis of the rise of ChatGPT in medical research
Barrington, N. M., Gupta, N., Musmar, B., Doyle, D., Panico, N., Godbole, N., Reardon, T., & D’Amico, R. S. (2023) · Medical Sciences · doi:10.3390/medsci11030061 · Cited by 69
“With its remarkable abilities, ChatGPT has already demonstrated success in passing specialized exams [17–19], including medical board exams [2], and has been credited with authorship on research articles [20], forcing educators and publishers to rapidly evolve teaching methods and publication policies to keep up with…”
BackgroundKumari, A., Kumari, A., Singh, A., Singh, S. K., Juhi, A., Dhanvijay, A. K. D., Pinjar, M. J., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.43861 · Cited by 62
“These models have demonstrated various levels of accuracy in performing medical examinations, solving complex medical issues, or interpreting radiology reports [4,5,6,7,8].”
BackgroundWu, R., Yu, C., Pan, X., Liu, Y., Zhang, N., Fu, Y., Wang, Y., Zheng, Z., Chen, L. Z., Jiang, Q., et al. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642790 · Cited by 87
“Recent studies have explored various health applications supported by LLMs, such as health information seeking [91], mental health support [47, 51], personal health coaching [62, 83], and health education [48, 62].”
BackgroundGenerative artificial intelligence in graduate medical education
Janumpally, R., Nanua, S., Ngo, A., & Youens, K. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2024.1525604 · Cited by 54
Hadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 53
Many bioinformatics programming tasks can be automated with ChatGPT
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S., & Ridge, P. G. (2023) · arXiv · doi:10.48550/arxiv.2303.13528 · Cited by 69
ChatGPT for Academic Purposes: Survey Among Undergraduate Healthcare Students in Malaysia
George Pallivathukal, R., Kyaw Soe, H. H., Donald, P. M., Samson, R. S., & Hj Ismail, A. R. (2024) · Cureus · doi:10.7759/cureus.53032 · Cited by 68
“…the accuracy of ChatGPT among the medical and dental educational fraternity, it is noteworthy that the healthcare students' perceptions regarding its utility for academic purposes received limited attention within the realm of research [2,17,18], particularly in the context of Southeast Asia.”
BackgroundTirado‐Olivares, S., Navío-Inglés, M., O’Connor-Jiménez, P., & Gutiérrez, R. C. (2023) · Education Sciences · doi:10.3390/educsci13080803 · Cited by 80
Mhlanga, D. (2023) · Education Research International · doi:10.1155/2023/7605075 · Cited by 75
“[31] have evaluated ChatGPT’s performance on the United States Medical Licensing Exam, demonstrating its ability to achieve or approach passing scores across all three assessments without specialized training.”
MethodCORAL: Expert-Curated Oncology Reports to Advance Language Model Inference
Sushil, M., Kennedy, V. E., Mandair, D., Miao, B. Y., Zack, T., & Butte, A. J. (2024) · NEJM AI · doi:10.1056/aidbp2300110 · Cited by 64
Answering real-world clinical questions using large language model based systems
Low, Y. S., Jackson, M. L., Hyde, R. J., Brown, R. E., Sanghavi, N. M., Baldwin, J. D., Pike, C. W., Muralidharan, J., Hui, G., Alexander, N., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.00541 · Cited by 68
“While LLMs have displayed impressive performance in responding to natural language queries in various medical domains 13,14 , they are prone to hallucinating reference materials or treatment guidelines 15,16 and may produce “recommendations” that are satirical or inappropriate 17 .”
BackgroundGhosh, A., & Bir, A. (2023) · Cureus · doi:10.7759/cureus.37023 · Cited by 67
“Moreover, ChatGPT demonstrated clear, logical thinking and provided accurate clinical insights in its responses [16].”
BackgroundEvaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment
Lai, U. H., Wu, K. S., Hsu, T.-Y., & Kan, J. K. C. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1240915 · Cited by 67
Böhm, R., Jörling, M., Reiter, L., & Fuchs, C. (2023) · Communications Psychology · doi:10.1038/s44271-023-00032-x · Cited by 49
“Other research evaluated the performance of ChatGPT on the United States Medical Licensing Exam (USMLE) and found that the tool performed near or at the passing threshold 8 .”
Strengths, Weaknesses, Opportunities, and Threats of Using ChatGPT in Scientific Research
Giray, L., Jacob, J., & Gumalin, D. L. (2024) · International Journal of Technology in Education · doi:10.46328/ijte.618 · Cited by 44
Student Interaction with NewtBot: An LLM-as-tutor Chatbot for Secondary Physics Education
Lieb, A., & Goel, T. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647957 · Cited by 69
Reasoning with large language models for medical question answering
Lucas, M. M., Yang, J., Pomeroy, J. K., & Yang, C. C. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae131 · Cited by 66
Evaluation of large language models as a diagnostic aid for complex medical cases
Ríos-Hoyo, A., Shan, N. L., Li, A., Pearson, A. T., Pusztai, L., & Howard, F. M. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1380148 · Cited by 66
Tran, V.-T., Gartlehner, G., Yaacoub, S., Boutron, I., Schwingshackl, L., Stadelmaier, J., Sommer, I., Alebouyeh, F., Afach, S., Meerpohl, J., et al. (2024) · Annals of Internal Medicine · doi:10.7326/m23-3389 · Cited by 66
“The LLMs are able to answer health questions (13, 14), diagnose disease (15), and perform near the passing threshold for the examinations of the United States Medical Licensing Examination (16).”
BackgroundRahimzadeh, V., Kostick-Quenet, K., Blumenthal Barby, J., & McGuire, A. L. (2023) · The American Journal of Bioethics · doi:10.1080/15265161.2023.2233358 · Cited by 66
Thromboembolic prophylaxis in spine surgery: an analysis of ChatGPT recommendations
Duey, A. H., Nietsch, K. S., Zaidat, B., Ren, R., Ndjonko, L. C. M., Shrestha, N., Rajjoub, R., Ahmed, W., Hoang, T., Saturno, M. P., et al. (2023) · The Spine Journal · doi:10.1016/j.spinee.2023.07.015 · Cited by 65
Ito, N., Kadomatsu, S., Fujisawa, M., Fukaguchi, K., Ishizawa, R., Kanda, N., Kasugai, D., Nakajima, M., Goto, T., & Tsugawa, Y. (2023) · JMIR Medical Education · doi:10.2196/47532 · Cited by 63
“Indeed, the latest study has reported that ChatGPT, an older model of GPT, passed the United States Medical Licensing Examination (USMLE) with moderate accuracy and high concordance [14].”
BackgroundShrestha, N., Shen, Z., Zaidat, B., Duey, A. H., Tang, J. E., Ahmed, W., Hoang, T., Restrepo Mejia, M., Rajjoub, R., Markowitz, J. S., et al. (2024) · Spine · doi:10.1097/brs.0000000000004915 · Cited by 64
“5,6 The increasing use of language models such as ChatGPT in health care, however, has led to concerns about their ability to provide accurate medical information, as they lack the clinical expertize and judgment of human health care providers.”
BackgroundChatGPT as an educational resource for civil engineering students
Uddin, S. M. J., Albert, A., Tamanna, M., Ovid, A., & Alsharef, A. F. (2024) · Computer Applications in Engineering Education · doi:10.1002/cae.22747 · Cited by 75
“On the other hand, a number of studies assessed the performance of ChatGPT for the US Medical Licensing Examination (USMLE) and they concluded that ChatGPT successfully passed all three steps of the exam without any additional training [18, 33, 36].”
Misra, S. M., & Suresh, S. (2024) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205241263475 · Cited by 73
Hayden, N., Gilbert, S., Poisson, L., Griffith, B. D., & Klochko, C. L. (2024) · Radiology · doi:10.1148/radiol.240153 · Cited by 78
Mavrych, V., Ganguly, P. K., & Bolgova, O. (2024) · Clinical Anatomy · doi:10.1002/ca.24244 · Cited by 71
“This constitutes a framework against which LLMs can be benchmarked in the medical context (Kung et al., 2023).”
The Roles of Artificial Intelligence in Teaching Anatomy: A Systematic Review
Joseph, T. S., Gowrie, S., Montalbano, M. J., Bandelow, S., Clunes, M. T., Dumont, A. S., Iwanaga, J., Tubbs, R. S., & Loukas, M. (2025) · Clinical Anatomy · doi:10.1002/ca.24272 · Cited by 72
“Furthermore, Kung et al. (2023) demonstrated that students using AI-powered tutoring systems showed a 20% improvement in test scores compared to those using traditional study methods, showcasing the transformative potential of AI in boosting learning outcomes.”
Large Language Model Performance and Clinical Reasoning Tasks
Rao, A. S., Esmail, K. P., Lee, R. S., Jiang, S., Arraiza Carlo, B., Gill, J., Khanna, P., Kalmowitz, E., Montagnese, B., Heydari, K., et al. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2026.4003 · Cited by 69
Long, C., Lowe, K., Zhang, J., Santos, A. D., Alanazi, A., O'Brien, D., Wright, E. D., & Cote, D. (2024) · JMIR Medical Education · doi:10.2196/49970 · Cited by 46
“ChatGPT has also been tested for its performance on the tasks of medical note-taking and answering consultations [2,14].”
BackgroundMethodInfluentialMore Capable, Less Benevolent: Trust Perceptions of AI Systems across Societal Contexts
Novozhilova, E., Mays, K. K., Paik, S., & Katz, J. E. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6010017 · Cited by 76
“ChatGPT specifically has been recognized as a valuable tool for helping the development of clinical practice guidelines for medical professionals [80], as well as an aid for patients in organizing information related to medications, lifestyle modifications, and treatment alternatives [79].”
MethodChatGPT versus Bard: A comparative study
Ahmed, I., Kajol, M. A., Hasan, U., Datta, P. P., Roy, A., & Reza, M. R. (2024) · Engineering Reports · doi:10.1002/eng2.12890 · Cited by 68
“136 Although ChatGPT demonstrates competence similar to that of a third-year medical student in addressing the questions of the US Medical Licensing Examination (USMLE) Steps 1 and 2, its overall performance across various disciplines remains uncertain.”
InfluentialComparing GPT-4 and Human Researchers in Health Care Data Analysis: Qualitative Description Study
Li, K. D., Fernandez, A., Schwartz, R., Ríos, N., Carlisle, M., Amend, G., Patel, H. V., & Breyer, B. N. (2024) · Journal of Medical Internet Research · doi:10.2196/56500 · Cited by 79
“Similarly, ChatGPT’s performance on the United States Medical Licensing Examination (USMLE) further showcases the potential use of AI in medical education, where it achieved scores near the passing threshold without specialized training [37].”
Correctness Comparison of ChatGPT ‐4, Gemini, Claude‐3, and Copilot for Spatial Tasks
Hochmair, H. H., Juhász, L., & Kemp, T. (2024) · Transactions in GIS · doi:10.1111/tgis.13233 · Cited by 82
Li, Z., Christoph, G., Inglin, Y., Steck, N., Streli, P., & Holz, C. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676470 · Cited by 82
Medical students and ChatGPT: analyzing attitudes, practices, and academic perceptions
Abdelhafiz, A. S., Farghly, M. I., Sultan, E. A., Abouelmagd, M. E., Ashmawy, Y. M., & Elsebaie, E. H. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06731-9 · Cited by 84
“Also, ChatGPT can be used as an assessment tool, giving students reasoned justifications for their wrong answers and enabling them to learn [11].”
Ke, Y., Yang, R., Lie, S. A., Lim, T. X. Y., Ning, Y., Li, I., Abdullah, H. R., Ting, D. S. W., & Liu, N. (2024) · Journal of Medical Internet Research · doi:10.2196/59439 · Cited by 64
“Notably, these models have demonstrated their proficiency by successfully passing advanced medical examinations [10] and scoring clinical risk gradings on par with experienced physicians [11].”
Coşkun, Ö., Kıyak, Y. S., & Budakoğlu, I. İ. (2025) · Medical Teacher · doi:10.1080/0142159x.2024.2327477 · Cited by 64
“It has also provided a high proportion of correct answers in the United States Medical Licensing Examination (Kung et al. 2023; Mihalache et al. 2023).”
Jin, Q., Leaman, R., & Lu, Z. (2023) · Journal of the American Society of Nephrology · doi:10.1681/asn.0000000000000166 · Cited by 64
Radiology Gets Chatty: The ChatGPT Saga Unfolds
Grewal, H., Dhillon, G., Monga, V., Sharma, P., Buddhavarapu, V. S., Sidhu, G., & Kashyap, R. (2023) · Cureus · doi:10.7759/cureus.40135 · Cited by 66
“Integration with electronic health records (EHRs) that adhere to privacy regulations can create an enhanced patient experience, with constant monitoring of performance and gathering user feedback for continuous service improvement and producing accurate, up-to-date, and relevant data [7].”
BackgroundRossettini, G., Cook, C., Palese, A., Pillastrini, P., & Turolla, A. (2023) · Journal of Orthopaedic & Sports Physical Therapy · doi:10.2519/jospt.2023.12000 · Cited by 58
Farhat, F., Chaudhry, B. M., Nadeem, M., Sohail, S. S., & Madsen, D. Ø. (2024) · JMIR Medical Education · doi:10.2196/51523 · Cited by 61
“In Gilson et al [30] and Kung et al [31], authors have scrutinized ChatGPT’s aptitude in the United States Medical Licensing Examination (USMLE), while Teebagy et al [32] conducted a comparative study of GPT-3.”
BackgroundInfluence on the accuracy in ChatGPT: Differences in the amount of information per medical field
Haze, T., Kawano, R., Takase, H., Suzuki, S., Hirawa, N., & Tamura, K. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105283 · Cited by 58
Sumbal, A., Sumbal, R., & Amir, A. (2024) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205241238641 · Cited by 63
“1,16,19 Another feature that differentiates ChatGPT from its predecessors is that users can ask follow-up questions.”
BackgroundChatGPT and medical education: a double-edged sword
Khairatun Hisan, U., & Miftahul Amri, M. (2023) · Journal of Pedagogy and Education Science · doi:10.56741/jpes.v2i01.302 · Cited by 63
ChatGPT as a tool for medical education and clinical decision-making on the wards: Case study
Skryd, A., & Lawrence, K. (2024) · JMIR Formative Research · doi:10.2196/51346 · Cited by 63
“…medical notes and academic abstracts and completing the United States Medical Licensing Examination (USMLE); a surfeit of opportunities to explore the impact of generative AI tools in medical education has also been identified at the undergraduate, graduate, and professional levels [5,6,19-22].”
BackgroundTanaka, Y., Nakata, T., Aiga, K., Etani, T., Muramatsu, R., Katagiri, S., Kawai, H., Higashino, F., Enomoto, M., Noda, M., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000433 · Cited by 62
“5 possesses sufficient ability to pass the United States Medical Licensing Examination [14,15].”
BackgroundDevelopment of a large-scale medical visual question-answering dataset
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W. (2024) · Communications Medicine · doi:10.1038/s43856-024-00709-2 · Cited by 115
Health Care Privacy Risks of AI Chatbots
Kanter, G. P., & Packel, E. A. (2023) · JAMA · doi:10.1001/jama.2023.9618 · Cited by 61
Reshaping medical education: Performance of ChatGPT on a PES medical examination
Wójcik, S., Rulkiewicz, A., Pruszczyk, P., Lisik, W., Poboży, M., & Domienik-Karłowicz, J. (2024) · Cardiology Journal · doi:10.5603/cj.97517 · Cited by 53
“It is worth noting that a similar system developed by the world’s largest database of scientific publica - tions, PubMed (PubMedGPT), could only answer up to 50% of the answers correctly in the same USMLE exam, far behind ChatGPT [11].”
BackgroundExamining ChatGPT Performance on USMLE Sample Items and Implications for Assessment
Yaneva, V., Baldwin, P., Jurich, D. P., Swygert, K., & Clauser, B. E. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005549 · Cited by 60
Albdrani, R. N., & Al-Shargabi, A. A. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.01411122 · Cited by 68
“Furthermore, Kung et al. [15] discussed the ability of ChatGPT to perform clinical reasoning by testing its performance on questions from the United States Medical Licensing Examination (USMLE) and the potential of ChatGPT to assist in medical education.”
BackgroundThe Content Analysis of the Lesson Plans Created by ChatGPT and Google Gemini
Baytak, A. (2024) · Research in Social Sciences and Technology · doi:10.46303/ressat.2024.19 · Cited by 83
“As it was described before, ChatGPT was found successful in most quiz-based exams (Choi et al., 2023; Kung et al., 2023; Malinka et al., 2023).”
MethodGenerative AI tools can enhance climate literacy but must be checked for biases and inaccuracies
Atkins, C., Girgente, G., Shirzaei, M., & Kim, J. (2024) · Communications Earth & Environment · doi:10.1038/s43247-024-01392-w · Cited by 58
“…capabilities of generative AI tools, such as ChatGPT and Bard, will likely grow in value, considering their quickly increasing role in climate literacy 25,26 and their potential — yet debated — bene fi cial applications in the general education sector 22,27,40 within countries where it is available.”
BackgroundGiuffrè, M., Kresevic, S., Pugliese, N., You, K., & Shung, D. L. (2024) · Liver International · doi:10.1111/liv.15974 · Cited by 68
Large language models can outperform humans in social situational judgments
Mittelstädt, J. M., Maier, J., Goerke, P., Zinn, F., & Hermes, M. (2024) · Scientific Reports · doi:10.1038/s41598-024-79048-0 · Cited by 45
Performance of ChatGPT on the MCAT: The Road to Personalized and Equitable Premedical Learning
Bommineni, V. L., Bhagwagar, S., Balcarcel, D., Davatzikos, C., & Boyer, D. (2023) · medRxiv · doi:10.1101/2023.03.05.23286533 · Cited by 54
Seeing ChatGPT Through Students' Eyes: An Analysis of TikTok Data
Haensch, A.-C., Ball, S., Herklotz, M., & Kreuter, F. (2023) · arXiv · doi:10.48550/arxiv.2303.05349 · Cited by 62
“ChatGPT represents a significant advancement in the usability of natural language processing technology, which has important implications for a variety of fields such as computer science, law, finance, education, and medicine – to only name a few (Choi et al., 2023; Dowling and Lucey, 2023; Kung et al., 2023; Haque et…”
BackgroundExperimental evidence of the effects of large language models versus web search on depth of learning
Melumad, S., & Yun, J. H. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf316 · Cited by 61
Collins, B. R., Black, E. W., & Rarey, K. E. (2024) · Clinical Anatomy · doi:10.1002/ca.24178 · Cited by 61
“The strong performance of both AnatomyGPT and ChatGPT is consistent with the findings of previous studies that explored the performance of ChatGPT on USMLE Sample Questions (Kung et al., 2023); ChatGPT on the Medical College Admission Test (MCAT) (Bommineni et al., 2023); GPT-4 on Medical Challenge Problems (Nori et a…”
InfluentialImproving radiology workflow using ChatGPT and artificial intelligence
Mese, I., Taslicay, C. A., & Sivrioglu, A. K. (2023) · Clinical Imaging · doi:10.1016/j.clinimag.2023.109993 · Cited by 58
GPT-4 can pass the Korean National Licensing Examination for Korean Medicine Doctors
Jang, D., Yun, T.-R., Lee, C.-Y., Kwon, Y.-K., & Kim, C.-E. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000416 · Cited by 59
Utility of artificial intelligence‐based large language models in ophthalmic care
Biswas, S., Davies, L. N., Sheppard, A. L., Logan, N. S., & Wolffsohn, J. S. (2024) · Ophthalmic and Physiological Optics · doi:10.1111/opo.13284 · Cited by 73
“22,23 ChatGPT-3.”
MethodInfluentialLLM-Based Framework for Administrative Task Automation in Healthcare
Gebreab, S. A., Salah, K., Jayaraman, R., Habib ur Rehman, M., & Ellaham, S. (2024) · 2024 12th International Symposium on Digital Forensics and Security (ISDFS) · doi:10.1109/isdfs60797.2024.10527275 · Cited by 53
“5 model, was evaluated on its ability to handle complex medical information considering United States Medical Licensing (USMLE) exam and it was able to achieve a score close to the passing threshold without any prompting [8].”
MethodFoundation models in ophthalmology
Chia, M. A., Antaki, F., Zhou, Y., Turner, A. W., Lee, A., & Keane, P. A. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2024-325459 · Cited by 71
“28 29 32 The performance of GPT-4 has been notably impressive, achieving a score of 72.9% on a multiple-choice question dataset, numerically surpassing the average historical human performance benchmarks.”
BackgroundInfluentialChatGPT and Other Large Language Models in Medical Education — Scoping Literature Review
Aster, A., Laupichler, M. C., Rockwell-Kollmann, T., Masala, G., Bala, E., & Raupach, T. (2024) · Medical Science Educator · doi:10.1007/s40670-024-02206-6 · Cited by 75
“Kung et al. [14] demonstrated relatively early on that ChatGPT can pass the United States Medical Licensing Examination (USMLE).”
Zhou, J., Zhang, J., Wan, R., Cui, X., Liu, Q., Guo, H., Shi, X., Fu, B., Meng, J., Yue, B., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06916-2 · Cited by 73
“However, LLMs are trained on large datasets of mixed quality, which can lead to issues such as disparities, biases, and incorrect associations [8–10], commonly referred to as “hallucinations“ [11].”
Framework for bias evaluation in large language models in healthcare settings
Templin, T., Fort, S., Padmanabham, P. S., Seshadri, P., Rimal, R., Oliva, J. B., Lich, K. H., Sylvia, S., & Sinnott-Armstrong, N. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01786-w · Cited by 80
Generating Multiple Choice Questions for Computing Courses Using Large Language Models
Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D. H., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342898 · Cited by 60
“Outside of the context of computing education, these models are also capable of passing many standardized tests, including the US Medical Licensing Exam [15].”
MethodCan ChatGPT Accurately Answer a PICOT Question? Assessing AI Response to a Clinical Question
Branum, C., & Schiavenato, M. (2023) · Nurse Educator · doi:10.1097/nne.0000000000001436 · Cited by 55
The model student: GPT-4 performance on graduate biomedical science exams
Stribling, D., Xia, Y., Amer, M. K., Graim, K. S., Mulligan, C. J., & Renne, R. (2024) · Scientific Reports · doi:10.1038/s41598-024-55568-7 · Cited by 52
“These include the bar exam for law and the United States Medical Licensing Examinations (USMLE) in medicine 19,21,22 .”
Quality, Accuracy, and Bias in ChatGPT-Based Summarization of Medical Abstracts
Hake, J., Crowley, M., Coy, A., Shanks, D., Eoff, A., Kirmer-Voss, K., Dhanda, G., & Parente, D. J. (2024) · The Annals of Family Medicine · doi:10.1370/afm.3075 · Cited by 47
Fan, H., Chen, G., Wang, X., & Peng, Z. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676390 · Cited by 86
Toward AI grading of student problem solutions in introductory physics: A feasibility study
Kortemeyer, G. (2023) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.19.020163 · Cited by 59
Malik, A., Khan, M. L., & Hussain, K. (2023) · doi:10.2139/ssrn.4413516 · Cited by 59
Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study
Shoufan, A. (2023) · ACM Transactions on Computing Education · doi:10.1145/3628162 · Cited by 59
ChatGPT passes German state examination in medicine with picture questions omitted
Jung, L. B., Gudera, J. A., Wiegand, T. L. T., Allmendinger, S., Dimitriadis, K., & Koerte, I. K. (2023) · Deutsches Ärzteblatt international · doi:10.3238/arztebl.m2023.0113 · Cited by 60
Metadata
- Authors
- Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-02-09
- Volume
- 2
- Issue
- 2
- Pages
- e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.