Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Open accessPublished version of a preprint
The DOI matches its Crossref record, with a small difference in title or year; 28 of the 28 references it cites were found in a registry or an index.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Kung et al. (2023) evaluated the performance of ChatGPT on the United States Medical Licensing Exam and demonstrated that the chatbot did not reach the passing threshold and achieved an overall accuracy of only 50% (Kung et al., 2023).”
From A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing (2025) · cites this paper as Kung et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [17]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI's capabilities, such as passing the United States Medical Licensing Exam [107], completing exams at the University of Minnesota Law School [108], and solving queries from Wharton School of Business tests [109], underscore its potential.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [107]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The HCI community has explored the application of LLMs for various healthcare practices, such as mental health support [66,68], health information seeking [81,117,128], and public health interventions [58]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
From DIALOGUE: A Generative AI-Based Pre–Post Simulation Study to Enhance Diagnostic Communication in Medical Students Through Virtual Type 2 Diabetes Scenarios (2025) · cites this paper as Kung et al., 2023
Experimental — accuracy not yet measured
52 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background
Cited by
Items 1,101–1,129 of 1,129
Karataş, G., & Karataş, M. E. (2026) · Strabismus · doi:10.1080/09273972.2025.2536782 · Cited by 7
Is ChatGPT a friend or foe in the war on misinformation?: A South African perspective
Senekal, B., & Brokensha, S. (2023) · Communicare: Journal for Communication Studies in Africa · doi:10.36615/jcsa.v42i2.2437 · Cited by 7
“Gilson et al. (2022) have found that the tool can perform at the level of a third-year medical student, while Kung et al. (2022) have demonstrated that ChatGPT has the ability to pass the United States Medical Licensing Exam (USMLE).”
BackgroundPotential effects of ChatGPT as a learning tool through students’ experiences
Hwang, J. Y. (2024) · Medical Teacher · doi:10.1080/0142159x.2023.2259068 · Cited by 6
Calabrese, G., Maselli, R., Maida, M., Barbaro, F., Morais, R., Nardone, O. M., Sinagra, E., Di Mitri, R., & Sferrazza, S. (2025) · iGIE · doi:10.1016/j.igie.2025.01.012 · Cited by 5
Hooshiar, M. H. (2026) · The Journal of Prosthetic Dentistry · doi:10.1016/j.prosdent.2025.11.005 · Cited by 5
Public Attitudes Toward ChatGPT on Twitter: Sentiments, Topics, and Occupations
Koonchanok, R., Pan, Y., & Jang, H. (2023) · arXiv · doi:10.48550/arxiv.2306.12951 · Cited by 5
Chetla, N., Samayamanthula, S., Chang, J., Leigh, A., Akosman, S., Tandon, M., Hage, T., & Cusick, M. (2025) · Clinical Ophthalmology · doi:10.2147/opth.s517238 · Cited by 5
“20 By using a clear, concise prompt in this familiar exam format, we aimed to harness ChatGPT’s high accuracy, concordance, and insight demonstrated during its USMLE performance.”
Wilhelm, C., Steckelberg, A., & Rebitschek, F. G. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02646-6 · Cited by 5
“Recently, Chat-GPT was evaluated for its clinical reasoning ability by testing its performance on questions from the United States Medical Licensing Examination, where it scored at or near the passing threshold on all three exams without any special training or reinforcement [21].”
MethodRose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Berg, R. C., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Potrebny, T. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02631-0 · Cited by 5
“For example, performing at or near the passing threshold for all three United States medical licensing exams [25], answering questions on complex topics such as statistical methodology (Table 1), offering opinions on Taylor Swift’s predominance in popular music in the vernacular of a bratty teenage…”
AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training
Saggar, A., Dimitrova, V., Sarikaya, D., Hogg, D., & Darling, J. C. (2026) · Archives of Disease in Childhood · doi:10.1136/archdischild-2025-329846 · Cited by 4
Performance of novel GPT-4 in otolaryngology knowledge assessment
Revercomb, L., Patel, A. M., Fu, D., & Filimonov, A. (2024) · Indian Journal of Otolaryngology and Head & Neck Surgery · doi:10.1007/s12070-024-04935-x · Cited by 4
Cabrera, C., & Neville, R. (2025) · Perspectives: Policy and Practice in Higher Education · doi:10.1080/13603108.2025.2595453 · Cited by 4
Exploring Student Perspectives on ChatGPT in Higher Education
Shard, Vaidya, P., Koul, S., Arora, A., Sonia, & Jain, P. (2024) · 2024 3rd Edition of IEEE Delhi Section Flagship Conference (DELCON) · doi:10.1109/delcon64804.2024.10866660 · Cited by 3
ChatGPT and Large Language Models (LLMs) in Healthcare: Opportunities and Risks
Ali, H., Qadir, J., Shah, Z., Alam, T., & Househ, M. (2023) · doi:10.36227/techrxiv.22579852.v2 · Cited by 3
Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
Liao, Y., Meng, Y., Wang, Y., Liu, H., Wang, Y., & Wang, Y. (2024) · arXiv · doi:10.48550/arxiv.2403.08495 · Cited by 15
“The second dataset, MedicalExam , encompasses 150 cases chosen from five public clinical examination datasets, including MedQA (Jin et al., 2020b), MedMCQA (Pal et al., 2022b), MMLU (Hendrycks et al., 2021), SelfExam (Kung et al., 2023), and QMAX.”
MethodAcademic Writing with GPT-3.5: Reflections on Practices, Efficacy and Transparency
Buruk, O. '. (2023) · doi:10.31222/osf.io/vg5f6 · Cited by 3
Gao, L., Zhang, L., Zhang, Y., Wu, M., Xu, J., & Lu, D. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09219-2 · Cited by 4
“Recent studies have highlighted the potential of LLMs to support clinical reasoning, generate patient narratives, and even demonstrate performance approaching passing thresholds on medical licensing examinations [6, 7].”
Automatically Enriching Content for a Behavioral Health Learning Management System: a First Look
Barish, G., Marlotte, L., Drayton, M., Mogil, C., & Lester, P. (2023) · World Congress on Electrical Engineering and Computer Systems and Science · doi:10.11159/cist23.125 · Cited by 2
Zancanaro, E., Giannopoulos, A., Gimelli, A., & Kresoja, K.-P. (2026) · European Heart Journal - Imaging Methods and Practice · doi:10.1093/ehjimp/qyag097 · Cited by 6
“4,5 Several reports have demonstrated that LLMs can generate coherent, detailed feedback on scientific manuscripts, 6,7 and recent comparative studies in surgery 8 and educational research 9 have suggested broad parity in review quality between AI and human reviewers.”
Boczkowski, D., Dolata, T., Radej, D., Sawina, P., Suleiman, R., Latkowska, A., Kowalczyk, A., Loson-Kawalec, M., Jaworski, W., Wielochowska, A., et al. (2025) · Cureus · doi:10.7759/cureus.95581 · Cited by 2
“In response to these challenges, studies are increasingly being conducted to verify the effectiveness of language models under controlled conditions, such as standardized medical exams [4-13].”
When AI speaks like a specialist: ChatGPT-4 in the management of inflammatory bowel disease
De Cristofaro, E., Zorzi, F., Abreu, M., Colella, A., Blanco, G. D. V., Fiorino, G., Lolli, E., Noor, N., Lopetuso, L. R., Pioche, M., et al. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1678320 · Cited by 1
Asoglu, H., Kose, S., Kayim, O., Abaci, A., Khatib, G., Vardar, M. A., Kilic Bagir, E., Gumurdulu, D., Kidi, M. M., Bayram, E., et al. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15124451 · Cited by 1
“Artificial intelligence (AI) has emerged as a transformative force in healthcare, with large language models (LLMs) representing one of the most significant recent developments in clinical decision support [1,2].”
Keskin, A., & Aygün, T. (2025) · Tıp Eğitimi Dünyası · doi:10.25282/ted.1716591 · Cited by 1
Designing AI-resilient assessment in higher education: a four-pillar conceptual framework
Nikolić, D., & Basta Nikolić, M. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1841682 · Cited by 1
“In professional education, a student may construct an exemplary case summary with AI assistance yet remain unable to justify a treatment decision under uncertainty (Kung et al., 2023).”
SEMA-RAG: A Self-Evolving Multi-Agent Retrieval-Augmented Generation Framework for Medical Reasoning
Huang, Y., Chen, R., & Cheng, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.917
Ethical and responsible use of artificial intelligence in medical education
Sriram, A., Nichols, E., Ganti, L., & Gue, S. (2026) · International Journal of Emergency Medicine · doi:10.1186/s12245-026-01389-6
Evidence on Demand: OpenEvidence versus Established LLMs on the Orthopaedic In-Training Examination
Khan, A. A., Lalvani, S. S., Pourarbab, S., Hyeamang, L. J., Lew, R. M., Hussain, Z. B., Nicholson, G. P., & Garrigues, G. E. (2026) · Journal of Orthopaedic Experience & Innovation · doi:10.60118/001c.162882
Xian, R. P., Lee, A. J., Lolla, S., Wang, V., Cui, Q., Ro, R., & Abbasi-Asl, R. (2024) · arXiv · doi:10.48550/arxiv.2402.10527
“Question answering (QA) has been the proving ground in recent achievements of biomedical language models (BLMs), here referring to GLMs adapted to the biomedical domain (Singhal et al., 2023a; Kung et al., 2023; Singhal et al., 2023b).”
BackgroundMethodGao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022) · doi:10.1101/2022.12.23.521610 · Cited by 1,099
Metadata
- Authors
- Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-02-09
- Volume
- 2
- Issue
- 2
- Pages
- e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.