How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. Objective: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. Methods: We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Results: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. Conclusions: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT’s capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Template of question posed to each large language model (LLM), including both AMBOSS Attending Tip and the response from Chat Generative Pre-trained Transformer (ChatGPT). The correct answer to this question is “E. Zidovudine (AZT).” In the case of GPT-3, prompt engineering was necessary, with: "Please answer this multiple choice question:" + question as described previously + "Correct answer is." As GPT-3 is inherently a nondialogic model, this was necessary to reduce model hallucinations and force a clear answer [17].](https://figures.cochat.ai/paper-figures/583a8f9972504138a5ac67eaf322285a/how-does-chatgpt-perform-on-the-united-states-medical-licensing-examination-figure-1.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, like GPT-4, have shown medical knowledge despite lacking medicine-specific training [10, 29].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, several valid concerns were raised, including the possibility of generating inaccurate content, along with ethical issues, including the risk of bias, plagiarism, and copyright issues [9,18,40,41].”
From Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study (2023) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, analyses of ChatGPT show structured, non-random response patterns consistent with trait-like reasoning abilities18.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the majority of clinical reasoning evaluations on LLMs have been conducted using standardized exams (the United States Medical Licensing Examination) or digital case reports [14,19].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [14]
Experimental — accuracy not yet measured
39 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Springer, L., Stanne, M. E., & Donovan, S. S. (1999) · Review of Educational Research · doi:10.3102/00346543069001021 · Cited by 1,943 · cited 1×
“Small group education has been shown to be a highly efficacious method of teaching [23,24].”
MethodThe use of reflection in medical education: AMEE Guide No. 44
Sandars, J. (2009) · Medical Teacher · doi:10.1080/01421590903050374 · Cited by 1,085
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
Research works printed but not linked: 2
- 10.48550/arxiv.2108.01764 · DOI
- Wang, DC, 2020, Yale J Biol Med, 93, 441
Cited by
Items 1,001–1,047 of 1,047
Cherif, H., Moussa, C., Missaoui, A. M., Salouage, I., Mokaddem, S., & Dhahri, B. (2024) · JMIR Medical Education · doi:10.2196/52818 · Cited by 18
Socrates, V., Wright, D. S., Huang, T., Fereydooni, S., Dien, C., Chi, L., Albano, J., Patterson, B., Sasidhar Kanaparthy, N., Wright, C. X., et al. (2025) · JMIR Aging · doi:10.2196/69504 · Cited by 17
Self-citationChatGPT: rumors of human providers’ demise have been greatly exaggerated
Alvero, R. (2023) · Fertility and Sterility · doi:10.1016/j.fertnstert.2023.03.010 · Cited by 17
Hopson, S., Mildon, C., Hassard, K., Kubalek, C., Laverty, L., Urie, P., & Corte, D. D. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07556-2 · Cited by 17
Jo, M. H., Kim, M.-J., Oh, H.-K., Choi, M. J., Shin, H.-R., Lee, T.-G., Ahn, H.-M., Kim, D.-W., & Kang, S.-B. (2024) · International Journal of Colorectal Disease · doi:10.1007/s00384-024-04670-3 · Cited by 17
Spennemann, D. H. R. (2023) · Publications · doi:10.3390/publications11030045 · Cited by 17
Can Large Language Models Safely Address Patient Questions Following Cataract Surgery?
Chowdhury, M., Lim, E., Higham, A., McKinnon, R., Ventoura, N., He, Y., & De Pennington, N. (2023) · Proceedings of the 5th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2023.clinicalnlp-1.17 · Cited by 17
The role of ChatGPT in enhancing ENT surgical training – a trainees’ perspective
Brennan, L., Balakumar, R., & Bennett, W. (2024) · The Journal of Laryngology & Otology · doi:10.1017/s0022215123001354 · Cited by 16
Navigating the AI era: university communication strategies and perspectives on generative AI tools
Henke, J. (2024) · Journal of Science Communication · doi:10.22323/2.23030205 · Cited by 16
Ali, H., Patel, P., Obaitan, I., Mohan, B. P., Sohail, A. H., Smith-Martinez, L., Lambert, K., Gangwani, M. K., Easler, J. J., & Adler, D. G. (2023) · iGIE · doi:10.1016/j.igie.2023.10.001 · Cited by 15
Large language models management of complex medication regimens: a case-based evaluation
Chase, A., Most, A., Xu, S., Barreto, E., Murray, B., Henry, K., Smith, S., Hedrick, T., Chen, X., Li, S., et al. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1514445 · Cited by 14
Artificial Intelligence: Tool or Teammate?
Díaz, B., & Delgado, C. (2024) · Journal of Research in Science Teaching · doi:10.1002/tea.21993 · Cited by 14
Unveiling the risks of ChatGPT in diagnostic surgical pathology
Guastafierro, V., Corbitt, D. N., Bressan, A., Fernandes, B., Mintemur, Ö., Magnoli, F., Ronchi, S., La Rosa, S., Uccella, S., & Renne, S. L. (2025) · Virchows Archiv · doi:10.1007/s00428-024-03918-1 · Cited by 14
Reina-Parrado, M., Román-Graván, P., & Hervás-Gómez, C. (2025) · European Journal of Educational Research · doi:10.12973/eu-jer.14.1.249 · Cited by 13
Scherr, R., Spina, A., Dao, A., Andalib, S., Halaseh, F. F., Blair, S., Wiechmann, W., & Rivera, R. (2025) · JMIR Formative Research · doi:10.2196/66478 · Cited by 10
“Early investigations assessed whether large language models (LLMs) could pass medical trainee licensing examinations [1-5].”
ChatGPT and its application in the field of mental health.
Bhattacharyya, R., Chakraborty, K., & Neogi, R. (2023) · Journal of SAARC Psychiatric Federation · doi:10.4103/jspf.jspf_9_23 · Cited by 12
Quantum leap in medical mentorship: exploring ChatGPT’s transition from textbooks to terabytes
Chokkakula, S., Chong, S., Yang, B., Jiang, H., Yu, J., Han, R., Attitalla, I. H., Yin, C., & Zhang, S. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1517981 · Cited by 12
Tong, L., Wang, J., Rapaka, S., & Garg, P. S. (2025) · Medical Teacher · doi:10.1080/0142159x.2024.2363486 · Cited by 12
Abouzeid, E., Wassef, R., Jawwad, A., & Harris, P. (2025) · JMIR Medical Education · doi:10.2196/69521 · Cited by 12
“…prompts and examination questions, and to mimic human writing styles, introducing a potential threat to the validity of traditional forms of medical student assessment including short answer assessment [8], it even successfully passed the USMLE (United States Medical Licensing Examination) [9].”
Mackey, B. P., Garabet, R., Maule, L., Tadesse, A., Cross, J., & Weingarten, M. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00135-2 · Cited by 11
ChatGPT in Research and Education: A SWOT Analysis of Its Academic Impact
Miah, A. S. M., Tusher, M. M. R., Hossain, M. M., Hossain, M. M., Rahim, M. A., Hamid, M. E., Islam, M. S., & Shin, J. (2025) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2025.064168 · Cited by 11
Meyer, B., Kfuri‐Rubens, R., Schmidt, G., Tariq, M., Riedel, C., Recker, F., Riedel, F., Kiechle, M., & Riedel, M. (2025) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.70251 · Cited by 10
“20 Our study demonstrates the feasibility of using GPT-4 to generate treatment recommendations for complex gynecologic patients.”
Battle of the Bots: Solving Clinical Cases in Osteoarticular Infections With Large Language Models
Borgonovo, F., Matsuo, T., Petri, F., Amin Alavi, S. M., Mazudie Ndjonko, L. C., Gori, A., & Berbari, E. F. (2025) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2025.100230 · Cited by 10
Yoon, S.-H., Oh, S. K., Lim, B. G., & Lee, H.-J. (2024) · JMIR Medical Education · doi:10.2196/56859 · Cited by 10
Plevris, V., Papazafeiropoulos, G., & Rios, A. J. (2023) · arXiv · doi:10.48550/arxiv.2305.18618 · Cited by 90
“They can be put to the test in various fields to evaluate their understanding and problem-solving abilities, and they can even be tested against actual professional examinations [8].”
BackgroundGandhi, A. P., Joesph, F. K., Rajagopal, V., Aparnavi, P., Katkuri, S., Dayama, S., Satapathy, P., Khatib, M. N., Gaidhane, S., Zahiruddin, Q. S., et al. (2024) · JMIR Formative Research · doi:10.2196/49964 · Cited by 9
“Previous studies have reported relatively more relevant, accurate, and congruent answers being provided by ChatGPT than the earlier AI systems [24,28].”
BackgroundResultIs ChatGPT a friend or foe in the war on misinformation?: A South African perspective
Senekal, B., & Brokensha, S. (2023) · Communicare: Journal for Communication Studies in Africa · doi:10.36615/jcsa.v42i2.2437 · Cited by 7
Tan, T. F., Elangovan, K., Jin, L., Jie, Y., Yong, L., Lim, J., Poh, S., Ng, W. Y., Lim, D., Ke, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.10083 · Cited by 7
Lim, J. S., & Hong, N. (2025) · Health & New Media Research · doi:10.22720/hnmr.2025.00115 · Cited by 6
Al-Khiami, M. I., & Jaeger, M. (2023) · doi:10.20944/preprints202307.0660.v1 · Cited by 6
Wang, A., Ruparel, R., Iurchenko, A., Jhun, P., Séguin, J. A., Strachan, P., Wong, R., Karthikesalingam, A., Matias, Y., Hassidim, A., et al. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3721208 · Cited by 6
Choudhury, A., & Shamszare, H. (2023) · doi:10.2196/preprints.47184 · Cited by 473
Caution regarding fabricated citations from artificial intelligence
Cohen, F., Vallimont, J., & Gelfand, A. A. (2024) · Headache: The Journal of Head and Face Pain · doi:10.1111/head.14649 · Cited by 4
Rethinking Productivity with GenAI: A Neurodivergent Students' Perspective
Jamshed, H., Naseem, M., Potluri, V., & Brewer, R. N. (2025) · Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3746329 · Cited by 5
“…in generative AI (GenAI), especially those leveraging large language models (LLMs) and employing natural language processing (NLP) technologies, have rapidly gained popularity, leading to shocking news reports like “ChatGPT Passes Whar-ton School’s MBA exam and US Medical Licensing Exam” [31, 64].”
AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training
Saggar, A., Dimitrova, V., Sarikaya, D., Hogg, D., & Darling, J. C. (2026) · Archives of Disease in Childhood · doi:10.1136/archdischild-2025-329846 · Cited by 4
Academic Integrity and Students’ Ethical Use of ChatGPT in Higher Education
Shishakly, R., & Nachouki, M. (2026) · Journal of Information Technology Education: Research · doi:10.28945/5730 · Cited by 15
“Scholars argue that the interactive and stylistic sophistication of ChatGPT complicates traditional notions of authorship and originality, thereby challenging existing academic norms (Gilson et al., 2023; Short & Short, 2023).”
Boczkowski, D., Dolata, T., Radej, D., Sawina, P., Suleiman, R., Latkowska, A., Kowalczyk, A., Loson-Kawalec, M., Jaworski, W., Wielochowska, A., et al. (2025) · Cureus · doi:10.7759/cureus.95581 · Cited by 2
Dilber, C., & Yoşumaz, İ. (2026) · Journal of Academic Ethics · doi:10.1007/s10805-025-09681-5 · Cited by 1
Zheng, T., Jiang, Z., Liu, J., & Feng, S. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02671-w · Cited by 3
“Moreover, analyses of ChatGPT show structured, non-random response patterns consistent with trait-like reasoning abilities 18 .”
InfluentialZhao, C., Zhu, J., Liu, J., Zhao, W., & Pang, Y. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09469-0 · Cited by 1
“Previous studies have shown that AI-assisted learning systems can enhance students’ engagement, promote self-directed learning, and improve academic performance in various disciplines [10, 11].”
Medical students' use of large language models: a national survey
Barr, A. A., Rozman, R. C., Liu, K., Pham, M., Klarenbach, Z., Chinna-Meyyappan, A., Hassan, A. Y., Zarychta, M., Ferri, O. E., Al-Khaz’Aly, A., et al. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106589 · Cited by 1
Ethical and responsible use of artificial intelligence in medical education
Sriram, A., Nichols, E., Ganti, L., & Gue, S. (2026) · International Journal of Emergency Medicine · doi:10.1186/s12245-026-01389-6
Explainability of decoder-only clinical large language models: A scoping review
Mishra, N., Abu-Hanna, A., & Calixto, I. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103501
Zheng, W., Turner, L., Kropczynski, J., Ozer, M., Nguyen, T., & Halse, S. (2025) · arXiv · doi:10.48550/arxiv.2506.11221
Eminovic, S., Levita, B., Schmidt, R., Lindholz, M., Desser, D., Penzkofer, T., Wattjes, M. P., & Nawabi, J. (2026) · DIGITAL HEALTH · doi:10.1177/20552076261464745
Med-KGMA: A novel AI-driven medical support system leveraging knowledge graphs and medical advisors
Varshney, S., Jain, B., Singh, P., Rani, D., & Mehra, S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110929
Metadata
- Authors
- Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D.
- Affiliations
- Yale University (US)
- University College Dublin (IE)
- Corresponding author
- Chartash, David
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-02-08
- Volume
- 9
- Pages
- e45312
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC9947764
- OpenAlex
- W4319460874
- Semantic Scholar
- cd4d112f3f9120d0715f22a9de2ce4720822368c
- Cited by
- 2,078
- References
- 21
- Citations by year
- 2026: 351 · 2025: 550 · 2024: 699 · 2023: 474
- FWCI
- 82.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- National Heart, Lung, and Blood Institute (T35HL007649)
- National Institute of Diabetes and Digestive and Kidney Diseases (T35DK104689)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE Step 1, USMLE Step 2, medical licensing examinations, question difficulty, GPT-3, medical education, knowledge assessment, AMBOSS, National Board of Medical Examiners, artificial intelligence in medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.