How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. Objective: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. Methods: We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Results: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. Conclusions: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT’s capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Template of question posed to each large language model (LLM), including both AMBOSS Attending Tip and the response from Chat Generative Pre-trained Transformer (ChatGPT). The correct answer to this question is “E. Zidovudine (AZT).” In the case of GPT-3, prompt engineering was necessary, with: "Please answer this multiple choice question:" + question as described previously + "Correct answer is." As GPT-3 is inherently a nondialogic model, this was necessary to reduce model hallucinations and force a clear answer [17].](https://figures.cochat.ai/paper-figures/583a8f9972504138a5ac67eaf322285a/how-does-chatgpt-perform-on-the-united-states-medical-licensing-examination-figure-1.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, like GPT-4, have shown medical knowledge despite lacking medicine-specific training [10, 29].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, several valid concerns were raised, including the possibility of generating inaccurate content, along with ethical issues, including the risk of bias, plagiarism, and copyright issues [9,18,40,41].”
From Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study (2023) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, analyses of ChatGPT show structured, non-random response patterns consistent with trait-like reasoning abilities18.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the majority of clinical reasoning evaluations on LLMs have been conducted using standardized exams (the United States Medical Licensing Examination) or digital case reports [14,19].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [14]
Experimental — accuracy not yet measured
39 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Springer, L., Stanne, M. E., & Donovan, S. S. (1999) · Review of Educational Research · doi:10.3102/00346543069001021 · Cited by 1,943 · cited 1×
“Small group education has been shown to be a highly efficacious method of teaching [23,24].”
MethodThe use of reflection in medical education: AMEE Guide No. 44
Sandars, J. (2009) · Medical Teacher · doi:10.1080/01421590903050374 · Cited by 1,085
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
Research works printed but not linked: 2
- 10.48550/arxiv.2108.01764 · DOI
- Wang, DC, 2020, Yale J Biol Med, 93, 441
Cited by
Items 601–700 of 1,047
A large language model digital patient system enhances ophthalmology history taking skills
Luo, M.-J., Bi, S., Pang, J., Liu, L., Tsui, C.-K., Lai, Y., Chen, W., Yang, Y., Xu, K., Zhao, L., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01841-6 · Cited by 39
“Through a literature search, we found that LLM applications in medicine such as answering medical examination questions 25,26 , providing diagnostic and treatment recommendations 27,28 , summarizing clinical research articles 29 , and generating patient education materials 30 are…”
Dr. ChatGPT: Utilizing Artificial Intelligence in Surgical Education
Lebhar, M. S., Velazquez, A. E., Goza, S. D., & Hoppe, I. C. (2023) · The Cleft Palate-Craniofacial Journal · doi:10.1177/10556656231193966 · Cited by 39
Sahin, M. C., Sozer, A., Kuzucu, P., Turkmen, T., Sahin, M. B., Sozer, E., Tufek, O. Y., Nernekli, K., Emmez, H., & Celtikci, E. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107807 · Cited by 36
Mohammadi, S. S., & Nguyen, Q. D. (2024) · Ophthalmology Science · doi:10.1016/j.xops.2024.100495 · Cited by 40
Tosun, B., & Yilmaz, Z. S. (2025) · Journal of Dental Sciences · doi:10.1016/j.jds.2025.01.025 · Cited by 39
Mavrych, V., Yousef, E. M., Yaqinuddin, A., & Bolgova, O. (2025) · Medical Education Online · doi:10.1080/10872981.2025.2534065 · Cited by 39
Bentegeac, R., Guellec, B. L., Kuchcinski, G., Amouyel, P., & Hamroun, A. (2025) · Journal of Medical Internet Research · doi:10.2196/64348 · Cited by 40
Evaluating the Current Ability of ChatGPT to Assist in Professional Otolaryngology Education
Zalzal, H. G., Cheng, J., & Shah, R. K. (2023) · OTO Open · doi:10.1002/oto2.94 · Cited by 39
Assessing ChatGPT’s orthopedic in-service training exam performance and applicability in the field
Jain, N., Gottlich, C., Fisher, J., Campano, D., & Winston, T. (2024) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-023-04467-0 · Cited by 39
The Evaluation of Generative AI Should Include Repetition to Assess Stability
Zhu, L., Mou, W., Hong, C., Yang, T., Lai, Y., Qi, C., Lin, A., Zhang, J., & Luo, P. (2024) · JMIR mHealth and uHealth · doi:10.2196/57978 · Cited by 38
“For example, Gilson et al [8] explored the performance of ChatGPT on the United States Medical Licensing Examination (USMLE) step 1 and step 2 exams, discovering that ChatGPT’s performance exceeded the passing score for third-year medical students in step 1.”
MethodAbi‐Rafeh, J., Hanna, S., Bassiri-Tehrani, B., Kazan, R., & Nahai, F. (2023) · Aesthetic Plastic Surgery · doi:10.1007/s00266-023-03538-1 · Cited by 38
Classification of RBA monetary policy announcements using ChatGPT
Smales, L. A. (2023) · Finance research letters · doi:10.1016/j.frl.2023.104514 · Cited by 38
Alzboon, M. S., Qawasmeh, S., Alqaraleh, M. K. S., Abuashour, A., Bader, A. F., & Al-Batah, M. S. (2023) · International Conference on Emerging Smart Technologies and Applications (eSmarTA) · doi:10.1109/esmarta59349.2023.10293294 · Cited by 38
Scott, I. A., & Zuccon, G. (2024) · Internal Medicine Journal · doi:10.1111/imj.16393 · Cited by 38
Assessing the Code Clone Detection Capability of Large Language Models
Zhang, Z., & Saber, T. (2024) · International Conference on Code Quality (ICCQ) · doi:10.1109/iccq60895.2024.10576803 · Cited by 38
Frenkel, M., & Emara, H. (2024) · Computer Applications in Engineering Education · doi:10.1002/cae.22781 · Cited by 38
Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm
Liu, L., Yang, X., Li, F., Chi, C., Shen, Y., Lyu, S., Zhang, M., Ma, X., Lv, X., Ma, L., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671575 · Cited by 38
PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion
Guo, Y., Zhang, Z., Liang, Y., Zhao, D., & Duan, N. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.514 · Cited by 38
Large language models in radiology: Fluctuating performance and decreasing discordance over time
Gupta, M., Virostko, J. M., & Kaufmann, C. R. (2024) · European Journal of Radiology · doi:10.1016/j.ejrad.2024.111842 · Cited by 38
Mahande, R. D., Fakhri, M. M., Suwahyu, I., & Sulaiman, D. R. A. (2025) · Journal of Applied Research in Higher Education · doi:10.1108/jarhe-07-2024-0372 · Cited by 39
Siam, K., Varela, A., Faruk, M. J. H., Cheng, J. Q., Gu, H., Maruf, A. A., & Aung, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-31010-4 · Cited by 39
Sammour, F., Xu, J., Wang, X., Hu, M., & Zhang, Z. (2025) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-16639 · Cited by 40
Afjal, M. (2023) · Library Hi Tech · doi:10.1108/lht-07-2023-0322 · Cited by 38
Ahmed, I., Hajiabadi, M., Raihan, A. S., Shafie, M. R., & Khosravi, H. (2023) · International Journal of Data Mining Modelling and Management · doi:10.1504/ijdmmm.2024.10061138 · Cited by 40
Kollitsch, L., Eredics, K., Marszalek, M., Rauchenwald, M., Brookman-May, S. D., Burger, M., Körner-Riffard, K., & May, M. (2024) · World Journal of Urology · doi:10.1007/s00345-023-04749-6 · Cited by 38
Fakhri, M. M., Ahmar, A. S., Isma, A., Rosidah, R., & Fadhilatunisa, D. (2024) · EduLine Journal of Education and Learning Innovation · doi:10.35877/454ri.eduline2592 · Cited by 39
Exploring the role of ChatGPT in medical image analysis
Handa, P., Chhabra, D., Goel, N., & Krishnan, S. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105292 · Cited by 38
Morreel, S., Verhoeven, V., & Mathysen, D. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000349 · Cited by 37
Artificial Intelligence in Physical Education and Sports: New Horizons with ChatGPT
GENÇ, N. (2023) · Akdeniz Spor Bilimleri Dergisi · doi:10.38021/asbid.1291604 · Cited by 37
Chaker, S. C., Hung, Y., Saad, M., Golinko, M. S., & Galdyn, I. A. (2024) · The Cleft Palate-Craniofacial Journal · doi:10.1177/10556656231223596 · Cited by 37
Welcome Your New AI Teammate: On Safety Analysis by Leashing Large Language Models
Nouri, A., Cabrero‐Daniel, B., Törner, F., Sivencrona, H., & Berger, C. (2024) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3644815.3644953 · Cited by 37
Law, S., Oldfield, B. J., Yang, W., & Collaborative, G. O. (2024) · Obesity Reviews · doi:10.1111/obr.13746 · Cited by 37
ChatGPT giving advice on how to cheat in university assignments: how workable are its suggestions?
Spennemann, D. H. R., Biles, J. M., Brown, L., Ireland, M. F., Longmore, L., Singh, C., Wallis, A., & Ward, C. (2024) · Interactive Technology and Smart Education · doi:10.1108/itse-10-2023-0195 · Cited by 37
Exploring the role of ChatGPT in higher education institutions: Where does Latin America stand?
Rivadeneira, L., de Luna, D. B., & Fernández-Corrales, C. (2024) · Digital Government Research and Practice · doi:10.1145/3689370 · Cited by 37
Evaluating ChatGPT and Google Gemini Performance and Implications in Turkish Dental Education
Kınıkoğlu, İ. (2025) · Cureus · doi:10.7759/cureus.77292 · Cited by 37
Yao, Z., Duan, L., Xu, S., Chi, L., & Sheng, D. (2025) · JMIR Medical Informatics · doi:10.2196/69485 · Cited by 38
Self-citationIdentifying ChatGPT-written OBGYN abstracts using a simple tool
Levin, G., Meyer, R., Kadoch, E., & Brezinov, Y. (2023) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2023.100936 · Cited by 37
Tajik, E., & Tajik, F. (2023) · doi:10.36227/techrxiv.22589497 · Cited by 37
Thomae, A. V., Witt, C. M., & Barth, J. (2024) · JMIR Medical Education · doi:10.2196/50545 · Cited by 36
Gin, B. C., LaForge, K., Burk-Rafel, J., & Boscardin, C. K. (2025) · Academic Medicine · doi:10.1097/acm.0000000000006117 · Cited by 36
Bibliometric Overview of ChatGPT: New Perspectives in Social Sciences
Oliński, M., Krukowski, K., & Sieciński, K. (2024) · Publications · doi:10.3390/publications12010009 · Cited by 36
Wang, S., Mo, C., Chen, Y., Dai, X., Wang, H., & Shen, X. (2024) · JMIR Medical Education · doi:10.2196/55595 · Cited by 36
Valentini, M., Szkandera, J., Smolle, M. A., Scheipl, S., Leithner, A., & Andreou, D. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1303319 · Cited by 36
ChatGPT's Gastrointestinal Tumor Board Tango: A limping dance partner?
Aghamaliyev, U., Karimbayli, J., Gießen-Jung, C., Ilmer, M., Unger, K., Andrade, D., Hofmann, F. O., Weniger, M., Angele, M. K., Westphalen, C. B., et al. (2024) · European Journal of Cancer · doi:10.1016/j.ejca.2024.114100 · Cited by 38
Locating and Mitigating Gender Bias in Large Language Models
Cai, Y., Cao, D., Guo, R., Wen, Y., Liu, G., & Chen, E. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5672-8_40 · Cited by 36
Sadeq, M. A., Ghorab, R. M. F., Ashry, M. H., Abozaid, A. M., Banihani, H. A., Salem, M., Aisheh, M. T. A., Abuzahra, S., Mourid, M. R., Assker, M. M., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-68996-2 · Cited by 36
How GPT models perform on the United States medical licensing examination: a systematic review
Brin, D., Sorin, V., Konen, E., Nadkarni, G. N., Glicksberg, B. S., & Klang, E. (2024) · Discover Applied Sciences · doi:10.1007/s42452-024-06194-5 · Cited by 36
Mahmoud, R., Shuster, A., Kleinman, S., Arbel, S., Ianculovici, C., & Peleg, O. (2024) · Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.joms.2024.11.007 · Cited by 36
A survey of datasets in medicine for large language models
Zhang, D., Xue, X., Gao, P., Jin, Z., Hu, M., Wu, Y., & Ying, X. (2024) · Intelligence & Robotics · doi:10.20517/ir.2024.27 · Cited by 36
Arvidsson, R., Gunnarsson, R. K., Entezarjou, A., Sundemo, D. S., & Wikberg, C. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-086148 · Cited by 36
Large-language-model-driven agents for fire evacuation simulation in a cellular automata environment
Dang, P., Zhu, J., Li, W., Xie, Y., & Zhang, H. (2025) · Safety Science · doi:10.1016/j.ssci.2025.106935 · Cited by 36
Doo, F. X., Cook, T. S., Siegel, E. L., Joshi, A., Parekh, V. S., Elahi, A., & Yi, P. H. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.07.007 · Cited by 36
Alshahrani, J. A., Alshahrani, A. S., Alshahrani, A. M., Alshalaan, A. M., Alhumam, M. N., & Alshahrani, N. Z. (2023) · Cureus · doi:10.7759/cureus.44216 · Cited by 36
Leveraging AI to enhance learning
Volante, L., DeLuca, C., & Klinger, D. A. (2023) · Phi Delta Kappan · doi:10.1177/00317217231197475 · Cited by 37
NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical NotesPreprint
Wang, J., Yao, Z., Yang, Z., Zhou, H., Li, R., Wang, X., Xu, Y., & Yu, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.15959 · Cited by 37
ChatGPT: A Conceptual Review of Applications and Utility in the Field of Medicine
Rao, S. J., Isath, A., Krishnan, P., Tangsrivimol, J. A., Virk, H. U. H., Wang, Z., Glicksberg, B. S., & Krittanawong, C. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02075-x · Cited by 35
Wiredu, J. K., Abuba, N. S., & Zakaria, H. (2024) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2024/v17i7491 · Cited by 35
Untangling Critical Interaction with AI in Students’ Written Assessment
Shibani, A., Knight, S., Kitto, K., Karunanayake, A., & Buckingham Shum, S. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3651083 · Cited by 35
Ghanem, D., Covarrubias, O., Raad, M., LaPorte, D. M., & Shafiq, B. (2023) · JBJS Open Access · doi:10.2106/jbjs.oa.23.00103 · Cited by 35
Accuracy and reproducibility of ChatGPT 's free version answers about endometriosis
Yüksel, B., & Simavi, M. A. (2023) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.15309 · Cited by 35
Babu, C. V. S., & Akshara, P. M. (2023) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-0502-7.ch011 · Cited by 35
AI and Human Reasoning: Qualitative Research in the Age of Large Language Models
Bano, M., Zowghi, D., & Whittle, J. (2024) · AI Ethics Journal · doi:10.47289/aiej20240122 · Cited by 35
Microcreativity with chat generative pre-trained transformer: Learnings in virtual space
Fialho, L. M. F., Neves, V. N. S., & do Nascimento, K. A. S. (2024) · Journal of Technology and Science Education · doi:10.3926/jotse.2338 · Cited by 35
ChatGPT Earns American Board Certification in Hand Surgery
Ghanem, D., Nassar, J. E., Bachour, J. E., & Hanna, T. (2024) · Hand surgery & rehabilitation · doi:10.1016/j.hansur.2024.101688 · Cited by 35
Zaidat, B., Shrestha, N., Rosenberg, A. M., Ahmed, W., Rajjoub, R., Hoang, T. J., Mejia, M. R., Duey, A. H., Tang, J. E., Kim, J., et al. (2024) · Neurospine · doi:10.14245/ns.2347310.655 · Cited by 35
Large Language Models in Pediatric Education: Current Uses and Future Potential
Suresh, S., & Misra, S. M. (2024) · PEDIATRICS · doi:10.1542/peds.2023-064683 · Cited by 35
Performance of Large Language Models on the Korean Dental Licensing Examination: A Comparative Study
Kim, W., Kim, B.-C., & Yeom, H.-G. (2024) · International Dental Journal · doi:10.1016/j.identj.2024.09.002 · Cited by 35
Ozkan, E., Tekin, A., Ozkan, M. C., Cabrera, D., Niven, A. S., & Dong, Y. (2025) · JMIR Medical Education · doi:10.2196/58801 · Cited by 35
A consensus statement on the use of digital twins in medicine
Iqbal, J. D., Krauthammer, M. O., Witt, C. M., Biller‐Andorno, N., & Christen, M. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01897-4 · Cited by 35
Ngo, T. T. A., Vo, T. T. A., & Phan, M. T. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105789 · Cited by 37
Knowing When to Abstain: Medical LLMs Under Clinical Uncertainty
Machcha, S., Yerra, S., Gupta, S., Sahoo, A., Sultana, S., Yu, H., & Yao, Z. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.291 · Cited by 35
Khorshidi, H., Mohammadi, A., Yousem, D. M., Abolghasemi, J., Ansari, G., Mirza-Aghazadeh-Attari, M., Acharya, U. R., & Abbasian Ardakani, A. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101314 · Cited by 30
Comparison of the Audiological Knowledge of Three Chatbots: ChatGPT, Bing Chat, and Bard
Jędrzejczak, W. W., & Kochanek, K. (2024) · Audiology and Neurotology · doi:10.1159/000538983 · Cited by 35
Passing is Great: Can ChatGPT Conduct USMLE Exams?
Biswas, S. S. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03224-y · Cited by 35
So, H., Jang, H., Kim, M., & Choi, J. (2023) · Asia Pacific Journal of Education · doi:10.1080/02188791.2023.2294699 · Cited by 34
Evaluation of ChatGPT as a Tool for Answering Clinical Questions in Pharmacy Practice
Munir, F., Gehres, A., Wai, D., & Song, L. (2024) · Journal of Pharmacy Practice · doi:10.1177/08971900241256731 · Cited by 32
Samala, A. D., Sokolova, E. V., Grassini, S., & Rawas, S. (2024) · International Journal of Evaluation and Research in Education (IJERE) · doi:10.11591/ijere.v13i4.28119 · Cited by 34
Large language models for diabetes training: a prospective study
Li, H., Jiang, Z., Guan, Z., Bao, Y., Liu, Y., Hu, T., Li, J., Liu, R., Wu, L., Cheng, D., et al. (2025) · 中国科学通报:英文版 · doi:10.1016/j.scib.2025.01.034 · Cited by 34
Zhang, Y., Tian, H., & Lu, J. (2025) · BMC Psychology · doi:10.1186/s40359-025-03404-6 · Cited by 34
Safavi‐Naini, S. A. A., Ali, S., Shahab, O., Shahhoseini, Z., Savage, T., Rafiee, S., Samaan, J. S., Shabeeb, R. Q. A., Ladak, F., Yang, J. O., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02174-0 · Cited by 34
Utilization of ChatGPT-4 in Plastic and Reconstructive Surgery: A Narrative Review
Aljindan, F., Shawosh, M. H., Altamimi, L. A., Arif, S., & Mortada, H. H. (2023) · Plastic & Reconstructive Surgery Global Open · doi:10.1097/gox.0000000000005305 · Cited by 34
Knoedler, L., Knoedler, S. A., Hoch, C. C., Prantl, L., Frank, K., Soiderer, L., Cotofana, S. C., Dorafshar, A. H., Schenck, T. L., Vollbach, F. H., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-63997-7 · Cited by 34
Morjaria, L., Burns, L., Bracken, K., Levinson, A. J., Ngo, Q. N., Lee, M., & Sibbald, M. (2024) · International Medical Education · doi:10.3390/ime3010004 · Cited by 33
“Shortly after the release of ChatGPT, it demonstrated the ability to pass well-known standardized multiple-choice assessments including the United States Medical Licensing Examination (USMLE) written by American medical students [6,7].”
Draschl, A., Hauer, G., Fischerauer, S. F., Kogler, A., Leitner, L., Andreou, D., Leithner, A., & Sadoghi, P. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12206655 · Cited by 29
Meşe, İ., Taşlıçay, C. A., Kuzan, B. N., Kuzan, T. Y., & Sivrioğlu, A. K. (2023) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2023.232496 · Cited by 33
Liang, R., Zhao, A., Lei, P., Xu, X., Zhong, J., Wu, F., Yi, F., Zhang, S., Wu, S., & Hou, J. (2024) · Annals of Surgical Oncology · doi:10.1245/s10434-024-15107-0 · Cited by 33
Beck, S. M., Kuhner, M., Haar, M., Daubmann, A., Semmann, M., & Kluge, S. (2024) · Scandinavian Journal of Trauma Resuscitation and Emergency Medicine · doi:10.1186/s13049-024-01266-2 · Cited by 34
Al-Khater, K. M. (2024) · Clinical Anatomy · doi:10.1002/ca.24243 · Cited by 33
The Practice of ChatGPT in English Teaching and Learning in Vietnam: A Systematic Review
Pham, M. T., & Cao, T. X. T. (2025) · International Journal of TESOL & Education · doi:10.54855/ijte.25513 · Cited by 33
On ChatGPT: what promise remains for multiple choice assessment?
Gonsalves, C. (2023) · Journal of Learning Development in Higher Education · doi:10.47408/jldhe.vi27.1009 · Cited by 23
KIZILGEÇİT, M., Çinici, M., & Okan, N. (2023) · Ağrı İbrahim Çeçen Üniversitesi Sosyal Bilimler Enstitüsü Dergisi · doi:10.31463/aicusbed.1275061 · Cited by 33
Assessing the applicability and appropriateness of ChatGPT in answering clinical pharmacy questions
Fournier, A., Fallet, C., Sadeghipour, F., & Perrottet, N. (2024) · Annales Pharmaceutiques Françaises · doi:10.1016/j.pharma.2023.11.001 · Cited by 28
Bartoli, A., May, A. T., Awadhi, A. A., & Schaller, K. (2023) · Brain and Spine · doi:10.1016/j.bas.2023.102715 · Cited by 33
Stengel, F. C., Stienen, M. N., Ivanov, M., Gandía-González, M. L., Raffa, G., Ganau, M., Whitfield, P. C., & Motov, S. (2024) · Brain and Spine · doi:10.1016/j.bas.2024.102765 · Cited by 33
Does the Information Quality of ChatGPT Meet the Requirements of Orthopedics and Trauma Surgery?
Kasapovic, A., Ali, T., Babasiz, M., Bojko, J., Gathen, M., Kaczmarczyk, R., & Roos, J. (2024) · Cureus · doi:10.7759/cureus.60318 · Cited by 33
An Evaluation of Reasoning Capabilities of Large Language Models in Financial Sentiment Analysis
Du, K., Xing, F., Mao, R., & Cambria, E. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00042 · Cited by 33
Komorowski, M., Lopez, M. D. P. A., & Chang, A. C. (2023) · Intensive Care Medicine · doi:10.1007/s00134-023-07096-7 · Cited by 33
Hudon, A., Kiepura, B., Pelletier, M., & Phan, V. (2024) · JMIR Medical Education · doi:10.2196/54067 · Cited by 32
Peng, Z., Ma, R., Zhang, Y., Yan, M., Lu, J. L., Cheng, Q., Liao, J., Zhang, Y., Wang, J., Zhao, Y., et al. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1323924 · Cited by 32
Impact of ChatGPT on the Academic Writing Quality of Senior High School Students
Maghamil, M. C., & Sieras, S. G. (2024) · Journal of English Language Teaching and Applied Linguistics · doi:10.32996/jeltal.2024.6.2.14 · Cited by 32
Metadata
- Authors
- Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D.
- Affiliations
- Yale University (US)
- University College Dublin (IE)
- Corresponding author
- Chartash, David
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-02-08
- Volume
- 9
- Pages
- e45312
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC9947764
- OpenAlex
- W4319460874
- Semantic Scholar
- cd4d112f3f9120d0715f22a9de2ce4720822368c
- Cited by
- 2,078
- References
- 21
- Citations by year
- 2026: 351 · 2025: 550 · 2024: 699 · 2023: 474
- FWCI
- 82.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- National Heart, Lung, and Blood Institute (T35HL007649)
- National Institute of Diabetes and Digestive and Kidney Diseases (T35DK104689)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE Step 1, USMLE Step 2, medical licensing examinations, question difficulty, GPT-3, medical education, knowledge assessment, AMBOSS, National Board of Medical Examiners, artificial intelligence in medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.