How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. Objective: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. Methods: We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Results: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. Conclusions: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT’s capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Template of question posed to each large language model (LLM), including both AMBOSS Attending Tip and the response from Chat Generative Pre-trained Transformer (ChatGPT). The correct answer to this question is “E. Zidovudine (AZT).” In the case of GPT-3, prompt engineering was necessary, with: "Please answer this multiple choice question:" + question as described previously + "Correct answer is." As GPT-3 is inherently a nondialogic model, this was necessary to reduce model hallucinations and force a clear answer [17].](https://figures.cochat.ai/paper-figures/583a8f9972504138a5ac67eaf322285a/how-does-chatgpt-perform-on-the-united-states-medical-licensing-examination-figure-1.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, like GPT-4, have shown medical knowledge despite lacking medicine-specific training [10, 29].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, several valid concerns were raised, including the possibility of generating inaccurate content, along with ethical issues, including the risk of bias, plagiarism, and copyright issues [9,18,40,41].”
From Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study (2023) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, analyses of ChatGPT show structured, non-random response patterns consistent with trait-like reasoning abilities18.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the majority of clinical reasoning evaluations on LLMs have been conducted using standardized exams (the United States Medical Licensing Examination) or digital case reports [14,19].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [14]
Experimental — accuracy not yet measured
39 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Springer, L., Stanne, M. E., & Donovan, S. S. (1999) · Review of Educational Research · doi:10.3102/00346543069001021 · Cited by 1,943 · cited 1×
“Small group education has been shown to be a highly efficacious method of teaching [23,24].”
MethodThe use of reflection in medical education: AMEE Guide No. 44
Sandars, J. (2009) · Medical Teacher · doi:10.1080/01421590903050374 · Cited by 1,085
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
Research works printed but not linked: 2
- 10.48550/arxiv.2108.01764 · DOI
- Wang, DC, 2020, Yale J Biol Med, 93, 441
Cited by
Items 501–600 of 1,047
Strzalkowski, P., Strzalkowska, A., Chhablani, J. K., Pfau, K., Errera, M., Roth, M., Schaub, F., Bechrakis, N. E., Hoerauf, H., Reiter, C., et al. (2024) · International Journal of Retina and Vitreous · doi:10.1186/s40942-024-00579-9 · Cited by 50
Evaluation of the performance of ChatGPT‐4 and ChatGPT‐4o as a learning tool in endodontics
Öztürk, E. A., Gökduman, C. T., & Çanakçı, B. C. (2025) · International Endodontic Journal · doi:10.1111/iej.14217 · Cited by 50
Shahzad, M. F., Xu, S., An, X., & Asif, M. (2025) · SAGE Open · doi:10.1177/21582440251368594 · Cited by 50
Angel, M. C., Rinehart, J. B., Canneson, M. P., & Baldi, P. (2024) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000006892 · Cited by 50
CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical StudyPreprint
Guan, Z., Wu, Z., Liu, Z., Wu, D., Ren, H., Li, Q., Li, X., & Liu, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.11346 · Cited by 50
Riedel, M., Kaefinger, K., Stuehrenberg, A., Ritter, V., Amann, N., Graf, A., Recker, F., Klein, E., Kiechle, M., Riedel, F., et al. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1296615 · Cited by 49
Meo, S. A., Al-Khlaiwi, T., AbuKhalaf, A. A., Meo, A. S., & Klonoff, D. C. (2025) · Journal of Diabetes Science and Technology · doi:10.1177/19322968231203987 · Cited by 49
Ilgaz, H. B., & Çelik, Z. (2023) · Cureus · doi:10.7759/cureus.45301 · Cited by 49
“These models have been helpful in solving complex medical problems, interpreting radiology reports, being used to diagnose diseases, writing scientific articles or answering and generating different medical exam questions and have shown varying degrees of accuracy in these fields [2-5,7,8,10,11,15].”
Can you spot the bot? Identifying AI-generated writing in college essays
Waltzer, T., Pilegard, C., & Heyman, G. D. (2024) · International Journal for Educational Integrity · doi:10.1007/s40979-024-00158-3 · Cited by 49
Kumah-Crystal, Y., Mankowitz, S., Embi, P., & Lehmann, C. U. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad104 · Cited by 44
Unproctored online exams provide meaningful assessment of student learning
Chan, J. C. K., & Ahn, D. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2302020120 · Cited by 50
Anastasio, A. T., Mills, F. B., Karavan, M. P., & Adams, S. B. (2023) · Foot & Ankle Orthopaedics · doi:10.1177/24730114231209919 · Cited by 49
Evaluating GPT-4V’s performance in the Japanese national dental examination: A challenge explored
Morishita, M., Fukuda, H., Muraoka, K., Nakamura, T., Hayashi, M., Yoshioka, I., Ono, K., & Awano, S. (2023) · Journal of Dental Sciences · doi:10.1016/j.jds.2023.12.007 · Cited by 49
Assessment of Pathology Domain-Specific Knowledge of ChatGPT and Comparison to Human Performance
Wang, A. Y., Lin, S., Tran, C., Homer, R. J., Wilsdon, D., Walsh, J. C., Goebel, E. A., Sansano, I., Sonawane, S., Cockenpot, V., et al. (2024) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2023-0296-oa · Cited by 43
Unveiling GPT-4V's hidden challenges behind high accuracy on USMLE questions: Observational Study
Yang, Z., Yao, Z., Tasmin, M., Vashisht, P., Jang, W. S., Ouyang, F., Wang, B., McManus, D. D., Berlowitz, D. R., & Yu, H. Q. (2025) · Journal of Medical Internet Research · doi:10.2196/65146 · Cited by 49
Generative Artificial Intelligence in Anatomic Pathology
Brodsky, V., Ullah, E., Bychkov, A., Song, A. H., Walk, E. E., Louis, P., Rasool, G., Singh, R. S., Mahmood, F., Bui, M. M., et al. (2025) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0215-ra · Cited by 48
Performance of ChatGPT on registered nurse license exam in Taiwan: a descriptive study
Huang, H. (2023) · Healthcare · doi:10.3390/healthcare11212855 · Cited by 48
Haddad, F., & Saade, J. S. (2024) · JMIR Medical Education · doi:10.2196/50842 · Cited by 48
Can ChatGPT pass the thoracic surgery exam?
Gencer, A., & Aydin, S. (2023) · The American Journal of the Medical Sciences · doi:10.1016/j.amjms.2023.08.001 · Cited by 46
Garabet, R., Mackey, B. P., Cross, J., & Weingarten, M. S. (2023) · Medical Science Educator · doi:10.1007/s40670-023-01956-z · Cited by 48
Canagasuriam, D., & Lukacik, E. (2024) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12491 · Cited by 48
Applying fine-tuned LLMs for reducing data needs in load profile analysis
Hu, Y., Kim, H., Ye, K., & Lü, N. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.124666 · Cited by 48
Assessing AI Simplification of Medical Texts: Readability and Content Fidelity
Picton, B., Andalib, S., Spina, A., Camp, B., Solomon, S. S., Liang, J., Chen, P. M., Chen, J. W., Hsu, F. P. K., & Oh, M. Y. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105743 · Cited by 48
Hasan, W. U., Zaman, K. T., Wang, X., Li, J., Xie, B., & Tao, C. (2024) · npj Biomedical Innovations. · doi:10.1038/s44385-024-00004-8 · Cited by 48
The Evolution of Learning: Assessing the Transformative Impact of Generative AI on Higher Education
Krause, S., Panchal, B. H., & Ubhe, N. (2024) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-97-9255-9_24 · Cited by 48
Multimodal Large Language Models for Medicine: A Comprehensive SurveyPreprint
Ye, J., & Tang, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.21051 · Cited by 49
Shin, H., De Gagne, J. C., Kim, S. S., & Hong, M. (2024) · CIN: Computers, Informatics, Nursing · doi:10.1097/cin.0000000000001177 · Cited by 47
Shi, W., Zhuang, Y., Zhu, Y., Iwinski, H., Wattenbarger, M., & Wang, M. D. (2023) · Proceedings of the 14th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics · doi:10.1145/3584371.3612956 · Cited by 47
Ali, T., & Kostakos, P. (2023) · arXiv · doi:10.48550/arxiv.2309.16021 · Cited by 47
Huynh, L. M., Bonebrake, B. T., Schultis, K., Quach, A., & Deibert, C. M. (2023) · Urology Practice · doi:10.1097/upj.0000000000000406 · Cited by 46
RetractedArtificial Intelligence in Ecology: A Commentary on a Chatbot's Perspective
Haghighi, S. R., Saqalaksari, M. P., & Johnson, S. N. (2023) · Bulletin of the Ecological Society of America · doi:10.1002/bes2.2097 · Cited by 47
Exploring the Latest Applications of OpenAI and ChatGPT: An In-Depth Survey
Hong, Z., & Shao, H. (2023) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.030649 · Cited by 47
Yin, M., Han, B., Ryu, S., & Hua, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48057-7_18 · Cited by 47
Ülkü, A. (2023) · Tourism & Management Studies · doi:10.18089/tms.2023.190402 · Cited by 48
Performance of ChatGPT and Bard in self-assessment questions for nephrology board renewal
Noda, R., Izaki, Y., Kitano, F., Komatsu, J., Ichikawa, D., & Shibagaki, Y. (2024) · Clinical and Experimental Nephrology · doi:10.1007/s10157-023-02451-w · Cited by 38
Lambert, R. A., Choo, Z.-Y., Gradwohl, K. M., Schroedl, L. M., & de Luzuriaga, A. M. R. (2024) · JMIR Dermatology · doi:10.2196/55898 · Cited by 48
A question-answering framework for automated abstract screening using large language models
Akinseloyin, O., Jiang, X., & Palade, V. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae166 · Cited by 28
Elbaz, A. M., Salem, I. E., Darwish, A., Alkathiri, N. A., Mathew, V., & Al-Kaaf, H. A. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100324 · Cited by 47
Uso de la inteligencia artificial en la educación universitaria: Exploración bibliométrica
Corzo-Zavaleta, J., Navarro-Castillo, Y. M., & Ugaz-Rivero, M. (2025) · Desde el Sur Revista de Ciencias Humanas y Sociales de la Universidad Científica del Sur · doi:10.21142/des-1701-2025-0010 · Cited by 47
Comparing diagnostic skills in endodontic cases: dental students versus ChatGPT-4o
Durmazpınar, P. M., & Ekmekci, E. (2025) · BMC Oral Health · doi:10.1186/s12903-025-05857-y · Cited by 47
Can ChatGPT pass the Vietnamese National High School Graduation Examination?Preprint
Dao, X.-Q., Le, N.-B., Phan, X.-D., & Ngo, B. B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.09170 · Cited by 47
ChatGPT in Education - A Global and Vietnamese Research OverviewPreprint
Trương, H. (2023) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/r4uhd · Cited by 47
Generative AI in Undergraduate Medical Education: A Rapid Review
Hale, J., Alexander, S., Wright, S. T., & Gilliland, K. (2024) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205241266697 · Cited by 46
Oon, M. L., Syn, N. L., Tan, C. L., Tan, K., & Ng, S. (2024) · Histopathology · doi:10.1111/his.15100 · Cited by 40
The Perception of ChatGPT among Educators: Preliminary Findings
Oğurlu, Ü., & Mossholder, J. (2023) · Research in Social Sciences and Technology · doi:10.46303/ressat.2023.39 · Cited by 46
How funny is ChatGPT? A comparison of human- and A.I.-produced jokes
Gorenz, D., & Schwarz, N. (2024) · PLoS ONE · doi:10.1371/journal.pone.0305364 · Cited by 47
On Cyber-Social Learning: A Critique of Artificial Intelligence in Education
Cope, B., & Kalantzis, M. (2024) · Postdigital science and education · doi:10.1007/978-3-031-64487-0_1 · Cited by 46
Builtjes, L., Bosma, J. S., Prokop, M., van Ginneken, B., & Hering, A. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf109 · Cited by 46
Frosolini, A., Gennaro, P., Cascino, F., & Gabriele, G. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03248-4 · Cited by 45
“Dear editor, we read with interest the letter by Som et al. “Role of Chat GPT in Public Health” which briefly resembled the possible role of Chat GPT to address inquiries about health promotion and disease prevention techniques [1].”
BackgroundMethodInfluentialSciberras, M., Farrugia, Y., Gordon, H., Furfaro, F., Allocca, M., Torres, J., Arebi, N., Fiorino, G., Iacucci, M., Verstockt, B., et al. (2024) · Journal of Crohn's and Colitis · doi:10.1093/ecco-jcc/jjae040 · Cited by 45
Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians
Hadi, A., Tran, E., Nagarajan, B., & Kirpalani, A. (2024) · PLOS ONE · doi:10.1371/journal.pone.0307383 · Cited by 45
A bibliometric analysis of publications on ChatGPT in education: Research patterns and topics
ZHELTUKHINA, M. R., Sergeeva, O. V., Masalimova, A. R., Budkevich, R. L., Kosarenko, N. N., & Nesterov, G. V. (2024) · Online Journal of Communication and Media Technologies · doi:10.30935/ojcmt/14103 · Cited by 46
Fujimoto, M., Kuroda, H., Katayama, T., Yamaguchi, A., Katagiri, N., Kagawa, K., Tsukimoto, S., Nakano, A., Imaizumi, U., Sato, A. B., et al. (2024) · Cureus · doi:10.7759/cureus.70302 · Cited by 45
Ohta, K., & Ohta, S. (2023) · Cureus · doi:10.7759/cureus.50369 · Cited by 45
Diagnostic and Management Performance of ChatGPT in Obstetrics and Gynecology
Allahqoli, L., Ghiasvand, M. M., Mazidimoradi, A., Salehiniya, H., & Alkatout, I. (2023) · Gynecologic and Obstetric Investigation · doi:10.1159/000533177 · Cited by 40
Ignjatović, A., & Stevanović, L. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.28 · Cited by 38
Thomas, A., Duggal, H. K., Khatri, P., & Corvello, V. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102619 · Cited by 45
The Roles of Social Perception and AI Anxiety in Individuals’ Attitudes Toward ChatGPT in Education
Wang, C., Li, X., Liang, Z., Sheng, Y., Zhao, Q., & Chen, S. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2365453 · Cited by 44
Unlocking innovation: how enjoyment drives GenAI use in higher education
Cano, J. R., & Núñez, N. A. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1483853 · Cited by 44
Maaß, L., Grab‐Kroll, C., Koerner, J., Öchsner, W., Schön, M., Messerer, D. A. C., Boeckers, T. M., & Böckers, A. (2024) · Journal of CME · doi:10.1080/28338073.2024.2437293 · Cited by 44
Steybe, D., Poxleitner, P. J., Aljohani, S., Herlofson, B. B., Nicolatou‐Galitis, O., Patel, V., Fedele, S., Kwon, T., Fusco, V., Pichardo, S. E. C., et al. (2025) · Journal of Cranio-Maxillofacial Surgery · doi:10.1016/j.jcms.2024.12.009 · Cited by 44
CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language ModelsPreprint
Fu, L., Chai, H., Luo, S., Du, K., Zhang, W., Fan, L., Lei, J., Rui, R., Lin, J., Fang, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.01940 · Cited by 44
Concerns About Using ChatGPT in Education
Lin, S.-M., Chung, H.-H., Chung, F., & Lan, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40113-8_4 · Cited by 44
On the Education Effect of CHATGPT: Is AI CHATGPT to Dominate Education Career Profession?
Ahmed, Y. A., & Sharo, A. (2023) · 2023 International Conference on Intelligent Computing, Communication, Networking and Services (ICCNS) · doi:10.1109/iccns58795.2023.10192993 · Cited by 34
Innovating Personalized Nephrology Care: Exploring the Potential Utilization of ChatGPT
Miao, J., Thongprayoon, C., Suppadungsuk, S., Garcia Valencia, O. A., Qureshi, F., & Cheungpasitporn, W. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13121681 · Cited by 36
Molla, C., Mani, L., Bhuiyan, M. R. I., & Hossain, R. (2023) · MIGRATION LETTERS · doi:10.59670/ml.v20is9.4918 · Cited by 44
Al-Sharif, E. M., Penteado, R. C., Dib El Jalbout, N., Topilow, N. J., Shoji, M. K., Kikkawa, D. O., Liu, C. Y., & Korn, B. S. (2024) · Ophthalmic Plastic & Reconstructive Surgery · doi:10.1097/iop.0000000000002567 · Cited by 40
Prospects for ChatGPT Application in Higher Education: A Scoping Review of International Research
Raitskaya, L., & Lambovska, M. (2024) · Integration of Education · doi:10.15507/1991-9468.114.028.202401.010-021 · Cited by 43
Application of ChatGPT-based blended medical teaching in clinical education of hepatobiliary surgery
Wu, C., Chen, L., Han, M., Li, Z., Yang, N., & Yu, C. (2025) · Medical Teacher · doi:10.1080/0142159x.2024.2339412 · Cited by 35
Szymanski, A., Wimer, B. L., Anuyah, O., Eicher‐Miller, H. A., & Metoyer, R. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641924 · Cited by 43
Assessing the Quality of ChatGPT’s Dietary Advice for College Students from Dietitians’ Perspectives
Liao, L., Chang, L., & Lai, I. (2024) · Nutrients · doi:10.3390/nu16121939 · Cited by 43
Huang, T. Y., Safranek, C. W., Socrates, V., Chartash, D., Wright, D., Dilip, M., Sangal, R. B., & Taylor, R. A. (2024) · Journal of Medical Internet Research · doi:10.2196/60336 · Cited by 43
Self-citationPreparing Educators and Students at Higher Education Institutions for an AI-Driven World
Magrill, J., & Magrill, B. (2024) · Teaching & Learning Inquiry The ISSOTL Journal · doi:10.20343/teachlearninqu.12.16 · Cited by 43
Performance of large language models (LLMs) in providing prostate cancer information
Alasker, A., Alsalamah, S., Alshathri, N. N., Almansour, N., Alsalamah, F., Alghafees, M. A., Alkhamees, M. A., & Alsaikhan, B. H. (2024) · BMC Urology · doi:10.1186/s12894-024-01570-0 · Cited by 45
FoodSky: A food-oriented large language model that can pass the chef and dietetic examinations
Zhou, P., Min, W., Fu, C., Jin, Y., Huang, M., Li, X., Mei, S., & Jiang, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101234 · Cited by 43
Woithe, J., & Filipec, O. (2023) · Jonkoping University Library (Jönköping University) · Cited by 43
Ji, Y., Zhan, Z., Li, T., Zou, X., & Lyu, S. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3554584 · Cited by 42
Parmanto, B., Aryoyudanta, B., Soekinto, T. W., Setiawan, I. M. A., Wang, Y., Hu, H., Saptono, A., & Choi, Y. K. (2024) · JMIR Formative Research · doi:10.2196/54633 · Cited by 42
Zhong, T., Zhu, G., Hou, C., Wang, Y., & Fan, X. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12787-9 · Cited by 42
Wysocka, M., Wysocki, O., Delmas, M., Mutel, V., & Freitas, A. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104724 · Cited by 44
Generative AI & machine learning in surgical education
Hla, D. A., & Hindin, D. I. (2024) · Current Problems in Surgery · doi:10.1016/j.cpsurg.2024.101701 · Cited by 42
Dual retrieving and ranking medical large language model with retrieval augmented generation
Yang, Q., Zuo, H., Su, R., Su, H., Zeng, T., Zhou, H., Wang, R., Chen, J., Lin, Y., Chen, Z., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-00724-w · Cited by 42
ChatGPT in Medical Education: Bibliometric and Visual Analysis
Zhang, Y., Xie, X., & Xu, Q. (2025) · JMIR Medical Education · doi:10.2196/72356 · Cited by 42
Professional Certification Benchmark Dataset: The First 500 Jobs for Large Language Models
Noever, D. A., & Ciolino, M. (2023) · doi:10.5121/csit.2023.131211 · Cited by 41
Alzaid, N., Ghulam, O., Albani, M., Alharbi, R., Othman, M., Taher, H., Albaradie, S. I., & Ahmed, S. (2023) · Cureus · doi:10.7759/cureus.47033 · Cited by 41
Vaishya, R., Iyengar, K. P., Patralekh, M. K., Botchu, R., Shirodkar, K. K., Jain, V. K., Vaish, A., & Scarlat, M. M. (2024) · International Orthopaedics · doi:10.1007/s00264-024-06182-9 · Cited by 41
Trindade, M. A. M., Edirisinghe, G. S., & Luo, L. (2024) · The International Journal of Management Education · doi:10.1016/j.ijme.2024.101104 · Cited by 41
Performance of 4 Artificial Intelligence Chatbots in Answering Endodontic Questions
Abdulrab, S., Abada, H., Mashyakhy, M., Mostafa, N. M., Alhadainy, H. A. H., & Halboub, E. S. (2025) · Journal of Endodontics · doi:10.1016/j.joen.2025.01.002 · Cited by 41
Zhang, K., Li, Z., Lin, Y., Guo, Y., Huang, R., Liu, G., & Zhang, B. (2025) · Scientific Reports · doi:10.1038/s41598-025-21535-z · Cited by 41
Mishra, A., Begley, S. L., Chen, A., Rob, M., Pelcher, I., Ward, M., & Schulder, M. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002598 · Cited by 41
Inteligencia artificial generativa y educación médica
Mayol, J. M. (2023) · Educación Médica · doi:10.1016/j.edumed.2023.100851 · Cited by 42
Sharma, P., Thapa, K., Thapa, D., Dhakal, P., Upadhaya, M. D., Adhikari, S., & Khanal, S. R. (2023) · arXiv · doi:10.48550/arxiv.2307.00112 · Cited by 40
Breeding, T., Martinez, B., Patel, H. S., Nasef, H., Arif, H., Nakayama, D. K., & Elkbuli, A. (2023) · The American Surgeon · doi:10.1177/00031348231180950 · Cited by 40
Artificial Intelligence and the Simulationists
Rodgers, D. L., Needler, M., Robinson, A., Barnes, R., Brosche, T., Hernandez, J., Poore, J., VandeKoppel, P., & Ahmed, R. (2023) · Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000747 · Cited by 34
Christy, M. N., Morris, M. T., Goldfarb, C. A., & Dy, C. J. (2023) · The Journal Of Hand Surgery · doi:10.1016/j.jhsa.2023.10.019 · Cited by 40
Taylor, W. L., Cheng, R., Weinblatt, A. I., Bergstein, V., & Long, W. J. (2024) · The Journal of Arthroplasty · doi:10.1016/j.arth.2024.02.005 · Cited by 40
Batool, I., Naved, N., Kazmi, S. M. R., & Umer, F. (2024) · BDJ Open · doi:10.1038/s41405-024-00226-3 · Cited by 40
Torre, D. M., & Schuwirth, L. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2409936 · Cited by 41
Open Artificial Intelligence Analysis using ChatGPT Integrated with Telegram Bot
Avisyah, G. F., Putra, I. J., & Hidayat, S. S. (2023) · Jurnal ELTIKOM · doi:10.31961/eltikom.v7i1.724 · Cited by 40
Auxiliary use of ChatGPT in surgical diagnosis and treatment
Au, K., & Yang, W. (2023) · International Journal of Surgery · doi:10.1097/js9.0000000000000686 · Cited by 40
Metadata
- Authors
- Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D.
- Affiliations
- Yale University (US)
- University College Dublin (IE)
- Corresponding author
- Chartash, David
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-02-08
- Volume
- 9
- Pages
- e45312
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC9947764
- OpenAlex
- W4319460874
- Semantic Scholar
- cd4d112f3f9120d0715f22a9de2ce4720822368c
- Cited by
- 2,078
- References
- 21
- Citations by year
- 2026: 351 · 2025: 550 · 2024: 699 · 2023: 474
- FWCI
- 82.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- National Heart, Lung, and Blood Institute (T35HL007649)
- National Institute of Diabetes and Digestive and Kidney Diseases (T35DK104689)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE Step 1, USMLE Step 2, medical licensing examinations, question difficulty, GPT-3, medical education, knowledge assessment, AMBOSS, National Board of Medical Examiners, artificial intelligence in medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.