How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. Objective: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. Methods: We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Results: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. Conclusions: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT’s capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Template of question posed to each large language model (LLM), including both AMBOSS Attending Tip and the response from Chat Generative Pre-trained Transformer (ChatGPT). The correct answer to this question is “E. Zidovudine (AZT).” In the case of GPT-3, prompt engineering was necessary, with: "Please answer this multiple choice question:" + question as described previously + "Correct answer is." As GPT-3 is inherently a nondialogic model, this was necessary to reduce model hallucinations and force a clear answer [17].](https://figures.cochat.ai/paper-figures/583a8f9972504138a5ac67eaf322285a/how-does-chatgpt-perform-on-the-united-states-medical-licensing-examination-figure-1.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, like GPT-4, have shown medical knowledge despite lacking medicine-specific training [10, 29].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, several valid concerns were raised, including the possibility of generating inaccurate content, along with ethical issues, including the risk of bias, plagiarism, and copyright issues [9,18,40,41].”
From Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study (2023) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, analyses of ChatGPT show structured, non-random response patterns consistent with trait-like reasoning abilities18.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the majority of clinical reasoning evaluations on LLMs have been conducted using standardized exams (the United States Medical Licensing Examination) or digital case reports [14,19].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [14]
Experimental — accuracy not yet measured
39 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Springer, L., Stanne, M. E., & Donovan, S. S. (1999) · Review of Educational Research · doi:10.3102/00346543069001021 · Cited by 1,943 · cited 1×
“Small group education has been shown to be a highly efficacious method of teaching [23,24].”
MethodThe use of reflection in medical education: AMEE Guide No. 44
Sandars, J. (2009) · Medical Teacher · doi:10.1080/01421590903050374 · Cited by 1,085
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
Research works printed but not linked: 2
- 10.48550/arxiv.2108.01764 · DOI
- Wang, DC, 2020, Yale J Biol Med, 93, 441
Cited by
Items 201–300 of 1,047
Generative artificial intelligence in medicine
Teo, Z. L., Thirunavukarasu, A. J., Elangovan, K., Cheng, H., Moova, P., Soetikno, B., Nielsen, C., Pollreisz, A., Ting, D. S. J., Morris, R. J. T., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03983-2 · Cited by 119
Exploring the opportunities and challenges of ChatGPT in academia
Islam, I., & Islam, M. N. (2024) · Discover Education · doi:10.1007/s44217-024-00114-w · Cited by 119
Could artificial intelligence write mental health nursing care plans?
Woodnutt, S., Allen, C., Snowden, J., Flynn, M., Hall, S., Libberton, P., & Purvis, F. (2024) · Journal of Psychiatric and Mental Health Nursing · doi:10.1111/jpm.12965 · Cited by 89
Huang, K.-L., Liu, Y.-C., & Dong, M.-Q. (2024) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2024.101508 · Cited by 95
Artificial intelligence tools expand scientists' impact but contract science's focus
Hao, Q., Xu, F., Li, Y., & Evans, J. (2026) · Nature · doi:10.1038/s41586-025-09922-y · Cited by 117
Students’ attitudes towards using ChatGPT as a learning tool: the case of the University of Jordan.
O. Ajlouni, A., Abd-Alkareem Wahba, F., & Salem Almahaireh, A. (2023) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v17i18.41753 · Cited by 116
Performance of large language models on a neurology board-style examination
Schubert, M. C., Wick, W., & Venkataramani, V. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.46721 · Cited by 115
HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge
Wang, H., Liu, C., Xi, N., Qiang, Z., Zhao, S., Qin, B., & Liu, T. (2023) · arXiv · doi:10.48550/arxiv.2304.06975 · Cited by 115
“While the generated instructions are required to be diverse enough for unseen tasks (Wang et al., 2022) in the general domain, the correctness of the fact in the responses from the large language model is of more concern in the biomedical domain (Gilson et al., 2023).”
BackgroundCan ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115
Giannos, P., & Delardas, O. (2023) · JMIR Medical Education · doi:10.2196/47737 · Cited by 95
Large language models streamline automated machine learning for clinical studies
Tayebi Arasteh, S., Han, T., Lotfinia, M., Kuhl, C., Kather, J. N., Truhn, D., & Nebelung, S. (2024) · Nature Communications · doi:10.1038/s41467-024-45879-8 · Cited by 114
“Beyond empowering clinicians to use the clinical data to their patients ’ advantage, utilizing LLMs for advanced data analysis provides a less costly and more ef fi cient alternative to hand-crafted ML models 32 .”
Zong, H., Li, J., Wu, E., Wu, R., Lu, J., & Shen, B. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05125-7 · Cited by 114
The Impact of ChatGPT on Learning Motivation: A Study Based on Self-Determination Theory
Zhou, L., & Li, J. (2023) · Education Science and Management · doi:10.56578/esm010103 · Cited by 116
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.2.26 · Cited by 65
Bobula, M. (2024) · Journal of Learning Development in Higher Education · doi:10.47408/jldhe.vi30.1137 · Cited by 113
Ng, F. Y. C., Thirunavukarasu, A. J., Cheng, H., Tan, T. F., Gutierrez, L., Lan, Y., Ong, J. C. L., Chong, Y. S., Ngiam, K. Y., Ho, D., et al. (2023) · Cell Reports Medicine · doi:10.1016/j.xcrm.2023.101230 · Cited by 113
Comparative Performance of ChatGPT and Bard in a Text-Based Radiology Knowledge Assessment
Patil, N. S., Huang, R. S., van der Pol, C. B., & Larocque, N. (2024) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231193716 · Cited by 94
Fatima, A., Shafique, M. A., Alam, K., Fadlalla Ahmed, T. K., & Mustafa, M. S. (2024) · Medicine · doi:10.1097/md.0000000000039250 · Cited by 81
Hadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741 · Cited by 83
Head‐to‐Head Comparison of ChatGPT Versus Google Search for Medical Knowledge Acquisition
Ayoub, N. F., Lee, Y., Grimm, D., & Divi, V. (2024) · Otolaryngology–Head and Neck Surgery · doi:10.1002/ohn.465 · Cited by 111
Meyer, A., Riese, J., & Streichert, T. (2024) · JMIR Medical Education · doi:10.2196/50965 · Cited by 110
“Thus, health care professionals could potentially benefit, especially from GPT-4’s conclusive and often nonobvious insights to multiple-choice questions, as these users have the ability to verify crucial details [13,14,41].”
Performance of ChatGPT on the pharmacist licensing examination in Taiwan
Wang, Y.-M., Shen, H.-W., & Chen, T.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000942 · Cited by 109
Thirunavukarasu, A. J., Mahmood, S., Malem, A., Foster, W. P., Sanghera, R., Hassan, R., Zhou, S., Wong, S. W., Wong, Y. L., Chong, Y. J., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000341 · Cited by 88
The impact of chatbots based on large language models on second language vocabulary acquisition
Zhang, Z., & Huang, X. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25370 · Cited by 107
ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis
Bicknell, B. T., Butler, D., Whalen, S., Ricks, J., Dixon, C. J., Clark, A. B., Spaedy, O., Skelton, A., Edupuganti, N., Dzubinski, L., et al. (2024) · JMIR Medical Education · doi:10.2196/63430 · Cited by 107
“Some studies suggest LLMs perform better on USMLE sample items compared to third-party question banks [20], and newer versions of LLMs such as ChatGPT 4 (GPT-4) outperform their earlier counterparts [22].”
Semrl, N., Feigl, S., Taumberger, N., Bracic, T., Fluhr, H., Blockeel, C., & Kollmann, M. (2023) · Human Reproduction · doi:10.1093/humrep/dead207 · Cited by 68
Using generative artificial intelligence/ChatGPT for academic communication: Students' perspectives
Liu, Y., Park, J., & McMinn, S. (2024) · International Journal of Applied Linguistics · doi:10.1111/ijal.12574 · Cited by 106
Pursnani, V., Sermet, Y., Kurt, M., & Demir, I. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100183 · Cited by 106
Caruccio, L., Cirillo, S., Polese, G., Solimando, G., Sundaramurthy, S., & Tortora, G. (2024) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2024.200336 · Cited by 107
Chatbots in education: Hype or help? A meta-analysis
Laun, M., & Wolff, F. (2025) · Learning and Individual Differences · doi:10.1016/j.lindif.2025.102646 · Cited by 106
Sallam, M., Barakat, M. M., & Sallam, M. (2024) · Interactive Journal of Medical Research · doi:10.2196/54704 · Cited by 105
Yuan, M., Bao, P., Yuan, J., Shen, Y., Chen, Z., Xie, Y., Zhao, J., Li, Q., Chen, Y., Zhang, L., et al. (2024) · Medicine Plus · doi:10.1016/j.medp.2024.100030 · Cited by 70
Ali, T., Kostakos, P., & Sheikhi, S. (2026) · Telecom · doi:10.3390/telecom7030073 · Cited by 104
Li, J., Zong, H., Wu, E., Wu, R., Peng, Z., Zhao, J., Yang, L., Xie, H., & Shen, B. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05738-y · Cited by 104
“Therefore, many researchers believe that Chat-GPT may improve students’ problem-solving ability and reflective learning [9].”
BackgroundAdvancing entity recognition in biomedicine via instruction tuning of large language models
Keloth, V. K., Hu, Y., Xie, Q., Peng, X., Wang, Y., Zheng, A., Selek, M., Raja, K., Wei, C. H., Jin, Q., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae163 · Cited by 104
ChatGPT as an important tool in organizational management: A review of the literature
Ayinde, L., Wibowo, M. P., Ravuri, B., & Emdad, F. B. (2023) · Business Information Review · doi:10.1177/02663821231187991 · Cited by 75
Rossettini, G., Rodeghiero, L., Corradi, F., Cook, C., Pillastrini, P., Turolla, A., Castellini, G., Chiappinotto, S., Gianola, S., & Palese, A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05630-9 · Cited by 103
The application of Chat Generative Pre-trained Transformer in nursing education
Liu, J., Liu, F., Fang, J., & Liu, S. (2023) · Nursing Outlook · doi:10.1016/j.outlook.2023.102064 · Cited by 103
ChatGPT and finetuned BERT: A comparative study for developing intelligent design support systems
Qiu, Y., & Jin, Y. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2023.200308 · Cited by 104
Graph Machine Learning in the Era of Large Language Models (LLMs)
Wang, S., Huang, J., Chen, Z., Song, Y., Tang, W., Mao, H., Fan, W., Liu, H., Liu, X., Yin, D., et al. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3732786 · Cited by 103
Performance and risks of ChatGPT used in drug information: an exploratory real-world analysis
Morath, B., Chiriac, U., Jaszkowski, E., Deiß, C., Nürnberg, H., Hörth, K., Hoppe-Tichy, T., & Green, K. (2024) · European Journal of Hospital Pharmacy · doi:10.1136/ejhpharm-2023-003750 · Cited by 79
Suárez, A., Jiménez, J., de Pedro, M. L., Vázquez, C. A., García, V. D., Sánchez, M. G., & Freire, Y. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.11.058 · Cited by 104
Artificial intelligence chatbot performance in triage of ophthalmic conditions
Lyons, R. J., Arepalli, S. R., Fromal, O., Choi, J. D., & Jain, N. (2024) · Canadian Journal of Ophthalmology · doi:10.1016/j.jcjo.2023.07.016 · Cited by 101
Jing, Y., Wang, H., Chen, X., & Wang, C. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-02751-w · Cited by 73
Benefits, limits, and risks of ChatGPT in medicine
Tangsrivimol, J. A., Darzidehkalani, E., Virk, H. U. H., Wang, Z., Egger, J., Wang, M., Hacking, S., Glicksberg, B. S., Strauss, M., & Krittanawong, C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1518049 · Cited by 76
ChatGPT failed Taiwan’s Family Medicine Board Exam
Weng, T.-L., Wang, Y.-M., Chang, S., Chen, T.-J., & Hwang, S.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000946 · Cited by 100
The Potential Usefulness of ChatGPT in Oral and Maxillofacial Radiology
Mago, J., & Sharma, M. (2023) · Cureus · doi:10.7759/cureus.42133 · Cited by 100
The beginning of ChatGPT – a systematic and bibliometric review of the literature
Baber, H., Nair, K., Gupta, R., & Gurjar, K. (2024) · Information and Learning Sciences · doi:10.1108/ils-04-2023-0035 · Cited by 66
Kıyak, Y. S., Çoşkun, Ö., Budakoğlu, I. İ., & Uluoğlu, C. (2024) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-024-03649-x · Cited by 100
Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis
Yu, H., Fan, L., Li, L., Zhou, J., Ma, Z., Xian, L., Hua, W., He, S., Jin, M., Zhang, Y., et al. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-024-00171-8 · Cited by 100
Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis
Zong, H., Wu, R., Cha, J., Wang, J., Wu, E., Li, J., Zhou, Y. K., Zhang, C., Feng, W.-Z., & Shen, B. (2024) · Journal of Medical Internet Research · doi:10.2196/66114 · Cited by 100
Can large language models write reflectively
Li, Y., Sha, L., Yan, L., Lin, J., Raković, M., Galbraith, K., Lyons, K., Gašević, D., & Chen, G. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100140 · Cited by 99
Tangadulrat, P., Sono, S., & Tangtrakulwanich, B. (2023) · JMIR Medical Education · doi:10.2196/50658 · Cited by 99
Reddy, S. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101304 · Cited by 99
Mooney, P., Cui, W., Guan, B., & Juhász, L. (2023) · ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery (GeoAI) · doi:10.1145/3615886.3627745 · Cited by 101
Exploring the potential of using ChatGPT in physics education
Liang, Y., Zou, D., Xie, H., & Wang, F. L. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00273-7 · Cited by 98
“Besides arithmetic reasoning, ChatGPT has examined its ability to perform clinical reasoning by testing its performance on the United States Medical Licensing Exam, where the results found that ChatGPT achieved near the passing threshold of 60% accuracy (Gilson et al., 2023; Kung et al., 2023).”
MethodLarge language models can outperform humans in social situational judgments
Mittelstädt, J. M., Maier, J., Goerke, P., Zinn, F., & Hermes, M. (2024) · Scientific Reports · doi:10.1038/s41598-024-79048-0 · Cited by 45
Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions About Code
Savelka, J., Agarwal, A., Bogart, C., & Sakr, M. (2023) · Proceedings of the 15th International Conference on Computer Supported Education · doi:10.5220/0011996900003470 · Cited by 56
Yanagita, Y., Yokokawa, D., Uchida, S., Tawara, J., & Ikusaka, M. (2023) · JMIR Formative Research · doi:10.2196/48023 · Cited by 92
Xu, X., Su, Y., Zhang, Y., Wu, Y., & Xu, X. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e26239 · Cited by 55
Artificial Intelligence in Endodontic Education
Aminoshariae, A., Nosrat, A., Nagendrababu, V., Dianat, O., Mohammad‐Rahimi, H., O'Keefe, A. W., & Setzer, F. C. (2024) · Journal of Endodontics · doi:10.1016/j.joen.2024.02.011 · Cited by 96
Potential Benefits and Risks of Artificial Intelligence in Education
Özer, M. (2024) · Bartın University Journal of Faculty of Education · doi:10.14686/buefad.1416087 · Cited by 97
ChatGPT in Clinical Toxicology
Sabry Abdel-Messih, M., & Kamel Boulos, M. N. (2023) · JMIR Medical Education · doi:10.2196/46876 · Cited by 95
“Since its public launch on November 30, 2022, ChatGPT, which ironically has not been specifically trained in medicine, has been taking the medical world by storm [1-3].”
BackgroundSevgi, U. T., Erol, G., Doğruel, Y., Sönmez, O. F., Tubbs, R. S., & Güngör, A. (2023) · Neurosurgical Review · doi:10.1007/s10143-023-01998-2 · Cited by 95
Hopkins, B. S., Nguyen, V. N., Dallas, J., Texakalidis, P., Yang, M., Renn, A., Guerra, G., Kashif, Z., Cheok, S., Zada, G., et al. (2023) · Journal of Neurosurgery · doi:10.3171/2023.2.jns23419 · Cited by 84
Zaleski, A. L., Berkowsky, R. S., Craig, K. J. T., & Pescatello, L. S. (2024) · JMIR Medical Education · doi:10.2196/51308 · Cited by 95
Knowledge, Perceptions and Attitude of Researchers Towards Using ChatGPT in Research
Abdelhafiz, A. S., Ali, A., Maaly, A. M., Ziady, H. H., Sultan, E. A., & Mahgoub, M. A. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02044-4 · Cited by 68
Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations
Quah, B., Zheng, L., Sng, T. J. H., Yong, C. W., & Islam, I. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05881-6 · Cited by 70
Spennemann, D. H. R. (2023) · Knowledge · doi:10.3390/knowledge3030032 · Cited by 93
ChatGPT in society: emerging issues
Farina, M., & Lavazza, A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1130913 · Cited by 66
Gödde, D., Nöhl, S., Wolf, C., Rupert, Y., Rimkus, L., Ehlers, J., Breuckmann, F., & Sellmann, T. (2023) · Journal of Medical Internet Research · doi:10.2196/49368 · Cited by 69
Seeing ChatGPT Through Students’ Eyes: An Analysis of TikTok Data
Haensch, A.-C., Ball, S., Herklotz, M., & Kreuter, F. (2023) · 2023 Big Data Meets Survey Science (BigSurv) · doi:10.1109/bigsurv59479.2023.10486710 · Cited by 62
Yamamoto, A., Koda, M., Ogawa, H., Miyoshi, T., Maeda, Y., Otsuka, F., & Ino, H. (2024) · JMIR Medical Education · doi:10.2196/58753 · Cited by 92
“One study demonstrated that LLMs can accurately answer questions of the United States Medical Licensing Examination (USMLE), demonstrating their use in medical education and assessment [10].”
BackgroundEmbracing ChatGPT for Medical Education: Exploring Its Impact on Doctors and Medical Students
Wu, Y., Zheng, Y., Feng, B., Yang, Y., Kang, K., & Zhao, A. (2024) · JMIR Medical Education · doi:10.2196/52483 · Cited by 92
Pu, D., & Demberg, V. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-srw.1 · Cited by 92
Almansour, M., & Alfhaid, F. M. (2024) · Medicine · doi:10.1097/md.0000000000038955 · Cited by 52
Evaluating the ChatGPT family of models for biomedical reasoning and classification
Chen, S., Li, Y., Lu, S., Van, H., Aerts, H. J. W. L., Savova, G. K., & Bitterman, D. S. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad256 · Cited by 91
Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification
Chen, S., Li, Y., Lu, S., Van, H., Aerts, H. J., Savova, G. K., & Bitterman, D. S. (2023) · arXiv · doi:10.48550/arxiv.2304.02496 · Cited by 91
Proposing a Principle-Based Approach for Teaching AI Ethics in Medical Education
Weidener, L., & Fischer, M. (2024) · JMIR Medical Education · doi:10.2196/55368 · Cited by 90
Zhang, J., Sun, K., Jagadeesh, A., Falakaflaki, P., Kayayan, E., Tao, G., Haghighat Ghahfarokhi, M., Gupta, D., Gupta, A., Gupta, V., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae184 · Cited by 70
E, K., S, P., R, G., R, K. L., A, B., M, G., T, O., S, R., V, R., H, M., et al. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04752-w · Cited by 89
The knowledge and innovation challenges of ChatGPT: A scoping review
Ali, O., Murray, P., Momin, M., & Al–Anzi, F. S. (2023) · Technology in Society · doi:10.1016/j.techsoc.2023.102402 · Cited by 90
The scholarly footprint of ChatGPT: a bibliometric analysis of the early outbreak phase
Farhat, F., Silva, E. S., Hassani, H., Madsen, D. Ø., Sohail, S. S., Himeur, Y., Alam, M. A., & Zafar, A. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1270749 · Cited by 48
Swisher, A. R., Wu, A. W., Liu, G. C., Lee, M., Carle, T. R., & Tang, D. M. (2024) · Otolaryngology · doi:10.1002/ohn.927 · Cited by 91
Shiferaw, M. W., Zheng, T., Winter, A., Mike, L. A., & Chan, L.-N. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02824-5 · Cited by 66
Evolution of Learning: Assessing the Transformative Impact of Generative AI on Higher Education
Krause, S., Panchal, B. H., & Ubhe, N. (2025) · Frontiers of digital education. · doi:10.1007/s44366-025-0058-7 · Cited by 89
Tseng, Y.-C., & Lin, Y.-H. (2024) · Electronic Journal of e-Learning · doi:10.34190/ejel.21.5.3329 · Cited by 88
Performance of ChatGPT on the Peruvian National Licensing Medical Examination: Cross-Sectional Study
Flores-Cohaila, J. A., García-Vicente, A., Vizcarra-Jiménez, S. F., De la Cruz-Galán, J. P., Gutiérrez-Arratia, J. D., Quiroga Torres, B. G., & Taype-Rondan, A. (2023) · JMIR Medical Education · doi:10.2196/48039 · Cited by 82
Toward sustainable smart cities: applications, challenges, and future directions
Ullah, A., Khan, S. H., Ullah, I., Mahmood, T., Nawazish, S., Ali, Z., & Rehman, A. U. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00856-2 · Cited by 87
Evaluating the performance of ChatGPT in answering questions related to pediatric urology
Çağlar, U., Yıldız, O., Meriç, A., Ayrancı, A., Gelmiş, M., Sarılar, Ö., & Özgör, F. (2023) · Journal of Pediatric Urology · doi:10.1016/j.jpurol.2023.08.003 · Cited by 86
Yilmaz, B. E., Yılmaz, B., & Özbey, F. (2025) · BMC Oral Health · doi:10.1186/s12903-025-05926-2 · Cited by 86
Democratizing Financial Knowledge with ChatGPT by OpenAI: Unleashing the Power of TechnologyPreprint
Yue, T., Au, D., Au, C. C., & Iu, K. Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4346152 · Cited by 86
Jin, H. K., Lee, H. E., & Kim, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05944-8 · Cited by 85
Analyzing the User's Sentiments of ChatGPT Using Twitter Data
KORKMAZ, A., AKTÜRK, C., & TALAN, T. (2023) · Iraqi Journal for Computer Science and Mathematics · doi:10.52866/ijcsm.2023.02.02.018 · Cited by 63
Faruk, L. I. D., Rohan, R., Ninrutsirikun, U., & Pal, D. (2023) · doi:10.1145/3628454.3629552 · Cited by 85
Park, H., & Ahn, D. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642785 · Cited by 84
“People are shocked by the continuous stream of news articles and reports with headlines like “ChatGPT Passes Wharton School’s MBA exam [107, 108] and US Medical Licensing Exam” [20, 46].”
BackgroundChow, J. C. L., & Li, K. (2025) · Information · doi:10.3390/info16070549 · Cited by 84
Current status of ChatGPT use in medical education: potentials, challenges, and strategies
Xu, T., Weng, H., Liu, F., Yang, L., Luo, Y., Ding, Z., & Wang, Q. (2024) · Journal of Medical Internet Research · doi:10.2196/57896 · Cited by 84
Chen, J., Zhuo, Z., & Lin, J. (2023) · Sustainability · doi:10.3390/su152416928 · Cited by 61
Applications of Artificial Intelligence in Medical Education: A Systematic Review
Hallquist, E., Gupta, I., Montalbano, M. J., & Loukas, M. (2025) · Cureus · doi:10.7759/cureus.79878 · Cited by 86
Metadata
- Authors
- Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D.
- Affiliations
- Yale University (US)
- University College Dublin (IE)
- Corresponding author
- Chartash, David
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-02-08
- Volume
- 9
- Pages
- e45312
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC9947764
- OpenAlex
- W4319460874
- Semantic Scholar
- cd4d112f3f9120d0715f22a9de2ce4720822368c
- Cited by
- 2,078
- References
- 21
- Citations by year
- 2026: 351 · 2025: 550 · 2024: 699 · 2023: 474
- FWCI
- 82.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- National Heart, Lung, and Blood Institute (T35HL007649)
- National Institute of Diabetes and Digestive and Kidney Diseases (T35DK104689)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE Step 1, USMLE Step 2, medical licensing examinations, question difficulty, GPT-3, medical education, knowledge assessment, AMBOSS, National Board of Medical Examiners, artificial intelligence in medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.