How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. Objective: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. Methods: We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Results: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. Conclusions: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT’s capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Template of question posed to each large language model (LLM), including both AMBOSS Attending Tip and the response from Chat Generative Pre-trained Transformer (ChatGPT). The correct answer to this question is “E. Zidovudine (AZT).” In the case of GPT-3, prompt engineering was necessary, with: "Please answer this multiple choice question:" + question as described previously + "Correct answer is." As GPT-3 is inherently a nondialogic model, this was necessary to reduce model hallucinations and force a clear answer [17].](https://figures.cochat.ai/paper-figures/583a8f9972504138a5ac67eaf322285a/how-does-chatgpt-perform-on-the-united-states-medical-licensing-examination-figure-1.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, like GPT-4, have shown medical knowledge despite lacking medicine-specific training [10, 29].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, several valid concerns were raised, including the possibility of generating inaccurate content, along with ethical issues, including the risk of bias, plagiarism, and copyright issues [9,18,40,41].”
From Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study (2023) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, analyses of ChatGPT show structured, non-random response patterns consistent with trait-like reasoning abilities18.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the majority of clinical reasoning evaluations on LLMs have been conducted using standardized exams (the United States Medical Licensing Examination) or digital case reports [14,19].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [14]
Experimental — accuracy not yet measured
39 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Springer, L., Stanne, M. E., & Donovan, S. S. (1999) · Review of Educational Research · doi:10.3102/00346543069001021 · Cited by 1,943 · cited 1×
“Small group education has been shown to be a highly efficacious method of teaching [23,24].”
MethodThe use of reflection in medical education: AMEE Guide No. 44
Sandars, J. (2009) · Medical Teacher · doi:10.1080/01421590903050374 · Cited by 1,085
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
Research works printed but not linked: 2
- 10.48550/arxiv.2108.01764 · DOI
- Wang, DC, 2020, Yale J Biol Med, 93, 441
Cited by
Items 101–200 of 1,047
Inteligencia artificial generativa y educación: Un análisis desde múltiples perspectivas
García‐Peñalvo, F. J. (2024) · Education in the Knowledge Society (EKS) · doi:10.14201/eks.31942 · Cited by 203
Educational Applications of the ChatGPT AI System: A Systematic Review Research
İpek, Z. H., Gözüm, A. İ. C., Papadakis, S., & Kallogiannakis, M. (2023) · Educational Process International Journal · doi:10.22521/edupij.2023.123.2 · Cited by 201
Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysis
Wei, Q., Yao, Z., Cui, Y., Wei, B., Jin, Z., & Xu, X. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104620 · Cited by 201
Suárez, A., Díaz‐Flores García, V., Algar, J., Gómez Sánchez, M., Llorente de Pedro, M., & Freire, Y. (2024) · International Endodontic Journal · doi:10.1111/iej.13985 · Cited by 200
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00097-9 · Cited by 199
Chakraborty, C., Pal, S., Bhattacharya, M., Dash, S., & Lee, S.-S. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1237704 · Cited by 152
Xie, Y., Seth, I., Hunter-Smith, D. J., Rozen, W. M., Ross, R., & Lee, M. (2023) · Aesthetic Plastic Surgery · doi:10.1007/s00266-023-03338-7 · Cited by 198
“ChatGPT has already demonstrated a passing performance equivalent to an undergraduate third-year medical school student on the US medical licensing exam [15].”
Transformers and large language models in healthcare: A review
Nerella, S., Bandyopadhyay, S., Zhang, J., Contreras, M., Siegel, S., Bumin, A., Silva, B., Sena, J., Shickel, B., Bihorac, A., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102900 · Cited by 197
ChatGPT: promise and challenges for deployment in low- and middle-income countries
Wang, X., Sanders, H. M., Liu, Y., Seang, K., Tran, B. X., Atanasov, A. G., Qiu, Y., Tang, S., Car, J., Wang, Y. X., et al. (2023) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2023.100905 · Cited by 138
Differentiating ChatGPT-Generated and Human-Written Medical Texts: Quantitative Study
Liao, W., Liu, Z., Dai, H., Xu, S., Wu, Z., Zhang, Y., Huang, X., Zhu, D., Cai, H., Li, Q., et al. (2023) · JMIR Medical Education · doi:10.2196/48904 · Cited by 116
Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination
Rosoł, M., Gąsior, J. S., Łaba, J., Korzeniewski, K., & Młyńczak, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-46995-z · Cited by 157
ChatGPT in Dentistry: A Comprehensive Review
Alhaidry, H. M., Fatani, B., Alrayes, J. O., Almana, A. M., & Alfhaed, N. K. (2023) · Cureus · doi:10.7759/cureus.38317 · Cited by 126
Sohail, S. S., Farhat, F., Himeur, Y., Nadeem, M., Madsen, D. Ø., Singh, Y., Atalla, S., & Mansoor, W. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101675 · Cited by 192
“(Huang et al., 2023; Ortega-Martín et al., 2023; Huang et al., 2023; Gao et al., researchers asked ChatGPT for antimicrobial advice in eight hypothetical infections scenario-based questions and evaluated the appropriateness, consistency, safety, and antimicrobial steward-ship implications of its…”
A systematic literature review of empirical research on ChatGPT in education
Albadarin, Y., Saqr, M., Pope, N., & Tukiainen, M. (2024) · Discover Education · doi:10.1007/s44217-024-00138-2 · Cited by 191
Megahed, F. M., Chen, Y.-J., Ferris, J. A., Knoth, S., & Jones-Farmer, L. A. (2024) · Quality Engineering · doi:10.1080/08982112.2023.2206479 · Cited by 190
Massey, P. A., Montgomery, C., & Zhang, A. S. (2023) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-23-00396 · Cited by 190
Friha, O., Amine Ferrag, M., Kantarci, B., Cakmak, B., Ozgun, A., & Ghoualmi-Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 189
Wachinger, J., Bärnighausen, K., Schäfer, L. N., Scott, K., & McMahon, S. A. (2024) · Qualitative Health Research · doi:10.1177/10497323241244669 · Cited by 190
AI and the future of humanity: ChatGPT-4, philosophy and education – Critical responses
Peters, M. A., Jackson, L., Papastephanou, M., Jandrić, P., Lazaroiu, G., Evers, C. W., Cope, B., Kalantzis, M., Araya, D., Tesar, M., et al. (2024) · Educational Philosophy and Theory · doi:10.1080/00131857.2023.2213437 · Cited by 188
“Ask GPT test questions, and it will give mostly correct answers—it can pass US standardized tests for medicine at the level of a third year student (Gilson et al., 2023).”
Applicability of ChatGPT in Assisting to Solve Higher Order Problems in Pathology
Sinha, R. K., Deb Roy, A., Kumar, N., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.35237 · Cited by 187
“They also reported its capability to give reasoning and informative context throughout the majority of replies owing to the dialogic character of the response to inquiries [13].”
BackgroundThirunavukarasu, A. J., Hassan, R., Mahmood, S., Sanghera, R., Barzangi, K., El Mukashfi, M., & Shah, S. (2023) · JMIR Medical Education · doi:10.2196/46599 · Cited by 187
Sequential diagnosis with language models
Nori, H., Daswani, M., Kelly, C. J., Lundberg, S., Ribeiro, M. T., Wilson, M., Liu, X., Sounderajah, V., Carlson, J. M., Lungren, M. P., et al. (2026) · Science · doi:10.48550/arxiv.2506.22405 · Cited by 185
ChatGPT Performs on the Chinese National Medical Licensing Examination
Wang, X., Gong, Z., Wang, G., Jia, J., Xu, Y., Zhao, J., Fan, Q., Wu, S., Hu, W., & Li, X. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01961-0 · Cited by 157
Implementing generative AI (GenAI) in higher education: A systematic review of case studies
Belkina, M., Daniel, S., Nikolic, S., Haque, R., Lyden, S., Neal, P., Grundy, S., & Hassan, G. M. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100407 · Cited by 183
Newton, P., & Xiromeriti, M. (2024) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2023.2299059 · Cited by 70
Temsah, O., Khan, S. A., Chaiah, Y., Senjab, A., Alhasan, K., Jamal, A., Aljamaan, F., Malki, K. H., Halwani, R., Al-Tawfiq, J. A., et al. (2023) · Cureus · doi:10.7759/cureus.37281 · Cited by 179
Understanding University Students’ Acceptance of ChatGPT: Insights from the UTAUT2 Model
Grassini, S., Aasen, M. L., & Møgelvang, A. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2371168 · Cited by 116
Knopp, M. I., Warm, E. J., Weber, D., Kelleher, M., Kinnear, B., Schumacher, D. J., Santen, S. A., Mendonça, E., & Turner, L. (2023) · JMIR Medical Education · doi:10.2196/50373 · Cited by 133
MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes
Abacha, A. B., Yim, W.-W., Fu, Y., Sun, Z., Yetişgen, M., Xia, F., & Lin, T. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1159 · Cited by 176
Wang, H., Wu, W., Dou, Z., He, L., & Yang, L. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105173 · Cited by 176
Exploring student perspectives on generative artificial intelligence in higher education learning
Baidoo-Anu, D., Asamoah, D., Amoako, I., & Mahama, I. (2024) · Discover Education · doi:10.1007/s44217-024-00173-z · Cited by 77
Influence of believed AI involvement on the perception of digital medical advice
Reis, M., Reis, F., & Kunde, W. (2024) · Nature Medicine · doi:10.1038/s41591-024-03180-7 · Cited by 118
Hashem, R., Ali, N., El Zein, F., Fidalgo, P., & Abu Khurma, O. (2023) · Research and Practice in Technology Enhanced Learning · doi:10.58459/rptel.2024.19023 · Cited by 173
Bernabei, M., Colabianchi, S., Falegnami, A., & Costantino, F. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100172 · Cited by 173
Choi, E. P. H., Lee, J. J., Ho, M.-H., Kwok, J. Y. Y., & Lok, K. Y. W. (2023) · Nurse Education Today · doi:10.1016/j.nedt.2023.105796 · Cited by 172
Digital transformation in engineering education: Exploring the potential of AI-assisted learning
Pham, T., Nguyen, T. B., Ha, S., & Nguyen Ngoc, N. T. (2023) · Australasian Journal of Educational Technology · doi:10.14742/ajet.8825 · Cited by 105
Xu, X., Chen, Y., & Miao, J. (2024) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2024.21.6 · Cited by 169
Are Lesson Plans Created by ChatGPT More Effective? An Experimental Study
Karaman, M. R., & Göksu, İ. (2024) · International Journal of Technology in Education · doi:10.46328/ijte.607 · Cited by 96
Let's have a chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Shahriar, S., & Hayawi, K. (2023) · arXiv · doi:10.48550/arxiv.2302.13817 · Cited by 167
Let’s Have a Chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Shahriar, S., & Hayawi, K. (2023) · Artificial Intelligence and Applications · doi:10.47852/bonviewaia3202939 · Cited by 167
Maheshwari, G. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12333-z · Cited by 167
Can ChatGPT improve investment decisions? From a portfolio management perspective
Ko, H., & Lee, J. (2024) · Finance research letters · doi:10.1016/j.frl.2024.105433 · Cited by 167
Feigerlova, E., Hani, H., & Hothersall-Davies, E. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06719-5 · Cited by 166
Generative AI: Implications and Applications for Education
Olga, A., Tzirides, A. O., Saini, A., Zapata, G. C., Searsmith, D., Cope, B., Kalantzis, M., de Castro, V. C., Kourkoulou, T., Jones, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.07605 · Cited by 60
Miao, J., Thongprayoon, C., Suppadungsuk, S., Garcia Valencia, O. A., & Cheungpasitporn, W. (2024) · Medicina · doi:10.3390/medicina60030445 · Cited by 164
Weidener, L., & Fischer, M. (2024) · JMIR Medical Education · doi:10.2196/51247 · Cited by 163
“Furthermore, this could be attributed to the availability of AI-based applications, such as ChatGPT, and increasing opportunities for education on AI in the medical context, as well as AI-based (chat) applications that are knowledgeable in the field of medicine [7,31-33].”
BackgroundVrdoljak, J., Boban, Z., Vilović, M., Kumrić, M., & Božić, J. (2025) · Healthcare · doi:10.3390/healthcare13060603 · Cited by 163
Early applications of ChatGPT in medical practice, education and research
Sedaghat, S. (2023) · Clinical Medicine · doi:10.7861/clinmed.2023-0078 · Cited by 163
Using ChatGPT for human–computer interaction research: a primer
Tabone, W., & de Winter, J. (2023) · Royal Society Open Science · doi:10.1098/rsos.231053 · Cited by 116
Evaluating ChatGPT Performance on the Orthopaedic In-Training Examination
Kung, J. E., Marshall, C., Gauthier, C., Gonzalez, T. A., & Jackson, J. B. (2023) · JBJS Open Access · doi:10.2106/jbjs.oa.23.00056 · Cited by 130
Leveraging ChatGPT to Aid Construction Hazard Recognition and Support Safety Education and Training
Uddin, S. M. J., Albert, A., Ovid, A., & Alsharef, A. (2023) · Sustainability · doi:10.3390/su15097121 · Cited by 158
Holderried, F., Stegemann–Philipps, C., Herschbach, L., Moldt, J.-A., Nevins, A., Griewatz, J., Holderried, M., Herrmann-Werner, A., Festl-Wietek, T., & Mahling, M. (2024) · JMIR Medical Education · doi:10.2196/53961 · Cited by 156
ChatGPT for Future Medical and Dental Research
Fatani, B. (2023) · Cureus · doi:10.7759/cureus.37285 · Cited by 121
Humar, P., Asaad, M., Bengur, F. B., & Nguyen, V. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad130 · Cited by 155
Hoch, C. C., Wollenberg, B., Lüers, J.-C., Knoedler, S., Knoedler, L., Frank, K., Cotofana, S., & Alfertshofer, M. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08051-4 · Cited by 154
“Gilson et al. demonstrated that ChatGPT performs comparably or even surpasses previous models when confronted with questions of similar difficulty and content [14].”
Ibrahim, H., Liu, F., Asim, R., Battu, B., Benabderrahmane, S., Alhafni, B., Adnan, W., Alhanai, T., AlShebli, B., Baghdadi, R., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-38964-3 · Cited by 154
The Capability of ChatGPT in Predicting and Explaining Common Drug-Drug Interactions
Juhi, A., Pipil, N., Santra, S., Mondal, S., Behera, J. K., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.36272 · Cited by 128
Holderried, F., Stegemann-Philipps, C., Herrmann-Werner, A., Festl-Wietek, T., Holderried, M., Eickhoff, C., & Mahling, M. (2024) · JMIR Medical Education · doi:10.2196/59213 · Cited by 152
Ali, K., Barhom, N., Tamimi, F., & Duggal, M. (2024) · European Journal of Dental Education · doi:10.1111/eje.12937 · Cited by 152
Prompt Engineering for Healthcare: Methodologies and Applications
Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.14670 · Cited by 150
“These include simple manually designed template-style prompts, prompts based on knowledge graphs [126], prompts generated based on natural language [127], and prompts that can be embedded as trainable parameters in the model for automatic generation [78].”
Plevris, V., Papazafeiropoulos, G., & Jiménez Rios, A. (2023) · AI · doi:10.3390/ai4040048 · Cited by 81
Salifu, I., Arthur, F., Arkorful, V., Abam Nortey, S., & Solomon Osei-Yaw, R. (2024) · Cogent Social Sciences · doi:10.1080/23311886.2023.2300177 · Cited by 98
ChatGPT in medicine: prospects and challenges: a review article
Tan, S., Xin, X., & Wu, D. (2024) · International Journal of Surgery · doi:10.1097/js9.0000000000001312 · Cited by 126
Artificial intelligence for teaching and learning in schools: The need for pedagogical intelligence
Díaz, B., & Nussbaum, M. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105071 · Cited by 148
Wang, P. Q. (2024) · Current Issues in Tourism · doi:10.1080/13683500.2023.2300030 · Cited by 150
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147
The Role of Artificial Intelligence in Higher Education: ChatGPT Assessment for Anatomy Course
TALAN, T., & KALINKARA, Y. (2023) · Uluslararası Yönetim Bilişim Sistemleri ve Bilgisayar Bilimleri Dergisi · doi:10.33461/uybisbbd.1244777 · Cited by 109
A study on prompt design, advantages and limitations of ChatGPT for deep learning program repair
Cao, J., Li, M., Wen, M., & Cheung, S.-C. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00492-x · Cited by 65
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v2 · Cited by 142
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141
LLM-assisted systematic review of large language models in clinical medicine
Chen, S. F., Alyakin, A., Seas, A., Yang, E., Choi, J. J., Lee, J. V., Chen, A. L., Warman, P. I., Bitolas, R., Steele, R. J., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04229-5 · Cited by 144
Suchman, K., Garg, S., & Trindade, A. J. (2023) · American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002320 · Cited by 139
ChatGPT in higher education - a synthesis of the literature and a future research agenda
Bhullar, P. S., Joshi, M., & Chugh, R. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12723-x · Cited by 139
“Gilson et al. ( 2023 ) evaluated the accuracy, speed and clarity of ChatGPT’s responses to questions on the United States Medical Licensing Examina - tion’s Step 1 and Step 2 tests.”
Toyama, Y., Harigai, A., Abe, M., Nagano, M., Kawabata, M., Seki, Y., & Takase, K. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01491-2 · Cited by 125
Evaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 68
GPT versus Resident Physicians — A Benchmark Based on Official Board Scores
Katz, U., Cohen, E., Shachar, E., Somer, J., Fink, A., Morse, E., Shreiber, B., & Wolf, I. (2024) · NEJM AI · doi:10.1056/aidbp2300192 · Cited by 135
Teaching AI Ethics in Medical Education: A Scoping Review of Current Literature and Practices
Weidener, L., & Fischer, M. (2023) · Perspectives on Medical Education · doi:10.5334/pme.954 · Cited by 133
“First released to the public in November 2022, ChatGPT can be considered as the first consumer-grade and broadly used AI application [10].”
BackgroundTwenty‐first century technologies and language education: Charting a path forward
Kern, R. (2024) · Modern Language Journal · doi:10.1111/modl.12924 · Cited by 133
Alkhaaldi, S. M. I., Kassab, C. H., Dimassi, Z., Oyoun Alsoud, L., Al Fahim, M., Al Hageh, C., & Ibrahim, H. (2023) · JMIR Medical Education · doi:10.2196/51302 · Cited by 132
“It is capable of amalgamating and processing large amounts of data and has received passing scores equivalent to a third-year medical student on steps 1 and 2 of the United States Medical Licensing Exam (USMLE) [7].”
BackgroundHuang, X., Estau, D., Liu, X., Yu, Y., Qin, J., & Li, Z. (2024) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.15896 · Cited by 112
Fraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., & Hilliard, R. (2023) · JMIR mHealth and uHealth · doi:10.2196/49995 · Cited by 128
“These include providing advice to patients on heart disease [12], performing tasks to support primary care physicians [13], and passing medical board exams based on performance in answering sample questions [14].”
BackgroundLi, B., Bonk, C. J., Wang, C., & Kou, X. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3386098 · Cited by 88
Suh, P. S., Shim, W. H., Suh, C. H., Heo, H., Park, C. R., Eom, H. J., Park, K. J., Choe, J., Kim, P. H., Park, H. J., et al. (2024) · Radiology · doi:10.1148/radiol.240273 · Cited by 128
Generative AI, Why, How, and Outcomes: A User Adoption Study
Huy, L. V., Nguyen, H. P. N., Vo‐Thanh, T., Thịnh, N. H., & Dung, T. T. T. (2017) · AIS Transactions on Human-Computer Interaction · doi:10.17705/1thci.00098 · Cited by 128
Firth, J., Torous, J., López‐Gil, J. F., Linardon, J., Milton, A., Lambert, J., Smith, L., Jarić, I., Fabian, H., Vancampfort, D., et al. (2024) · World Psychiatry · doi:10.1002/wps.21188 · Cited by 80
Al-Ashwal, F. Y., Zawiah, M., Gharaibeh, L., Abu-Farha, R., & Bitar, A. N. (2023) · Drug Healthcare and Patient Safety · doi:10.2147/dhps.s425858 · Cited by 125
Jeyaraman, M., Ramasubramanian, S., Balaji, S., Jeyaraman, N., Nallakumarasamy, A., & Sharma, S. (2023) · World Journal of Methodology · doi:10.5662/wjm.v13.i4.170 · Cited by 125
Ayre, J., Mac, O., McCaffery, K., McKay, B. R., Liu, M., Shi, Y., Rezwan, A., & Dunn, A. G. (2024) · Journal of General Internal Medicine · doi:10.1007/s11606-023-08469-w · Cited by 124
“19, 20 21–24 This study sought to investigate the capacity for ChatGPT (GPT-3.”
BackgroundGuerra, G. A., Hofmann, H., Sobhani, S., Hofmann, G., Gomez, D., Soroudi, D., Hopkins, B. S., Dallas, J., Pangal, D. J., Cheok, S., et al. (2023) · World Neurosurgery · doi:10.1016/j.wneu.2023.08.042 · Cited by 123
Herrmann-Werner, A., Festl-Wietek, T., Holderried, F., Herschbach, L., Griewatz, J., Masters, K., Zipfel, S., & Mahling, M. (2024) · Journal of Medical Internet Research · doi:10.2196/52113 · Cited by 96
Mihalache, A., Huang, R. S., Popovic, M. M., & Muni, R. H. (2024) · Medical Teacher · doi:10.1080/0142159x.2023.2249588 · Cited by 111
Kayalı, B., Yavuz, M., Balat, Ş., & Çalışan, M. (2023) · Australasian Journal of Educational Technology · doi:10.14742/ajet.8915 · Cited by 75
Automated Essay Scoring and Revising Based on Open-Source Large Language Models
Song, Y., Zhu, Q., Wang, H., & Zheng, Q. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3396873 · Cited by 121
Opportunities, Challenges, and Strategies for Using ChatGPT in Higher Education: A Literature Review
Sok, S., & Heng, K. (2024) · Journal of Digital Educational Technology · doi:10.30935/jdet/14027 · Cited by 120
“…al., 2023), biased responses or outputs (Halaweh, 2023; Kitamura, 2023; Lund et al., 2023), a lack of creativity and originality of AI-generated responses (Chakravarti, 2023; Dwivedi et al., 2023), and the restricted use of training data for ChatGPT (Gilson et al., 2023; Talan & Kalinkara, 2023).”
MethodOn the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
ChatGPT's influence on customer experience in digital marketing: Investigating the moderating roles
Abdelkader, O. A. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e18770 · Cited by 71
Sun, H., Zhang, K., Lan, W., Gu, Q., Jiang, G., Yang, X., Qin, W., & Han, D. (2023) · Journal of Medical Internet Research · doi:10.2196/51300 · Cited by 101
Taloni, A., Borselli, M., Scarsi, V., Rossi, C., Coco, G., Scorcia, V., & Giannaccare, G. (2023) · Scientific Reports · doi:10.1038/s41598-023-45837-2 · Cited by 104
ChatGPT has Aced the Test of Understanding in College Economics: Now What?
Geerling, W., Mateer, G. D., Wooten, J., & Damodaran, N. (2023) · The American Economist · doi:10.1177/05694345231169654 · Cited by 119
“Several studies have demonstrated ChatGPT ’ s ability to pass standardized tests in various fi elds such as mathematics (Leswing, 2023), medicine (Gilson et al., 2023), law (Choi et al., 2023), and physics (West, 2023).”
BackgroundDifferentiate ChatGPT-generated and Human-written Medical Texts
Liao, W., Liu, Z., Dai, H., Xu, S., Wu, Z., Zhang, Y., Huang, X., Zhu, D., Cai, H., Liu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.11567 · Cited by 119
Metadata
- Authors
- Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D.
- Affiliations
- Yale University (US)
- University College Dublin (IE)
- Corresponding author
- Chartash, David
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-02-08
- Volume
- 9
- Pages
- e45312
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC9947764
- OpenAlex
- W4319460874
- Semantic Scholar
- cd4d112f3f9120d0715f22a9de2ce4720822368c
- Cited by
- 2,078
- References
- 21
- Citations by year
- 2026: 351 · 2025: 550 · 2024: 699 · 2023: 474
- FWCI
- 82.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- National Heart, Lung, and Blood Institute (T35HL007649)
- National Institute of Diabetes and Digestive and Kidney Diseases (T35DK104689)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE Step 1, USMLE Step 2, medical licensing examinations, question difficulty, GPT-3, medical education, knowledge assessment, AMBOSS, National Board of Medical Examiners, artificial intelligence in medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.