How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. Objective: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. Methods: We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Results: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. Conclusions: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT’s capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Template of question posed to each large language model (LLM), including both AMBOSS Attending Tip and the response from Chat Generative Pre-trained Transformer (ChatGPT). The correct answer to this question is “E. Zidovudine (AZT).” In the case of GPT-3, prompt engineering was necessary, with: "Please answer this multiple choice question:" + question as described previously + "Correct answer is." As GPT-3 is inherently a nondialogic model, this was necessary to reduce model hallucinations and force a clear answer [17].](https://figures.cochat.ai/paper-figures/583a8f9972504138a5ac67eaf322285a/how-does-chatgpt-perform-on-the-united-states-medical-licensing-examination-figure-1.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, like GPT-4, have shown medical knowledge despite lacking medicine-specific training [10, 29].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, several valid concerns were raised, including the possibility of generating inaccurate content, along with ethical issues, including the risk of bias, plagiarism, and copyright issues [9,18,40,41].”
From Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study (2023) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, analyses of ChatGPT show structured, non-random response patterns consistent with trait-like reasoning abilities18.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the majority of clinical reasoning evaluations on LLMs have been conducted using standardized exams (the United States Medical Licensing Examination) or digital case reports [14,19].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [14]
Experimental — accuracy not yet measured
39 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Springer, L., Stanne, M. E., & Donovan, S. S. (1999) · Review of Educational Research · doi:10.3102/00346543069001021 · Cited by 1,943 · cited 1×
“Small group education has been shown to be a highly efficacious method of teaching [23,24].”
MethodThe use of reflection in medical education: AMEE Guide No. 44
Sandars, J. (2009) · Medical Teacher · doi:10.1080/01421590903050374 · Cited by 1,085
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
Research works printed but not linked: 2
- 10.48550/arxiv.2108.01764 · DOI
- Wang, DC, 2020, Yale J Biol Med, 93, 441
Cited by
Items 901–1,000 of 1,047
ChatGPT is Good but Bing Chat is Better for Vietnamese StudentsPreprint
Dao, X.-Q., & Le, N.-B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.08272 · Cited by 23
Gill, G. S., Tsai, J., Moxam, J., Sanghvi, H. A., & Gupta, S. (2024) · Cureus · doi:10.7759/cureus.69612 · Cited by 22
The Role of Artificial Intelligence (ChatGPT-4o) in Supporting Tumor Board Decisions
Karabuğa, B., Karaçin, C., Büyükkör, M., Bayram, D., Aydemir, E., Kaya, O. B., Yılmaz, M. E., Çamöz, E. S., & Ergün, Y. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14103535 · Cited by 22
Artificial intelligence in respiratory care: perspectives on critical opportunities and challenges
Drummond, D., Adejumo, I., Hansen, K., Poberezhets, V., Slabaugh, G., & Hui, C. Y. (2024) · Breathe · doi:10.1183/20734735.0189-2023 · Cited by 22
Chen, R.-J., Wu, M.-S., Tsai, L.-W., Chang, S.-S., Hsiao, S. S., & Lo, Y. S. (2026) · Journal of Medical Internet Research · doi:10.2196/81604 · Cited by 23
The role of artificial intelligence in headache medicine: Potential and peril
Cohen, F. (2023) · Headache The Journal of Head and Face Pain · doi:10.1111/head.14495 · Cited by 22
Jiao, Z., Du, X., Liu, Z., Liu, L., Sun, Z., Shi, G., & Liu, R. (2023) · Sustainability · doi:10.3390/su15129680 · Cited by 22
Can ChatGPT pass the "Iranian Endodontics Specialist Board" exam?
Farajollahi, M., & Modaberi, A. (2023) · Iranian endodontic Journal/Iranian endodontic journal. · doi:10.22037/iej.v18i3.42154 · Cited by 22
Moskatel, L. S., & Zhang, N. (2023) · Frontiers in Neurology · doi:10.3389/fneur.2023.1225223 · Cited by 22
Barclay, K. S., You, J. Y., Coleman, M. J., Mathews, P. M., Ray, V., Riaz, K. M., Rojas, J. O. D., Wang, A. S., Watson, S. H., Koo, E. H., et al. (2023) · Cornea · doi:10.1097/ico.0000000000003439 · Cited by 22
Gritti, M. N., AlTurki, H., Farid, P., & Morgan, C. T. (2024) · Pediatric Cardiology · doi:10.1007/s00246-023-03385-6 · Cited by 22
Giunti, G., & Doherty, C. P. (2024) · JMIR Medical Education · doi:10.2196/48949 · Cited by 22
Pereyra, L., Schlottmann, F., Steinberg, L., & Lasa, J. (2024) · Journal of Clinical Gastroenterology · doi:10.1097/mcg.0000000000001979 · Cited by 22
Halawani, A., Mitchell, A., Saffarzadeh, M., Wong, V. K. F., Chew, B. H., & Forbes, C. M. (2024) · Urology · doi:10.1016/j.urology.2023.11.042 · Cited by 22
Pham, C., Govender, R., Tehami, S., Chavez, S., Adepoju, O. E., & Liaw, W. R. (2024) · Journal of Medical Internet Research · doi:10.2196/55037 · Cited by 22
Analysis of ChatGPT Performance in Computer Engineering Exams
Rodríguez-Echeverría, R., Gutiérrez, J. D., Conejero, J. M., & Prieto, Á. E. (2024) · IEEE Revista Iberoamericana de Tecnologias del Aprendizaje · doi:10.1109/rita.2024.3381842 · Cited by 22
Kröplin, J., Maier, L., Lenz, J., & Romeike, B. F. M. (2024) · JMIR Medical Education · doi:10.2196/51389 · Cited by 22
Using large language models for safety-related table summarization in clinical study reports
Landman, R., Healey, S. P., Loprinzo, V., Kochendoerfer, U., Winnier, A. R., Henstock, P. V., Lin, W., Chen, A., Rajendran, A., Penshanwar, S., et al. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae043 · Cited by 22
Siebielec, J., Ordak, M., Oskroba, A., Dworakowska, A. M., & Bujalska‐Zadrożny, M. (2024) · Healthcare · doi:10.3390/healthcare12161637 · Cited by 22
Lotto, C., Sheppard, S. C., Anschuetz, W., Stricker, D., Molinari, G., Huwendiek, S., & Anschuetz, L. (2024) · OTO Open · doi:10.1002/oto2.70018 · Cited by 22
Huang, Y., Wang, W., Zhou, J., Zhang, L., Lin, J., Liu, H., Hu, X., Zhou, Z., & Dong, W. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104152 · Cited by 22
From Transformers to ChatGPT: An Analysis of Large Language Models Research
Cotfas, L., Sandu, A., Delcea, C., Diaconu, P., Frăsineanu, C., & Stănescu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3600739 · Cited by 22
Barandoni, S., Cascone, L., Marrale, E., Puccio, S., & Chiarello, F. (2026) · Computers in Industry · doi:10.1016/j.compind.2026.104448 · Cited by 22
Arbel, Y., Gimmon, Y., & Shmueli, L. (2025) · Physical Therapy · doi:10.1093/ptj/pzaf010 · Cited by 21
Evaluating Chatbot responses to patient questions in the field of glaucoma
Tan, D. N. H., Tham, Y. C., Koh, V., Loon, S. C., Aquino, M. C., Lun, K., Cheng, C., Ngiam, K. Y., & Tan, M. C. J. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1359073 · Cited by 21
Google Gemini’s Performance in Endodontics: A Study on Answer Precision and Reliability
García, V. D., Freire, Y., Tortosa, M., Tejedor, B., Estévez, R., & Suárez, A. (2024) · Applied Sciences · doi:10.3390/app14156390 · Cited by 21
Avsar, D. B., & Ertan, A. A. (2024) · Turkiye Klinikleri Journal of Dental Sciences · doi:10.5336/dentalsci.2024-104610 · Cited by 21
Kottlors, J., Hahnfeldt, R., Goertz, L., Iuga, A.-I., Fervers, P., Bremm, J., Zopfs, D., Laukamp, K. R., Onur, O. A., Lennartz, S., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/48328 · Cited by 21
Semantic Clinical Artificial Intelligence vs Native Large Language Model Performance on the USMLE
Elkin, P. L., Mehta, G., LeHouillier, F., Resnick, M., Mullin, S., Tomlin, C., Resendez, S. D., Liu, J., Nebeker, J. R., & Brown, S. H. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.6359 · Cited by 21
Exploring the occupational biases and stereotypes of Chinese large language models
Jiang, L., Zhu, G., Sun, J., Cao, J., & Wu, J. X. (2025) · Scientific Reports · doi:10.1038/s41598-025-03893-w · Cited by 21
ChatGPT as a Learning Tool for Medical Students: Results From a Randomized Controlled Trial
Kalam, K. A., Masoud, F. D., Muntaser, A., Ranga, R., Geng, X., & Goyal, M. K. (2025) · Cureus · doi:10.7759/cureus.85767 · Cited by 21
Elzayyat, M., Mohammad, J. N., & Zaqout, S. (2025) · Medical Education Online · doi:10.1080/10872981.2025.2554678 · Cited by 22
How to Use Generative AI in Educational Research
Roe, J. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009675338 · Cited by 22
Lawan, A. A., Muhammad, B. R., Tahir, A. M., Yarima, K. I., Zakari, A., Abdullahi, A. H., Hussaini, A., Kademi, H. I., Danlami, A. A., Sani, M. A., et al. (2023) · Education Journal · doi:10.11648/j.edu.20231204.14 · Cited by 21
Talking Abortion (Mis)information with ChatGPT on TikTok
Sharevski, F., Loop, J. V., Jachim, P., Devine, A., & Pieroni, E. (2023) · IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) · doi:10.1109/eurospw59978.2023.00071 · Cited by 21
The Hazards of Using ChatGPT: A Call to Action for Medical Education Researchers
Liaw, W. R., Chavez, S., Pham, C., Tehami, S., & Govender, R. (2023) · PRiMER · doi:10.22454/primer.2023.295710 · Cited by 21
Large language models and the future of rheumatology: assessing impact and emerging opportunities
Mannstadt, I., & Mehta, B. Y. (2023) · Current Opinion in Rheumatology · doi:10.1097/bor.0000000000000981 · Cited by 21
Investigating AI languages’ ability to solve undergraduate finance problems
Yang, C., & Stivers, A. (2023) · Journal of Education for Business · doi:10.1080/08832323.2023.2253963 · Cited by 21
Ghosh, A., Jindal, N. M., Gupta, V. K., Bansal, E., Bajwa, N. K., & Sett, A. (2023) · Cureus · doi:10.7759/cureus.47329 · Cited by 21
Sood, A., Mansoor, N. M., Memmi, C., Lynch, M., & Lynch, J. (2024) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-024-03071-9 · Cited by 21
Performance of GPT-4 Vision on kidney pathology exam questions
Miao, J., Thongprayoon, C., Cheungpasitporn, W., & Cornell, L. D. (2024) · American Journal of Clinical Pathology · doi:10.1093/ajcp/aqae030 · Cited by 21
ChatGPT and Medicine: Together We Embrace the AI Renaissance
Hacking, S. M. (2024) · JMIR Bioinformatics and Biotechnology · doi:10.2196/52700 · Cited by 21
Polat, E., Polat, Y. B., Şentürk, E., Doğan, R., Yenigün, A., Tuğrul, S., Eren, S. B., Aksoy, F., & Özturan, O. (2024) · International Journal of Pediatric Otorhinolaryngology · doi:10.1016/j.ijporl.2024.111998 · Cited by 21
Schoch, J., Schmelz, H. U., Strauch, A., Borgmann, H., & Nestler, T. (2024) · World Journal of Urology · doi:10.1007/s00345-024-05137-4 · Cited by 21
Abi‐Rafeh, J., Bassiri-Tehrani, B., Kazan, R., Furnas, H. J., Hammond, D. C., Adams, W. P., & Nahai, F. (2024) · Aesthetic Surgery Journal Open Forum · doi:10.1093/asjof/ojae062 · Cited by 21
Guerra, G. A., Hofmann, H. L., Le, J. L., Wong, A. M., Fathi, A., Mayfield, C. K., Petrigliano, F. A., & Liu, J. N. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.08.023 · Cited by 21
Harnessing the power of ChatGPT to promote Construction Hazard Prevention through Design (CHPtD)
Uddin, S. M. J., Albert, A., & Tamanna, M. (2024) · Engineering Construction & Architectural Management · doi:10.1108/ecam-03-2024-0314 · Cited by 21
Miyazaki, Y., Hata, M., Omori, H., Hirashima, A., Nakagawa, Y., Eto, M., Takahashi, S., & Ikeda, M. (2024) · JMIR Medical Education · doi:10.2196/63129 · Cited by 21
Penny, P., Bane, R., & Riddle, V. (2024) · Cureus · doi:10.7759/cureus.76309 · Cited by 21
Liu, Y., Li, X., Luo, Y., Du, J., Zhang, Y., Lv, T., Yin, H., Tang, X., & Liu, H. (2025) · Engineering · doi:10.1016/j.eng.2025.02.010 · Cited by 21
Yue, Y., Ng, S. I., & Basha, N. K. (2024) · Pakistan Journal of Commerce and Social Sciences · doi:10.64534/commer.2024.040 · Cited by 21
ChatGPT and GPTZero in Research and Social Media: A Sentiment-and Topic-based AnalysisPreprint
Heumann, M., Kraschewski, T., & Breitner, M. H. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4467646 · Cited by 21
Sallam, M., Al‐Farajat, A., & Egger, J. (2024) · Jordan Medical Journal · doi:10.35516/jmj.v58i1.2285 · Cited by 21
FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic ExaminationPreprint
Zhou, P., Min, W., Fu, C., Jin, Y., Huang, M., Li, X., Mei, S., & Jiang, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.10261 · Cited by 21
Integrating AI Technology Into Language Teacher Education: Challenges, Potentials, and Assumptions
Case, R. E., Liu, L., & Mintz, J. (2025) · Computers in the Schools · doi:10.1080/07380569.2025.2458950 · Cited by 22
Strasser, L. M., Anschuetz, W., Dennstädt, F., & Hastings, J. (2026) · JMIR Medical Education · doi:10.2196/81399 · Cited by 20
Jung, J. Y., Tyack, L., & von Davier, M. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100375 · Cited by 20
Lafourcade, C., Kérourédan, O., Ballester, B., & Richert, R. (2025) · Journal of Dentistry · doi:10.1016/j.jdent.2025.105764 · Cited by 20
Students Collaboratively Prompting ChatGPT
Perifanou, M. A., & Economides, A. A. (2025) · Computers · doi:10.3390/computers14050156 · Cited by 20
Cao, J., Wu, Z., Wu, Z., Hu, Y., Dai, S., & Liu, Y. (2026) · Digital Health · doi:10.1177/20552076261435836 · Cited by 20
Zou, J., Qin, Y., & Rhee, M. (2024) · Innovation and technology studies. · doi:10.61784/its4026 · Cited by 27
Chat generative pre-trained transformer (ChatGPT): potential implications for rheumatology practice
Nune, A., Iyengar, K. P., Manzo, C., Barman, B., & Botchu, R. (2023) · Rheumatology International · doi:10.1007/s00296-023-05340-3 · Cited by 20
Is the laboratory report dead? AI and ChatGPT
Wang, J. T. H. (2023) · Microbiology Australia · doi:10.1071/ma23042 · Cited by 20
Performance of ChatGPT in Otolaryngology knowledge assessment
Revercomb, L., Patel, A. M., Choudhry, H. S., & Filimonov, A. (2023) · American Journal of Otolaryngology · doi:10.1016/j.amjoto.2023.104082 · Cited by 20
Javidan, A. P., Feridooni, T., Gordon, L., & Crawford, S. A. (2023) · JVS-Vascular Insights · doi:10.1016/j.jvsvi.2023.100049 · Cited by 20
Wong, L. H., Park, H., & Looi, C. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12962 · Cited by 20
Touma, N. J., Caterini, J. E., & Liblk, K. (2024) · Canadian Urological Association Journal · doi:10.5489/cuaj.8800 · Cited by 20
Cautious Optimism: The Influence of Generative AI Tools in Software Development Projects
Mbizo, T., Oosterwyk, G., Tsibolane, P., & Kautondokwa, P. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-64881-6_21 · Cited by 20
Varady, N. H., Lu, A. Z., Mazzucco, M. R., Dines, J. S., Altchek, D. W., Williams, R. J., & Kunze, K. N. (2024) · Orthopaedic Journal of Sports Medicine · doi:10.1177/23259671241257516 · Cited by 20
Ehlert, A., Ehlert, B., Cao, B., & Morbitzer, K. A. (2024) · American Journal of Pharmaceutical Education · doi:10.1016/j.ajpe.2024.101294 · Cited by 20
Based on Medicine, The Now and Future of Large Language Models
Su, Z., Tang, G., Huang, R., Yang, Q. W., Zhang, Z., & Dai, X. (2024) · Cellular and Molecular Bioengineering · doi:10.1007/s12195-024-00820-3 · Cited by 20
Huang, R. S., Benour, A., Kemppainen, J., & Leung, F. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06115-5 · Cited by 20
Roos, J., Martin, R., & Kaczmarczyk, R. (2024) · JMIR Formative Research · doi:10.2196/57592 · Cited by 20
Implementation and evaluation of an optimized surgical clerkship teaching model utilizing ChatGPT
Huang, Y., Xu, B., Wang, X., Luo, Y.-C., Teng, M.-M., & Weng, X. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06575-9 · Cited by 20
Comparison of AI applications and anesthesiologist’s anesthesia method choices
Çelik, E., Turgut, M. A., Aydoğan, M., Kılınç, M., Toktaş, İ., & Akelma, H. (2025) · BMC Anesthesiology · doi:10.1186/s12871-024-02882-2 · Cited by 20
HERNÁNDEZ-FLORES, L. A., López‐Martínez, J. B., la Rosa, J. J. R.-D., Aillaud‐De‐Uriarte, D., Contreras‐Garduño, S., & Cortés‐González, R. (2025) · Journal of Surgical Oncology · doi:10.1002/jso.28121 · Cited by 20
Ethics of AI: Toward a “Design for Values” Approach
Buijsman, S., Klenk, M., & van den Hoven, J. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009367783.005 · Cited by 20
Evaluating the Use of Artificial Intelligence as a Study Tool for Preclinical Medical School Exams
Sakelaris, P. G., Novotny, K. V., Borvick, M. S., Lagasca, G., & Simanton, E. G. (2025) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205251320150 · Cited by 20
Integration of Artificial Intelligence and Machine Learning in Education: A Systematic Review
Reina-Parrado, M., Graván, P. R., & Hervás-Gómez, C. (2025) · International Journal of Educational Methodology · doi:10.12973/ijem.11.2.203 · Cited by 20
Ramlogan, S., Raman, V., & Ramlogan, S. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07195-7 · Cited by 20
Exploring the impact of ChatGPT on Wikipedia engagement
Reeves, N., Yin, W., & Simperl, E. P. B. (2025) · Collective Intelligence · doi:10.1177/26339137251372599 · Cited by 20
FROM TEXT TO DIAGNOSE: CHATGPT’S EFFICACY IN MEDICAL DECISION-MAKING
Mykhalko, Y. O., Kish, P. P., Rubtsova, Y. I., Kutsin, O. O., & Koval, V. (2023) · Wiadomości Lekarskie · doi:10.36740/wlek202311101 · Cited by 20
Aubignat, M., & Diab, E. (2023) · Revue Neurologique · doi:10.1016/j.neurol.2023.03.004 · Cited by 20
ChatGPT in Medicine; a Disruptive Innovation or Just One Step Forward?
Parsa, A., & Ebrahimzadeh, M. A. (2023) · PubMed Central · doi:10.22038/abjs.2023.22042 · Cited by 20
Leveraging large language models in dermatology
Matin, R., Linos, E., & Rajan, N. (2023) · British Journal of Dermatology · doi:10.1093/bjd/ljad230 · Cited by 20
Can generative artificial intelligence pass the orthopaedic board examination?
Isleem, U. N., Zaidat, B., Ren, R., Geng, E. A., Burapachaisri, A., Tang, J. E., Kim, J. S., & Cho, S. K. (2024) · Journal of Orthopaedics · doi:10.1016/j.jor.2023.10.026 · Cited by 19
Dastani, M., Mardaneh, J., & Rostamian, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-03074-9 · Cited by 19
Shaping the future of education: a cluster analysis of generative AI’s transformative impact
Issa, H.-E. A., & Maheshwari, G. (2025) · Interactive Technology and Smart Education · doi:10.1108/itse-01-2025-0013 · Cited by 19
Khurana, D., Rashed, A., Tambil, Z., Firek, A. F., Parasher, P., & Khurana, S. (2026) · Journal of Dental Education · doi:10.1002/jdd.70218 · Cited by 19
ChatGPT-Enabled daVinci Surgical Robot Prototype: Advancements and Limitations
Pandya, A. D. (2023) · Robotics · doi:10.3390/robotics12040097 · Cited by 19
Anagnostopoulos, C. (2023) · F1000Research · doi:10.12688/f1000research.141958.1 · Cited by 19
Surapaneni, K. M. (2023) · JMIR Medical Education · doi:10.2196/47191 · Cited by 19
How Can the Clinical Aptitude of AI Assistants Be Assayed?
Thirunavukarasu, A. J. (2023) · Journal of Medical Internet Research · doi:10.2196/51603 · Cited by 19
Chatbot Performance in Defining and Differentiating Palliative Care, Supportive Care, Hospice Care
Kim, M. J., Admane, S., Chang, Y. K., Shih, K. K., Reddy, A. S., Tang, M. J., de la Cruz, M., Taylor, T. P., Bruera, E. D., & Hui, D. (2024) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2024.01.008 · Cited by 19
A Cybernetic Perspective on Generative AI in Education: From Transmission to Coordination.
Griffiths, D., Frías-Martínez, E., Tlili, A., & Burgos, D. (2024) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2024.02.008 · Cited by 19
Evaluation of ChatGPT in Predicting 6-Month Outcomes After Traumatic Brain Injury*
Gakuba, C., Barbey, C. L., Sar, A., Bonnet, G., Cerasuolo, D., Giabicani, M., & Moyer, J.-D. (2024) · Critical Care Medicine · doi:10.1097/ccm.0000000000006236 · Cited by 19
Takagi, S., Koda, M., & Watari, T. (2024) · JMIR Medical Education · doi:10.2196/54283 · Cited by 19
DiDonna, N., Shetty, P. N., Khan, K., & Damitz, L. (2024) · Plastic & Reconstructive Surgery Global Open · doi:10.1097/gox.0000000000005929 · Cited by 19
Hepatic encephalopathy post-TIPS: Current status and prospects in predictive assessment
Xu, X., Yang, Y. S., Tan, X., Zhang, Z., Wang, B., Yang, X., Weng, C., Yu, R., Zhao, Q., & Quan, S. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.07.008 · Cited by 19
Anki Tagger: A Generative AI Tool for Aligning Third-Party Resources to Preclinical Curriculum
Pendergrast, T., & Chalmers, Z. (2023) · JMIR Medical Education · doi:10.2196/48780 · Cited by 18
Metadata
- Authors
- Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D.
- Affiliations
- Yale University (US)
- University College Dublin (IE)
- Corresponding author
- Chartash, David
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-02-08
- Volume
- 9
- Pages
- e45312
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC9947764
- OpenAlex
- W4319460874
- Semantic Scholar
- cd4d112f3f9120d0715f22a9de2ce4720822368c
- Cited by
- 2,078
- References
- 21
- Citations by year
- 2026: 351 · 2025: 550 · 2024: 699 · 2023: 474
- FWCI
- 82.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- National Heart, Lung, and Blood Institute (T35HL007649)
- National Institute of Diabetes and Digestive and Kidney Diseases (T35DK104689)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE Step 1, USMLE Step 2, medical licensing examinations, question difficulty, GPT-3, medical education, knowledge assessment, AMBOSS, National Board of Medical Examiners, artificial intelligence in medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.