How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. Objective: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. Methods: We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Results: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. Conclusions: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT’s capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Template of question posed to each large language model (LLM), including both AMBOSS Attending Tip and the response from Chat Generative Pre-trained Transformer (ChatGPT). The correct answer to this question is “E. Zidovudine (AZT).” In the case of GPT-3, prompt engineering was necessary, with: "Please answer this multiple choice question:" + question as described previously + "Correct answer is." As GPT-3 is inherently a nondialogic model, this was necessary to reduce model hallucinations and force a clear answer [17].](https://figures.cochat.ai/paper-figures/583a8f9972504138a5ac67eaf322285a/how-does-chatgpt-perform-on-the-united-states-medical-licensing-examination-figure-1.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, like GPT-4, have shown medical knowledge despite lacking medicine-specific training [10, 29].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, several valid concerns were raised, including the possibility of generating inaccurate content, along with ethical issues, including the risk of bias, plagiarism, and copyright issues [9,18,40,41].”
From Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study (2023) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, analyses of ChatGPT show structured, non-random response patterns consistent with trait-like reasoning abilities18.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the majority of clinical reasoning evaluations on LLMs have been conducted using standardized exams (the United States Medical Licensing Examination) or digital case reports [14,19].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [14]
Experimental — accuracy not yet measured
39 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Springer, L., Stanne, M. E., & Donovan, S. S. (1999) · Review of Educational Research · doi:10.3102/00346543069001021 · Cited by 1,943 · cited 1×
“Small group education has been shown to be a highly efficacious method of teaching [23,24].”
MethodThe use of reflection in medical education: AMEE Guide No. 44
Sandars, J. (2009) · Medical Teacher · doi:10.1080/01421590903050374 · Cited by 1,085
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
Research works printed but not linked: 2
- 10.48550/arxiv.2108.01764 · DOI
- Wang, DC, 2020, Yale J Biol Med, 93, 441
Cited by
Items 401–500 of 1,047
Jiang, H., Xia, S., Yang, Y., Xu, J., Hua, Q., Mei, Z., Hou, Y., Wei, M., Lai, L., Li, N., et al. (2024) · European Journal of Radiology · doi:10.1016/j.ejrad.2024.111458 · Cited by 62
Impact of Large Language Models on Medical Education and Teaching Adaptations
Zhui, L., Yhap, N., Liping, L., Wang, Z., Zhonghao, X., Xiaoshu, Y., Hong, C., Liu, X., & Wei, R. (2024) · JMIR Medical Informatics · doi:10.2196/55933 · Cited by 62
Suh, P. S., Shim, W. H., Suh, C. H., Heo, H., Park, K. J., Kim, P. H., Choi, S. J., Ahn, Y., Park, S., Park, H. Y., et al. (2024) · Radiology · doi:10.1148/radiol.241668 · Cited by 62
Bitew, S. K., Deleu, J., Develder, C., & Demeester, T. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-74627-7_4 · Cited by 62
Claude, ChatGPT, Copilot, and Gemini performance versus students in different topics of neuroscience
Mavrych, V., Yaqinuddin, A., & Bolgova, O. (2025) · AJP Advances in Physiology Education · doi:10.1152/advan.00093.2024 · Cited by 62
Yang, Z., Yao, Z., Tasmin, M., Vashisht, P., Jang, W. S., Ouyang, F., Wang, B., Berlowitz, D., & Yu, H. (2023) · medRxiv · doi:10.1101/2023.10.26.23297629 · Cited by 36
Liu, H., Azam, M., Bin Naeem, S., & Faiola, A. (2023) · Health Information & Libraries Journal · doi:10.1111/hir.12509 · Cited by 47
Legal and ethical considerations regarding the use of ChatGPT in education
Panagopoulou, F., Parpoula, C., & Karpouzis, K. (2023) · doi:10.31235/osf.io/c5hf3 · Cited by 35
Durmuş Sarıkahya, S., Özbay, Ö., Torpuş, K., Usta, G., & Çınar Özbay, S. (2025) · Nurse Education Today · doi:10.1016/j.nedt.2025.106755 · Cited by 60
Carrasco, J. P., García, E., Martínez, D. A. S., Estrella-Porter, P., la Puente, L. D., Navarro, J., & Cerame, Á. (2023) · Revista Española de Educación Médica · doi:10.6018/edumed.556511 · Cited by 61
Assessment of ChatGPT’s performance on neurology written board examination questions
Chen, T. C., Multala, E., Kearns, P., Delashaw, J., Dumont, A., Maraganore, D., & Wang, A. (2023) · BMJ Neurology Open · doi:10.1136/bmjno-2023-000530 · Cited by 51
Nghi, T. T., & Tran, H. P. (2023) · Proceedings of the AsiaCALL International Conference · doi:10.54855/paic.2341 · Cited by 60
ChatGPT vs. web search for patient questions: what does ChatGPT do better?
Shen, S. A., Perez-Heydrich, C. A., Xie, D. X., & Nellis, J. C. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-024-08524-0 · Cited by 52
ChatGPT and other AI Users: Innovative and Creative Utilitarian Value and Mindset Shift
Duhaylungsod, A. V., & Chavez, J. V. (2023) · Journal of Namibian Studies : History Politics Culture · doi:10.59670/jns.v33i.2791 · Cited by 59
Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study
Shoufan, A. (2023) · ACM Transactions on Computing Education · doi:10.1145/3628162 · Cited by 59
Malik, A., Khan, M. L., & Hussain, K. (2023) · doi:10.2139/ssrn.4413516 · Cited by 59
Leite, B. S. (2023) · Química Nova · doi:10.21577/0100-4042.20230059 · Cited by 59
Hofmann, H. L., Guerra, G. A., Le, J. L., Wong, A. M., Hofmann, G., Mayfield, C. K., Petrigliano, F. A., & Liu, J. N. (2023) · Orthopedics · doi:10.3928/01477447-20230922-05 · Cited by 59
Potential Applications and Impact of ChatGPT in Radiology
Bajaj, S., Gandhi, D., & Nayar, D. (2023) · Academic Radiology · doi:10.1016/j.acra.2023.08.039 · Cited by 59
Rizzo, M. G., Cai, N., & Constantinescu, D. S. (2023) · Journal of Orthopaedics · doi:10.1016/j.jor.2023.11.056 · Cited by 59
An integrative decision-making framework to guide policies on regulating ChatGPT usage
Bukar, U. A., Sayeed, M. S., Abdul Razak, S. F., Yogarayan, S., & Ahmed Amodu, O. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1845 · Cited by 43
Pasupuleti, R. S., & Thiyyagura, D. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12573-7 · Cited by 60
Unveiling destination evangelism through generative AI tools
Soliman, M. S. A., & Balushi, M. K. A. (2023) · ROBONOMICS The Journal of the Automated Economy · doi:10.67723/rj.v4i.54 · Cited by 59
Performance of ChatGPT on free-response, clinical reasoning exams
Strong, E., DiGiammarino, A., Weng, Y., Basaviah, P., Hosamani, P., Kumar, A., Nevins, A., Kugler, J., Hom, J., & Chen, J. H. (2023) · medRxiv · doi:10.1101/2023.03.24.23287731 · Cited by 48
Hadi, M. U., Tashi, Q. A., Qureshi, R., Shah, A., Muneer, A., Irfan, M. I., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2025) · doi:10.36227/techrxiv.23589741.v8 · Cited by 59
ChatGPT in Medical Education and Research: A Boon or a Bane?
Jeyaraman, M., K, S. P., Jeyaraman, N., Nallakumarasamy, A., Yadav, S., & Bondili, S. K. (2023) · Cureus · doi:10.7759/cureus.44316 · Cited by 58
“Technology breakthroughs are advancing medical education, and AI like ChatGPT can serve several beneficial functions, which are as follows [12]:”
BackgroundLin, Y., Luo, Z., Ye, Z., Zhong, N., Zhao, L., Zhang, L., Li, X., Chen, Z., & Chen, Y. (2025) · JMIR Medical Education · doi:10.2196/71125 · Cited by 58
“Across all models, performance tended to decline with increased difficulty, especially for higher-level questions that required multistep reasoning, where accuracy dropped markedly [39-45].”
Shaw, K., Henning, M. A., & Webster, C. S. (2025) · Medical Science Educator · doi:10.1007/s40670-025-02373-0 · Cited by 58
Baskara, F. R. (2023) · Prosiding Seminar Nasional Fakultas Tarbiyah dan Ilmu Keguruan IAIM Sinjai · doi:10.47435/sentikjar.v2i0.1825 · Cited by 28
Bushuven, S., Bentele, M., Bentele, S., Gerber, B., Bansbach, J., Ganter, J., Trifunovic-Koenig, M., & Ranisch, R. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-02019-x · Cited by 47
Yang, J., Ardavanis, K. S., Slack, K. E., Fernando, N. D., Valle, C. J. D., & Hernandez, N. M. (2024) · The Journal of Arthroplasty · doi:10.1016/j.arth.2024.01.029 · Cited by 59
Understanding the perception of design students towards ChatGPT
Chellappa, V., & Luximon, Y. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100281 · Cited by 58
Zada, T., Tam, N., Barnard, F., Sittert, M. V., Bhat, V., & Rambhatla, S. (2025) · JMIR Formative Research · doi:10.2196/66207 · Cited by 58
Urzúa, C. A. C., Ranjan, R., Saavedra, E. E. M., Quintana, M. G. B., Lepe-Martínez, N. F., & Philominraj, A. S. (2025) · Education Sciences · doi:10.3390/educsci15101396 · Cited by 59
Malik, A., Khan, M. L., Hussain, K., Qadir, J., & Tarhini, A. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2024.2409407 · Cited by 57
Kao, H.-J., Chien, T.-W., Wang, W.-C., Chou, W., & Chow, J. C. (2023) · Medicine · doi:10.1097/md.0000000000034068 · Cited by 57
“[12] ChatGPT has achieved accuracy on various datasets, including answering yes-or-no questions from PubMed-available abstracts and questions on the US Medical Licensing Examination.”
BackgroundMethodInfluentialBrameier, D. T., Alnasser, A. A., Carnino, J. M., Bhashyam, A. R., von Keudell, A. G., & Weaver, M. J. (2023) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.23.00473 · Cited by 46
Chakraborty, C., Bhattacharya, M., Pal, S., Chatterjee, S., Das, A., & Lee, S. (2025) · Journal of Advanced Research · doi:10.1016/j.jare.2025.02.011 · Cited by 59
Navigating Generative AI (ChatGPT) in Higher Education: Opportunities and Challenges
Wang, T. (2023) · Lecture Notes in Educational Technology · doi:10.1007/978-981-99-5961-7_28 · Cited by 56
Liu, J. Q. J., Hui, K. T. K., Al Zoubi, F., Zhou, Z. Z. X., Samartzis, D., Yu, C. C. H., Chang, J. R., & Wong, A. Y. L. (2024) · International Journal for Educational Integrity · doi:10.1007/s40979-024-00155-6 · Cited by 56
Huang, R. S., Lu, K. J. Q., Meaney, C., Kemppainen, J., Punnett, A., & Leung, F.-H. (2023) · JMIR Medical Education · doi:10.2196/50514 · Cited by 56
Al-Dujaili, Z., Omari, S., Pillai, J., & Al Faraj, A. (2023) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2023.08.012 · Cited by 56
Lower, K., Seth, I., Lim, B., & Seth, N. (2023) · Indian Journal of Orthopaedics · doi:10.1007/s43465-023-00967-7 · Cited by 56
Choi, W. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04832-x · Cited by 48
Cognitive Biases and Artificial Intelligence
Wang, J., & Redelmeier, D. A. (2024) · NEJM AI · doi:10.1056/aics2400639 · Cited by 56
Wu, J., Wang, Z., & Qin, Y. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02213-z · Cited by 56
Exploring Qualitative Research Using LLMsPreprint
Bano, M., Zowghi, D., & Whittle, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.13298 · Cited by 55
A systematic review of current AI techniques used in the context of the SDGs
Greif, L., Röckel, F., Kimmig, A., & Ovtcharova, J. (2025) · International Journal of Environmental Research · doi:10.1007/s41742-024-00668-5 · Cited by 55
Xie, Y., Seth, I., Hunter‐Smith, D. J., Rozen, W. M., & Seifman, M. A. (2024) · ANZ Journal of Surgery · doi:10.1111/ans.18666 · Cited by 55
Can ChatGPT improve communication in hospitals?
Santandreu-Calonge, D., Medina-Aguerrebere, P., Hultberg, P., & Shah, M.-A. (2023) · El Profesional de la información · doi:10.3145/epi.2023.mar.19 · Cited by 39
Abi-Rafeh, J., Xu, H. H., Kazan, R., Tevlin, R., & Furnas, H. (2024) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad260 · Cited by 49
Ittarat, M., Cheungpasitporn, W., & Chansangpetch, S. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13121679 · Cited by 55
Maitland, A., Fowkes, R., & Maitland, S. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-080558 · Cited by 47
How Good Is ChatGPT in Giving Advice on Your Visualization Design?
Kim, N. W., Ahn, Y., Myers, G., & Bach, B. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3745768 · Cited by 55
Yao, Z., Zhang, Z., Tang, C., Bian, X., Zhao, Y., Yang, Z., Wang, J., Zhou, H., Jang, W. S., Ouyang, F., et al. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.292 · Cited by 55
Llms for Malware Detection: Review, Framework Design, and Countermeasure ApproachesPreprint
Khan, M. A.-Z., Al-Karaki, J. N., & Omar, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4995252 · Cited by 56
Deebel, N. A., & Terlecki, R. (2023) · Urology · doi:10.1016/j.urology.2023.05.010 · Cited by 54
Watari, T., Takagi, S., Sakaguchi, K., Nishizaki, Y., Shimizu, T., Yamamoto, Y., & Tokuda, Y. (2023) · JMIR Medical Education · doi:10.2196/52202 · Cited by 54
Daher, M., Koa, J., Boufadel, P., Singh, J., Fares, M. Y., & Abboud, J. A. (2023) · JSES International · doi:10.1016/j.jseint.2023.07.018 · Cited by 49
Aljindan, F. K., Al Qurashi, A. A., Albalawi, I. A. S., Alanazi, A. M. M., Aljuhani, H. A. M., Falah Almutairi, F., Aldamigh, O. A., Halawani, I. R., & K. Zino Alarki, S. M. (2023) · Cureus · doi:10.7759/cureus.45043 · Cited by 47
Su, M.-C., Lin, L.-E., Lin, L.-H., & Chen, Y. (2024) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2024.104717 · Cited by 54
ChatGPT-enabled two-stage auctions for electric vehicle battery recycling
Feng, J., Ning, Y., Wang, Z., Guo, L., & Xu, S. X. (2024) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2024.103453 · Cited by 54
Jyothy, S. N., Kolil, V. K., Raman, R., & Achuthan, K. (2024) · Cogent Engineering · doi:10.1080/23311916.2024.2353494 · Cited by 54
Bhattacharya, M., Pal, S., Chatterjee, S., Lee, S., & Chakraborty, C. (2024) · Molecular Therapy — Nucleic Acids · doi:10.1016/j.omtn.2024.102255 · Cited by 54
Liu, Y., Park, Y., & Wang, H. (2024) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2024.104176 · Cited by 54
García-Alonso, E. M., León-Mejía, A. C., Sánchez-Cabrero, R., & Guzmán-Ordaz, R. (2024) · Behavioral Sciences · doi:10.3390/bs14070612 · Cited by 53
Szabó, A. (2023) · Baltic Journal of Sport and Health Sciences · doi:10.33607/bjshs.v1i128.1341 · Cited by 53
Detecting the corruption of online questionnaires by artificial intelligence
Lebrun, B., Temtsin, S., Vonasch, A. J., & Bartneck, C. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1277635 · Cited by 54
Rosenberg, G. S., Magnéli, M., Barle, N., Kontakis, M. G., Müller, A. M., Wittauer, M., Gordon, M., & Brodén, C. (2024) · Acta Orthopaedica · doi:10.2340/17453674.2024.40182 · Cited by 53
Luo, D., Liu, M., Yu, R., Liu, Y., Jiang, W., Fan, Q., Kuang, N., Gao, Q., Yin, T., & Zheng, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-98949-2 · Cited by 53
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4Preprint
Kalyan, K. S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.12321 · Cited by 55
AI in imaging: the regulatory landscape
Hill, D. L. G. (2024) · British Journal of Radiology · doi:10.1093/bjr/tqae002 · Cited by 52
Analyzing evaluation methods for large language models in the medical field: a scoping review
Lee, J., Park, S., Shin, J., & Cho, B. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02709-7 · Cited by 52
Assessing ChatGPT's ability to pass the FRCS orthopaedic part A exam: A critical analysis
Saad, A., Iyengar, K. P., Kurisunkal, V., & Botchu, R. (2023) · The Surgeon · doi:10.1016/j.surge.2023.07.001 · Cited by 49
Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis
Rojas-Carabali, W., Cifuentes-González, C., Wei, X., Putera, I., Sen, A., Thng, Z. X., Agrawal, R., Elze, T., Sobrin, L., Kempen, J. H., et al. (2024) · Ocular Immunology and Inflammation · doi:10.1080/09273948.2023.2253471 · Cited by 47
Almeida, L. C., Farina, E. M. J. D. M., Kuriki, P. E. D. A., Abdala, N., & Kitamura, F. (2023) · Radiology Artificial Intelligence · doi:10.1148/ryai.230103 · Cited by 52
ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions
Buhr, C. R., Smith, H., Huppertz, T., Bahr-Hamm, K., Matthias, C., Blaikie, A., Kelsey, T., Kuhn, S., & Eckrich, J. (2023) · JMIR Medical Education · doi:10.2196/49183 · Cited by 45
Ebrahimian, M., Behnam, B., Ghayebi, N., & Sobhrakhshankhah, E. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100815 · Cited by 47
PAIGE: A generative AI-based framework for promoting assignment integrity in higher education
Shanto, S. S., Ahmed, Z., & Jony, A. I. (2023) · STEM Education · doi:10.3934/steme.2023018 · Cited by 42
Knoedler, L., Alfertshofer, M., Knoedler, S., Hoch, C. C., Funk, P. F., Cotofana, S., Maheta, B., Frank, K., Brébant, V., Prantl, L., et al. (2024) · JMIR Medical Education · doi:10.2196/51148 · Cited by 49
Generative AI as Virtual Healthcare Assistant for Enhancing Patient Care Quality
Samala, A. D., & Rawas, S. (2024) · International Journal of Online and Biomedical Engineering (iJOE) · doi:10.3991/ijoe.v20i05.45937 · Cited by 42
Zhu, L., Mou, W., Lai, Y., Chen, J., Lin, S., Xu, L., Lin, J., Guo, Z., Yang, T., Lin, A., et al. (2024) · International Journal of Surgery · doi:10.1097/js9.0000000000001359 · Cited by 52
Readability analysis of ChatGPT's responses on lung cancer
Gencer, A. (2024) · Scientific Reports · doi:10.1038/s41598-024-67293-2 · Cited by 52
Mortlock, R., & Lucas, C. (2024) · Exploratory Research in Clinical and Social Pharmacy · doi:10.1016/j.rcsop.2024.100481 · Cited by 51
Wang, L., Li, J., Zhuang, B., Huang, S., Fang, M., Wang, C., Li, W., Zhang, M., & Gong, S. (2025) · Journal of Medical Internet Research · doi:10.2196/64486 · Cited by 51
Hung, Y.-C., Chaker, S. C., Sigel, M., Saad, M., & Slater, E. D. (2023) · Annals of Plastic Surgery · doi:10.1097/sap.0000000000003634 · Cited by 43
Agarwal, M., Goswami, A., & Sharma, P. (2023) · Cureus · doi:10.7759/cureus.46222 · Cited by 34
Kunitsu, Y. (2023) · JMIR Medical Education · doi:10.2196/48452 · Cited by 43
Chat bots to Virtual Tutors: An Overview of Chat GPT's Role in the Future of Education
Graefen, B., & Fazal, N. (2024) · Archives of Pharmacy Practice · doi:10.51847/touppjedsx · Cited by 28
Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection
Li, Y., Liu, Y., Deng, G., Zhang, Y., Song, W., Shi, L., Wang, K., Li, Y., Liu, Y., & Wang, H. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660799 · Cited by 51
Analysis of ChatGPT Responses to Ophthalmic Cases: Can ChatGPT Think like an Ophthalmologist?
Chen, J., Reddy, A. J., Al-Sharif, E. M., Shoji, M. K., Kalaw, F. G. P., Eslani, M., Lang, P. Z., Arya, M., Koretz, Z. A., Bolo, K. A., et al. (2024) · Ophthalmology Science · doi:10.1016/j.xops.2024.100600 · Cited by 53
ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?
Chen, C., Yu, J., Chen, S., Liu, C., Wan, Z., Zhou, S., Luo, Y., Zhang, R., Bitterman, D., Wang, F., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818831 · Cited by 52
ChatGPT translation vs. human translation: an examination of a literary text
Rousan, R. M. A., Jaradat, R., & Malkawi, M. (2025) · Cogent Social Sciences · doi:10.1080/23311886.2025.2472916 · Cited by 53
Can Artificial Intelligence Language Models Effectively Address Dental Trauma Questions?
Kuru, E., Aşık, A., & Demir, D. M. (2025) · Dental Traumatology · doi:10.1111/edt.13063 · Cited by 52
Hsu, H.-Y., Hsu, K.-C., Hou, S.-Y., Wu, C.-L., Hsieh, Y.-W., & Cheng, Y.-D. (2023) · JMIR Medical Education · doi:10.2196/48433 · Cited by 50
Matsui, K., Utsumi, T., Aoki, Y., Maruki, T., Takeshima, M., & Takaesu, Y. (2024) · Journal of Medical Internet Research · doi:10.2196/52758 · Cited by 50
Chen, Y., Huang, X., Yang, F., Lin, H., Lin, H., Zheng, Z., Liang, Q., Zhang, J., & Li, X. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06309-x · Cited by 50
Kufel, J., Paszkiewicz, I., Bielówka, M., Bartnikowska, W., Janik, M., Stencel, M., Czogalik, Ł., Gruszczyńska, K., & Mielcarska, S. (2023) · Polish Journal of Radiology · doi:10.5114/pjr.2023.131215 · Cited by 50
Zhou, S., Luo, X., Chen, C., Jiang, H., Yang, C., Ran, G., Yu, J., & Yin, C. (2024) · International Journal of Surgery · doi:10.1097/js9.0000000000001850 · Cited by 50
Taufiq-Hail, G. A.-M., Yusof, S. A. M., Rashid, A., El-Shekeil, I., & Lutfi, A. (2024) · Emerging Science Journal · doi:10.28991/esj-2024-08-03-02 · Cited by 50
Metadata
- Authors
- Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D.
- Affiliations
- Yale University (US)
- University College Dublin (IE)
- Corresponding author
- Chartash, David
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-02-08
- Volume
- 9
- Pages
- e45312
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC9947764
- OpenAlex
- W4319460874
- Semantic Scholar
- cd4d112f3f9120d0715f22a9de2ce4720822368c
- Cited by
- 2,078
- References
- 21
- Citations by year
- 2026: 351 · 2025: 550 · 2024: 699 · 2023: 474
- FWCI
- 82.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- National Heart, Lung, and Blood Institute (T35HL007649)
- National Institute of Diabetes and Digestive and Kidney Diseases (T35DK104689)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE Step 1, USMLE Step 2, medical licensing examinations, question difficulty, GPT-3, medical education, knowledge assessment, AMBOSS, National Board of Medical Examiners, artificial intelligence in medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.