How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. Objective: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. Methods: We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Results: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. Conclusions: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT’s capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Template of question posed to each large language model (LLM), including both AMBOSS Attending Tip and the response from Chat Generative Pre-trained Transformer (ChatGPT). The correct answer to this question is “E. Zidovudine (AZT).” In the case of GPT-3, prompt engineering was necessary, with: "Please answer this multiple choice question:" + question as described previously + "Correct answer is." As GPT-3 is inherently a nondialogic model, this was necessary to reduce model hallucinations and force a clear answer [17].](https://figures.cochat.ai/paper-figures/583a8f9972504138a5ac67eaf322285a/how-does-chatgpt-perform-on-the-united-states-medical-licensing-examination-figure-1.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, like GPT-4, have shown medical knowledge despite lacking medicine-specific training [10, 29].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, several valid concerns were raised, including the possibility of generating inaccurate content, along with ethical issues, including the risk of bias, plagiarism, and copyright issues [9,18,40,41].”
From Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study (2023) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, analyses of ChatGPT show structured, non-random response patterns consistent with trait-like reasoning abilities18.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the majority of clinical reasoning evaluations on LLMs have been conducted using standardized exams (the United States Medical Licensing Examination) or digital case reports [14,19].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [14]
Experimental — accuracy not yet measured
39 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Springer, L., Stanne, M. E., & Donovan, S. S. (1999) · Review of Educational Research · doi:10.3102/00346543069001021 · Cited by 1,943 · cited 1×
“Small group education has been shown to be a highly efficacious method of teaching [23,24].”
MethodThe use of reflection in medical education: AMEE Guide No. 44
Sandars, J. (2009) · Medical Teacher · doi:10.1080/01421590903050374 · Cited by 1,085
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
Research works printed but not linked: 2
- 10.48550/arxiv.2108.01764 · DOI
- Wang, DC, 2020, Yale J Biol Med, 93, 441
Cited by
Items 301–400 of 1,047
ChatGPT for assessment writing
Zuckerman, M., Flood, R., Tan, R. J. B., Kelp, N., Ecker, D. J., Menke, J., & Lockspeiser, T. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2249239 · Cited by 83
Zhang, X., Zhang, P., Shen, Y., Liu, M., Wang, Q., Gašević, D., & Fan, Y. (2024) · Frontiers of Digital Education · doi:10.1007/s44366-024-0028-5 · Cited by 83
Beaulieu-Jones, B. R., Berrigan, M. T., Shah, S., Marwaha, J. S., Lai, S.-L., & Brat, G. A. (2024) · Surgery · doi:10.1016/j.surg.2023.12.014 · Cited by 83
How Machine Learning (ML) is Transforming Higher Education: A Systematic Literature Review
Pinto, A. S., Abreu, A., Costa, E., & Paiva, J. (2023) · Journal of Information Systems Engineering and Management · doi:10.55267/iadt.07.13227 · Cited by 49
NoteChat: A Dataset of Synthetic Patient-Physician Conversations Conditioned on Clinical Notes
Wang, J., Yao, Z., Yang, Z., Zhou, H., Li, R., Wang, X., Xu, Y., & Yu, H. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.901 · Cited by 83
Medical students and ChatGPT: analyzing attitudes, practices, and academic perceptions
Abdelhafiz, A. S., Farghly, M. I., Sultan, E. A., Abouelmagd, M. E., Ashmawy, Y. M., & Elsebaie, E. H. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06731-9 · Cited by 83
Şişmanoğlu, S., & Çapan, B. Ş. (2025) · BMC Medical Education · doi:10.1186/s12909-024-06389-9 · Cited by 83
Saleh, Z. T., Rababa, M., Elshatarat, R. A., Alharbi, M., Alhumaidi, B. N., Al-Za'areer, M. S., Jarrad, R. A., niarat, T. F. A., Almagharbeh, W. T., Al‐Sayaghi, K. M., et al. (2025) · BMC Nursing · doi:10.1186/s12912-025-03082-0 · Cited by 83
On the Trustworthiness of Generative Foundation Models: Guideline, Assessment, and PerspectivePreprint
Huang, Y., Gao, C., Wu, S., Wang, H., Wang, X., Zhou, Y., Wang, Y., Ye, J., Shi, J., Zhang, Q., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.14296 · Cited by 85
A systematic review and meta-analysis on ChatGPT and its utilization in medical and dental research
Bagde, H., Dhopte, A., Alam, M. K., & Basri, R. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e23050 · Cited by 66
Strengths, Weaknesses, Opportunities, and Threats of Using ChatGPT in Scientific Research
Giray, L., Jacob, J., & Gumalin, D. L. (2024) · International Journal of Technology in Education · doi:10.46328/ijte.618 · Cited by 44
Arthur, F., Salifu, I., & Nortey, S. A. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2024.2362805 · Cited by 83
Nia, M. F., Ahmadi, M., & Irankhah, E. (2025) · Frontiers in Dental Medicine · doi:10.3389/fdmed.2024.1456208 · Cited by 82
Moradi, H. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-025-00530-4 · Cited by 82
Chat or cheat? Academic dishonesty, risk perceptions, and ChatGPT usage in higher education students
Ortiz-Bonnin, S., & Blahopoulou, J. (2025) · Social Psychology of Education · doi:10.1007/s11218-025-10080-2 · Cited by 80
“…in the training data, potentially reproducing discriminatory narratives or reinforcing structural inequalities (Borji & Ai, 2023 ; Comas-Forgas et al., 2025 ; Farrokhnia et al., 2023 ; Gilson et al., 2023; Gonzalo & Enriquez, 2023 ; Gulati et al., 2024; Kitamura, 2023 ; van Dis et al., 2023 ).”
Artificial Intelligence in Health Professions Education assessment: AMEE Guide No. 178
Masters, K., MacNeil, H., Benjamin, J., Carver, T., Nemethy, K., Valanci-Aroesty, S., Taylor, D. C. M., Thoma, B., & Thesen, T. (2025) · Medical Teacher · doi:10.1080/0142159x.2024.2445037 · Cited by 80
Proof of Concept: Using ChatGPT to Teach Emergency Physicians How to Break Bad News
Webb, J. J. (2023) · Cureus · doi:10.7759/cureus.38755 · Cited by 80
“It has even passed the Step 1 United States Medical Licensing Exam [6].”
BackgroundPerformance of ChatGPT on Nephrology Test Questions
Miao, J., Thongprayoon, C., Garcia Valencia, O. A., Krisanapan, P., Sheikh, M. S., Davis, P. W., Mekraksakit, P., Suarez, M. G., Craici, I. M., & Cheungpasitporn, W. (2024) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.0000000000000330 · Cited by 68
Hindelang, M., Sitaru, S., & Zink, A. (2024) · JMIR Medical Informatics · doi:10.2196/56628 · Cited by 79
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78
Evaluating GPT and BERT models for protein–protein interaction identification in biomedical text
Rehana, H., Çam, N. B., Basmaci, M., Zheng, J., Jemiyo, C., He, Y. O., Özgür, A., & Hur, J. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbae133 · Cited by 78
Rane, N. L., Choudhary, S., & Rane, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4645601 · Cited by 78
Uribe, S. E., Maldupa, I., Kavadella, A., El Tantawi, M., Chaurasia, A., Fontana, M., Marino, R., Innes, N., & Schwendicke, F. (2024) · European Journal of Dental Education · doi:10.1111/eje.13009 · Cited by 77
Factors Affecting the Adoption and Use of ChatGPT in Higher Education
Alshammari, S. H., & Alshammari, M. H. (2024) · International Journal of Information and Communication Technology Education · doi:10.4018/ijicte.339557 · Cited by 36
Generative AI in Education: Mapping the Research Landscape Through Bibliometric Analysis
Ng, S. L., & Ho, C. (2025) · Information · doi:10.3390/info16080657 · Cited by 77
Misra, D. P., & Chandwar, K. (2023) · Journal of the Royal College of Physicians of Edinburgh · doi:10.1177/14782715231181023 · Cited by 46
Taira, K., Itaya, T., & Hanada, A. (2023) · JMIR Nursing · doi:10.2196/47305 · Cited by 76
Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical Scenarios
Qu, R. W., Qureshi, U., Petersen, G., & Lee, S. C. (2023) · OTO Open · doi:10.1002/oto2.67 · Cited by 62
Large language models in oncology: a review
Chen, D., Parsa, R., Swanson, K. E., Nuñez, J.-J., Critch, A., Bitterman, D. S., Liu, F., & Raman, S. (2025) · BMJ Oncology · doi:10.1136/bmjonc-2025-000759 · Cited by 76
ChatGPT in ophthalmology: the dawn of a new era?
Ting, D. S. J., Tan, T. F., & Ting, D. S. W. (2023) · Eye · doi:10.1038/s41433-023-02619-4 · Cited by 76
ChatGPT as an educational resource for civil engineering students
Uddin, S. M. J., Albert, A., Tamanna, M., Ovid, A., & Alsharef, A. F. (2024) · Computer Applications in Engineering Education · doi:10.1002/cae.22747 · Cited by 75
Zhou, W., & Kim, Y. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12705-z · Cited by 75
ChatGPT and Other Large Language Models in Medical Education — Scoping Literature Review
Aster, A., Laupichler, M. C., Rockwell-Kollmann, T., Masala, G., Bala, E., & Raupach, T. (2024) · Medical Science Educator · doi:10.1007/s40670-024-02206-6 · Cited by 75
Meo, S. A., Al-Masri, A. A., Alotaibi, M., Meo, M. Z. S., & Meo, M. O. S. (2023) · Healthcare · doi:10.3390/healthcare11142046 · Cited by 74
“, 2023 [25] evaluated ChatGPT’s performance on questions which were within the bounds of the United States Medical Licensing Examination Step 1 and 2 examinations.”
BackgroundAl-Worafi, Y. M., Goh, K. W., Hermansyah, A., Tan, C. S., & Ming, L. C. (2024) · JMIR Medical Education · doi:10.2196/47339 · Cited by 74
ChatGPT for Clinical Vignette Generation, Revision, and Evaluation
Benoit, J. R. A. (2023) · doi:10.1101/2023.02.04.23285478 · Cited by 74
Farhat, F., Chaudhry, B. M., Nadeem, M., Sohail, S. S., & Madsen, D. Ø. (2024) · JMIR Medical Education · doi:10.2196/51523 · Cited by 61
Panagoulias, D. P., Virvou, M. K., & Tsihrintzis, G. A. (2024) · Electronics · doi:10.3390/electronics13020320 · Cited by 74
Liu, M., Okuhara, T., Dai, Z., Huang, W., Gu, L., Okada, H., Furukawa, E., & Kiuchi, T. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105673 · Cited by 73
Ganjavi, C., Eppler, M., O’Brien, D., Ramacciotti, L. S., Ghauri, M. S., Anderson, I., Choi, J., Dwyer, D., Stephens, C., Shi, V., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000596 · Cited by 73
“While there have been publications on ChatGPT usage in education [10,11], the opinions and current use of the technology by medical students in real-world settings has not been studied.”
Data-Centric Foundation Models in Computational Healthcare: A Survey
Zhang, Y., Gao, J. G., Tan, Z., Zhou, L., Ding, K., Zhou, M., Zhang, S., & Wang, D. (2026) · ACM Computing Surveys · doi:10.1145/3800677 · Cited by 74
Ogundiya, O., Rahman, T. J., Valnarov-Boulter, I., & Young, T. (2024) · Journal of Medical Internet Research · doi:10.2196/60312 · Cited by 73
Magalhães Araujo, S., & Cruz-Correia, R. (2024) · JMIR Medical Education · doi:10.2196/51151 · Cited by 72
Agarwal, M., Sharma, P., & Goswami, A. (2023) · Cureus · doi:10.7759/cureus.40977 · Cited by 72
“ChatGPT not only exhibited the capacity to pass the USMLE independently without any human assistance but also showcased comprehensible reasoning and provided valid clinical insights in its responses [11,12].”
BackgroundGuillen-Grima, F., Guillen-Aguinaga, S., Guillen-Aguinaga, L., Alas-Brun, R., Onambele, L., Ortega, W., Montejo, R., Aguinaga-Ontoso, E., Barach, P., & Aguinaga-Ontoso, I. (2023) · Clinics and Practice · doi:10.3390/clinpract13060130 · Cited by 72
“However, in a scientific investigation involving GPT-4, researchers applied National Board of Medical Examiners (NBME) questions to the model and analyzed their explanations for the answers [71].”
MethodChatGPT and Librarians for Reference Consultations
Adetayo, A. J. (2023) · Internet Reference Services Quarterly · doi:10.1080/10875301.2023.2203681 · Cited by 50
Hadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 53
Evaluating the performance of ChatGPT in answering questions related to urolithiasis
Çakır, H., Çağlar, U., Yıldız, O., Meriç, A., Ayrancı, A., & Özgör, F. (2023) · International Urology and Nephrology · doi:10.1007/s11255-023-03773-0 · Cited by 71
Long, C., Lowe, K., Zhang, J., Santos, A. D., Alanazi, A., O'Brien, D., Wright, E. D., & Cote, D. (2024) · JMIR Medical Education · doi:10.2196/49970 · Cited by 46
Collaborative, T. C. (2025) · BMJ · doi:10.1136/bmj-2024-083305 · Cited by 72
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
Balloccu, S., Schmidtová, P., Lango, M., & Dusek, O. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.5 · Cited by 70
Examining ChatGPT Performance on USMLE Sample Items and Implications for Assessment
Yaneva, V., Baldwin, P., Jurich, D. P., Swygert, K., & Clauser, B. E. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005549 · Cited by 60
Tanaka, Y., Nakata, T., Aiga, K., Etani, T., Muramatsu, R., Katagiri, S., Kawai, H., Higashino, F., Enomoto, M., Noda, M., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000433 · Cited by 62
The model student: GPT-4 performance on graduate biomedical science exams
Stribling, D., Xia, Y., Amer, M. K., Graim, K. S., Mulligan, C. J., & Renne, R. (2024) · Scientific Reports · doi:10.1038/s41598-024-55568-7 · Cited by 52
Williams, A. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00485-y · Cited by 48
Li, J., Yin, K., Wang, Y., Jiang, X., & Chen, D. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07750-2 · Cited by 70
Shao, Y., Tang, X., Sohn, J., Chen, J., Liao, Y., Zhang, J., Xiang, J., Wu, F., Zhao, Y., Wu, C., et al. (2026) · Patterns · doi:10.1016/j.patter.2026.101658 · Cited by 70
The Emerging Role of Generative Artificial Intelligence in Medical Education, Research, and Practice
Shoja, M. M., Van de Ridder, J. M. M., & Rajput, V. (2023) · Cureus · doi:10.7759/cureus.40883 · Cited by 69
“suggested that ChatGPT has the potential to facilitate the development of an interactive, externally supported, and on-demand learning environment for students, which could enhance problem-solving skills and reflective learning [15].”
ChatGPT’s Response Consistency: A Study on Repeated Queries of Medical Examination Questions
Funk, P. F., Hoch, C. C., Knoedler, S., Knoedler, L., Cotofana, S., Sofo, G., Bashiri Dezfouli, A., Wollenberg, B., Guntinas-Lichius, O., & Alfertshofer, M. (2024) · European Journal of Investigation in Health, Psychology and Education · doi:10.3390/ejihpe14030043 · Cited by 69
A bibliometric analysis of the rise of ChatGPT in medical research
Barrington, N. M., Gupta, N., Musmar, B., Doyle, D., Panico, N., Godbole, N., Reardon, T., & D’Amico, R. S. (2023) · Medical Sciences · doi:10.3390/medsci11030061 · Cited by 69
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
Maillard, A., Micheli, G., Lefevre, L., Guyonnet, C., Poyart, C., Canouï, E., Belan, M., & Charlier, C. (2024) · Clinical Infectious Diseases · doi:10.1093/cid/ciad632 · Cited by 64
ChatGPT for Academic Purposes: Survey Among Undergraduate Healthcare Students in Malaysia
George Pallivathukal, R., Kyaw Soe, H. H., Donald, P. M., Samson, R. S., & Hj Ismail, A. R. (2024) · Cureus · doi:10.7759/cureus.53032 · Cited by 68
“…the accuracy of ChatGPT among the medical and dental educational fraternity, it is noteworthy that the healthcare students' perceptions regarding its utility for academic purposes received limited attention within the realm of research [2,17,18], particularly in the context of Southeast Asia.”
BackgroundThe Intersection of ChatGPT, Clinical Medicine, and Medical Education
Wong, R. S.-Y., Ming, L. C., & Ali, R. A. R. (2023) · JMIR Medical Education · doi:10.2196/47274 · Cited by 69
Seo, J., Choi, D., Kim, T., Cha, W. C., Kim, M., Yoo, H., Oh, N., Yi, Y., Lee, K. H., & Choi, E. (2024) · Journal of Medical Internet Research · doi:10.2196/58329 · Cited by 52
Safety analysis in the era of large language models: A case study of STPA using ChatGPT
Qi, Y., Zhao, X., Khastgir, S., & Huang, X. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100622 · Cited by 68
Duah, J. E., & McGivern, P. (2024) · The International Journal of Information and Learning Technology · doi:10.1108/ijilt-11-2023-0213 · Cited by 67
Ghosh, A., & Bir, A. (2023) · Cureus · doi:10.7759/cureus.37023 · Cited by 67
Watters, C., & Lemański, M. K. (2023) · Frontiers in Big Data · doi:10.3389/fdata.2023.1224976 · Cited by 69
Zhang, S., Liau, Z. Q. G., Tan, K. L. M., & Chua, W. L. (2024) · Knee Surgery and Related Research · doi:10.1186/s43019-024-00218-5 · Cited by 67
Brondani, M., Alves, C. M. C., Ribeiro, C. C. C., Braga, M. M., Garcia, R. C. M. R., Ardenghi, T. M., & Pattanaporn, K. (2024) · Journal of Dental Education · doi:10.1002/jdd.13663 · Cited by 67
Collaborative Uses of GenAI Tools in Project-Based Learning
Perifanou, M. A., & Economides, A. A. (2025) · Education Sciences · doi:10.3390/educsci15030354 · Cited by 67
Reasoning with large language models for medical question answering
Lucas, M. M., Yang, J., Pomeroy, J. K., & Yang, C. C. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae131 · Cited by 66
Rahimzadeh, V., Kostick-Quenet, K., Blumenthal Barby, J., & McGuire, A. L. (2023) · The American Journal of Bioethics · doi:10.1080/15265161.2023.2233358 · Cited by 66
Chat Generative Pre-Trained Transformer (ChatGPT) usage in healthcare
Zhang, Y., Pei, H., Zhen, S., Li, Q., & Liang, F. (2023) · Gastroenterology & Endoscopy · doi:10.1016/j.gande.2023.07.002 · Cited by 46
Hsiao, Y.-P., Klijn, N., & Chiu, M.-S. (2023) · Learning: Research and Practice · doi:10.1080/23735082.2023.2257234 · Cited by 36
Influence on the accuracy in ChatGPT: Differences in the amount of information per medical field
Haze, T., Kawano, R., Takase, H., Suzuki, S., Hirawa, N., & Tamura, K. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105283 · Cited by 58
Empirical assessment of ChatGPT’s answering capabilities in natural science and engineering
Balhorn, L. S., Weber, J. M., Buijsman, S., Hildebrandt, J. R., Ziefle, M., & Schweidtmann, A. M. (2024) · Scientific Reports · doi:10.1038/s41598-024-54936-7 · Cited by 66
Pre-trained language models in medicine: A survey
Luo, X., Deng, Z., Yang, B., & Luo, M. Y. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102904 · Cited by 66
Visualizing the knowledge mapping of artificial intelligence in education: A systematic review
Qin, Q., & Zhang, S. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13076-1 · Cited by 68
Status and perceptions of ChatGPT utilization among medical students: a survey-based study
Hu, N., Jiang, X. L., Wang, Y. D., Kang, Y. M., Xia, Z., Chen, H. H., Duan, S. N., & Chen, D. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07438-7 · Cited by 69
MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence
Liu, J., Wang, W., Ma, Z., Huang, G., Yihang, S., Chang, K.-J., Li, H., Shen, L., Lyu, M. R.-T., & Chen, W. (2025) · neural information processing systems · doi:10.52202/085713-2498 · Cited by 66
Two artificial intelligence models underperform on examinations in a veterinary curriculum
Coleman, M. C., & Moore, J. N. (2024) · Journal of the American Veterinary Medical Association · doi:10.2460/javma.23.12.0666 · Cited by 65
A Bibliometric Analysis of Research on ChatGPT in Education
Polat, H., Topuz, A. C., Yıldız, M., Taşlıbeyaz, E., & Kurşun, E. (2024) · International Journal of Technology in Education · doi:10.46328/ijte.606 · Cited by 35
Waldock, W. J., Zhang, J., Guni, A., Nabeel, A., Darzi, A. W., & Ashrafian, H. (2024) · Journal of Medical Internet Research · doi:10.2196/56532 · Cited by 65
Xu, P., Wu, Y., Jin, K., Chen, X., He, M., & Shi, D. (2025) · Advances in Ophthalmology Practice and Research · doi:10.1016/j.aopr.2025.05.001 · Cited by 65
Exploring Capabilities of Large Language Models such as ChatGPT in Radiation Oncology
Dennstädt, F., Hastings, J., Putora, P. M., Vu, E., Fischer, G. F., Süveg, K., Glatzer, M., Riggenbach, E., Hà, H.-L., & Cihoric, N. (2024) · Advances in Radiation Oncology · doi:10.1016/j.adro.2023.101400 · Cited by 48
Assessing Ability for ChatGPT to Answer Total Knee Arthroplasty-Related Questions
Magruder, M. L., Rodriguez, A. N., Wong, J., Erez, O., Piuzzi, N. S., Scuderi, G. R., Slover, J. D., Oh, J. H., Schwarzkopf, R. M., Chen, A. F., et al. (2024) · The Journal of Arthroplasty · doi:10.1016/j.arth.2024.02.023 · Cited by 64
Sumbal, A., Sumbal, R., & Amir, A. (2024) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205241238641 · Cited by 63
Assessing the accuracy of ChatGPT references in head and neck and ENT disciplines
Frosolini, A., Franz, L., Benedetti, S., Vaira, L. A., de Filippis, C., Gennaro, P., Marioni, G., & Gabriele, G. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08205-4 · Cited by 63
The Role and Potential Contributions of the Artificial Intelligence Language Model ChatGPT
Berşe, S., Akça, K., Dirgar, E., & Kaplan Serin, E. (2024) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03296-w · Cited by 63
“The effects on nursing education have not yet been determined, but research has shown that ChatGPT can be effectively used to explain difficult concepts, manage student time and assignments, create patient scenarios [10].”
ChatGPT as a tool for medical education and clinical decision-making on the wards: Case study
Skryd, A., & Lawrence, K. (2024) · JMIR Formative Research · doi:10.2196/51346 · Cited by 63
Potential Use of ChatGPT for Patient Information in Periodontology: A Descriptive Pilot Study
Babayiğit, O., Tastan Eroglu, Z., Ozkan Sen, D., & Ucan Yarkac, F. (2023) · Cureus · doi:10.7759/cureus.48518 · Cited by 42
Hermann, C. E., Patel, J. M., Boyd, L., Growdon, W. B., Aviki, E., & Stasenko, M. (2023) · Gynecologic Oncology · doi:10.1016/j.ygyno.2023.11.008 · Cited by 56
Mihalache, A., Grad, J., Patil, N. S., Huang, R. S., Popovic, M. M., Mallipatna, A. C., Kertes, P. J., & Muni, R. H. (2024) · Eye · doi:10.1038/s41433-024-03067-4 · Cited by 63
Strategic view on the current role of AI in advancing environmental sustainability: a SWOT analysis
Greif, L., Kimmig, A., Bobbou, S. E., Jurisch, P., & Ovtcharova, J. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00146-z · Cited by 64
Asy’ari, M., & Шаров, С. (2024) · International Journal of Essential Competencies in Education · doi:10.36312/ijece.v3i2.2424 · Cited by 63
Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench
Huang, J.-T., Wang, W., Li, E. J., Lam, M. H., Ren, S., Yuan, Y., Jiao, W., Tu, Z., & Lyu, M. R.-T. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.01386 · Cited by 64
Seeing ChatGPT Through Students' Eyes: An Analysis of TikTok Data
Haensch, A.-C., Ball, S., Herklotz, M., & Kreuter, F. (2023) · arXiv · doi:10.48550/arxiv.2303.05349 · Cited by 62
Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration
Qian, C., & Wexler, J. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645198 · Cited by 62
Metadata
- Authors
- Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D.
- Affiliations
- Yale University (US)
- University College Dublin (IE)
- Corresponding author
- Chartash, David
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-02-08
- Volume
- 9
- Pages
- e45312
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC9947764
- OpenAlex
- W4319460874
- Semantic Scholar
- cd4d112f3f9120d0715f22a9de2ce4720822368c
- Cited by
- 2,078
- References
- 21
- Citations by year
- 2026: 351 · 2025: 550 · 2024: 699 · 2023: 474
- FWCI
- 82.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- National Heart, Lung, and Blood Institute (T35HL007649)
- National Institute of Diabetes and Digestive and Kidney Diseases (T35DK104689)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE Step 1, USMLE Step 2, medical licensing examinations, question difficulty, GPT-3, medical education, knowledge assessment, AMBOSS, National Board of Medical Examiners, artificial intelligence in medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.