Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Open accessPublished version of a preprint
The DOI matches its Crossref record, with a small difference in title or year; 28 of the 28 references it cites were found in a registry or an index.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Kung et al. (2023) evaluated the performance of ChatGPT on the United States Medical Licensing Exam and demonstrated that the chatbot did not reach the passing threshold and achieved an overall accuracy of only 50% (Kung et al., 2023).”
From A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing (2025) · cites this paper as Kung et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [17]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI's capabilities, such as passing the United States Medical Licensing Exam [107], completing exams at the University of Minnesota Law School [108], and solving queries from Wharton School of Business tests [109], underscore its potential.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [107]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The HCI community has explored the application of LLMs for various healthcare practices, such as mental health support [66,68], health information seeking [81,117,128], and public health interventions [58]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
From DIALOGUE: A Generative AI-Based Pre–Post Simulation Study to Enhance Diagnostic Communication in Medical Students Through Virtual Type 2 Diabetes Scenarios (2025) · cites this paper as Kung et al., 2023
Experimental — accuracy not yet measured
52 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background
Cited by
Items 601–700 of 1,129
Cuthbert, R., & Simpson, A. I. (2023) · Postgraduate Medical Journal · doi:10.1093/postmj/qgad053 · Cited by 56
Assessing Biases in Medical Decisions via Clinician and AI Chatbot Responses to Patient Vignettes
Kim, J., Cai, Z. R., Chen, M. L., Simard, J. F., & Linos, E. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.38050 · Cited by 59
Su, J., & Yang, W. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2023.2266490 · Cited by 42
“Kung et al. (2023) assessed the performance of ChatGPT in the US Medical Licensing Exam (USMLE), which examines doctors ’ application of knowledge, principles and basic patient-centered skills (USMLE, 2023).”
Kim, J., Lee, J., Jang, K. M., & Lourentzou, I. (2023) · Telematics and Informatics · doi:10.1016/j.tele.2023.102085 · Cited by 63
Fagbohun, O., Harrison, R. M., & Dereventsov, A. (2023) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/15 · Cited by 69
“To comprehend, generate, and interpret human language, leading to profound impacts across diverse domains from tech-focused areas like finance 14,15 and programming 16,17 to human-centric sectors like education 18,19 and healthcare 20,21 .”
International Scientific Report on the Safety of Advanced AI (Interim Report)Preprint
Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Goldfarb, D. A., Heidari, H., Khalatbari, L., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.05282 · Cited by 79
Abbas, A., Rehman, M. S., & Rehman, S. S. (2024) · Cureus · doi:10.7759/cureus.55991 · Cited by 58
“Although previous studies have shown the ability of individual LLMs to pass certain medical licensing exams [6], there is limited research comparing the performance of different LLMs.”
BackgroundChatGPT in Medical Education and Research: A Boon or a Bane?
Jeyaraman, M., K, S. P., Jeyaraman, N., Nallakumarasamy, A., Yadav, S., & Bondili, S. K. (2023) · Cureus · doi:10.7759/cureus.44316 · Cited by 58
“According to new research by Kung et al., ChatGPT performed at or near the passing level for all three United States Medical Licensure Exams (USMLE) [3].”
BackgroundAlanzi, T. M. (2023) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s419847 · Cited by 58
Shaw, K., Henning, M. A., & Webster, C. S. (2025) · Medical Science Educator · doi:10.1007/s40670-025-02373-0 · Cited by 58
“More significantly, it reliably generated comprehensive clinical reasoning and novel insights in 90% of responses, suggesting a role for AI in medical education [5].”
Nomura, A. (2023) · Hypertension Research · doi:10.1038/s41440-023-01317-8 · Cited by 66
“Surprisingly, the ChatGPT has already been scored at or near the passing threshold on the United States Medical Licensing Exam [16].”
BackgroundAyub, I., Hamann, D., Hamann, C. R., & Davis, M. J. (2023) · Cureus · doi:10.7759/cureus.43717 · Cited by 53
“Furthermore, ChatGPT has been employed for taking licensing examinations and responding to specialty board review questions, demonstrating an average accuracy rate close to passing thresholds [6,7].”
MethodHarnessing customized AI to create voice of customer via GPT3.5
Shahin, M., Chen, F. F., & Hosseinzadeh, A. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102462 · Cited by 67
Visualizing the knowledge mapping of artificial intelligence in education: A systematic review
Qin, Q., & Zhang, S. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13076-1 · Cited by 68
“Gilson et al. (2023) and Kung et al. (2023) both explore the evaluative role of GPT in education.”
Wang, C., Li, S., Lin, N., Zhang, X., Han, Y., Wang, X., Liu, D., Tan, X., Pu, D., Li, K., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/59435 · Cited by 69
The rise of small language models in healthcare: A comprehensive survey
Garg, M., Raza, S., Rayana, S., Liu, X., & Sohn, S. (2026) · Computer Science Review · doi:10.1016/j.cosrev.2026.100999 · Cited by 81
Malik, A., Khan, M. L., Hussain, K., Qadir, J., & Tarhini, A. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2024.2409407 · Cited by 57
“According to these tests, the responses produced average passing scores and demonstrated ChatGPT ’ s ability to contextualize and provide pertinent responses (Choi et al., 2023; Kung et al., 2023).”
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv · doi:10.48550/arxiv.2305.10415 · Cited by 66
“Notably, even in domains where expert knowledge is supposed to play a critical role, like medical diagnosis, these language models have also achieved impressive success, passing the United States Medical Licensing Examination (USMLE) [13, 17, 26, 32].”
BackgroundFaithful AI in Medicine: A Systematic Review with Large Language Models and Beyond
Xie, Q., Schenck, E. J., Yang, H. S., Chen, Y., Peng, Y., & Wang, F. (2023) · doi:10.21203/rs.3.rs-3661764/v1 · Cited by 57
Future of AI & Generation Alpha: ChatGPT beyond Boundaries
Rathore, D. B. (2023) · Eduzone : international peer reviewed/refereed academic multidisciplinary journal · doi:10.56614/eiprmj.v12i1y23.254 · Cited by 41
Pre-trained language models in medicine: A survey
Luo, X., Deng, Z., Yang, B., & Luo, M. Y. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102904 · Cited by 66
Kopka, M., von Kalckreuth, N., & Feufel, M. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01566-6 · Cited by 69
Kao, H.-J., Chien, T.-W., Wang, W.-C., Chou, W., & Chow, J. C. (2023) · Medicine · doi:10.1097/md.0000000000034068 · Cited by 57
“[53,54] The results showed that ChatGPT performed at or near the passing threshold for all 3 exams, without any specialized training or reinforcement.”
BackgroundSuppadungsuk, S., Thongprayoon, C., Krisanapan, P., Tangpanithandee, S., Garcia Valencia, O., Miao, J., Mekraksakit, P., Kashani, K., & Cheungpasitporn, W. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12175550 · Cited by 56
Huang, R. S., Lu, K. J. Q., Meaney, C., Kemppainen, J., Punnett, A., & Leung, F.-H. (2023) · JMIR Medical Education · doi:10.2196/50514 · Cited by 56
“5 performed at or near the passing threshold on all 3 exams: Step 1, Step 2 CK, and Step 3 [8].”
Aye, AI! ChatGPT passes multiple-choice family medicine exam
Morreel, S., Mathysen, D., & Verhoeven, V. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2187684 · Cited by 56
Nghi, T. T., & Tran, H. P. (2023) · Proceedings of the AsiaCALL International Conference · doi:10.54855/paic.2341 · Cited by 62
“AI tools have the potential to offer customized and adaptable language learning experiences that cater to the specific needs of individual learners (Grassini, 2023; Kasneci et al., 2023; Kung et al., 2023).”
BackgroundYang, J., Ardavanis, K. S., Slack, K. E., Fernando, N. D., Valle, C. J. D., & Hernandez, N. M. (2024) · The Journal of Arthroplasty · doi:10.1016/j.arth.2024.01.029 · Cited by 59
Kim, J. H., Kim, S. K., Choi, J., & Lee, Y. (2024) · DIGITAL HEALTH · doi:10.1177/20552076241227132 · Cited by 47
“11 Given its predictive power, previous studies have examined its performance in medical licensing exams 20 and the use of a data set for medical questions.”
BackgroundStrategic view on the current role of AI in advancing environmental sustainability: a SWOT analysis
Greif, L., Kimmig, A., Bobbou, S. E., Jurisch, P., & Ovtcharova, J. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00146-z · Cited by 66
“In the short term, current AI models, such as LLMs, are already being positively evaluated for their potential as assistants in medical education [95, 96] showing a high level of concordance and insight in its explanations.”
Masanneck, L., Meuth, S. G., & Pawlitzki, M. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01536-y · Cited by 60
A systematic review of current AI techniques used in the context of the SDGs
Greif, L., Röckel, F., Kimmig, A., & Ovtcharova, J. (2025) · International Journal of Environmental Research · doi:10.1007/s41742-024-00668-5 · Cited by 55
“In the short term, the use of LLMs for roles such as educational aids and analytical tools shows promise, as evidenced in sectors such as medical education (Kung et al. 2023; Gilson et al. 2023).”
ChatGPT in higher education—a student's perspective
Shuhaiber, A., Kuhail, M. A., & Salman, S. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2024.100565 · Cited by 55
Cowling, M., Crawford, J., Allen, K.-A., & Wehmeyer, M. (2023) · Australasian Journal of Educational Technology · doi:10.14742/ajet.8598 · Cited by 55
“Although prior AIs have focused primarily on a process of forecasting and predicting, the new breed of generative AI tools embody a level of creativity that has been unseen in the past (Kung et al., 2023).”
BackgroundMethodXie, Y., Seth, I., Hunter‐Smith, D. J., Rozen, W. M., & Seifman, M. A. (2024) · ANZ Journal of Surgery · doi:10.1111/ans.18666 · Cited by 55
ChatGPT and conversational artificial intelligence: Friend, foe, or future of research?
Gottlieb, M., Kline, J. A., Schneider, A. J., & Coates, W. C. (2023) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2023.05.018 · Cited by 51
ChatGPT and Patient Information in Nuclear Medicine: GPT-3.5 Versus GPT-4
Currie, G., Robbie, S., & Tually, P. (2023) · Journal of Nuclear Medicine Technology · doi:10.2967/jnmt.123.266151 · Cited by 54
Lim, D. Y. Z., Tan, Y. B., Koh, J. T. E., Tung, J. Y. M., Sng, G. G. R., Tan, D. M. Y., & Tan, C. (2024) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.16375 · Cited by 53
Datta, S., Lee, K., Paek, H., Manion, F. J., Ofoegbu, N., Du, J., Li, Y., Huang, L., Wang, J., Lin, B., et al. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad218 · Cited by 76
Fu, X., Wang, R., & Li, C. (2023) · Journal of the American Planning Association · doi:10.1080/01944363.2023.2271893 · Cited by 64
“…in line with the common findings in the existing literature that ChatGPT generally under-performs in answering domain-specific questions that require domain expert knowledge, such as ophthalmology (Antaki et al., 2023), education (Kasneci et al., 2023; Kung et al., 2023), and law (Chalkidis, 2023).”
ResultChatGPT: Artificial Intelligence as a Potential Tool for Parents Seeking Information About Autism
McFayden, T. C., Bristol, S. K., Putnam, O. C., & Harrop, C. (2024) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2023.0202 · Cited by 61
How well do large language model-based chatbots perform in oral and maxillofacial radiology?
Jeong, H., Han, S., Yu, Y., Kim, S., & Jeon, K. J. (2024) · Dentomaxillofacial Radiology · doi:10.1093/dmfr/twae021 · Cited by 60
Perceptions of ChatGPT in healthcare: usefulness, trust, and risk
Chen, S.-Y., Kuo, H. Y., & Chang, S.-H. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1457131 · Cited by 41
Waldock, W. J., Zhang, J., Guni, A., Nabeel, A., Darzi, A. W., & Ashrafian, H. (2024) · Journal of Medical Internet Research · doi:10.2196/56532 · Cited by 65
Deebel, N. A., & Terlecki, R. (2023) · Urology · doi:10.1016/j.urology.2023.05.010 · Cited by 54
Watari, T., Takagi, S., Sakaguchi, K., Nishizaki, Y., Shimizu, T., Yamamoto, Y., & Tokuda, Y. (2023) · JMIR Medical Education · doi:10.2196/52202 · Cited by 54
“However, these accomplishments have been attained exclusively in English, and investigations conducted until 2022 cast doubt on its ability to provide medically reliable responses in non-English languages [6].”
BackgroundChatGPT in Plastic and Reconstructive Surgery
Sharma, S. C., Ramchandani, J. P., Thakker, A., & Lahiri, A. (2023) · Indian Journal of Plastic Surgery · doi:10.1055/s-0043-1771514 · Cited by 52
Exploring Capabilities of Large Language Models such as ChatGPT in Radiation Oncology
Dennstädt, F., Hastings, J., Putora, P. M., Vu, E., Fischer, G. F., Süveg, K., Glatzer, M., Riggenbach, E., Hà, H.-L., & Cihoric, N. (2024) · Advances in Radiation Oncology · doi:10.1016/j.adro.2023.101400 · Cited by 48
“11 Furthermore, ChatGPT performed near the level of the passing threshold on the United States Medical Licensing Examination in a study by Kung et al. 12 In another study by Ayers et al, evaluators preferred the responses of ChatGPT over physician responses to patient questions from a social media forum in 78.6% of ca…”
BackgroundInfluentialChoi, W. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04832-x · Cited by 48
“After ChatGPT met the passing threshold on the United States Medical Licensing Examination (USMLE) [7, 8], many authors applied ChatGPT on answering other multiple-choice questions (MCQs) in the medical domain such as physiology [9], anesthesiology [10, 11], ophthalmology [12], and parasitology…”
MethodRizzo, M. G., Cai, N., & Constantinescu, D. S. (2023) · Journal of Orthopaedics · doi:10.1016/j.jor.2023.11.056 · Cited by 59
Mejia, M. R., Arroyave, J. S., Saturno, M., Ndjonko, L. C. M., Zaidat, B., Rajjoub, R., Ahmed, W., Zapolsky, I., & Cho, S. K. (2024) · Neurospine · doi:10.14245/ns.2347052.526 · Cited by 52
Mahmud, A., Sarower, A. H., Sohel, A., Assaduzzaman, M., & Bhuiyan, T. (2024) · Array · doi:10.1016/j.array.2024.100339 · Cited by 65
Cui, P., & Alias, B. S. (2024) · Journal of Infrastructure Policy and Development · doi:10.24294/jipd.v8i11.8390 · Cited by 63
“Blended learning and AI-based learning have emerged as pivotal topics within education due to ongoing societal progress (Kung et al., 2023).”
BackgroundShay, D., Kumar, B., Bellamy, D., Palepu, A. K., Dershwitz, M., Walz, J. M., Schaefer, M. S., & Beam, A. L. (2023) · British Journal of Anaesthesia · doi:10.1016/j.bja.2023.04.017 · Cited by 55
Assessing the Accuracy of ChatGPT on Core Questions in Glomerular Disease
Miao, J., Thongprayoon, C., & Cheungpasitporn, W. (2023) · Kidney International Reports · doi:10.1016/j.ekir.2023.05.014 · Cited by 53
“The news that ChatGPT has approached or even exceeded the passing threshold (approximately 60%) for the United States Medical Licensing Examination, a high-stakes, comprehensive 3-step standardized testing program covering all topics of medical knowledge from basic science, clinical knowledge and management to bioethi…”
BackgroundMethodChatGPT encounters multiple opportunities and challenges in neurosurgery
Kuang, Y.-R., Zou, M.-X., Niu, H.-Q., Zheng, B.-Y., Zhang, T.-L., & Zheng, B.-W. (2023) · International Journal of Surgery · doi:10.1097/js9.0000000000000571 · Cited by 49
Xiao, D., Meyers, P., Upperman, J. S., & Robinson, J. R. (2023) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2023.07.008 · Cited by 51
Russe, M. F., Fink, A., Ngo, H., Tran, H., Bamberg, F., Reisert, M., & Rau, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-41512-8 · Cited by 49
Potential Applications and Impact of ChatGPT in Radiology
Bajaj, S., Gandhi, D., & Nayar, D. (2023) · Academic Radiology · doi:10.1016/j.acra.2023.08.039 · Cited by 59
Heart-to-heart with ChatGPT: the impact of patients consulting AI for cardiovascular health advice
Lautrup, A. D., Hyrup, T., Schneider-Kamp, A., Dahl, M., Lindholt, J. S., & Schneider-Kamp, P. (2023) · Open Heart · doi:10.1136/openhrt-2023-002455 · Cited by 48
Credibility of ChatGPT in the assessment of obesity in type 2 diabetes according to the guidelines
Barlas, T., Altınova, A. E., Aktürk, M., & Törüner, F. B. (2023) · International Journal of Obesity · doi:10.1038/s41366-023-01410-5 · Cited by 59
“Large language models (LLMs) are a specialized form of arti fi cial intelligence (AI) algorithm trained for natural language processing, with a focus on predicting word sequences based on context [7].”
Empirical assessment of ChatGPT’s answering capabilities in natural science and engineering
Balhorn, L. S., Weber, J. M., Buijsman, S., Hildebrandt, J. R., Ziefle, M., & Schweidtmann, A. M. (2024) · Scientific Reports · doi:10.1038/s41598-024-54936-7 · Cited by 66
“Previous studies show that ChatGPT reaches near-passing grades on medical licensing exams 23,24 and passes with a low grade in law school exams 25 and a first-year mechanics course 26 .”
Kim, J., Wang, K., Weng, C., & Liu, C. (2024) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2024.08.010 · Cited by 53
Liu, X., Wu, J., Shao, A., Shen, W., Ye, P., Wang, Y., Ye, J., Jin, K., & Yang, J. (2024) · Journal of Medical Internet Research · doi:10.2196/51926 · Cited by 52
“While numerous studies have demonstrated promising outcomes in complex medical tasks, including the United States Medical Licensing Exam (USMLE) [17,43], simplifying imaging reports for patients [27] and aiding decision-making [44,45], it is crucial to note that ChatGPT exhibits certain limitations.”
Sahu, P. K., Benjamin, L. A., Singh Aswal, G., & Williams-Persad, A. (2023) · Postgraduate Medical Journal · doi:10.1093/postmj/qgad090 · Cited by 52
Analyzing evaluation methods for large language models in the medical field: a scoping review
Lee, J., Park, S., Shin, J., & Cho, B. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02709-7 · Cited by 52
Russe, M. F., Reisert, M., Bamberg, F., & Rau, A. (2024) · RöFo - Fortschritte auf dem Gebiet der Röntgenstrahlen und der bildgebenden Verfahren · doi:10.1055/a-2264-5631 · Cited by 52
Abi-Rafeh, J., Xu, H. H., Kazan, R., Tevlin, R., & Furnas, H. (2024) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad260 · Cited by 49
Assessment of ChatGPT’s performance on neurology written board examination questions
Chen, T. C., Multala, E., Kearns, P., Delashaw, J., Dumont, A., Maraganore, D., & Wang, A. (2023) · BMJ Neurology Open · doi:10.1136/bmjno-2023-000530 · Cited by 51
Su, M.-C., Lin, L.-E., Lin, L.-H., & Chen, Y. (2024) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2024.104717 · Cited by 54
Academic Surgery in the Era of Large Language Models: A Review
Rengers, T. A., Thiels, C. A., & Salehinejad, H. (2024) · JAMA Surgery · doi:10.1001/jamasurg.2023.6496 · Cited by 62
Generative artificial intelligence in surgery
Rodler, S., Ganjavi, C., BACKER, P. D., Magoulianitis, V., Ramacciotti, L. S., Abreu, A. L., Gill, I. S., & Cacciamani, G. E. (2024) · Surgery · doi:10.1016/j.surg.2024.02.019 · Cited by 59
Fatima, S. S., Sheikh, N. A., & Osama, A. (2024) · Postgraduate Medical Journal · doi:10.1093/postmj/qgae088 · Cited by 46
Tang, T., Sha, J., Zhao, Y., Wang, S., Wang, Z., & Shen, S. (2024) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2024.101607 · Cited by 52
Suh, P. S., Shim, W. H., Suh, C. H., Heo, H., Park, K. J., Kim, P. H., Choi, S. J., Ahn, Y., Park, S., Park, H. Y., et al. (2024) · Radiology · doi:10.1148/radiol.241668 · Cited by 62
Using Generative AI in nursing education: Students’ perceptions
Khlaif, Z. N., Salameh, N., Ajouz, M., Mousa, A., Itmazi, J. A., Alwawi, A. A., & Alkaissi, A. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07416-z · Cited by 63
Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks
Kang, M., Lee, S., Baek, J., Kawaguchi, K., & Hwang, S. J. (2023) · neural information processing systems · doi:10.52202/075280-2109 · Cited by 133
Automatic Interactive Evaluation for Large Language Models with State Aware Patient SimulatorPreprint
Liao, Y., Meng, Y., Wang, Y., Li, L., Liu, H., Wang, Y., & Wang, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4890649 · Cited by 58
Low, Y. S., Jackson, M. L., Hyde, R. J., Brown, R. E., Sanghavi, N. M., Baldwin, J. D., Pike, C. W., Muralidharan, J., Hui, G., Alexander, N., et al. (2025) · DIGITAL HEALTH · doi:10.1177/20552076251348850 · Cited by 51
“12,13 Large language models have displayed impressive performance summarizing relevant literature 14 – 16 and responding to natural language queries.”
Cross, J., Robinson, R., Devaraju, S., Vaughans, A., Hood, R., Kayalackakom, T., Honnavar, P., Naik, S., & Sebastian, R. (2023) · Cureus · doi:10.7759/cureus.41399 · Cited by 38
“5, an LLM developed by OpenAI (San Francisco CA) which features 175 billion parameters and leverages extensive data from the internet [2].”
BackgroundInfluentialBrameier, D. T., Alnasser, A. A., Carnino, J. M., Bhashyam, A. R., von Keudell, A. G., & Weaver, M. J. (2023) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.23.00473 · Cited by 46
Exploring the Ethical, Legal, and Social Implications of ChatGPT in Urology
Adhikari, K., Naik, N., Hameed, B. M. Z., Raghunath, S. K., & Somani, B. K. (2023) · Current Urology Reports · doi:10.1007/s11934-023-01185-2 · Cited by 57
Gajjar, A. A., Kumar, R. P., Paliwoda, E. D., Kuo, C. C., Adida, S., Legarreta, A. P., Deng, H., Anand, S. K., Hamilton, D. K., Buell, T. J., et al. (2024) · Neurosurgery · doi:10.1227/neu.0000000000002856 · Cited by 57
Is ChatGPT a trusted source of information for total hip and knee arthroplasty patients?
Wright, B. M., Bodnar, M. S., Moore, A. D., Maseda, M., Kucharik, M. P., Diaz, C. C., Schmidt, C. M., & Mir, H. R. (2024) · Bone & Joint Open · doi:10.1302/2633-1462.52.bjo-2023-0113.r1 · Cited by 54
“This includes passing the USA Medical Licensing Examinations, 6 and generating diagnostically accurate differential diagnoses from clinical cases.”
MethodRosenberg, G. S., Magnéli, M., Barle, N., Kontakis, M. G., Müller, A. M., Wittauer, M., Gordon, M., & Brodén, C. (2024) · Acta Orthopaedica · doi:10.2340/17453674.2024.40182 · Cited by 53
Evaluation of ChatGPT’s responses to information needs and information seeking of dementia patients
Saeidnia, H. R., Kozak, M., Lund, B. D., & Hassanzadeh, M. (2024) · Scientific Reports · doi:10.1038/s41598-024-61068-5 · Cited by 55
Li, B., Gross, O., Crampton, N., Kapoor, M., Tauseef, S., Jain, M., Truong, K. N., & Mariakakis, A. T. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641913 · Cited by 61
AI-empowered scale development: Testing the potential of ChatGPT
Hoffmann, S., Lasarov, W., & Dwivedi, Y. K. (2024) · Technological forecasting & social change · doi:10.1016/j.techfore.2024.123488 · Cited by 42
Bevilacqua, M., Oketch, K., Qin, R., Stamey, W., Zhang, X., Gan, Y., Yang, K., & Abbasi, A. (2025) · ACM Transactions on Information Systems · doi:10.1145/3702639 · Cited by 56
“In taking the United States Medical Licensing Exam (USMLE), which consists of three separate tests, ChatGPT performed ”at or near the passing threshold” and displayed both consistency among its responses and comprehension of the presented topic [45].”
Status and perceptions of ChatGPT utilization among medical students: a survey-based study
Hu, N., Jiang, X. L., Wang, Y. D., Kang, Y. M., Xia, Z., Chen, H. H., Duan, S. N., & Chen, D. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07438-7 · Cited by 69
“Although, ChatGPT offers a personalized learning experience for medical students and residents, providing a secure environment with immediate feedback [7].”
A large language model improves clinicians’ diagnostic performance in complex critical illness cases
Wu, X. C., Huang, Y., & He, Q. (2025) · Critical Care · doi:10.1186/s13054-025-05468-7 · Cited by 66
“Recent studies demonstrated that LLMs without specialized training could pass all three stages of the United States Medical Licensing Exam (USMLE) [3].”
A Ghostwriter for the Masses: ChatGPT and the Future of Writing
Zimmerman, A. (2023) · Annals of Surgical Oncology · doi:10.1245/s10434-023-13436-0 · Cited by 44
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v2 · Cited by 142
Hsu, H.-Y., Hsu, K.-C., Hou, S.-Y., Wu, C.-L., Hsieh, Y.-W., & Cheng, Y.-D. (2023) · JMIR Medical Education · doi:10.2196/48433 · Cited by 50
Hu, J.-M., Liu, F.-C., Chu, C.-M., & Chang, Y.-T. (2023) · Journal of Medical Internet Research · doi:10.2196/49385 · Cited by 50
Matsui, K., Utsumi, T., Aoki, Y., Maruki, T., Takeshima, M., & Takaesu, Y. (2024) · Journal of Medical Internet Research · doi:10.2196/52758 · Cited by 50
Chen, Y., Huang, X., Yang, F., Lin, H., Lin, H., Zheng, Z., Liang, Q., Zhang, J., & Li, X. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06309-x · Cited by 50
Szabó, A. (2023) · Baltic Journal of Sport and Health Sciences · doi:10.33607/bjshs.v1i128.1341 · Cited by 54
Lower, K., Seth, I., Lim, B., & Seth, N. (2023) · Indian Journal of Orthopaedics · doi:10.1007/s43465-023-00967-7 · Cited by 56
“11 Pulmonary embolism information to assist with clinical decision-making [7].”
BackgroundEbrahimian, M., Behnam, B., Ghayebi, N., & Sobhrakhshankhah, E. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100815 · Cited by 47
“9 Additionally, multiple studies have assessed the performance of public AI language models, such as ChatGPT, in medical licensing examinations, particularly in various specialties and subspecialties.”
Background
Metadata
- Authors
- Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-02-09
- Volume
- 2
- Issue
- 2
- Pages
- e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.