Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Open accessPublished version of a preprint
The DOI matches its Crossref record, with a small difference in title or year; 28 of the 28 references it cites were found in a registry or an index.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Kung et al. (2023) evaluated the performance of ChatGPT on the United States Medical Licensing Exam and demonstrated that the chatbot did not reach the passing threshold and achieved an overall accuracy of only 50% (Kung et al., 2023).”
From A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing (2025) · cites this paper as Kung et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [17]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI's capabilities, such as passing the United States Medical Licensing Exam [107], completing exams at the University of Minnesota Law School [108], and solving queries from Wharton School of Business tests [109], underscore its potential.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [107]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The HCI community has explored the application of LLMs for various healthcare practices, such as mental health support [66,68], health information seeking [81,117,128], and public health interventions [58]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
From DIALOGUE: A Generative AI-Based Pre–Post Simulation Study to Enhance Diagnostic Communication in Medical Students Through Virtual Type 2 Diabetes Scenarios (2025) · cites this paper as Kung et al., 2023
Experimental — accuracy not yet measured
52 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background
Cited by
Items 201–300 of 1,129
Role of activity-based learning and ChatGPT on students' performance in education
Al Shloul, T., Mazhar, T., Abbas, Q., Iqbal, M., Ghadi, Y. Y., Shahzad, T., Mallek, F., & Hamam, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100219 · Cited by 173
“Using the United States Medical Licencing Examination data, the author (Kung et al., 2023) assessed ChatGPT.”
BackgroundMethodChoi, E. P. H., Lee, J. J., Ho, M.-H., Kwok, J. Y. Y., & Lok, K. Y. W. (2023) · Nurse Education Today · doi:10.1016/j.nedt.2023.105796 · Cited by 172
Caruccio, L., Cirillo, S., Polese, G., Solimando, G., Sundaramurthy, S., & Tortora, G. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121186 · Cited by 153
The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies
Swanson, K., Wu, W., Bulaong, N. L., Pak, J. E., & Zou, J. (2025) · Nature · doi:10.1038/s41586-025-09442-9 · Cited by 169
Xu, X., Chen, Y., & Miao, J. (2024) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2024.21.6 · Cited by 169
“Some scholars believe that ChatGPT can be applied to self-directed learning and exam preparation, such as helping students review, facilitating group learning, and creating exam simulation questions [31,32,36,37] (Fig.”
BackgroundSison, A. J. G., Daza, M. T., Gozalo-Brizuela, R., & Garrido-Merchán, E. C. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2225931 · Cited by 169
“Lastly, ChatGPT can also be used, for instance, to imper-sonate young, desirable males or females to ensnare targets, extorting money or information from them (Brewster, 2023).”
Let’s Have a Chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Shahriar, S., & Hayawi, K. (2023) · Artificial Intelligence and Applications · doi:10.47852/bonviewaia3202939 · Cited by 167
“Take on medical exam [23] Asked ChatGPT questions”
BackgroundMethodLet's have a chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Shahriar, S., & Hayawi, K. (2023) · arXiv · doi:10.48550/arxiv.2302.13817 · Cited by 167
“Take on medical exam [23] Asked ChatGPT questions”
BackgroundMethodSng, G. G. R., Tung, J. Y. M., Lim, D. Y. Z., & Bee, Y. M. (2023) · Diabetes Care · doi:10.2337/dc23-0197 · Cited by 159
Wood, D. A., Achhpilia, M. P., Adams, M. T., Aghazadeh, S., Akinyele, K., Akpan, M., Allee, K. D., Allen, A. M., Almer, E. D., Ames, D., et al. (2023) · Issues in Accounting Education · doi:10.2308/issues-2023-013 · Cited by 166
ChatGPT Performs on the Chinese National Medical Licensing Examination
Wang, X., Gong, Z., Wang, G., Jia, J., Xu, Y., Zhao, J., Fan, Q., Wu, S., Hu, W., & Li, X. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01961-0 · Cited by 157
“Based on a three-year assessment, ChatGPT demonstrated higher performance in Unit 4 (51.8%), whereas its results in the other three units were relatively lower (Unit 1 43.3%, Unit 2 36.7%, Unit 3 40.7%), indicating a statistical difference (P = 0.001).”
ResultInfluentialHS Kumar, A. (2023) · Biology, Engineering, Medicine and Science Reports · doi:10.5530/bems.9.1.5 · Cited by 165
Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination
Rosoł, M., Gąsior, J. S., Łaba, J., Korzeniewski, K., & Młyńczak, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-46995-z · Cited by 157
Early applications of ChatGPT in medical practice, education and research
Sedaghat, S. (2023) · Clinical Medicine · doi:10.7861/clinmed.2023-0078 · Cited by 163
Weidener, L., & Fischer, M. (2024) · JMIR Medical Education · doi:10.2196/51247 · Cited by 163
“Furthermore, this could be attributed to the availability of AI-based applications, such as ChatGPT, and increasing opportunities for education on AI in the medical context, as well as AI-based (chat) applications that are knowledgeable in the field of medicine [7,31-33].”
BackgroundLarge language models and their impact in ophthalmology
Betzler, B. K., Chen, H., Cheng, C.-Y., Lee, C. S., Ning, G., Song, S. J., Lee, A. Y., Kawasaki, R., van Wijngaarden, P., Grzybowski, A., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00201-7 · Cited by 161
Newton, P., & Xiromeriti, M. (2024) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2023.2299059 · Cited by 70
“Students could ask ChatGPT to explain the answers it gives, and generate practice test questions (Bommineni et al. 2023; Cuthbert and Simpson 2023; Friederichs, Friederichs and März 2023; Fijačko et al. 2023; Kung et al. 2023) b. Teaching Assistant .”
A Preliminary Investigation of Fake Peer-Reviewed Citations and References Generated by ChatGPT
Day, T. (2023) · The Professional Geographer · doi:10.1080/00330124.2023.2190373 · Cited by 160
“It has quickly become a cultural sensation (Thorp 2023) since it trial launched in November 2022, with, “ far-reaching consequences for science and society ” (van Dis et al. 2023), and with “ remarkable performance ” on medical licensing exams (Kung et al. 2023).”
BackgroundRau, A., Rau, S., Zöller, D., Fink, A., Tran, H., Wilpert, C., Nattenmüller, J., Neubauer, J., Bamberg, F., Reisert, M., et al. (2023) · Radiology · doi:10.1148/radiol.230970 · Cited by 150
ChatGPT and Publication Ethics
Rahimi, F., & Talebi Bezmin Abadi, A. (2023) · Archives of Medical Research · doi:10.1016/j.arcmed.2023.03.004 · Cited by 158
Leveraging ChatGPT to Aid Construction Hazard Recognition and Support Safety Education and Training
Uddin, S. M. J., Albert, A., Ovid, A., & Alsharef, A. (2023) · Sustainability · doi:10.3390/su15097121 · Cited by 158
Knopp, M. I., Warm, E. J., Weber, D., Kelleher, M., Kinnear, B., Schumacher, D. J., Santen, S. A., Mendonça, E., & Turner, L. (2023) · JMIR Medical Education · doi:10.2196/50373 · Cited by 133
“Additionally, GLMs models have demonstrated the ability to pass medical licensing examinations with increasing reliability and in multiple languages [6-11], which implies that their potential in advancing health care cannot be ignored.”
BackgroundMasanneck, L., Schmidt, L., Seifert, A., Kölsche, T., Huntemann, N., Jansen, R., Mehsin, M., Bernhard, M., Meuth, S. G., Böhm, L., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/53297 · Cited by 124
“…many medical professionals, it is remarkable that systems such as ChatGPT without any specialized training have been able to pass US medical exams [5,6]; provide answers to questions from a web-based platform that are preferred over doctors’ replies [7]; and in most cases, offer appropriate…”
Humar, P., Asaad, M., Bengur, F. B., & Nguyen, V. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad130 · Cited by 155
Lum, Z. C. (2023) · Clinical Orthopaedics & Related Research · doi:10.1097/corr.0000000000002704 · Cited by 152
“This is different from other chatbots or domain-speci fi c trained AI that allow online database searches or additional information access [9].”
BackgroundHoch, C. C., Wollenberg, B., Lüers, J.-C., Knoedler, S., Knoedler, L., Frank, K., Cotofana, S., & Alfertshofer, M. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08051-4 · Cited by 154
“Accord-ingly, a recent study evaluating ChatGPT’s performance across all three USMLE steps (namely, Step 1, Step 2CK, and Step 3) revealed a substantial level of agreement and provided valuable insights through the comprehensive explanations generated by ChatGPT [15].”
BackgroundIbrahim, H., Liu, F., Asim, R., Battu, B., Benabderrahmane, S., Alhafni, B., Adnan, W., Alhanai, T., AlShebli, B., Baghdadi, R., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-38964-3 · Cited by 154
“For instance, Kung et al. evaluated the performance of ChatGPT on the United States Medical Licensing Exam, where they show that ChatGPT performed at or near a passing level on all three stages of the test, without the need for additional specialized training or reinforcement 21 .”
Is AI the Future of Mental Healthcare?
Minerva, F., & Giubilini, A. (2023) · Topoi · doi:10.1007/s11245-023-09932-3 · Cited by 182
“Very recently, the development of ChatGPT has shown how AI could have applications in education (Kung et al. 2023) the judicial system (Parikh et al.”
BackgroundAli, K., Barhom, N., Tamimi, F., & Duggal, M. (2024) · European Journal of Dental Education · doi:10.1111/eje.12937 · Cited by 152
Interactive computer-aided diagnosis on medical image using large language models
Wang, S., Zhao, Z., Ouyang, X., Liu, T., Wang, Q., & Shen, D. (2024) · Communications Engineering · doi:10.1038/s44172-024-00271-8 · Cited by 147
Zhu, L., Mou, W., & Chen, R. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04123-5 · Cited by 143
Assessing the capabilities of ChatGPT to improve additive manufacturing troubleshooting
Badini, S., Regondi, S., Frontoni, E., & Pugliese, R. (2023) · Advanced Industrial and Engineering Polymer Research · doi:10.1016/j.aiepr.2023.03.003 · Cited by 134
ChatGPT: promise and challenges for deployment in low- and middle-income countries
Wang, X., Sanders, H. M., Liu, Y., Seang, K., Tran, B. X., Atanasov, A. G., Qiu, Y., Tang, S., Car, J., Wang, Y. X., et al. (2023) · The Lancet Regional Health - Western Pacific · doi:10.1016/j.lanwpc.2023.100905 · Cited by 138
Use of a Large Language Model to Assess Clinical Acuity of Adults in the Emergency Department
Williams, C. Y. K., Zack, T., Miao, B. Y., Sushil, M., Wang, M., Kornblith, A. E., & Butte, A. J. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.8895 · Cited by 148
A comparison of ChatGPT-generated articles with human-written articles
Ariyaratne, S., Iyengar, K. P., Nischal, N., Chitti Babu, N., & Botchu, R. (2023) · Skeletal Radiology · doi:10.1007/s00256-023-04340-5 · Cited by 148
“It has also been successful in clearing USMLE (US Medical Licensing Examination) and certain law exams [6].”
MethodNovel research and future prospects of artificial intelligence in cancer diagnosis and treatment
Zhang, C., Xu, J., Tang, R., Yang, J., Wang, W., Yu, X., & Shi, S. (2023) · Journal of Hematology & Oncology · doi:10.1186/s13045-023-01514-5 · Cited by 148
“[35] evaluated ChatGPT on the United States Medical Licensing Examination (USMLE) and found that ChatGPT performed at or near the passing threshold of 60% accuracy.”
MethodGenerative AI: Implications and Applications for Education
Olga, A., Tzirides, A. O., Saini, A., Zapata, G. C., Searsmith, D., Cope, B., Kalantzis, M., de Castro, V. C., Kourkoulou, T., Jones, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.07605 · Cited by 60
“5 was able to pass the US National Board of Medical Examiners test at the level of a third-year medical student (Gilson et al. 2023, Kung et al. 2023) and GPT-4 can pass the Multistate Bar Examination taken by law students (Katz et al. 2023).”
BackgroundTowards building multilingual language model for medicine
Qiu, P., Wu, C., Zhang, X., Lin, W., Wang, H., Zhang, Y., Wang, Y., & Xie, W. (2024) · Nature Communications · doi:10.1038/s41467-024-52417-z · Cited by 147
“In detail, given GPT-4’s demonstrated capability to outperform human experts by providing detailed explanations in Chain of Thought (CoT) experiments [13], we utilize GPT-4 to generate rationales for our dataset.”
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147
An explorative assessment of ChatGPT as an aid in medical education: Use it with caution
Han, Z., Battaglia, F., Udaiyar, A., Fooks, A., & Terlecky, S. R. (2024) · Medical Teacher · doi:10.1080/0142159x.2023.2271159 · Cited by 131
Prompt Engineering Paradigms for Medical Applications: Scoping Review
Zaghir, J., Naguib, M., Bjelogrlic, M., Névéol, A., Tannier, X., & Lovis, C. (2024) · Journal of Medical Internet Research · doi:10.2196/60501 · Cited by 143
“These papers spanned across various medical fields, with a greater emphasis on clinical (including specialties) rather than biomedical disciplines.”
Scaffolding Computational Thinking With ChatGPT
Liao, J., Zhong, L., Zhe, L., Xu, H., Liu, M., & Xie, T. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3392896 · Cited by 108
“Students who have used ChatGPT are more prone to plagiarism compared with those who have not [19], posing serious challenges to academic integrity and fair learning for students [20].”
BackgroundSavelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141
“Others have used GPT models to solve various MCQ-based exams, including the United States Medical Licensing Examination (USMLE), with accuracy around 50% [13, 22, 27], the Multistate Bar Examination (MBE) [3, 18], and the American Institute of Certified Public Accountants’ (AICPA) Regulation (REG) exam [2].”
MethodPerformance of Generative Artificial Intelligence in Dental Licensing Examinations
Chau, R. C. W., Thu, K. M., Yu, O. Y., Hsung, T.-C., Lo, E. C. M., & Lam, W. Y. H. (2024) · International Dental Journal · doi:10.1016/j.identj.2023.12.007 · Cited by 156
“10,29,30 This study extends these results to dentistry and demonstrates that GenAI can be a valuable tool in dental practice and education.”
Exploring the use of artificial intelligence (AI) in the delivery of effective feedback
Venter, J., Coetzee, S. A., & Schmulian, A. (2024) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2024.2415649 · Cited by 156
“These advantages potentially make LLMs a suitable tool for enhancing textual feedback processes (Kung et al. 2023), particularly in large-scale educational contexts, while freeing up time for instructors to create more innovative lesson plans, engage in professional development, promote research…”
Performance of Generative Large Language Models on Ophthalmology Board–Style Questions
Cai, L. Z., Shaheen, A., Jin, A., Fukui, R., Yi, J. S., Yannuzzi, N., & Alabiad, C. (2023) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2023.05.024 · Cited by 139
Suchman, K., Garg, S., & Trindade, A. J. (2023) · American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002320 · Cited by 139
“Mostrecently,ithas garneredattentionfromthemedicalcommunityforpassingtheUS MedicalLicensingExamination(5)andassessedothermedicalex-aminations(6).”
BackgroundAwareness and level of digital literacy among students receiving health-based education
Aydınlar, A., Mavi, A., Kütükçü, E., Kırımlı, E. E., Alis, D. C., Akın, A., & Altıntaş, L. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05025-w · Cited by 163
“It is seen as a practical online teaching assistant due to its instant responses and online availability [15].”
BackgroundBrügge, E., Ricchizzi, S., Arenbeck, M., Keller, M. N., Schur, L., Stummer, W., Holling, M., Lu, M. H., & Darici, D. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06399-7 · Cited by 137
Automatic assessment of text-based responses in post-secondary education: A systematic review
Gao, R., Merzdorf, H. E., Anwar, S., Hipwell, M. C., & Srinivasa, A. R. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100206 · Cited by 137
ChatGPT in Higher Education and Science: a Threat or a Valuable Resource?
Ivakhnenko, E. N., & Nikolskiy, V. S. (2023) · Высшее образование в России · doi:10.31992/0869-3617-2023-32-4-9-22 · Cited by 199
Evaluating ChatGPT Performance on the Orthopaedic In-Training Examination
Kung, J. E., Marshall, C., Gauthier, C., Gonzalez, T. A., & Jackson, J. B. (2023) · JBJS Open Access · doi:10.2106/jbjs.oa.23.00056 · Cited by 130
Choi, H. S., Song, J. Y., Shin, K. H., Chang, J. H., & Jang, B.-S. (2023) · Radiation Oncology Journal · doi:10.3857/roj.2023.00633 · Cited by 132
“We expect that LLMs specialized in medical tasks may elevate the accuracy of extraction even better [18,19].”
Kim, T. W. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.38 · Cited by 101
“ChatGPT’s performance on the United States Medical Licensing Exam, consisting of Step 1, Step 2 CK, and Step 3, was at or near the cut score for all 3 examinations [6].”
GPT versus Resident Physicians — A Benchmark Based on Official Board Scores
Katz, U., Cohen, E., Shachar, E., Somer, J., Fink, A., Morse, E., Shreiber, B., & Wolf, I. (2024) · NEJM AI · doi:10.1056/aidbp2300192 · Cited by 135
Assessing the utility of ChatGPT throughout the entire clinical workflow
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K. J., & Succi, M. D. (2023) · doi:10.1101/2023.02.21.23285886 · Cited by 134
Concerns with the Usage of ChatGPT in Academia and Medicine: A Viewpoint
Alser, M., & Waisberg, E. (2023) · American Journal of Medicine Open · doi:10.1016/j.ajmo.2023.100036 · Cited by 120
Teaching AI Ethics in Medical Education: A Scoping Review of Current Literature and Practices
Weidener, L., & Fischer, M. (2023) · Perspectives on Medical Education · doi:10.5334/pme.954 · Cited by 133
“Owing to the capabilities of ChatGPT in the medical field, for example, being able to pass the written portion of the US medical licensing examination (USMLE) [11], AI-based chat applications are expected to have a significant impact on medical practice [10, 12].”
BackgroundMedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Tang, X., Zou, A., Zhang, Z., Li, Z., Zhao, Y., Zhang, X., Cohan, A., & Gerstein, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.33 · Cited by 376
Önder, Ç. E., Koç, G., Gökbulut, P., Taşkaldıran, I., & Kuşkonmaz, Ş. M. (2024) · Scientific Reports · doi:10.1038/s41598-023-50884-w · Cited by 135
“In the United States Medical Licensing Examination (USMLE), ChatGPT was found to perform at or close to passing on multiple-choice single answer questions 8 .”
Zhai, X., Nyaaba, M., & Ma, W. (2025) · Science & Education · doi:10.1007/s11191-024-00496-1 · Cited by 87
“Research on GAI and problem-solving explores various aspects of GAI, its applications, challenges, and impact on different areas of life, with a focus on how GAI can be used to solve complex problems by imitating human behavior using advanced algorithms and techniques (Kung et al., 2023).”
AI-based avatars are changing the way we learn and teach: benefits and challenges
Fink, M. C., Robinson, S. A., & Ertl, B. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1416307 · Cited by 195
Alkhaaldi, S. M. I., Kassab, C. H., Dimassi, Z., Oyoun Alsoud, L., Al Fahim, M., Al Hageh, C., & Ibrahim, H. (2023) · JMIR Medical Education · doi:10.2196/51302 · Cited by 132
“Moreover, ChatGPT can be used as a testing tool, providing learners with logical explanations for incorrect responses and allowing them to gain knowledge [8].”
BackgroundMethodChatGPT in radiology: A systematic review of performance, pitfalls, and future perspectives
Keshavarz, P., Bagherieh, S., Nabipoorashrafi, S. A., Chalian, H., Rahsepar, A. A., Kim, G. H. J., Hassani, C., Raman, S. S., & Bedayat, A. (2024) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2024.04.003 · Cited by 132
Nastasi, A. J., Courtright, K. R., Halpern, S. D., & Weissman, G. E. (2023) · Scientific Reports · doi:10.1038/s41598-023-45223-y · Cited by 93
Nakaura, T., Ito, R., Ueda, D., Nozaki, T., Fushimi, Y., Matsui, Y., Yanagawa, M., Yamada, A., Tsuboyama, T., Fujima, N., et al. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-024-01552-0 · Cited by 150
“…data into a matrix using a tokenizer, transforming it into another matrix using a Large Language Model, and then converting it back into output data ◂ (USMLE) revealed that it performed at or near the passing threshold across all three exams without any specialized training or reinforcement [40].”
BackgroundMethodInfluentialQuality of Large Language Model Responses to Radiation Oncology Patient Care Questions
Yalamanchili, A., Sengupta, B., Song, J., Lim, S., Thomas, T. O., Mittal, B. B., Abazeed, M. E., & Teo, P. T. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.4630 · Cited by 123
Artificial intelligence and digital health in global eye health: opportunities and challenges
Tan, T. F., Thirunavukarasu, A. J., Jin, L., Lim, J., Poh, S. S. J., Teo, Z. L., Ang, M., Chan, R. V. P., Ong, J., Turner, A. W., et al. (2023) · The Lancet Global Health · doi:10.1016/s2214-109x(23)00323-6 · Cited by 142
Jo, H. (2023) · Telematics and Informatics · doi:10.1016/j.tele.2023.102067 · Cited by 145
Hirosawa, T., Kawamura, R., Harada, Y., Mizuta, K., Tokumasu, K., Kaji, Y., Suzuki, T., & Shimizu, T. (2023) · JMIR Medical Informatics · doi:10.2196/48808 · Cited by 130
“5) to the fourth-generation GPT (GPT-4), the model’s accuracy has improved in professional examinations [15] and multiple-choice problems across various languages [16].”
BackgroundSallam, M., Salim, N. A., Al-Tammemi, A. B., Barakat, M., Fayyad, D., Hallit, S., Harapan, H., Hallit, R., & Mahafzah, A. (2023) · Cureus · doi:10.7759/cureus.35029 · Cited by 130
“Scientists must be able to assess the correctness of AI-generated medical information and develop credibly, validated information for patients and the general public [4,7].”
BackgroundLukac, S., Dayan, D., Fink, V., Leinert, E., Hartkopf, A., Veselinovic, K., Janni, W., Rack, B., Pfister, K., Heitmeir, B., et al. (2023) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-023-07130-5 · Cited by 130
“Additionally, the ChatGPT can pass the United States Medical Licensing Exam [10].”
BackgroundInfluentialNaidu, K., & Sevnarayan, K. (2023) · Online Journal of Communication and Media Technologies · doi:10.30935/ojcmt/13291 · Cited by 75
Leveraging large language models for generating responses to patient messages—a subjective analysis
Liu, S., McCoy, A. B., Wright, A. P., Carew, B., Genkins, J. Z., Huang, S. S., Peterson, J. F., Steitz, B., & Wright, A. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae052 · Cited by 86
“10 14–16 In particular, a recent study used ChatGPT to generate responses to 195 patient questions from social media forums.”
BackgroundFraiwan, M., & Khasawneh, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.00237 · Cited by 56
“ChatGPT was able to pass bar exams, United States medical licensing exams, and job interviews, among others [1].”
BackgroundTassoti, S. (2024) · Journal of Chemical Education · doi:10.1021/acs.jchemed.4c00212 · Cited by 129
Addressing 6 challenges in generative AI for digital health: A scoping review
Templin, T., Perez, M. W., Sylvia, S., Leek, J., & Sinnott-Armstrong, N. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000503 · Cited by 99
ChatGPT in Medical Imaging Higher Education
Currie, G., Singh, C., Nelson, T., Nabasenja, C., Al-Hayek, Y., & Spuur, K. (2023) · Radiography · doi:10.1016/j.radi.2023.05.011 · Cited by 128
ChatGPT takes on the European Exam in Core Cardiology: an artificial intelligence success story?
Skalidis, I., Cagnina, A., Luangphiphat, W., Mahendiran, T., Muller, O., Abbe, E., & Fournier, S. (2023) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztad029 · Cited by 128
“Recent preliminary studies have shown that ChatGPT can achieve a passing grade in exams such as the United States Medical Licensing Examination and the final exam in an MBA course at the University of Pennsylvania.”
Artificial intelligence in sport management education: Playing the AI game with ChatGPT
Keiper, M. C., Fried, G. B., Lupinek, J. M., & Nordstrom, H. (2023) · Journal of Hospitality Leisure Sport & Tourism Education · doi:10.1016/j.jhlste.2023.100456 · Cited by 149
Toyama, Y., Harigai, A., Abe, M., Nagano, M., Kawabata, M., Seki, Y., & Takase, K. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01491-2 · Cited by 125
“Two LLMs that have demonstrated exceptional performance in the US Medical Licensing Examination (USMLE)- style questions, surpassing the approximate passing threshold of 60%, are GPT-4 and Med-PaLM 2 [13, 16, 17, 21].”
BackgroundAl-Ashwal, F. Y., Zawiah, M., Gharaibeh, L., Abu-Farha, R., & Bitar, A. N. (2023) · Drug Healthcare and Patient Safety · doi:10.2147/dhps.s425858 · Cited by 125
“20 The discrepancies between different LLM chatbots were demonstrated in a study conducted by Raimondi et al on the ophthalmologists’ fellowship exams.”
Wagner, M. W., & Ertl-Wagner, B. B. (2024) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231171125 · Cited by 125
Jeyaraman, M., Ramasubramanian, S., Balaji, S., Jeyaraman, N., Nallakumarasamy, A., & Sharma, S. (2023) · World Journal of Methodology · doi:10.5662/wjm.v13.i4.170 · Cited by 125
Wang, D., Feng, L., Ye, J., Zou, J., & Zheng, Y. (2023) · MedComm – Future Medicine · doi:10.1002/mef2.43 · Cited by 120
Can ChatGPT improve investment decisions? From a portfolio management perspective
Ko, H., & Lee, J. (2024) · Finance research letters · doi:10.1016/j.frl.2024.105433 · Cited by 167
Roles and potential of Large language models in healthcare: A comprehensive review
Lin, C., & Kuo, C. (2025) · Biomedical Journal · doi:10.1016/j.bj.2025.100868 · Cited by 141
“For example, Kung et al. reported that ChatGPT passed the United States Medical Licensing Examination (USMLE) with a score above the passing threshold, suggesting that LLMs possess medical knowledge comparable to medical graduates [19].”
Ayre, J., Mac, O., McCaffery, K., McKay, B. R., Liu, M., Shi, Y., Rezwan, A., & Dunn, A. G. (2024) · Journal of General Internal Medicine · doi:10.1007/s11606-023-08469-w · Cited by 124
“22 Due to ChatGPT’s reliance on human input for training, users should also carefully reflect on its potential to perpetuate biases relating to, e.g. race, age, gender, and ethnicity.”
BackgroundMoshirfar, M., Altaf, A. W., Stoakes, I. M., Tuttle, J. J., & Hoopes, P. C. (2023) · Cureus · doi:10.7759/cureus.40822 · Cited by 124
“It has gained significant recognition in the healthcare field, particularly due to its noteworthy performance in passing all three parts of the United States Medical Licensing Exam (USMLE), achieving scores near or at 60% in each examination [2].”
BackgroundMedical image captioning via generative pretrained transformers
Selivanov, A., Rogov, O. Y., Chesakov, D., Shelmanov, A., Fedulova, I., & Dylov, D. V. (2023) · Scientific Reports · doi:10.1038/s41598-023-31223-5 · Cited by 107
Cadamuro, J., Cabitza, F., Debeljak, Z., De Bruyne, S., Frans, G., Perez, S. M., Ozdemir, H., Tolios, A., Carobene, A., & Padoan, A. (2023) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2023-0355 · Cited by 116
“This chatbot has been demonstrated to pass the United States Medical Licensing Exam (USMLE), a set of three standardized testsofexpert-levelknowledge,whicharerequiredformedical licensure in this country [5].”
MethodLi, S. W., Kemp, M. W., Logan, S. J. S., Dimri, P. S., Singh, N., Mattar, C. N. Z., Dashraath, P., Ramlal, H., Mahyuddin, A. P., Kanayan, S., et al. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.04.020 · Cited by 116
Generative Artificial Intelligence Through ChatGPT and Other Large Language Models in Ophthalmology
Tan, T. F., Thirunavukarasu, A. J., Campbell, J. P., Keane, P. A., Pasquale, L. R., Abramoff, M. D., Kalpathy-Cramer, J., Lum, F., Kim, J. E., Baxter, S. L., et al. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100394 · Cited by 116
ChatGPT and Generative AI Technology: A Mixed Bag of Concerns and New Opportunities
Lambert, J., & Stevens, M. (2024) · Computers in the Schools · doi:10.1080/07380569.2023.2256710 · Cited by 93
“Similarly, Kung et al. (2022) found the application performed at or near the passing threshold for all three portions of the U.S. Medical Licensing Exam.”
Flodén, J. (2025) · British Educational Research Journal · doi:10.1002/berj.4069 · Cited by 82
Guerra, G. A., Hofmann, H., Sobhani, S., Hofmann, G., Gomez, D., Soroudi, D., Hopkins, B. S., Dallas, J., Pangal, D. J., Cheok, S., et al. (2023) · World Neurosurgery · doi:10.1016/j.wneu.2023.08.042 · Cited by 123
ChatGPT interactive medical simulations for early clinical education: Case study
Scherr, R., Halaseh, F. F., Spina, A., Andalib, S., & Rivera, R. (2023) · JMIR Medical Education · doi:10.2196/49877 · Cited by 122
“ChatGPT has already entered the medical education arena, having passed curated, publicly available versions of the United States Medical Licensing Examination (USMLE) Step 1, Step 2 Clinical Knowledge, and Step 3 questions earlier this year [4].”
BackgroundThe Use of ChatGPT to Assist in Diagnosing Glaucoma Based on Clinical Case Reports
Delsoz, M., Raja, H., Madadi, Y., Tang, A. A., Wirostko, B. M., Kahook, M. Y., & Yousefi, S. (2023) · Ophthalmology and Therapy · doi:10.1007/s40123-023-00805-x · Cited by 116
“The capabilities of ChatGPT in responding to multiple-choice questions from the USMLE was investigated and it was reported that ChatGPT not only responded correctly to over 50% of questions but also provided reasonable supporting explanations for the selected choices [22].”
MethodGenerative AI and large language models in health care: pathways to implementation
Raza, M. M., Venkatesh, K. P., & Kvedar, J. C. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00988-4 · Cited by 107
“LLMs have already been used to pass the United States Medical Licensing Examination 4 , write research articles 5 , and interpret electronic medical record data 6 .”
MethodOpportunities, Challenges, and Strategies for Using ChatGPT in Higher Education: A Literature Review
Sok, S., & Heng, K. (2024) · Journal of Digital Educational Technology · doi:10.30935/jdet/14027 · Cited by 120
“…of data derived from multiple internet sources, including websites, journal articles, and textbooks (Adiguzel et al., 2023; Choi et al., 2023; Kung et al., 2023), ChatGPT is capable of generating precise responses to an array of prompts and inquiries instantaneously (Qadir, 2022) and…”
Background
Metadata
- Authors
- Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-02-09
- Volume
- 2
- Issue
- 2
- Pages
- e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.