Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Open accessPublished version of a preprint
The DOI matches its Crossref record, with a small difference in title or year; 28 of the 28 references it cites were found in a registry or an index.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Kung et al. (2023) evaluated the performance of ChatGPT on the United States Medical Licensing Exam and demonstrated that the chatbot did not reach the passing threshold and achieved an overall accuracy of only 50% (Kung et al., 2023).”
From A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing (2025) · cites this paper as Kung et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [17]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI's capabilities, such as passing the United States Medical Licensing Exam [107], completing exams at the University of Minnesota Law School [108], and solving queries from Wharton School of Business tests [109], underscore its potential.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [107]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The HCI community has explored the application of LLMs for various healthcare practices, such as mental health support [66,68], health information seeking [81,117,128], and public health interventions [58]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
From DIALOGUE: A Generative AI-Based Pre–Post Simulation Study to Enhance Diagnostic Communication in Medical Students Through Virtual Type 2 Diabetes Scenarios (2025) · cites this paper as Kung et al., 2023
Experimental — accuracy not yet measured
52 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background
Cited by
Items 101–200 of 1,129
Májovský, M., Černý, M., Kasal, M., Komarc, M., & Netuka, D. (2023) · Journal of Medical Internet Research · doi:10.2196/46924 · Cited by 301
“Interestingly, Kung et al [14] evaluated the performance of ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of 3 exams: Step 1, Step 2 Clinical Knowledge, and Step 3.”
PMC-LLaMA: toward building open-source language models for medicine
Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae045 · Cited by 300
Sezgin, E. (2023) · DIGITAL HEALTH · doi:10.1177/20552076231186520 · Cited by 261
“10,11 The increased capabilities also spread concerning conversations.”
Essel, H. B., Vlachopoulos, D., Essuman, A. B., & Amankwa, J. O. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100198 · Cited by 296
Generative AI: A systematic review using topic modelling techniques
Gupta, P., Ding, B., Guan, C., & Ding, D. (2024) · Data and Information Management · doi:10.1016/j.dim.2024.100066 · Cited by 218
ChatGPT: Vision and challenges
Gill, S. S., & Kaur, R. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.05.004 · Cited by 224
“This section investigates these challenges and looks ahead to the potential of ChatGPT in the scientific community in the form of future trends [8, 11, 17, 25, 26, 29, 33, 34, 37, 38, 41, 45, 46, 47].”
BackgroundMethodYu, P., Xu, H., Hu, X., & Deng, C. (2023) · Healthcare · doi:10.3390/healthcare11202776 · Cited by 257
“A key application area of LLMs is medical communication and education for healthcare and medical student, staff, and consumers alike [6,15,16,46].”
BackgroundResultInfluentialAssessment of Artificial Intelligence Chatbot Responses to Top Searched Queries About Cancer
Pan, A., Musheyev, D., Bockelman, D., Loeb, S., & Kabarriti, A. E. (2023) · JAMA Oncology · doi:10.1001/jamaoncol.2023.2947 · Cited by 271
Artificial Intelligence–Generated Draft Replies to Patient Inbox Messages
Garcia, P., Ma, S. P., Shah, S., Smith, M., Jeong, Y., Devon-Sand, A., Tai-Seale, M., Takazawa, K., Clutter, D., Vogt, K., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.3201 · Cited by 266
A systematic review of large language models and their implications in medical education
Lucas, H. C., Upperman, J. S., & Robinson, J. R. (2024) · Medical Education · doi:10.1111/medu.15402 · Cited by 287
Artificial intelligence in surgery
Varghese, C., Harrison, E. M., O’Grady, G., & Topol, E. J. (2024) · Nature Medicine · doi:10.1038/s41591-024-02970-3 · Cited by 287
“These models have attained high scores on medical entrance exams 69,70 and contextualized complex information as competently as surgeons 71 , and there is the potential for patients to interact with them as an initial clinical contact point 72,73 .”
Prompting Higher Education Towards AI-Augmented Teaching and Learning Practice
Eager, B., & Brunton, R. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.5.02 · Cited by 287
“Resistance likely has origins in concerns that AI tools can produce essays, answer questions to exams, and augment student capability to achieve an advantage when completing any form of assessment task involving computer-based work (Cotton et al., 2023; Kung et al., 2023).”
BackgroundRaman, R., Mandal, S., Das, P., Kaur, T., Sanjanasri, J. P., & Nedungadi, P. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/3085910 · Cited by 214
Draxler, F., Werner, A., Lehmann, F., Hoppe, M., Schmidt, A., Buschek, D., & Welsch, R. (2023) · arXiv · doi:10.48550/arxiv.2303.03283 · Cited by 284
Academic integrity and artificial intelligence: is ChatGPT hype, hero or heresy?
Currie, G. M. (2023) · Seminars in Nuclear Medicine · doi:10.1053/j.semnuclmed.2023.04.008 · Cited by 283
Generative artificial intelligence (ChatGPT): Implications for management educators
Ratten, V., & Jones, P. (2023) · The International Journal of Management Education · doi:10.1016/j.ijme.2023.100857 · Cited by 214
Use of ChatGPT at University as a Tool for Complex Thinking: Students’ Perceived Usefulness
Romero-Rodríguez, J.-M., Ramírez-Montoya, M.-S., Buenestado-Fernández, M., & Lara-Lara, F. (2023) · Journal of New Approaches in Educational Research · doi:10.7821/naer.2023.7.1458 · Cited by 277
“This result is explained by the complexity of the language model developed (Kung et al., 2023), giving rise to the ease of access and use, the variety of brief and concrete responses, the number of information sources, user confidentiality, and compatibility with other devices, among others…”
BackgroundAssessing GPT-4 for cell type annotation in single-cell RNA-seq analysis
Hou, W., & Ji, Z. (2024) · Nature Methods · doi:10.1038/s41592-024-02235-4 · Cited by 277
“Recent studies have demonstrated the competitive performance of GPT models in answering biomedical questions 29–32 .”
ChatGPT passing USMLE shines a spotlight on the flaws of medical education
Mbakwe, A. B., Lourentzou, I., Celi, L. A., Mechanic, O. J., & Dagan, A. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000205 · Cited by 273
“What does this kind of performance mean for us? [2] As we learn more about its capabilities, we must consider what ChatGPT’s success on a medical examination says about testing and current medical education.”
BackgroundPerformance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations
Ali, R., Tang, O. Y., Connolly, I. D., Zadnik Sullivan, P. L., Shin, J. H., Fridley, J. S., Asaad, W. F., Cielo, D., Oyelese, A. A., Doberstein, C. E., et al. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002632 · Cited by 273
Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023) · JMIR Medical Education · doi:10.2196/48002 · Cited by 271
Transformers and large language models in healthcare: A review
Nerella, S., Bandyopadhyay, S., Zhang, J., Contreras, M., Siegel, S., Bumin, A., Silva, B., Sena, J., Shickel, B., Bihorac, A., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102900 · Cited by 197
“Scaled-up models such as ChatGPT, PaLM, PALM2, and GPT-4 have been shown to answer medical questions and successfully pass or achieve near-passing scores on medical licensing examinations [43, 46-49].”
ChatGPT and Software Testing Education: Promises & Perils
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023) · 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw58534.2023.00078 · Cited by 270
“ChatGPT produces responses that are at or near the passing threshold for all three parts of the US Medical Licensing Exam without any additional information or prompt engineering [26].”
BackgroundResultA Bibliometric Review of Large Language Models Research from 2017 to 2023
Fan, L., Li, L., Ma, Z., Lee, S., Yu, H., & Hemphill, L. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3664930 · Cited by 196
“They are also demonstrated to have the potential to provide AI-assisted medical education (Kung et al., 2023).”
Sai, S., Gaur, A., Sai, R., Chamola, V., Guizani, M., & Rodrigues, J. J. P. C. (2024) · IEEE Access · doi:10.1109/access.2024.3367715 · Cited by 268
Prompt Engineering in Medical Education
Heston, T., & Khun, C. (2023) · International Medical Education · doi:10.3390/ime2030019 · Cited by 232
“These results suggest that GLMs have significant potential to assist with medical education and even potentially aid in clinical decision-making [5,6].”
BackgroundMirza, A., Alampara, N., Kunchapu, S., Ríos-García, M., Emoekabu, B., Krishnan, A., Gupta, T., Schilling-Wilhelmi, M., Okereke, M., Aneesh, A., et al. (2025) · Nature Chemistry · doi:10.1038/s41557-025-01815-x · Cited by 380
“Aggressive scaling of these models has led to a rapid increase in their capabilities 1,2 , with the leading models now being able to pass the US Medical Licensing Examination 3 or other professional licensing exams.”
Savage, T., Nayak, A., Gallo, R., Rangan, E., & Chen, J. H. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01010-1 · Cited by 265
“Within medicine, initial efforts have demonstrated that LLMs can write clinical notes 3 , pass standardized medical exams 4 , and draft responses to patient questions 5,6 .”
BackgroundLarge AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 233
“The generative capability of such LAMs may augment medical student learning and help them gain additional insights from AI-generated content as recently pointed out in [131].”
BackgroundAlawida, M., Mejri, S., Mehmood, A., Chikhaoui, B., & Isaac Abiodun, O. (2023) · Information · doi:10.3390/info14080462 · Cited by 210
“ChatGPT performance was evaluated in different domains such as ophthalmology [64], medicine [65], programming [66], etc.”
BackgroundEvaluating ChatGPT as an Adjunct for Radiologic Decision-Making
Rao, A., Kim, J., Kamineni, M., Pang, M., Lie, W., & Succi, M. D. (2023) · doi:10.1101/2023.02.02.23285399 · Cited by 257
Karataş, F., Abedi, F. Y., Ozek Gunyel, F., Karadeniz, D., & Kuzgun, Y. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12574-6 · Cited by 252
Use of Artificial Intelligence Chatbots for Cancer Treatment Information
Chen, S., Kann, B. H., Foote, M. B., Aerts, H. J. W. L., Savova, G. K., Mak, R. H., & Bitterman, D. S. (2023) · JAMA Oncology · doi:10.1001/jamaoncol.2023.2954 · Cited by 251
Large language models in medical and healthcare fields: applications, advances, and challenges
Wang, D., & Zhang, S. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10921-0 · Cited by 246
“5 The workflow of generating results (Kung et al. 2023) authentic test questions and found that the USMLE test questions and answers generated by GPT-4 were not significantly different from those crafted by human physicians, and the generated questions and answers were deemed highly effective.”
A comprehensive survey of ChatGPT: Advancements, applications, prospects, and challenges
Nazir, A., & Wang, Z. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100022 · Cited by 245
Akinci D’Antonoli, T., Stanzione, A., Bluethgen, C., Vernuccio, F., Ugga, L., Klontzas, M. E., Cuocolo, R., Cannella, R., & Koçak, B. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2023.232417 · Cited by 227
“49 Second, LLMs can rapidly develop specialized knowledge for different medical disciplines and generate answers to clinical questions by analyzing large amounts of medical data, and with Technique: Transabdominal ultrasonography.”
BackgroundMoulaei, K., Yadegari, A., Baharestani, M., Farzanbakhsh, S., Sabet, B., & Reza Afrash, M. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105474 · Cited by 244
Fine-tuning ChatGPT for automatic scoring
Latif, E., & Zhai, X. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100210 · Cited by 179
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 633
Liu, M., Okuhara, T., Chang, X., Shirabe, R., Nishiie, Y., Okada, H., & Kiuchi, T. (2024) · Journal of Medical Internet Research · doi:10.2196/60807 · Cited by 217
“Regarding the accuracy of ChatGPT on MCQs, while 2 previous studies conducted meta-analyses that yielded accuracy rates of 61% and 56%, respectively, we noted that these accuracy rates reflected the performance of all versions of ChatGPT without differentiation by version [54,56].”
Wang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E., Malin, B., & Yin, Z. (2024) · Journal of Medical Internet Research · doi:10.2196/22769 · Cited by 165
An exploratory survey about using ChatGPT in education, healthcare, and research
Hosseini, M., Gao, C. A., Liebovitz, D. M., Carvalho, A. M., Ahmad, F. S., Luo, Y., MacDonald, N., Holmes, K. L., & Kho, A. (2023) · doi:10.1101/2023.03.31.23287979 · Cited by 238
An exploratory survey about using ChatGPT in education, healthcare, and research
Hosseini, M., Gao, C. A., Liebovitz, D. M., Carvalho, A. M., Ahmad, F. S., Luo, Y., MacDonald, N., Holmes, K. L., & Kho, A. (2023) · PLOS ONE · doi:10.1371/journal.pone.0292216 · Cited by 238
“Some educators fed ChatGPT questions from a freely available United States Medical Licensing Examination (USMLE) and reported a near or at passing range performance [3].”
La nueva realidad de la educación ante los avances de la inteligencia artificial generativa
García Peñalvo, F. J., Llorens-Largo, F., & Vidal, J. (2023) · RIED-Revista Iberoamericana de Educación a Distancia · doi:10.5944/ried.27.1.37716 · Cited by 238
ChatGPT: The cognitive effects on learning and memory
Bai, L., Liu, X., & Su, J. (2023) · Brain‐X · doi:10.1002/brx2.30 · Cited by 180
Artificial Intelligence for Drug Discovery: Are We There Yet?
Hasselgren, C., & Oprea, T. I. (2024) · Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-040323-040828 · Cited by 236
Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study
Sallam, M., Salim, N. A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A. B., Malaeb, D., Hallit, R., & Hallit, S. (2023) · JMIR Medical Education · doi:10.2196/48254 · Cited by 234
Suárez, A., Díaz‐Flores García, V., Algar, J., Gómez Sánchez, M., Llorente de Pedro, M., & Freire, Y. (2024) · International Endodontic Journal · doi:10.1111/iej.13985 · Cited by 200
The Role of Large Language Models in Medical Education: Applications and Implications
Safranek, C. W., Sidamon-Eristoff, A. E., Gilson, A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/50945 · Cited by 233
“Recent research using multiple-choice questions sourced from the United States Medical Licensing Exam as a proxy for medical knowledge found that ChatGPT could approximate the performance of a third-year medical student [22,23].”
MethodLai, V. D., Ngo, N. T., Veyseh, A. P. B., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. H. (2023) · arXiv · doi:10.48550/arxiv.2304.05613 · Cited by 233
“…impacts and risks of the technology for the broader society, e.g., in education (Susnjak, 2022; Khalil and Er, 2023), law (Choi et al., 2023), medical (Kung et al., 2022), ethnics (Shen et al., 2023), human-computer collaboration (Lanzi and Loia-cono, 2023), and cognition (Mahowald et al., 2023).”
BackgroundInfluentialLai, V., Ngo, N., Pouran Ben Veyseh, A., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.878 · Cited by 233
Design Principles for Generative AI Applications
Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R., & Geyer, W. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642466 · Cited by 232
Tsai, M.-L., Ong, C. W., & Chen, C.-L. (2023) · Education for Chemical Engineers · doi:10.1016/j.ece.2023.05.001 · Cited by 197
Qureshi, R., Shaughnessy, D., Gill, K. A. R., Robinson, K. A., Li, T., & Agai, E. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02243-z · Cited by 201
Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks
Sandmann, S., Riepenhausen, S., Plagwitz, L., & Varghese, J. (2024) · Nature Communications · doi:10.1038/s41467-024-46411-8 · Cited by 212
“Research has shown that ChatGPT can pass the United States Medical Licensing Exam (USMLE) or Advanced Cardio-vascularLifeSupport(ACLS)exam 12,13 .”
Lin, X. (2024) · Adult Learning · doi:10.1177/10451595231184928 · Cited by 180
“While some (Kung et al., 2023; Cotton et al., 2023) argues that ChatGPT would make traditional online exams outdated and causes concern about the future automation of the teaching profession.”
Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysis
Wei, Q., Yao, Z., Cui, Y., Wei, B., Jin, Z., & Xu, X. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104620 · Cited by 201
Empowering education development through AIGC: A systematic literature review
Chen, X., Hu, Z., & Wang, C. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12549-7 · Cited by 203
Friha, O., Amine Ferrag, M., Kantarci, B., Cakmak, B., Ozgun, A., & Ghoualmi-Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 189
Large language model (ChatGPT) as a support tool for breast tumor board
Sorin, V., Klang, E., Sklair-Levy, M., Cohen, I., Zippel, D. B., Balint Lahat, N., Konen, E., & Barash, Y. (2023) · npj Breast Cancer · doi:10.1038/s41523-023-00557-8 · Cited by 218
“These include the United States Medical Licensing Exam (USMLE) 3 , imaging reports simpli fi cation for patients 4 , decision-making on the appropriate breast imaging examinations 5 , and scienti fi c manuscripts generation 6 .”
MethodDalalah, D., & Dalalah, O. M. A. (2023) · The International Journal of Management Education · doi:10.1016/j.ijme.2023.100822 · Cited by 218
Graphologue: Exploring Large Language Model Responses with Interactive Diagrams
Jiang, P., Rayan, J., Dow, S. P., & Xia, H. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606737 · Cited by 135
“mains such as programming [21, 67, 82], writing support [25, 35, 92], learning [7, 50, 56], and many others [11, 18, 30, 74], and further promoting the notion of natural language interfaces the HCI com-”
BackgroundShahriar, S., Lund, B. D., Mannuru, N. R., Arshad, M. A., Hayawi, K., Bevara, R. V. K., Mannuru, A., & Batool, L. (2024) · Applied Sciences · doi:10.3390/app14177782 · Cited by 156
Homolak, J. (2023) · Croatian Medical Journal · doi:10.3325/cmj.2023.64.1 · Cited by 208
Can ChatGPT pass the life support exams without entering the American heart association course?
Fijačko, N., Gosak, L., Štiglic, G., Picard, C. T., & John Douma, M. (2023) · Resuscitation · doi:10.1016/j.resuscitation.2023.109732 · Cited by 206
Impact of ChatGPT on medical chatbots as a disruptive technology
Chow, J. C. L., Sanders, L., & Li, K. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1166014 · Cited by 188
“Recently, ChatGPT has been reported to be capable of passing the gold-standard US medical exam, suggesting that is has potentially significant applications in the field of medicine (Kung et al., 2023).”
BackgroundThe Potential and Concerns of Using AI in Scientific Research: ChatGPT Performance Evaluation
Khlaif, Z. N., Mousa, A., Hattab, M. K., Itmazi, J., Hassan, A. A., Sanmugam, M., & Ayyoub, A. (2023) · JMIR Medical Education · doi:10.2196/47049 · Cited by 205
“AI-generated text bridges the knowledge gap by providing XSL • FO RenderX accurate and current information from reputable sources, facilitating access to relevant medical literature, research studies, and clinical guidelines [14].”
BackgroundInfluentialAlshami, A., Elsayed, M., Ali, E., Eltoukhy, A. E. E., & Zayed, T. (2023) · Systems · doi:10.3390/systems11070351 · Cited by 202
An evaluation framework for clinical use of large language models in patient interaction tasks
Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03328-5 · Cited by 201
Oh, N., Choi, G.-S., & Lee, W. Y. (2023) · Annals of Surgical Treatment and Research · doi:10.4174/astr.2023.104.5.269 · Cited by 201
Defending ChatGPT against jailbreak attack via self-reminders
Xie, Y., Yi, J., Shao, J., Curl, J., Lyu, L., Chen, Q., Xie, X., & Wu, F. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00765-8 · Cited by 149
“The remarkable success of ChatGPT 1 spans a wide range of applications, and it has amassed a rapidly expanding user base 2–4 .”
BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical TextPreprint
Bolton, E., Venigalla, A., Yasunaga, M., Hall, D., Xiong, B., Lee, T., Daneshjou, R., Frankle, J., Liang, P., Carbin, M., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.18421 · Cited by 276
“Large, general models like GPT-4 and Med-PaLM 2 have set new standards for performance on question-answering and information extraction (Kung et al., 2022; Singhal et al., 2023a,b), but there are several drawbacks to these models.”
MethodLevine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00097-9 · Cited by 199
Wang, M., Wang, M., Xu, X., Yang, L., Cai, D., & Yin, M. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3324714 · Cited by 156
“However, when applied in fields, such as law [33], [48] and medical education [49], [50], its performance is about average.”
BackgroundMassey, P. A., Montgomery, C., & Zhang, A. S. (2023) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-23-00396 · Cited by 190
ChatGPT: Challenges and Benefits in Software Programming for Higher Education
Silva, C. A. G. D., Ramos, F. N., de Moraes, R. V., & Santos, E. L. D. (2024) · Sustainability · doi:10.3390/su16031245 · Cited by 149
“ChatGPT is a versatile tool that is being increasingly used in education across a variety of fields, including dentistry [68], medicine [10,52,69–71], chemistry [72,73], education science [74], journalism and media [75], and many others.”
MethodCheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T.-H. (2023) · PLOS ONE · doi:10.1371/journal.pone.0290691 · Cited by 195
Miao, J., Thongprayoon, C., Suppadungsuk, S., Garcia Valencia, O. A., & Cheungpasitporn, W. (2024) · Medicina · doi:10.3390/medicina60030445 · Cited by 164
Sohail, S. S., Farhat, F., Himeur, Y., Nadeem, M., Madsen, D. Ø., Singh, Y., Atalla, S., & Mansoor, W. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101675 · Cited by 192
“With the aid of this information, ChatGPT will be better able to provide customers in the medical industry with personalized answers by better comprehending the linguistic conventions and terminologies used there (Eysenbach et al., 2023; Kung et al., 2023).”
The clinicians’ guide to large language models: a general perspective with a focus on hallucinations
Roustan, D., & Bastardot, F. (2025) · Interactive Journal of Medical Research · doi:10.2196/59823 · Cited by 191
“It can also be an interesting tool for creating a dynamic learning experience [6].”
Temsah, O., Khan, S. A., Chaiah, Y., Senjab, A., Alhasan, K., Jamal, A., Aljamaan, F., Malki, K. H., Halwani, R., Al-Tawfiq, J. A., et al. (2023) · Cureus · doi:10.7759/cureus.37281 · Cited by 179
“This effect could not only possibly limit its capabilities and could produce counterfactual results [43, 44].”
BackgroundChatGPT in Computer Science Curriculum Assessment: An analysis of Its Successes and Shortcomings
Qureshi, B. (2023) · 2023 9th International Conference on e-Society e-Learning and e-Technologies · doi:10.1145/3613944.3613946 · Cited by 109
Better by You, better than Me? ChatGPT-3 as writing assistance in students’ essays.
Bašić, Ž., Banovac, A., Kružić, I., & Jerković, I. (2023) · doi:10.35542/osf.io/n5m7s · Cited by 189
Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., Afvari, S., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-50043-3 · Cited by 189
“Over the past few months, the fi eld of large language models (LLMs) has seen signi fi cant advancements 40,41 , offering remarkable language comprehension abilities and the potential to perform complex linguistic tasks.”
Momenaei, B., Wakabayashi, T., Shahlaee, A., Durrani, A. F., Pandit, S. A., Wang, K., Mansour, H. A., Abishek, R. M., Xu, D., Sridhar, J., et al. (2023) · Ophthalmology Retina · doi:10.1016/j.oret.2023.05.022 · Cited by 178
Albayati, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100203 · Cited by 188
Hadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 103
Applicability of ChatGPT in Assisting to Solve Higher Order Problems in Pathology
Sinha, R. K., Deb Roy, A., Kumar, N., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.35237 · Cited by 187
“Furthermore, ChatGPT displayed comprehensible reasoning and valid clinical insights in the responses [14].”
BackgroundThirunavukarasu, A. J., Hassan, R., Mahmood, S., Sanghera, R., Barzangi, K., El Mukashfi, M., & Shah, S. (2023) · JMIR Medical Education · doi:10.2196/46599 · Cited by 187
“The ability of ChatGPT to accurately answer questions, provide useful advice, and triage based on clinical vignettes consistently exceeds that of a layperson [5,18].”
BackgroundDraxler, F., Werner, A., Lehmann, F., Hoppe, M., Schmidt, A., Buschek, D., & Welsch, R. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3637875 · Cited by 186
Exploring ChatGPT Capabilities and Limitations: A Survey
Koubaa, A., Boulila, W., Ghouti, L., Alzahem, A., & Latif, S. (2023) · IEEE Access · doi:10.1109/access.2023.3326474 · Cited by 97
Vargas-Murillo, A. R., Pari-Bedoya, I. N. M. D. L. A., & Guevara-Soto, F. D. J. (2023) · International Journal of Learning, Teaching and Educational Research · doi:10.26803/ijlter.22.7.7 · Cited by 141
ChatGPT is a Remarkable Tool—For Experts
Azaria, A., Azoulay, R., & Reches, S. (2024) · Data Intelligence · doi:10.1162/dint_a_00235 · Cited by 130
“[44], ChatGPT’s aptitude in processing intricate medical and clinical information resulted in a commendable performance on the United States Medical Licensing Examination (USMLE), often surpassing the passing threshold.”
BackgroundA generalist medical language model for disease diagnosis assistance
Liu, X., Liu, H., Yang, G., Jiang, Z., Cui, S., Zhang, Z., Wang, H., Tao, L., Sun, Y., Song, Z., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03416-6 · Cited by 183
“GPT-4o is the latest version of ChatGPT, which is reported to have a broader knowledge base and enhanced problem-solving abilities, showing promise in diagnostic tasks.”
A study on ChatGPT for Industry 4.0: Background, potentials, challenges, and eventualities
Javaid, M., Haleem, A., & Singh, R. P. (2023) · Journal of Economy and Technology · doi:10.1016/j.ject.2023.08.001 · Cited by 138
Temsah, M.-H., Aljamaan, F., Malki, K. H., Alhasan, K., Altamimi, I., Aljarbou, R., Bazuhair, F., Alsubaihin, A., Abdulmajeed, N., Alshahrani, F. S., et al. (2023) · Healthcare · doi:10.3390/healthcare11131812 · Cited by 179
“Furthermore, ChatGPT demonstrated that it is capable of passing the United States Medical Licensing Exam (USMLE) [17].”
BackgroundInfluentialAn empirical investigation of the impact of ChatGPT on creativity
Lee, B. C., & Chung, J. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01953-1 · Cited by 179
Raj, R., Singh, A., Kumar, V., & Verma, P. (2023) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.1016/j.tbench.2023.100140 · Cited by 211
Yan, L., Martinez-Maldonado, R., & Gasevic, D. (2024) · Proceedings of the 14th Learning Analytics and Knowledge Conference · doi:10.1145/3636555.3636856 · Cited by 102
Kavadella, A., Dias da Silva, M. A., Kaklamanos, E. G., Stamatopoulos, V., & Giannakopoulos, K. (2024) · JMIR Medical Education · doi:10.2196/51344 · Cited by 133
“Nevertheless, students thought that the LLM must evolve to provide images, videos, accurate and relevant citations, and browse the internet [31,41,42].”
Influential
Metadata
- Authors
- Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-02-09
- Volume
- 2
- Issue
- 2
- Pages
- e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.