Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Open accessPublished version of a preprint
The DOI matches its Crossref record, with a small difference in title or year; 28 of the 28 references it cites were found in a registry or an index.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Kung et al. (2023) evaluated the performance of ChatGPT on the United States Medical Licensing Exam and demonstrated that the chatbot did not reach the passing threshold and achieved an overall accuracy of only 50% (Kung et al., 2023).”
From A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing (2025) · cites this paper as Kung et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [17]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI's capabilities, such as passing the United States Medical Licensing Exam [107], completing exams at the University of Minnesota Law School [108], and solving queries from Wharton School of Business tests [109], underscore its potential.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [107]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The HCI community has explored the application of LLMs for various healthcare practices, such as mental health support [66,68], health information seeking [81,117,128], and public health interventions [58]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
From DIALOGUE: A Generative AI-Based Pre–Post Simulation Study to Enhance Diagnostic Communication in Medical Students Through Virtual Type 2 Diabetes Scenarios (2025) · cites this paper as Kung et al., 2023
Experimental — accuracy not yet measured
52 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background
Cited by
Items 901–1,000 of 1,129
Kunze, K. N., Nwachukwu, B. U., Cote, M. P., & Ramkumar, P. N. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.12.010 · Cited by 40
Haag, D., Kumar, D., Gruber, S., Hofer, D. P., Sareban, M., Treff, G., Niebauer, J., Bull, C. N., Schmidt, A., & Smeddinck, J. D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713307 · Cited by 49
Dedicated AI Expert System vs Generative AI With Large Language Model for Clinical Diagnoses
Feldman, M. J., Hoffer, E. P., Conley, J. J., Chang, J. P., Chung, J. A., Jernigan, M. C., Lester, W. T., Strasser, Z. H., & Chueh, H. C. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.12994 · Cited by 42
Jeon, S., & Kim, H. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110614 · Cited by 56
Public use of a generalist LLM chatbot for health queries
Costa-Gomes, B., Tolmachev, P., Taysom, E., Sounderajah, V., Richardson, H. B., Schoenegger, P., Liu, X., Nour, M. M., Spielman, S., Way, S. F., et al. (2026) · Nature Health · doi:10.1038/s44360-026-00117-x · Cited by 59
“LLMs perform competitively on medical licensing examinations 6–8 and clinical reasoning benchmarks 9–11 .”
Beyond open book versus closed book: a taxonomy of restrictions in online examinations
Dawson, P., Nicola‐Richmond, K., & Partridge, H. (2023) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2023.2209298 · Cited by 42
“Spell checking tools might be allowed in one online examination, but not the ChatgPT generative artificial intelligence, which is claimed in one recent paper (Kung et al. 2022) to be capable of satisfying much of the requirements of the united States medical licensing Examination.”
BackgroundOrganizational preparedness for the use of large language models in pathology informatics
Hart, S. N., Hoffman, N. G., Gershkovich, P., Christenson, C., McClintock, D. S., Miller, L. J., Jackups, R., Azimi, V., Spies, N., & Brodsky, V. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100338 · Cited by 36
Thomae, A. V., Witt, C. M., & Barth, J. (2024) · JMIR Medical Education · doi:10.2196/50545 · Cited by 36
“Besides endeavors such as exams in the field of medicine [1,2], there are many other opportunities to implement ChatGPT in medical education [3,4].”
BackgroundDorta-González, P., López-Puig, A. J., Dorta-González, M. I., & González-Betancor, S. M. (2024) · Telematics and Informatics · doi:10.1016/j.tele.2024.102187 · Cited by 36
“For example, ChatGPT has been shown to support clinical decision-making (Kung et al., 2023).”
Gin, B. C., LaForge, K., Burk-Rafel, J., & Boscardin, C. K. (2025) · Academic Medicine · doi:10.1097/acm.0000000000006117 · Cited by 36
Harnessing language models for streamlined postcolonoscopy patient management: a novel approach
Gorelik, Y., Ghersin, I., Maza, I., & Klein, A. (2023) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2023.06.025 · Cited by 37
van Wyk, M. M., Adarkwah, M. A., & Amponsah, S. (2023) · Open Praxis · doi:10.55982/openpraxis.15.3.563 · Cited by 42
Artificial Intelligence and the Simulationists
Rodgers, D. L., Needler, M., Robinson, A., Barnes, R., Brosche, T., Hernandez, J., Poore, J., VandeKoppel, P., & Ahmed, R. (2023) · Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000747 · Cited by 34
Koga, S. (2023) · Pathology International · doi:10.1111/pin.13382 · Cited by 41
Hussain, M. (2023) · IEEE Access · doi:10.1109/access.2023.3332468 · Cited by 57
Ohta, K., & Ohta, S. (2023) · Cureus · doi:10.7759/cureus.50369 · Cited by 45
“In English-speaking countries, GPT-4 has been reported to meet the passing criteria for both the United States Medical Licensing Examination and the United Kingdom Medical Licensing Assessment [6-8].”
BackgroundSinger, M. B., Fu, J. J., Chow, J., & Teng, C. C. (2024) · Journal of Surgical Education · doi:10.1016/j.jsurg.2023.11.019 · Cited by 34
RetractedChatGPT: towards AI subjectivity
D’Amato, K. (2024) · AI & Society · doi:10.1007/s00146-024-01898-z · Cited by 41
“…impact of the technology on various domains: e.g. education (Baidoo-Anu and Ansah 2023), public health (Biswas 2023), the medical industry (Kung et al. 2023), business and finance (AlAfan et al. 2023), law (Choi et al. 2023), creative writing (Cox and Tzoc 2023), software development…”
Scott, I. A., & Zuccon, G. (2024) · Internal Medicine Journal · doi:10.1111/imj.16393 · Cited by 39
“14 The need for clinical LLMs to be fi ne-tuned using examples curated by clinical domain experts may incur costs, workload and accountability imposts on human experts.”
BackgroundTorre, D. M., & Schuwirth, L. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2409936 · Cited by 42
“At the same time the performance of AI on traditional assessments like multiple choice-based assessment (USMLE or other licensing exams) creates some apprehension that will likely require us to rethink uses and interpretation of specific assessments and overall programs of assessment [115, 116].”
A Bayesian regularization neural network procedure to solve the language learning system
Sabir, Z., Khansa, S., Baltaji, G., & Saeed, T. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.112997 · Cited by 36
Impact of ChatGPT on English Academic Writing Ability and Engagement of Chinese EFL Undergraduates
Hongxia, H., & Razali, A. B. (2025) · International Journal of Instruction · doi:10.29333/iji.2025.18218a · Cited by 51
“ChatGPT improved writing self-efficacy by providing continuous feedback and cultivating a sense of achievement, and self-efficacy was positively correlated with long-term engagement in ChatGPT (Bouzar et al., 2024; Ghafouri et al., 2024; Kung et al., 2023).”
Chen, C., & Cui, Z. (2025) · Journal of Medical Internet Research · doi:10.2196/66083 · Cited by 46
“Generative AI models excel in conducting clinical reasoning and helping medical education [18,19], diagnosing complex cases [3], and enabling text-to-protein generation [20].”
Large Language Models and Text Embeddings for Detecting Depression and Suicide in Patient Narratives
Lho, S. K., Park, S. C., Lee, H., Oh, D. Y., Kim, H., Jang, S., Jung, H. Y., Yoo, S. Y., Park, S. M., & Lee, J. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.11922 · Cited by 49
Çekiç, E. C., & Tavşan, O. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07896-z · Cited by 37
“…vast text corpora from books, scientific articles, and websites [1], LLMs are applied in medical and dental education for clinical decision support, telemedicine, administrative tasks, patient education, scientific writing, and multi-lingual communication and enhancing educational processes [2–5].”
Itani, A., Gronseth, S. L., Musaad, S. M. A., Nguyen, T. T., Mirabile, Y. Z., & Beech, B. M. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-025-00563-9 · Cited by 49
Breeding, T., Martinez, B., Patel, H. S., Nasef, H., Arif, H., Nakayama, D. K., & Elkbuli, A. (2023) · The American Surgeon · doi:10.1177/00031348231180950 · Cited by 40
Future of Textile: Sustainable Manufacturing & Prediction via ChatGPT
Rathore, B. (2023) · Eduzone International peer reviewed/refereed academic multidisciplinary journal · doi:10.56614/eiprmj.v12i1y23.253 · Cited by 88
Tsoutsanis, P., & Tsoutsanis, A. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107794 · Cited by 40
Kunze, K. N., Jang, S. J., Fullerton, M. A., Vigdorchik, J. M., & Haddad, F. S. (2023) · The Bone & Joint Journal · doi:10.1302/0301-620x.105b6.bjj-2023-0156 · Cited by 38
Exploring the role of ChatGPT in medical image analysis
Handa, P., Chhabra, D., Goel, N., & Krishnan, S. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105292 · Cited by 38
ChatGPT: A Conceptual Review of Applications and Utility in the Field of Medicine
Rao, S. J., Isath, A., Krishnan, P., Tangsrivimol, J. A., Virk, H. U. H., Wang, Z., Glicksberg, B. S., & Krittanawong, C. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02075-x · Cited by 35
“The study demonstrated that ChatGPT could generate high-quality medical content, provide expla - nations for complex medical concepts, and help students better understand the material [47 ].”
Wardi, G., Owens, R., Josef, C., Malhotra, A., Longhurst, C., & Nemati, S. (2023) · Critical Care Medicine · doi:10.1097/ccm.0000000000005894 · Cited by 35
“Such triumphs in computer-based technologies are common today as artificial intelligence (AI) applications, such as ChatGPT and DALL-E, are mimicking human capabilities, even passing medical board examinations (2).”
BackgroundA Case Study Investigating the Utilization of ChatGPT in Online Discussions
Lin, X., Luterbach, K., Gregory, K. H., & Sconyers, S. E. (2024) · Online Learning · doi:10.24059/olj.v28i2.4407 · Cited by 35
“…(2022), believe that ChatGPT has the potential to promote independent learning practices by facilitating customized learning plans, others, like Kung et al. (2023) and Cotton et al. (2023), raise concerns about the obsolescence of traditional online exams and the automation of the teaching…”
BackgroundWill ChatGPT Match to Your Program?
Zumsteg, J. M., & Junn, C. (2023) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000002238 · Cited by 35
ChatGPT’s performance in dentistry and allergyimmunology assessments: a comparative study
Fuchs, A., Trachsel, T., Weiger, R., & Eggmann, F. (2023) · SWISS DENTAL JOURNAL SSO – Science and Clinical Topics · doi:10.61872/sdj-2024-06-01 · Cited by 32
The Perception of ChatGPT among Educators: Preliminary Findings
Oğurlu, Ü., & Mossholder, J. (2023) · Research in Social Sciences and Technology · doi:10.46303/ressat.2023.39 · Cited by 46
An Investigation of Applying Large Language Models to Spoken Language Learning
Gao, Y., Nuchged, B., Li, Y., & Peng, L. (2023) · Applied Sciences · doi:10.3390/app14010224 · Cited by 41
Assessing ChatGPT’s orthopedic in-service training exam performance and applicability in the field
Jain, N., Gottlich, C., Fisher, J., Campano, D., & Winston, T. (2024) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-023-04467-0 · Cited by 39
“Kung et al. used adjucators to examine the responses set forth by ChatGPT based on the criteria of “novelty, obviousness, and validity" [5].”
BackgroundMethodResultInfluentialWang, L., Bi, W., Zhao, S., Ma, Y., Lv, L., Meng, C., Fu, J., & Lv, H. (2024) · JMIR Formative Research · doi:10.2196/53216 · Cited by 37
Cadeddu, A., Chessa, A., Leo, V. D., Fenu, G., Motta, E., Osborne, F., Recupero, D. R., Salatino, A. A., & Secchi, L. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108166 · Cited by 44
How I GPT It: Development of Custom Artificial Intelligence (AI) Chatbots for Surgical Education
Sathe, T. S., Roshal, J. A., Naaseh, A., L’Huillier, J. C., Navarro, S. M., & Silvestri, C. J. (2024) · Journal of surgical education · doi:10.1016/j.jsurg.2024.03.004 · Cited by 37
Preparing Educators and Students at Higher Education Institutions for an AI-Driven World
Magrill, J., & Magrill, B. (2024) · Teaching & Learning Inquiry The ISSOTL Journal · doi:10.20343/teachlearninqu.12.16 · Cited by 43
Frenkel, M., & Emara, H. (2024) · Computer Applications in Engineering Education · doi:10.1002/cae.22781 · Cited by 40
“LLMs like ChatGPT are trained on massive text data sets and aim to predict the next word in a sequence [24], as cited in [19].”
Acharya, P. C., Alba, R. M., Krisanapan, P., Acharya, C., Suppadungsuk, S., Csongrádi, É., Mao, M. A., Craici, I. M., Miao, J., Thongprayoon, C., et al. (2024) · Diseases · doi:10.3390/diseases12080185 · Cited by 38
Harašta, J., Novotná, T., & Šavelka, J. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09422-w · Cited by 47
Asy’ari, M., & Шаров, С. (2024) · International Journal of Essential Competencies in Education · doi:10.36312/ijece.v3i2.2424 · Cited by 64
“Previous studies have explored ChatGPT’s integration into professional training and certification processes, including the USMLE (Exintaris et al., 2023; Kung et al., 2023).”
Stephan, D., Bertsch, A., Schumacher, S., Puladi, B., Burwinkel, M., Al‐Nawas, B., Kämmerer, P. W., & Thiem, D. G. E. (2025) · Journal of Medical Internet Research · doi:10.2196/73337 · Cited by 44
“ChatGPT’s potential in medical applications is underscored by its strong performance in standardized examinations [12,13], demonstrating its ability to process complex medical content at a level comparable to human expertise.”
Kortemeyer, G., Babayeva, M., Polverini, G., Widenhorn, R., & Gregorcic, B. (2025) · Physical Review Physics Education Research · doi:10.1103/98hg-rkrf · Cited by 43
Salimi, A., & Saheb, H. (2023) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2023.06.004 · Cited by 32
Nikdel, M., Ghadimi, H., Tavakoli, M., & Suh, D. W. (2023) · Journal of Pediatric Ophthalmology & Strabismus · doi:10.3928/01913913-20231005-02 · Cited by 38
Kollitsch, L., Eredics, K., Marszalek, M., Rauchenwald, M., Brookman-May, S. D., Burger, M., Körner-Riffard, K., & May, M. (2024) · World Journal of Urology · doi:10.1007/s00345-023-04749-6 · Cited by 38
Exploring the Efficacy of ChatGPT in Analyzing Student Teamwork Feedback with an Existing TaxonomyPreprint
Katz, A., Wei, S., Nanda, G., Brinton, C. G., & Ohland, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.11882 · Cited by 41
ChatGPT-Based Learning: Generative Artificial Intelligence in Medical Education
Stretton, B., Kovoor, J., Arnold, M., & Bacchi, S. (2023) · Medical Science Educator · doi:10.1007/s40670-023-01934-5 · Cited by 34
“…has been demonstrated across a diverse range of implementations including its ability to pass the United States Medical Licensing Exams, accurately provide a list of differentials for cases written by internal medicine physicians and even contribute to writing published medical case reports [1–3].”
Ichikawa, T., Olsen, E., Vinod, A., Glenn, N., Hanna, K., Lund, G. C., & Pierce-Talsma, S. (2025) · JMIR Medical Education · doi:10.2196/58766 · Cited by 34
Artificial Intelligence in Pediatrics: Learning to Walk Together
Demirbaş, K. C., Yıldız, M., Saygılı, S., Canpolat, N., & Kasapçopur, Ö. (2024) · Turkish Archives of Pediatrics · doi:10.5152/turkarchpediatr.2024.24002 · Cited by 41
Zaidat, B., Shrestha, N., Rosenberg, A. M., Ahmed, W., Rajjoub, R., Hoang, T. J., Mejia, M. R., Duey, A. H., Tang, J. E., Kim, J., et al. (2024) · Neurospine · doi:10.14245/ns.2347310.655 · Cited by 35
ChatGPT giving advice on how to cheat in university assignments: how workable are its suggestions?
Spennemann, D. H. R., Biles, J. M., Brown, L., Ireland, M. F., Longmore, L., Singh, C., Wallis, A., & Ward, C. (2024) · Interactive Technology and Smart Education · doi:10.1108/itse-10-2023-0195 · Cited by 37
Shanshan, S., & Sen, G. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12996 · Cited by 29
Pan, G., & Ni, J. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05871-8 · Cited by 37
“In addition, Kung et al. have demonstrated that ChatGPT can effectively process intricate medical and clinical information with a high level of accuracy [14].”
BackgroundMethodSteybe, D., Poxleitner, P. J., Aljohani, S., Herlofson, B. B., Nicolatou‐Galitis, O., Patel, V., Fedele, S., Kwon, T., Fusco, V., Pichardo, S. E. C., et al. (2025) · Journal of Cranio-Maxillofacial Surgery · doi:10.1016/j.jcms.2024.12.009 · Cited by 44
Unveiling GPT-4V's hidden challenges behind high accuracy on USMLE questions: Observational Study
Yang, Z., Yao, Z., Tasmin, M., Vashisht, P., Jang, W. S., Ouyang, F., Wang, B., McManus, D. D., Berlowitz, D. R., & Yu, H. Q. (2025) · Journal of Medical Internet Research · doi:10.2196/65146 · Cited by 49
“5 Turbo and GPT-4 have achieved commendable results in the United States Medical Licensing Examination (USMLE), leading to discussions about the readiness of LLM applications for integration into clinical [17-19] and educational [20-22] environments.”
Bariatric Evaluation Through AI: a Survey of Expert Opinions Versus ChatGPT-4 (BETA-SEOV)
Jazi, A. H. D., Mahjoubi, M., Shahmiri, S. S., Alqahtani, A. R., Haddad, A. J., Pazouki, A., Prasad, A., Safadi, B. Y., Chiappetta, S., Taşkın, H. E., et al. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06903-w · Cited by 39
“The potential applications of AI in healthcare are now being analyzed more closely, marking a new era of interest [4, 5].”
BackgroundCan ChatGPT Aid Clinicians in Educating Patients on the Surgical Management of Glaucoma?
Kianian, R., Sun, D., & Giaconi, J. A. (2023) · Journal of Glaucoma · doi:10.1097/ijg.0000000000002338 · Cited by 36
Assessing ChatGPT’s capability as a new age standardized patient: qualitative study
Cross, J., Kayalackakom, T., Robinson, R. E., Vaughans, A., Sebastian, R., Hood, R., Lewis, C., Devaraju, S., Honnavar, P., Naik, S., et al. (2025) · JMIR Medical Education · doi:10.2196/63353 · Cited by 33
“5 model, has an enormous 175 billion parameters and displays a remarkable capacity for understanding and reasoning, bordering on human-like proficiency [9].”
Morjaria, L., Burns, L., Bracken, K., Levinson, A. J., Ngo, Q. N., Lee, M., & Sibbald, M. (2024) · International Medical Education · doi:10.3390/ime3010004 · Cited by 33
“Shortly after the release of ChatGPT, it demonstrated the ability to pass well-known standardized multiple-choice assessments including the United States Medical Licensing Examination (USMLE) written by American medical students [6,7].”
Evaluating the Artificial Intelligence Performance Growth in Ophthalmic Knowledge
Jiao, C., Edupuganti, N., Patel, P. A., Bui, T., & Sheth, V. S. (2023) · Cureus · doi:10.7759/cureus.45700 · Cited by 46
“Furthermore, over 90% of AI responses provided substantial insights, demonstrating deductive reasoning that could be beneficial for human learners [6].”
BackgroundComparing ChatGPT and clinical nurses’ performances on tracheostomy care: A cross-sectional study
Wang, T., Mu, J., Chen, J., & Lin, C.-C. (2024) · International Journal of Nursing Studies Advances · doi:10.1016/j.ijnsa.2024.100181 · Cited by 30
Evaluation of ChatGPT as a Tool for Answering Clinical Questions in Pharmacy Practice
Munir, F., Gehres, A., Wai, D., & Song, L. (2024) · Journal of Pharmacy Practice · doi:10.1177/08971900241256731 · Cited by 32
AlShehri, Y., McConkey, M. O., & Lodhia, P. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.06.017 · Cited by 34
Xiao, L., Wang, D., Zhong, H., Wang, Q., Li, R., Jia, R., & Wan, B. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103805 · Cited by 39
Harnessing the Power of LLMs for Service Quality Assessment From User-Generated Content
Falatouri, T. N., Hrušecká, D., & Fischer, T. (2024) · IEEE Access · doi:10.1109/access.2024.3429290 · Cited by 36
Hetz, M. J., Carl, N., Haggenmüller, S., Wies, C., Kather, J. N., Michel, M., Wessels, F., & Brinker, T. J. (2024) · ESMO Real World Data and Digital Oncology · doi:10.1016/j.esmorw.2024.100078 · Cited by 38
Effects of Applicants' Use of Generative AI in Personnel Selection: Towards a More Nuanced View?
Lievens, F., & Dunlop, P. D. (2024) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12516 · Cited by 35
Distinguishing LLM-Generated from Human-Written Code by Contrastive Learning
Xu, X., Ni, C., Guo, X., Liu, S., Wang, X., Liu, K., & Yang, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705300 · Cited by 57
ChatGPT in Medical Education: Bibliometric and Visual Analysis
Zhang, Y., Xie, X., & Xu, Q. (2025) · JMIR Medical Education · doi:10.2196/72356 · Cited by 43
Bixler, K., & Ceballos, M. (2025) · Leadership and Policy in Schools · doi:10.1080/15700763.2024.2428297 · Cited by 43
“…suggest that ChatGPT could be used as a tool for learning for test-taking due to its ability to refer learners to primary sources of information and to explain lines of reasoning through justifications of correct answers and wrong answers to specific examinations (Choi, 2023; Kung et al., 2023).”
ChatGPT: The next-gen tool for triaging?
Bhattaram, S., Shinde, V., & Khumujam, P. P. (2023) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2023.03.027 · Cited by 33
Altamimi, I., Altamimi, A., Alhumimidi, A. S., Altamimi, A., & Temsah, M.-H. A. (2023) · Cureus · doi:10.7759/cureus.40351 · Cited by 37
“Alternative chatbots were not considered for this study, as they demonstrated inferior capabilities when compared to ChatGPT [11].”
ResultTao, B. K.-L., Hua, N. J., Milkovich, J., & Micieli, J. A. (2024) · Eye · doi:10.1038/s41433-024-03037-w · Cited by 38
“This is in keeping with works in the literature using pre-existing official resources for collating examination questions [15, 16, 19, 21].”
Knoedler, L., Knoedler, S. A., Hoch, C. C., Prantl, L., Frank, K., Soiderer, L., Cotofana, S. C., Dorafshar, A. H., Schenck, T. L., Vollbach, F. H., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-63997-7 · Cited by 34
“Lastly, it is important to acknowledge the limitation stemming from our study’s focus on text-based questions and the exclusion of artificially created queries designed to assess ChatGPT’s ability to navigate different levels of question complexity and questioning techniques.”
BackgroundEvaluating Large Language Models in extracting cognitive exam dates and scores
Zhang, H., Jethani, N., Jones, S., Genes, N. G., Major, V. J., Jaffe, I. S., Cardillo, A. B., Heilenbach, N., Ali, N. F., Bonanni, L. J., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000685 · Cited by 39
“These models have demonstrated impressive clinical reasoning abilities [7], successfully passing medical licensing exams [8–10] and generating medical advice on distinct subjects, including cardio-vascular disease [11], breast cancer [12], colonoscopy [13], and general health inquiries [6,14– 16].”
Passing is Great: Can ChatGPT Conduct USMLE Exams?
Biswas, S. S. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03224-y · Cited by 35
Banerjee, P., Srivastava, A. K., Adjeroh, D. A., Reddy, R., & Karimian, N. (2025) · IEEE Access · doi:10.1109/access.2024.3524102 · Cited by 32
ChatGPT applications in Academic Research: A Review of Benefits, Concerns, and Recommendations
AlZaabi, A., ALAmri, A., Albalushi, H., Aljabri, R., & AalAbdulsalam, A. (2023) · doi:10.1101/2023.08.17.553688 · Cited by 32
“Currently, there are only four articles that have listed ChatGPT as an author [183]–[186], and one of them has submitted a corrigendum to remove ChatGPT from authorship based on the journal's guide for authors and Elsevier's Publishing Ethics Policies145.”
BackgroundDraschl, A., Hauer, G., Fischerauer, S. F., Kogler, A., Leitner, L., Andreou, D., Leithner, A., & Sadoghi, P. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12206655 · Cited by 29
“Among these questions, eight (Q1–8) were related to PJI prevention, and five (Q9–13) focused on PJI diagnosis, while one question each (Q14 and Q15) addressed pathogen factors and fungal PJI, respectively.”
BackgroundAlzboon, M. S., Qawasmeh, S., Alqaraleh, M. K. S., Abuashour, A., Bader, A. F., & Al-Batah, M. S. (2023) · International Conference on Emerging Smart Technologies and Applications (eSmarTA) · doi:10.1109/esmarta59349.2023.10293294 · Cited by 38
“Artificial intelligence can change education by providing tailored learning experiences, automating tedious processes, and freeing instructors to focus on more essential tasks [27].”
BackgroundChaker, S. C., Hung, Y., Saad, M., Golinko, M. S., & Galdyn, I. A. (2024) · The Cleft Palate-Craniofacial Journal · doi:10.1177/10556656231223596 · Cited by 37
“22,23,25 As suggested by our model, there is an increase in utility (or at least the consideration of the utility) of AI in the medical fi eld.”
Wang, S., Mo, C., Chen, Y., Dai, X., Wang, H., & Shen, X. (2024) · JMIR Medical Education · doi:10.2196/55595 · Cited by 36
“The broad applicability of AI chatbots in health care extends beyond patient interaction, serving clinicians, researchers, and students, with ChatGPT showing effectiveness in personalizing patient interactions and providing consumer health education [8,9].”
BackgroundChatGPT Earns American Board Certification in Hand Surgery
Ghanem, D., Nassar, J. E., Bachour, J. E., & Hanna, T. (2024) · Hand surgery & rehabilitation · doi:10.1016/j.hansur.2024.101688 · Cited by 35
Evaluation and Analysis of Large Language Models for Clinical Text Augmentation and Generation
Latif, A., & Kim, J. (2024) · IEEE Access · doi:10.1109/access.2024.3384496 · Cited by 51
“[ 17], [18], [19].”
Szymanski, A., Wimer, B. L., Anuyah, O., Eicher‐Miller, H. A., & Metoyer, R. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641924 · Cited by 43
Teaching and testing in the era of text-generative AI: exploring the needs of students and teachers
Jochim, J., & Lenz-Kesekamp, V. (2024) · Information and Learning Sciences · doi:10.1108/ils-10-2023-0165 · Cited by 33
Rhetoric of/with AI: An Introduction
Majdik, Z. P., & Graham, S. S. (2024) · Rhetoric Society Quarterly · doi:10.1080/02773945.2024.2343264 · Cited by 32
Cao, M., Wang, Q., Zhang, X., Liang, Z., Qiu, J., Yung, P. S., & Ong, M. T. (2024) · Journal of sport and health science/Journal of Sport and Health Science · doi:10.1016/j.jshs.2024.101016 · Cited by 37
“Specifically, multiple unproven supplements were recommended for OA patients, such as S-adenosylmethionine, methylsulfonylme-thane, etc. (Supplementary Tables 2 (cid:1) 4).”
Hasan, W. U., Zaman, K. T., Wang, X., Li, J., Xie, B., & Tao, C. (2024) · npj Biomedical Innovations. · doi:10.1038/s44385-024-00004-8 · Cited by 48
Mahande, R. D., Fakhri, M. M., Suwahyu, I., & Sulaiman, D. R. A. (2025) · Journal of Applied Research in Higher Education · doi:10.1108/jarhe-07-2024-0372 · Cited by 39
Espartinez, A. S. (2025) · Social Sciences · doi:10.3390/socsci14060359 · Cited by 38
“Faculty exhibit greater ambivalence, acknowledging benefits while worrying about academic integrity, critical thinking erosion, and shifts in student–teacher dynamics (Al-khresheh 2024; Kung et al. 2023).”
Shyr, C., Ren, B., Hsu, C.-Y., Nguyen, M.-H., Enger, M., Yan, C., Tinker, R. J., Cassini, T. A., Hamid, R., Wright, A. T., et al. (2026) · Journal of the American Medical Informatics Association · doi:10.1101/2025.08.06.25333170 · Cited by 43
“Among these, diagnostic support has gained increasing attention, with recent studies demonstrating strong LLM performance in diagnostic reasoning and clinical decision-making [4–9].”
LogLM: From Task-Based to Instruction-Based Automated Log Analysis
Liu, Y., Ji, Y., Tao, S., He, M., Meng, W., Zhang, S., Sun, Y., Xie, Y., Chen, B., & Yang, H. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00041 · Cited by 45
“A score of 100 indicates a perfect match between the generated and reference text, while a score of 0 indicates no overlap.”
Metadata
- Authors
- Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-02-09
- Volume
- 2
- Issue
- 2
- Pages
- e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.