Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Open accessPublished version of a preprint
The DOI matches its Crossref record, with a small difference in title or year; 28 of the 28 references it cites were found in a registry or an index.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Kung et al. (2023) evaluated the performance of ChatGPT on the United States Medical Licensing Exam and demonstrated that the chatbot did not reach the passing threshold and achieved an overall accuracy of only 50% (Kung et al., 2023).”
From A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing (2025) · cites this paper as Kung et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [17]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI's capabilities, such as passing the United States Medical Licensing Exam [107], completing exams at the University of Minnesota Law School [108], and solving queries from Wharton School of Business tests [109], underscore its potential.”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [107]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The HCI community has explored the application of LLMs for various healthcare practices, such as mental health support [66,68], health information seeking [81,117,128], and public health interventions [58]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [66]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The recent emergence of generative AI models capable of producing coherent, unscripted dialogue opens new possibilities for scalable, on-demand simulations (Kung et al., 2023).”
From DIALOGUE: A Generative AI-Based Pre–Post Simulation Study to Enhance Diagnostic Communication in Medical Students Through Virtual Type 2 Diabetes Scenarios (2025) · cites this paper as Kung et al., 2023
Experimental — accuracy not yet measured
52 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“Indeed today, Inception-V3 serves as the basic foundation of many of the top medical imaging models currently published, ranging from ophthalmology [5,6] and pathology [7] to dermatology [8].”
Background
Cited by
Items 301–400 of 1,129
Reimagining Healthcare: Unleashing the Power of Artificial Intelligence in Medicine
Iqbal, J., Cortés Jaimes, D. C., Makineni, P., Subramani, S., Hemaida, S., Thugu, T. R., Butt, A. N., Sikto, J. T., Kaur, P., Lak, M. A., et al. (2023) · Cureus · doi:10.7759/cureus.44658 · Cited by 94
“The study concluded that large language models have significant potential to assist medical education and clinical decision-making [125].”
BackgroundWilliams, C. Y. K., Miao, B. Y., Kornblith, A. E., & Butte, A. J. (2024) · Nature Communications · doi:10.1038/s41467-024-52415-1 · Cited by 119
“They instead apply these models to answerquestionsfrommedicalexaminationssuchastheUSMLE,solve publicly available clinical diagnostic challenges such as the New England Journal of Medicine (NEJM) clinicopathologic conferences, or evaluate performance on existing clinical benchmarks 3,6 – 9 .”
BackgroundGenerative artificial intelligence in medicine
Teo, Z. L., Thirunavukarasu, A. J., Elangovan, K., Cheng, H., Moova, P., Soetikno, B., Nielsen, C., Pollreisz, A., Ting, D. S. J., Morris, R. J. T., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03983-2 · Cited by 119
“Medical GAI garnered initial interest after LLM chatbots achieved passing level marks in examinations taken by medical students and doctors 75,76 .”
Foundation model of neural activity predicts response to new stimulus types
Wang, E. Y., Fahey, P. G., Ding, Z., Papadopoulos, S., Ponder, K., Weis, M. A., Chang, A., Muhammad, T., Patel, S. S., Ding, Z., et al. (2025) · Nature · doi:10.1038/s41586-025-08829-y · Cited by 169
“These tasks include answering unstructured questions and even passing medical licensing exams (Kung et al., 2023).”
InfluentialThe generative era of medical AI
Fahrner, J., Chen, E., Topol, E. J., & Rajpurkar, P. (2025) · Cell · doi:10.1016/j.cell.2025.05.018 · Cited by 144
Huang, Y., Gomaa, A., Semrau, S., Haderlein, M., Lettmaier, S., Weissmann, T., Grigo, J., Tkhayat, H. B., Frey, B., Gaipl, U., et al. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1265024 · Cited by 112
“For domain knowledge in medicine, ChatGPT-3 achieved better than 50% accuracy across all the exams of the United States Medical Licensing Exam (USMLE) and exceeding 60% in certain analyses (17); ChatGPT-3.”
Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries
Jin, Y., Chandra, M., Verma, G., Hu, Y., De Choudhury, M., & Kumar, S. (2024) · Proceedings of the ACM Web Conference 2024 · doi:10.1145/3589334.3645643 · Cited by 75
Eppler, M., Ganjavi, C., Ramacciotti, L. S., Piazza, P., Rodler, S., Checcucci, E., Gomez Rivas, J., Kowalewski, K. F., Belenchón, I. R., Puliatti, S., et al. (2024) · European Urology · doi:10.1016/j.eururo.2023.10.014 · Cited by 117
Beyond human expertise: the promise and limitations of ChatGPT in suicide risk assessment
Elyoseph, Z., & Levkovich, I. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1213141 · Cited by 116
Yang, Z., Xu, X., Yao, B., Rogers, E., Zhang, S., Intille, S., Shara, N., Gao, G. G., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3659625 · Cited by 120
Das, D., Kumar, N., Longjam, L. A., Sinha, R., Deb Roy, A., Mondal, H., & Gupta, P. (2023) · Cureus · doi:10.7759/cureus.36034 · Cited by 116
“ChatGPT is one such artificial intelligence (AI) language model that has been trained to understand the nuances of natural language and provide accurate answers to a wide range of questions [2,3,4].”
BackgroundLoh, E. (2024) · BMJ Leader · doi:10.1136/leader-2023-000797 · Cited by 116
Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4
Krusche, M., Callhoff, J., Knitza, J., & Ruffer, N. (2023) · Rheumatology International · doi:10.1007/s00296-023-05464-6 · Cited by 116
Lind, C. M., Abtahi, F., & Forsman, M. (2023) · Sensors · doi:10.3390/s23094259 · Cited by 136
Comparative Evaluation of LLMs in Clinical Oncology
Rydzewski, N. R., Dinakaran, D., Zhao, S. G., Ruppin, E., Turkbey, B., Citrin, D. E., & Patel, K. R. (2024) · NEJM AI · doi:10.1056/aioa2300151 · Cited by 106
Performance of large language models on a neurology board-style examination
Schubert, M. C., Wick, W., & Venkataramani, V. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.46721 · Cited by 115
Can ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115
“Kung et al. (2023 ) corroborated this finding by demonstrating that the model achieved scores near the passing threshold for the United States Medical Licensing Exam (USMLE).”
Kong, S.-C., Lee, J. C.-K., & Tsang, O. (2024) · Research and Practice in Technology Enhanced Learning · doi:10.58459/rptel.2024.19030 · Cited by 69
“Indeed, although ChatGPT, one of the most popular text-based generative AI tools, passed the United States Medical Licensing Exam, it only did so marginally with a 60% accuracy (Kung et al., 2023).”
BackgroundLarge language models streamline automated machine learning for clinical studies
Tayebi Arasteh, S., Han, T., Lotfinia, M., Kuhl, C., Kather, J. N., Truhn, D., & Nebelung, S. (2024) · Nature Communications · doi:10.1038/s41467-024-45879-8 · Cited by 114
“In medicine, the principle feasibility and use of AutoML platforms, such as the Classi fi cation Learner of MATLAB (MathWorks Inc.), Vertex AI (Google LLC), and Azure (Microsoft Corporation), have been demonstrated 6 – 11 , enabling non-technical experts to create ML models.”
Zong, H., Li, J., Wu, E., Wu, R., Lu, J., & Shen, B. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05125-7 · Cited by 114
“Recent studies shown that Chat-GPT passed United States Medical Licensing Examination [19, 20].”
ResultMihalache, A., Huang, R. S., Popovic, M. M., & Muni, R. H. (2024) · Medical Teacher · doi:10.1080/0142159x.2023.2249588 · Cited by 111
Consistency Analysis of ChatGPT
Jang, M., & Lukasiewicz, T. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.991 · Cited by 67
Rabbani, S. A., El-Tanani, M., Sharma, S., Rabbani, S. S., El-Tanani, Y., Kumar, R., & Saini, M. (2025) · BioMedInformatics · doi:10.3390/biomedinformatics5030037 · Cited by 80
“%) in some analyses [3].”
Exploring ChatGPT for information of cardiopulmonary resuscitation
Ahn, C. (2023) · Resuscitation · doi:10.1016/j.resuscitation.2023.109729 · Cited by 112
People Overtrust AI-Generated Medical Advice despite Low Accuracy
Shekar, S., Pataranutaporn, P., Sarabu, C. R., Cecchi, G., & Maes, P. (2025) · NEJM AI · doi:10.1056/aioa2300015 · Cited by 139
The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2023) · doi:10.1101/2023.01.30.23285067 · Cited by 106
Meyer, A., Riese, J., & Streichert, T. (2024) · JMIR Medical Education · doi:10.2196/50965 · Cited by 110
“Thus, health care professionals could potentially benefit, especially from GPT-4’s conclusive and often nonobvious insights to multiple-choice questions, as these users have the ability to verify crucial details [13,14,41].”
Zhui, L., Fenghe, L., Xuehu, W., Qining, F., & Wei, R. (2024) · Journal of Medical Internet Research · doi:10.2196/60083 · Cited by 91
“A well-known test of the accuracy of LLMs in the context of medical questions is ChatGPT's successful passing of the United States Medical Licensing Exam [40].”
History, development, and principles of large language models: an introductory survey
Wang, Z., Chu, Z., Doan, T. V., Ni, S., Yang, M., & Zhang, W. (2025) · AI and Ethics · doi:10.1007/s43681-024-00583-7 · Cited by 109
“…ICSE [35, 36], ICAIF [37, 38], ICDM [39–41], IJCAI [42–44], AAAI [45–47], ECAI [48–50], BEA [51], Financial Innovation [52], PLOS Digital Health [53, 54], Applied Sciences [55], Interspeech [56–58], Journal of Machine Learning Research [59–62], Journal of AI [63], etc., with an emphasis on…”
Performance of ChatGPT on the pharmacist licensing examination in Taiwan
Wang, Y.-M., Shen, H.-W., & Chen, T.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000942 · Cited by 109
Savelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588792 · Cited by 109
“Other applications of GPT to solve MCQ-based exams include United States Medical Licensing Examination (USMLE) [15, 22, 25], Multistate Bar Examination [6], and American Institute of Certified Public Accountants’ (AICPA) Regulation exam [5].”
MethodPractical Applications of ChatGPT in Undergraduate Medical Education
Tsang, R. (2023) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205231178449 · Cited by 104
“Following its release, ChatGPT gained attention for its ability to “perform at or near the passing threshold” of the three-step, United States Medical Licensing Exam (USMLE) per Kung et al.(2)”
BackgroundInfluentialSrivastav, S., Chandrakar, R., Gupta, S., Babhulkar, V., Agrawal, S., Jaiswal, A., Prasad, R., & Wanjari, M. B. (2023) · Cureus · doi:10.7759/cureus.41435 · Cited by 103
“By analyzing a patient's medical data, ChatGPT can help oncologists identify the most effective treatments and tailor them to their unique characteristics [18].”
BackgroundMethodInfluentialHerrmann-Werner, A., Festl-Wietek, T., Holderried, F., Herschbach, L., Griewatz, J., Masters, K., Zipfel, S., & Mahling, M. (2024) · Journal of Medical Internet Research · doi:10.2196/52113 · Cited by 96
“This confirms results from other studies, which also show high exam scores in, for example, the USMLE (United States Medical Licensing Examination) [14] and other board exams [27].”
Zhu, J., Dang, P., Cao, Y., Lai, J., Guo, Y., Wang, P., & Li, W. (2024) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2024.2306167 · Cited by 121
Recent Advances in Large Language Models for Healthcare
Nassiri, K., & Akhloufi, M. A. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020062 · Cited by 134
“This enables them to generate differential diagnoses [128,129], predict specific diagnoses [130,131], and provide answers to physicians’ queries [98,132].”
BackgroundUse of an ambient artificial intelligence tool to improve quality of clinical documentation
Balloch, J., Sridharan, S., Oldham, G., Wray, J., Gough, P., Robinson, R., Sebire, N. J., Khalil, S., Asgari, E., Tan, C., et al. (2024) · Future Healthcare Journal · doi:10.1016/j.fhj.2024.100157 · Cited by 107
ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis
Bicknell, B. T., Butler, D., Whalen, S., Ricks, J., Dixon, C. J., Clark, A. B., Spaedy, O., Skelton, A., Edupuganti, N., Dzubinski, L., et al. (2024) · JMIR Medical Education · doi:10.2196/63430 · Cited by 107
“Evaluations of ChatGPT 3.0 found it was able to accurately answer USMLE sample items 36.7% of the time [23], improving to more than 50% correct in a matter of months [21].”
InfluentialClinical and Surgical Applications of Large Language Models: A Systematic Review
Pressman, S. M., Borna, S., Gomez-Cabello, C. A., Haider, S. A., Haider, C. R., & Forte, A. J. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13113041 · Cited by 107
“Through iterative exposure to a training dataset, LLMs refine their predictive abilities by anticipating subsequent words in a sequence, evaluating their predictions against actual outcomes, and adjusting their parameters accordingly until they achieve a high level of proficiency [4,5].”
Pursnani, V., Sermet, Y., Kurt, M., & Demir, I. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100183 · Cited by 106
Using generative artificial intelligence/ChatGPT for academic communication: Students' perspectives
Liu, Y., Park, J., & McMinn, S. (2024) · International Journal of Applied Linguistics · doi:10.1111/ijal.12574 · Cited by 106
Giannos, P., & Delardas, O. (2023) · JMIR Medical Education · doi:10.2196/47737 · Cited by 95
Sun, H., Zhang, K., Lan, W., Gu, Q., Jiang, G., Yang, X., Qin, W., & Han, D. (2023) · Journal of Medical Internet Research · doi:10.2196/51300 · Cited by 101
“They have already shown their potential in medical writing and education [10,11].”
A Mathematical Investigation of Hallucination and Creativity in GPT Models
Lee, M. (2023) · Mathematics · doi:10.3390/math11102320 · Cited by 105
Shieh, A., Tran, B., He, G., Kumar, M., Freed, J. A., & Majety, P. (2024) · Scientific Reports · doi:10.1038/s41598-024-58760-x · Cited by 101
Li, R., Kumar, A., & Chen, J. H. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1835 · Cited by 104
Li, J., Zong, H., Wu, E., Wu, R., Peng, Z., Zhao, J., Yang, L., Xie, H., & Shen, B. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05738-y · Cited by 104
“Since its inception, ChatGPT has been widely applied in medicine domain, especially after testing, it can demonstrate the medical level that meets the requirements of passing the United States Medical Licensing Exam [6].”
BackgroundSuh, P. S., Shim, W. H., Suh, C. H., Heo, H., Park, C. R., Eom, H. J., Park, K. J., Choe, J., Kim, P. H., Park, H. J., et al. (2024) · Radiology · doi:10.1148/radiol.240273 · Cited by 128
Rossettini, G., Rodeghiero, L., Corradi, F., Cook, C., Pillastrini, P., Turolla, A., Castellini, G., Chiappinotto, S., Gianola, S., & Palese, A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05630-9 · Cited by 103
“Conventional preparation methods involve attending classes, studying textbooks, and completing practical exercises [2].”
BackgroundMethodInfluentialThe application of Chat Generative Pre-trained Transformer in nursing education
Liu, J., Liu, F., Fang, J., & Liu, S. (2023) · Nursing Outlook · doi:10.1016/j.outlook.2023.102064 · Cited by 103
Complement or Substitute? How AI Increases the Demand for Human SkillsPreprint
Mäkelä, E., & Stephany, F. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5153230 · Cited by 115
ChatGPT: The transformative influence of generative AI on science and healthcare
Varghese, J., & Chapiro, J. (2024) · Journal of Hepatology · doi:10.1016/j.jhep.2023.07.028 · Cited by 89
Carlà, M. M., Gambini, G., Baldascino, A., Giannuzzi, F., Boselli, F., Crincoli, E., D’Onofrio, N. C., & Rizzo, S. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-325143 · Cited by 95
Claman, D. B., & Sezgın, E. (2024) · JMIR Medical Education · doi:10.2196/52346 · Cited by 108
“Currently well-known LLMs and FMs, such as GPT (OpenAI) [15], LaMDA and PaLM (Google) [16], and LLaMA (Meta) [17], have shown potential in medical education and practice, including problem-solving, question and answering, summarization, and content creation [13,18,19].”
BackgroundArtificial intelligence chatbot performance in triage of ophthalmic conditions
Lyons, R. J., Arepalli, S. R., Fromal, O., Choi, J. D., & Jain, N. (2024) · Canadian Journal of Ophthalmology · doi:10.1016/j.jcjo.2023.07.016 · Cited by 101
Kendall, G., & Teixeira da Silva, J. A. (2024) · Learned Publishing · doi:10.1002/leap.1578 · Cited by 100
“In one example, a preprint lists ChatGPT as an author (Kung et al., 2022) whereas the published version of the paper excluded it as an author (Kung et al., 2023).”
BackgroundOn the cusp: considering the impact of artificial intelligence language models in healthcare
Goodman, R. S., Patrinely, J. R., Osterman, T., Wheless, L., & Johnson, D. B. (2023) · Med · doi:10.1016/j.medj.2023.02.008 · Cited by 100
Harnessing the potential of large language models in medical education: promise and pitfalls
Benítez, T. M., Xu, Y., Boudreau, J. D., Kow, A. W. C., Bello, F., Van Phuoc, L., Wang, X., Sun, X., Leung, G. K.-K., Lan, Y., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad252 · Cited by 100
Assessment of artificial intelligence applications in responding to dental trauma
Özden, İ., Gökyar, M., Ozden, M. E., & Öveçoğlu, H. S. (2024) · Dental Traumatology · doi:10.1111/edt.12965 · Cited by 111
Bayesian Optimization of Catalysis with In-Context Learning
Ramos, M. C., Michtavy, S. S., White, A. D., & Porosoff, M. D. (2026) · ACS Central Science · doi:10.1021/acscentsci.5c02418 · Cited by 37
ChatGPT failed Taiwan’s Family Medicine Board Exam
Weng, T.-L., Wang, Y.-M., Chang, S., Chen, T.-J., & Hwang, S.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000946 · Cited by 100
Tangadulrat, P., Sono, S., & Tangtrakulwanich, B. (2023) · JMIR Medical Education · doi:10.2196/50658 · Cited by 99
“However, the results of using ChatGPT were quite varied, from passing the exam to failing some [7,8,10,30-33].”
ResultFuture of the Language Models in Healthcare: The Role of Chatgpt
Tustumi, F., Andreollo, N. A., & Aguilar-Nascimento, J. E. D. (2023) · ABCD. Arquivos Brasileiros de Cirurgia Digestiva (São Paulo) · doi:10.1590/0102-672020230002e1727 · Cited by 99
“ChatGPT is designed to understand and respond to text-based inputs from users in order to provide helpful and informative responses to their questions and comments 4,11,12 .”
BackgroundReddy, S. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101304 · Cited by 99
Can large language models write reflectively
Li, Y., Sha, L., Yan, L., Lin, J., Raković, M., Galbraith, K., Lyons, K., Gašević, D., & Chen, G. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100140 · Cited by 99
What Should ChatGPT Mean for Bioethics?
Cohen, I. G. (2023) · The American Journal of Bioethics · doi:10.1080/15265161.2023.2233357 · Cited by 98
Nakaura, T., Yoshida, N., Kobayashi, N., Shiraishi, K., Nagayama, Y., Uetani, H., Kidoh, M., Hokamura, M., Funama, Y., & Hirai, T. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-023-01487-y · Cited by 96
“Additionally, recent research has shown that GPT performed at or near the passing threshold for all three exams of the USMLE without any specialized training or reinforcement [11], and the usefulness for the transformation and the summarization of radiology reports [12, 13].”
BackgroundHarnessing progress in radiotherapy for global cancer control
Jaffray, D. A., Knaul, F. M., Baumann, M. A., & Gospodarowicz, M. K. (2023) · Nature Cancer · doi:10.1038/s43018-023-00619-7 · Cited by 107
Analyzing ChatGPT adoption drivers with the TOEK framework
Jo, H., & Bang, Y. (2023) · Scientific Reports · doi:10.1038/s41598-023-49710-0 · Cited by 83
Thirunavukarasu, A. J., Mahmood, S., Malem, A., Foster, W. P., Sanghera, R., Hassan, R., Zhou, S., Wong, S. W., Wong, Y. L., Chong, Y. J., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000341 · Cited by 88
“Our null hypothesis was that LLMs and doctors would exhibit similar performance, supported by results in a wide range of medical examinations [3, 6].”
BackgroundResultAbeltino, A., Riente, A., Bianchetti, G., Serantoni, C., Spirito, M. D., Capezzone, S., Esposito, R., & Maulucci, G. (2024) · Nutrition Reviews · doi:10.1093/nutrit/nuae035 · Cited by 112
LLM-assisted systematic review of large language models in clinical medicine
Chen, S. F., Alyakin, A., Seas, A., Yang, E., Choi, J. J., Lee, J. V., Chen, A. L., Warman, P. I., Bitolas, R., Steele, R. J., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04229-5 · Cited by 145
“The rarest models studied were Inflection’s ‘Pi’ chatbot Licensing Examination 8–10 .”
Hadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741 · Cited by 83
Exploring the potential of using ChatGPT in physics education
Liang, Y., Zou, D., Xie, H., & Wang, F. L. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00273-7 · Cited by 98
“Besides arithmetic reasoning, ChatGPT has examined its ability to perform clinical reasoning by testing its performance on the United States Medical Licensing Exam, where the results found that ChatGPT achieved near the passing threshold of 60% accuracy (Gilson et al., 2023; Kung et al., 2023).”
MethodChatGPT in Research: Balancing Ethics, Transparency and Advancement
Graf, A., & Bernardi, R. E. (2023) · Neuroscience · doi:10.1016/j.neuroscience.2023.02.008 · Cited by 109
Gala, D., & Makaryus, A. N. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20156438 · Cited by 88
“It is worth noting that ChatGPT has demonstrated impressive capabilities, including passing medical board exams, which highlights the potential of language models to assist and augment healthcare professionals [34].”
BackgroundGlobal Cancer Surgery: pragmatic solutions to improve cancer surgery outcomes worldwide
Are, C., Murthy, S. S., Sullivan, R. J., Schissel, M. E., Chowdhury, S., Alatise, O., Anaya, D., Are, M., Balch, C. M., Bartlett, D. L., et al. (2023) · The Lancet Oncology · doi:10.1016/s1470-2045(23)00412-6 · Cited by 107
Small language models learn enhanced reasoning skills from medical textbooks
Kim, H., Hwang, H., Lee, J., Park, S., Kim, D., Lee, T., Yoon, C., Sohn, J., Park, J., Reykhart, O., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01653-8 · Cited by 59
Capabilities of GPT‑5 on multimodal medical reasoning
Wang, S., Hu, M., Li, Q., Safari, M., Qiu, R. L. J., & Yang, X. (2026) · SPIE Medical Imaging: Imaging Informatics for Healthcare, Research, and Applications · doi:10.1117/12.3086794 · Cited by 125
“• USMLE Self Assessment [28] from official practice materials provided by the U.S. Medical Licensing Examination (USMLE) program * .”
Zhang, Y., He, Z., Li, J., Lin, J., Guan, Q., & Yu, W. (2024) · Cartography and Geographic Information Science · doi:10.1080/15230406.2024.2404868 · Cited by 116
“…application in more intricate (Hadi et al., 2023), such as medical education, radiologic decision-making, clinical genetics in the medical field (Kung et al., 2023; Sallam, 2023), as well as risk assessment, algorithmic trading, and market prediction in the finance sector (S. Wu et al., 2023).”
Kolding, S., Lundin, R. M., Hansen, L., & Østergaard, S. D. (2024) · Acta Neuropsychiatrica · doi:10.1017/neu.2024.50 · Cited by 114
“…many aspects of modern society, including the field of medicine (Haug and Drazen, 2023), where it may aid, e.g., training of medical professionals (Kung et al., 2023), informing/educating patients (Ayers et al., 2023), diagnostic processes (Lee, et al ., 2023), clinical note taking/summarization…”
Revolutionizing Medical Education: Can ChatGPT Boost Subjective Learning and Expression?
Seetharaman, R. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01957-w · Cited by 106
“Chat-GPT can provide a safe and supportive environment for students to practice subjective expression of their knowledge, helping them to become competent and skilled medical professionals [6].”
BackgroundThüs, D., Malone, S., & Brünken, R. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1474892 · Cited by 66
ChatGPT in Clinical Toxicology
Sabry Abdel-Messih, M., & Kamel Boulos, M. N. (2023) · JMIR Medical Education · doi:10.2196/46876 · Cited by 95
The Role of Artificial Intelligence in the Future of Pharmacy Education
Cain, J., Malcom, D. R., & Aungst, T. D. (2023) · American Journal of Pharmaceutical Education · doi:10.1016/j.ajpe.2023.100135 · Cited by 80
ChatGPT as an important tool in organizational management: A review of the literature
Ayinde, L., Wibowo, M. P., Ravuri, B., & Emdad, F. B. (2023) · Business Information Review · doi:10.1177/02663821231187991 · Cited by 75
Cheong, R. C. T., Unadkat, S. N., Mcneillis, V., Williamson, A., Joseph, J., Randhawa, P. S., Andrews, P. J., & Paleri, V. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08319-9 · Cited by 99
ChatGPT: A brief narrative review
Gupta, B., Mufti, T., Sohail, S. S., & Madsen, D. Ø. (2023) · Cogent Business & Management · doi:10.1080/23311975.2023.2275851 · Cited by 107
“Studies also show that ChatGPT performs increasingly well on different entrance and standardized exams and tests across academic subjects (Gilson et al., 2022; Huh, 2023; Kung et al., 2023; Wood et al., 2023).”
BackgroundChatCAD+: Toward a Universal and Reliable Interactive CAD Using LLMs
Zhao, Z., Wang, S., Gu, J., Zhu, Y., Mei, L., Zhuang, Z., Cui, Z., Wang, Q., & Shen, D. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3398350 · Cited by 75
“Notably, ChatGPT, which did not receive specific medical training, successfully passed all three parts of the USMLE and achieved an overall accuracy of over 50% across all exams, with several exams surpassing 60% accuracy [18].”
BackgroundFatima, A., Shafique, M. A., Alam, K., Fadlalla Ahmed, T. K., & Mustafa, M. S. (2024) · Medicine · doi:10.1097/md.0000000000039250 · Cited by 81
“[10,11] Regarding healthcare education and scientific writing, ChatGPT also exhibits bright, revolutionary potential to enhance and accelerate the processes of teaching, learning, and writing processes.”
BackgroundMethodInfluentialEvaluating the effectiveness of biomedical fine-tuning for large language models on clinical tasks
Dorfner, F. J., Dada, A., Busch, F., Makowski, M. R., Han, T., Truhn, D., Kleesiek, J., Sushil, M., Adams, L. C., & Bressem, K. K. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf045 · Cited by 113
Derraz, B., Breda, G., Kaempf, C., Baenke, F., Cotte, F., Reiche, K., Köhl, U., Kather, J. N., Eskenazy, D., & Gilbert, S. (2024) · npj Precision Oncology · doi:10.1038/s41698-024-00517-w · Cited by 94
“Large language model (LLM)- based CDS approaches have recently been proposed including in the area of oncology 48 – 50 , which show impressive but variable performance 51,52 .”
MethodAn overview and a roadmap for artificial intelligence in hematology and oncology
Rösler, W., Altenbuchinger, M., Baeßler, B., Beißbarth, T., Beutel, G., Bock, R., von Bubnoff, N. C. C., Eckardt, J., Foersch, S., Loeffler, C. M. L., et al. (2023) · Journal of Cancer Research and Clinical Oncology · doi:10.1007/s00432-023-04667-5 · Cited by 102
“Modern LLMs can converse like humans, can respond to questions in medical examinations (Kung, et al. 2022) and generally can be used as a search tool, in particular to answer medical questions.”
BackgroundDo ChatGPT and Other AI Chatbots Pose a Cybersecurity Risk?: An Exploratory Study
Sebastian, G. (2023) · International Journal of Security and Privacy in Pervasive Computing · doi:10.4018/ijsppc.320225 · Cited by 51
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S. H., & Ridge, P. G. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011511 · Cited by 69
Harnessing large vision and language models in agriculture: a review
Zhu, H., Qin, S., Su, M., Lin, C., Li, A., & Gao, J. (2025) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1579355 · Cited by 120
Pal, S., Bhattacharya, M., Lee, S., & Chakraborty, C. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03306-x · Cited by 114
“They reported that it is powerful enough to reach the passing cutoff score [13].”
Fang, C., Wu, Y., Fu, W., Ling, J., Wang, Y., Liu, X., Jiang, Y., Wu, Y., Chen, Y., Zhou, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000397 · Cited by 91
Suárez, A., Jiménez, J., de Pedro, M. L., Vázquez, C. A., García, V. D., Sánchez, M. G., & Freire, Y. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.11.058 · Cited by 105
“In medicine, although ChatGPT is not specifically trained to answer questions related to this field, the model has been shown to outperform official exams such as the United States Medical Licensing Exam (USMLE) [14,15] or the examination for Internal Medical Residents in Spain [16].”
MethodCarlà, M. M., Gambini, G., Baldascino, A., Boselli, F., Giannuzzi, F., Margollicci, F., & Rizzo, S. (2024) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-024-06470-5 · Cited by 110
Metadata
- Authors
- Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- Venue
- PLOS Digital Health
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-02-09
- Volume
- 2
- Issue
- 2
- Pages
- e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.