How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. Objective: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. Methods: We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Results: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. Conclusions: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT’s capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Template of question posed to each large language model (LLM), including both AMBOSS Attending Tip and the response from Chat Generative Pre-trained Transformer (ChatGPT). The correct answer to this question is “E. Zidovudine (AZT).” In the case of GPT-3, prompt engineering was necessary, with: "Please answer this multiple choice question:" + question as described previously + "Correct answer is." As GPT-3 is inherently a nondialogic model, this was necessary to reduce model hallucinations and force a clear answer [17].](https://figures.cochat.ai/paper-figures/583a8f9972504138a5ac67eaf322285a/how-does-chatgpt-perform-on-the-united-states-medical-licensing-examination-figure-1.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, like GPT-4, have shown medical knowledge despite lacking medicine-specific training [10, 29].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, several valid concerns were raised, including the possibility of generating inaccurate content, along with ethical issues, including the risk of bias, plagiarism, and copyright issues [9,18,40,41].”
From Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study (2023) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, analyses of ChatGPT show structured, non-random response patterns consistent with trait-like reasoning abilities18.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the majority of clinical reasoning evaluations on LLMs have been conducted using standardized exams (the United States Medical Licensing Examination) or digital case reports [14,19].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [14]
Experimental — accuracy not yet measured
39 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Springer, L., Stanne, M. E., & Donovan, S. S. (1999) · Review of Educational Research · doi:10.3102/00346543069001021 · Cited by 1,943 · cited 1×
“Small group education has been shown to be a highly efficacious method of teaching [23,24].”
MethodThe use of reflection in medical education: AMEE Guide No. 44
Sandars, J. (2009) · Medical Teacher · doi:10.1080/01421590903050374 · Cited by 1,085
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
Research works printed but not linked: 2
- 10.48550/arxiv.2108.01764 · DOI
- Wang, DC, 2020, Yale J Biol Med, 93, 441
Cited by
Items 701–800 of 1,047
Use of a large language model (LLM) for ambulance dispatch and triage
Shekhar, A. C., Kimbrell, J. M., Saharan, A., Stebel, J., Ashley, E. R., & Abbott, E. E. (2024) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2024.12.032 · Cited by 32
Ko, J. S., Heo, H., Suh, C. H., Yi, J., & Shim, W. H. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2024.1161 · Cited by 33
Chauhan, A., Khaliq, F., & Nayak, K. R. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00471-z · Cited by 32
Applications of ChatGPT and Large Language Models in Medicine and Health Care: Benefits and Pitfalls
Borkowski, A., Jakey, C. E., Mastorides, S., Kraus, A. L., Vidyarthi, G., Viswanadhan, N., & Lezama, J. L. (2023) · Federal Practitioner · doi:10.12788/fp.0386 · Cited by 32
Mannam, S. S., Subtirelu, R. C., Chauhan, D., Ahmad, H. S., Matache, I., Bryan, K., Chitta, S., Bathula, S. C., Turlip, R., Wathen, C. A., et al. (2023) · World Neurosurgery · doi:10.1016/j.wneu.2023.10.043 · Cited by 32
Gao, Z., Li, L., Ma, S., Wang, Q., Hemphill, L., & Xu, R. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03385-w · Cited by 33
Goyanes, M., & Lopezosa, C. (2024) · Anuario ThinkEPI · doi:10.3145/thinkepi.2024.e18a04 · Cited by 32
Lin, S.-Y., Chan, P. K., Hsu, W., & Kao, C. (2024) · Digital Health · doi:10.1177/20552076241237678 · Cited by 32
Shojaee-Mend, H., Mohebbati, R., Amiri, M., & Atarodi, A. (2024) · Scientific Reports · doi:10.1038/s41598-024-60405-y · Cited by 32
ChatGPT in early childhood STEM education: Can it be an innovative tool to overcome challenges?
Uğraş, H., & Uğraş, M. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12960-0 · Cited by 32
Villarreal‐Espinosa, J. B., Berreta, R. S., Allende, F., García, J. R., Ayala, S., Familiari, F., & Chahla, J. A. (2024) · The Knee · doi:10.1016/j.knee.2024.08.014 · Cited by 32
Exploring the Potential of Large Language Models in Personalized Diabetes Treatment StrategiesPreprint
Yang, H., Li, J., Liu, S., Du, L., Liu, X., Huang, Y., Shi, Q., & Liu, J. (2023) · Research Square · doi:10.1101/2023.06.30.23292034 · Cited by 32
Morjaria, L., Burns, L., Bracken, K., Ngo, Q. N., Lee, M., Levinson, A. J., Smith, J., Thompson, P., & Sibbald, M. (2023) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205231204178 · Cited by 31
“This lack of trend in ChatGPT response quality with question dif fi culty or complexity is in contrast with one of the USMLE studies 9 and warrants further focused research.”
ResultAvsheniuk, N., Lutsenko, O., Svyrydiuk, T., & Seminikhyna, N. (2024) · Arab World English Journal · doi:10.24093/awej/chatgpt.14 · Cited by 31
Assessment of Large Language Models (LLMs) in Decision-Making Support for Gynecologic Oncology
Gumilar, K. E., Indraprasta, B. R., Faridzi, A. S., Wibowo, B. M., Herlambang, A., Rahestyningtyas, E., Irawan, B., Tambunan, Z., Bustomi, A. F., Brahmantara, B. N., et al. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.10.050 · Cited by 31
How Well Did ChatGPT Perform in Answering Questions on Different Topics in Gross Anatomy?
Bolgova, O., Shypilova, I., Sankova, L., & Mavrych, V. (2023) · European Journal of Medical and Health Sciences · doi:10.24018/ejmed.2023.5.6.1989 · Cited by 31
Oliveira, J. A., Eskandar, K. O., Kar, D. E., de Oliveira, F. R., & Filho, A. L. D. S. (2024) · JMIR AI · doi:10.2196/64593 · Cited by 31
Nguyen, T. Y. P., Nguyen, N. T., & Phan, N. K. H. (2025) · International journal of AI in language education. · doi:10.54855/ijaile.25212 · Cited by 31
Artificial intelligence chatbots in endodontic education—Concepts and potential applications
Mohammad‐Rahimi, H., Setzer, F. C., Aminoshariae, A., Dummer, P. M. H., Duncan, H. F., & Nosrat, A. (2025) · International Endodontic Journal · doi:10.1111/iej.14231 · Cited by 31
Pavone, M., Goglia, M., Rosati, A., Innocenzi, C., Bizzarri, N., Seeliger, B., Mascagni, P., Ferrari, F. A., Forgione, A., Testa, A. C., et al. (2025) · Facts Views and Vision in ObGyn · doi:10.52054/fvvo.2024.13522 · Cited by 31
Caner, S. (2025) · SAGE Open · doi:10.1177/21582440251343340 · Cited by 31
AI tools such as ChatGPT will disrupt hydrology, too
Halloran, L. J. S., Mhanna, S., & Brunner, P. (2023) · Hydrological Processes · doi:10.1002/hyp.14843 · Cited by 31
Kleinig, O., Gao, C., & Bacchi, S. (2023) · Medical Journal of Australia · doi:10.5694/mja2.52061 · Cited by 28
Analyzing the Performance of ChatGPT About Osteoporosis
Çınar, Ç. (2023) · Cureus · doi:10.7759/cureus.45890 · Cited by 31
ChatGPT4PCG Competition: Character-like Level Generation for Science Birds
Taveekitworachai, P., Abdullah, F., Dewantoro, M. F., Thawonmas, R., Togelius, J., & Renz, J. (2023) · IEEE Conference on Games (CoG) · doi:10.1109/cog57401.2023.10333206 · Cited by 31
Çağlar, U., Yıldız, O., Meriç, A., Ayrancı, A., Yusuf, R., Sarılar, Ö., & Özgör, F. (2023) · Minerva Urology and Nephrology · doi:10.23736/s2724-6051.23.05450-2 · Cited by 31
Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability
Akyürek, A. F., Akyürek, E., Choshen, L., Wijaya, D. T., & Andreas, J. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.584 · Cited by 32
Rise of ChatGPT: It May Be Time to Reassess How We Teach and Test Radiology Residents
Lourenço, A. P., Slanetz, P. J., & Baird, G. L. (2023) · Radiology · doi:10.1148/radiol.231053 · Cited by 31
Chang, H., Liu, B., Zhao, Y., Li, Y., & He, F. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2024.2331646 · Cited by 30
Chen, L., Tophel, A., Hettiyadura, U., & Kodikara, J. K. (2024) · Geotechnics · doi:10.3390/geotechnics4020026 · Cited by 30
Ming, S., Guo, Q., Cheng, W., & Lei, B. (2024) · JMIR Medical Education · doi:10.2196/52784 · Cited by 30
Fervers, P., Hahnfeldt, R., Kottlors, J., Wagner, A., Maintz, D. C., dos Santos, D. P., Lennartz, S., & Persigehl, T. (2024) · Frontiers in Radiology · doi:10.3389/fradi.2024.1390774 · Cited by 30
Stadler, R. D., Sudah, S. Y., Moverman, M. A., Denard, P. J., Duralde, X. A., Garrigues, G. E., Klifto, C. S., Levy, J. C., Namdari, S., Sánchez‐Sotelo, J., et al. (2024) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2024.06.045 · Cited by 30
Sorenson, B., & Hanson, K. (2024) · Journal of Chemical Education · doi:10.1021/acs.jchemed.4c00165 · Cited by 30
ChatGPT’s Impact on Education and Healthcare: Insights, Challenges, and Ethical Consideration
Hamad, Z. T., Jamil, N., & Belkacem, A. N. (2024) · IEEE Access · doi:10.1109/access.2024.3437374 · Cited by 30
Performance of Publicly Available Large Language Models on Internal Medicine Board-style Questions
Tarabanis, C., Zahid, S., Mamalis, M. E., Zhang, K., Kalampokis, E., & Jankelson, L. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000604 · Cited by 30
Armitage, R. C. (2024) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.14250 · Cited by 30
Bibliometric Analysis on ChatGPT Research with CiteSpace
Nan, D., Zhao, X., Chen, C., Sun, S., Lee, K. R., & Kim, J. H. (2025) · Information · doi:10.3390/info16010038 · Cited by 30
Can Large Language Models Serve as Reliable Tools for Information in Dentistry? A Systematic Review
Alhazmi, N., Alshehri, A., BaHammam, F. A., Philip, M. R., Nadeem, M., & Khanagar, S. B. (2025) · International Dental Journal · doi:10.1016/j.identj.2025.04.015 · Cited by 30
Bužančić, I., Belec, D., Držaić, M., Kummer, I., Brkić, J., Fialová, D., & Hadžiabdić, M. O. (2023) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.15963 · Cited by 30
Yang, W., Lee, H., Wu, R., Zhang, R., & Pan, Y. (2023) · Electronics · doi:10.3390/electronics12234813 · Cited by 30
Scaioli, G., Moro, G. L., Conrado, F., Rosset, L., Bert, F., & Siliquini, R. (2023) · Annali dell Istituto Superiore di Sanità · doi:10.4415/ann_23_04_05 · Cited by 30
Bai, J. Y. H., Zawacki‐Richter, O., & Müskens, W. (2024) · Turkish Online Journal of Distance Education · doi:10.17718/tojde.1248901 · Cited by 30
Fisch, U., Kliem, P. S. C., Grzonka, P., & Sutter, R. (2024) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100978 · Cited by 31
Peled, T., Sela, H. Y., Weiss, A., Grisaru‐Granovsky, S., Agrawal, S., & Rottenstreich, M. (2024) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.15501 · Cited by 30
Use of ChatGPT in Urology and its Relevance in Clinical Practice: Is it useful?
Braga, A. V. N. M., Nunes, N. C., Santos, E. N., Veiga, M. L., Braga, A. A. N. M., de Abreu, G. E., de Bessa, J., Braga, L. H., Kirsch, A. J., & Barroso, U. (2024) · International braz j urol · doi:10.1590/s1677-5538.ibju.2023.0570 · Cited by 30
Meyer, A., Soleman, A., Riese, J., & Streichert, T. (2024) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2024-0246 · Cited by 30
Detection of ChatGPT fake science with the xFakeSci learning algorithm
Hamed, A. A., & Wu, X. (2024) · Scientific Reports · doi:10.1038/s41598-024-66784-6 · Cited by 30
Mukhlis, M. (2024) · Journal of Languages and Language Teaching · doi:10.33394/jollt.v12i3.11433 · Cited by 30
Moglia, A., Georgiou, K. E., Cerveri, P., Mainardi, L. T., Satava, R. M., & Cuschieri, A. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10849-5 · Cited by 30
Evaluating large language models in medical applications: a surveyPreprint
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.07468 · Cited by 30
Ajalo, E., Mukunya, D., Nantale, R., Kayemba, F., Pangholi, K., Babuya, J., Langoya Akuu, S., Namiiro, A. M., Nsubuga, Y. B., Mpagi, J. L., et al. (2025) · PLOS ONE · doi:10.1371/journal.pone.0313776 · Cited by 29
Kasthuri, V. S., Glueck, J., Pham, H., Daher, M., Balmaceno-Criss, M., McDonald, C. L., Diebo, B. G., & Daniels, A. H. (2024) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.23.00914 · Cited by 29
Alkhamees, A. (2024) · Cureus · doi:10.7759/cureus.65820 · Cited by 29
Koroleva, D. O., & Jogezai, N. A. (2024) · Information and Learning Sciences · doi:10.1108/ils-10-2023-0137 · Cited by 29
Evaluation of ChatGPT-4o’s answers to questions about hip arthroscopy from the patient perspective
Ayık, G., Ercan, N., Demirtaş, Y., Yıldırım, T., & Çakmak, G. (2024) · Joint Diseases and Related Surgery · doi:10.52312/jdrs.2025.1961 · Cited by 29
Joseph, G., Bhatti, N., Mittal, R., & Bhatti, A. (2025) · Cureus · doi:10.7759/cureus.77313 · Cited by 30
Liu, Q., Hu, A., Gladman, T., & Gallagher, S. (2025) · Innovative Higher Education · doi:10.1007/s10755-025-09790-4 · Cited by 29
Neittaanmäki, N. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1517116 · Cited by 29
Krumsvik, R. J. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1441747 · Cited by 29
A review of ChatGPT in medical education: exploring advantages and limitations
Cheng, Y., & Zhu, L. (2025) · International Journal of Surgery · doi:10.1097/js9.0000000000002505 · Cited by 29
Taşsöker, M. (2025) · BMC Oral Health · doi:10.1186/s12903-025-06444-x · Cited by 29
Li, L., Zhang, W., Zhang, K., Yang, Y., Wang, L., Zuo, L., Sun, Y., & Peng, Q. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07567-z · Cited by 29
Appropriateness of ophthalmic symptoms triage by a popular online artificial intelligence chatbot
Tsui, J. C., Wong, M. B., Kim, B. J., Maguire, A. M., Scoles, D., VanderBeek, B. L., & Brucker, A. J. (2023) · Eye · doi:10.1038/s41433-023-02556-2 · Cited by 29
ChatGPT underperforms in triaging appropriate use of Mohs surgery for cutaneous neoplasms
O’Hern, K., Yang, E., & Vidal, N. Y. (2023) · JAAD International · doi:10.1016/j.jdin.2023.06.002 · Cited by 29
Exhibiting the Heritage of COVID-19—A Conversation with ChatGPT
Spennemann, D. H. R. (2023) · Heritage · doi:10.3390/heritage6080302 · Cited by 29
Cung, M., Sosa, B. R., Yang, H. S., McDonald, M. M., Matthews, B. G., Veldhuis‐Vlug, A. G., Imel, E. A., Wein, M. N., Stein, E. M., & Greenblatt, M. B. (2024) · Journal of Bone and Mineral Research · doi:10.1093/jbmr/zjad007 · Cited by 29
Davies, N., Wilson, R., Winder, M. S., Tunster, S. J., McVicar, K., Thakrar, S., Williams, J., & Reid, A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05042-9 · Cited by 29
Potential applications of ChatGPT in obstetrics and gynecology in Korea: a review article
Lee, Y., & Kim, S. Y. (2024) · Obstetrics & Gynecology Science · doi:10.5468/ogs.23231 · Cited by 29
Evaluating ChatGPT ability to answer urinary tract Infection-Related questions
Çakır, H., Çağlar, U., Sekkeli, S., Zerdali, E. Y., Sarılar, Ö., Yıldız, O., & Özgör, F. (2024) · Infectious Diseases Now · doi:10.1016/j.idnow.2024.104884 · Cited by 29
ChatGPT for GTFS: benchmarking LLMs on GTFS semantics... and retrieval
Devunuri, S., Qiam, S., & Lehe, L. (2024) · Public Transport · doi:10.1007/s12469-024-00354-x · Cited by 29
Tran, C. G., Chang, J., Sherman, S. K., & Andrade, J. P. D. (2024) · Journal of Surgical Research · doi:10.1016/j.jss.2024.04.060 · Cited by 29
RetractedLaohawetwanit, T., Apornvirat, S., & Kantasiripitak, C. (2024) · American Journal of Clinical Pathology · doi:10.1093/ajcp/aqae062 · Cited by 29
Naureen, S., & Kiani, H. G. (2024) · Journal of College of Physicians And Surgeons Pakistan · doi:10.29271/jcpsp.2024.07.761 · Cited by 29
İş, E. E., & Menekşeoğlu, A. K. (2024) · Clinical Rheumatology · doi:10.1007/s10067-024-07154-5 · Cited by 30
Making a revolution in physics learning in high schools with ChatGPT: A case study in UAE
Alarbi, K., Halaweh, M., Tairab, H., Alsalhi, N. R., Annamalai, N., & Aldarmaki, F. (2024) · Eurasia Journal of Mathematics, Science and Technology Education · doi:10.29333/ejmste/14983 · Cited by 28
Chen, J., Cadiente, A., Kasselman, L. J., & Pilkington, B. (2024) · Journal of Medical Ethics · doi:10.1136/jme-2023-109366 · Cited by 28
Cao, J. J., Kwon, D. H., Ghaziani, T. T., Kwo, P., Tse, G., Kesselman, A., Kamaya, A., & Tse, J. R. (2024) · Abdominal Radiology · doi:10.1007/s00261-024-04501-7 · Cited by 28
Bosco, C., Otenen, E., Torres, J. O., Nguyen, V. V., Chheda, D., Peng, X., Jessup, N. M., Himes, A. K., Cureton, B., Lu, Y. Y.-F., et al. (2024) · JMIR Aging · doi:10.2196/60566 · Cited by 28
Healthcare ethics and artificial intelligence: a UK doctor survey
Warrington, D. J., & Holm, S. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-089090 · Cited by 28
Luordo, D., Arrese, M. T., Calvo, C., Shani, K. D. S., Cruz, L. M. R., Sánchez, F. J. G., Lagares, A., Rubio, R., Delgado, J. F., Carreras, M. P., et al. (2025) · Applied Sciences · doi:10.3390/app15031153 · Cited by 28
Roshani, M. A., Zhou, X., Qiang, Y., Suresh, S., Hicks, S. D., Sethuraman, U., & Zhu, D. (2025) · JMIR AI · doi:10.2196/67363 · Cited by 28
Ilicki, J. (2023) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2023.03.006 · Cited by 20
Yoshiyasu, Y., Wu, F. M., Dhanda, A. K., Gorelik, D., Takashima, M., & Ahmed, O. G. (2023) · International Forum of Allergy & Rhinology · doi:10.1002/alr.23201 · Cited by 28
People's Perceptions Toward Bias and Related Concepts in Large Language Models: A Systematic Review
Wang, L., Song, M., Rezapour, R., Kwon, B. C., & Huh, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.14504 · Cited by 29
Floyd, W., Kleber, T. J., Carpenter, D. J., Pasli, M., Qazi, J. J., Huang, C. C., Leng, J. X., Ackerson, B. G., Pierpoint, M., Salama, J. K., et al. (2023) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2023.10.020 · Cited by 28
Lee, G. U., Hong, D. Y., Kim, S. Y., Kim, J. W., Lee, Y. H., Park, S. O., & Lee, K. R. (2024) · Medicine · doi:10.1097/md.0000000000037325 · Cited by 28
ChatGPT performance on the American Shoulder and Elbow Surgeons maintenance of certification exam
Fiedler, B., Azua, E. N., Phillips, T. W., & Ahmed, A. S. (2024) · Journal of Shoulder and Elbow Surgery · doi:10.1016/j.jse.2024.02.029 · Cited by 28
Calle, P., Shao, R., Liu, Y., Hébert, E. T., Kendzor, D. E., Neil, J. M., Businelle, M. S., & Pan, C. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641965 · Cited by 28
Evaluation of AI Chatbots for Patient-Specific EHR Questions
Hamidi, A., & Roberts, K. E. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.02549 · Cited by 8
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2023) · arXiv · doi:10.48550/arxiv.2310.05694 · Cited by 27
Wang, Y., Chen, Y., & Sheng, J. (2024) · JMIR Medical Informatics · doi:10.2196/56426 · Cited by 27
Baskara, F. R., Vasudevan, A., Sain, Z. H., Tee, M., Vasumathi, A., Parahakaran, S., & Balakrishnan, R. (2024) · Journal of Infrastructure Policy and Development · doi:10.24294/jipd.v8i12.6385 · Cited by 27
Chan, V. C. (2025) · Journal of Professional Nursing · doi:10.1016/j.profnurs.2025.01.003 · Cited by 27
Wang, X., Ye, H., Zhang, S., Yang, M., & Wang, X. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02152-9 · Cited by 27
Fancong, W. A. N. G., Tang, X., & Yu, S. (2025) · Acta Psychologica Sinica · doi:10.3724/sp.j.1041.2025.0967 · Cited by 27
A survey on biomedical automatic text summarization with large language models
Huang, Z., Chen, X., Wang, Y., Huang, J., & Zhao, X. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104216 · Cited by 27
Wójcik, D., Adamiak, O., Czerepak, G., Tokarczuk, O. D., & Szalewski, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-17030-0 · Cited by 27
Yapay Zekâ Destekli Dil İşleme Teknolojisi Olan ChatGPT’nin Sağlık Hizmetlerinde Kullanımı
Yiğit, S., Berşe, S., & Dіrgar, E. (2023) · Eurasian Journal of Health Technology Assessment · doi:10.52148/ehta.1302000 · Cited by 27
Göktaş, L. S. (2023) · Journal of Tourism and Gastronomy Studies · doi:10.21325/jotags.2023.1224 · Cited by 27
Metadata
- Authors
- Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D.
- Affiliations
- Yale University (US)
- University College Dublin (IE)
- Corresponding author
- Chartash, David
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-02-08
- Volume
- 9
- Pages
- e45312
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC9947764
- OpenAlex
- W4319460874
- Semantic Scholar
- cd4d112f3f9120d0715f22a9de2ce4720822368c
- Cited by
- 2,078
- References
- 21
- Citations by year
- 2026: 351 · 2025: 550 · 2024: 699 · 2023: 474
- FWCI
- 82.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- National Heart, Lung, and Blood Institute (T35HL007649)
- National Institute of Diabetes and Digestive and Kidney Diseases (T35DK104689)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE Step 1, USMLE Step 2, medical licensing examinations, question difficulty, GPT-3, medical education, knowledge assessment, AMBOSS, National Board of Medical Examiners, artificial intelligence in medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.