How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. Objective: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. Methods: We used 2 sets of multiple-choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT’s performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Results: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT’s answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. Conclusions: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT’s capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Template of question posed to each large language model (LLM), including both AMBOSS Attending Tip and the response from Chat Generative Pre-trained Transformer (ChatGPT). The correct answer to this question is “E. Zidovudine (AZT).” In the case of GPT-3, prompt engineering was necessary, with: "Please answer this multiple choice question:" + question as described previously + "Correct answer is." As GPT-3 is inherently a nondialogic model, this was necessary to reduce model hallucinations and force a clear answer [17].](https://figures.cochat.ai/paper-figures/583a8f9972504138a5ac67eaf322285a/how-does-chatgpt-perform-on-the-united-states-medical-licensing-examination-figure-1.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“where LLMs score well above passing8–10,17–20”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs, like GPT-4, have shown medical knowledge despite lacking medicine-specific training [10, 29].”
From Generative AI in healthcare: an implementation science informed translational path on application, integration and governance (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, several valid concerns were raised, including the possibility of generating inaccurate content, along with ethical issues, including the risk of bias, plagiarism, and copyright issues [9,18,40,41].”
From Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study (2023) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, analyses of ChatGPT show structured, non-random response patterns consistent with trait-like reasoning abilities18.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the majority of clinical reasoning evaluations on LLMs have been conducted using standardized exams (the United States Medical Licensing Examination) or digital case reports [14,19].”
From Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (2025) · cites this paper as [14]
Experimental — accuracy not yet measured
39 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Springer, L., Stanne, M. E., & Donovan, S. S. (1999) · Review of Educational Research · doi:10.3102/00346543069001021 · Cited by 1,943 · cited 1×
“Small group education has been shown to be a highly efficacious method of teaching [23,24].”
MethodThe use of reflection in medical education: AMEE Guide No. 44
Sandars, J. (2009) · Medical Teacher · doi:10.1080/01421590903050374 · Cited by 1,085
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
Research works printed but not linked: 2
- 10.48550/arxiv.2108.01764 · DOI
- Wang, DC, 2020, Yale J Biol Med, 93, 441
Cited by
Items 801–900 of 1,047
Accelerating medical education with ChatGPT: an implementation guide
Peacock, J. G., Austin, A. L., Shapiro, M., Battista, A., & Samuel, A. (2023) · MedEdPublish · doi:10.12688/mep.19732.1 · Cited by 27
Evaluation of ChatGPT's performance in providing treatment recommendations for pediatric diseases
Wei, Q., Wang, Y., Yao, Z., Cui, Y., Wei, B., Li, T., & Xu, X. (2023) · Pediatric Discovery · doi:10.1002/pdi3.42 · Cited by 27
Panagoulias, D. P., Palamidas, F. A., Virvou, M. K., & Tsihrintzis, G. A. (2023) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai59109.2023.00018 · Cited by 27
Sosa, B. R., Cung, M., Suhardi, V. J., Morse, K. W., Thomson, A. L., Yang, H. S., Iyer, S., & Greenblatt, M. B. (2024) · Journal of Orthopaedic Research® · doi:10.1002/jor.25782 · Cited by 27
Hamid, E. A. H. A., Maskur, H., & Mutalib, R. A. (2023) · Malaysian Journal of Information and Communication Technology (MyJICT) · doi:10.53840/myjict8-2-98 · Cited by 27
Prediction of outcomes after cardiac arrest by a generative artificial intelligence model
Amacher, S. A., Arpagaus, A., Sahmer, C., Becker, C., Groß, S., Urben, T., Tisljar, K., Sutter, R., Marsch, S. C. U., & Hunziker, S. (2024) · Resuscitation Plus · doi:10.1016/j.resplu.2024.100587 · Cited by 27
Apornvirat, S., Namboonlue, C., & Laohawetwanit, T. (2024) · American Journal of Clinical Pathology · doi:10.1093/ajcp/aqae036 · Cited by 27
ChatGPT’s adherence to otolaryngology clinical practice guidelines
Tessler, I., Wolfovitz, A., Alon, E. E., Gecel, N. A., Livneh, N., Zimlichman, E., & Klang, E. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-024-08634-9 · Cited by 27
Ardyansyah, A., Yuwono, A. B., Rahayu, S. L., Alsulami, N. M., & Sulistina, O. (2024) · Journal of Chemical Education · doi:10.1021/acs.jchemed.4c00220 · Cited by 27
Wang, S., Wang, Y., Jiang, L., Chang, Y., Zhang, S., Zhao, K., Chen, L., & Gao, C. (2025) · European journal of medical research · doi:10.1186/s40001-025-02296-x · Cited by 27
Evaluating the clinical benefits of LLMs
Bedi, S., Jain, S. S., & Shah, N. H. (2024) · Nature Medicine · doi:10.1038/s41591-024-03181-6 · Cited by 28
Epstein, R. H., & Dexter, F. (2023) · JMIR Medical Education · doi:10.2196/48305 · Cited by 26
Bonsu, E., & Baffour-Koduah, D. (2023) · doi:10.2139/ssrn.4387107 · Cited by 26
“, 2023; Hu, 2023; Pavlik, 2023) and empirical-based (Gilson et al., 2023; Khalil & Er, 2023; Tlili et al., 2023).”
BackgroundO uso da ferramenta ChatGPT no suporte à educação e à produção acadêmica
Celestino, M. S., & Valente, V. C. P. N. (2024) · ETD - Educação Temática Digital · doi:10.20396/etd.v26i00.8673464 · Cited by 26
Ahmed, F. R., Rushdan, E. E., Al‐Yateem, N., Almaazmi, A. N., Subu, M. A., Hijazi, H. H. A., Abdelbasset, W. K., Mottershead, R., Ahmed, A. A., & AbuRuz, M. E. (2024) · Teaching and learning in nursing · doi:10.1016/j.teln.2024.12.002 · Cited by 26
SDoH-GPT: using large language models to extract social determinants of health
Consoli, B. S., Wang, H., Wu, X., Wang, S., Zhao, X., Wang, Y., Rousseau, J. F., Hartvigsen, T., Shen, L., Wu, H., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf094 · Cited by 27
Ophthalmological Question Answering and Reasoning Using OpenAI o1 vs Other Large Language Models
Srinivasan, S., Ai, X., Zou, M., Zou, K., Kim, H., Lo, T. W. S., Pushpanathan, K., Yang, G. D., Goh, J. H. L., Kong, Y., et al. (2025) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2025.2413 · Cited by 26
Tracking ChatGPT Research: Insights From the Literature and the Web
Mubin, O., Alnajjar, F., Trabelsi, Z., Ali, L., Parambil, M. M. A., & Zou, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3356584 · Cited by 26
ChatGPT Yields a Passing Score on a Pediatric Board Preparatory Exam but Raises Red Flags
Le, M., & Davis, M. K. (2024) · Global Pediatric Health · doi:10.1177/2333794x241240327 · Cited by 26
Mert, S., Stoerzer, P., Brauer, J., Fuchs, B., Haas-Lützenberger, E. M., Demmer, W., Giunta, R. E., & Nürnberger, T. (2024) · Archives of Orthopaedic and Trauma Surgery · doi:10.1007/s00402-024-05298-2 · Cited by 26
RetractedAI Chatbots and Linguistic Injustice
Park, S. (2024) · Journal of Universal Language · doi:10.22425/jul.2024.25.1.99 · Cited by 26
The use of AI Chatbots in higher education: the problem of plagiarism
Prokhorova, Y., Gujrati, R., & Uygun, H. (2024) · Review of Artificial Intelligence in Education · doi:10.37497/rev.artif.intell.educ.v5i00.31 · Cited by 26
Expert assessment of ChatGPT’s ability to generate illness scripts: an evaluative study
Yanagita, Y., Yokokawa, D., Fukuzawa, F., Uchida, S., Uehara, T., & Ikusaka, M. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05534-8 · Cited by 17
Cahill, C., & McCabe, K. T. (2024) · PS Political Science & Politics · doi:10.1017/s1049096524000155 · Cited by 26
Zarei, M., Zarei, M., Hamzehzadeh, S., Oliyaei, S. S. B., & Hosseini, M. (2024) · Shiraz E-Medical Journal · doi:10.5812/semj-145840 · Cited by 26
Riding the Generative AI Tsunami: Addressing the Teaching and Learning Crisis in Higher Education
Tan, S., Rudolph, J., & Tan, S. (2024) · Springer eBooks · doi:10.1007/978-3-031-54509-2_8 · Cited by 26
Use of the artificial intelligence in teaching the concept of magnetism in preschool education
Samara, V., & Kotsis, K. Τ. (2024) · Journal of Digital Educational Technology · doi:10.30935/jdet/14864 · Cited by 26
Hayes, D. S., Foster, B. K., Makar, G. S., Manzar, S., Ozdag, Y., Shultz, M. J., Klena, J. C., & Grandizio, L. C. (2024) · Journal of surgical education · doi:10.1016/j.jsurg.2024.08.002 · Cited by 26
Snigdha, N. T., Batul, R., Karobari, M. I., Adil, A. H., Dawasaz, A. A., Hameed, M. S., Mehta, V., & Noorani, T. Y. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/1119816 · Cited by 26
A comparison of drug information question responses by a drug information center and by ChatGPT
Triplett, S., Ness-Engle, G. L., & Behnen, E. M. (2024) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxae316 · Cited by 26
Genovese, A., Prabha, S., Borna, S., Gomez-Cabello, C. A., Haider, S. A., Trabilsy, M., Tao, C., Aziz, K. T., Murray, P. M., & Forte, A. J. (2025) · Aesthetic Surgery Journal · doi:10.1093/asj/sjaf038 · Cited by 26
Taking Advice from ChatGPTPreprint
Zhang, P. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/b53vn · Cited by 26
Vaid, A., Lampert, J. M., Lee, J., Sawant, A. S., Apakama, D. U., Sakhuja, A., Soroush, A., Bick, S., Abbott, E., Gomez, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.02851 · Cited by 26
Jaleel, A., Aziz, U., Farid, G., Zahid Bashir, M., Mirza, T. R., Khizar Abbas, S. M., Aslam, S., & Sikander, R. M. H. (2025) · JMIR Medical Education · doi:10.2196/68070 · Cited by 25
Dolenc, K., & Brumen, M. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100285 · Cited by 25
Artificial Intelligence (ChatGPT) and Bloom’s Taxonomy in Theoretical Computer Science Education
Habiballa, H., Kotyrba, M., Volná, E., Bradáč, V., & Dusek, M. (2025) · Applied Sciences · doi:10.3390/app15020581 · Cited by 25
Revolution of AAV in Drug Discovery: From Delivery System to Clinical Application
Yin, L., He, H., Zhang, H., Shang, Y., Fu, C., Wu, S., & Jin, T. (2025) · Journal of Medical Virology · doi:10.1002/jmv.70447 · Cited by 26
Evaluation of performance of generative large language models for stroke care
Lee, J. T., Li, V. C.-S., Wu, J.-J., Chen, H.-H., Su, S. S.-Y., Chang, B. P.-H., Lai, R. L., Liu, C.-H., Chen, C.-T., Tanapima, V., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01830-9 · Cited by 25
Foundation models for radiology—the position of the AI for Health Imaging (AI4HI) network
de Almeida, J. G., Alberich, L. C., Tsakou, G., Marias, K., Tsiknakis, M., Lekadir, K., Martí‐Bonmatí, L., & Papanikolaou, N. K. (2025) · Insights into Imaging · doi:10.1186/s13244-025-02056-9 · Cited by 26
Nanua, S., Steward, R., Neely, B., Datto, M., & Youens, K. E. (2025) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2025.100520 · Cited by 25
Kasagga, A., Sapkota, A., Changaramkumarath, G., Abucha, J. M., Wollel, M. M., Somannagari, N., Husami, M. Y., Hailu, K. T., & Kasagga, E. (2025) · Cureus · doi:10.7759/cureus.94300 · Cited by 26
Hassanein, F. E. A., Hussein, R. R., Ahmed, Y., Elguindy, J., ahmed, D., & Abou‐Bakr, A. (2026) · BMC Oral Health · doi:10.1186/s12903-026-07665-4 · Cited by 25
What’s Important: The Next Academic—ChatGPT AI?
Bi, A. S. (2023) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.23.00269 · Cited by 25
The potential application of ChatGPT in gastrointestinal pathology
Ma, Y. (2023) · Gastroenterology & Endoscopy · doi:10.1016/j.gande.2023.05.002 · Cited by 25
Clinical Decision Support: Moving Beyond Interruptive “Pop-up” Alerts
Sangal, R. B., Sharifi, M., Rhodes, D. J., & Melnick, E. R. (2023) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2023.05.025 · Cited by 25
Reyes-Moreno, E. R., Londoño-Gallego, J. A., Andrade-Martelo, I. C., Villar-Vega, H. F., & Maldonado, J. J. C. (2023) · Aibi revista de investigación administración e ingeniería · doi:10.15649/2346030x.3245 · Cited by 25
Sato, H., & Ogasawara, K. (2024) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2024.21.4 · Cited by 25
GPT-4 Performance for Neurologic Localization
Lee, J.-H., Choi, E., McDougal, R. A., & Lytton, W. W. (2024) · Neurology Clinical Practice · doi:10.1212/cpj.0000000000200293 · Cited by 25
Fukuda, H., Morishita, M., Muraoka, K., Yamaguchi, S., Nakamura, T., Yoshioka, I., Awano, S., & Ono, K. (2024) · Journal of Dental Sciences · doi:10.1016/j.jds.2024.06.015 · Cited by 25
Is ChatGPT Enhancing Youth’s Learning, Engagement and Satisfaction?
Shah, C. S., Mathur, S., & Vishnoi, S. K. (2024) · Journal of Computer Information Systems · doi:10.1080/08874417.2024.2380698 · Cited by 25
Al-Naser, Y., Halka, F., Ng, B., Mountford, D., Sharma, S., Niure, K., Yong-Hing, C. J., Khosa, F., & der Pol, C. V. (2024) · Academic Radiology · doi:10.1016/j.acra.2024.08.009 · Cited by 25
Dzuali, F., Seiger, K., Novoa, R. A., Aleshin, M. A., Teng, J. M. C., Lester, J., & Daneshjou, R. (2024) · JMIR Medical Education · doi:10.2196/51435 · Cited by 25
Artificial Intelligence in Anatomy Teaching and Learning: A Literature Review
Pandurangam, G., Gurajala, S., & Nagajyothi, D. (2024) · National Journal of Clinical Anatomy · doi:10.4103/njca.njca_103_24 · Cited by 25
Performance of ChatGPT and Dental Students on Concepts of Periodontal Surgery
Li, C., Zhang, J., Abdul‐Masih, J., Zhang, S., & Yang, J. (2024) · European Journal Of Dental Education · doi:10.1111/eje.13047 · Cited by 25
Nakaura, T., Yoshida, N., Kobayashi, N., Nagayama, Y., Uetani, H., Kidoh, M., Oda, S., Funama, Y., & Hirai, T. (2024) · Academic Radiology · doi:10.1016/j.acra.2024.10.035 · Cited by 25
Ma, M. D., Ye, C., Yan, Y., Wang, X., Ping, P., Chang, T. S., & Wang, W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.09923 · Cited by 25
Yan, Z., Fan, K.-Q., Zhang, Q., Wu, X., Chen, Y., Wu, X., Yu, T., Su, N., Zou, Y., Chi, H., et al. (2025) · World Journal of Urology · doi:10.1007/s00345-025-05757-4 · Cited by 24
Large language models in physical therapy: time to adapt and adept
Naqvi, W. M., Shaikh, S. Z., & Mishra, G. V. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1364660 · Cited by 24
Zeljković, I., Novák, A., Lisičić, A., Jordán, A., Šerman, A., Jurin, I., Pavlović, N., & Manola, Š. (2025) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2025.01.036 · Cited by 24
Performance of artificial intelligence chatbots in National dental licensing examination
Lin, C. C.-C., Sun, J., Chang, C.-H., Chang, Y.-H., & Chang, J. Z. (2025) · Journal of Dental Sciences · doi:10.1016/j.jds.2025.05.012 · Cited by 24
AcuKG: a comprehensive knowledge graph for medical acupuncture
Li, Y., Peng, X., Peng, S., Li, J., Pei, D., Zhang, Q., Lu, Y., Hu, Y., Li, F., Zhou, L., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf179 · Cited by 24
Roeschl, T., Hoffmann, M., Hashemi, D., Rarreck, F., Hinrichs, N., Trippel, T. D., Gröschel, M. I., Unbehaun, A., Klein, C., Kempfert, J., et al. (2025) · JMIRx Med · doi:10.2196/74899 · Cited by 25
Zhu, Y., Gao, J., Wang, Z., Liao, W., Zheng, X., Liang, L., Bernabéu, M. O., Wang, Y., Yu, L., Pan, C., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02539-z · Cited by 24
Exploring the Capabilities and Possible Applications of Large Language Models for Education
Čávojský, M., Bugár, G., Kormaník, T., & Hasin, M. (2023) · International Conference on Emerging eLearning Technologies and Applications (ICETA) · doi:10.1109/iceta61311.2023.10344166 · Cited by 24
Desafiando el estado del arte en el uso de ChatGPT en educación en el año 2023
Saz-Pérez, F., & Pizà-Mir, B. (2023) · REIRE Revista d Innovació i Recerca en Educació · doi:10.1344/reire.44018 · Cited by 24
Bonfitto, G. R., Roletto, A., Savardi, M., Fasulo, S. V., Catania, D., & Signoroni, A. (2024) · Radiography · doi:10.1016/j.radi.2024.02.015 · Cited by 24
Iqbal, U., Lee, L. T., Rahmanti, A. R., Celi, L. A., & Li, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae067 · Cited by 24
Tsai, C.-Y., Hsieh, S.-J., Huang, H.-H., Deng, J., Huang, Y., & Cheng, P. (2024) · World Journal of Urology · doi:10.1007/s00345-024-04957-8 · Cited by 24
Li, B., Wang, A., Strachan, P., Séguin, J. A., Lachgar, S., Schroeder, K. C., Fleck, M. S., Wong, R., Karthikesalingam, A., Natarajan, V., et al. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651891 · Cited by 25
Kaya, K., Gietzen, C., Hahnfeldt, R., Zoubi, M. S. A., Emrich, T., Halfmann, M. C., Sieren, M. M., Elser, Y., Krumm, P., Brendel, J. M., et al. (2024) · Journal of Cardiovascular Magnetic Resonance · doi:10.1016/j.jocmr.2024.101068 · Cited by 24
Singh, R. G., & Ngai, C. S. B. (2024) · Discover Education · doi:10.1007/s44217-024-00211-w · Cited by 24
Holland, A. M., Lorenz, W. R., Cavanagh, J. C., Smart, N. J., Ayuso, S. A., Scarola, G. T., Kercher, K. W., Jørgensen, L. N., Janis, J. E., Fischer, J. P., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.25373 · Cited by 24
Jung, Y. S., Chae, Y. K., Kim, M. S., Lee, H., Choi, S. C., & Nam, O. H. (2024) · THE JOURNAL OF THE KOREAN ACADEMY OF PEDTATRIC DENTISTRY · doi:10.5933/jkapd.2024.51.3.299 · Cited by 24
Survey of Large Language Models for Answering Questions Across Various Fields
Shailendra, P., Ghosh, R. C., Kumar, R., & Sharma, N. (2024) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs60874.2024.10717078 · Cited by 24
SYNFAC-EDIT: Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization
Mishra, P., Yao, Z., Vashisht, P., Ouyang, F., Wang, B., Mody, V. D., & Hong, Y. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1120 · Cited by 25
Lubis, S., Nurmandi, A., Ahmad, J., Purnomo, E. P., Purwaningsih, T., & Jovita, H. (2025) · Frontiers in Sustainable Cities · doi:10.3389/frsc.2025.1553816 · Cited by 24
Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters
Exler, D., Schutera, M., Reischl, M., & Rettenberger, L. (2025) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.48550/arxiv.2505.04393 · Cited by 24
Performance of ChatGPT on Chinese master’s degree entrance examination in clinical medicine
Li, K.-C., Bu, Z.-J., Shahjalal, M., He, B.-X., Zhuang, Z.-F., Li, C., Liu, J.-P., Wang, B., & Liu, Z.-L. (2024) · PLOS ONE · doi:10.1371/journal.pone.0301702 · Cited by 23
Leitão, C. A., Salvador, G. L. D. O., Rabelo, L. M., & Escuissato, D. L. (2024) · Radiologia Brasileira · doi:10.1590/0100-3984.2023.0083 · Cited by 23
Albalawi, F., Khanagar, S. B., Iyer, K., Alhazmi, N., Alayyash, A., Alhazmi, A. S., Awawdeh, M. A., & Singh, O. G. (2025) · Applied Sciences · doi:10.3390/app15020893 · Cited by 23
Salmanpour, F., & Akpınar, M. (2025) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2025.02.017 · Cited by 23
Revolutionizing Online Learning: The Potential of ChatGPT in Massive Open Online Courses
Alseddiqi, M., Al–Mofleh, A., Albalooshi, L., & Najam, O. (2023) · European Journal of Education and Pedagogy · doi:10.24018/ejedu.2023.4.4.686 · Cited by 23
Performance of ChatGPT in French language Parcours d'Accès Spécifique Santé test and in OBGYN
Guigue, P., Meyer, R., Thivolle‐Lioux, G., Brezinov, Y., & Levin, G. (2024) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.15083 · Cited by 20
Assessment of ChatGPT-3.5's Knowledge in Oncology: Comparative Study with ASCO-SEP Benchmarks
Odabashian, R., Bastin, D., Jones, G., Manzoor, M., Tangestaniapour, S., Assad, M., Lakhani, S., Odabashian, M., & McGee, S. (2024) · JMIR AI · doi:10.2196/50442 · Cited by 18
Generative AI in Terms of Its Ethical Problems for Both Teachers and Learners: Striking a Balance
Le-Nguyen, H.-T., & Tran, T. T. (2023) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-0074-9.ch006 · Cited by 23
OpenAI's GPT‐4 performs to a high degree on board‐style dermatology questions
Elias, M. L., Burshtein, J. G., & Sharon, V. R. (2023) · International Journal of Dermatology · doi:10.1111/ijd.16913 · Cited by 23
Marshall, R. F., Mallem, K., Xu, H., Thorne, J. E., Burkholder, B. M., Chaon, B., Liberman, P., & Berkenstock, M. (2024) · Ocular Immunology and Inflammation · doi:10.1080/09273948.2024.2317417 · Cited by 23
Evaluation of ChatGPT for Pelvic Floor Surgery Counseling
Johnson, C. M., Bradley, C. S., Kenne, K. A., Rabice, S. R., Takacs, E. B., Vollstedt, A. J., & Kowalski, J. T. (2024) · Urogynecology · doi:10.1097/spv.0000000000001459 · Cited by 23
Xue, E., Bracken-Clarke, D., Iannantuono, G. M., Choo-Wosoba, H., Gulley, J. L., & Floudas, C. S. (2024) · Journal of Medical Internet Research · doi:10.2196/54758 · Cited by 22
Emdad, F. B., Ravuri, B., Ayinde, L. A., & Rahman, M. I. (2024) · IEEE International Conference on Interdisciplinary Approaches in Technology and Management for Social Innovation (IATMSI) · doi:10.1109/iatmsi60426.2024.10502836 · Cited by 23
Tong, W., Zhang, X., Zeng, H., Pan, J., Gong, C., & Zhang, H. (2024) · JMIR Medical Education · doi:10.2196/48594 · Cited by 23
Parental concerns about oral health of children: Is ChatGPT helpful in finding appropriate answers?
Gugnani, N., Pandit, I. K., Gupta, M., Gugnani, S., & Kathuria, S. (2024) · Journal of Indian Society of Pedodontics and Preventive Dentistry · doi:10.4103/jisppd.jisppd_110_24 · Cited by 23
Naqvi, H. A., Delungahawatta, T., Atarere, J. O., Bandaru, S. K., Barrow, J., & Mattar, M. C. (2024) · European Journal of Gastroenterology & Hepatology · doi:10.1097/meg.0000000000002815 · Cited by 23
Prompt matters: evaluation of large language model chatbot responses related to Peyronie’s disease
Warren, C. J., Edmonds, V. S., Payne, N. G., Voleti, S. S., Wu, S. Y., Colquitt, J., Sadeghi‐Nejad, H., & Punjani, N. (2024) · Sexual Medicine · doi:10.1093/sexmed/qfae055 · Cited by 23
Alsayed, A. A., Aldajani, M. B., Aljohani, M., Alamri, H., Alwadi, M. A. M., Alshammari, B. Z., & Alshammari, F. R. (2024) · The Saudi Dental Journal · doi:10.1016/j.sdentj.2024.09.009 · Cited by 23
Telemedicine in nephrology: future perspective and solutions
Sugawara, Y., Hirakawa, Y., & Nangaku, M. (2024) · Clinical Kidney Journal · doi:10.1093/ckj/sfae267 · Cited by 23
Applications and Future Prospects of Medical LLMs: A Survey Based on the M-KAT Conceptual Framework
Chang, Y., Yin, J., Li, J., Liu, C., Cao, L., & Lin, S. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02132-5 · Cited by 23
Street-Level Bureaucrats & AI Interactions in Public Organizations: An Identity Based Framework
Jung, H., & Camarena, L. (2024) · Public Performance & Management Review · doi:10.1080/15309576.2024.2447352 · Cited by 23
Zhou, B., Pek, L. S., Wang, C., Tiannan, L., Krishnasamy, H. N., Ne’matullah, K. F., & Arar, H. (2025) · International Journal of Learning Teaching and Educational Research · doi:10.26803/ijlter.24.3.9 · Cited by 23
RobotGPT: From ChatGPT to Robot IntelligencePreprint
He, H. (2023) · doi:10.36227/techrxiv.22569247 · Cited by 23
Metadata
- Authors
- Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D.
- Affiliations
- Yale University (US)
- University College Dublin (IE)
- Corresponding author
- Chartash, David
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-02-08
- Volume
- 9
- Pages
- e45312
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC9947764
- OpenAlex
- W4319460874
- Semantic Scholar
- cd4d112f3f9120d0715f22a9de2ce4720822368c
- Cited by
- 2,078
- References
- 21
- Citations by year
- 2026: 351 · 2025: 550 · 2024: 699 · 2023: 474
- FWCI
- 82.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Yale University
- National Institutes of Health (UL1TR001863)
- National Heart, Lung, and Blood Institute (T35HL007649)
- National Institute of Diabetes and Digestive and Kidney Diseases (T35DK104689)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE Step 1, USMLE Step 2, medical licensing examinations, question difficulty, GPT-3, medical education, knowledge assessment, AMBOSS, National Board of Medical Examiners, artificial intelligence in medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.