Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models
Open accessPreprint — published as 10.1371/journal.pdig.0000198 · Paper page
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT performance on the United States Medical Licensing Exam was at or near the passing threshold without any domain fine-tuning.
“According to the evaluation results, the performance of ChatGPT on this test was at or near the passing threshold without any domain fine-tuning.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [28]
Large language models might be a powerful tool to assist medical education and eventually clinical decision-making processes.
“Based on these results, the authors argue that large language models might be a powerful tool to assist medical education and eventually clinical decision-making processes [28].”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [28]
Researchers in recent publications struggle with how they should report text generation [69] and policies vary from banning text generation to mandatory labeling of AI-based text generation [32].
“Echoing this, researchers in recent publications struggle with how they should report text generation [69] and policies vary from banning text generation to mandatory labeling of AI-based text generation [32].”
From The AI Ghostwriter Effect: When Users do not Perceive Ownership of AI-Generated Text but Self-Declare as Authors (2024) · cites this paper as [69]
The citation to this paper is attached to the claim that policies range from banning AI-generated text to requiring its labeling. The Introduction and Artificial Intelligence sections describe ChatGPT’s text generation, but the passages found do not address publication policies; searches for bans, disclosure, and labeling found no relevant passage. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 26 · Cited by 4,115
Research works printed but not linked: 22
- Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for com- puter vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE; 2016. https://doi.org/10.1109/cvpr.2016.308 · DOI · Paper page
- Zhang W, Feng Y, Meng F, You D, Liu Q. Bridging the gap between training and inference for neural machine translation. Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics. Stroudsburg, PA, USA: Association for Computational Linguistics; 2019. https://doi.org/10. 18653/v1/p19-1426 · DOI
- Bhatia Y, Bajpayee A, Raghuvanshi D, Mittal H. Image captioning using Google's inception-resnet-v2 and recurrent neural network. 2019 Twelfth International Conference on Contemporary Computing (IC3). IEEE; 2019. https://doi.org/10.1109/ic3.2019.8844921 · DOI
- McDermott MBA, Wang S, Marinsek N, Ranganath R, Foschini L, Ghassemi M. Reproducibility in machine learning for health research: Still a ways to go. Sci Transl Med. 2021;13. https://doi.org/10. 1126/scitranslmed.abb1655 PMID: 33762434 · DOI · Paper page
- Chen P-HC, Liu Y, Peng L. How to develop machine learning models for healthcare. Nat Mater. 2019; 18: 410-414. https://doi.org/10.1038/s41563-019-0345-0 PMID: 31000806 · DOI · Paper page
- Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and valida- tion of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016; 316: 2402. https://doi.org/10.1001/jama.2016.17216 PMID: 27898976 · DOI · Paper page
- Nagpal K, Foote D, Liu Y, Chen P-HC, Wulczyn E, Tan F, et al. Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. NPJ Digit Med. 2019; 2: 48. https:// doi.org/10.1038/s41746-019-0112-2 PMID: 31304394 · DOI · Paper page
- Liu Y, Jain A, Eng C, Way DH, Lee K, Bui P, et al. A deep learning system for differential diagnosis of skin diseases. Nat Med. 2020; 26: 900-908. https://doi.org/10.1038/s41591-020-0842-3 PMID: 32424212 9. [cited 26 Jan 2023]. Available: https://openai.com/blog/chatgpt/ 10 · DOI · Paper page
- Burk-Rafel J, Santen SA, Purkiss J. Study Behaviors and USMLE Step 1 Performance: Implications of a Student Self-Directed Parallel Curriculum. Acad Med. 2017; 92: S67-S74. https://doi.org/10.1097/ ACM.0000000000001916 PMID: 29065026 · DOI
- Lie ´vin V, Hother CE, Winther O. Can large language models reason about medical questions? arXiv [cs.CL]. 2022. Available: http://arxiv.org/abs/2207.08143 · arXiv
- Jin D, Pan E, Oufattole N, Weng W-H, Fang H, Szolovits P. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. arXiv [cs.CL]. 2020. Available: http://arxiv.org/abs/2009.13081 · arXiv
- Densen P. Challenges and opportunities facing medical education. Trans Am Clin Climatol Assoc. 2011; 122: 48-58. Available: https://www.ncbi.nlm.nih.gov/pubmed/21686208 PMID: 21686208
- Prasad V, Vandross A, Toomey C, Cheung M, Rho J, Quinn S, et al. A decade of reversal: an analysis of 146 contradicted medical practices. Mayo Clin Proc. 2013; 88: 790-798. https://doi.org/10.1016/j. mayocp.2013.05.012 PMID: 23871230 · DOI · Paper page
- Herrera-Perez D, Haslam A, Crain T, Gill J, Livingston C, Kaestner V, et al. A comprehensive review of randomized clinical trials in three medical journals reveals 396 medical reversals. Elife. 2019; 8. https:// doi.org/10.7554/eLife.45183 PMID: 31182188 · DOI
- Abou-Hanna JJ, Owens ST, Kinnucan JA, Mian SI, Kolars JC. Resuscitating the Socratic Method: Stu- dent and Faculty Perspectives on Posing Probing Questions During Clinical Teaching. Acad Med. 2021; 96: 113-117. https://doi.org/10.1097/ACM.0000000000003580 PMID: 33394663 · DOI
- Plana D, Shung DL, Grimshaw AA, Saraf A, Sung JJY, Kann BH. Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review. JAMA Netw Open. 2022; 5: e2233946. https://doi.org/10.1001/jamanetworkopen.2022.33946 PMID: 36173632 · DOI · Paper page
- Kan HJ, Kharrazi H, Chang H-Y, Bodycombe D, Lemke K, Weiner JP. Exploring the use of machine learning for risk adjustment: A comparison of standard and penalized linear regression models in pre- dicting health care costs in older adults. PLoS One. 2019; 14: e0213258. https://doi.org/10.1371/ journal.pone.0213258 PMID: 30840682 · DOI
- Delahanty RJ, Kaufman D, Jones SS. Development and Evaluation of an Automated Machine Learning Algorithm for In-Hospital Mortality Risk Adjustment Among Critical Care Patients. Crit Care Med. 2018; 46: e481-e488. https://doi.org/10.1097/CCM.0000000000003011 PMID: 29419557 · DOI · Paper page
- Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE- AI. Nat Med. 2022; 28: 924-933. https://doi.org/10.1038/s41591-022-01772-9 PMID: 35585198 · DOI · Paper page
- Garcia-Vidal C, Sanjuan G, Puerta-Alcalde P, Moreno-Garcı ´a E, Soriano A. Artificial intelligence to sup- port clinical decision-making processes. EBioMedicine. 2019; 46: 27-29. https://doi.org/10.1016/j. ebiom.2019.07.019 PMID: 31303500 · DOI
- Bala S, Keniston A, Burden M. Patient Perception of Plain-Language Medical Notes Generated Using Artificial Intelligence Software: Pilot Mixed-Methods Study. JMIR Form Res. 2020; 4: e16670. https:// doi.org/10.2196/16670 PMID: 32442148 · DOI
- Milne-Ives M, de Cock C, Lim E, Shehadeh MH, de Pennington N, Mole G, et al. The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review. J Med Internet Res. 2020; 22: e20346. https://doi.org/10.2196/20346 PMID: 33090118 · DOI · Paper page
- Authors
- Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V.
- Affiliations
- Harvard University (US)
- Brown University (US)
- Corresponding author
- Tseng, Victor
- ORCID
- Cheatham, Morgan
- Venue
- PLOS Digital Health
- Publisher
- openRxiv
- Published
- 2022-12-20
- Volume
- 2
- Issue
- 2
- Pages
- e0000198-e0000198
- ISSN
- 2767-3170 (ISSN-L)
- PMC
- PMC9931230
- OpenAlex
- W4319662928
- Cited by
- 4,115
- References
- 26
- Citations by year
- 2026: 802 · 2025: 1010 · 2024: 1213 · 2023: 1079 · 2022: 4 · 2021: 1 · 2013: 1 · 2012: 1
- FWCI
- 163.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- PLOS Digital Health (publishedVersion, cc-by)
- medRxiv (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, USMLE, medical licensing examinations, clinical decision-making, exam performance, model explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
