Performance of ChatGPT on USMLE: Unlocking the Potential of Large Language Models for AI-Assisted Medical Education
Abstract
Artificial intelligence is gaining traction in more ways than ever before. The popularity of language models and AI-based businesses has soared since ChatGPT was made available to the general public via OpenAI. It is becoming increasingly common for people to use ChatGPT both professionally and personally. Considering the widespread use of ChatGPT and the reliance people place on it, this study determined how reliable ChatGPT can be for answering complex medical and clinical questions. Harvard University gross anatomy along with the United States Medical Licensing Examination (USMLE) questionnaire were used to accomplish the objective. The paper evaluated the obtained results using a 2-way ANOVA and posthoc analysis. Both showed systematic covariation between format and prompt. Furthermore, the physician adjudicators independently rated the outcome's accuracy, concordance, and insight. As a result of the analysis, ChatGPT-generated answers were found to be more context-oriented and represented a better model for deductive reasoning than regular Google search results. Furthermore, ChatGPT obtained 58.8% on logical questions and 60% on ethical questions. This means that the ChatGPT is approaching the passing range for logical questions and has crossed the threshold for ethical questions. The paper believes ChatGPT and other language learning models can be invaluable tools for e-learners; however, the study suggests that there is still room to improve their accuracy. In order to improve ChatGPT's performance in the future, further research is needed to better understand how it can answer different types of questions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Answers generated by ChatGPT are more context-aware with better deductive reasoning abilities compared to Google search results.
“Moreover, Sharma et al. [173] found that answers generated by ChatGPT are more context-aware with better deductive reasoning abilities compared to Google search results.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [173]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 17
- Choi, J. H. a. H., Kristin E. and Monahan, Amy and Schwarcz, Daniel B. (2023). ChatGPT Goes to Law School. Minnesota Legal Studies Research Paper No. 23-03. https://doi.org/http://dx.doi.org/10.2139/ssrn.4335905 · DOI
- Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2022). How Does ChatGPT Perform on the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. medRxiv, 2022.2012.2023.22283901. https://doi.org/10.1101/2022.12.23.22283901 · DOI · Paper page
- Kasneci, E., Seßler, K., Kuechemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., & Kasneci, G. (2023). ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education. 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274 · DOI · Paper page
- Khanna, A., Pandey, B., Vashishta, K., Kalia, K., Bhale, P., & Das, T. (2015). A Study of Today's A.I. through Chatbots and Rediscovery of Machine Intelligence. International Journal of u-and e- Service, Science and Technology, 8, 277-284. https://doi.org/10.14257/ijunesst.2015.8.7.28 · DOI · Paper page
- Kung, T., Cheatham, M., Medinilla, A., Sillos, C., Leon, L., Elepano, C., Madriaga, M., Aggabao, R., Diaz- Candido, G., Maningo, J., & Tseng, V. (2022). Performance of ChatGPT on USMLE: Potential for AI- Assisted Medical Education Using Large Language Models. https://doi.org/10.1101/2022.12.19.22283643 · DOI · Paper page
- Susnjak, T. (2022). ChatGPT: The End of Online Exam Integrity? https://doi.org/10.48550/arXiv.2212.09292 · DOI · Paper page
- Terwiesch, C. (2023). Would Chat GPT3 Get a Wharton MBA? A Prediction Based on Its Performance in the Operations Management Course. In U. o. P. Mack Institute for Innovation Management at the Wharton School (Ed.).
- Thurzo, A., Strunga, M., Urban, R., Surovkova, J., & ???????????? (2023). Impact of Artificial Intelligence on Dental Education: A Review and Guide for Curriculum Update. Education Sciences, 13, 150. https://doi.org/10.3390/educsci13020150 · DOI
- Wallace, R. S. (2009). The Anatomy of A.L.I.C.E. In R. Epstein, G. Roberts, & G. Beber (Eds.), Parsing the Turing Test: Philosophical and Methodological Issues in the Quest for the Thinking Computer (pp. 181- 210). Springer Netherlands. https://doi.org/10.1007/978-1- 4020-6710-5_13 · DOI
- Weizenbaum, J. (1966). ELIZA-a computer program for the study of natural language communication between man and machine. Commun. ACM, 9(1), 36-45. https://doi.org/10.1145/365153.365168 · DOI · Paper page
- Wu, B. G. a. X. Z. a. Z. W. a. M. J. a. J. N. a. Y. D. a. J. Y. a. Y. (2023). How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. ArXiv.
- Abdul-Kader, S. A., & Woods, J. C. (2015). Survey on chatbot design techniques in speech conversation systems. International Journal of Advanced Computer Science and Applications, 6(7).
- De Angeli, A., & Carpenter, R. (2005). Stupid computer! Abuse and social identities. In Proc. INTERACT 2005 workshop Abuse: The darker side of Human-Computer Interaction (No. 4, pp. 19-25).
- Zhang, C., Zhang, C., Li, C., Qiao, Y., Zheng, S., Dam, S. K., ... & Hong, C. S. (2023). One small step for generative ai, one giant leap for agi: A complete survey on chatgpt in aigc era. arXiv preprint arXiv:2304.06488. · arXiv
- Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. 2020. Language Models Are Few- shot Learners. arXiv preprint arXiv:2005.14165 (2020) · arXiv
- Zarifhonarvar, A. (2023). Economics of chatgpt: A labor market view on the occupational impact of artificial intelligence. Available at SSRN 4350925.
- Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Available at SSRN 4337484.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
- Authors
- Sharma, P., Thapa, K., Thapa, D., Dhakal, P., Upadhaya, M. D., Adhikari, S., & Khanal, S. R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- arXiv
- 2307.00112
- References
- 0
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
