Can ChatGPT-3.5 Pass a Medical Exam? A Systematic Review of ChatGPT’s Performance in Academic Testing
Open accessAbstract
OBJECTIVE: We, therefore, aim to conduct a systematic review to assess the academic potential of ChatGPT-3.5, along with its strengths and limitations when giving medical exams. METHOD: Following PRISMA guidelines, a systemic search of the literature was performed using electronic databases PUBMED/MEDLINE, Google Scholar, and Cochrane. Articles from their inception till April 4, 2023, were queried. A formal narrative analysis was conducted by systematically arranging similarities and differences between individual findings together. RESULTS: After rigorous screening, 12 articles underwent this review. All the selected papers assessed the academic performance of ChatGPT-3.5. One study compared the performance of ChatGPT-3.5 with the performance of ChatGPT-4 when giving a medical exam. Overall, ChatGPT performed well in 4 tests, averaged in 4 tests, and performed badly in 4 tests. ChatGPT's performance was directly proportional to the level of the questions’ difficulty but was unremarkable on whether the questions were binary, descriptive, or MCQ-based. ChatGPT's explanation, reasoning, memory, and accuracy were remarkably good, whereas it failed to understand image-based questions, and lacked insight and critical thinking. CONCLUSION: ChatGPT-3.5 performed satisfactorily in the exams it took as an examinee. However, there is a need for future related studies to fully explore the potential of ChatGPT in medical education.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT performs well on standardized tasks but often fails in complex cognitive challenges when not paired with sound pedagogical design.
“A systematic review by Sumbal et al. [62] highlights that while ChatGPT performs well on standardized tasks, it often fails in complex cognitive challenges when not paired with sound pedagogical design.”
From Artificial Intelligence in Higher Education: A State-of-the-Art Overview of Pedagogical Integrity, Artificial Intelligence Literacy, and Policy Integration (2025) · cites this paper as [62]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.03.001 · Cited by 4,265 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 3× · see citing works
“15,22 This means that ChatGPT has limited knowledge regarding information present on visual diagrams.”
BackgroundGilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842 · see citing works
ChatGPT and other large language models are double-edged swords
Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023) · Radiology · doi:10.1148/radiol.230163 · Cited by 1,008 · see citing works
What ChatGPT and generative AI mean for science
Stokel-Walker, C., & Van Noorden, R. (2023) · Nature · doi:10.1038/d41586-023-00340-6 · Cited by 828 · see citing works
Can artificial intelligence help for scientific writing?
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04380-2 · Cited by 827 · see citing works
The rise of <scp>ChatGPT</scp> : Exploring its potential in medical education
Lee, H. (2024) · Anatomical Sciences Education · doi:10.1002/ase.2270 · Cited by 596 · see citing works
Generating scholarly content with ChatGPT: Ethical challenges for medical publishing
Liebrenz, M., Schleifer, R., Buadze, A., Bhugra, D., & Smith, A. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00019-5 · Cited by 594 · see citing works
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563 · see citing works
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.1 · Cited by 278 · see citing works
Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591703 · Cited by 275 · see citing works
Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations
Ali, R., Tang, O. Y., Connolly, I. D., Sullivan, P. L. Z., Shin, J. H., Fridley, J., Asaad, W. F., Cielo, D., Oyelese, A. A., Doberstein, C. E., et al. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002632 · Cited by 273 · see citing works
The exciting potential for ChatGPT in obstetrics and gynecology
Grünebaum, A., Chervenak, J., Pollet, S. L., Katz, A., & Chervenak, F. A. (2023) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2023.03.009 · Cited by 234 · see citing works
How does peer teaching compare to faculty teaching? A systematic review and meta-analysis
Rees, E. L., Quinn, P., Davies, B., et al. (2015) · Medical Teacher · doi:10.3109/0142159x.2015.1112888 · Cited by 223 · see citing works
Can ChatGPT pass the life support exams without entering the American heart association course?
Fijačko, N., Gosak, L., Štiglic, G., Picard, C. T., & John Douma, M. (2023) · Resuscitation · doi:10.1016/j.resuscitation.2023.109732 · Cited by 206 · see citing works
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.01 · Cited by 199 · see citing works
Massey, P. A., Montgomery, C., & Zhang, A. S. (2023) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-23-00396 · Cited by 194 · see citing works
Applicability of ChatGPT in Assisting to Solve Higher Order Problems in Pathology
Sinha, R. K., Deb Roy, A., Kumar, N., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.35237 · Cited by 187 · see citing works
Analysis of large-language model versus human performance for genetics questions
Duong, D., & Solomon, B. D. (2024) · European Journal of Human Genetics · doi:10.1038/s41431-023-01396-8 · Cited by 159 · see citing works
Analysis of large-language model versus human performance for genetics questions
Duong, D., & Solomon, B. D. (2023) · doi:10.1101/2023.01.27.23285115 · Cited by 159 · cited 3× · see citing works
“However, Duong and Solomon 16 found that ChatGPT performed poorly on critical thinking questions compared to questions requiring memorization.”
BackgroundUsing ChatGPT for language editing in scientific articles
Kim, S.-G. (2023) · Maxillofacial Plastic and Reconstructive Surgery · doi:10.1186/s40902-023-00381-x · Cited by 153 · see citing works
ChatGPT, An Artificial Intelligence Chatbot, Is Impacting Medical Literature
Lubowitz, J. H. (2023) · Arthroscopy · doi:10.1016/j.arthro.2023.01.015 · Cited by 132 · see citing works
ChatGPT interactive medical simulations for early clinical education: Case study
Scherr, R., Halaseh, F. F., Spina, A., Andalib, S., & Rivera, R. (2023) · JMIR Medical Education · doi:10.2196/49877 · Cited by 122 · see citing works
Performance of ChatGPT in medical examinations: A systematic review and a meta‐analysis
Levin, G., Horesh, N., Brezinov, Y., & Meyer, R. (2023) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.17641 · Cited by 92 · see citing works
Evaluating the performance of ChatGPT in medical physiology university examination of phase I MBBS
Mahila, S., Jaleel, I., & Surapaneni, K. M. (2023) · AJP Advances in Physiology Education · doi:10.1152/advan.00036.2023 · Cited by 72 · see citing works
The Intersection of ChatGPT, Clinical Medicine, and Medical Education
Wong, R. S.-Y., Ming, L. C., & Ali, R. A. R. (2023) · JMIR Medical Education · doi:10.2196/47274 · Cited by 60 · see citing works
Aye, AI! ChatGPT passes multiple-choice family medicine exam
Morreel, S., Mathysen, D., & Verhoeven, V. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2187684 · Cited by 56 · see citing works
Benchmarking large language models for genomic knowledge with GeneTuringPreprint
Shang, X., Liao, X., Ji, Z., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.03.11.532238 · Cited by 41 · see citing works
ChatGPT to Enhance Learning in Dental Education at a Historically Black Medical College
Rahad, K., Martin, K., Amugo, I., Ferguson, S., Curtis, A., Davis, A., Gangula, P. R., & Wang, Q. (2024) · Dental Research and Oral Health · doi:10.26502/droh.0069 · Cited by 27 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [17]Hou W Ji Z. GeneTuring tests GPT models in genomics. bioRxiv. 2023.
Cited by
Sorted by citations
Jin, H. K., Lee, H. E., & Kim, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05944-8 · Cited by 85 · see citing works
Liu, M., Okuhara, T., Dai, Z., Huang, W., Gu, L., Okada, H., Furukawa, E., & Kiuchi, T. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105673 · Cited by 73 · see citing works
Adamakis, M., & Rachiotis, T. (2025) · Encyclopedia · doi:10.3390/encyclopedia5040180 · Cited by 76 · see citing works
Jaleel, A., Aziz, U., Farid, G., Zahid Bashir, M., Mirza, T. R., Khizar Abbas, S. M., Aslam, S., & Sikander, R. M. H. (2025) · JMIR Medical Education · doi:10.2196/68070 · Cited by 25 · see citing works
- Authors
- Sumbal, A., Sumbal, R., & Amir, A.
- Affiliations
- Dow University of Health Sciences (PK)
- Corresponding author
- Sumbal, Anusha
- Venue
- Journal of Medical Education and Curricular Development
- Publisher
- SAGE Publications
- Published
- 2024-01
- Volume
- 11
- Pages
- 23821205241238641-23821205241238641
- ISSN
- 2382-1205 (ISSN-L)
- PMC
- PMC10938614
- OpenAlex
- W4392749797
- Cited by
- 63
- References
- 31
- Citations by year
- 2026: 13 · 2025: 30 · 2024: 19 · 2022: 1
- FWCI
- 1.41
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Journal of Medical Education and Curricular Development (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- ChatGPT-3.5, large language models, medical examination, automated assessment, academic testing, question difficulty, multiple-choice questions, image-based questions, critical thinking, medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
