Examining the Efficacy of ChatGPT in Marking Short-Answer Assessments in an Undergraduate Medical Program
Open accessAbstract
Traditional approaches to marking short-answer questions face limitations in timeliness, scalability, inter-rater reliability, and faculty time costs. Harnessing generative artificial intelligence (AI) to address some of these shortcomings is attractive. This study aims to validate the use of ChatGPT for evaluating short-answer assessments in an undergraduate medical program. Ten questions from the pre-clerkship medical curriculum were randomly chosen, and for each, six previously marked student answers were collected. These sixty answers were evaluated by ChatGPT in July 2023 under four conditions: with both a rubric and standard, with only a standard, with only a rubric, and with neither. ChatGPT displayed good Spearman correlations with a single human assessor (r = 0.6–0.7, p < 0.001) across all conditions, with the absence of a standard or rubric yielding the best correlation. Scoring differences were common (65–80%), but score adjustments of more than one point were less frequent (20–38%). Notably, the absence of a rubric resulted in systematically higher scores (p < 0.001, partial η2 = 0.33). Our findings demonstrate that ChatGPT is a viable, though imperfect, assistant to human assessment, performing comparably to a single expert assessor. This study serves as a foundation for future research on AI-based assessment techniques with potential for further optimization and increased reliability.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT has potential as an assistive tool for grading.
“ChatGPT correlated well with human assessors when grading short-answer questions, suggesting its potential as an assistive tool for grading [141].”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [141]
ChatGPT correlated well with human assessors when grading short-answer questions.
“ChatGPT correlated well with human assessors when grading short-answer questions, suggesting its potential as an assistive tool for grading [141].”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [141]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
1 retracted
Huynh, L. M., Bonebrake, B. T., Schultis, K. E., Quach, A., & Deibert, C. M. (2023) · Urology Practice · doi:10.1097/upj.0000000000000406 · Cited by 46 · see citing works
RetractedLikert scales, levels of measurement, and the “laws” of statistics
Norman, G. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9222-y · Cited by 4,850 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 1× · see citing works
“The use of large language models (LLMs) has, in particular, drawn consideration for potential use in medical education since the public launch of ChatGPT (OpenAI, CA, USA) in November 2022 [3–5].”
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions
Abd-alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P. M., Latifi, S., Aziz, S., Damseh, R., Alabed Alrazak, S., & Sheikh, J. (2023) · JMIR Medical Education · doi:10.2196/48291 · Cited by 722 · see citing works
ChatGPT - Reshaping medical education and clinical management
Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.39.2.7653 · Cited by 628 · see citing works
The rise of <scp>ChatGPT</scp> : Exploring its potential in medical education
Lee, H. (2024) · Anatomical Sciences Education · doi:10.1002/ase.2270 · Cited by 596 · see citing works
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563 · see citing works
Artificial intelligence in medical education
Masters, K. (2019) · Medical Teacher · doi:10.1080/0142159x.2019.1595557 · Cited by 469 · see citing works
Assessment methods in medical education
Norcini, J. J., & McKinley, D. W. (2007) · Teaching and Teacher Education · doi:10.1016/j.tate.2006.12.021 · Cited by 366 · see citing works
Winkler-Schwartz, A., Bissonnette, V., Mirchi, N., Ponnudurai, N., Yilmaz, R., Ledwos, N., Siyar, S., Azarnoush, H., Karlık, B., & Maestro, R. F. D. (2019) · Journal of surgical education · doi:10.1016/j.jsurg.2019.05.015 · Cited by 295 · see citing works
Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023) · JMIR Medical Education · doi:10.2196/48002 · Cited by 271 · see citing works
Reimagining Medical Education in the Age of AI
(2019) · AMA Journal of Ethics · doi:10.1001/amajethics.2019.146 · Cited by 263 · see citing works
The Advent of Generative Language Models in Medical Education
Karabacak, M., Ozkara, B. B., Margetis, K., Wintermark, M., & Bisdas, S. (2023) · JMIR Medical Education · doi:10.2196/48163 · Cited by 208 · see citing works
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T.-H. (2023) · PLOS ONE · doi:10.1371/journal.pone.0290691 · Cited by 195 · see citing works
Applicability of ChatGPT in Assisting to Solve Higher Order Problems in Pathology
Sinha, R. K., Deb Roy, A., Kumar, N., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.35237 · Cited by 187 · see citing works
Wang, H., Wu, W., Dou, Z., He, L., & Yang, L. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105173 · Cited by 156 · see citing works
Humar, P., Asaad, M., Bengur, F. B., & Nguyen, V. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad130 · Cited by 155 · see citing works
ChatGPT in medical school: how successful is AI in progress testing?
Friederichs, H., Friederichs, W. J., & März, M. (2023) · Medical Education Online · doi:10.1080/10872981.2023.2220920 · Cited by 122 · see citing works
Das, D., Kumar, N., Longjam, L. A., Sinha, R., Deb Roy, A., Mondal, H., & Gupta, P. (2023) · Cureus · doi:10.7759/cureus.36034 · Cited by 116 · see citing works
Agarwal, M., Sharma, P., & Goswami, A. (2023) · Cureus · doi:10.7759/cureus.40977 · Cited by 72 · see citing works
Ghosh, A., & Bir, A. (2023) · Cureus · doi:10.7759/cureus.37023 · Cited by 67 · see citing works
Alfertshofer, M. G., Hoch, C. C., Funk, P. F., Hollmann, K., Wollenberg, B., Knoedler, S. A., & Knoedler, L. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03338-3 · Cited by 60 · see citing works
Patients don’t come with multiple choice options: essay-based assessment in UME
Bird, J. B., Olvet, D. M., Willey, J. M., & Brenner, J. M. (2019) · Medical Education Online · doi:10.1080/10872981.2019.1649959 · Cited by 59 · see citing works
Aye, AI! ChatGPT passes multiple-choice family medicine exam
Morreel, S., Mathysen, D., & Verhoeven, V. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2187684 · Cited by 56 · see citing works
Xie, Y., Seth, I., Hunter‐Smith, D. J., Rozen, W. M., & Seifman, M. A. (2024) · ANZ Journal of Surgery · doi:10.1111/ans.18666 · Cited by 55 · see citing works
Ayub, I., Hamann, D. J., Hamann, C. R., & Davis, M. J. (2023) · Cureus · doi:10.7759/cureus.43717 · Cited by 54 · see citing works
Assessment Methods in Medical Education
Tabish, S. A. (2008) · Cited by 49 · see citing works
The Evolving Importance of Artificial Intelligence and Radiology in Medical Trainee Education
Fischetti, C., Bhatter, P., Frisch, E., Sidhu, A., Helmy, M., Lungren, M., & Duhaime, E. (2022) · Academic Radiology · doi:10.1016/j.acra.2021.03.023 · Cited by 47 · see citing works
Kumah-Crystal, Y. A., Mankowitz, S. L., Embí, P. J., & Lehmann, C. U. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad104 · Cited by 45 · see citing works
Borchert, R. J., Hickman, C. R., Pepys, J., & Sadler, T. J. (2023) · JMIR Medical Education · doi:10.2196/48978 · Cited by 39 · see citing works
Morjaria, L., Burns, L., Bracken, K., Ngo, Q. N., Lee, M., Levinson, A. J., Smith, J., Thompson, P., & Sibbald, M. (2023) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205231204178 · Cited by 31 · cited 2× · see citing works
“…not formally tested in this study, there also exists some level of inter-rater variability with independent human tutors; our group formally investigated in a previous work and found a Cronbach alpha value of 0.816 for a team of six human assessors on past student-generated CAE responses [29].”
InfluentialSelf-citationUse of ChatGPT on Taiwan’s Examination for Medical Doctors
Kao, Y., Chuang, W.-K., & Yang, J. C. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03308-9 · Cited by 28 · see citing works
Development of clinical reasoning exercises in a problem-based curriculum
Neville, A. J., Cunnington, J. P. W., & Norman, G. R. (1996) · Academic Medicine · doi:10.1097/00001888-199601000-00058 · Cited by 21 · see citing works
Performance of ChatGPT in Israeli Hebrew OBGYN national residency examinations
Cohen, A., Alter, R., Lessans, N., Meyer, R., Brezinov, Y., & Levin, G. (2023) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-023-07185-4 · Cited by 17 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Abd-Alrazaq, A.; AlSaad, R.; Alhuwail, D.; Ahmed, A.; Healy, P.M.; Latifi, S.; Aziz, S.; Damseh, R.; Alrazak, S.A.; Sheikh, J. Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions. JMIR Med. Educ. 2023, 9, e48291.
Cited by
Sorted by citations
Flodén, J. (2024) · British Educational Research Journal · doi:10.1002/berj.4069 · Cited by 110 · see citing works
Seßler, K., Fürstenberg, M., Bühler, B., & Kasneci, E. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706527 · Cited by 84 · see citing works
Artificial intelligence in undergraduate medical education: an updated scoping review
Simoni, J., Urtubia-Fernandez, J., Mengual, E., Simoni, D. A., Royo, M., Egaña-Yin, D., Hertog, O. L. A., López-Ortiz, L., Muñoz-Tomás, A., Santiago-Martínez, P., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08188-2 · Cited by 62 · see citing works
“ChatGPT correlated well with human assessors when grading short-answer questions, suggesting its potential as an assistive tool for grading [141].”
University Educators Perspectives on ChatGPT: A Technology Acceptance Model-Based Study
Barakat, M. M., Salim, N. A., & Sallam, M. (2025) · Open Praxis · doi:10.55982/openpraxis.17.1.718 · Cited by 45 · see citing works
Leveraging prompt-based LLMs for automated scoring and feedback generation in higher education
AlGhamdi, E., Li, Y., Gašević, D., & Chen, G. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105511 · Cited by 30 · see citing works
Henkel, O., Hills, L., Roberts, B., & McGrane, J. A. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00431-z · Cited by 37 · see citing works
An agent-based approach for the automatic generation of valid SysMLv2 Models in industrial contexts
Cibrián, E., Olivert-Iserte, J., Lloréns, J., & Álvarez-Rodríguez, J. M. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104350 · Cited by 25 · see citing works
Hassanein, F. E. A., Hussein, R. R., Ahmed, Y., Elguindy, J., ahmed, D., & Abou‐Bakr, A. (2026) · BMC Oral Health · doi:10.1186/s12903-026-07665-4 · Cited by 18 · see citing works
Song, E. S., & Lee, S. (2024) · International Journal of Dental Hygiene · doi:10.1111/idh.12848 · Cited by 10 · see citing works
Examining the Reliability of ChatGPT as an Assessment Tool Compared to Human Evaluators
Poláková, P., Ivenz, P., & Klímová, B. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.543 · Cited by 10 · see citing works
Student perspectives on AI‐supported formative assessment in pharmacology
Berg, J. A., Repstad, Ø., Serkland, T. T., Mork, T. E., Bru, C., Tenstad, O., & Kvernenes, M. (2025) · British Journal of Clinical Pharmacology · doi:10.1002/bcp.70267 · Cited by 7 · see citing works
Reimagining pharmacology education
Guilding, C., Kelly-Laubscher, R. F., Cunningham, M., Dilles, T., Kennedy, D., Brinkman, D. J., Eid, A. H., Quesnelle, K. M., Engels, F., Maxwell, S., et al. (2026) · Pharmacological Reviews · doi:10.1016/j.pharmr.2026.100126 · Cited by 8 · see citing works
Huang, S., Cheng, H., & Luo, M. (2024) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer19030088 · Cited by 6 · see citing works
An Investigation Into the Use of Large Language Models for Adaptive Grading of Course Projects
Addison, L. M., Ramdial, P. T., & Hosein, P. A. (2024) · IEEE International Conference on Technology Management, Operations and Decisions (ICTMOD) · doi:10.1109/ictmod63116.2024.10878239 · Cited by 6 · see citing works
Chuenjitwongsa, S., Arunratanothai, T., Rewthamrongsris, P., Sumrejkanchanakij, P., Tavedhikul, K., Samaranayake, L. P., & Osathanon, T. (2026) · International Dental Journal · doi:10.1016/j.identj.2026.109733 · Cited by 6 · see citing works
Generative Artificial Intelligence and Assessment of Learning
Kaddouri, M., Mhamdi, K., Jabri, A., Boukare, M., & Jmad, S. (2025) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-6527-4.ch010 · Cited by 6 · see citing works
Human vs. Machine Marking: A Comparative Study of Chemistry Assessments
Ade-Ibijola, A., Chikezie, I. J., & Oyelere, S. S. (2025) · Journal of Science Education and Technology · doi:10.1007/s10956-025-10223-2 · Cited by 4 · see citing works
Use of large language models in undergraduate medical education a scoping review
Uzoechina, G., & Osajiuba, T. (2026) · Discover Education · doi:10.1007/s44217-026-01239-w · Cited by 5 · see citing works
Olvet, D. M., Kruidering, M., Fulton, T. B., Truong, B., Endo, K., Lucito, R., & Willey, J. M. (2026) · Medical Science Educator · doi:10.1007/s40670-026-02638-2 · Cited by 4 · see citing works
Yap, W. S., Saw, P. S., Yeap, L. L., Lee, S. W. H., Wong, W. J., & Lee, R. F. S. (2025) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2025.22.20 · Cited by 2 · see citing works
Ratniyom, J., Boonphadung, S., Intaraprasit, M., & Chumkaeo, P. (2025) · Chemistry Teacher International · doi:10.1515/cti-2025-0048 · Cited by 2 · see citing works
Morjaria, L., Burns, L., Gandhi, B., Bracken, K., Farooq, M. S., Levinson, A. J., Ngo, Q., & Sibbald, M. (2026) · Applied Sciences · doi:10.3390/app16052535 · Cited by 2 · see citing works
Self-citationGenerative AI: A Disruptor to Health Professions Learner Assessment
Morjaria, L., Gandhi, B., Burns, L., & Sibbald, M. (2025) · Cureus · doi:10.7759/cureus.96034 · Cited by 1 · see citing works
Self-citationEvaluating ChatGPT for Automated Creation and Grading of Essay Questions in Higher Education
Stowell, J. R., & Zhu, J. (2025) · Teaching of Psychology · doi:10.1177/00986283251408813 · Cited by 2 · see citing works
Gunawardana, H. M. R. S. S. (2025) · Springer eBooks · doi:10.1007/978-981-96-6001-8_7
Meshcheryakov, I. A., & Lagutina, N. S. (2025) · Modeling and Analysis of Information Systems · doi:10.18255/1818-1015-2025-4-396-416
Navigating Challenges and Seizing Opportunities in AI-Driven Assessment
Ahmad, M., & Delda, M. J. M. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7057-6.ch002
Pranoto, D. C. Y., Hussien, S. B., Sabariah, S., Bandono, A., & Bahrawi, A. (2026) · Research Square · doi:10.21203/rs.3.rs-8985839/v1
Kara, A., & Yıldırım, S. (2026) · Sakarya University Journal of Computer and Information Sciences · doi:10.35377/saucis...1835608
Fianko, S. K., Addo, A., Agyemang, K., Gbadago, F. Y., Adjaye-Gyamfi, O., Quartey, O., Agropah, F., Larttey, R., Amoah, N., & Nooni, I. K. (2026) · Discover Education · doi:10.1007/s44217-026-01837-8
Nahir, M., Kasap, A., & Nahir, C. B. (2026) · Surgical and Radiologic Anatomy · doi:10.1007/s00276-026-03986-9
Özdemir, Y., & AŞÇI, E. (2026) · Journal of Educational Technology and Online Learning · doi:10.31681/jetol.1882892
- Authors
- Morjaria, L., Burns, L., Bracken, K., Levinson, A. J., Ngo, Q. N., Lee, M., & Sibbald, M.
- Affiliations
- McMaster University (CA)
- Corresponding author
- Sibbald, Matthew
- Venue
- International Medical Education
- Publisher
- MDPI AG
- Published
- 2024-01-19
- Volume
- 3
- Issue
- 1
- Pages
- 32-43
- ISSN
- 2813-141X (ISSN-L)
- OpenAlex
- W4391026420
- Cited by
- 33
- References
- 36
- Citations by year
- 2026: 12 · 2025: 15 · 2024: 6
- FWCI
- 3.12
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- International Medical Education (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- ChatGPT, automated assessment, short-answer questions, generative AI, medical education, rubrics, scoring standards, inter-rater reliability, pre-clerkship curriculum
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
