Examining the Threat of ChatGPT to the Validity of Short Answer Assessments in an Undergraduate Medical Program
Open accessAbstract
OBJECTIVES: ChatGPT is an artificial intelligence model that can interpret free-text prompts and return detailed, human-like responses across a wide domain of subjects. This study evaluated the extent of the threat posed by ChatGPT to the validity of short-answer assessment problems used to examine pre-clerkship medical students in our undergraduate medical education program. METHODS: Forty problems used in prior student assessments were retrieved and stratified by levels of Bloom's Taxonomy. Thirty of these problems were submitted to ChatGPT-3.5. For the remaining 10 problems, we retrieved past minimally passing student responses. Six tutors graded each of the 40 responses. Comparison of performance between student-generated and ChatGPT-generated answers aggregated as a whole and grouped by Bloom's levels of cognitive reasoning, was done using t-tests, ANOVA, Cronbach's alpha, and Cohen's d. Scores for ChatGPT-generated responses were also compared to historical class average performance. RESULTS: ChatGPT-generated responses received a mean score of 3.29 out of 5 (n = 30, 95% CI 2.93-3.65) compared to 2.38 for a group of students meeting minimum passing marks (n = 10, 95% CI 1.94-2.82), representing higher performance ( P = .008, η 2 = 0.169), but was outperformed by historical class average scores on the same 30 problems (mean 3.67, P = .018) when including all past responses regardless of student performance level. There was no statistically significant trend in performance across domains of Bloom's Taxonomy. CONCLUSION: While ChatGPT was able to pass short answer assessment problems spanning the pre-clerkship curriculum, it outperformed only underperforming students. We remark that tutors in several cases were convinced that ChatGPT-produced responses were produced by students. Risks to assessment validity include uncertainty in identifying struggling students and inability to intervene in a timely manner. The performance of ChatGPT on problems requiring increasing demands of cognitive reasoning warrants further research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several studies examined the performance of LLMs on various medical exams; comparisons between older and newer versions indicated that newer versions exhibited better accuracy; one study comparing ChatGPT and Google's Bard in answering multiple-choice questionnaires related to endocrinology and diabetes found ChatGPT to be slightly superior, but both models lacked up-to-date information; another
“Several studies examined the performance of LLMs on various medical exams [97–99]. Comparisons between older and newer versions indicated that newer versions exhibited better accuracy, highlighting the need for ongoing assessment of AI tools in medical education [100–102]. One study comparing ChatGPT and Google’s Bard in answering multiple-choice questionnaires (MCQs) related to endocrinology and”
From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [98]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931 · see citing works
Lakens, D. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00863 · Cited by 10,303 · cited 1× · see citing works
“8 large.(18,19) Reliability was estimated with Cronbach alpha.”
MethodMeasuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1× · see citing works
“8,9 While licensing examinations in Canada and the United States are in the multiple-choice format, assessment throughout the medical curriculum includes other formats.”
BackgroundSallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · cited 1× · see citing works
“This lack of trend in ChatGPT response quality with question dif fi culty or complexity is in contrast with one of the USMLE studies 9 and warrants further focused research.”
ResultChatGPT - Reshaping medical education and clinical management
Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.39.2.7653 · Cited by 628 · see citing works
The rise of <scp>ChatGPT</scp> : Exploring its potential in medical education
Lee, H. (2024) · Anatomical Sciences Education · doi:10.1002/ase.2270 · Cited by 596 · see citing works
Artificial intelligence in medical education
Masters, K. (2019) · Medical Teacher · doi:10.1080/0142159x.2019.1595557 · Cited by 469 · cited 1× · see citing works
“Conversely, the exponential rate of progress of AI-powered tools can pose challenges to their thoughtful integration into routine teaching practices.(3,4) In November 2022, ChatGPT (OpenAI, California, USA), a large language model (LLM), was released for public use.”
BackgroundArtificial Intelligence in Undergraduate Medical Education: A Scoping Review
Lee, J., Wu, A. S., Li, D., & Kulasegaram, K. (. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004291 · Cited by 324 · see citing works
Ethical use of Artificial Intelligence in Health Professions Education: AMEE Guide No. 158
Masters, K. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2186203 · Cited by 302 · cited 1× · see citing works
“Our group supports the thoughtful and ethical incorporation of arti fi cial intelligence in medical education 28 and are actively exploring ways that these tools might be used to support underperforming students.”
BackgroundWinkler-Schwartz, A., Bissonnette, V., Mirchi, N., Ponnudurai, N., Yilmaz, R., Ledwos, N., Siyar, S., Azarnoush, H., Karlık, B., & Maestro, R. F. D. (2019) · Journal of surgical education · doi:10.1016/j.jsurg.2019.05.015 · Cited by 295 · cited 1× · see citing works
“Whether it be facilitating the retrieval and summary of the medical literature, analyzing trends in student responses in real-time, or enhancing simulation exercises with virtual reality, AI technology has already begun to be embedded in education.(1,2) Conversely, the exponential rate of progress of AI-powered tools…”
BackgroundReimagining Medical Education in the Age of AI
(2019) · AMA Journal of Ethics · doi:10.1001/amajethics.2019.146 · Cited by 263 · cited 1× · see citing works
“Conversely, the exponential rate of progress of AI-powered tools can pose challenges to their thoughtful integration into routine teaching practices.(3,4) In November 2022, ChatGPT (OpenAI, California, USA), a large language model (LLM), was released for public use.”
BackgroundHuisman, M., Ranschaert, E. R., Parker, W., Mastrodicasa, D., Kočí, M., dos Santos, D. P., Coppola, F., Morozov, S., Zins, M., Bohyn, C., et al. (2021) · European Radiology · doi:10.1007/s00330-021-07781-5 · Cited by 245 · cited 1× · see citing works
“22 Such training can include how to interpret the output of models and the consideration of bias in their underlying machine-learning models.”
BackgroundApplicability of ChatGPT in Assisting to Solve Higher Order Problems in Pathology
Sinha, R. K., Deb Roy, A., Kumar, N., & Mondal, H. (2023) · Cureus · doi:10.7759/cureus.35237 · Cited by 187 · see citing works
2020 ACR Data Science Institute Artificial Intelligence Survey
Allen, B., Agarwal, S., Coombs, L. P., Wald, C., & Dreyer, K. J. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.04.002 · Cited by 154 · see citing works
Pucchio, A., Rathagirishnan, R., Caton, N., Gariscsak, P. J., Del Papa, J., Nabhen, J. J., Vo, V., Lee, W., & Moraes, F. Y. (2022) · BMC Medical Education · doi:10.1186/s12909-022-03896-5 · Cited by 138 · see citing works
Das, D., Kumar, N., Longjam, L. A., Sinha, R., Deb Roy, A., Mondal, H., & Gupta, P. (2023) · Cureus · doi:10.7759/cureus.36034 · Cited by 116 · see citing works
Liu, D. S., Sawyer, J., Luna, A., Aoun, J., Wang, J., Boachie, L., Halabi, S., & Joe, B. (2022) · JMIR Medical Education · doi:10.2196/38325 · Cited by 90 · cited 1× · see citing works
“Despite the very limited presence of autonomous AI in medical practice at this time, training in AI is likely to benefit medical learners given that clinical groups are beginning to actively explore how to further harness AI in their practices,(24) especially in the context of the known current knowledge gap for medic…”
BackgroundGhosh, A., & Bir, A. (2023) · Cureus · doi:10.7759/cureus.37023 · Cited by 67 · see citing works
Patients don’t come with multiple choice options: essay-based assessment in UME
Bird, J. B., Olvet, D. M., Willey, J. M., & Brenner, J. M. (2019) · Medical Education Online · doi:10.1080/10872981.2019.1649959 · Cited by 59 · cited 1× · see citing works
“Short answer assessments are a widely used form of student evaluation in medical education, and are critical in evaluating student knowledge, communication skills, and other essential competencies required for delivering compassionate care.(10,11) The ability of ChatGPT to quickly and accurately apply known concepts i…”
BackgroundInsights from teaching artificial intelligence to medical students in Canada
Hu, R., Fan, K. Y., Pandey, P., Hu, Z., Yau, O., Teng, M., Wang, P., Li, T., Ashraf, M., & Singla, R. (2022) · Communications Medicine · doi:10.1038/s43856-022-00125-4 · Cited by 58 · cited 1× · see citing works
“Moreover, calls to incorporate knowledge of artificial intelligence in medical education, including appropriate use, continue to build(29,30) and there is now gradual incorporation into Canadian medical training.(31,32) Conclusions The ability of ChatGPT to meet the standard on our institution’s short answer assessmen…”
BackgroundThe Evolving Importance of Artificial Intelligence and Radiology in Medical Trainee Education
Fischetti, C., Bhatter, P., Frisch, E., Sidhu, A., Helmy, M., Lungren, M., & Duhaime, E. (2022) · Academic Radiology · doi:10.1016/j.acra.2021.03.023 · Cited by 47 · see citing works
Assessment methods in medical education.
(2008) · Europe PMC (PubMed Central) · Cited by 47 · see citing works
Development of clinical reasoning exercises in a problem-based curriculum
Neville, A. J., Cunnington, J. P. W., & Norman, G. R. (1996) · Academic Medicine · doi:10.1097/00001888-199601000-00058 · Cited by 21 · cited 1× · see citing works
“Pre-clerkship students at our medical school write a concept application exercise (CAE) assessment approximately every 4 weeks covering the most recently covered curriculum content.(16) CAEs feature four or five clinical scenarios each with multiple prompts and questions.”
BackgroundAtalay, M., Baird, G., Stib, M., George, P., Oueidat, K., & Cronan, J. (2022) · Academic Radiology · doi:10.1016/j.acra.2022.07.003 · Cited by 14 · see citing works
Hathaway, Q. A., Hogg, J. P., & Lakhani, D. A. (2022) · Academic Radiology · doi:10.1016/j.acra.2022.11.018 · Cited by 8 · see citing works
Canadian Conference on Medical Education 2022 abstracts
Hickey, H. (2022) · Canadian Medical Education Journal · doi:10.36834/cmej.75002 · Cited by 6 · see citing works
Fostering Artificial Intelligence Education within Radiology Residencies: A Two-Tiered Approach
Hathaway, Q. A., & Lakhani, D. A. (2022) · Academic Radiology · doi:10.1016/j.acra.2022.12.009 · Cited by 2 · cited 1× · see citing works
“Such training can include how to interpret the output of models and the consideration of bias in their underlying machine-learning models.(23) Despite the very limited presence of autonomous AI in medical practice at this time, training in AI is likely to benefit medical learners given that clinical groups are beginni…”
BackgroundOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 1×
“21 Furthermore, the ability of this new iteration to interpret images has led to further outperform-ance on many exams.”
Background
Flagged first, then by citations
Cited by
Sorted by citations
A Scoping Review on How Generative Artificial Intelligence Transforms Assessment in Higher Education
Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. F. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00468-z · Cited by 493 · see citing works
“The use of AI tools like ChatGPT in education requires rethinking assessment policies (Gorichanaz, 2023; Morjaria et al., 2023), as traditional examinations and assignments may become obsolete due to AI-generated content (Geerling et al., 2023; Gorichanaz, 2023).”
ChatGPT in the higher education: A systematic literature review and research challenges
Baig, M. I., & Yadegaridehkordi, E. (2024) · International Journal of Educational Research · doi:10.1016/j.ijer.2024.102411 · Cited by 208 · see citing works
Lubbe, A., Marais, E., & Kruger, D. J. D. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13476-x · Cited by 89 · see citing works
ChatGPT and Other Large Language Models in Medical Education — Scoping Literature Review
Aster, A., Laupichler, M. C., Rockwell-Kollmann, T., Masala, G., Bala, E., & Raupach, T. (2024) · Medical Science Educator · doi:10.1007/s40670-024-02206-6 · Cited by 59 · see citing works
Artificial intelligence in undergraduate medical education: an updated scoping review
Simoni, J., Urtubia-Fernandez, J., Mengual, E., Simoni, D. A., Royo, M., Egaña-Yin, D., Hertog, O. L. A., López-Ortiz, L., Muñoz-Tomás, A., Santiago-Martínez, P., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08188-2 · Cited by 62 · see citing works
BURKE, H. B., Hoang, A., Lopreiato, J. O., King, H. B., Hemmer, P. A., Montgomery, M., & Gagarin, V. (2024) · JMIR Medical Education · doi:10.2196/56342 · Cited by 36 · see citing works
“The ChatGPT-human Spearman correlations for a single assessor ranged from 0.6 to 0.7 [12].”
MethodInfluentialBolgova, O., Ganguly, P. K., Ikram, M. F., & Mavrych, V. (2025) · Medical Education Online · doi:10.1080/10872981.2025.2550751 · Cited by 35 · see citing works
“While preliminary research suggests the potential of LLMs in supporting SAQ grading, there remains a significant gap in our understanding of how these models perform specifically in medical education [5,15].”
InfluentialMorjaria, L., Burns, L., Bracken, K., Levinson, A. J., Ngo, Q. N., Lee, M., & Sibbald, M. (2024) · International Medical Education · doi:10.3390/ime3010004 · Cited by 33 · see citing works
“…not formally tested in this study, there also exists some level of inter-rater variability with independent human tutors; our group formally investigated in a previous work and found a Cronbach alpha value of 0.816 for a team of six human assessors on past student-generated CAE responses [29].”
InfluentialSelf-citationChen, J., Zhu, L., Mou, W., Lin, A., Zeng, D., Qi, C., Liu, Z., Jiang, A., Tang, B., Shi, W., et al. (2024) · iMetaOmics · doi:10.1002/imo2.7 · Cited by 33 · see citing works
“…AI in the medical field encompasses a broad spectrum, ranging from assessing its grasp of medical knowledge and ability to pass medical examinations [8, 9] to aiding in providing initial medical counseling [10, 11] and swiftly providing pertinent medical information and recommendations [12, 13].”
Jaleel, A., Aziz, U., Farid, G., Zahid Bashir, M., Mirza, T. R., Khizar Abbas, S. M., Aslam, S., & Sikander, R. M. H. (2025) · JMIR Medical Education · doi:10.2196/68070 · Cited by 25 · see citing works
“These results suggest that ChatGPT may potentially serve as an adjunct learning tool in surgical education [35,45,59].”
Teckwani, S. H., Wong, A. H., Luke, N. V., & Low, I. C. C. (2024) · AJP Advances in Physiology Education · doi:10.1152/advan.00137.2024 · Cited by 21 · see citing works
One year in the classroom with ChatGPT: empirical insights and transformative impacts
Guo, F., Li, T., & Cunningham, C. J. L. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1574477 · Cited by 32 · see citing works
Duong‐Trung, N., Wang, X., & Kravčík, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72312-4_11 · Cited by 15 · see citing works
Abouzeid, E., Wassef, R., Jawwad, A., & Harris, P. (2025) · JMIR Medical Education · doi:10.2196/69521 · Cited by 12 · see citing works
“…prompts and examination questions, and to mimic human writing styles, introducing a potential threat to the validity of traditional forms of medical student assessment including short answer assessment [8], it even successfully passed the USMLE (United States Medical Licensing Examination) [9].”
Ghimire, S. N., Baral, R. K., & Bhattarai, U. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2025.2611124 · Cited by 6 · see citing works
“However, most of them were focused on HE contexts belonging to the United States and Australia, where many universities have formulated publicly available guidelines on GenAI use in assessment practices (Luo, 2024; Morjaria et al., 2023; Perkins et al., 2024).”
Wani, T. A., Liem, M. K., Prasad, N., Robinson, K. M., Nexhip, A., Tassos, M., Gjorgioski, S., Khan, U. R., Boyd, J., & Riley, M. M. (2026) · JMIR Medical Education · doi:10.2196/82988 · Cited by 4 · see citing works
Is ChatGPT reliable in education?
Alibrahim, A. A. (2024) · South African Journal of Education · doi:10.15700/saje.v44n4a2557 · Cited by 3 · see citing works
“In another case, ChatGPT could pass a short-answer test in a pre-clerkship medical educational programme (mean 3.29); however, its performance was low compared to the students’ answers with a mean of 3.67 (Morjaria et al., 2023).”
InfluentialMorjaria, L., Burns, L., Gandhi, B., Bracken, K., Farooq, M. S., Levinson, A. J., Ngo, Q., & Sibbald, M. (2026) · Applied Sciences · doi:10.3390/app16052535 · Cited by 2 · see citing works
Self-citationA Systematic Review of the Role of Artificial Intelligence in Teaching and Learning
Wang, J., Xing, Q., Qian, Y., & Akbar, A. (2024) · International Conference on Educational Technology (ICET) · doi:10.1109/icet62460.2024.10868945 · Cited by 2 · see citing works
Generative AI: A Disruptor to Health Professions Learner Assessment
Morjaria, L., Gandhi, B., Burns, L., & Sibbald, M. (2025) · Cureus · doi:10.7759/cureus.96034 · Cited by 1 · see citing works
“Within undergraduate medical programs, ChatGPT 3.5, a now-outdated model, was shown to pass short-answer assessments at the pre-clinical level [12].”
InfluentialSelf-citationLopiano, S., Cohen, E. S., Holan, G., & Grachan, J. J. (2026) · Cureus · doi:10.7759/cureus.107397 · Cited by 1 · see citing works
“…potential across a range of educational applications, including simulating clinical scenarios and patient interactions, supporting active learning modalities, interpreting anatomical images, and producing assessment content [2-10], their integration into medical education has accelerated.”
Khake, A. M., Gokhale, S., Ramchandra, D. D. P., Khake, S., & Desai, M. M. (2025) · Cureus · doi:10.7759/cureus.86288 · Cited by 1 · see citing works
Huang, K., Liu, C., Wu, C.-C., Sheu, B., & Yang, C. (2024) · doi:10.2196/preprints.60565
Bridging the gaps in AI performance evaluation in medical education
Dharmasaroja, P. (2025) · Medical Teacher · doi:10.1080/0142159x.2025.2467301
A Inteligência Artificial como ferramenta transformadora na aprendizagem de acadêmicos de medicina
da Silva, F. J. A., Karigyo, M. J., Machado, K. C., de Carvalho, D. A. B., & Nyland, J. J. A. O. L. (2025) · Revista Caribeña de Ciencias Sociales · doi:10.55905/rcssv14n6-022
Torres-Coronas, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-01429-0_16
Merino, M., & Duarte, S. R. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7574-8.ch004
Gao, Q., Zhou, H., & Lu, H. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1881713
“Educational administrators must therefore reconstruct the existing academic assessment systems (Gorichanaz, 2023; Morjaria et al., 2023).”
Maini, E., Kumar, S. A., Marwaha, D., & Maini, B. (2026) · Smart innovation, systems and technologies · doi:10.1007/978-981-95-9101-5_41
Azzubaidi, K. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1895483
- Authors
- Morjaria, L., Burns, L., Bracken, K., Ngo, Q. N., Lee, M., Levinson, A. J., Smith, J., Thompson, P., & Sibbald, M.
- Affiliations
- McMaster University (CA)
- Corresponding author
- Sibbald, Matthew
- Venue
- Journal of Medical Education and Curricular Development
- Publisher
- SAGE Publications
- Published
- 2023-01
- Volume
- 10
- Pages
- 23821205231204178-23821205231204178
- ISSN
- 2382-1205 (ISSN-L)
- PMC
- PMC10540597
- OpenAlex
- W4387124372
- Cited by
- 31
- References
- 29
- Citations by year
- 2026: 7 · 2025: 13 · 2024: 11
- FWCI
- 1.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Journal of Medical Education and Curricular Development (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- ChatGPT, assessment validity, artificial intelligence in education, preclinical medical students, Bloom's taxonomy, cognitive reasoning, student performance, undergraduate medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
