Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Problem: Creating medical exam questions is time consuming, but well-written questions can be used for test-enhanced learning, which has been shown to have a positive effect on student learning. The automated generation of high-quality questions using large language models (LLMs), such as ChatGPT, would therefore be desirable. However, there are no current studies that compare students’ performance on LLM-generated questions to questions developed by humans. Approach: The authors compared student performance on questions generated by ChatGPT (LLM questions) with questions created by medical educators (human questions). Two sets of 25 multiple-choice questions (MCQs) were created, each with 5 answer options, 1 of which was correct. The first set of questions was written by an experienced medical educator, and the second set was created by ChatGPT 3.5 after the authors identified learning objectives and extracted some specifications from the human questions. Students answered all questions in random order in a formative paper-and-pencil test that was offered leading up to the final summative neurophysiology exam (summer 2023). For each question, students also indicated whether they thought it had been written by a human or ChatGPT. Outcomes: The final data set consisted of 161 participants and 46 MCQs (25 human and 21 LLM questions). There was no statistically significant difference in item difficulty between the 2 question sets, but discriminatory power was statistically significantly higher in human than LLM questions (mean = .36, standard deviation [SD] = .09 vs mean = .24, SD = .14; P = .001). On average, students identified 57% of question sources (human or LLM) correctly. Next Steps: Future research should replicate the study procedure in other contexts (e.g., other medical subjects, semesters, countries, and languages). In addition, the question of whether LLMs are suitable for generating different question types, such as key feature questions, should be investigated.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Critical Importance of Retrieval for Learning
Karpicke, J. D., & Roediger, H. L. (2008) · Science · doi:10.1126/science.1152408 · Cited by 1,791
ChatGPT - Reshaping medical education and clinical management
Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.39.2.7653 · Cited by 628
The rise of ChatGPT : Exploring its potential in medical education
Lee, H. (2024) · Anatomical Sciences Education · doi:10.1002/ase.2270 · Cited by 596
Testing the testing effect in the classroom
McDaniel, M. A., Anderson, J. L., Derbish, M. H., & Morrisette, N. (2007) · European Journal of Cognitive Psychology · doi:10.1080/09541440701326154 · Cited by 563
Transfer of test- enhanced learning: meta-analytic review and synthesis
Pan, S. C., & Rickard, T. C. (2018) · Psychological Bulletin · doi:10.1037/bul0000151 · Cited by 286
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T.-H. (2023) · PLOS ONE · doi:10.1371/journal.pone.0290691 · Cited by 195
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T. H. (2023) · doi:10.1101/2023.05.13.23289943 · Cited by 10
Research works printed but not linked: 4
- Coughlin, 2017, How to write a high quality multiple choice question (MCQ): a guide for clinicians, Eur J Vasc Endovasc Surg, 54, 654, 10.1016/j.ejvs.2017.07.012 · DOI
- Thirunavukarasu, 2023, Large language models in medicine, Nat Med, 29, 1930, 10.1038/s41591-023-02448-8 · DOI · Paper page
- von Davier, 2023, Advancing Natural Language Processing in Natural Assessment
- Berens, 2022, Effects of elaboration and instructor feedback on retention of clinical reasoning competence among undergraduate medical students: a randomized crossover trial, JAMA Netw Open, 5, e2245491, 10.1001/jamanetworkopen.2022.45491 · DOI
Cited by
Large language models for generating medical examinations: systematic review
Artsi, Y., Sorin, V., Konen, E., Glicksberg, B. S., Nadkarni, G., & Klang, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05239-y · Cited by 128
“A paper published after our search period by Laupichler et al. conducted this comparison in student performance in answering AI vs. human generated MCQs [63].”
MethodDecoding medical educators’ perceptions on generative artificial intelligence in medical education
Cervantes, J., Smith, B., Ramadoss, T., D’Amario, V., Shoja, M. M., & Rajput, V. (2024) · Journal of Investigative Medicine · doi:10.1177/10815589241257215 · Cited by 47
Grévisse, C., Pavlou, M. A. S., & Schneider, J. G. (2024) · SN Computer Science · doi:10.1007/s42979-024-02963-6 · Cited by 41
“Laupichler et al. compared student performance of 25 MCQs generated by ChatGPT (GPT − 3.5) against 25 MCQs created by human medical educators, each with 1 key and 4 distractors [24].”
InfluentialLee, W., Chang, S., & Suh, J. (2025) · Education Sciences · doi:10.3390/educsci15111478 · Cited by 5
“Produces grammatically correct and standardized questions at scale (Laupichler et al., 2024).”
Influential
Metadata
- Authors
- Laupichler, M. C., Rother, J. F., Grunwald Kadow, I. C., Ahmadi, S., & Raupach, T.
- Venue
- Academic Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-12-28
- Volume
- 99
- Issue
- 5
- Pages
- 508-512
- ISSN
- 1040-2446 (ISSN-L)
- 1938-808X
- OpenAlex
- W4390546602
- Cited by
- 107
- References
- 7
- FWCI
- 3.70
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions. https://app.cochat.ai/papers/10.1097/acm.0000000000005626
- The paper
- Laupichler, M. C., Rother, J. F., Grunwald Kadow, I. C., Ahmadi, S., & Raupach, T. (2024). Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions. Academic Medicine. https://doi.org/10.1097/acm.0000000000005626
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.