The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 6 of the 40 sources its claims cite were checked for support.
Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine
Open accessAbstract
Assessment is an essential part of education, both for teachers who assess their students as well as learners who may evaluate themselves. Multiple-choice questions (MCQ) are one of the most popular types of knowledge assessment, e.g., in medical education, as they can be automatically graded and can cover a wide range of learning items. However, the creation of high-quality MCQ items is a time-consuming task. The recent advent of Large Language Models (LLM), such as Generative Pre-trained Transformer (GPT), caused a new momentum for automatic question generation solutions. Still, evaluating generated questions according to the best practices for MCQ item writing is needed to ensure docimological quality. In this article, we propose an analysis of the quality of LLM-generated MCQs. We employ zero-shot approaches in two domains, namely computer science and medicine. In the former, we make use of 3 GPT-based services to generate MCQs. In the latter, we developed a plugin for the Moodle learning management system that generates MCQs based on learning material. We compare the generated MCQs against common multiple-choice item writing guidelines. Among the major challenges, we determined that while LLMs are certainly useful in generating MCQs more efficiently, they sometimes create broad items with ambiguous keys or implausible distractors. Human oversight is also necessary to ensure instructional alignment between generated items and course contents. Finally, we propose solutions for AQG developers.
Overview
CoChat Summary
## TL;DR
This paper evaluates the docimological quality of LLM-generated multiple-choice questions (MCQs) in two zero-shot case studies: computer science and medicine. Across both domains, many questions were acceptable, but key problems remained with ambiguous or wrong keys, implausible distractors, and weaker instructional alignment.
## Key findings
- In the computer science case study, 55 of 150 questions (37%) had at least one flaw. By service, flaws were 19/50 (38%) for Quiz Wizard, 19/50 (38%) for Quizicist, and 17/50 (34%) for ChatGPT.
- The most common computer science issues were ambiguous keys, followed by keys that were too long and implausible distractors. The paper notes that None of the above and K-type questions did not appear in this dataset.
- In the medicine case study, 13 of 100 questions had 1 flaw and 1 question had 2 flaws. Among subfields, there was 1 flawed question out of 20 neurology MCQs (5%) and 13 flawed questions out of 80 endocrinology MCQs (16%).
- For medicine, the main issue was implausible distractors, followed by no key, wrong key, or ambiguous key. Endocrinology questions were generally judged pertinent and aligned, while neurology questions were often not pertinent, not well-aligned, too easy, and too general.
## Methods
- The study used two case studies with zero-shot LLM-based question generation: one in computer science and one in medicine.
- In computer science, the authors used 3 GPT-based services: Quiz Wizard, Quizicist, and ChatGPT. They generated 10 questions per chapter for a course with 5 chapters, yielding 150 questions total.
- In medicine, the authors developed a Moodle plugin using OpenAI’s Assistants API and generated 10 questions per slidedeck from 8 endocrinology slidedecks and 2 neurology slidedecks, yielding 100 MCQs total. Questions were tagged by domain experts for docimological flaws, and in medicine they were also rated on 5-point Likert scales for pertinence, difficulty, specificity, ambiguity, and instructional alignment.
## Limitations
- The paper explicitly states that questions were only evaluated by a single domain expert each time.
- It also states there were relatively few questions in neurology compared to endocrinology or computer science.
- Evident from the design: the study uses zero-shot prompts and fixed output settings in the Moodle plugin (10 questions, 4 options, 1 key), which limits generalizability to other prompting strategies and item formats.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
6 Valid · 0 Not valid · 18 Not checked · 53 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“The creation of high-quality MCQs is a time-consuming task [5, 6], even more considering that changes in the curriculum or students collecting questions from previous years requires teachers to create new sets of questions every once in a while [7].”
What the sources say
- Supportedcites [5] Doughty J. et al. A comparative study of AI-generated (GPT-4) and human-crafted MCQs in programming education. In: Herbert N, Seton C, editors. Proceedings of the 26th Australasian Computing Education Conference, ACE ’24. New York:Association for Computing Machinery. 2024. p. 114–123 source
The Introduction directly states that crafting high-quality MCQs is time-consuming. The later clause about needing new questions carries a separate citation and is not part of this verdict. (LLM, conf=0.95)
- Not checked — no full text of the sourcecites [6] Indran IR, N G, Paramanathan P, Mustafa N. Twelve tips to leverage AI for efficient and effective medical question generation: a guide for educators using Chat GPT. Medical Teacher 2023;2:1–6. source
- Not checked — no full text of the sourcecites [7] Zuckerman M, et al. ChatGPT for assessment writing. Med Teach. 2023;45:1224–7. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Cheung et al. reported that manual creation took 10 times longer than automatic generation [4].”
What the sources say
- Supportedcites [4] Cheung BHH et al. ChatGPT versus human in generating medical graduate exam multiple choice questions-A multinational prospective study. In: Hong Kong SAR, Singapore, Ireland, and the United Kingdom. PLOS ONE 2023;18:1–12 . source
The Question construction section reports 211 minutes 33 seconds for two human examiners to draft 50 questions, versus 20 minutes 25 seconds for ChatGPT to generate 50—about 10.4 times as long. No page number is provided in our copy. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the neurology-related questions, the domain expert did not perceive them as pertinent topic-wise or well-aligned with the course content, found them oftentimes too easy and too general.”
- Also stated in: § Discussion · plain
“However, for neurology, the majority of questions were considered neither pertinent nor well-aligned, and oftentimes too broad and hence too easy.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the neurology-related questions, the domain expert did not perceive them as pertinent topic-wise or well-aligned with the course content, found them oftentimes too easy and too general.”
- Also stated in: § Discussion · plain
“However, for neurology, the majority of questions were considered neither pertinent nor well-aligned, and oftentimes too broad and hence too easy.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The majority of questions were not ambiguous.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 77 claims and to verify the 3 references still open.
- Probably not a substantial claim. 45
- It may state two separate claims as one. 16
Citations
Cites
The assessment of clinical skills/competence/performance
Miller, G. E. (1990) · Academic Medicine · doi:10.1097/00001888-199009000-00045 · Cited by 4,883 · cited 1×
“Models like Bloom’s taxonomy [1] or Miller’s pyramid [2] present different levels of learning objectives and competences.”
A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1503_5 · Cited by 973 · cited 2×
“• All of the above : Usage of this option is generally discouraged [11].”
Post-examination analysis of objective tests
Tavakol, M., & Dennick, R. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.564682 · Cited by 198 · cited 1×
“A more advanced tool for determining the discriminative power of an item is the point bi-serial correlation coefficient [14].”
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T.-H. (2023) · PLOS ONE · doi:10.1371/journal.pone.0290691 · Cited by 195
Mulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132 · cited 4×
“AQG techniques can be classified into rule-based and neural network-based approaches [9].”
InfluentialA Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132 · cited 3×
“The creation of high-quality MCQs is a time-consuming task [5, 6], even more considering that changes in the curriculum or students collecting questions from previous years requires teachers to create new sets of questions every once in a while [7].”
Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions
Laupichler, M. C., Rother, J. F., Kadow, I. C. G., Ahmadi, S., & Raupach, T. (2023) · Academic Medicine · doi:10.1097/acm.0000000000005626 · Cited by 132 · cited 1×
“Laupichler et al. compared student performance of 25 MCQs generated by ChatGPT (GPT − 3.5) against 25 MCQs created by human medical educators, each with 1 key and 4 distractors [24].”
InfluentialIndran, I. R., Paranthaman, P., Gupta, N., & Mustafa, N. (2024) · Medical Teacher · doi:10.1080/0142159x.2023.2294703 · Cited by 89 · cited 5×
“Where possible, GPT-4 should be preferred [6].”
InfluentialChatGPT for assessment writing
Zuckerman, M., Flood, R., Tan, R. J. B., Kelp, N., Ecker, D. J., Menke, J., & Lockspeiser, T. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2249239 · Cited by 83 · cited 5×
“Zuckerman et al. used ChatGPT 2 to create USMLE 3 -style MCQ items with a clinical vignette, vital signs and exam findings [7].”
InfluentialGenerating Multiple Choice Questions for Computing Courses Using Large Language Models
Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D. H., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342898 · Cited by 60
Research works printed but not linked: 6
- Bloom BS. Taxonomy of educational objectives: the classification of educational goals. Boston: Allyn and Bacon; 1956.
- Bertrand C. et al. In: Pelaccia T (ed) Choisir un outil d’évaluationComment (mieux) former et évaluer les étudiants en médecine et en sciences de la santé? De Boeck Supérieur. 2016. pp. 357–370
- Dijkstra R, Genç Z, Kayal S, Kamps J. Reading comprehension quiz generation using generative pre-trained transformers. In: Sosnovsky SA, Brusilovsky P, Lan AS, editors, Proceedings of the Fourth International Workshop on Intelligent Textbooks 2022 co-located with 23d International Conference on Artificial Intelligence in Education (AIED 2022), Durham, UK, July 27, 2022, Vol. 3192 of CEUR Workshop Proceedings, 4–17 (CEUR-WS.org, 2022). https://ceur-ws.org/Vol-3192/itb22_p1_full5439.pdf.
- Gilal AR et al. Question guru: an automated multiple-choice question generation system. In: Al-Sharafi MA, Al-Emran M, Al-Kabi MN, Shaalan K, editors Proceedings of the 2nd International Conference on Emerging Technologies and Intelligent Systems. Cham: Springer International Publishing 2023. p. 501–514
- Grévisse C, Florez H, Leon M. Comparative quality analysis of GPT-based multiple choice question generation. In: Florez H, Leon M, editors. Applied informatics. Cham: Springer Nature Switzerland; 2023. p. 435–47.
- Lewis P et al. In: Larochelle H, Ranzato M, Hadsell R, Balcan M, Lin H (eds) Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in neural information processing systems, Vol. 33, 9459–9474 (Curran Associates, Inc., 2020). https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf.
Cited by
Large Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 219
“Our findings also reveal that LLMs are employed not only for code generation but also for code-related Q&A tasks [82, 116, 119, 124] and MCQs [48, 49, 105, 109] across various CS courses.”
Humble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 55
“Therefore, oversight by, for example, a teacher is recommended when using GAI in educational context to ensure that student use and system output is in educational alignment with course content (Grévisse et al., 2024).”
InfluentialMulti-step retrieval and reasoning improves radiology question answering with large language models
Wind, S., Sopa, J., Truhn, D., Lotfinia, M., Nguyen, T.-T., Bressem, K. K., Adams, L. C., Rusu, M., Koestler, H., Wellein, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02250-5 · Cited by 40
Automatic distractor generation in multiple-choice questions: a systematic literature review
Awalurahman, H. W., & Budi, I. D. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2441 · Cited by 24
Awalurahman, H. W., Aji, R. F., & Budi, I. D. (2025) · IEEE Access · doi:10.1109/access.2025.3590423 · Cited by 24
Metadata
- Authors
- Grévisse, C., Pavlou, M. A. S., & Schneider, J. G.
- Affiliations
- University of Luxembourg (LU)
- Corresponding author
- Grévisse, Christian
- Venue
- SN Computer Science
- Published
- 2024-06-10
- Volume
- 5
- Issue
- 5
- ISSN
- 2661-8907 (ISSN-L)
- OpenAlex
- W4399487593
- Cited by
- 41
- References
- 20
- Citations by year
- 2026: 27 · 2025: 11 · 2024: 3
- FWCI
- 15.35
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- SN Computer Science (publishedVersion, cc-by)
- Open Repository and Bibliography (University of Luxembourg) (submittedVersion)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- multiple-choice question generation, large language models, automatic question generation, GPT, item writing guidelines, distractor quality, Moodle plugin, computer science education, medical education, human oversight, instructional alignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine. https://app.cochat.ai/papers/10.1007/s42979-024-02963-6
- The paper
- Grévisse, C., Pavlou, M. A. S., & Schneider, J. G. (2024). Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine. SN Computer Science. https://doi.org/10.1007/s42979-024-02963-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.