Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 6 of the 40 sources its claims cite were checked for support.

Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine

Open access

Abstract

Assessment is an essential part of education, both for teachers who assess their students as well as learners who may evaluate themselves. Multiple-choice questions (MCQ) are one of the most popular types of knowledge assessment, e.g., in medical education, as they can be automatically graded and can cover a wide range of learning items. However, the creation of high-quality MCQ items is a time-consuming task. The recent advent of Large Language Models (LLM), such as Generative Pre-trained Transformer (GPT), caused a new momentum for automatic question generation solutions. Still, evaluating generated questions according to the best practices for MCQ item writing is needed to ensure docimological quality. In this article, we propose an analysis of the quality of LLM-generated MCQs. We employ zero-shot approaches in two domains, namely computer science and medicine. In the former, we make use of 3 GPT-based services to generate MCQs. In the latter, we developed a plugin for the Moodle learning management system that generates MCQs based on learning material. We compare the generated MCQs against common multiple-choice item writing guidelines. Among the major challenges, we determined that while LLMs are certainly useful in generating MCQs more efficiently, they sometimes create broad items with ambiguous keys or implausible distractors. Human oversight is also necessary to ensure instructional alignment between generated items and course contents. Finally, we propose solutions for AQG developers.

Overview

CoChat Summary

## TL;DR

This paper evaluates the docimological quality of LLM-generated multiple-choice questions (MCQs) in two zero-shot case studies: computer science and medicine. Across both domains, many questions were acceptable, but key problems remained with ambiguous or wrong keys, implausible distractors, and weaker instructional alignment.

## Key findings

  • In the computer science case study, 55 of 150 questions (37%) had at least one flaw. By service, flaws were 19/50 (38%) for Quiz Wizard, 19/50 (38%) for Quizicist, and 17/50 (34%) for ChatGPT.
  • The most common computer science issues were ambiguous keys, followed by keys that were too long and implausible distractors. The paper notes that None of the above and K-type questions did not appear in this dataset.
  • In the medicine case study, 13 of 100 questions had 1 flaw and 1 question had 2 flaws. Among subfields, there was 1 flawed question out of 20 neurology MCQs (5%) and 13 flawed questions out of 80 endocrinology MCQs (16%).
  • For medicine, the main issue was implausible distractors, followed by no key, wrong key, or ambiguous key. Endocrinology questions were generally judged pertinent and aligned, while neurology questions were often not pertinent, not well-aligned, too easy, and too general.

## Methods

  • The study used two case studies with zero-shot LLM-based question generation: one in computer science and one in medicine.
  • In computer science, the authors used 3 GPT-based services: Quiz Wizard, Quizicist, and ChatGPT. They generated 10 questions per chapter for a course with 5 chapters, yielding 150 questions total.
  • In medicine, the authors developed a Moodle plugin using OpenAI’s Assistants API and generated 10 questions per slidedeck from 8 endocrinology slidedecks and 2 neurology slidedecks, yielding 100 MCQs total. Questions were tagged by domain experts for docimological flaws, and in medicine they were also rated on 5-point Likert scales for pertinence, difficulty, specificity, ambiguity, and instructional alignment.

## Limitations

  • The paper explicitly states that questions were only evaluated by a single domain expert each time.
  • It also states there were relatively few questions in neurology compared to endocrinology or computer science.
  • Evident from the design: the study uses zero-shot prompts and fixed output settings in the Moodle plugin (10 questions, 4 options, 1 key), which limits generalizability to other prompting strategies and item formats.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s42979-024-02963-6

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

6 Valid · 0 Not valid · 18 Not checked · 53 Own results and interpretations

19 more claims

Create a free account to see all 77 claims and to verify the 3 references still open.

See all claims — free account
  • Probably not a substantial claim. 45
  • It may state two separate claims as one. 16

Create a free account to see these claims.

Citations

Cites

Show all 20 references

Research works printed but not linked: 6

  1. Bloom BS. Taxonomy of educational objectives: the classification of educational goals. Boston: Allyn and Bacon; 1956.
  2. Bertrand C. et al. In: Pelaccia T (ed) Choisir un outil d’évaluationComment (mieux) former et évaluer les étudiants en médecine et en sciences de la santé? De Boeck Supérieur. 2016. pp. 357–370
  3. Dijkstra R, Genç Z, Kayal S, Kamps J. Reading comprehension quiz generation using generative pre-trained transformers. In: Sosnovsky SA, Brusilovsky P, Lan AS, editors, Proceedings of the Fourth International Workshop on Intelligent Textbooks 2022 co-located with 23d International Conference on Artificial Intelligence in Education (AIED 2022), Durham, UK, July 27, 2022, Vol. 3192 of CEUR Workshop Proceedings, 4–17 (CEUR-WS.org, 2022). https://ceur-ws.org/Vol-3192/itb22_p1_full5439.pdf.
  4. Gilal AR et al. Question guru: an automated multiple-choice question generation system. In: Al-Sharafi MA, Al-Emran M, Al-Kabi MN, Shaalan K, editors Proceedings of the 2nd International Conference on Emerging Technologies and Intelligent Systems. Cham: Springer International Publishing 2023. p. 501–514
  5. Grévisse C, Florez H, Leon M. Comparative quality analysis of GPT-based multiple choice question generation. In: Florez H, Leon M, editors. Applied informatics. Cham: Springer Nature Switzerland; 2023. p. 435–47.
  6. Lewis P et al. In: Larochelle H, Ranzato M, Hadsell R, Balcan M, Lin H (eds) Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in neural information processing systems, Vol. 33, 9459–9474 (Curran Associates, Inc., 2020). https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf.

Cited by

See all 41

Metadata

Authors
Grévisse, C., Pavlou, M. A. S., & Schneider, J. G.
Affiliations
University of Luxembourg (LU)
Corresponding author
Grévisse, Christian
Venue
SN Computer Science
Published
2024-06-10
Volume
5
Issue
5
ISSN
2661-8907 (ISSN-L)
OpenAlex
W4399487593
Cited by
41
References
20
Citations by year
2026: 27 · 2025: 11 · 2024: 3
FWCI
15.35
Type
journal article
Language
en
Open access
Licence
Locations
  • SN Computer Science (publishedVersion, cc-by)
  • Open Repository and Bibliography (University of Luxembourg) (submittedVersion)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
multiple-choice question generation, large language models, automatic question generation, GPT, item writing guidelines, distractor quality, Moodle plugin, computer science education, medical education, human oversight, instructional alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine. https://app.cochat.ai/papers/10.1007/s42979-024-02963-6
The paper
Grévisse, C., Pavlou, M. A. S., & Schneider, J. G. (2024). Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine. SN Computer Science. https://doi.org/10.1007/s42979-024-02963-6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.