The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Generating Multiple Choice Questions for Computing Courses Using Large Language Models
Abstract
Generating high-quality multiple-choice questions (MCQs) is a time-consuming activity that has led practitioners and researchers to develop community question banks and reuse the same questions from semester to semester. This results in generic MCQs which are not relevant to every course. Template-based methods for generating MCQs require less effort but are similarly limited. At the same time, advances in natural language processing have resulted in large language models (LLMs) that are capable of doing tasks previously reserved for people, such as generating code, code explanations, and programming assignments. In this paper, we investigate whether these generative capabilities of LLMs can be used to craft high-quality M CQs more efficiently, thereby enabling instructors to focus on personalizing MCQs to each course and the associated learning goals. We used two LLMs, GPT-3 and GPT-4, to generate isomorphic MCQs based on MCQs from the Canterbury Question Bank and an Introductory to Low-level C Programming Course. We evaluated the resulting MCQs to assess their ability to generate correct answers based on the question stem, a task that was previously not possible. Finally, we investigate whether there is a correlation between model performance and the discrimination score of the associated MCQ to understand whether low discrimination questions required the model to do more inference and therefore perform poorly. GPT-4 correctly generated the answer for 78.5% of MCQs based on…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Tran et al. reported that GPT-3 presented poor performance at generating correct answers [28].”
From Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine (2024) · cites this paper as [28]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“These capabilities have recently been dramatically improved with the introduction of large language models (LLMs) [35].”
BackgroundKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“Outside of the context of computing education, these models are also capable of passing many standardized tests, including the US Medical Licensing Exam [15].”
MethodKung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Peer Instruction: Ten years of experience and results
Crouch, C. H., & Mazur, E. (2001) · American Journal of Physics · doi:10.1119/1.1374249 · Cited by 2,682 · cited 1×
“For example, during peer instruction [47], quiz questions could be generated just-in-time based on students’ misconceptions.”
BackgroundEssentials of educational measurement
Hanson, R. A. (1973) · Journal of School Psychology · doi:10.1016/0022-4405(73)90057-5 · Cited by 525
Introduction to measurement with theory
Barnett, W. A., Erwin Diewert, W., & Zellner, A. (2011) · Journal of Econometrics · doi:10.1016/j.jeconom.2010.09.001 · Cited by 514
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 3×
“For example, these models perform well on a variety of computing education pedagogical tasks, including generating code [36], explanations of code [12], [38], [41], coding concepts [42], and programming assignments [13].”
BackgroundMethodThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · cited 2×
“Recently, Large Language Models (LLMs) have demonstrated near human-level performance across a variety of tasks such as generating code [10], code explanations [11], [12], and programming assignments [13], [14].”
Background
Research works printed but not linked: 1
- Mosteller, 1989, The ‘muddiest point in the lecture’ as a feedback device, On Teaching and Learning: The Journal of the Harvard-Danforth Center, 3, 10
Cited by
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
“Note that some of the LOs extracted from the courses were not well-defined (e.g., no action verb) and, hence, it was not possible to categorize them in terms of Bloom’s taxonomy.”
BackgroundPrather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 208
Self-citationLarge Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 219
“They can also generate programming problems [101, 112], MCQs [109, 113], and detect AI-generated code despite having false positives [61].”
Wang, P., Jing, Y., & Shen, S. (2025) · The Internet and Higher Education · doi:10.1016/j.iheduc.2025.100996 · Cited by 117
Metadata
- Authors
- Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D. H., & MacNeil, S.
- Affiliations
- Temple University (US)
- Venue
- 2023 IEEE Frontiers in Education Conference (FIE)
- Publisher
- IEEE
- Published
- 2023-10-18
- Pages
- 1-8
- ISSN
- 0190-5848 (ISSN-L)
- OpenAlex
- W4390621416
- Cited by
- 60
- References
- 47
- Citations by year
- 2026: 16 · 2025: 28 · 2024: 16
- FWCI
- 15.42
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Frontiers in Education Conference (FIE) (publishedVersion)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Keywords
- multiple-choice question generation, large language models, GPT-3, GPT-4, computing education, C programming, item discrimination, quiz generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Generating Multiple Choice Questions for Computing Courses Using Large Language Models. https://app.cochat.ai/papers/10.1109/fie58773.2023.10342898
- The paper
- Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D. H., & MacNeil, S. (2023). Generating Multiple Choice Questions for Computing Courses Using Large Language Models. 2023 IEEE Frontiers in Education Conference (FIE). https://doi.org/10.1109/fie58773.2023.10342898
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.