The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Optimizing Large Language Models for Auto-Generation of Programming Quizzes
Abstract
This study analyzes the use of Large Language Models (LLMs) like ChatGPT in creating quizzes for Java programming courses, specifically Object-Oriented Programming (CS1) and Data Structures (CS2). It aims to evaluate the accuracy of LLM-generated assessments, understand the benefits and drawbacks of using LLMs in CS education from educators' viewpoints, and identify effective prompt engineering strategies to enhance the quality of educational materials. The research compares quizzes made by LLMs against human-created content to assess their consistency with Java programming principles, alignment with CS1 and CS2 learning goals, and their impact on student engagement and comprehension, providing insights into LLMs' effectiveness in academic assessment creation for computer science education.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In the process of generating CS1 and CS2 quizzes using GenAI, Kumar et al. [194] suggested to consider the copyright and ownership of content and verify whether the material falls under 'fair use' exemptions”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [194]
In the citing paper
“Both students and educators have expressed concerns about the violation of property rights [194, 324].”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [194]
In the citing paper
“but they also stated that 'We know that generated content relies heavily on the training cases, and we assume that the LLM providers have resolved the copyright issues regarding the training and the generated content.”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [194]
Citations
Cites
Evaluating Academic Answers Generated Using ChatGPT
Fergus, S., Botha, M., & Ostovar, M. (2023) · Journal of Chemical Education · doi:10.1021/acs.jchemed.3c00087 · Cited by 300
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
Research works printed but not linked: 20
- 10.1007/978-3-031-36336-8_108 · DOI
- Lu, 2021, Expert-authored and machine-generated short-answer questions, Educational Technology & Society
- Gan, 2019, AI-tutor: Generating tailored remedial questions and answers
- 10.1080/00131857.2020.1728732 · DOI · Paper page
- Denny, 2023, Can We Trust AI-Generated Educational Content?, arXiv preprint
- Clune, 2019, AI-GAs: AI-generating algorithms, arXiv preprint
- Speth, 2023, Investigating the Use of AI-Generated Exercises for Programming Courses
- 10.1080/08993400500150747 · DOI
- Orenstrakh, 2023, Detecting LLM -generated text in computing education, arXiv preprint
- Denny, 2023, Human vs Machine: Comparison of Student-generated and AI-generated Educational Content, arXiv preprint
- 10.1145/3629296.3629298 · DOI
- Sutherland, 2023, Exploring the Role of AI-Generated Feedback
- Thomas, 2023, Comparative analysis of learner sourced responses
- Aboalela, 2023, ChatGPT for generating questions and assessments, arXiv preprint
- 2024, Human-AI interaction regarding the assignments
- 2024, Human-AI interaction
- Patel, 2022, Implementing Inclusive Software Design in the CS Curriculum, 2, 1272
- 10.1109/ISEC57711.2023.10402302 · DOI
- 2024, OpenAI GitHub Repository
- 2024, Link to QuickStart Tutorial Instructions
Cited by
Sorted by citations
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
Metadata
- Authors
- Kumar, Y., Manikandan, A., Li, J. J., & Morreale, P.
- Affiliations
- Kean University (US)
- Venue
- 2024 IEEE Integrated STEM Education Conference (ISEC)
- Publisher
- IEEE
- Published
- 2024-03-09
- Pages
- 1-5
- OpenAlex
- W4402572069
- Cited by
- 9
- References
- 2
- Citations by year
- 2026: 6 · 2025: 3
- FWCI
- 3.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Integrated STEM Education Conference (ISEC) (publishedVersion)
- Funders
- NSF (1834620,2137791)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- large language models, Java programming, CS1, carbon disulfide, object-oriented programming, data structures, prompt engineering, computer science education, student engagement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
