Using GPT-4 to Provide Tiered, Formative Code Feedback
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) have shown promise in generating sensible code explanation and feedback in programming exercises. In this experience report, we discuss the process of using one of these models (OpenAI's GPT-4) to generate individualized feedback for students' Java code and pseudocode. We instructed GPT-4 to generate feedback for 113 submissions to four programming problems in an Algorithms and Data Structures class. We prompted the model with example feedback (few-shot learning) and instruction to (1) give feedback on conceptual understanding, syntax, and time complexity, and (2) suggest follow-up actions based on students' code or provide guiding questions. Overall, GPT-4 provided accurate feedback and successfully built on students' ideas in most submissions. Human evaluators (computer science instructors and tutors) rated GPT-4's hints as useful in guiding students' next steps. Model performance varied with programming problems but not submission quality. We reflect on where the model performed well and fell short, and discuss the potential of integrating LLM-generated, individualized feedback into computer science instruction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167 · cited 1×
“Two evaluators (a CS instructor and a senior CS undergraduate) used the rubric to rate a subset of 20 feedback separately and achieved substantial agreement (percent agreement 75%-100%; Gwet’s AC1 range [17]).”
MethodVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
A systematic literature review of automated feedback generation for programming exercises
Keuning, H., Jeuring, J., & Heeren, B. (2019) · ACM Transactions on Computing Education · doi:10.1145/3231711 · Cited by 370 · cited 3×
“Meanwhile, formative feedback systems remind students of task requirements, give examples to illustrate concepts, or provide hints and solutions to help students resolve errors and identify next steps [20, 21, 27, 28, 31].”
BackgroundDenny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
Corbett, A. T., & Anderson, J. R. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365111 · Cited by 280 · cited 1×
“Receiving effective feedback when programming can enhance students’ learning, engagement, and motivation to pursue computer science [7, 16, 24].”
BackgroundExploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145
Sonkar, S., Liu, N., Mallick, D., & Baraniuk, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.130 · Cited by 35
Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations
Al-Hossami, E., Bunescu, R., Teehan, R., Powell, L., Mahajan, K., & Dorodchi, M. (2023) · Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) · doi:10.18653/v1/2023.bea-1.57 · Cited by 23
Cited by
A GPT-Based Code Review System With Accurate Feedback for Programming Education
Lee, D.-K., & Joe, I. (2025) · IEEE Access · doi:10.1109/access.2025.3581139 · Cited by 20
Generative AI in Computer Science Education: Challenges and Opportunities
Franklin, D., Denny, P., Gonzalez-Maldonado, D. A., & Tran, M. (2025) · doi:10.1017/9781009581738 · Cited by 20
Mapeamento Sistemático sobre o Uso de IAs Generativas no Ensino de Programação
Ferreira de Sousa, J., & Silveira, F. R. D. V. (2026) · Revista Eletrônica de Iniciação Científica em Computação · doi:10.5753/reic.2026.7169
Assessing the Impact and Underlying Pathways of Sequenced AI feedback on Student Learning
Cao, J., Zhao, C. Q., Schunn, C., McLaughlin, E. A., Lin, J., & Koedinger, K. R. (2026) · arXiv · doi:10.48550/arxiv.2604.07469
Metadata
- Authors
- Nguyen, H., & Allan, V.
- Venue
- Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2024-03-07
- Pages
- 958-964
- OpenAlex
- W4392564315
- Cited by
- 37
- References
- 10
- FWCI
- 16.78
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using GPT-4 to Provide Tiered, Formative Code Feedback. https://app.cochat.ai/papers/10.1145/3626252.3630960
- The paper
- Nguyen, H., & Allan, V. (2024). Using GPT-4 to Provide Tiered, Formative Code Feedback. Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1. https://doi.org/10.1145/3626252.3630960
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.