Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments
Open accessAbstract
Recent advances in artificial intelligence have led to the development of large language models (LLMs), which are able to generate text, images, and source code based on prompts provided by humans. In this paper, we explore the capabilities of an LLM - OpenAI's GPT-3 model to provide feedback for student written code. Specifically, we examine the feasibility of GPT-3 to check, critique and suggest changes to code written by learners in an online programming exam of an undergraduate Python programming course.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When used to generate questions or hints to support students in solving problems, the research is inconclusive.
“When used to generate questions or hints to support students in solving problems, the research is inconclusive [21,30,82,151,192]”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [30]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs
Cadar, C., Dunbar, D., & Engler, D. (2008) · doi:10.5555/1855741.1855756 · Cited by 2,655 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
(2000) · doi:10.1145/351240.351266 · Cited by 753 · see citing works
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 561 · see citing works
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · see citing works
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · see citing works
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287 · see citing works
Grading student programs using ASSYST
Jackson, D., & Usher, M. (1997) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/268084.268210 · Cited by 254 · see citing works
Using Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235 · see citing works
Generating Diverse Code Explanations using the GPT-3 Large Language Model
MacNeil, S., Tran, A., Mogil, D., Bernstein, S., Ross, E., & Huang, Z. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 2 · doi:10.1145/3501709.3544280 · Cited by 200 · see citing works
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195 · see citing works
Using GitHub Copilot to Solve Simple Programming Problems
Wermelinger, M. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569830 · Cited by 191 · see citing works
Improving student performance by evaluating how well students test their own programs
Edwards, S. H. (2003) · Journal on Educational Resources in Computing · doi:10.1145/1029994.1029995 · Cited by 179 · see citing works
Running students' software tests against each others' code
Edwards, S. H., Shams, Z., Cogswell, M., et al. (2012) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2157136.2157202 · Cited by 39 · see citing works
The Implications of Large Language Models for CS Teachers and Students
MacNeil, S., Kim, J., Leinonen, J., Denny, P. C., Bernstein, S., Becker, B. A., Michel Wermelinger, B., Hellas, A., Tran, A., Sarsa, S., et al. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545947.3573358 · Cited by 37 · see citing works
A gimmick to integrate software testing throughout the curriculum
Goldwasser, M. H. (2002) · ACM SIGCSE Bulletin · doi:10.1145/563517.563446 · Cited by 27 · see citing works
Wrenn, J., Krishnamurthi, S., & Fisler, K. (2018) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3230977.3230999 · Cited by 25 · see citing works
LEGenT: Localizing Errors and Generating Testcases for CS1
Agarwal, N., & Karkare, A. (2022) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3491140.3528282 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- James Finnie-Ansley, Paul Denny, Brett A Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of Ope- nAI Codex on Introductory Programming. In Australasian Computing Education Conference. 10-19.
Cited by
Sorted by citations
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · see citing works
Budhiraja, R., Joshi, I., Challa, J. S., Akolekar, H. D., & Kumar, D. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636257 · Cited by 52 · see citing works
Jordan, M., Ly, K., & Soosai Raj, A. G. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630897 · Cited by 39 · see citing works
Singh, A., Brooks, C., Wang, X., Li, W., Kim, J., & Wilson, D. (2024) · Proceedings of the 14th Learning Analytics and Knowledge Conference · doi:10.1145/3636555.3636853 · Cited by 18 · see citing works
Chang, C. I., Choi, W. C., & Choi, I. C. (2024) · Proceedings of the 2024 7th Artificial Intelligence and Cloud Computing Conference · doi:10.1145/3719384.3719439 · Cited by 11 · see citing works
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12 · see citing works
- Authors
- Balse, R., Valaboju, B., Singhal, S., Warriem, J. M., & Prasad, P.
- Affiliations
- Indian Institute of Technology Madras (IN)
- Flame University (IN)
- Venue
- Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2023-06-29
- Pages
- 292-298
- OpenAlex
- W4382654255
- Cited by
- 63
- References
- 18
- Citations by year
- 2026: 11 · 2025: 20 · 2024: 26 · 2023: 6
- FWCI
- 19.05
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) (publishedVersion)
- Funders
- Indian Institute of Technology Madras
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- GPT-3, large language models, automated feedback, programming assessment, code review, student code, Python programming, automated grading
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
