Cited by 370Teaching and Learning ProgrammingStudent Assessment and FeedbackIntelligent Tutoring Systems and Adaptive Learning

The DOI matches its Crossref record; 200 of the 200 references it cites were found in a registry or an index.

A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

Open access

Keuning, Hieke, Jeuring, Johan, Heeren, Bastiaan·2019·ACM Transactions on Computing Education

DOI 10.1145/3231711

Open this paper in CoChat — free account

Abstract

Formative feedback, aimed at helping students to improve their work, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss’ feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.

Integrity signals

Registry

Crossref · 10.1145/3231711

Cites

200 of 200 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The Power of Feedback
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · cited 2×
“The authors see a role for automated feedback at all three levels as defined by Hattie and Timperley [70]: “task level,” “process level,” and “self-regulation level,” discarding feedback at the “self level” because of its limited effect on learning.”
Background
Focus on Formative Feedback
Shute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631
Knowledge tracing: Modeling the acquisition of procedural knowledge
Corbett, A. T., & Anderson, J. R. (1995) · User Modelling and User-Adapted Interaction · doi:10.1007/bf01099821 · Cited by 2,324
QuickCheck
Claessen, K., & Hughes, J. (2011) · ACM SIGPLAN Notices · doi:10.1145/1988042.1988046 · Cited by 1,018
Lowering the barriers to programming
Kelleher, C. L., & Pausch, R. F. (2005) · ACM Computing Surveys · doi:10.1145/1089733.1089734 · Cited by 887
The Behavior of Tutoring Systems
VanLehn, K. A. (2006) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2006-16(3)02 · Cited by 836
Finding bugs is easy
Hovemeyer, D., & Pugh, W. (2004) · Companion to the ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA Companion) · doi:10.1145/1028664.1028717 · Cited by 770
Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science
Sahami, M., Roach, S., Cuadros-Vargas, E., Danyluk, A., Dodge, R., Fisher, K. S., France, R., Grossman, D., Hawthorne, B., Katz, R., et al. (2013) · ACM eBooks · doi:10.1145/2534860 · Cited by 733
Failure rates in introductory programming
Bennedsen, J., & Caspersen, M. E. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1272848.1272879 · Cited by 677
A multi-national, multi-institutional study of assessment of programming skills of first-year CS students
McCracken, M., Almstrum, V. L., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/572133.572137 · Cited by 628

Flagged first, then by citations

Show all 200 references

Research works printed but not linked: 31

10.5555/1018423.1019977 · DOI
Beierle Christoph, 2004, Proceedings of the International Workshop on Teaching Logic Programming. 18--34
10.5555/977403.978384 · DOI
Brusilovsky Peter, 1996, Proceedings of the Conference on Learning Sciences. 357--362
Julio, Proceedings of the International Technology, Education and Development Conference. 5691--5700
de Barros Leliane Nunes, 2006, Proceedings of the Monet Workshop on Model-Based System at ECAI.
Foxley Eric, Proceedings of the CAA Conference.
Ghosh Moumita, Proceedings of the Conference on Information Technology and Applications.
Holland Jay, 2009, Proceedings of the Conference on Computers in Education. 142--146
Innocenti Carlo, 1991, Proceedings of the PEG Conference on Knowledge Based Environments for Teaching and Learning. 322--329
Lewis Johnson W., 1984, Proceedings of the AAAI Conference. 162--168
Köllmann Carsten, 2006, Proceedings of the Workshop on Graph Based Tools, Electronic Communications of the EASST, 1
Le Nguyen-Thinh, 2006, Proceedings of the Conference on Computers in Education. 63--70
Le Nguyen-Thinh, 2009, Proceedings of the Conference on Computers in Education. 51--58
Le Nguyen-Thinh, 2011, Proceedings of the Conference on Cognition and Exploratory Learning in Digital Age. 43--50
Le Nguyen-Thinh, 2014, Proceedings of the Workshop on AI-supported Education for Computer Science. 51--60
MacNish Cara, 2002, Proceedings of the IEEE Conference on Advanced Learning Technologies. 317--321
Matloobi Roozbeh, 2009, Proceedings of the Interactive Computer Aided Learning Conference, 1
Mitrovic Antonija, User Modeling
Perelman Daniel, 2014, Proceedings of the Workshop on Data Mining for Educational Assessment and Feedback.
Peylo Christoph, 2000, Proceedings of the Workshop on Adaptive and Intelligent Web-Based Education Systems, ITS. 70--80
Pribela Ivan, 2011, Proceedings of the Conference on Information Technology.
Rahman Khirulnizam Abd, 2007, Proceedings of the National Conference on Programming.
del Pino Juan Carlos, 2012, Proceedings of the Conference on e-Learning, e-Business, Entreprise Information Systems, 8 e-Government.
Romli Rohaida, 2010, Proceedings of the International Symposium in Information Technology. 1186--1192
Matthew, Proceedings of the ASEE Annual Conference and Exposition.
Striewe Michael, 2009, Proceedings of the Conference on Computer Supported Education 2
10.5555/276953.276964 · DOI
Truong Nghi, 2004, Proceedings of the Australasian Conference on Computing Education, 30
10.5555/1434921.1434923 · DOI
Vizcaíno Aurora, 1839, Intelligent Tutoring Systems. LNCS

Sorted by citations

Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Artificial Intelligence for Assessment and Feedback to Enhance Student Success in Higher Education
Hooda, M., Rana, C., Dahiya, O., Rizwan, A., & Hossain, M. S. (2022) · Mathematical Problems in Engineering · doi:10.1155/2022/5215722 · Cited by 444
Automatic feedback in online learning environments: A systematic literature review
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100027 · Cited by 440
A review of automated feedback systems for learners: Classification framework, challenges and opportunities
Deeva, G., Bogdanova, D., Serral, E., Snoeck, M., & Weerdt, J. D. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.104094 · Cited by 292
“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“attempts to provide more contextualized help to students learning to program, although effective programming support remains a challenge [43].”
BackgroundInfluential
Automated Grading and Feedback Tools for Programming Education: A Systematic Review
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2024) · ACM Transactions on Computing Education · doi:10.1145/3636515 · Cited by 262
Personalized feedback in digital learning environments: Classification framework and literature review
Maier, U., & Klotz, C. (2022) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2022.100080 · Cited by 218
CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145
How Novices Use LLM-based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment
Kazemitabaar, M., Hou, X., Henley, A. Z., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631806 · Cited by 202
Exploring the Potential of Large Language Models to Generate Formative Programming Feedback
Kiesler, N., Lohr, D., & Keuning, H. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343457 · Cited by 113
Self-citation
AI or Human? Evaluating Student Feedback Perceptions in Higher Education
Nazaretsky, T., Mejia-Domenzain, P., Swamy, V., Frej, J., & Käser, T. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72315-5_20 · Cited by 76
Towards understanding the effective design of automated formative feedback for programming assignments
Hao, Q., Smith, D. H., Ding, L., Ko, A. J., Ottaway, C., Wilson, J., Arakawa, K., Turcan, A., Poehlman, T., & Greer, T. (2021) · Computer Science Education · doi:10.1080/08993408.2020.1860408 · Cited by 90
Is technology always helpful?: A critical review of the impact on learning outcomes of education technology in supporting formative assessment in schools
See, B. H., Gorard, S. A. C., Lu, B., Dong, L., & Siddiqui, N. (2021) · Research Papers in Education · doi:10.1080/02671522.2021.1907778 · Cited by 121
Building a Comprehensive Automated Programming Assessment System
Mekterović, I., Brkić, L., Milašinović, B., & Baranović, M. (2020) · IEEE Access · doi:10.1109/access.2020.2990980 · Cited by 74
Formative assessment strategies for students' conceptions—The potential of learning analytics
Stanja, J., Gritz, W., Krugel, J., Hoppe, A., & Dannemann, S. (2022) · British Journal of Educational Technology · doi:10.1111/bjet.13288 · Cited by 70
Exploring differences in self-regulated learning strategy use between high- and low-performing students in introductory programming: An analysis of eye-tracking and retrospective think-aloud data from program comprehension
Cheng, G., Zou, D., Xie, H., & Wang, F. L. (2023) · Computers & Education · doi:10.1016/j.compedu.2023.104948 · Cited by 74
Automated Code Assessment for Education: Review, Classification and Perspectives on Techniques and Tools
Combéfis, S. (2022) · Software · doi:10.3390/software1010002 · Cited by 65
Feedback-Generation for Programming Exercises With GPT-4
Azaiz, I., Kiesler, N., & Strickroth, S. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653594 · Cited by 91
A Tutoring System to Learn Code Refactoring
Keuning, H., Heeren, B. J., & Jeuring, J. T. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3432526 · Cited by 60
Self-citation
Next-Step Hint Generation for Introductory Programming Using Large Language Models
Roest, L., Keuning, H., & Jeuring, J. T. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636259 · Cited by 95
Self-citation
Evaluation of LLM Tools for Feedback Generation in a Course on Concurrent Programming
Ayres, I. E., Callejo, P., Hombrados-Herrera, M. Á., Alario‐Hoyos, C., & Kloos, C. D. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00406-0 · Cited by 73
CodeTailor: LLM-Powered Personalized Parsons Puzzles for Engaging Support While Learning Programming
Hou, X., Wu, Z., Wang, X., & Ericson, B. J. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662032 · Cited by 74
Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge
Koutcheme, C., Dainese, N., Sarsa, S., Hellas, A., Leinonen, J., & Denny, P. C. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653612 · Cited by 82
Automated Assessment and Microlearning Units as Predictors of At-Risk Students and Students’ Outcomes in the Introductory Programming Courses
Skalka, J., & Drlík, M. (2020) · Applied Sciences · doi:10.3390/app10134566 · Cited by 46
Promoting knowledge elaboration, socially shared regulation, and group performance in collaborative learning: an automated assessment and feedback approach based on knowledge graphs
Zheng, L.-Q., Long, M., Chen, B.-D., & Fan, Y. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00415-4 · Cited by 46
Large language models meet user interfaces: The case of provisioning feedback
Pozdniakov, S., Brazil, J., Abdi, S., Bakharia, A., Sadiq, S., Gašević, D., Denny, P., & Khosravi, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100289 · Cited by 43
Highly informative feedback using learning analytics: how feedback literacy moderates student perceptions of feedback
Weidlich, J., Fink, A., Frey, A., Jivet, I., Gombert, S., Menzel, L., Giorgashvili, T., Yau, J. Y.-K., & Drachsler, H. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-025-00539-9 · Cited by 70
An experience of automated assessment in a large‐scale introduction programming course
Zampirolli, F. D. A., Josko, J. M. B., Venero, M. L. F., Kobayashi, G., Fraga, F. J., Goya, D., & Savegnago, H. R. (2021) · Computer Applications in Engineering Education · doi:10.1002/cae.22385 · Cited by 42
A Survey of Automated Programming Hint Generation: The HINTS Framework
McBroom, J., Koprinska, I., & Yacef, K. (2022) · ACM Computing Surveys · doi:10.1145/3469885 · Cited by 39
Towards automated content analysis of educational feedback: A multi-language study
Osakwe, I., Chen, G., Whitelock‐Wainwright, A., Gašević, D., Cavalcanti, A. P., & Mello, R. F. (2022) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2022.100059 · Cited by 40
Towards Giving Timely Formative Feedback and Hints to Novice Programmers
Jeuring, J., Keuning, H., Marwan, S., Bouvier, D., Izu, C., Kiesler, N., Lehtinen, T., Lohr, D., Peterson, A., & Sarsa, S. (2022) · Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3571785.3574124 · Cited by 40
Self-citation
Using GPT-4 to Provide Tiered, Formative Code Feedback
Nguyen, H., & Allan, V. H. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630960 · Cited by 51
Combining LLM-Generated and Test-Based Feedback in a MOOC for Programming
Gabbay, H., & Cohen, A. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662040 · Cited by 47
Evaluating the Quality of LLM-Generated Explanations for Logical Errors in CS1 Student Programs
Balse, R., Kumar, V., Prasad, P., & Warriem, J. M. (2023) · Bangalore Annual Compute Conference · doi:10.1145/3627217.3627233 · Cited by 42
You're (Not) My Type‐ Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?
Lohr, D., Keuning, H., & Kiesler, N. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13107 · Cited by 51
Self-citation
Your Answer is Incorrect... Would you like to know why? Introducing a Bilingual Short Answer Feedback Dataset
Filighera, A., Parihar, S., Steuer, T., Meuser, T., & Ochs, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.587 · Cited by 33
Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices
Xiao, R., Hou, X., & Stamper, J. C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650937 · Cited by 46
A Comparison of Immediate and Scheduled Feedback in Introductory Programming Projects
Leinonen, J., Denny, P., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499372 · Cited by 32
Evaluating the Application of Large Language Models to Generate Feedback in Programming Education
Jacobs, S., & Jaschke, S. (2024) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon60312.2024.10578838 · Cited by 37
Rubric Is All You Need: Improving LLM-Based Code Evaluation With Question-Specific Rubrics
Pathak, A., Gandhi, R., Uttam, V. G., Ramamoorthy, A., Ghosh, P., Jindal, A. R., Verma, S., Mittal, A., Ased, A., Khatri, C., et al. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744220 · Cited by 62
Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics
Pathak, A., Gandhi, R., Uttam, V., Ramamoorthy, A., Ghosh, P., Jindal, A. R., Verma, S., Mittal, A., Ased, A., Khatri, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.23989 · Cited by 30
Student Refactoring Behaviour in a Programming Tutor
Keuning, H., Heeren, B. J., & Jeuring, J. T. (2020) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3428029.3428043 · Cited by 29
Self-citation
Digitalization of Multistep Chemistry Exercises with Automated Formative Feedback
Eitemüller, C., Trauten, F., Striewe, M., & Walpuski, M. (2023) · Journal of Science Education and Technology · doi:10.1007/s10956-023-10043-2 · Cited by 37
Code Quality Defects across Introductory Programming Topics
Effenberger, T., & Pelánek, R. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499415 · Cited by 28
Who Gives Feedback Matters: Student Biases Towards Human and AI ‐Generated Formative Feedback
Nazaretsky, T., Mejia‐Domenzain, P., Swamy, V., Frej, J., & Käser, T. (2026) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70153 · Cited by 26
Adaptive Assessment and Content Recommendation in Online Programming Courses: On the Use of Elo-rating
Vesin, B., Mangaroska, K., Akhüseyinoğlu, K., & Giannakos, M. N. (2022) · ACM Transactions on Computing Education · doi:10.1145/3511886 · Cited by 37
Promoting Early Engagement with Programming Assignments Using Scheduled Automated Feedback
Denny, P., Whalley, J., & Leinonen, J. (2021) · Proceedings of the 23rd Australasian Computing Education Conference · doi:10.1145/3441636.3442309 · Cited by 24
Customizable and scalable automated assessment of C/C++ programming assignments
Delgado‐Pérez, P., & Medina‐Bulo, I. (2020) · Computer Applications in Engineering Education · doi:10.1002/cae.22317 · Cited by 25
Learning From Errors: Exploring the Effectiveness of Enhanced Error Messages in Learning to Program
Zhou, Z., Wang, S.-J., & Qian, Y. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.768962 · Cited by 24
Evaluating Language Models for Generating and Judging Programming Feedback
Koutcheme, C., Dainese, N., Sarsa, S., Hellas, A., Leinonen, J., Ashraf, S., & Denny, P. C. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701791 · Cited by 43
Intelligent Deep-Learning Tutoring System to Assist Instructors in Programming Courses
Álvarez, D. R., & Mesa, F. J. (2023) · IEEE Transactions on Education · doi:10.1109/te.2023.3331055 · Cited by 22
Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming
Alkafaween, U., Al-Bluwi, I., & Denny, P. C. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13100 · Cited by 23
What Are Cybersecurity Education Papers About?
Švábenský, V., Vykopal, J., & Čeleda, P. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366816 · Cited by 21
Leveraging ChatGPT for personalized reflective learning in programming education: effects on self-efficacy, higher-order thinking, and project implementation skills
Lee, H.-Y., Chen, P.-H., Lin, C.-J., Huang, Y.-M., & Wu, T.-T. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13733-z · Cited by 21
Code-quality evaluation scheme for assessment of student contributions to programming projects
Chen, H., Nguyen, B.-A., & Dow, C. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111273 · Cited by 21
Non-Expert Programmers in the Generative AI Future
Feldman, M. Q., & Anderson, C. J. (2024) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3663384.3663393 · Cited by 34
Building a Corpus of Task-Based Grading and Feedback Systems for Learning and Teaching Programming
Strickroth, S., & Striewe, M. (2022) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v12i5.31283 · Cited by 20
A Feasible Study of a Deep Learning Model Supporting Human–Machine Collaborative Learning of Object-Oriented Programming
Wang, F.-H. (2022) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2022.3226345 · Cited by 21
Machine Learning-Based Automated Grading and Feedback Tools for Programming: A Meta-Analysis
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588822 · Cited by 20
GRADESTYLE: GitHub-Integrated and Automated Assessment of Java Code Style
Iddon, C., Giacaman, N., & Terragni, V. (2023) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet58685.2023.00024 · Cited by 20
The Responsible Development of Automated Student Feedback with Generative AI
Lindsay, E. D., Zhang, M., Johri, A., & Bjerva, J. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016572 · Cited by 42
Experiences from Integrating Large Language Model Chatbots into the Classroom
Hellas, A., Leinonen, J., & Leppänen, L. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690101 · Cited by 32
Let’s Ask Students About Their Programs, Automatically
Lehtinen, T., Santos, A. L., & Sorva, J. (2021) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00054 · Cited by 19
Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring with Feedback
Fateen, M., Wang, B., & Mine, T. (2024) · IEEE Access · doi:10.1109/access.2024.3508747 · Cited by 29
Annete: An Intelligent Tutoring Companion Embedded into the Eclipse IDE
Day, M., Penumala, M. R., & Gonzalez-Sanchez, J. (2019) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi48466.2019.00018 · Cited by 18
One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks
Birillo, A., Artser, E., Potriasaeva, A., Vlasov, I., Dzialets, K., Golubev, Y., Герасимов, И. В., Keuning, H., & Bryksin, T. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699556 · Cited by 30
Self-citation
Software Analytics to Support Students in Object-Oriented Programming Tasks: An Empirical Study
Ardimento, P., Bernardi, M. L., & Cimitile, M. (2020) · IEEE Access · doi:10.1109/access.2020.3010172 · Cited by 17
tAIfa: Enhancing Team Effectiveness and Cohesion with AI-Generated Automated Feedback
Almutairi, M., Chiang, C., Bai, Y., & Gómez-Zará, D. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729197 · Cited by 19
Development of a Programming Course for Students of a Teacher Training Higher Education Institution Using the Programming Language Python
Prokopyev, M. S., Vlasova, E. Z., Tretyakova, T., Sorochinsky, M. A., & Soloveva, R. A. (2020) · Propósitos y Representaciones · doi:10.20511/pyr2020.v8n3.484 · Cited by 15
Studying the Impact of Auto-Graders Giving Immediate Feedback in Programming Assignments
Mitra, J. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569726 · Cited by 14
Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitions
Furia, C. A., Torkar, R., & Feldt, R. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3611667 · Cited by 14
Learning with Style: Improving Student Code-Style Through Better Automated Feedback
Saliba, L., Shioji, E., de Oliveira, E. A., Cohney, S., & Qi, J. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630889 · Cited by 14
Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
Lehtinen, T., Koutcheme, C., & Hellas, A. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640058 · Cited by 24
Exploring undergraduates’ computational thinking and human-computer interaction patterns in generative progressive prompt-assisted programming learning
Gong, X., Xu, W., & Qiao, A. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-025-00552-y · Cited by 18
Automated Assessment: Experiences From the Trenches
Luxton-Reilly, A., Tempero, E. D., Arachchilage, N. A. G., Angela, C., Denny, P. C., Fowler, A., Giacaman, N., Kontorovich, I., Lottridge, D. M., Manoharan, S., et al. (2023) · Australasian Computing Education Conference · doi:10.1145/3576123.3576124 · Cited by 13
Does Peer Code Review Change My Mind on My Submission?
Strickroth, S. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588802 · Cited by 15
How problem difficulty and order influence programming education outcomes in online judge systems
Wang, J., Lin, P., Tang, Z., & Chen, S. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e20947 · Cited by 13
Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generation
Jacobs, S., & Jaschke, S. (2024) · 2024 36th International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet62301.2024.10663001 · Cited by 13
Scaling Automated Programming Assessment Systems
Mekterović, I., Brkić, L., & Horvat, M. (2023) · Electronics · doi:10.3390/electronics12040942 · Cited by 13
Automated Assessment of Learning Objectives in Programming Assignments
Rump, A., Fehnker, A., & Mader, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80421-3_33 · Cited by 15
Exploring Feedback and Gamification in a Data Modeling Learning Tool
Dæhli, O., Kristoffersen, B., Lauvås, P., & Sandnes, T. (2021) · The Electronic Journal of e-Learning · doi:10.34190/ejel.19.6.2501 · Cited by 13
On the use of feedback in learning computer programming by novices: a systematic literature mapping
Rocha, H. J. B., Tedesco, P. C. D. A. R., & Costa, E. D. B. (2022) · Informatics in Education · doi:10.15388/infedu.2023.09 · Cited by 13
Advancing data justice in education: some suggestions towards a deontological framework
Perrotta, C. (2022) · Learning Media and Technology · doi:10.1080/17439884.2022.2156536 · Cited by 16
Automated feedback for participants of hands-on cybersecurity training
Švábenský, V., Vykopal, J., Čeleda, P., & Dovjak, J. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-12265-8 · Cited by 12
An Adaptive Approach to Provide Feedback for Students in Programming Problem Solving
Silva, P., Costa, E. D. B., & de Araújo, J. M. F. R. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-22244-4_3 · Cited by 12
AI assistance in peer feedback provision: Pedagogically sound, but minimally adopted
Pozdniakov, S., Brazil, J., Banihashem, S. K., Noroozi, O., Gašević, D., Sadiq, S., & Khosravi, H. (2026) · Computers & Education · doi:10.1016/j.compedu.2026.105591 · Cited by 11
Evaluating CodeClusters for Effectively Providing Feedback on Code Submissions
Koivisto, T., & Hellas, A. (2022) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie56618.2022.9962751 · Cited by 11
Academic procrastination, incentivized and self-selected spaced practice, and quiz performance in an online programming problem system: An intensive longitudinal investigation
Zhang, Y., Paquette, L., & Hu, X. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105029 · Cited by 13
An Analysis of Programming Course Evaluations Before and After the Introduction of an Autograder
Hagerer, G., Lahesoo, L., Anschütz, M., Krusche, S., & Groh, G. (2021) · International Conference on Information Technology Based Higher Education and Training (ITHET) · doi:10.1109/ithet50392.2021.9759809 · Cited by 11
Using Programming Autograder Formative Data to Understand Student Growth
Acuña, R., & Bansal, A. (2022) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie56618.2022.9962650 · Cited by 11
Investigating the Use and Effects of Feedback in CodingBat Exercises: An Exploratory Thinking Aloud Study
Kiesler, N. (2023) · Future of Educational Innovation-Workshop Series Data in Action · doi:10.1109/ieeeconf56852.2023.10104622 · Cited by 11
Development of Automatic Source Code Evaluation Tests Using Grey-Box Methods: A Programming Education Case Study
Skalka, J., & Drlík, M. (2023) · IEEE Access · doi:10.1109/access.2023.3317694 · Cited by 10
The Trees in the Forest: Characterizing Computing Students' Individual Help-Seeking Approaches
Ko, S.-H., & Stephens-Martinez, K. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671099 · Cited by 11
PerFuSIT: Personalized Fuzzy Logic Strategies for Intelligent Tutoring of Programming
Chrysafiadi, K., & Virvou, M. K. (2024) · Electronics · doi:10.3390/electronics13234827 · Cited by 11
Partnering with AI: A Pedagogical Feedback System for LLM Integration Into Programming Education
Scholz, N., Nguyen, M. H., Singla, A. K., & Nagashima, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-03873-9_31 · Cited by 10
Café
Liénardy, S., Leduc, L., Verpoorten, D., & Donnet, B. (2020) · Australasian Computing Education Conference · doi:10.1145/3373165.3373176 · Cited by 10
Model‐based automatic grading of object‐oriented programming assignments
Le, D. M. (2021) · Computer Applications in Engineering Education · doi:10.1002/cae.22464 · Cited by 9
Efficient generation of text feedback in object-oriented programming education using cached performer revision
Wang, F.-H. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100481 · Cited by 9
Learning UML database design and modeling with AutoER
Foss, S., Urazova, T., & Lawrence, R. (2022) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) · doi:10.1145/3550356.3559091 · Cited by 9

Authors: Keuning, H., Jeuring, J., & Heeren, B.
Affiliations: Windesheim University of Applied Sciences (NL)
Utrecht University (NL)
Open University of the Netherlands (NL)
ORCID: Keuning, Hieke
Jeuring, Johan
Heeren, Bastiaan
Venue: ACM Transactions on Computing Education
Publisher: Association for Computing Machinery (ACM)
Published: 2018-09-28
Volume: 19
Issue: 1
Pages: 1-43
ISSN: 1946-6226 (ISSN-L)
DOI: 10.1145/3231711
OpenAlex: W2894540915
Cited by: 370
References: 200
Citations by year: 2026: 72 · 2025: 71 · 2024: 60 · 2023: 58 · 2022: 46 · 2021: 29 · 2020: 22 · 2019: 12
FWCI: 2.95
Type: journal article
Language: en
Open access: green
dspace.library.uu.nl
Licence: other-oa
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: ACM Transactions on Computing Education (publishedVersion)
Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
DSpace (Open University in the Netherlands) (submittedVersion, other-oa)
Utrecht University Repository (Utrecht University) (submittedVersion)
Funders: Netherlands Organisation for Scientific Research (023.005.063)
Topics: Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Social Sciences › Education › Student Assessment and Feedback
Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords: automated feedback, programming exercises, formative feedback, programming education, adaptive feedback, tool evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks