Cited by 370Teaching and Learning ProgrammingStudent Assessment and FeedbackIntelligent Tutoring Systems and Adaptive Learning

The DOI matches its Crossref record; 200 of the 200 references it cites were found in a registry or an index.

A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

Open access

Keuning, Hieke, Jeuring, Johan, Heeren, Bastiaan·2019·ACM Transactions on Computing Education

DOI 10.1145/3231711

Open this paper in CoChat — free account

Abstract

Formative feedback, aimed at helping students to improve their work, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss’ feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.

Integrity signals

Registry

Crossref · 10.1145/3231711

Cites

200 of 200 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The Power of Feedback
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · cited 2×
“The authors see a role for automated feedback at all three levels as defined by Hattie and Timperley [70]: “task level,” “process level,” and “self-regulation level,” discarding feedback at the “self level” because of its limited effect on learning.”
Background
Focus on Formative Feedback
Shute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631
Knowledge tracing: Modeling the acquisition of procedural knowledge
Corbett, A. T., & Anderson, J. R. (1995) · User Modelling and User-Adapted Interaction · doi:10.1007/bf01099821 · Cited by 2,324
QuickCheck
Claessen, K., & Hughes, J. (2011) · ACM SIGPLAN Notices · doi:10.1145/1988042.1988046 · Cited by 1,018
Lowering the barriers to programming
Kelleher, C. L., & Pausch, R. F. (2005) · ACM Computing Surveys · doi:10.1145/1089733.1089734 · Cited by 887
The Behavior of Tutoring Systems
VanLehn, K. A. (2006) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2006-16(3)02 · Cited by 836
Finding bugs is easy
Hovemeyer, D., & Pugh, W. (2004) · Companion to the ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA Companion) · doi:10.1145/1028664.1028717 · Cited by 770
Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science
Sahami, M., Roach, S., Cuadros-Vargas, E., Danyluk, A., Dodge, R., Fisher, K. S., France, R., Grossman, D., Hawthorne, B., Katz, R., et al. (2013) · ACM eBooks · doi:10.1145/2534860 · Cited by 733
Failure rates in introductory programming
Bennedsen, J., & Caspersen, M. E. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1272848.1272879 · Cited by 677
A multi-national, multi-institutional study of assessment of programming skills of first-year CS students
McCracken, M., Almstrum, V. L., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/572133.572137 · Cited by 628

Flagged first, then by citations

Show all 200 references

Research works printed but not linked: 31

10.5555/1018423.1019977 · DOI
Beierle Christoph, 2004, Proceedings of the International Workshop on Teaching Logic Programming. 18--34
10.5555/977403.978384 · DOI
Brusilovsky Peter, 1996, Proceedings of the Conference on Learning Sciences. 357--362
Julio, Proceedings of the International Technology, Education and Development Conference. 5691--5700
de Barros Leliane Nunes, 2006, Proceedings of the Monet Workshop on Model-Based System at ECAI.
Foxley Eric, Proceedings of the CAA Conference.
Ghosh Moumita, Proceedings of the Conference on Information Technology and Applications.
Holland Jay, 2009, Proceedings of the Conference on Computers in Education. 142--146
Innocenti Carlo, 1991, Proceedings of the PEG Conference on Knowledge Based Environments for Teaching and Learning. 322--329
Lewis Johnson W., 1984, Proceedings of the AAAI Conference. 162--168
Köllmann Carsten, 2006, Proceedings of the Workshop on Graph Based Tools, Electronic Communications of the EASST, 1
Le Nguyen-Thinh, 2006, Proceedings of the Conference on Computers in Education. 63--70
Le Nguyen-Thinh, 2009, Proceedings of the Conference on Computers in Education. 51--58
Le Nguyen-Thinh, 2011, Proceedings of the Conference on Cognition and Exploratory Learning in Digital Age. 43--50
Le Nguyen-Thinh, 2014, Proceedings of the Workshop on AI-supported Education for Computer Science. 51--60
MacNish Cara, 2002, Proceedings of the IEEE Conference on Advanced Learning Technologies. 317--321
Matloobi Roozbeh, 2009, Proceedings of the Interactive Computer Aided Learning Conference, 1
Mitrovic Antonija, User Modeling
Perelman Daniel, 2014, Proceedings of the Workshop on Data Mining for Educational Assessment and Feedback.
Peylo Christoph, 2000, Proceedings of the Workshop on Adaptive and Intelligent Web-Based Education Systems, ITS. 70--80
Pribela Ivan, 2011, Proceedings of the Conference on Information Technology.
Rahman Khirulnizam Abd, 2007, Proceedings of the National Conference on Programming.
del Pino Juan Carlos, 2012, Proceedings of the Conference on e-Learning, e-Business, Entreprise Information Systems, 8 e-Government.
Romli Rohaida, 2010, Proceedings of the International Symposium in Information Technology. 1186--1192
Matthew, Proceedings of the ASEE Annual Conference and Exposition.
Striewe Michael, 2009, Proceedings of the Conference on Computer Supported Education 2
10.5555/276953.276964 · DOI
Truong Nghi, 2004, Proceedings of the Australasian Conference on Computing Education, 30
10.5555/1434921.1434923 · DOI
Vizcaíno Aurora, 1839, Intelligent Tutoring Systems. LNCS

Sorted by citations

Items 101–200 of 370

Advancing Automated Assessment Tools - Opportunities for Innovations in Upper-level Computing Courses: A Position Paper
Hooper, S., Wünsche, B. C., Luxton-Reilly, A., Denny, P. C., & Feng, H. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630772 · Cited by 8
Opening a conversation on responsible environmental data science in the age of large language models
Oliver, R. Y., Chapman, M., Emery, N. C., Gillespie, L., Gownaris, N. J., Leiker, S., Nisi, A. C., Ayers, D., Breckheimer, I., Blondin, H. E., et al. (2024) · Environmental Data Science · doi:10.1017/eds.2024.12 · Cited by 11
A Clustering-Based Computational Model to Group Students With Similar Programming Skills From Automatic Source Code Analysis Using Novel Features
Silva, D. B., Carvalho, D. R., & Junior, C. N. S. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3273926 · Cited by 10
Effects of Automated Feedback in Scratch Programming Tutorials
Obermüller, F., Greifenstein, L., & Fraser, G. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588803 · Cited by 8
A Systematic Literature Review of Methodology of Learning Evaluation Based on Item Response Theory in the Context of Programming Teaching
dos Santos, J. S., Andrade, W. L., Brunet, J., & Melo, M. R. A. (2020) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie44824.2020.9274068 · Cited by 8
Current State and Next Steps on Automated Hints for Students Learning to Code
Toll, D., Wingkvist, A., & Ericsson, M. (2020) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie44824.2020.9274053 · Cited by 8
Investigating the Proficiency of Large Language Models in Formative Feedback Generation for Student Programmers
Kumar, S. S., Lones, M. A., Maarek, M., & Zantout, H. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648380 · Cited by 7
Howzat? Appealing to Expert Judgement for Evaluating Human and AI Next-Step Hints for Novice Programmers
Brown, N. C. C., Weill-Tessier, P., Leinonen, J., Denny, P. C., & Kölling, M. (2025) · ACM Transactions on Computing Education · doi:10.1145/3737885 · Cited by 11
Impact of Adaptive Feedback on Learning Programming with a Serious Game in High Schools’ Classes
Branthôme, M., & Lallé, S. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3699682.3728326 · Cited by 7
Enhancing Student Feedback in Data Science Education: Harnessing the Power of AI-Generated Approaches
Letteri, I., & Vittorini, P. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00492-8 · Cited by 13
Direct Automated Feedback Delivery for Student Submissions based on LLMs
Sölch, M., Dietrich, F. T. J., & Krusche, S. (2025) · SIGSOFT FSE Companion · doi:10.1145/3696630.3727247 · Cited by 10
How Helpful do Novice Programmers Find the Feedback of an Automated Repair Tool?
Kurniawan, O., POSKITT, C. M., Hoque, I. A., Lee, N. T. S., Jégourel, C., & Sockalingam, N. (2023) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale56641.2023.10398393 · Cited by 7
QuizMaster: An Adaptive Formative Assessment System
Lin, F. O., Morland, R., & Yan, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63028-6_5 · Cited by 8
Improving Student Learning with Automated Assessment
Acuña, R., & Bansal, A. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653603 · Cited by 7
DETECT: A Hierarchical Clustering Algorithm for Behavioural Trends in Temporal Educational Data
McBroom, J., Yacef, K., & Koprinska, I. (2020) · International Conference on Artificial Intelligence in Education · doi:10.1007/978-3-030-52237-7_30 · Cited by 8
Student and Lecturer Perceptions of Usability of the Virtual Programming Lab Module for Moodle
Ramos, V. F. C., Cechinel, C., MAGÉ, L., & Lemos, R. R. (2021) · Informatics in Education · doi:10.15388/infedu.2021.14 · Cited by 7
Identifying nonconformities in contributions to programming projects: from an engagement perspective in improving code quality
Nguyen, B.-A., Chen, H., & Dow, C. (2021) · Behaviour and Information Technology · doi:10.1080/0144929x.2021.2017483 · Cited by 7
Jask
Santos, A. L., Soares, T., Garrido, N. M. D. F., & Lehtinen, T. (2022) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3502718.3524761 · Cited by 7
Towards Generating Counterfactual Examples as Automatic Short Answer Feedback
Filighera, A., Tschesche, J., Steuer, T., Tregel, T., & Wernet, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-11644-5_17 · Cited by 7
Impact of Hint Content on Performance and Learning: A Study with Primary School Children in a Scratch Course
Greifenstein, L., Brune, M., Fuchs, T., Heuer, U., & Fraser, G. (2023) · Workshop in Primary and Secondary Computing Education · doi:10.1145/3605468.3605498 · Cited by 7
An Early Review of Generative Language Models in Automated Writing Evaluation: Advancements, Challenges, and Future Directions for Automated Essay Scoring and Feedback Generation
Yue, H., Palermo, C., Liu, R., & He, Y. (2025) · Chinese/English Journal of Educational Measurement and Evaluation · doi:10.59863/famj7696 · Cited by 9
Feedback Formativo Automatizado com LLMs: Desenvolvimento e Análise de um Sistema para Aprendizagem Progressiva em Programação
Silva, F. G., & Aranha, E. H. D. S. (2025) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2025.12807 · Cited by 6
Evaluation of Submission Limits and Regression Penalties to Improve Student Behavior with Automatic Assessment Systems
Lawrence, R., Foss, S., & Urazova, T. (2023) · ACM Transactions on Computing Education · doi:10.1145/3591210 · Cited by 6
An Approach to Generating Adaptive Feedback for Online Formative Assessment
Lin, F. O., & Silva, S. D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-32883-1_8 · Cited by 6
"Let Them Try to Figure It Out First" - Reasons Why Experts (Do Not) Provide Feedback to Novice Programmers
Lohr, D., Kiesler, N., Keuning, H., & Jeuring, J. T. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653530 · Cited by 6
Self-citation
That's Not the Feedback I Need! - Student Engagement with GenAI Feedback in the Tutor Kai
Jacobs, S., Kempf, M., & Kiesler, N. (2025) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3754508.3754512 · Cited by 6
A Systematic Literature Review of Gameful Feedback in Computer Science Education
Willert, N. (2021) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2021.11.10.1551 · Cited by 6
Review of Programming Assignments Automated Assessment Systems
Tarek, M., Ashraf, A., Heidar, M., & Eliwa, E. (2022) · International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC) · doi:10.1109/miucc55081.2022.9781736 · Cited by 6
MAATSE: Prototyping and Evaluating an Open and Modular E-Assessment Tool for STEM Education
Rolwes, A., Stellbauer, P., Lungershausen, U., Čubela, D., Böhm, K., & Neis, P. (2023) · IEEE German Education Conference (GECon) · doi:10.1109/gecon58119.2023.10295151 · Cited by 6
Scalable Feedback for Student Live Coding in Large Courses Using Automatic Error Grouping
Strickroth, S. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653620 · Cited by 6
An Image-Based User Interface Testing Method for Flutter Programming Learning Assistant System
Aung, S. T., Funabiki, N., Aung, L. H., Kinari, S. A., Wai, K. H., & Mentari, M. (2024) · Information · doi:10.3390/info15080464 · Cited by 6
Case Studies in Game-Based Complex Learning
Miller, J. A., & Cooper, S. (2021) · Multimodal Technologies and Interaction · doi:10.3390/mti5120072 · Cited by 6
On How Novices Approach Programming Exercises Before and During Coding
Sáenz, J. P., & Russis, L. D. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3519655 · Cited by 6
Die Zukunft des MINT-Lernens – Band 1
Röth, J., Baum, M., Eilerts, K., Hornung, G., & Trefzger, T. (2022) · Springer eBooks · doi:10.1007/978-3-662-66131-4 · Cited by 9
TESTed—An educational testing framework with language-agnostic test suites for programming exercises
Strijbol, N., Petegem, C. V., Maertens, R., Sels, B., Scholliers, C., Dawyndt, P., & Mesuere, B. (2023) · SoftwareX · doi:10.1016/j.softx.2023.101404 · Cited by 6
Code Quality Defects in Introductory Programming
Řechtáčková, A., Pelánek, R., & Effenberger, T. (2025) · ACM Transactions on Computing Education · doi:10.1145/3749996 · Cited by 5
Automated Grading Method of Python Code Submissions Using Large Language Models and Machine Learning
Mahdaoui, M., Nouh, S., Alaoui, S. E. K., & Kandali, K. (2025) · Information · doi:10.3390/info16080674 · Cited by 5
Bridging awareness and behavior: decoding implicit metacognitive behaviors in AI-assisted programming via fine-grained log analysis
Hou, X., Liu, Z., Wang, G., & Shi, N. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1790196 · Cited by 5
Towards a Success Model for Automated Programming Assessment Systems Used as a Formative Assessment Tool
Sauerwein, C., Antensteiner, T., Oppl, S., Groher, I., Meschtscherjakov, A., Zech, P., & Breu, R. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588848 · Cited by 5
A Systematic Literature Review of Student Assessment Framework in Software Engineering Courses
Fauzan, R., Siahaan, D., Solekhah, M., Saputra, V. W., Bagaskara, A. E., & Karimi, M. I. (2023) · Journal of Information Systems Engineering and Business Intelligence · doi:10.20473/jisebi.9.2.264-275 · Cited by 6
Rethinking Computing Students’ Help Resource Utilization through Sequentiality
Ko, S.-H., & Stephens-Martinez, K. (2025) · ACM Transactions on Computing Education · doi:10.1145/3716860 · Cited by 7
Assisted learning of C programming through automated program repair and feed-back generation
Arifi, S. M., Abbou, R. B., & Zahi, A. (2020) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v20.i1.pp454-464 · Cited by 5
Describing Console I/O Behavior for Testing Student Submissions in Haskell
Westphal, O., & Voigtländer, J. (2020) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.321.2 · Cited by 5
Supporting the Semi-automatic Feedback Provisioning on Programming Assignments
Strickroth, S., & Holzinger, F. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-20617-7_3 · Cited by 5
AP-Coach: formative feedback generation for learning introductory programming concepts
Duong, T. N. B., Shar, L. K., & Shankararaman, V. (2022) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale54877.2022.00060 · Cited by 6
The Diagnosing Behaviour of Intelligent Tutoring Systems
van der Bent, R., Jeuring, J. T., & Heeren, B. J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29736-7_9 · Cited by 5
Relationships Between Computing Students' Characteristics, Help-Seeking Approaches, and Help-Seeking Behavior in Introductory Courses and Beyond
Ko, S.-H., Zahn, M., Stephens-Martinez, K., Velasco, Y., Battestilli, L., & Heckman, S. S. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744214 · Cited by 5
LLM-based Multimodal Feedback Produces Equivalent Learning and Better Student Perceptions than Educator Feedback
Zhao, C. Q., Cao, J., Lin, J. X. C., & Koedinger, K. R. (2026) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3785022.3785124 · Cited by 4
Feedback Assisted by Technology: A Systematic Review of Empirical Research
Huang, W., Stephens, J. M., & Brown, G. (2025) · International Journal of Technology in Education · doi:10.46328/ijte.1061 · Cited by 7
Fine-Tuning GPT-3.5-Turbo for Automatic Feedback Generation
Mazzullo, E., Bulut, O., Walsh, C., Sitarenios, G., & MacIntosh, A. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707735 · Cited by 4
Teaming in the AI Era: AI-Augmented Frameworks for Forming, Simulating, and Optimizing Human Teams
Almutairi, M. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3699682.3727574 · Cited by 4
Integrating Computational Thinking Diagnostic Mechanism and Reflective Learning: An Innovative Approach to Enhance Learning Outcomes in Introductory Programming
Wu, T., Lee, H., Chen, P., Wang, W., & Huang, Y. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70121 · Cited by 5
GenAI Voice Mode in Programming Education
Jacobs, S., & Kiesler, N. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770001 · Cited by 7
A Review of Automatic Question Generation in Teaching Programming
Alshboul, J., & Varga, E. B. (2022) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2022.0131006 · Cited by 7
A Selective Review of Computing Education Research
Malmi, L., & Johri, A. (2023) · Routledge eBooks · doi:10.4324/9781003287483-31 · Cited by 5
The Y-Model - Formalization of Computer Science Tasks in the Context of Adaptive Learning Systems
Lohr, D., Berges, M., Kohlhase, M., Müller, D., & Rapp, M. G. (2023) · IEEE German Education Conference (GECon) · doi:10.1109/gecon58119.2023.10295148 · Cited by 4
FEEDBOT: Formative Design Feedback on Programming Assignments
Zhu, E., Teja, S., Coombes, C., & PATTERSON, D. C. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729063 · Cited by 4
Work In Progress: Sleuth, a programming environment for testing gamification
Katan, S., & Anstead, E. (2020) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon45650.2020.9125098 · Cited by 4
Analysis of typical programming mistakes made by first and second year IT students
Kaczorowska, M. (2020) · Journal of Computer Sciences Institute · doi:10.35784/jcsi.1681 · Cited by 4
Os estudantes leem as mensagens de feedback estendido exibidas em juízes online?
Oliveira, J. D., Salem, F., Oliveira, E., de Oliveira, D. B. F., de Carvalho, L. S. G., & Pereira, F. D. (2020) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/cbie.sbie.2020.1723 · Cited by 4
TextCode: A Tool to Support Problem Solving Among Novice Programmers
Corno, F., Russis, L. D., & Sáenz, J. P. (2021) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc51201.2021.9576164 · Cited by 4
An automated system with a versatile test oracle for assessing student programs
Tang, C. M., Yu, Y. T., & Poon, C. K. (2022) · Computer Applications in Engineering Education · doi:10.1002/cae.22577 · Cited by 4
Microlearning and Automated Assessment – A Framework Implementation of Dissimilar Elements to Achieve Better Educational Outcomes
Skalka, J. (2022) · Springer eBooks · doi:10.1007/978-3-031-13359-6_1 · Cited by 4
When is Automated Feedback a Barrier to Timely Feedback?
DeOrio, A., & Keefer, C. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--40405 · Cited by 4
Clustering Recurrent and Semantically Cohesive Program Statements in Introductory Programming Assignments
Marin, V. J., & Rivero, C. R. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3357960 · Cited by 4
An Epistemic Model-Based Tutor for Imperative Programming
Kumar, A. N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78270-2_38 · Cited by 4
Combining Summative and Formative Evaluation Using Automated Assessment
English, J., & English, T. (2019) · Issues in Informing Science and Information Technology · doi:10.28945/4293 · Cited by 4
CoderBot 2.0: Integrating LLM and Prompt Engineering in the Evolution of an Educational Chatbot
Mendes, A., Parizotto, A., Garcia, R., Garcia, M., Guedes, G., Vilela, R., Valle, P., Balancieri, R., & Silva, W. (2025) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2025.12321 · Cited by 3
A Systematic Review of Studies on the Use of Generative Artificial Intelligence Tools in Programming Education
ÖZGÜL, E. (2026) · Kastamonu Eğitim Dergisi · doi:10.24106/kefdergi.1878122 · Cited by 3
The Dimensions of Abundance in AI-Generated Feedback
Lindsay, E. D., Rodda, A. E., Lindqvist, A. L., Quince, Z., Lim, M., & Jiang, D. (2026) · Education Sciences · doi:10.3390/educsci16030465 · Cited by 3
A Cloud-Based Technology for Conducting In-class Exercises in Data Science and Machine Learning Courses
Pahi, K., & Phan, V. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569838 · Cited by 3
Addressing Misconceptions in Introductory Programming: Automated Feedback in Integrated Development Environments
Fischer, B., Birk, F. T., Iwer, E.-M., Panitz, S. E., & Dörner, R. (2023) · International Conference on Education Technology and Computer · doi:10.1145/3629296.3629297 · Cited by 3
“Questomatic”: Automated Question Formulation System
Jadhav, P., Deore, S., Aute, S., Gaikwad, R., Barphe, S. S., & Meshram, E. (2024) · International Conference on Sustainable Expert Systems (ICSES) · doi:10.1109/icses63445.2024.10763124 · Cited by 3
Automated Formative Feedback for Algorithm and Data Structure Self-Assessment
Araujo, L., López-Ostenero, F., Plaza, L., & Martínez-Romo, J. (2025) · Electronics · doi:10.3390/electronics14051034 · Cited by 9
The Impact of Data-Driven Feedback on Programming Learning: A Cognitive Load Perspective
Liu, M., Zhu, L., Wang, J., & Qian, Y. (2025) · International Conference on Computer Science and Technologies in Education (CSTE) · doi:10.1109/cste64638.2025.11092261 · Cited by 3
A Framework for Generating Diverse Haskell-I/O Exercise Tasks
Westphal, O. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-75333-7_6 · Cited by 4
PERSONALIZED FEEDBACKS BASED ON LEARNING ANALYTICS TO ENHANCE THE LEARNING OF PROGRAMMING
Peter, Z. H. A. Y. (2020) · doi:10.33965/el2020_202007c018 · Cited by 3
Automatic Feedback in the Teaching of Programming in Undergraduate Courses: a Literature Mapping
Conceicao, W., de Holanda, M. T., Macedo, F., Ishikawa, E., Nunes, V., & Silva, D. D. (2022) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie56618.2022.9962723 · Cited by 3
An automatic feedback model for learning programming via block‐based programming platforms
Çakıroğlu, Ü., & Mumcu, S. (2023) · Computer Applications in Engineering Education · doi:10.1002/cae.22652 · Cited by 3
Evaluating Distance Measures for Program Repair
Koutcheme, C., Sarsa, S., Leinonen, J., Haaranen, L., & Hellas, A. (2023) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3568813.3600130 · Cited by 5
JUnit++: An Open Educational Tool for Simplifying Unit Testing
Adeyemi, O., Adiraju, A., Akins, S., Khademi, K., & Hui, B. (2023) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt58122.2023.00012 · Cited by 3
Leveraging Generative Artificial Intelligence to Broaden Participation in Computer Science
Jayachandran, D., Maldikar, P. S., Love, T. S., & Blum, J. J. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31262 · Cited by 3
Generating Next Step Hints for Task Oriented Programs Using Symbolic Execution
Naus, N., & Steenvoorden, T. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-57761-2_3 · Cited by 3
Lecturers’ and Students’ Experiences with an Automated Programming Assessment System
Sauerwein, C., Priller, S., Dobiasch, M., Oppl, S., Felderer, M., & Breu, R. (2023) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2023.008 · Cited by 3
Exploring Students' Self-Confidence in Their Programming Solutions
Strickroth, S. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653589 · Cited by 2
Understanding Optimal Interactions Between Students and A Chatbot During A Programming Task
Shin, J., Castro, L. M. C., Yang, Z., Castelblanco, G., Aggarwal, A., Leite, W. L., & Carroll, B. F. (2024) · Winter Simulation Conference · doi:10.1109/wsc63780.2024.10838799 · Cited by 4
Provide personalized programming learning for individuals based on large language models
Li, C., Lee, X., & Wu, X. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.10.026 · Cited by 4
Challenges, Multiple Attempts, and Trump Cards: A Practice Report of Student’s Exposure to an Automated Correction System for a Programming Challenges Activity
Liénardy, S., Leduc, L., Verpoorten, D., & Donnet, B. (2021) · Revue internationale des technologies en pédagogie universitaire · doi:10.18162/ritpu-2021-v18n2-03 · Cited by 2
Customizing Feedback for Introductory Programming Courses Using Semantic Clusters
Marin, V. J., Hosseini, H., & Rivero, C. R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80421-3_30 · Cited by 2
Write a Line: Tests with Answer Templates and String Completion Hints for Self-Learning in a CS1 Course
Александрович, С. О. (2022) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet55299.2022.9794157 · Cited by 2
Simplifying the creation and maintenance of automated assessments of programming tasks via Test Specific Language
Muuli, E., Palm, R., & Lepp, M. (2022) · International Conference on Education and E-Learning (ICEEL) · doi:10.1145/3578837.3578840 · Cited by 2
Seeing Program Output Improves Novice Learning Gains
Leinonen, J., Hellas, A., & Edwards, J. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588796 · Cited by 2
Witter: A Library for White-Box Testing of Introductory Programming Algorithms
Caniço, A. B., & Santos, A. L. (2023) · ACM SIGPLAN International Symposium on SPLASH-E (SPLASH-E) · doi:10.1145/3622780.3623650 · Cited by 2
Exploring Machine Learning Methods to Identify Patterns in Students' Solutions to Programming Assignments
Liu, X., Castellanos, H., Wiese, L. J., & Magana, A. J. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342972 · Cited by 2
Utilization of Information Entropy in Training and Evaluation of Students’ Abstraction Performance and Algorithm Efficiency in Programming
Wu, Z., Liu, H., & Xiao, C. (2024) · IEEE Transactions on Education · doi:10.1109/te.2024.3354297 · Cited by 2
ASSIST: Automated Feedback Generation for Syntax and Logical Errors in Programming Exercises
Praet, L. V., Hoobergs, J., & Schrijvers, T. (2024) · ACM SIGPLAN International Symposium on SPLASH-E (SPLASH-E) · doi:10.1145/3689493.3689981 · Cited by 2
A Software Framework Based on JUnit for Automated Software Testing in Computer Science Courses
Zmuda, M. A. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892903 · Cited by 2
Requirements for an Automated Assessment Tool for Learning Programming by Doing
Rump, A., Zaytsev, V., & Mader, A. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988998 · Cited by 3
Learning from Teachers: AI-Driven Feedback for a High School Python Serious Game
Branthôme, M., & Lallé, S. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729543 · Cited by 2
Navigating the Landscape of Automated Feedback Generation Techniques for Programming Exercises
Kumar, S. S., Lones, M. A., Maarek, M., & Zantout, H. (2025) · ACM Transactions on Computing Education · doi:10.1145/3764593 · Cited by 2

Authors: Keuning, H., Jeuring, J., & Heeren, B.
Affiliations: Windesheim University of Applied Sciences (NL)
Utrecht University (NL)
Open University of the Netherlands (NL)
ORCID: Keuning, Hieke
Jeuring, Johan
Heeren, Bastiaan
Venue: ACM Transactions on Computing Education
Publisher: Association for Computing Machinery (ACM)
Published: 2018-09-28
Volume: 19
Issue: 1
Pages: 1-43
ISSN: 1946-6226 (ISSN-L)
DOI: 10.1145/3231711
OpenAlex: W2894540915
Cited by: 370
References: 200
Citations by year: 2026: 72 · 2025: 71 · 2024: 60 · 2023: 58 · 2022: 46 · 2021: 29 · 2020: 22 · 2019: 12
FWCI: 2.95
Type: journal article
Language: en
Open access: green
dspace.library.uu.nl
Licence: other-oa
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: ACM Transactions on Computing Education (publishedVersion)
Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
DSpace (Open University in the Netherlands) (submittedVersion, other-oa)
Utrecht University Repository (Utrecht University) (submittedVersion)
Funders: Netherlands Organisation for Scientific Research (023.005.063)
Topics: Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Social Sciences › Education › Student Assessment and Feedback
Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords: automated feedback, programming exercises, formative feedback, programming education, adaptive feedback, tool evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks