CoChat
Sign Up
Cited by 4Teaching and Learning ProgrammingIntelligent Tutoring Systems and Adaptive LearningOnline Learning and Analytics

The DOI matches its Crossref record; the references it cites are not checked yet.

Exploring the role of automated feedback in programming education: a systematic literature review

Open access
Jung, Yeonji, Lee, Yunseo, Bae, Jiyeong, Kim, Doyong, Choi, Heungsoo, Kang, Minji, Lee, Unggi·2026·Interactive Learning Environments
DOI 10.1080/10494820.2026.2701430
Open this paper in CoChat — free accountOpen PDF

Abstract

Automated feedback systems have become increasingly central to programming education, where learners iteratively write and refine code. Despite rapid technical advances, research on automated feedback remains fragmented across technological, pedagogical, and interactional dimensions. This systematic literature review synthesizes 58 studies published between 2005 and 2024, following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) protocol. Studies were identified through searches of Scopus, ACM Digital Library, and IEEE Xplore, screened against predefined criteria, and analyzed using a five-category coding framework encompassing system architecture, feedback content, learner interaction, educational context, and evaluation. Across the reviewed studies, feedback systems were typically fully automated and delivered immediate feedback focused on task-level support for code correctness and error detection. Recent systems have begun to incorporate adaptive mechanisms and feedback generated by large language models; however, relatively few offer learners meaningful control over when or how feedback is received. Implementation was concentrated in higher education, often in blended course formats, and evaluation emphasized short-term gains, with limited attention to long-term learning, classroom integration, or effects on teaching. Together, these findings contribute a multidimensional understanding of automated feedback as a pedagogical and interactional process embedded in programming learning environments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1080/10494820.2026.2701430

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

References 88 · Cited by 4

Research works printed but not linked: 85

  1. 10.1109/rita.2018.2801898 · DOI
  2. Azcona, D. , Hsiao, I. , & Smeaton, A. F. (2018). Personalizing computer science education by leveraging multimodal learning analytics. In Proceedings of 2018 IEEE frontiers in education conference (FIE) (pp. 1–9). IEEE. https://doi.org/10.1109/fie.2018.8658596 · DOI
  3. Berry, V. , Castelltort, A. , Pelissier, C. , Rousseau, M. , & Tibermacine, C. (2022). ShellOnYou: Learning by doing Unix command line. In Proceedings of the 27th ACM conference on innovation and technology in computer science education (pp. 379–385). Association for Computing Machinery. https://doi.org/10.1145/3502718.3524753. · DOI
  4. Bey, A. , & Bensebaa, T. (2011). ALGO+, an assessment tool for algorithmic competencies. In 2011 IEEE global engineering education conference (EDUCON) (pp. 941–946). IEEE. https://doi.org/10.1109/educon.2011.5773260 · DOI
  5. 10.12700/aph.19.9.2022.9.6 · DOI
  6. Buffardi, K. , & Edwards, S. H. (2015). Reconsidering automated feedback: A test-driven approach. Proceedings of the 46th ACM technical symposium on computer science education (pp. 416–420). Association for Computing Machinery. https://doi.org/10.1145/2676723.2677313 · DOI
  7. 10.3102/00346543065003245 · DOI · Paper page
  8. Casseau, C. , Falleri, J. , Blanc, X. , & Degueule, T. (2021). Immediate feedback for students to solve notebook reproducibility problems in the classroom. In 2021 IEEE symposium on visual languages and human-centric computing (VL/HCC) (pp. 1–5). IEEE. https://doi.org/10.1109/vl/hcc51201.2021.9576363 · DOI
  9. 10.1016/j.caeai.2021.100027 · DOI · Paper page
  10. 10.1007/978-3-319-61911-8_7 · DOI
  11. Croft, D. , & England, M. (2019). Computing with Codio at Coventry University: Online virtual Linux boxes and automated formative feedback. In Proceedings of the 3rd conference on computing education practice (pp. 1–4). IEEE. https://doi.org/10.1145/3294016.3294018 · DOI
  12. 10.1007/978-3-319-17461-7_124 · DOI · Paper page
  13. de Carvalho Filho, R. , Neto, C. , & Santos, D. (2024). Expanding code assessment: A qualitative feedback system for beginning students. In Proceedings of the 16th international conference on computer supported education (pp. 578–584). https://doi.org/10.5220/0012741600003693 · DOI
  14. 10.1016/j.compedu.2020.104094 · DOI
  15. Duong, T. N. B. , Shar, L. K. , & Shankararaman, V. (2022). AP-Coach: Formative feedback generation for learning introductory programming concepts. In Proceedings of 2022 IEEE international conference on teaching, assessment and learning for engineering (TALE) (pp. 323–330). IEEE. https://doi.org/10.1109/tale54877.2022.00060 · DOI
  16. Edwards, S. H. (2014, March). Work-in-progress: Program grading and feedback generation with Web-CAT. In Proceedings of the first ACM conference on learning@ scale conference (pp. 215–216). Association for Computing Machinery. https://doi.org/10.1145/2556325.2567888 · DOI
  17. Edwards, S. H. , & Murali, K. P. (2017). CodeWorkout: Short programming exercises with built-in data collection. In Proceedings of the 2017 ACM conference on innovation and technology in computer science education (pp. 188–193). Association for Computing Machinery. https://doi.org/10.1145/3059009.3059055 · DOI
  18. 10.1007/978-3-030-50436-6_45 · DOI
  19. 10.1007/978-3-031-16290-9_44 · DOI
  20. 10.1145/3629296.3629297 · DOI
  21. 10.1109/fie49875.2021.9637292 · DOI
  22. Gabbay, H. , & Cohen, A. (2022). Investigating the effect of automated feedback on learning behavior in MOOCs for programming. In Proceedings of the 15th international conference on educational data mining (pp. 376–383). https://doi.org/10.5281/zenodo.6853125 · DOI
  23. Gabbay, H. , & Cohen, A. (2024, July). Combining LLM-generated and test-based feedback in a MOOC for programming. In Proceedings of the eleventh ACM conference on learning@ scale (pp. 177–187). Association for Computing Machinery. https://doi.org/10.1145/3657604.3662040 · DOI
  24. Gamper, P. , Heinemann, B. , Ehlenz, M. , & Schroeder, U. (2021). Identifying problem solving strategies of programming novices in a serious game. In Proceedings of 2021 international conference on advanced learning technologies (ICALT) (pp. 91–93). https://doi.org/10.1109/icalt52272.2021.00035 · DOI
  25. 10.1145/2843043.2843070 · DOI
  26. 10.3102/0013189x12463051 · DOI
  27. 10.1145/2807442.2807469 · DOI
  28. 10.1145/3341719 · DOI
  29. 10.1080/02602938.2014.893418 · DOI
  30. 10.1145/3539618.3591680 · DOI
  31. 10.3102/003465430298487 · DOI · Paper page
  32. 10.1177/1469787419872393 · DOI
  33. 10.1177/07356331221087773 · DOI
  34. Iquira, D. , Galarza, M. , & Sharhorodska, O. (2021). Enhancing software engineering courses with a mobile gamified platform: Results of a mixed approach. In Proceedings of 2021 XVI Latin American conference on learning technologies (LACLO) (pp. 534–537). https://doi.org/10.1109/laclo54177.2021.00097 · DOI
  35. Jacobs, S. , & Jaschke, S. (2024, May). Evaluating the application of large language models to generate feedback in programming education. In Proceedings of 2024 IEEE global engineering education conference (EDUCON) (pp. 1–5). IEEE. https://doi.org/10.1109/EDUCON60312.2024.10578838 · DOI
  36. Jansen J. Oprescu A. & Bruntink M. (2017). The impact of automated code quality feedback in programming education. In Proceedings of the 10th seminar on advanced techniques & tools for software evolution (SATToSE 2017) CEUR workshop proceedings (Vol. 2070). https://ceur-ws.org/Vol-2070/paper-04.pdf
  37. 10.1145/3571785.3574124 · DOI · Paper page
  38. 10.1145/3613904.3642773 · DOI · Paper page
  39. 10.1145/3231711 · DOI · Paper page
  40. 10.1145/1822090.1822167 · DOI
  41. 10.1109/educon45650.2020.9125323 · DOI
  42. Krusche, S. , & Berrezueta-Guzman, J. (2023). Introduction to programming using interactive learning. In Proceedings of the 2023 IEEE 35th international conference on software engineering education and training (CSEE&T) (pp. 178–182). IEEE. https://doi.org/10.1109/CSEET58097.2023.00037 · DOI
  43. 10.1109/tale56641.2023.10398393 · DOI
  44. Laakso, M. J. , Salakoski, T. , & Korhonen, A. (2005). The feasibility of automatic assessment and feedback. In D. Kinshuk , G. Sampson , & P. Isaias (Eds.), Proceedings of the IADIS International conference on cognition and exploratory learning in digital age (CELDA) (pp. 113–122). IADIS Press.
  45. 10.1145/3287324.3287378 · DOI · Paper page
  46. 10.1145/3587102.3588796 · DOI
  47. 10.1109/compsac48688.2020.00032 · DOI
  48. 10.1145/3636243.3636262 · DOI
  49. 10.1145/3487050 · DOI · Paper page
  50. 10.1109/icse-seet55299.2022.9794188 · DOI
  51. Marin, V. J. , Pereira, T. , Sridharan, S. , & Rivero, C. R. (2017). Automated personalized feedback in introductory Java programming MOOCs. In Proceedings of 2017 IEEE 33rd international conference on data engineering (ICDE) (pp. 1259–1270). IEEE Computer Society. https://doi.org/10.1109/ICDE.2017.169 · DOI
  52. 10.1145/3636515 · DOI · Paper page
  53. Mirmotahari, O. , Damsa, C. , & Berg, Y. (2018). Individualized assessment and automated feedback in undergraduate computer science education. In J. Kay & R. Luckin (Eds.), Proceedings of the 13th international conference of the learning sciences (ICLS 2018) (pp. 1577–1578). International Society of the Learning Sciences.
  54. 10.1109/iiai-aai.2019.00059 · DOI
  55. Narciss, S. (2008). Feedback strategies for interactive learning tasks. In J. M. Spector , M. D. Merrill , J. van Merrienboer , & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 125–143). Lawrence Erlbaum Associates Edition.
  56. Nascimento, M. , Araujo, E. , Serey, D. , & Figueiredo, J. (2020). The role of source code vocabulary in programming teaching and learning. In Proceedings of 2020 IEEE frontiers in education conference (FIE) (pp. 1–8). IEEE. https://doi.org/10.1109/fie44824.2020.9274137 · DOI
  57. 10.1145/3657604.3664660 · DOI
  58. 10.1080/03075070600572090 · DOI · Paper page
  59. Nieminen, J. H. , Tai, J. , Boud, D. , & Henderson, M. (2022). Student agency in feedback: Beyond the individual. Assessment and Evaluation in Higher Education , 47 , 95–108. · DOI
  60. 10.1136/bmj.n71 · DOI · Paper page
  61. 10.1145/3568812.3603476 · DOI · Paper page
  62. 10.1145/3197091.3197117 · DOI
  63. 10.1145/3372782.3406263 · DOI
  64. 10.1177/2158244019885136 · DOI · Paper page
  65. 10.1109/FIE43999.2019.9028418 · DOI
  66. 10.1145/3377814.3381712 · DOI
  67. 10.1145/3636243.3636259 · DOI
  68. Roy Choudhury R. Yin H. & Fox A. (2016). Scale-driven automatic hint generation for coding style. In Proceedings of intelligent tutoring systems: 13th international conference ITS 2016 Zagreb Croatia June 7–10 2016. Proceedings 13 (pp. 122–132). Springer International Publishing. https://doi.org/10.1007/978-3-319-39583-8_12 · DOI
  69. 10.1109/vlhcc.2018.8506508 · DOI
  70. 10.3102/0034654307313795 · DOI · Paper page
  71. 10.1109/access.2023.3317694 · DOI
  72. 10.1145/3649217.3653620 · DOI
  73. 10.1145/2325296.2325319 · DOI
  74. 10.1109/ICETSIS61505.2024.10459432 · DOI
  75. 10.1109/icse-seet55299.2022.9794157 · DOI
  76. 10.1145/2610384.2628054 · DOI
  77. Tuo, F. , Bai, Y. , Long, S. , Luo, Y. , Wang, T. , Wang, R. , & Yin, G. (2018). A new model of Docker-based e-learning in Hadoop. In Proceedings of the 2018 International conference on distance education and learning (pp. 22–31). Association for Computing Machinery (ACM). https://doi.org/10.1145/3231848.3231875 · DOI
  78. Venables, A. , & Haywood, L. (2003, January). Programming students need instant feedback! In T. Greening & R. Lister (Eds.), Proceedings of the fifth Australasian conference on computing education-volume 20 (pp. 267–272). Australian Computer Society.
  79. 10.1145/3430665.3456367 · DOI
  80. 10.1007/978-3-642-25813-8_2 · DOI
  81. 10.1145/1118178.1118215 · DOI · Paper page
  82. 10.1145/3674399.3674430 · DOI
  83. 10.1109/te.2024.3354297 · DOI
  84. 10.1038/s41562-024-02004-5 · DOI · Paper page
  85. 10.1609/aaai.v38i21.30372 · DOI · Paper page

Metadata

Authors
Jung, Y., Lee, Y., Bae, J., Kim, D., Choi, H., Kang, M., & Lee, U.
Affiliations
  • Texas A&M University (US)
  • University of Wisconsin–Madison (US)
  • Enuma, Inc.
  • Gyeongin National University of Education (KR)
  • Korea University (KR)
  • Daegu Gyeongdong Elementary School
Corresponding author
Lee, Unggi
ORCID
Venue
Interactive Learning Environments
Publisher
Informa UK Limited
Published
2026-08-26
Pages
1-27
ISSN
  • 1049-4820 (ISSN-L)
  • 1744-5191
DOI
10.1080/10494820.2026.2701430
OpenAlex
W7127331836
Cited by
4
References
88
Citations by year
2026: 4
FWCI
3.28
Type
journal article
Language
en
Open access
Licence
doi.org/10.1080/policies-text-data-mining (tdm)
Locations
  • Interactive Learning Environments (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Keywords
automated feedback, programming education, large language models, adaptive feedback, learner control, program correctness, error detection, blended learning, higher education, feedback systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Exploring the role of automated feedback in programming education: a systematic literature review. https://app.cochat.ai/papers/10.1080/10494820.2026.2701430
The paper
Jung, Y., Lee, Y., Bae, J., Kim, D., Choi, H., Kang, M., & Lee, U. (2026). Exploring the role of automated feedback in programming education: a systematic literature review. Interactive Learning Environments. https://doi.org/10.1080/10494820.2026.2701430