The DOI matches its Crossref record; 200 of the 200 references it cites were found in a registry or an index.
A Systematic Literature Review of Automated Feedback Generation for Programming Exercises
Open accessAbstract
Formative feedback, aimed at helping students to improve their work, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss’ feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Findings showed that mostly automated feedback for programming exercises has been used to identify mistakes rather than fixing them.”
From Artificial Intelligence for Assessment and Feedback to Enhance Student Success in Higher Education (2022) · cites this paper as [76]
Citations
Cites
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · cited 2×
“The authors see a role for automated feedback at all three levels as defined by Hattie and Timperley [70]: “task level,” “process level,” and “self-regulation level,” discarding feedback at the “self level” because of its limited effect on learning.”
BackgroundShute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631
Knowledge tracing: Modeling the acquisition of procedural knowledge
Corbett, A. T., & Anderson, J. R. (1995) · User Modelling and User-Adapted Interaction · doi:10.1007/bf01099821 · Cited by 2,324
Claessen, K., & Hughes, J. (2011) · ACM SIGPLAN Notices · doi:10.1145/1988042.1988046 · Cited by 1,018
Lowering the barriers to programming
Kelleher, C. L., & Pausch, R. F. (2005) · ACM Computing Surveys · doi:10.1145/1089733.1089734 · Cited by 887
The Behavior of Tutoring Systems
VanLehn, K. A. (2006) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2006-16(3)02 · Cited by 836
Hovemeyer, D., & Pugh, W. (2004) · Companion to the ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA Companion) · doi:10.1145/1028664.1028717 · Cited by 770
Sahami, M., Roach, S., Cuadros-Vargas, E., Danyluk, A., Dodge, R., Fisher, K. S., France, R., Grossman, D., Hawthorne, B., Katz, R., et al. (2013) · ACM eBooks · doi:10.1145/2534860 · Cited by 733
Failure rates in introductory programming
Bennedsen, J., & Caspersen, M. E. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1272848.1272879 · Cited by 677
McCracken, M., Almstrum, V. L., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/572133.572137 · Cited by 628
Research works printed but not linked: 31
- 10.5555/1018423.1019977 · DOI
- Beierle Christoph, 2004, Proceedings of the International Workshop on Teaching Logic Programming. 18--34
- 10.5555/977403.978384 · DOI
- Brusilovsky Peter, 1996, Proceedings of the Conference on Learning Sciences. 357--362
- Julio, Proceedings of the International Technology, Education and Development Conference. 5691--5700
- de Barros Leliane Nunes, 2006, Proceedings of the Monet Workshop on Model-Based System at ECAI.
- Foxley Eric, Proceedings of the CAA Conference.
- Ghosh Moumita, Proceedings of the Conference on Information Technology and Applications.
- Holland Jay, 2009, Proceedings of the Conference on Computers in Education. 142--146
- Innocenti Carlo, 1991, Proceedings of the PEG Conference on Knowledge Based Environments for Teaching and Learning. 322--329
- Lewis Johnson W., 1984, Proceedings of the AAAI Conference. 162--168
- Köllmann Carsten, 2006, Proceedings of the Workshop on Graph Based Tools, Electronic Communications of the EASST, 1
- Le Nguyen-Thinh, 2006, Proceedings of the Conference on Computers in Education. 63--70
- Le Nguyen-Thinh, 2009, Proceedings of the Conference on Computers in Education. 51--58
- Le Nguyen-Thinh, 2011, Proceedings of the Conference on Cognition and Exploratory Learning in Digital Age. 43--50
- Le Nguyen-Thinh, 2014, Proceedings of the Workshop on AI-supported Education for Computer Science. 51--60
- MacNish Cara, 2002, Proceedings of the IEEE Conference on Advanced Learning Technologies. 317--321
- Matloobi Roozbeh, 2009, Proceedings of the Interactive Computer Aided Learning Conference, 1
- Mitrovic Antonija, User Modeling
- Perelman Daniel, 2014, Proceedings of the Workshop on Data Mining for Educational Assessment and Feedback.
- Peylo Christoph, 2000, Proceedings of the Workshop on Adaptive and Intelligent Web-Based Education Systems, ITS. 70--80
- Pribela Ivan, 2011, Proceedings of the Conference on Information Technology.
- Rahman Khirulnizam Abd, 2007, Proceedings of the National Conference on Programming.
- del Pino Juan Carlos, 2012, Proceedings of the Conference on e-Learning, e-Business, Entreprise Information Systems, 8 e-Government.
- Romli Rohaida, 2010, Proceedings of the International Symposium in Information Technology. 1186--1192
- Matthew, Proceedings of the ASEE Annual Conference and Exposition.
- Striewe Michael, 2009, Proceedings of the Conference on Computer Supported Education 2
- 10.5555/276953.276964 · DOI
- Truong Nghi, 2004, Proceedings of the Australasian Conference on Computing Education, 30
- 10.5555/1434921.1434923 · DOI
- Vizcaíno Aurora, 1839, Intelligent Tutoring Systems. LNCS
Cited by
Sorted by citations
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Artificial Intelligence for Assessment and Feedback to Enhance Student Success in Higher Education
Hooda, M., Rana, C., Dahiya, O., Rizwan, A., & Hossain, M. S. (2022) · Mathematical Problems in Engineering · doi:10.1155/2022/5215722 · Cited by 444
Automatic feedback in online learning environments: A systematic literature review
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100027 · Cited by 440
Deeva, G., Bogdanova, D., Serral, E., Snoeck, M., & Weerdt, J. D. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.104094 · Cited by 292
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“attempts to provide more contextualized help to students learning to program, although effective programming support remains a challenge [43].”
BackgroundInfluential
Metadata
- Authors
- Keuning, H., Jeuring, J., & Heeren, B.
- Affiliations
- Windesheim University of Applied Sciences (NL)
- Utrecht University (NL)
- Open University of the Netherlands (NL)
- Venue
- ACM Transactions on Computing Education
- Published
- 2018-09-28
- Volume
- 19
- Issue
- 1
- Pages
- 1-43
- ISSN
- 1946-6226 (ISSN-L)
- OpenAlex
- W2894540915
- Cited by
- 370
- References
- 200
- Citations by year
- 2026: 72 · 2025: 71 · 2024: 60 · 2023: 58 · 2022: 46 · 2021: 29 · 2020: 22 · 2019: 12
- FWCI
- 2.95
- Type
- journal article
- Language
- en
- Open access
- green
- dspace.library.uu.nl
- Locations
- ACM Transactions on Computing Education (publishedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- DSpace (Open University in the Netherlands) (submittedVersion, other-oa)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Funders
- Netherlands Organisation for Scientific Research (023.005.063)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Student Assessment and Feedback
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- automated feedback, programming exercises, formative feedback, programming education, adaptive feedback, tool evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
