CoChat
Sign Up
Cited by 370Teaching and Learning ProgrammingStudent Assessment and FeedbackIntelligent Tutoring Systems and Adaptive Learning

The DOI matches its Crossref record; 200 of the 200 references it cites were found in a registry or an index.

A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

Open access
Keuning, Hieke, Jeuring, Johan, Heeren, Bastiaan·2019·ACM Transactions on Computing Education
DOI 10.1145/3231711
Open this paper in CoChat — free account

Abstract

Formative feedback, aimed at helping students to improve their work, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss’ feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.

Overview

Integrity signals

Registry

Crossref · 10.1145/3231711

Cites

200 of 200 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Flagged first, then by citations

Show all 200 references

Research works printed but not linked: 31

  1. 10.5555/1018423.1019977 · DOI
  2. Beierle Christoph, 2004, Proceedings of the International Workshop on Teaching Logic Programming. 18--34
  3. 10.5555/977403.978384 · DOI
  4. Brusilovsky Peter, 1996, Proceedings of the Conference on Learning Sciences. 357--362
  5. Julio, Proceedings of the International Technology, Education and Development Conference. 5691--5700
  6. de Barros Leliane Nunes, 2006, Proceedings of the Monet Workshop on Model-Based System at ECAI.
  7. Foxley Eric, Proceedings of the CAA Conference.
  8. Ghosh Moumita, Proceedings of the Conference on Information Technology and Applications.
  9. Holland Jay, 2009, Proceedings of the Conference on Computers in Education. 142--146
  10. Innocenti Carlo, 1991, Proceedings of the PEG Conference on Knowledge Based Environments for Teaching and Learning. 322--329
  11. Lewis Johnson W., 1984, Proceedings of the AAAI Conference. 162--168
  12. Köllmann Carsten, 2006, Proceedings of the Workshop on Graph Based Tools, Electronic Communications of the EASST, 1
  13. Le Nguyen-Thinh, 2006, Proceedings of the Conference on Computers in Education. 63--70
  14. Le Nguyen-Thinh, 2009, Proceedings of the Conference on Computers in Education. 51--58
  15. Le Nguyen-Thinh, 2011, Proceedings of the Conference on Cognition and Exploratory Learning in Digital Age. 43--50
  16. Le Nguyen-Thinh, 2014, Proceedings of the Workshop on AI-supported Education for Computer Science. 51--60
  17. MacNish Cara, 2002, Proceedings of the IEEE Conference on Advanced Learning Technologies. 317--321
  18. Matloobi Roozbeh, 2009, Proceedings of the Interactive Computer Aided Learning Conference, 1
  19. Mitrovic Antonija, User Modeling
  20. Perelman Daniel, 2014, Proceedings of the Workshop on Data Mining for Educational Assessment and Feedback.
  21. Peylo Christoph, 2000, Proceedings of the Workshop on Adaptive and Intelligent Web-Based Education Systems, ITS. 70--80
  22. Pribela Ivan, 2011, Proceedings of the Conference on Information Technology.
  23. Rahman Khirulnizam Abd, 2007, Proceedings of the National Conference on Programming.
  24. del Pino Juan Carlos, 2012, Proceedings of the Conference on e-Learning, e-Business, Entreprise Information Systems, 8 e-Government.
  25. Romli Rohaida, 2010, Proceedings of the International Symposium in Information Technology. 1186--1192
  26. Matthew, Proceedings of the ASEE Annual Conference and Exposition.
  27. Striewe Michael, 2009, Proceedings of the Conference on Computer Supported Education 2
  28. 10.5555/276953.276964 · DOI
  29. Truong Nghi, 2004, Proceedings of the Australasian Conference on Computing Education, 30
  30. 10.5555/1434921.1434923 · DOI
  31. Vizcaíno Aurora, 1839, Intelligent Tutoring Systems. LNCS

Cited by

Sorted by citations

Items 301–370 of 370

Related

See all

Metadata

Authors
Keuning, H., Jeuring, J., & Heeren, B.
Affiliations
  • Windesheim University of Applied Sciences (NL)
  • Utrecht University (NL)
  • Open University of the Netherlands (NL)
ORCID
Venue
ACM Transactions on Computing Education
Publisher
Association for Computing Machinery (ACM)
Published
2018-09-28
Volume
19
Issue
1
Pages
1-43
ISSN
1946-6226 (ISSN-L)
DOI
10.1145/3231711
OpenAlex
W2894540915
Cited by
370
References
200
Citations by year
2026: 72 · 2025: 71 · 2024: 60 · 2023: 58 · 2022: 46 · 2021: 29 · 2020: 22 · 2019: 12
FWCI
2.95
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Transactions on Computing Education (publishedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • DSpace (Open University in the Netherlands) (submittedVersion, other-oa)
  • Utrecht University Repository (Utrecht University) (submittedVersion)
Funders
Netherlands Organisation for Scientific Research (023.005.063)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Social Sciences › Education › Student Assessment and Feedback
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
automated feedback, programming exercises, formative feedback, programming education, adaptive feedback, tool evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review