The DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
A Survey of Automated Programming Hint Generation: The HINTS Framework
Open accessAbstract
Automated tutoring systems offer the flexibility and scalability necessary to facilitate the provision of high-quality and universally accessible programming education. To realise the potential of these systems, recent work has proposed a diverse range of techniques for automatically generating feedback in the form of hints to assist students with programming exercises. This article integrates these apparently disparate approaches into a coherent whole. Specifically, it emphasises that all hint techniques can be understood as a series of simpler components with similar properties. Using this insight, it presents a simple framework for describing such techniques, the Hint Iteration by Narrow-down and Transformation Steps framework, and surveys recent work in the context of this framework. Findings from this survey include that (1) hint techniques share similar properties, which can be used to visualise them together, (2) the individual steps of hint techniques should be considered when designing and evaluating hint systems, (3) more work is required to develop and improve evaluation methods, and (4) interesting relationships, such as the link between automated hints and data-driven evaluation, should be further investigated. Ultimately, this article aims to facilitate the development, extension, and comparison of automated programming hint techniques to maximise their educational potential.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Proceedings of the 10th Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 554 · cited 1×
“ture of programming hints. Our work is situated more broadly in the area of automated programming tutors. Other reviews in this area have considered techniques for automated assessment (e.g. [22] and [23]), or approaches to 2 tutoring that can be supported by AI techniques [24], such as \example", \simulation&quo…”
MethodA Survey of Automated Assessment Approaches for Programming Assignments
Ala-Mutka, K. M. (2005) · Computer Science Education · doi:10.1080/08993400500150747 · Cited by 546
A systematic literature review of automated feedback generation for programming exercises
Keuning, H., Jeuring, J., & Heeren, B. (2019) · ACM Transactions on Computing Education · doi:10.1145/3231711 · Cited by 370
Automatic Software Repair: A Bibliography
Monperrus, M. (2019) · ACM Computing Surveys · doi:10.1145/3105906 · Cited by 321
Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor
Rivers, K., & Koedinger, K. R. (2017) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0070-z · Cited by 258 · cited 13×
“The set of features can also be searched until some correctness test is passed as in [48, 6, 1, 11], or a model of correct features can be used to narrow them down as in [46, 4, 45, 47]”
BackgroundMethodInfluentialLearning Program Embeddings to Propagate Feedback on Student CodePreprint
Piech, C., Huang, J., Nguyen, A., Phulsuksombati, M., Sahami, M., & Guibas, L. J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05969 · Cited by 205
Intelligent tutoring systems for programming education: a systematic review
Crow, T., Luxton-Reilly, A., & Wuensche, B. (2018) · Proceedings of the 20th Australasian Computing Education Conference · doi:10.1145/3160489.3160492 · Cited by 166 · cited 1×
“or \dialogue"- based approaches. In addition, some work has focused on the general features of programming tutors, such as the programming languages taught, or primary and supplementary features [25]. While some of these reviews also reference automated programming feedback, it is generally brief or not the main…”
BackgroundFifteen years of constraint-based tutors: what we have achieved and where we are going
Mitrović, A. (2011) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9105-9 · Cited by 151
Codewebs: scalable homework search for massive open online programming courses
Nguyen, A., Piech, C., Huang, J., & Guibas, L. J. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568023 · Cited by 150 · cited 5×
“values of some expressions in the program. When a student submits an incorrect program following a particular strategy, the teacher feedback associated with that strategy is given. Patterns Codewebs [38] - teacher feedback is written to address particular bugs in peer data. Then, when these bugs are identied in a new…”
BackgroundMethodInfluentialTowards a Systematic Review of Automated Feedback Generation for Programming Exercises
Keuning, H., Jeuring, J., & Heeren, B. (2016) · Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/2899415.2899422 · Cited by 142 · cited 2×
“Single File vs. Multi File Analysis") and the program representations they require as input (e.g. \Trees vs. Graphs", "Source Code vs. Byte Code Analysis"). In [20] (extended from [21]), all three types are used separately to label programming tools. The tools are labelled based on the technique us…”
BackgroundMethod
Research works printed but not linked: 2
- 10.5555/3045118.3045235 · doi:10.5555/3045118.3045235
- John Stamper , Tiffany Barnes , Lorrie Lehmann , and Marvin Croy . 2008 . The hint factory: Automatic generation of contextualized help for existing computer aided instruction . In Proceedings of the 9th International Conference on Intelligent Tutoring Systems Young Researchers Track. 71–78 . John Stamper, Tiffany Barnes, Lorrie Lehmann, and Marvin Croy. 2008. The hint factory: Automatic generation of contextualized help for existing computer aided instruction. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems Young Researchers Track. 71–78.
Cited by
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“More recently, automated hint generation systems have been employed to generate hints for any programming exercises [58].”
MethodCodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192
“Most of the work has been devoted to various approaches for the generation of effective hints [21, 22] and feedback [18].”
BackgroundKochmar, E., Vu, D. D., Belfer, R., Gupta, V., Serban, I. V., & Pineau, J. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00267-x · Cited by 112
“There is a growing body of research on automated hint generation for programming exercises (McBroom et al., 2019; Price et al., 2019).”
BackgroundYang, A. C. M., Lin, J.-Y., Lin, C.-Y., & Ogata, H. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100309 · Cited by 95
Can Language Models Employ the Socratic Method? Experiments with Code Debugging
Al-Hossami, E., Bunescu, R., Smith, J., & Teehan, R. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630799 · Cited by 60
Metadata
- Authors
- McBroom, J., Koprinska, I., & Yacef, K.
- Affiliations
- The University of Sydney (AU)
- Venue
- ACM Computing Surveys
- Published
- 2021-10-04
- Volume
- 54
- Issue
- 8
- Pages
- 1-27
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2970680470
- Cited by
- 39
- References
- 52
- Citations by year
- 2026: 1 · 2025: 10 · 2024: 9 · 2023: 11 · 2022: 4 · 2021: 3
- FWCI
- 5.71
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- automated feedback, programming education, data-driven evaluation, programming exercises
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Survey of Automated Programming Hint Generation: The HINTS Framework. https://app.cochat.ai/papers/10.1145/3469885
- The paper
- McBroom, J., Koprinska, I., & Yacef, K. (2022). A Survey of Automated Programming Hint Generation: The HINTS Framework. ACM Computing Surveys. https://doi.org/10.1145/3469885
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.