A systematic literature review of automated feedback generation for programming exercises
Open accessAbstract
Formative feedback, aimed at helping students to improve their work, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss’ feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Mostly automated feedback for programming exercises has been used to identify mistakes rather than fixing them.
“Findings showed that mostly automated feedback for programming exercises has been used to identify mistakes rather than fixing them.”
From Artificial Intelligence for Assessment and Feedback to Enhance Student Success in Higher Education (2022) · cites this paper as [76]
Cites
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · cited 2× · see citing works
“The authors see a role for automated feedback at all three levels as defined by Hattie and Timperley [70]: “task level,” “process level,” and “self-regulation level,” discarding feedback at the “self level” because of its limited effect on learning.”
BackgroundShute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631 · see citing works
Knowledge tracing: Modeling the acquisition of procedural knowledge
Corbett, A. T., & Anderson, J. R. (1995) · User Modelling and User-Adapted Interaction · doi:10.1007/bf01099821 · Cited by 2,324 · see citing works
Claessen, K., & Hughes, J. (2011) · ACM SIGPLAN Notices · doi:10.1145/1988042.1988046 · Cited by 1,018 · see citing works
Lowering the barriers to programming
Kelleher, C. L., & Pausch, R. F. (2005) · ACM Computing Surveys · doi:10.1145/1089733.1089734 · Cited by 887 · see citing works
The Behavior of Tutoring Systems
VanLehn, K. A. (2006) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2006-16(3)02 · Cited by 821 · see citing works
Hovemeyer, D., & Pugh, W. (2004) · Companion to the ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA Companion) · doi:10.1145/1028664.1028717 · Cited by 770 · see citing works
Sahami, M., Roach, S., Cuadros-Vargas, E., Danyluk, A., Dodge, R., Fisher, K. S., France, R., Grossman, D., Hawthorne, B., Katz, R., et al. (2013) · ACM eBooks · doi:10.1145/2534860 · Cited by 733 · see citing works
Failure rates in introductory programming
Bennedsen, J., & Caspersen, M. E. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1272848.1272879 · Cited by 634 · see citing works
McCracken, M., Almstrum, V. L., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/572133.572137 · Cited by 628 · see citing works
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 561 · see citing works
A Survey of Automated Assessment Approaches for Programming Assignments
Ala-Mutka, K. (2005) · Computer Science Education · doi:10.1080/08993400500150747 · Cited by 550 · see citing works
ELM-ART: An Adaptive Versatile System for Web-Based Instruction
Weber, G., & Brusilovsky, P. L. (2001) · Cited by 546 · see citing works
Students’ Misconceptions and Other Difficulties in Introductory Programming: A Literature Review
Qian, Y., & Lehman, J. (2018) · ACM Transactions on Computing Education · doi:10.1145/3077618 · Cited by 529 · see citing works
A survey of literature on the teaching of introductory programming
Pears, A., Seidman, S. B., Malmi, L., Mannila, L., Adams, E. S., Bennedsen, J., Devlin, M., & Paterson, J. H. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1345375.1345441 · Cited by 481 · see citing works
Feedback in Higher and Professional Education
Boud, D. J., & Molloy, E. (2012) · Routledge eBooks · doi:10.4324/9780203074336 · Cited by 454 · see citing works
Automatic test-based assessment of programming
Douce, C., Livingstone, D., & Orwell, J. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163409 · Cited by 416 · see citing works
Hovemeyer, D., & Pugh, W. (2004) · ACM SIGPLAN Notices · doi:10.1145/1052883.1052895 · Cited by 397 · see citing works
Automated feedback generation for introductory programming assignments
Singh, R. K., Gulwani, S., & Solar-Lezama, A. (2013) · Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2491956.2462195 · Cited by 388 · see citing works
Feedback Strategies for Interactive Learning Tasks
Narciss, S. (2007) · doi:10.4324/9780203880869.ch11 · Cited by 388 · see citing works
A Review of Generic Program Visualization Systems for Introductory Programming Education
Sorva, J., Karavirta, V., & Malmi, L. (2013) · ACM Transactions on Computing Education · doi:10.1145/2490822 · Cited by 367 · see citing works
Effective Tutoring Techniques: A Comparison of Human Tutors and Intelligent Tutoring Systems
Merrill, D. C., Reiser, B. J., Ranney, M., & Trafton, J. G. (1992) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0203_2 · Cited by 298 · see citing works
User Modeling and Adaptive Navigation Support in WWW-Based Tutoring Systems
Weber, G., & Specht, M. M. (1997) · Springer eBooks · doi:10.1007/978-3-7091-2670-7_30 · Cited by 288 · see citing works
PROUST: Knowledge-Based Program Understanding
Johnson, W. L., & Soloway, E. M. (1985) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1985.232210 · Cited by 285 · see citing works
Corbett, A. T., & Anderson, J. R. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365111 · Cited by 280 · see citing works
A New Paradigm for Intelligent Tutoring Systems: Example-Tracing Tutors
Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2009) · doi:10.5555/1734243.1734245 · Cited by 279 · see citing works
Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor
Rivers, K., & Koedinger, K. R. (2015) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0070-z · Cited by 258 · see citing works
Grading student programs using ASSYST
Jackson, D., & Usher, M. (1997) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/268084.268210 · Cited by 254 · see citing works
On automated grading of programming assignments in an academic institution
Cheang, B., Kurnia, A., Lim, A. E. B., & Oon, W.-C. (2003) · Computers & Education · doi:10.1016/s0360-1315(03)00030-7 · Cited by 227 · see citing works
The automated tutoring of introductory computer programming
Anderson, J. R., & Skwarecki, E. J. (1986) · Communications of the ACM · doi:10.1145/6592.6593 · Cited by 225 · see citing works
Strategies for Computer-Based Programming Instruction: Program Completion vs. Program Generation
Van Merriënboer, J. J. G., & De Croock, M. B. M. (1992) · Journal of Educational Computing Research · doi:10.2190/mjdx-9pp4-kfmt-09pm · Cited by 207 · see citing works
Improving student performance by evaluating how well students test their own programs
Edwards, S. H. (2003) · Journal on Educational Resources in Computing · doi:10.1145/1029994.1029995 · Cited by 179 · see citing works
The computer as a tutorial laboratory: the Stanford BIP project
Barr, A. B., Beard, M., & Atkinson, R. C. (1976) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(76)80021-1 · Cited by 177 · see citing works
A Comparative Analysis of Cognitive Tutoring and Constraint-Based Modeling
Mitrović, A., Koedinger, K. R., & Martin, B. (2003) · Lecture notes in computer science · doi:10.1007/3-540-44963-9_42 · Cited by 166 · see citing works
Spacco, J. W., Hovemeyer, D., Pugh, W., Emad, F., Hollingsworth, J. K., & Padua-Perez, N. (2006) · ACM SIGCSE Bulletin · doi:10.1145/1140123.1140131 · Cited by 163 · see citing works
Automated assessment and experiences of teaching programming
Higgins, C., Gray, G. R., Symeonidis, P., & Tsintsifas, A. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163410 · Cited by 163 · see citing works
Instructional design of a programming course
Caspersen, M. E., & Bennedsen, J. (2007) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/1288580.1288595 · Cited by 155 · see citing works
Fully automatic assessment of programming exercises
Saikkonen, R., Malmi, L., & Korhonen, A. (2001) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/377435.377666 · Cited by 154 · see citing works
Intention-Based Diagnosis of Novice Programming Errors
Johnson, W. L., & Gladwin, L. A. (1987) · IEEE Expert · doi:10.1109/mex.1987.4307101 · Cited by 150 · see citing works
Towards a Systematic Review of Automated Feedback Generation for Programming Exercises
Keuning, H., Jeuring, J. T., & Heeren, B. (2016) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2899415.2899422 · Cited by 142 · see citing works
Self-citationAbility-training-oriented automated assessment in introductory programming course
Wang, T., Su, X., Ma, P., Wang, Y., & Wang, K. (2010) · Computers & Education · doi:10.1016/j.compedu.2010.08.003 · Cited by 138 · see citing works
Automated feedback generation for introductory programming assignments
Singh, R. K., Gulwani, S., & Solar-Lezama, A. (2013) · ACM SIGPLAN Notices · doi:10.1145/2499370.2462195 · Cited by 137 · see citing works
Scaffolding students' learning using test my code
Vihavainen, A., Vikberg, T., Luukkainen, M., & Pärtel, M. (2013) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2462476.2462501 · Cited by 135 · see citing works
Nguyen, A., Piech, C., Huang, J., & Guibas, L. J. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568023 · Cited by 108 · see citing works
Understanding and debugging novice programs
Johnson, W. L. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90094-g · Cited by 108 · see citing works
Grading student programs using ASSYST
Jackson, D., & Usher, M. (1997) · ACM SIGCSE Bulletin · doi:10.1145/268085.268210 · Cited by 107 · see citing works
Programming environments for novices
du Boulay, B. (1992) · Lecture notes in computer science · doi:10.1007/3-540-55606-0_4 · Cited by 106 · see citing works
Translating Principles of Effective Feedback for Students into the CS1 Context
Ott, C., Robins, A. V., & Shephard, K. L. (2016) · ACM Transactions on Computing Education · doi:10.1145/2737596 · Cited by 106 · see citing works
LAURA, a system to debug student programs
Adam, A., & Laurent, J.-P. (1980) · Artificial Intelligence · doi:10.1016/0004-3702(80)90023-5 · Cited by 105 · see citing works
Automatic program debugging for intelligent tutoring systems
Murray, W. R. (1987) · Computational Intelligence · doi:10.1111/j.1467-8640.1987.tb00169.x · Cited by 104 · see citing works
Teaching the tacit knowledge of programming to noviceswith natural language tutoring
Lane, H. C., & VanLehn, K. A. (2005) · Computer Science Education · doi:10.1080/08993400500224286 · Cited by 101 · see citing works
The Teaching and Learning of Programming: A Survey of Supporting Software Tools
Gómez-Albarrán, M. (2005) · The Computer Journal · doi:10.1093/comjnl/bxh080 · Cited by 94 · see citing works
Fully automatic assessment of programming exercises
Saikkonen, R., Malmi, L., & Korhonen, A. (2001) · ACM SIGCSE Bulletin · doi:10.1145/507758.377666 · Cited by 90 · see citing works
Software verification and graph similarity for automated evaluation of students’ assignments
Janičić, M. V., Nikolić, M., Tošić, D., & Kunčak, V. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.12.005 · Cited by 89 · see citing works
Teaching and learning programming and software engineering via interactive gaming
Tillmann, N., de Halleux, J., Xie, T., Gulwani, S., & Bishop, J. (2013) · doi:10.5555/2486788.2486941 · Cited by 89 · see citing works
Program Representation for Automatic Hint Generation for a Data-Driven Novice Programming Tutor
Jin, W., Barnes, T. M., Stamper, J. C., Eagle, M. J., Johnson, M. W., & Lehmann, L. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30950-2_40 · Cited by 86 · see citing works
A Virtual Programming Lab for Moodle with automatic assessment and anti-plagiarism features
Rodríguez-del-Pino, J. C., Royo, E. R., & Figueroa, Z. J. H. (2012) · Acceda (Universidad de Las Palmas de Gran Canaria) · Cited by 85 · see citing works
Evaluating assessments of novice programming environments
Gross, P., & Powers, K. D. (2005) · doi:10.1145/1089786.1089796 · Cited by 80 · see citing works
PROGRAMMING ASSIGNMENTS AUTOMATIC GRADING: REVIEW OF TOOLS AND IMPLEMENTATIONS
Caiza, J. C., & Álamo, J. M. D. (2013) · Hispana · Cited by 78 · see citing works
Increasing Adoption of Smart Learning Content for Computer Science Education
Brusilovsky, P. L., Edwards, S. H., Kumar, A. N., Malmi, L., Benotti, L., Buck, D., Ihantola, P., Prince, R., Sirkiä, T., Александрович, С. С., et al. (2014) · doi:10.1145/2713609.2713611 · Cited by 75 · see citing works
The marking system for CourseMaster
Higgins, C., Symeonidis, P., & Tsintsifas, A. (2002) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/544414.544431 · Cited by 73 · see citing works
Assessment process for programming assignments
Ala-Mutka, K., & Järvinen, H.-M. (2004) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2004.1357399 · Cited by 71 · see citing works
Feedback generation for performance problems in introductory programming assignments
Gulwani, S., Radiček, I., & Zuleger, F. (2014) · ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) · doi:10.1145/2635868.2635912 · Cited by 70 · see citing works
Teaching and learning computer programming
Ulloa, M. A. (1980) · ACM SIGCSE Bulletin · doi:10.1145/989253.989263 · Cited by 69 · see citing works
Intelligent Tutor, Environment and Manual for Introductory Programming
Brusilovsky, P. L. (1992) · Programmed Learning and Educational Technology · doi:10.1080/0954730920290104 · Cited by 68 · see citing works
Transformation-based diagnosis of student programs for programming tutoring systems
Xu, S., & Chee, Y. S. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1191799 · Cited by 68 · see citing works
WebToTeach: an interactive focused programming exercise system
Arnow, D. M., & Barshay, O. (2003) · doi:10.1109/fie.1999.839303 · Cited by 66 · see citing works
Learning to program through the web
Truong, N., Bancroft, P., & Roe, P. (2005) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1067445.1067452 · Cited by 64 · see citing works
PASS: An automated system for program assessment
Thorburn, G., & Rowe, G. (1997) · Computers & Education · doi:10.1016/s0360-1315(97)00021-3 · Cited by 64 · see citing works
Interface-based programming assignments and automatic grading of java programs
Helmick, M. T. (2007) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1268784.1268805 · Cited by 62 · see citing works
Automatic programming assessment and test data generation a review on its approaches
Romli, R., Sulaiman, S., & Zamli, K. Z. (2010) · International Symposium on Information Technology · doi:10.1109/itsim.2010.5561488 · Cited by 61 · see citing works
Proceedings 1989 Frontiers in Education Conference
(1989) · doi:10.1109/fie.1989 · Cited by 61 · see citing works
A semi-automated approach to online assessment
Jackson, D. (2000) · doi:10.1145/343048.343160 · Cited by 60 · see citing works
Weber, G. (1996) · Cognitive Science · doi:10.1016/s0364-0213(99)80006-8 · Cited by 57 · see citing works
Developing intelligent programming tutors for novice programmers
Pillay, N. (2003) · ACM SIGCSE Bulletin · doi:10.1145/782941.782986 · Cited by 55 · see citing works
J-Latte: a Constraint-Based Tutor for Java
Holland, J., Mitrović, A., & Martin, B. (2009) · UC Research Repository (University of Canterbury) · Cited by 55 · see citing works
A Review of AI-Supported Tutoring Approaches for Learning Programming
Le, N.-T., Strickroth, S., Groß, S., & Niels Pinkwart, C. (2013) · Studies in computational intelligence · doi:10.1007/978-3-319-00293-4_20 · Cited by 52 · see citing works
A Survey and Critical Analysis of Tools for Learning Programming
Deek, F. P., & McHugh, J. A. M. (1998) · Computer Science Education · doi:10.1076/csed.8.2.130.3820 · Cited by 50 · see citing works
A Classification of Adaptive Feedback in Educational Systems for Programming
Le, N.-T. (2016) · Systems · doi:10.3390/systems4020022 · Cited by 48 · see citing works
A software system for grading student computer programs
Jackson, D. (1996) · Computers & Education · doi:10.1016/s0360-1315(96)00025-5 · Cited by 47 · see citing works
A programming learning system for beginners-a completion strategy approach
Chang, K., Chiao, B.-C., Chen, S.-W., & Hsiao, R.-S. (2000) · IEEE Transactions on Education · doi:10.1109/13.848075 · Cited by 46 · see citing works
The marking system for CourseMaster
Higgins, C., Symeonidis, P., & Tsintsifas, A. (2002) · ACM SIGCSE Bulletin · doi:10.1145/637610.544431 · Cited by 44 · see citing works
Automated feedback on programs means students need less help from teachers
Odekirk-Hash, E., & Zachary, J. L. (2001) · doi:10.1145/364447.364537 · Cited by 43 · see citing works
An interactive functional programming tutor
Gerdes, A., Jeuring, J. T., & Heeren, B. (2012) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2325296.2325356 · Cited by 43 · see citing works
An intelligent tutoring system for introductory C language course
Song, J., Hahn, S. H., Tak, K., & Kim, J. H. (1997) · Computers & Education · doi:10.1016/s0360-1315(97)00003-1 · Cited by 43 · see citing works
MENO-II: An AI-Based Programming Tutor.
Soloway, E. M., Rubin, E., Woolf, B. P., Bonar, J. G., & Johnson, W. L. (1983) · Cited by 42 · see citing works
de Souza, D. M., Maldonado, J. C., & Barbosa, E. F. (2011) · doi:10.1109/cseet.2011.5876088 · Cited by 42 · see citing works
Chee, Y. S. (1995) · Instructional Science · doi:10.1007/bf00890449 · Cited by 41 · see citing works
A Simulated Student Can Improve Collaborative Learning
Vizcaíno, A. (2005) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2005-15(1)02 · Cited by 40 · see citing works
Experiences using test-driven development with an automated grader
Edwards, S. H., & Pérez-Quiñones, M. A. (2007) · Journal of computing sciences in colleges · doi:10.5555/1181849.1181855 · Cited by 39 · see citing works
Guided programming and automated error analysis in an intelligent Prolog tutor
Hong, J. S. (2004) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2004.02.001 · Cited by 38 · see citing works
Ruth, G. R. (1976) · Artificial Intelligence · doi:10.1016/0004-3702(76)90022-9 · Cited by 37 · see citing works
The Behavior of Tutoring Systems
VanlehnKurt (2006) · International Journal of Artificial Intelligence in Education · doi:10.5555/1435351.1435353 · Cited by 37 · see citing works
Analysing Student Programs in the PHP Intelligent Tutoring System
Weragama, D., & Reye, J. (2014) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-014-0014-z · Cited by 36 · see citing works
GAME: a Generic Automated Marking Environment for programming assessment
Blumenstein, M., Green, S., Nguyen, A. T. A., & Muthukkumarasamy, V. (2004) · International Conference on Information Technology: Coding and Computing (ITCC) · doi:10.1109/itcc.2004.1286454 · Cited by 36 · see citing works
Weber, G. (1996) · Cognitive Science · doi:10.1207/s15516709cog2002_2 · Cited by 36 · see citing works
A Review of Static Analysis Approaches for Programming Exercises
Striewe, M., & Goedicke, M. (2014) · Communications in computer and information science · doi:10.1007/978-3-319-08657-6_10 · Cited by 35 · see citing works
Identifying Effective Moves in Tutoring: On the Refinement of Dialogue Act Annotation Schemes
Vail, A. K., & Boyer, K. E. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_24 · Cited by 35 · see citing works
Programming students NEED instant feedback
Venables, A. T., & Haywood, L. (2003) · Australasian Computing Education Conference · Cited by 34 · see citing works
The Ceilidh system for the automatic grading of students on programming courses
Benford, S., Burke, E. K., Foxley, E., & Higgins, C. (1995) · ACM Southeast Regional Conference (ACMSE) · doi:10.1145/1122018.1122050 · Cited by 32 · see citing works
Automatic debugging of Prolog programs in a Prolog Intelligent Tutoring System
Looi, C. (1991) · Instructional Science · doi:10.1007/bf00120883 · Cited by 32 · see citing works
Jurado, F., Redondo, M. Á., & Ortega, M. (2011) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2011.11.002 · Cited by 32 · see citing works
Data-Driven Program Synthesis for Hint Generation in Programming Tutors
Lazar, T., & Ivan Bratko, A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_38 · Cited by 32 · see citing works
Intelligent Learning Environments wthin Blended Learning for Ensuring Effective C Programming Course
Kose, U. (2012) · International Journal of Artificial Intelligence & Applications · doi:10.5121/ijaia.2012.3109 · Cited by 31 · see citing works
Using strategies for assessment of programming exercises
Gerdes, A., Jeuring, J. T., & Heeren, B. (2010) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1734263.1734412 · Cited by 30 · see citing works
A FLEXIBLE AND MODULAR SOFTWARE ARCHITECTURE FOR COMPUTER AIDED ASSESSMENTS AND AUTOMATED MARKING
Striewe, M., Balz, M., & Goedicke, M. (2009) · International Conference on Computer Supported Education · doi:10.5220/0001966900540061 · Cited by 29 · see citing works
Example-based feedback provision using structured solution spaces
Groß, S., Mokbel, B., Paaßen, B., Hammer, B., & Niels Pinkwart, C. (2014) · International Journal of Learning Technology · doi:10.1504/ijlt.2014.065752 · Cited by 29 · see citing works
Learning Feedback in Intelligent Tutoring Systems
Groß, S., Mokbel, B., Hammer, B., & Niels Pinkwart, C. (2015) · Künstliche Intell. · doi:10.1007/s13218-015-0367-y · Cited by 28 · see citing works
Performance analysis of GAME: A generic automated marking environment
Blumenstein, M., Green, S., Fogelman, S., Nguyen, A. T. A., & Muthukkumarasamy, V. (2007) · Computers & Education · doi:10.1016/j.compedu.2006.11.006 · Cited by 28 · see citing works
An Incremental Hint System For Automated Programming Assignments
Antonucci, P., Estler, C., Nikolić, Đ., Piccioni, M., & Meyer, B. (2015) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2729094.2742607 · Cited by 28 · see citing works
Towards an Integrative Learning Environment for Java Programming
Groß, S., & Pinkwart, N. (2015) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2015.75 · Cited by 27 · see citing works
Moghadam, J., Choudhury, R. R., Yin, H., & Fox, A. (2015) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2724660.2728672 · Cited by 27 · see citing works
Collaborative example selection in an intelligent example-based programming environment
Brusilovsky, P. L., & Weber, G. (1996) · International Conference of Learning Sciences · doi:10.22318/icls1996.357 · Cited by 26 · see citing works
Shaffer, S. C. (2005) · ACM SIGCSE Bulletin · doi:10.1145/1083431.1083464 · Cited by 26 · see citing works
Learning to construct quality software with the Ceilidh system
Benford, S., Burke, E. K., & Foxley, E. (1993) · Software Quality Journal · doi:10.1007/bf00402268 · Cited by 26 · see citing works
McCalla, G., Greer, J. E., Barrie, B., & Pospisil, P. (1992) · Computers & Mathematics with Applications · doi:10.1016/0898-1221(92)90148-b · Cited by 25 · see citing works
Interface-based programming assignments and automatic grading of java programs
Helmick, M. T. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1269900.1268805 · Cited by 25 · see citing works
Automatic grading of students' ALGOL programming
Naur, P. (1964) · BIT Numerical Mathematics · doi:10.1007/bf01956028 · Cited by 24 · see citing works
Teaching novice programmers using ProgTest
de Souza, D. M., Isotani, S., & Barbosa, E. F. (2015) · International Journal of Knowledge and Learning · doi:10.1504/ijkl.2015.071054 · Cited by 23 · see citing works
Bridge: Intelligent Tutoring with Intermediate Representations
Bonar, J. G., & Cunningham, R. K. (1988) · doi:10.21236/ada218938 · Cited by 22 · see citing works
The PHP Intelligent Tutoring System
Weragama, D., & Reye, J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_64 · Cited by 22 · see citing works
Jurado, F., Redondo, M. Á., & Ortega, M. (2012) · Computer Applications in Engineering Education · doi:10.1002/cae.21569 · Cited by 21 · see citing works
Suzuki, R., Soares, G., Glassman, E. L., Head, A., D’Antoni, L., & Hartmann, B. (2017) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3027063.3053187 · Cited by 21 · see citing works
Submit! a web-based system for automatic program critiquing
Pisan, Y., Richards, D., Sloane, A. M., Koncek, H., & Mitchell, S. L. (2003) · Australasian Computing Education Conference · Cited by 20 · see citing works
Binary instant feedback on programming exercises can reduce student engagement and promote cheating
Kyrilov, A., & Noelle, D. C. (2015) · doi:10.1145/2828959.2828968 · Cited by 20 · see citing works
A rationale and description of a CAI program to teach the BASIC programming language
Barr, A. B., Beard, M., & Atkinson, R. C. (1975) · Instructional Science · doi:10.1007/bf00157068 · Cited by 19 · see citing works
Gotel, O., Scharff, C., & Wildenberg, A. (2008) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1384271.1384329 · Cited by 19 · see citing works
Pettit, R., & Prather, J. E. (2017) · Journal of computing sciences in colleges · doi:10.5555/3055338.3079060 · Cited by 19 · see citing works
Using functional models in automatic debugging
Allemang, D. (1991) · IEEE Expert · doi:10.1109/64.108947 · Cited by 18 · see citing works
Pärtel, M., Luukkainen, M., Vihavainen, A., & Vikberg, T. (2013) · International Journal of Technology Enhanced Learning · doi:10.1504/ijtel.2013.059495 · Cited by 18 · see citing works
Strategy-based feedback in a programming tutor
Keuning, H., Heeren, B., & Jeuring, J. T. (2014) · doi:10.1145/2691352.2691356 · Cited by 17 · see citing works
Self-citationUsing run time traces in automated programming tutoring
Striewe, M., & Goedicke, M. (2011) · doi:10.1145/1999747.1999832 · Cited by 17 · see citing works
Student Modeling in an Intelligent Programming Tutor
Corbett, A. T., & Anderson, J. R. (1993) · NATO ASI series. Series F : Computer and system sciences · doi:10.1007/978-3-662-11334-9_13 · Cited by 16 · see citing works
Automatic analysis of programming assignments
Beierle, C., Kulaa, M., & Widera, M. (2003) · Cited by 16 · see citing works
Computer Aided Assessments and Programming Exercises with JACKPreprint
Goedicke, M., Striewe, M., & Balz, M. (2018) · RePEc: Research Papers in Economics · doi:10.17185/duepublico/47108 · Cited by 16 · see citing works
Ramadhan, H. A., Deek, F. P., & Shihab, K. (2001) · Artificial Intelligence Review · doi:10.1023/a:1011078011415 · Cited by 16 · see citing works
Evaluation of Guided-Planning and Assisted-Coding with Task Relevant Dynamic Hinting
Jin, W., Corbett, A. T., Lloyd, W., Baumstark, L., & Rolka, C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_40 · Cited by 15 · see citing works
An Automatic Assessment Marking and Plagiarism Detection System
Ghosh, M. R., Verma, B., & Nguyen, A. T. A. (2002) · Griffith Research Online · Cited by 15 · see citing works
A semi-automated approach to online assessment
Jackson, D. (2000) · ACM SIGCSE Bulletin · doi:10.1145/353519.343160 · Cited by 15 · see citing works
AnalyseC: A Framework for Assessing Students' Programs at Structural and Semantic Level
Wu, W., Li, G., Sun, Y., Wang, J., & Lai, T. (2007) · International Conference on Control and Automation · doi:10.1109/icca.2007.4376454 · Cited by 14 · see citing works
Intelligent tutoring system for learning PHP
Weragama, D. (2013) · Cited by 14 · see citing works
The design of the SCENT automated advisor
McCalla, G. I., Bunt, R. B., & Harms, J. J. (1986) · Computational Intelligence · doi:10.1111/j.1467-8640.1986.tb00073.x · Cited by 13 · see citing works
Learning to program through the web
Truong, N., Bancroft, P., & Roe, P. (2005) · ACM SIGCSE Bulletin · doi:10.1145/1151954.1067452 · Cited by 13 · see citing works
Automated feedback on programs means students need less help from teachers
Odekirk-Hash, E., & Zachary, J. L. (2001) · ACM SIGCSE Bulletin · doi:10.1145/366413.364537 · Cited by 13 · see citing works
Early experiences of computer-aided assessment and administration when teaching computer programming
Benford, S., Burke, E., Foxley, E., Gutteridge, N., & Zın, A. M. (2011) · Research in Learning Technology · doi:10.3402/rlt.v1i2.9481 · Cited by 12 · see citing works
A critical analysis and evaluation of Web-based environments for program development
Deek, F. P., Ho, K.-W., & Ramadhan, H. A. (2000) · The Internet and Higher Education · doi:10.1016/s1096-7516(01)00038-0 · Cited by 12 · see citing works
Improving the quality of programming education by online assessment
Fischer, G., & von Gudenberg, J. W. (2006) · doi:10.1145/1168054.1168085 · Cited by 12 · see citing works
Java Facilities for Automating Analysis, Feedback and Assessment of Laboratory Work
MacNish, C. K. (2000) · Computer Science Education · doi:10.1076/0899-3408(200008)10:2;1-c;ft147 · Cited by 12 · see citing works
An Affective and Cognitive Tutoring System for Learning Programming
Estrada, M. L. B., Cabada, R. Z., González-Hernández, F., Bustillos, R. O., & Reyes-Garcia, C. A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-27101-9_12 · Cited by 12 · see citing works
Qualitative Evaluation of the Java Intelligent Tutoring System
Sykes, E. R. (2005) · SOURCE Sheridan's Institutional Repository (Sheridan College) · Cited by 12 · see citing works
Design, Development and Evaluation of the Java Intelligent Tutoring System.
Sykes, E. R. (2010) · Cited by 12 · see citing works
Evaluating the Synergies of Integrating E-Assessment and Software Testing
Majchrzak, T. A., & Usener, C. A. (2013) · Springer eBooks · doi:10.1007/978-1-4614-4951-5_15 · Cited by 11 · see citing works
Exploiting program schemata in a Prolog tutoring system
Gegg-Harrison, T. S. (1993) · Cited by 11 · see citing works
The Grace Tutor: A qualified success
McKendree, J., Radlinski, B., & Atwood, M. E. (1992) · Lecture notes in computer science · doi:10.1007/3-540-55606-0_78 · Cited by 11 · see citing works
Model solutions and properties for diagnosing student programs in Ask-Elle
Jeuring, J. T., van Binsbergen, L. T., Gerdes, A., & Heeren, B. J. (2014) · Computer Science Education Research Conference · doi:10.1145/2691352.2691355 · Cited by 11 · see citing works
Karkalas, S., & Gutiérrez-Santos, S. (2014) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2014.35 · Cited by 11 · see citing works
A Generalized Knowledge approach to comprehend Pascal and C Programs
Ueno, H. (2000) · IEICE Transactions on Information and Systems · Cited by 10 · see citing works
Sant, J. A. (2009) · doi:10.1145/1562877.1562971 · Cited by 10 · see citing works
Towards the use of an automatic assessment system in the teaching of software testing
de Souza, D. M., Oliveira, B. H., Maldonado, J. C., Souza, S. R. S., & Barbosa, E. F. (2014) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2014.7044375 · Cited by 10 · see citing works
Mansouri, F. Z., Gibbon, C. A., & Higgins, C. (1998) · doi:10.1145/282991.283108 · Cited by 10 · see citing works
Problem-based learning in formal and informal learning environments
Šimić, G., & Jevremović, A. (2010) · Interactive Learning Environments · doi:10.1080/10494820.2010.486685 · Cited by 10 · see citing works
Global perceptions on the use of WeBWorK as an online tutor for computer science
Gotel, O., Scharff, C., Wildenberg, A., Bousso, M., Bunthoeurn, C., Phal, D., Kulkarni, V., Palakvangsa-Na-Ayudhya, S., Sarr, C., & Sunetnanta, T. T. (2008) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2008.4720331 · Cited by 10 · see citing works
Incorporating anchored learning in a C# intelligent tutoring system
Hartanto, B., & Reye, J. (2013) · International Conference on Computers in Education · Cited by 9 · see citing works
Keuning, H., Jeuring, J. T., & Heeren, B. (2016) · Utrecht University Repository (Utrecht University) · Cited by 9 · see citing works
Self-citationA System for Automatic Evaluation of Programs for Correctness and Performance
Mandal, A. K., Mandal, C., & Reade, C. (2007) · Lecture notes in business information processing · doi:10.1007/978-3-540-74063-6_29 · Cited by 8 · see citing works
Meaningful learning in the tutoring system for programming
Dadić, T., Stankov, S., & Rosić, M. (2008) · doi:10.1109/iti.2008.4588458 · Cited by 8 · see citing works
Intelligent Tutoring System for Learning Programming
Dadić, T. (2010) · IGI Global eBooks · doi:10.4018/978-1-61692-008-1.ch009 · Cited by 8 · see citing works
Gotel, O., Scharff, C., & Wildenberg, A. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1597849.1384329 · Cited by 8 · see citing works
Advanced Computational Methods for Knowledge Engineering
Le, N.-T., Do, T. V., Nguyên, N. T., & Thi, H. A. L. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-61911-8 · Cited by 8 · see citing works
Rosenthal, T., Suppes, P., & Ben‐Zvi, N. (2003) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2002.1157893 · Cited by 7 · see citing works
Experiences of Using Automated Assessment in Computer Science Courses
English, J., & English, T. (2015) · Journal of Information Technology Education Innovations in Practice · doi:10.28945/2304 · Cited by 7 · see citing works
MacNish, C. K. (2002) · UWA Profiles and Research Repository (University of Western Australia) · Cited by 6 · see citing works
An instructional interpreter for basic
Barr, A., & Beard, M. (1976) · doi:10.1145/800107.803494 · Cited by 6 · see citing works
Radlinski, R., & McKendree, J. (1992) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/142750.142829 · Cited by 6 · see citing works
A specification language for static analysis of student exercises
Köllmann, C., & Goedicke, M. (2008) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2008.47 · Cited by 6 · see citing works
Automation of Java Code Analysis for Programming Exercises
Köllmann, C., & Goedicke, M. (2024) · Technische Universität Berlin – Universitätsbibliothek · doi:10.14279/tuj.eceasst.1.78 · Cited by 5 · see citing works
Extensions to Generic Automated Marking Environment: GAME-2+
Matloobi, R., Blumenstein, M., & Green, S. (2009) · Griffith Research Online · Cited by 5 · see citing works
Computer Assisted Assessment. Research into E-Assessment
Ras, E., & Brinke, D. J.-T. (2015) · Communications in computer and information science · doi:10.1007/978-3-319-27704-2 · Cited by 5 · see citing works
Adding Weights to Constraints in Intelligent Tutoring Systems: Does It Improve the Error Diagnosis?
Le, N.-T., & Pinkwart, N. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23985-4_19 · Cited by 4 · see citing works
Pattie: an automated tutor for top-down programming.
Danielson, R. L. (1975) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 4 · see citing works
Helping novice programmers bridge the conceptual gap
He, Y., Ikeda, M., & Mizoguchi, R. (2002) · doi:10.1109/icesd.1994.302282 · Cited by 4 · see citing works
A hybrid approach for program understanding based on graph parsing and expectation-driven analysis
Kim, S.-M., & Kim, J. H. (1998) · Applied Artificial Intelligence · doi:10.1080/088395198117659 · Cited by 4 · see citing works
INCOM: A WEB-BASED HOMEWORK COACHING SYSTEM FOR LOGIC PROGRAMMING
Le, N.-T., & Pinkwart, N. (2011) · Cited by 4 · see citing works
Sant, J. A. (2009) · ACM SIGCSE Bulletin · doi:10.1145/1595496.1562971 · Cited by 4 · see citing works
Problem Solving Process Oriented Diagnosis in Logic Programming
Le, N.-T., & Menzel, W. (2006) · Cited by 3 · see citing works
Assessment of student programming assignments in COBOL
Hasan, H. M. (1988) · Education and Computing · doi:10.1016/s0167-9287(88)90574-2 · Cited by 3 · see citing works
Knowledge base compilation and the language design game
Sack, W. (1992) · Lecture notes in computer science · doi:10.1007/3-540-55606-0_29 · Cited by 3 · see citing works
CSTutor: An Intelligent Tutoring System that Supports Natural Learning
Hartanto, B., & Reye, J. (2013) · doi:10.5176/2251-2195_cseit13.09 · Cited by 3 · see citing works
Automated Feedback and Recognition through Data Mining in Code Hunt
Perelman, D., Bishop, J., Gulwani, S., & Grossman, D. (2015) · Cited by 3 · see citing works
Proceedings of the Australasian conference on Computing education
(2000) · doi:10.1145/359369 · Cited by 3 · see citing works
PRAM: A Courseware System for the Automatic Assessment of AI Programs
Higgins, C., & Mansouri, F. Z. (2000) · Studies in fuzziness and soft computing · doi:10.1007/978-3-7908-1868-0_10 · Cited by 2 · see citing works
An Enhanced Generic Automated Marking Environment: GAME-2
Matloobi, R., Blumenstein, M., & Green, S. (2007) · Griffith Research Online · Cited by 2 · see citing works
The use of reverse engineering to analyse student computer programs
Vanneste, P., Bertels, K., & Decker, B. D. (1996) · Instructional Science · doi:10.1007/bf00119977 · Cited by 2 · see citing works
An Expert System For Education: IPTS
Yongqing, C., Qing, H., & Jingyu, Y. (2005) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1988.712842 · Cited by 2 · see citing works
Partial Specifications of Program Properties
Beierle, C., Kulaš, M., & Widera, M. (2004) · Cited by 2 · see citing works
AutoTeach: incremental hints for programming exercises
Antonucci, P. (2014) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-a-010255485 · Cited by 2 · see citing works
From PROUST to CHIRON: ITS Design as Iterative Engineering; Intermediate Results are Important!
Sack, W., & Soloway, E. M. (2021) · Routledge eBooks · doi:10.4324/9781315044361-9 · Cited by 2 · see citing works
Koh, H., & Wu, D. (1988) · International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems (IEA/AIE) · doi:10.1145/51909.51930 · Cited by 1 · see citing works
A String Of Perls: Using Perl To Teach Perl
Smith, M. A., & Ekstrom, J. J. (2020) · doi:10.18260/1-2--13091 · Cited by 1 · see citing works
An instructional interpreter for basic
Barr, A., & Beard, M. (1976) · ACM SIGCSE Bulletin · doi:10.1145/952989.803494 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · see citing works
Artificial Intelligence for Assessment and Feedback to Enhance Student Success in Higher Education
Hooda, M., Rana, C., Dahiya, O., Rizwan, A., & Hossain, M. S. (2022) · Mathematical Problems in Engineering · doi:10.1155/2022/5215722 · Cited by 444 · see citing works
Automatic feedback in online learning environments: A systematic literature review
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100027 · Cited by 440 · see citing works
Deeva, G., Bogdanova, D., Serral, E., Snoeck, M., & Weerdt, J. D. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.104094 · Cited by 292 · see citing works
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256 · see citing works
“attempts to provide more contextualized help to students learning to program, although effective programming support remains a challenge [43].”
BackgroundInfluentialAutomated Grading and Feedback Tools for Programming Education: A Systematic Review
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2024) · ACM Transactions on Computing Education · doi:10.1145/3636515 · Cited by 262 · see citing works
Maier, U., & Klotz, C. (2022) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2022.100080 · Cited by 218 · see citing works
CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192 · see citing works
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145 · see citing works
Kazemitabaar, M., Hou, X., Henley, A. Z., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631806 · Cited by 132 · see citing works
Exploring the Potential of Large Language Models to Generate Formative Programming Feedback
Kiesler, N., Lohr, D., & Keuning, H. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343457 · Cited by 82 · see citing works
Self-citationAI or Human? Evaluating Student Feedback Perceptions in Higher Education
Nazaretsky, T., Mejia-Domenzain, P., Swamy, V., Frej, J., & Käser, T. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72315-5_20 · Cited by 76 · see citing works
Hao, Q., Smith, D. H., Ding, L., Ko, A. J., Ottaway, C., Wilson, J., Arakawa, K., Turcan, A., Poehlman, T., & Greer, T. (2021) · Computer Science Education · doi:10.1080/08993408.2020.1860408 · Cited by 81 · see citing works
See, B. H., Gorard, S. A. C., Lu, B., Dong, L., & Siddiqui, N. (2021) · Research Papers in Education · doi:10.1080/02671522.2021.1907778 · Cited by 99 · see citing works
Building a Comprehensive Automated Programming Assessment System
Mekterović, I., Brkić, L., Milašinović, B., & Baranović, M. (2020) · IEEE Access · doi:10.1109/access.2020.2990980 · Cited by 74 · see citing works
Formative assessment strategies for students' conceptions—The potential of learning analytics
Stanja, J., Gritz, W., Krugel, J., Hoppe, A., & Dannemann, S. (2022) · British Journal of Educational Technology · doi:10.1111/bjet.13288 · Cited by 70 · see citing works
Cheng, G., Zou, D., Xie, H., & Wang, F. L. (2023) · Computers & Education · doi:10.1016/j.compedu.2023.104948 · Cited by 74 · see citing works
Combéfis, S. (2022) · Software · doi:10.3390/software1010002 · Cited by 65 · see citing works
Feedback-Generation for Programming Exercises With GPT-4
Azaiz, I., Kiesler, N., & Strickroth, S. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653594 · Cited by 61 · see citing works
A Tutoring System to Learn Code Refactoring
Keuning, H., Heeren, B. J., & Jeuring, J. T. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3432526 · Cited by 60 · see citing works
Self-citationNext-Step Hint Generation for Introductory Programming Using Large Language Models
Roest, L., Keuning, H., & Jeuring, J. T. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636259 · Cited by 57 · see citing works
Self-citationEvaluation of LLM Tools for Feedback Generation in a Course on Concurrent Programming
Ayres, I. E., Callejo, P., Hombrados-Herrera, M. Á., Alario‐Hoyos, C., & Kloos, C. D. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00406-0 · Cited by 67 · see citing works
CodeTailor: LLM-Powered Personalized Parsons Puzzles for Engaging Support While Learning Programming
Hou, X., Wu, Z., Wang, X., & Ericson, B. J. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662032 · Cited by 51 · see citing works
Koutcheme, C., Dainese, N., Sarsa, S., Hellas, A., Leinonen, J., & Denny, P. C. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653612 · Cited by 47 · see citing works
Skalka, J., & Drlík, M. (2020) · Applied Sciences · doi:10.3390/app10134566 · Cited by 46 · see citing works
Zheng, L.-Q., Long, M., Chen, B.-D., & Fan, Y. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00415-4 · Cited by 46 · see citing works
Large language models meet user interfaces: The case of provisioning feedback
Pozdniakov, S., Brazil, J., Abdi, S., Bakharia, A., Sadiq, S., Gašević, D., Denny, P., & Khosravi, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100289 · Cited by 43 · see citing works
Weidlich, J., Fink, A., Frey, A., Jivet, I., Gombert, S., Menzel, L., Giorgashvili, T., Yau, J. Y.-K., & Drachsler, H. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-025-00539-9 · Cited by 62 · see citing works
An experience of automated assessment in a large‐scale introduction programming course
Zampirolli, F. D. A., Josko, J. M. B., Venero, M. L. F., Kobayashi, G., Fraga, F. J., Goya, D., & Savegnago, H. R. (2021) · Computer Applications in Engineering Education · doi:10.1002/cae.22385 · Cited by 42 · see citing works
A Survey of Automated Programming Hint Generation: The HINTS Framework
McBroom, J., Koprinska, I., & Yacef, K. (2022) · ACM Computing Surveys · doi:10.1145/3469885 · Cited by 39 · see citing works
Towards automated content analysis of educational feedback: A multi-language study
Osakwe, I., Chen, G., Whitelock‐Wainwright, A., Gašević, D., Cavalcanti, A. P., & Mello, R. F. (2022) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2022.100059 · Cited by 40 · see citing works
Towards Giving Timely Formative Feedback and Hints to Novice Programmers
Jeuring, J., Keuning, H., Marwan, S., Bouvier, D., Izu, C., Kiesler, N., Lehtinen, T., Lohr, D., Peterson, A., & Sarsa, S. (2022) · Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3571785.3574124 · Cited by 40 · see citing works
Self-citationUsing GPT-4 to Provide Tiered, Formative Code Feedback
Nguyen, H., & Allan, V. H. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630960 · Cited by 39 · see citing works
Combining LLM-Generated and Test-Based Feedback in a MOOC for Programming
Gabbay, H., & Cohen, A. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662040 · Cited by 36 · see citing works
Evaluating the Quality of LLM-Generated Explanations for Logical Errors in CS1 Student Programs
Balse, R., Kumar, V., Prasad, P., & Warriem, J. M. (2023) · Bangalore Annual Compute Conference · doi:10.1145/3627217.3627233 · Cited by 34 · see citing works
Lohr, D., Keuning, H., & Kiesler, N. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13107 · Cited by 33 · see citing works
Self-citationFilighera, A., Parihar, S., Steuer, T., Meuser, T., & Ochs, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.587 · Cited by 33 · see citing works
Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices
Xiao, R., Hou, X., & Stamper, J. C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650937 · Cited by 32 · see citing works
A Comparison of Immediate and Scheduled Feedback in Introductory Programming Projects
Leinonen, J., Denny, P., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499372 · Cited by 32 · see citing works
Evaluating the Application of Large Language Models to Generate Feedback in Programming Education
Jacobs, S., & Jaschke, S. (2024) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon60312.2024.10578838 · Cited by 37 · see citing works
Rubric Is All You Need: Improving LLM-Based Code Evaluation With Question-Specific Rubrics
Pathak, A., Gandhi, R., Uttam, V. G., Ramamoorthy, A., Ghosh, P., Jindal, A. R., Verma, S., Mittal, A., Ased, A., Khatri, C., et al. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744220 · Cited by 38 · see citing works
Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics
Pathak, A., Gandhi, R., Uttam, V., Ramamoorthy, A., Ghosh, P., Jindal, A. R., Verma, S., Mittal, A., Ased, A., Khatri, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.23989 · Cited by 30
Student Refactoring Behaviour in a Programming Tutor
Keuning, H., Heeren, B. J., & Jeuring, J. T. (2020) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3428029.3428043 · Cited by 29 · see citing works
Self-citationDigitalization of Multistep Chemistry Exercises with Automated Formative Feedback
Eitemüller, C., Trauten, F., Striewe, M., & Walpuski, M. (2023) · Journal of Science Education and Technology · doi:10.1007/s10956-023-10043-2 · Cited by 34 · see citing works
Code Quality Defects across Introductory Programming Topics
Effenberger, T., & Pelánek, R. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499415 · Cited by 28 · see citing works
Nazaretsky, T., Mejia‐Domenzain, P., Swamy, V., Frej, J., & Käser, T. (2026) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70153 · Cited by 26 · see citing works
Vesin, B., Mangaroska, K., Akhüseyinoğlu, K., & Giannakos, M. N. (2022) · ACM Transactions on Computing Education · doi:10.1145/3511886 · Cited by 26 · see citing works
Promoting Early Engagement with Programming Assignments Using Scheduled Automated Feedback
Denny, P., Whalley, J., & Leinonen, J. (2021) · Proceedings of the 23rd Australasian Computing Education Conference · doi:10.1145/3441636.3442309 · Cited by 24 · see citing works
Customizable and scalable automated assessment of C/C++ programming assignments
Delgado‐Pérez, P., & Medina‐Bulo, I. (2020) · Computer Applications in Engineering Education · doi:10.1002/cae.22317 · Cited by 25 · see citing works
Learning From Errors: Exploring the Effectiveness of Enhanced Error Messages in Learning to Program
Zhou, Z., Wang, S.-J., & Qian, Y. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.768962 · Cited by 24 · see citing works
Evaluating Language Models for Generating and Judging Programming Feedback
Koutcheme, C., Dainese, N., Sarsa, S., Hellas, A., Leinonen, J., Ashraf, S., & Denny, P. C. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701791 · Cited by 30 · see citing works
Intelligent Deep-Learning Tutoring System to Assist Instructors in Programming Courses
Álvarez, D. R., & Mesa, F. J. (2023) · IEEE Transactions on Education · doi:10.1109/te.2023.3331055 · Cited by 22 · see citing works
Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming
Alkafaween, U., Al-Bluwi, I., & Denny, P. C. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13100 · Cited by 23 · see citing works
What Are Cybersecurity Education Papers About?
Švábenský, V., Vykopal, J., & Čeleda, P. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366816 · Cited by 21 · see citing works
Lee, H.-Y., Chen, P.-H., Lin, C.-J., Huang, Y.-M., & Wu, T.-T. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13733-z · Cited by 21 · see citing works
Code-quality evaluation scheme for assessment of student contributions to programming projects
Chen, H., Nguyen, B.-A., & Dow, C. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111273 · Cited by 21 · see citing works
Non-Expert Programmers in the Generative AI Future
Feldman, M. Q., & Anderson, C. J. (2024) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3663384.3663393 · Cited by 21 · see citing works
Building a Corpus of Task-Based Grading and Feedback Systems for Learning and Teaching Programming
Strickroth, S., & Striewe, M. (2022) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v12i5.31283 · Cited by 20 · see citing works
Wang, F.-H. (2022) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2022.3226345 · Cited by 21 · see citing works
Machine Learning-Based Automated Grading and Feedback Tools for Programming: A Meta-Analysis
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588822 · Cited by 20 · see citing works
GRADESTYLE: GitHub-Integrated and Automated Assessment of Java Code Style
Iddon, C., Giacaman, N., & Terragni, V. (2023) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet58685.2023.00024 · Cited by 20 · see citing works
The Responsible Development of Automated Student Feedback with Generative AI
Lindsay, E. D., Zhang, M., Johri, A., & Bjerva, J. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016572 · Cited by 33 · see citing works
Experiences from Integrating Large Language Model Chatbots into the Classroom
Hellas, A., Leinonen, J., & Leppänen, L. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690101 · Cited by 19 · see citing works
Let’s Ask Students About Their Programs, Automatically
Lehtinen, T., Santos, A. L., & Sorva, J. (2021) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00054 · Cited by 19 · see citing works
Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring with Feedback
Fateen, M., Wang, B., & Mine, T. (2024) · IEEE Access · doi:10.1109/access.2024.3508747 · Cited by 22 · see citing works
Annete: An Intelligent Tutoring Companion Embedded into the Eclipse IDE
Day, M., Penumala, M. R., & Gonzalez-Sanchez, J. (2019) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi48466.2019.00018 · Cited by 18 · see citing works
Birillo, A., Artser, E., Potriasaeva, A., Vlasov, I., Dzialets, K., Golubev, Y., Герасимов, И. В., Keuning, H., & Bryksin, T. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699556 · Cited by 23 · see citing works
Self-citationSoftware Analytics to Support Students in Object-Oriented Programming Tasks: An Empirical Study
Ardimento, P., Bernardi, M. L., & Cimitile, M. (2020) · IEEE Access · doi:10.1109/access.2020.3010172 · Cited by 17 · see citing works
tAIfa: Enhancing Team Effectiveness and Cohesion with AI-Generated Automated Feedback
Almutairi, M., Chiang, C., Bai, Y., & Gómez-Zará, D. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729197 · Cited by 15 · see citing works
Prokopyev, M. S., Vlasova, E. Z., Tretyakova, T., Sorochinsky, M. A., & Soloveva, R. A. (2020) · Propósitos y Representaciones · doi:10.20511/pyr2020.v8n3.484 · Cited by 15 · see citing works
Studying the Impact of Auto-Graders Giving Immediate Feedback in Programming Assignments
Mitra, J. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569726 · Cited by 14 · see citing works
Furia, C. A., Torkar, R., & Feldt, R. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3611667 · Cited by 14 · see citing works
Learning with Style: Improving Student Code-Style Through Better Automated Feedback
Saliba, L., Shioji, E., de Oliveira, E. A., Cohney, S., & Qi, J. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630889 · Cited by 14 · see citing works
Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
Lehtinen, T., Koutcheme, C., & Hellas, A. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640058 · Cited by 14 · see citing works
Gong, X., Xu, W., & Qiao, A. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-025-00552-y · Cited by 17 · see citing works
Automated Assessment: Experiences From the Trenches
Luxton-Reilly, A., Tempero, E. D., Arachchilage, N. A. G., Angela, C., Denny, P. C., Fowler, A., Giacaman, N., Kontorovich, I., Lottridge, D. M., Manoharan, S., et al. (2023) · Australasian Computing Education Conference · doi:10.1145/3576123.3576124 · Cited by 13 · see citing works
Does Peer Code Review Change My Mind on My Submission?
Strickroth, S. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588802 · Cited by 15 · see citing works
How problem difficulty and order influence programming education outcomes in online judge systems
Wang, J., Lin, P., Tang, Z., & Chen, S. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e20947 · Cited by 13 · see citing works
Jacobs, S., & Jaschke, S. (2024) · 2024 36th International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet62301.2024.10663001 · Cited by 13 · see citing works
Scaling Automated Programming Assessment Systems
Mekterović, I., Brkić, L., & Horvat, M. (2023) · Electronics · doi:10.3390/electronics12040942 · Cited by 13 · see citing works
Automated Assessment of Learning Objectives in Programming Assignments
Rump, A., Fehnker, A., & Mader, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80421-3_33 · Cited by 13 · see citing works
Exploring Feedback and Gamification in a Data Modeling Learning Tool
Dæhli, O., Kristoffersen, B., Lauvås, P., & Sandnes, T. (2021) · The Electronic Journal of e-Learning · doi:10.34190/ejel.19.6.2501 · Cited by 13 · see citing works
On the use of feedback in learning computer programming by novices: a systematic literature mapping
Rocha, H. J. B., Tedesco, P. C. D. A. R., & Costa, E. D. B. (2022) · Informatics in Education · doi:10.15388/infedu.2023.09 · Cited by 13 · see citing works
Advancing data justice in education: some suggestions towards a deontological framework
Perrotta, C. (2022) · Learning Media and Technology · doi:10.1080/17439884.2022.2156536 · Cited by 15 · see citing works
Automated feedback for participants of hands-on cybersecurity training
Švábenský, V., Vykopal, J., Čeleda, P., & Dovjak, J. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-12265-8 · Cited by 12 · see citing works
An Adaptive Approach to Provide Feedback for Students in Programming Problem Solving
Silva, P., Costa, E. D. B., & de Araújo, J. M. F. R. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-22244-4_3 · Cited by 12 · see citing works
AI assistance in peer feedback provision: Pedagogically sound, but minimally adopted
Pozdniakov, S., Brazil, J., Banihashem, S. K., Noroozi, O., Gašević, D., Sadiq, S., & Khosravi, H. (2026) · Computers & Education · doi:10.1016/j.compedu.2026.105591 · Cited by 11 · see citing works
Evaluating CodeClusters for Effectively Providing Feedback on Code Submissions
Koivisto, T., & Hellas, A. (2022) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie56618.2022.9962751 · Cited by 11 · see citing works
Zhang, Y., Paquette, L., & Hu, X. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105029 · Cited by 13 · see citing works
An Analysis of Programming Course Evaluations Before and After the Introduction of an Autograder
Hagerer, G., Lahesoo, L., Anschütz, M., Krusche, S., & Groh, G. (2021) · International Conference on Information Technology Based Higher Education and Training (ITHET) · doi:10.1109/ithet50392.2021.9759809 · Cited by 11 · see citing works
Using Programming Autograder Formative Data to Understand Student Growth
Acuña, R., & Bansal, A. (2022) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie56618.2022.9962650 · Cited by 11 · see citing works
Kiesler, N. (2023) · Future of Educational Innovation-Workshop Series Data in Action · doi:10.1109/ieeeconf56852.2023.10104622 · Cited by 11 · see citing works
Skalka, J., & Drlík, M. (2023) · IEEE Access · doi:10.1109/access.2023.3317694 · Cited by 10 · see citing works
The Trees in the Forest: Characterizing Computing Students' Individual Help-Seeking Approaches
Ko, S.-H., & Stephens-Martinez, K. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671099 · Cited by 10 · see citing works
PerFuSIT: Personalized Fuzzy Logic Strategies for Intelligent Tutoring of Programming
Chrysafiadi, K., & Virvou, M. K. (2024) · Electronics · doi:10.3390/electronics13234827 · Cited by 10 · see citing works
Partnering with AI: A Pedagogical Feedback System for LLM Integration Into Programming Education
Scholz, N., Nguyen, M. H., Singla, A. K., & Nagashima, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-03873-9_31 · Cited by 10 · see citing works
Liénardy, S., Leduc, L., Verpoorten, D., & Donnet, B. (2020) · Australasian Computing Education Conference · doi:10.1145/3373165.3373176 · Cited by 10 · see citing works
Model‐based automatic grading of object‐oriented programming assignments
Le, D. M. (2021) · Computer Applications in Engineering Education · doi:10.1002/cae.22464 · Cited by 9 · see citing works
Wang, F.-H. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100481 · Cited by 9 · see citing works
Learning UML database design and modeling with AutoER
Foss, S., Urazova, T., & Lawrence, R. (2022) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) · doi:10.1145/3550356.3559091 · Cited by 9 · see citing works
- Authors
- Keuning, H., Jeuring, J., & Heeren, B.
- Affiliations
- Windesheim University of Applied Sciences (NL)
- Utrecht University (NL)
- Open University of the Netherlands (NL)
- Venue
- ACM Transactions on Computing Education
- Published
- 2018-09-28
- Volume
- 19
- Issue
- 1
- Pages
- 1-43
- ISSN
- 1946-6226 (ISSN-L)
- OpenAlex
- W2894540915
- Cited by
- 370
- References
- 200
- Citations by year
- 2026: 72 · 2025: 71 · 2024: 60 · 2023: 58 · 2022: 46 · 2021: 29 · 2020: 22 · 2019: 12
- FWCI
- 2.95
- Type
- journal article
- Language
- en
- Open access
- green
- dspace.library.uu.nl
- Locations
- ACM Transactions on Computing Education (publishedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- DSpace (Open University in the Netherlands) (submittedVersion, other-oa)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Funders
- Netherlands Organisation for Scientific Research (023.005.063)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Student Assessment and Feedback
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- automated feedback, programming exercises, formative feedback, programming education, adaptive feedback, tool evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
