The DOI matches its Crossref record; 200 of the 200 references it cites were found in a registry or an index.
A Systematic Literature Review of Automated Feedback Generation for Programming Exercises
Open accessAbstract
Formative feedback, aimed at helping students to improve their work, is an important factor in learning. Many tools that offer programming exercises provide automated feedback on student solutions. We have performed a systematic literature review to find out what kind of feedback is provided, which techniques are used to generate the feedback, how adaptable the feedback is, and how these tools are evaluated. We have designed a labelling to classify the tools, and use Narciss’ feedback content categories to classify feedback messages. We report on the results of coding a total of 101 tools. We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs. However, the diversity of feedback types has increased over the past decades and new techniques are being applied to generate feedback that is increasingly helpful for students.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Findings showed that mostly automated feedback for programming exercises has been used to identify mistakes rather than fixing them.”
From Artificial Intelligence for Assessment and Feedback to Enhance Student Success in Higher Education (2022) · cites this paper as [76]
Citations
Cites
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · cited 2×
“The authors see a role for automated feedback at all three levels as defined by Hattie and Timperley [70]: “task level,” “process level,” and “self-regulation level,” discarding feedback at the “self level” because of its limited effect on learning.”
BackgroundShute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631
Knowledge tracing: Modeling the acquisition of procedural knowledge
Corbett, A. T., & Anderson, J. R. (1995) · User Modelling and User-Adapted Interaction · doi:10.1007/bf01099821 · Cited by 2,324
Claessen, K., & Hughes, J. (2011) · ACM SIGPLAN Notices · doi:10.1145/1988042.1988046 · Cited by 1,018
Lowering the barriers to programming
Kelleher, C. L., & Pausch, R. F. (2005) · ACM Computing Surveys · doi:10.1145/1089733.1089734 · Cited by 887
The Behavior of Tutoring Systems
VanLehn, K. A. (2006) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2006-16(3)02 · Cited by 836
Hovemeyer, D., & Pugh, W. (2004) · Companion to the ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA Companion) · doi:10.1145/1028664.1028717 · Cited by 770
Sahami, M., Roach, S., Cuadros-Vargas, E., Danyluk, A., Dodge, R., Fisher, K. S., France, R., Grossman, D., Hawthorne, B., Katz, R., et al. (2013) · ACM eBooks · doi:10.1145/2534860 · Cited by 733
Failure rates in introductory programming
Bennedsen, J., & Caspersen, M. E. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1272848.1272879 · Cited by 677
McCracken, M., Almstrum, V. L., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/572133.572137 · Cited by 628
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 561
A Survey of Automated Assessment Approaches for Programming Assignments
Ala-Mutka, K. (2005) · Computer Science Education · doi:10.1080/08993400500150747 · Cited by 550
ELM-ART: An Adaptive Versatile System for Web-Based Instruction
Weber, G., & Brusilovsky, P. L. (2001) · Cited by 546
Students’ Misconceptions and Other Difficulties in Introductory Programming: A Literature Review
Qian, Y., & Lehman, J. (2018) · ACM Transactions on Computing Education · doi:10.1145/3077618 · Cited by 529
A survey of literature on the teaching of introductory programming
Pears, A., Seidman, S. B., Malmi, L., Mannila, L., Adams, E. S., Bennedsen, J., Devlin, M., & Paterson, J. H. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1345375.1345441 · Cited by 481
Feedback in Higher and Professional Education
Boud, D. J., & Molloy, E. (2012) · Routledge eBooks · doi:10.4324/9780203074336 · Cited by 454
Automatic test-based assessment of programming
Douce, C., Livingstone, D., & Orwell, J. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163409 · Cited by 416
A Review of Generic Program Visualization Systems for Introductory Programming Education
Sorva, J., Karavirta, V., & Malmi, L. (2013) · ACM Transactions on Computing Education · doi:10.1145/2490822 · Cited by 400
Hovemeyer, D., & Pugh, W. (2004) · ACM SIGPLAN Notices · doi:10.1145/1052883.1052895 · Cited by 397
Automated feedback generation for introductory programming assignments
Singh, R., Gulwani, S., & Solar-Lezama, A. (2013) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2491956.2462195 · Cited by 390
Feedback Strategies for Interactive Learning Tasks
Narciss, S. (2007) · doi:10.4324/9780203880869.ch11 · Cited by 389
Effective Tutoring Techniques: A Comparison of Human Tutors and Intelligent Tutoring Systems
Merrill, D. C., Reiser, B. J., Ranney, M., & Trafton, J. G. (1992) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0203_2 · Cited by 298
User Modeling and Adaptive Navigation Support in WWW-Based Tutoring Systems
Weber, G., & Specht, M. M. (1997) · Springer eBooks · doi:10.1007/978-3-7091-2670-7_30 · Cited by 288
PROUST: Knowledge-Based Program Understanding
Johnson, W. L., & Soloway, E. M. (1985) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1985.232210 · Cited by 285
Corbett, A. T., & Anderson, J. R. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365111 · Cited by 280
A New Paradigm for Intelligent Tutoring Systems: Example-Tracing Tutors
Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2009) · doi:10.5555/1734243.1734245 · Cited by 279
Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor
Rivers, K., & Koedinger, K. R. (2015) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0070-z · Cited by 258
Grading student programs using ASSYST
Jackson, D., & Usher, M. (1997) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/268084.268210 · Cited by 254
On automated grading of programming assignments in an academic institution
Cheang, B., Kurnia, A., Lim, A. E. B., & Oon, W.-C. (2003) · Computers & Education · doi:10.1016/s0360-1315(03)00030-7 · Cited by 227
The automated tutoring of introductory computer programming
Anderson, J. R., & Skwarecki, E. J. (1986) · Communications of the ACM · doi:10.1145/6592.6593 · Cited by 225
Strategies for Computer-Based Programming Instruction: Program Completion vs. Program Generation
Van Merriënboer, J. J. G., & De Croock, M. B. M. (1992) · Journal of Educational Computing Research · doi:10.2190/mjdx-9pp4-kfmt-09pm · Cited by 207
Improving student performance by evaluating how well students test their own programs
Edwards, S. H. (2003) · Journal on Educational Resources in Computing · doi:10.1145/1029994.1029995 · Cited by 179
The computer as a tutorial laboratory: the Stanford BIP project
Barr, A. B., Beard, M., & Atkinson, R. C. (1976) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(76)80021-1 · Cited by 177
A Comparative Analysis of Cognitive Tutoring and Constraint-Based Modeling
Mitrović, A., Koedinger, K. R., & Martin, B. (2003) · Lecture notes in computer science · doi:10.1007/3-540-44963-9_42 · Cited by 166
Spacco, J. W., Hovemeyer, D., Pugh, W., Emad, F., Hollingsworth, J. K., & Padua-Perez, N. (2006) · ACM SIGCSE Bulletin · doi:10.1145/1140123.1140131 · Cited by 163
Automated assessment and experiences of teaching programming
Higgins, C., Gray, G. R., Symeonidis, P., & Tsintsifas, A. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163410 · Cited by 163
Instructional design of a programming course
Caspersen, M. E., & Bennedsen, J. (2007) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/1288580.1288595 · Cited by 160
Fully automatic assessment of programming exercises
Saikkonen, R., Malmi, L., & Korhonen, A. (2001) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/377435.377666 · Cited by 154
Intention-Based Diagnosis of Novice Programming Errors
Johnson, W. L., & Gladwin, L. A. (1987) · IEEE Expert · doi:10.1109/mex.1987.4307101 · Cited by 150
Towards a Systematic Review of Automated Feedback Generation for Programming Exercises
Keuning, H., Jeuring, J. T., & Heeren, B. (2016) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2899415.2899422 · Cited by 142
Self-citationAbility-training-oriented automated assessment in introductory programming course
Wang, T., Su, X., Ma, P., Wang, Y., & Wang, K. (2010) · Computers & Education · doi:10.1016/j.compedu.2010.08.003 · Cited by 138
Automated feedback generation for introductory programming assignments
Singh, R. K., Gulwani, S., & Solar-Lezama, A. (2013) · ACM SIGPLAN Notices · doi:10.1145/2499370.2462195 · Cited by 137
Scaffolding students' learning using test my code
Vihavainen, A., Vikberg, T., Luukkainen, M., & Pärtel, M. (2013) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2462476.2462501 · Cited by 135
Translating Principles of Effective Feedback for Students into the CS1 Context
Ott, C., Robins, A. V., & Shephard, K. L. (2016) · ACM Transactions on Computing Education · doi:10.1145/2737596 · Cited by 113
Nguyen, A., Piech, C., Huang, J., & Guibas, L. J. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568023 · Cited by 108
Understanding and debugging novice programs
Johnson, W. L. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90094-g · Cited by 108
Grading student programs using ASSYST
Jackson, D., & Usher, M. (1997) · ACM SIGCSE Bulletin · doi:10.1145/268085.268210 · Cited by 107
Programming environments for novices
du Boulay, B. (1992) · Lecture notes in computer science · doi:10.1007/3-540-55606-0_4 · Cited by 106
LAURA, a system to debug student programs
Adam, A., & Laurent, J.-P. (1980) · Artificial Intelligence · doi:10.1016/0004-3702(80)90023-5 · Cited by 105
Automatic program debugging for intelligent tutoring systems
Murray, W. R. (1987) · Computational Intelligence · doi:10.1111/j.1467-8640.1987.tb00169.x · Cited by 104
Teaching the tacit knowledge of programming to noviceswith natural language tutoring
Lane, H. C., & VanLehn, K. A. (2005) · Computer Science Education · doi:10.1080/08993400500224286 · Cited by 101
The Teaching and Learning of Programming: A Survey of Supporting Software Tools
Gómez-Albarrán, M. (2005) · The Computer Journal · doi:10.1093/comjnl/bxh080 · Cited by 94
Fully automatic assessment of programming exercises
Saikkonen, R., Malmi, L., & Korhonen, A. (2001) · ACM SIGCSE Bulletin · doi:10.1145/507758.377666 · Cited by 90
Software verification and graph similarity for automated evaluation of students’ assignments
Janičić, M. V., Nikolić, M., Tošić, D., & Kunčak, V. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.12.005 · Cited by 89
Teaching and learning programming and software engineering via interactive gaming
Tillmann, N., de Halleux, J., Xie, T., Gulwani, S., & Bishop, J. (2013) · doi:10.5555/2486788.2486941 · Cited by 89
Program Representation for Automatic Hint Generation for a Data-Driven Novice Programming Tutor
Jin, W., Barnes, T. M., Stamper, J. C., Eagle, M. J., Johnson, M. W., & Lehmann, L. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30950-2_40 · Cited by 86
A Virtual Programming Lab for Moodle with automatic assessment and anti-plagiarism features
Rodríguez-del-Pino, J. C., Royo, E. R., & Figueroa, Z. J. H. (2012) · Acceda (Universidad de Las Palmas de Gran Canaria) · Cited by 85
Evaluating assessments of novice programming environments
Gross, P., & Powers, K. D. (2005) · doi:10.1145/1089786.1089796 · Cited by 80
PROGRAMMING ASSIGNMENTS AUTOMATIC GRADING: REVIEW OF TOOLS AND IMPLEMENTATIONS
Caiza, J. C., & Álamo, J. M. D. (2013) · Hispana · Cited by 78
Increasing Adoption of Smart Learning Content for Computer Science Education
Brusilovsky, P. L., Edwards, S. H., Kumar, A. N., Malmi, L., Benotti, L., Buck, D., Ihantola, P., Prince, R., Sirkiä, T., Александрович, С. С., et al. (2014) · doi:10.1145/2713609.2713611 · Cited by 75
The marking system for CourseMaster
Higgins, C., Symeonidis, P., & Tsintsifas, A. (2002) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/544414.544431 · Cited by 73
Assessment process for programming assignments
Ala-Mutka, K., & Järvinen, H.-M. (2004) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2004.1357399 · Cited by 71
Feedback generation for performance problems in introductory programming assignments
Gulwani, S., Radiček, I., & Zuleger, F. (2014) · ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) · doi:10.1145/2635868.2635912 · Cited by 70
Teaching and learning computer programming
Ulloa, M. A. (1980) · ACM SIGCSE Bulletin · doi:10.1145/989253.989263 · Cited by 69
Intelligent Tutor, Environment and Manual for Introductory Programming
Brusilovsky, P. L. (1992) · Programmed Learning and Educational Technology · doi:10.1080/0954730920290104 · Cited by 68
Transformation-based diagnosis of student programs for programming tutoring systems
Xu, S., & Chee, Y. S. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1191799 · Cited by 68
WebToTeach: an interactive focused programming exercise system
Arnow, D. M., & Barshay, O. (2003) · doi:10.1109/fie.1999.839303 · Cited by 66
Learning to program through the web
Truong, N., Bancroft, P., & Roe, P. (2005) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1067445.1067452 · Cited by 64
PASS: An automated system for program assessment
Thorburn, G., & Rowe, G. (1997) · Computers & Education · doi:10.1016/s0360-1315(97)00021-3 · Cited by 64
Interface-based programming assignments and automatic grading of java programs
Helmick, M. T. (2007) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1268784.1268805 · Cited by 62
Automatic programming assessment and test data generation a review on its approaches
Romli, R., Sulaiman, S., & Zamli, K. Z. (2010) · International Symposium on Information Technology · doi:10.1109/itsim.2010.5561488 · Cited by 61
Proceedings 1989 Frontiers in Education Conference
(1989) · doi:10.1109/fie.1989 · Cited by 61
A semi-automated approach to online assessment
Jackson, D. (2000) · doi:10.1145/343048.343160 · Cited by 60
Weber, G. (1996) · Cognitive Science · doi:10.1016/s0364-0213(99)80006-8 · Cited by 57
Developing intelligent programming tutors for novice programmers
Pillay, N. (2003) · ACM SIGCSE Bulletin · doi:10.1145/782941.782986 · Cited by 55
J-Latte: a Constraint-Based Tutor for Java
Holland, J., Mitrović, A., & Martin, B. (2009) · UC Research Repository (University of Canterbury) · Cited by 55
A Review of AI-Supported Tutoring Approaches for Learning Programming
Le, N.-T., Strickroth, S., Groß, S., & Niels Pinkwart, C. (2013) · Studies in computational intelligence · doi:10.1007/978-3-319-00293-4_20 · Cited by 52
A Survey and Critical Analysis of Tools for Learning Programming
Deek, F. P., & McHugh, J. A. M. (1998) · Computer Science Education · doi:10.1076/csed.8.2.130.3820 · Cited by 50
A Classification of Adaptive Feedback in Educational Systems for Programming
Le, N.-T. (2016) · Systems · doi:10.3390/systems4020022 · Cited by 48
A software system for grading student computer programs
Jackson, D. (1996) · Computers & Education · doi:10.1016/s0360-1315(96)00025-5 · Cited by 47
A programming learning system for beginners-a completion strategy approach
Chang, K., Chiao, B.-C., Chen, S.-W., & Hsiao, R.-S. (2000) · IEEE Transactions on Education · doi:10.1109/13.848075 · Cited by 46
The marking system for CourseMaster
Higgins, C., Symeonidis, P., & Tsintsifas, A. (2002) · ACM SIGCSE Bulletin · doi:10.1145/637610.544431 · Cited by 44
Automated feedback on programs means students need less help from teachers
Odekirk-Hash, E., & Zachary, J. L. (2001) · doi:10.1145/364447.364537 · Cited by 43
An interactive functional programming tutor
Gerdes, A., Jeuring, J. T., & Heeren, B. (2012) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2325296.2325356 · Cited by 43
An intelligent tutoring system for introductory C language course
Song, J., Hahn, S. H., Tak, K., & Kim, J. H. (1997) · Computers & Education · doi:10.1016/s0360-1315(97)00003-1 · Cited by 43
MENO-II: An AI-Based Programming Tutor.
Soloway, E. M., Rubin, E., Woolf, B. P., Bonar, J. G., & Johnson, W. L. (1983) · Cited by 42
Chee, Y. S. (1995) · Instructional Science · doi:10.1007/bf00890449 · Cited by 42
de Souza, D. M., Maldonado, J. C., & Barbosa, E. F. (2011) · doi:10.1109/cseet.2011.5876088 · Cited by 42
A Simulated Student Can Improve Collaborative Learning
Vizcaíno, A. (2005) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2005-15(1)02 · Cited by 40
Experiences using test-driven development with an automated grader
Edwards, S. H., & Pérez-Quiñones, M. A. (2007) · Journal of computing sciences in colleges · doi:10.5555/1181849.1181855 · Cited by 39
Guided programming and automated error analysis in an intelligent Prolog tutor
Hong, J. S. (2004) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2004.02.001 · Cited by 38
Ruth, G. R. (1976) · Artificial Intelligence · doi:10.1016/0004-3702(76)90022-9 · Cited by 37
The Behavior of Tutoring Systems
VanlehnKurt (2006) · International Journal of Artificial Intelligence in Education · doi:10.5555/1435351.1435353 · Cited by 37
Analysing Student Programs in the PHP Intelligent Tutoring System
Weragama, D., & Reye, J. (2014) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-014-0014-z · Cited by 36
GAME: a Generic Automated Marking Environment for programming assessment
Blumenstein, M., Green, S., Nguyen, A. T. A., & Muthukkumarasamy, V. (2004) · International Conference on Information Technology: Coding and Computing (ITCC) · doi:10.1109/itcc.2004.1286454 · Cited by 36
Weber, G. (1996) · Cognitive Science · doi:10.1207/s15516709cog2002_2 · Cited by 36
A Review of Static Analysis Approaches for Programming Exercises
Striewe, M., & Goedicke, M. (2014) · Communications in computer and information science · doi:10.1007/978-3-319-08657-6_10 · Cited by 35
Identifying Effective Moves in Tutoring: On the Refinement of Dialogue Act Annotation Schemes
Vail, A. K., & Boyer, K. E. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_24 · Cited by 35
Programming students NEED instant feedback
Venables, A. T., & Haywood, L. (2003) · Australasian Computing Education Conference · Cited by 34
The Ceilidh system for the automatic grading of students on programming courses
Benford, S., Burke, E. K., Foxley, E., & Higgins, C. (1995) · ACM Southeast Regional Conference (ACMSE) · doi:10.1145/1122018.1122050 · Cited by 32
Automatic debugging of Prolog programs in a Prolog Intelligent Tutoring System
Looi, C. (1991) · Instructional Science · doi:10.1007/bf00120883 · Cited by 32
Jurado, F., Redondo, M. Á., & Ortega, M. (2011) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2011.11.002 · Cited by 32
Data-Driven Program Synthesis for Hint Generation in Programming Tutors
Lazar, T., & Ivan Bratko, A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_38 · Cited by 32
Intelligent Learning Environments wthin Blended Learning for Ensuring Effective C Programming Course
Kose, U. (2012) · International Journal of Artificial Intelligence & Applications · doi:10.5121/ijaia.2012.3109 · Cited by 31
Using strategies for assessment of programming exercises
Gerdes, A., Jeuring, J. T., & Heeren, B. (2010) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1734263.1734412 · Cited by 30
A FLEXIBLE AND MODULAR SOFTWARE ARCHITECTURE FOR COMPUTER AIDED ASSESSMENTS AND AUTOMATED MARKING
Striewe, M., Balz, M., & Goedicke, M. (2009) · International Conference on Computer Supported Education · doi:10.5220/0001966900540061 · Cited by 29
Example-based feedback provision using structured solution spaces
Groß, S., Mokbel, B., Paaßen, B., Hammer, B., & Niels Pinkwart, C. (2014) · International Journal of Learning Technology · doi:10.1504/ijlt.2014.065752 · Cited by 29
Learning Feedback in Intelligent Tutoring Systems
Groß, S., Mokbel, B., Hammer, B., & Niels Pinkwart, C. (2015) · Künstliche Intell. · doi:10.1007/s13218-015-0367-y · Cited by 28
Performance analysis of GAME: A generic automated marking environment
Blumenstein, M., Green, S., Fogelman, S., Nguyen, A. T. A., & Muthukkumarasamy, V. (2007) · Computers & Education · doi:10.1016/j.compedu.2006.11.006 · Cited by 28
Moghadam, J., Choudhury, R. R., Yin, H., & Fox, A. (2015) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2724660.2728672 · Cited by 28
An Incremental Hint System For Automated Programming Assignments
Antonucci, P., Estler, C., Nikolić, Đ., Piccioni, M., & Meyer, B. (2015) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2729094.2742607 · Cited by 28
Towards an Integrative Learning Environment for Java Programming
Groß, S., & Pinkwart, N. (2015) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2015.75 · Cited by 27
Collaborative example selection in an intelligent example-based programming environment
Brusilovsky, P. L., & Weber, G. (1996) · International Conference of Learning Sciences · doi:10.22318/icls1996.357 · Cited by 26
Shaffer, S. C. (2005) · ACM SIGCSE Bulletin · doi:10.1145/1083431.1083464 · Cited by 26
Learning to construct quality software with the Ceilidh system
Benford, S., Burke, E. K., & Foxley, E. (1993) · Software Quality Journal · doi:10.1007/bf00402268 · Cited by 26
McCalla, G., Greer, J. E., Barrie, B., & Pospisil, P. (1992) · Computers & Mathematics with Applications · doi:10.1016/0898-1221(92)90148-b · Cited by 25
Interface-based programming assignments and automatic grading of java programs
Helmick, M. T. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1269900.1268805 · Cited by 25
Automatic grading of students' ALGOL programming
Naur, P. (1964) · BIT Numerical Mathematics · doi:10.1007/bf01956028 · Cited by 24
Teaching novice programmers using ProgTest
de Souza, D. M., Isotani, S., & Barbosa, E. F. (2015) · International Journal of Knowledge and Learning · doi:10.1504/ijkl.2015.071054 · Cited by 23
Bridge: Intelligent Tutoring with Intermediate Representations
Bonar, J. G., & Cunningham, R. K. (1988) · doi:10.21236/ada218938 · Cited by 22
The PHP Intelligent Tutoring System
Weragama, D., & Reye, J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_64 · Cited by 22
Jurado, F., Redondo, M. Á., & Ortega, M. (2012) · Computer Applications in Engineering Education · doi:10.1002/cae.21569 · Cited by 21
Suzuki, R., Soares, G., Glassman, E. L., Head, A., D’Antoni, L., & Hartmann, B. (2017) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3027063.3053187 · Cited by 21
Submit! a web-based system for automatic program critiquing
Pisan, Y., Richards, D., Sloane, A. M., Koncek, H., & Mitchell, S. L. (2003) · Australasian Computing Education Conference · Cited by 20
Binary instant feedback on programming exercises can reduce student engagement and promote cheating
Kyrilov, A., & Noelle, D. C. (2015) · doi:10.1145/2828959.2828968 · Cited by 20
A rationale and description of a CAI program to teach the BASIC programming language
Barr, A. B., Beard, M., & Atkinson, R. C. (1975) · Instructional Science · doi:10.1007/bf00157068 · Cited by 19
Gotel, O., Scharff, C., & Wildenberg, A. (2008) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1384271.1384329 · Cited by 19
Pettit, R., & Prather, J. E. (2017) · Journal of computing sciences in colleges · doi:10.5555/3055338.3079060 · Cited by 19
Using functional models in automatic debugging
Allemang, D. (1991) · IEEE Expert · doi:10.1109/64.108947 · Cited by 18
Pärtel, M., Luukkainen, M., Vihavainen, A., & Vikberg, T. (2013) · International Journal of Technology Enhanced Learning · doi:10.1504/ijtel.2013.059495 · Cited by 18
Student Modeling in an Intelligent Programming Tutor
Corbett, A. T., & Anderson, J. R. (1993) · NATO ASI series. Series F : Computer and system sciences · doi:10.1007/978-3-662-11334-9_13 · Cited by 17
Strategy-based feedback in a programming tutor
Keuning, H., Heeren, B., & Jeuring, J. T. (2014) · doi:10.1145/2691352.2691356 · Cited by 17
Self-citationUsing run time traces in automated programming tutoring
Striewe, M., & Goedicke, M. (2011) · doi:10.1145/1999747.1999832 · Cited by 17
Automatic analysis of programming assignments
Beierle, C., Kulaa, M., & Widera, M. (2003) · Cited by 16
Computer Aided Assessments and Programming Exercises with JACKPreprint
Goedicke, M., Striewe, M., & Balz, M. (2018) · RePEc: Research Papers in Economics · doi:10.17185/duepublico/47108 · Cited by 16
Ramadhan, H. A., Deek, F. P., & Shihab, K. (2001) · Artificial Intelligence Review · doi:10.1023/a:1011078011415 · Cited by 16
Evaluation of Guided-Planning and Assisted-Coding with Task Relevant Dynamic Hinting
Jin, W., Corbett, A. T., Lloyd, W., Baumstark, L., & Rolka, C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_40 · Cited by 15
An Automatic Assessment Marking and Plagiarism Detection System
Ghosh, M. R., Verma, B., & Nguyen, A. T. A. (2002) · Griffith Research Online · Cited by 15
A semi-automated approach to online assessment
Jackson, D. (2000) · ACM SIGCSE Bulletin · doi:10.1145/353519.343160 · Cited by 15
AnalyseC: A Framework for Assessing Students' Programs at Structural and Semantic Level
Wu, W., Li, G., Sun, Y., Wang, J., & Lai, T. (2007) · International Conference on Control and Automation · doi:10.1109/icca.2007.4376454 · Cited by 14
Intelligent tutoring system for learning PHP
Weragama, D. (2013) · Cited by 14
The design of the SCENT automated advisor
McCalla, G. I., Bunt, R. B., & Harms, J. J. (1986) · Computational Intelligence · doi:10.1111/j.1467-8640.1986.tb00073.x · Cited by 13
Learning to program through the web
Truong, N., Bancroft, P., & Roe, P. (2005) · ACM SIGCSE Bulletin · doi:10.1145/1151954.1067452 · Cited by 13
Automated feedback on programs means students need less help from teachers
Odekirk-Hash, E., & Zachary, J. L. (2001) · ACM SIGCSE Bulletin · doi:10.1145/366413.364537 · Cited by 13
Early experiences of computer-aided assessment and administration when teaching computer programming
Benford, S., Burke, E., Foxley, E., Gutteridge, N., & Zın, A. M. (2011) · Research in Learning Technology · doi:10.3402/rlt.v1i2.9481 · Cited by 12
A critical analysis and evaluation of Web-based environments for program development
Deek, F. P., Ho, K.-W., & Ramadhan, H. A. (2000) · The Internet and Higher Education · doi:10.1016/s1096-7516(01)00038-0 · Cited by 12
Improving the quality of programming education by online assessment
Fischer, G., & von Gudenberg, J. W. (2006) · doi:10.1145/1168054.1168085 · Cited by 12
Java Facilities for Automating Analysis, Feedback and Assessment of Laboratory Work
MacNish, C. K. (2000) · Computer Science Education · doi:10.1076/0899-3408(200008)10:2;1-c;ft147 · Cited by 12
An Affective and Cognitive Tutoring System for Learning Programming
Estrada, M. L. B., Cabada, R. Z., González-Hernández, F., Bustillos, R. O., & Reyes-Garcia, C. A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-27101-9_12 · Cited by 12
Qualitative Evaluation of the Java Intelligent Tutoring System
Sykes, E. R. (2005) · SOURCE Sheridan's Institutional Repository (Sheridan College) · Cited by 12
Design, Development and Evaluation of the Java Intelligent Tutoring System.
Sykes, E. R. (2010) · Cited by 12
Evaluating the Synergies of Integrating E-Assessment and Software Testing
Majchrzak, T. A., & Usener, C. A. (2013) · Springer eBooks · doi:10.1007/978-1-4614-4951-5_15 · Cited by 11
Exploiting program schemata in a Prolog tutoring system
Gegg-Harrison, T. S. (1993) · Cited by 11
The Grace Tutor: A qualified success
McKendree, J., Radlinski, B., & Atwood, M. E. (1992) · Lecture notes in computer science · doi:10.1007/3-540-55606-0_78 · Cited by 11
Model solutions and properties for diagnosing student programs in Ask-Elle
Jeuring, J. T., van Binsbergen, L. T., Gerdes, A., & Heeren, B. J. (2014) · Computer Science Education Research Conference · doi:10.1145/2691352.2691355 · Cited by 11
Karkalas, S., & Gutiérrez-Santos, S. (2014) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2014.35 · Cited by 11
A Generalized Knowledge approach to comprehend Pascal and C Programs
Ueno, H. (2000) · IEICE Transactions on Information and Systems · Cited by 10
Sant, J. A. (2009) · doi:10.1145/1562877.1562971 · Cited by 10
Towards the use of an automatic assessment system in the teaching of software testing
de Souza, D. M., Oliveira, B. H., Maldonado, J. C., Souza, S. R. S., & Barbosa, E. F. (2014) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2014.7044375 · Cited by 10
Mansouri, F. Z., Gibbon, C. A., & Higgins, C. (1998) · doi:10.1145/282991.283108 · Cited by 10
Problem-based learning in formal and informal learning environments
Šimić, G., & Jevremović, A. (2010) · Interactive Learning Environments · doi:10.1080/10494820.2010.486685 · Cited by 10
Global perceptions on the use of WeBWorK as an online tutor for computer science
Gotel, O., Scharff, C., Wildenberg, A., Bousso, M., Bunthoeurn, C., Phal, D., Kulkarni, V., Palakvangsa-Na-Ayudhya, S., Sarr, C., & Sunetnanta, T. T. (2008) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2008.4720331 · Cited by 10
Incorporating anchored learning in a C# intelligent tutoring system
Hartanto, B., & Reye, J. (2013) · International Conference on Computers in Education · Cited by 9
Keuning, H., Jeuring, J. T., & Heeren, B. (2016) · Utrecht University Repository (Utrecht University) · Cited by 9
Self-citationA System for Automatic Evaluation of Programs for Correctness and Performance
Mandal, A. K., Mandal, C., & Reade, C. (2007) · Lecture notes in business information processing · doi:10.1007/978-3-540-74063-6_29 · Cited by 8
Meaningful learning in the tutoring system for programming
Dadić, T., Stankov, S., & Rosić, M. (2008) · doi:10.1109/iti.2008.4588458 · Cited by 8
Intelligent Tutoring System for Learning Programming
Dadić, T. (2010) · IGI Global eBooks · doi:10.4018/978-1-61692-008-1.ch009 · Cited by 8
Gotel, O., Scharff, C., & Wildenberg, A. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1597849.1384329 · Cited by 8
Advanced Computational Methods for Knowledge Engineering
Le, N.-T., Do, T. V., Nguyên, N. T., & Thi, H. A. L. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-61911-8 · Cited by 8
Rosenthal, T., Suppes, P., & Ben‐Zvi, N. (2003) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2002.1157893 · Cited by 7
Experiences of Using Automated Assessment in Computer Science Courses
English, J., & English, T. (2015) · Journal of Information Technology Education Innovations in Practice · doi:10.28945/2304 · Cited by 7
MacNish, C. K. (2002) · UWA Profiles and Research Repository (University of Western Australia) · Cited by 6
An instructional interpreter for basic
Barr, A., & Beard, M. (1976) · doi:10.1145/800107.803494 · Cited by 6
Radlinski, R., & McKendree, J. (1992) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/142750.142829 · Cited by 6
A specification language for static analysis of student exercises
Köllmann, C., & Goedicke, M. (2008) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2008.47 · Cited by 6
Automation of Java Code Analysis for Programming Exercises
Köllmann, C., & Goedicke, M. (2024) · Technische Universität Berlin – Universitätsbibliothek · doi:10.14279/tuj.eceasst.1.78 · Cited by 5
Extensions to Generic Automated Marking Environment: GAME-2+
Matloobi, R., Blumenstein, M., & Green, S. (2009) · Griffith Research Online · Cited by 5
Computer Assisted Assessment. Research into E-Assessment
Ras, E., & Brinke, D. J.-T. (2015) · Communications in computer and information science · doi:10.1007/978-3-319-27704-2 · Cited by 5
Adding Weights to Constraints in Intelligent Tutoring Systems: Does It Improve the Error Diagnosis?
Le, N.-T., & Pinkwart, N. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23985-4_19 · Cited by 4
Pattie: an automated tutor for top-down programming.
Danielson, R. L. (1975) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 4
Helping novice programmers bridge the conceptual gap
He, Y., Ikeda, M., & Mizoguchi, R. (2002) · doi:10.1109/icesd.1994.302282 · Cited by 4
A hybrid approach for program understanding based on graph parsing and expectation-driven analysis
Kim, S.-M., & Kim, J. H. (1998) · Applied Artificial Intelligence · doi:10.1080/088395198117659 · Cited by 4
INCOM: A WEB-BASED HOMEWORK COACHING SYSTEM FOR LOGIC PROGRAMMING
Le, N.-T., & Pinkwart, N. (2011) · Cited by 4
Sant, J. A. (2009) · ACM SIGCSE Bulletin · doi:10.1145/1595496.1562971 · Cited by 4
Problem Solving Process Oriented Diagnosis in Logic Programming
Le, N.-T., & Menzel, W. (2006) · Cited by 3
Assessment of student programming assignments in COBOL
Hasan, H. M. (1988) · Education and Computing · doi:10.1016/s0167-9287(88)90574-2 · Cited by 3
Knowledge base compilation and the language design game
Sack, W. (1992) · Lecture notes in computer science · doi:10.1007/3-540-55606-0_29 · Cited by 3
CSTutor: An Intelligent Tutoring System that Supports Natural Learning
Hartanto, B., & Reye, J. (2013) · doi:10.5176/2251-2195_cseit13.09 · Cited by 3
Automated Feedback and Recognition through Data Mining in Code Hunt
Perelman, D., Bishop, J., Gulwani, S., & Grossman, D. (2015) · Cited by 3
Proceedings of the Australasian conference on Computing education
(2000) · doi:10.1145/359369 · Cited by 3
PRAM: A Courseware System for the Automatic Assessment of AI Programs
Higgins, C., & Mansouri, F. Z. (2000) · Studies in fuzziness and soft computing · doi:10.1007/978-3-7908-1868-0_10 · Cited by 2
An Enhanced Generic Automated Marking Environment: GAME-2
Matloobi, R., Blumenstein, M., & Green, S. (2007) · Griffith Research Online · Cited by 2
The use of reverse engineering to analyse student computer programs
Vanneste, P., Bertels, K., & Decker, B. D. (1996) · Instructional Science · doi:10.1007/bf00119977 · Cited by 2
An Expert System For Education: IPTS
Yongqing, C., Qing, H., & Jingyu, Y. (2005) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1988.712842 · Cited by 2
Partial Specifications of Program Properties
Beierle, C., Kulaš, M., & Widera, M. (2004) · Cited by 2
AutoTeach: incremental hints for programming exercises
Antonucci, P. (2014) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-a-010255485 · Cited by 2
From PROUST to CHIRON: ITS Design as Iterative Engineering; Intermediate Results are Important!
Sack, W., & Soloway, E. M. (2021) · Routledge eBooks · doi:10.4324/9781315044361-9 · Cited by 2
Koh, H., & Wu, D. (1988) · International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems (IEA/AIE) · doi:10.1145/51909.51930 · Cited by 1
A String Of Perls: Using Perl To Teach Perl
Smith, M. A., & Ekstrom, J. J. (2020) · doi:10.18260/1-2--13091 · Cited by 1
An instructional interpreter for basic
Barr, A., & Beard, M. (1976) · ACM SIGCSE Bulletin · doi:10.1145/952989.803494 · Cited by 1
Flagged first, then by citations
Research works printed but not linked: 31
- 10.5555/1018423.1019977 · DOI
- Beierle Christoph, 2004, Proceedings of the International Workshop on Teaching Logic Programming. 18--34
- 10.5555/977403.978384 · DOI
- Brusilovsky Peter, 1996, Proceedings of the Conference on Learning Sciences. 357--362
- Julio, Proceedings of the International Technology, Education and Development Conference. 5691--5700
- de Barros Leliane Nunes, 2006, Proceedings of the Monet Workshop on Model-Based System at ECAI.
- Foxley Eric, Proceedings of the CAA Conference.
- Ghosh Moumita, Proceedings of the Conference on Information Technology and Applications.
- Holland Jay, 2009, Proceedings of the Conference on Computers in Education. 142--146
- Innocenti Carlo, 1991, Proceedings of the PEG Conference on Knowledge Based Environments for Teaching and Learning. 322--329
- Lewis Johnson W., 1984, Proceedings of the AAAI Conference. 162--168
- Köllmann Carsten, 2006, Proceedings of the Workshop on Graph Based Tools, Electronic Communications of the EASST, 1
- Le Nguyen-Thinh, 2006, Proceedings of the Conference on Computers in Education. 63--70
- Le Nguyen-Thinh, 2009, Proceedings of the Conference on Computers in Education. 51--58
- Le Nguyen-Thinh, 2011, Proceedings of the Conference on Cognition and Exploratory Learning in Digital Age. 43--50
- Le Nguyen-Thinh, 2014, Proceedings of the Workshop on AI-supported Education for Computer Science. 51--60
- MacNish Cara, 2002, Proceedings of the IEEE Conference on Advanced Learning Technologies. 317--321
- Matloobi Roozbeh, 2009, Proceedings of the Interactive Computer Aided Learning Conference, 1
- Mitrovic Antonija, User Modeling
- Perelman Daniel, 2014, Proceedings of the Workshop on Data Mining for Educational Assessment and Feedback.
- Peylo Christoph, 2000, Proceedings of the Workshop on Adaptive and Intelligent Web-Based Education Systems, ITS. 70--80
- Pribela Ivan, 2011, Proceedings of the Conference on Information Technology.
- Rahman Khirulnizam Abd, 2007, Proceedings of the National Conference on Programming.
- del Pino Juan Carlos, 2012, Proceedings of the Conference on e-Learning, e-Business, Entreprise Information Systems, 8 e-Government.
- Romli Rohaida, 2010, Proceedings of the International Symposium in Information Technology. 1186--1192
- Matthew, Proceedings of the ASEE Annual Conference and Exposition.
- Striewe Michael, 2009, Proceedings of the Conference on Computer Supported Education 2
- 10.5555/276953.276964 · DOI
- Truong Nghi, 2004, Proceedings of the Australasian Conference on Computing Education, 30
- 10.5555/1434921.1434923 · DOI
- Vizcaíno Aurora, 1839, Intelligent Tutoring Systems. LNCS
Cited by
Sorted by citations
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Artificial Intelligence for Assessment and Feedback to Enhance Student Success in Higher Education
Hooda, M., Rana, C., Dahiya, O., Rizwan, A., & Hossain, M. S. (2022) · Mathematical Problems in Engineering · doi:10.1155/2022/5215722 · Cited by 444
Automatic feedback in online learning environments: A systematic literature review
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100027 · Cited by 440
Deeva, G., Bogdanova, D., Serral, E., Snoeck, M., & Weerdt, J. D. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.104094 · Cited by 292
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“attempts to provide more contextualized help to students learning to program, although effective programming support remains a challenge [43].”
BackgroundInfluential
Metadata
- Authors
- Keuning, H., Jeuring, J., & Heeren, B.
- Affiliations
- Windesheim University of Applied Sciences (NL)
- Utrecht University (NL)
- Open University of the Netherlands (NL)
- Venue
- ACM Transactions on Computing Education
- Published
- 2018-09-28
- Volume
- 19
- Issue
- 1
- Pages
- 1-43
- ISSN
- 1946-6226 (ISSN-L)
- OpenAlex
- W2894540915
- Cited by
- 370
- References
- 200
- Citations by year
- 2026: 72 · 2025: 71 · 2024: 60 · 2023: 58 · 2022: 46 · 2021: 29 · 2020: 22 · 2019: 12
- FWCI
- 2.95
- Type
- journal article
- Language
- en
- Open access
- green
- dspace.library.uu.nl
- Locations
- ACM Transactions on Computing Education (publishedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- DSpace (Open University in the Netherlands) (submittedVersion, other-oa)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Funders
- Netherlands Organisation for Scientific Research (023.005.063)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Student Assessment and Feedback
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- automated feedback, programming exercises, formative feedback, programming education, adaptive feedback, tool evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
