Promoting Early Engagement with Programming Assignments Using Scheduled Automated Feedback
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Programming assignments are a common form of assessment in introductory courses and often require substantial work to complete. Students must therefore plan and manage their time carefully, especially leading up to published deadlines. Although time management is an important metacognitive skill that students must develop, it is rarely taught explicitly. Prior research has explored various approaches for reducing procrastination and other unproductive behaviours in students, but these are often ineffective or impractical in large courses. In this work, we investigate a scalable intervention that incentivizes students to begin work early. We provide automatically generated feedback to students who submit their work-in-progress prior to two fixed deadlines scheduled earlier than the final deadline for the assignment. Although voluntary, we find that many students welcome this early feedback and improve the quality of their work across each iteration. Especially for at-risk students, who have failed an earlier module in the course, engaging with the early feedback opportunities results in significantly better work at the time of final submission.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Learning to Teach in Higher Education
Ramsden, P., & Ramsden, P. (2003) · Routledge eBooks · doi:10.4324/9780203507711 · Cited by 5,106
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Proceedings of the 10th Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 554 · cited 1×
“A 2010 systematic review of the literature on automated programming assessment tools reported that many systems produce feedback based around test cases that are either automatically generated or taken from instructor-provided test suites [20].”
BackgroundA systematic literature review of automated feedback generation for programming exercises
Keuning, H., Jeuring, J., & Heeren, B. (2019) · ACM Transactions on Computing Education · doi:10.1145/3231711 · Cited by 370
Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies
Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., Isohanni, E., Korhonen, A., Petersen, A., Rivers, K., et al. (2015) · Proceedings of the 2015 ITiCSE on Working Group Reports · doi:10.1145/2858796.2858798 · Cited by 332 · cited 1×
“interface throughwhich code can be uploaded or submitted, provide convenience for students and the ability to collect fine-grained process data for instructors and educational researchers [21, 33].”
BackgroundWeb-CAT: automatically grading programming assignments
Edwards, S. H., & Perez-Quinones, M. A. (2008) · Proceedings of the 13th annual conference on Innovation and technology in computer science education · doi:10.1145/1384271.1384371 · Cited by 256
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199 · cited 1×
“As a result, there has been a recent surge in interest focused on improving the quality and usefulness of such error messages [2, 11].”
BackgroundSelf-citationGoda, Y., Yamada, M., Kato, H., Matsuda, T., Saito, Y., & Miyagawa, H. (2014) · Learning and Individual Differences · doi:10.1016/j.lindif.2014.11.001 · Cited by 135 · cited 1×
“Moreover, procrastination is a common and widely acknowledged problem for many students [7, 18].”
BackgroundWhat Do We Think We Think We Are Doing?: Metacognition and Self-Regulation in Programming
Prather, J., Becker, B. A., Craig, M., Denny, P., Loksa, D., & Margulieux, L. (2020) · Proceedings of the 2020 ACM Conference on International Computing Education Research · doi:10.1145/3372782.3406263 · Cited by 131 · cited 1×
“In the programming education literature, a range of activities aimed at supporting metacognitive and self-regulatory skills have specifically focused on time management [31].”
BackgroundSelf-citationInflammatory Myofibroblastic Bladder Tumor in a Patient with Wolf-Hirschhorn Syndrome
Marte, A., Indolfi, P., Ficociello, C., Russo, D., Oreste, M., Bottigliero, G., Gualdiero, G., Barone, C., Vigliar, E., Indolfi, C., et al. (2013) · Case Reports in Urology · doi:10.1155/2013/675059 · Cited by 117
Comparing effective and ineffective behaviors of student programmers
Edwards, S. H., Snyder, J., Pérez-Quiñones, M. A., Allevato, A., Kim, D., & Tretola, B. (2009) · Proceedings of the fifth international workshop on Computing education research workshop · doi:10.1145/1584322.1584325 · Cited by 115 · cited 2×
“effect on their performance [29], there is much more evidence that starting early has a positive effect [1, 15, 17, 39].”
BackgroundInflammatory Myofibroblastic Bladder Tumor in a Patient with Wolf-Hirschhorn Syndrome
Marte, A., Casale, F., Indolfi, P., et al. (2013) · Dr RGF - RGF Repository · doi:10.1155/2013 · Cited by 97
Malmi, L., Karavirta, V., Korhonen, A., & Nikander, J. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163412 · Cited by 90 · cited 1×
“Such formative re-submissions allow students to learn from their mistakes, reduce careless errors, and can reduce the anxiety associated with the act of submitting an assignment [28].”
BackgroundCodeWrite: supporting student-driven practice of java
Denny, P., Luxton-Reilly, A., Tempero, E., & Hendrickx, J. (2011) · Proceedings of the 42nd ACM technical symposium on Computer science education · doi:10.1145/1953163.1953299 · Cited by 88 · cited 3×
“int MakeMove(int warehouse[10][10], char move)”
BackgroundInfluentialSelf-citationAnalyzing Student Work Patterns Using Programming Exercise Data
Spacco, J., Denny, P., Richards, B., Babcock, D., Hovemeyer, D., Moscola, J., & Duvall, R. (2015) · Proceedings of the 46th ACM Technical Symposium on Computer Science Education · doi:10.1145/2676723.2677297 · Cited by 74 · cited 1×
“In his best-selling book on teaching in higher education, Ramsden provides a section on the effect of feedback on performance [32, p. 1930].”
BackgroundSelf-citationHelping students appreciate test-driven development (TDD)
Spacco, J. W., & Pugh, W. (2006) · Companion to the ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA Companion) · doi:10.1145/1176617.1176743 · Cited by 74 · cited 1×
“In a similar vein, Spacco and Pugh describe an auto-grading tool called Marmoset, which allows students to submit their code which is executed against a set of public test cases [34].”
BackgroundCodeWorkout: Short Programming Exercises with Built-in Data Collection
Edwards, S. H., & Murali, K. P. (2017) · Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3059009.3059055 · Cited by 62 · cited 1×
“A wide variety of such systems exist, ranging from practice tools that provide automatic feedback on small fragments of student-written code [10, 13, 30], to more sophisticated frameworks designed to automatically grade large multi-file assignments [14].”
BackgroundMining student CVS repositories for performance indicators
Mierle, K., Laven, K., Roweis, S., & Wilson, G. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083150 · Cited by 51 · cited 1×
“Although some studies involving novice programmers have reported that the time at which a student starts their assignment has little effect on their performance [29], there is much more evidence that starting early has a positive effect [1, 15, 17, 39].”
BackgroundError Message Readability and Novice Debugging Performance
Denny, P., Prather, J., & Becker, B. A. (2020) · Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3341525.3387384 · Cited by 50 · cited 1×
“As a result, there has been a recent surge in interest focused on improving the quality and usefulness of such error messages [2, 11].”
BackgroundSelf-citationQuantifying Incremental Development Practices and Their Relationship to Procrastination
Kazerouni, A. M., Edwards, S. H., & Shaffer, C. A. (2017) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3105726.3106180 · Cited by 48 · cited 1×
“There is also clear evidence that starting work early, and making incremental progress, is strongly correlated with successful outcomes [9, 25].”
BackgroundCormack, S. H., Eagle, L. A., & Davies, M. S. (2020) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2019.1705244 · Cited by 48 · cited 1×
“Moreover, procrastination is a common and widely acknowledged problem for many students [7, 18].”
BackgroundHow can software metrics help novice programmers
Cardell‐Oliver, R. (2011) · UWA Profiles and Research Repository (UWA) · Cited by 47
Another look at the behaviors of novice programmers
Fenwick, J. B., Norris, C., Barry, F. E., Rountree, J., Spicer, C. J., & Cheek, S. D. (2009) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1508865.1508973 · Cited by 47 · cited 1×
“effect on their performance [29], there is much more evidence that starting early has a positive effect [1, 15, 17, 39].”
BackgroundSupporting Self-Regulated Learning with Visualizations in Online Learning Environments
Ilves, K., Leinonen, J., & Hellas, A. (2018) · Proceedings of the 49th ACM Technical Symposium on Computer Science Education · doi:10.1145/3159450.3159509 · Cited by 46 · cited 1×
“In another study by Ilves et al. visualizations targeting time management were only helpful for low-performing students and even harmful for performance-oriented students [22].”
BackgroundSelf-citationDenny, P., Becker, B. A., Craig, M., Wilson, G., & Banaszkiewicz, P. (2019) · Proceedings of the 2019 ACM Conference on International Computing Education Research · doi:10.1145/3291279.3339402 · Cited by 41 · cited 1×
“Providing convenient and actionable feedback, which is simple for students to understand, remains an ongoing challenge of great interest to computing educators [8].”
BackgroundSelf-citationOn study habits on an introductory course on programming
Willman, S., Lindén, R., Kaila, E., Rajala, T., Laakso, M., & Salakoski, T. (2015) · Computer Science Education · doi:10.1080/08993408.2015.1073829 · Cited by 38 · cited 1×
“The assignment, a game inspired by Sokoban [38], was presented as ten smaller tasks in the context of an overarching larger project.”
MethodExamining Classroom Interventions to Reduce Procrastination
Edwards, S. H., Martin, J., & Shaffer, C. A. (2015) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2729094.2742632 · Cited by 36 · cited 1×
“In a study by Edwards et al. [12] comparing three interventions, only regular automated email alerts which included a reminder of upcoming deadlines seemed to have an effect.”
BackgroundMining student CVS repositories for performance indicators
Mierle, K., Laven, K., Roweis, S. T., & Wilson, G. (2005) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/1083142.1083150 · Cited by 32
Harmful Study Habits in Online Learning Environments with Automatic Assessment
Auvinen, T. (2015) · International Conference on Learning and Teaching in Computing and Engineering (LaTICE) · doi:10.1109/latice.2015.31 · Cited by 30 · cited 1×
“Although some studies involving novice programmers have reported that the time at which a student starts their assignment has little effect on their performance [29], there is much more evidence that starting early has a positive effect [1, 15, 17, 39].”
BackgroundCan Mobile Gaming Psychology Be Used to Improve Time Management on Programming Assignments?
Irwin, M. S., & Edwards, S. H. (2019) · Proceedings of the ACM Conference on Global Computing Education · doi:10.1145/3300115.3309517 · Cited by 27 · cited 1×
“Irwin and Edwards explored an ingenious way of using mobile gaming psychology to encourage better time management by students when working on programming assignments [23].”
BackgroundCompareCFG: Providing Visual Feedback on Code Quality Using Control Flow Graphs
Jiang, L., Rewcastle, R., Denny, P. C., & Tempero, E. D. (2020) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3341525.3387362 · Cited by 24
Self-citationVivian, R., Falkner, K. E. K., & Falkner, N. (2013) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2526968.2526982 · Cited by 23 · cited 1×
“Time management has been identified as one of the primary reasons that students struggle in their programming courses [36].”
BackgroundUnderstanding the Effects of Lecturer Intervention on Computer Science Student Behaviour
Szabo, C., Falkner, N., Knutas, A., & Dorodchi, M. (2018) · ITiCSE-WGR · doi:10.1145/3174781.3174787 · Cited by 22 · cited 1×
“A review of a range of educational interventions in computer science classrooms shows significant evidence for the benefits of automated feedback [35].”
BackgroundImproving complex task performance using a sequence of simple practice tasks
Denny, P., Luxton-Reilly, A., Craig, M., & Petersen, A. (2018) · Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3197091.3197141 · Cited by 21 · cited 1×
“There is also clear evidence that starting work early, and making incremental progress, is strongly correlated with successful outcomes [9, 25].”
BackgroundSelf-citationParlante, N. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1272848.1272876 · Cited by 21 · cited 1×
“A wide variety of such systems exist, ranging from practice tools that provide automatic feedback on small fragments of student-written code [10, 13, 30], to more sophisticated frameworks designed to automatically grade large multi-file assignments [14].”
BackgroundA unit testing approach to building novice programmers' skills and confidence
Whalley, J. L., & Philpott, A. (2011) · Australasian Computing Education Conference · Cited by 18
Self-citationAssignment submission, student behaviour and experience
Gregory, K., & Morón-García, S. (2009) · Engineering Education · doi:10.11120/ened.2009.04010016 · Cited by 17 · cited 1×
“As students progress through their degree, their time management skills tend to improve [19].”
BackgroundAutoStyle: Toward Coding Style Feedback At Scale
Choudhury, R. R., Yin, H., Moghadam, J., & Fox, A. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818052.2874315 · Cited by 16 · cited 1×
“organised code, and future work incorporating some automatic feedback on code style is warranted [6].”
BackgroundMariaye, H., Murden-Louise, M., & Ramasawmy, S. J. (2023) · Scholarship of Teaching and Learning in the South · doi:10.36615/sotls.v7i2.317 · Cited by 13
Feedback for Programming Assignments Using Software-Metrics and Reference Code
Koyya, P., Lee, Y., & Yang, J. Y. (2013) · ISRN Software Engineering · doi:10.1155/2013/805963 · Cited by 11 · cited 2×
“Work in this area also suggests that students find it difficult to interpret metric-based feedback [27], and has lead researchers to explore a variety of feedback formats [24].”
BackgroundAutomated Feedback for Quality Assurance in Software Engineering Education
Cardell‐Oliver, R., Zhang, L., Barady, R., Lim, Y. H., Naveed, A., & Woodings, T. (2010) · Australasian Software Engineering Conference (ASWEC) · doi:10.1109/aswec.2010.24 · Cited by 10 · cited 1×
“This approach has been found to afford some benefits, as the feed-back students receive enables them to write fully compliant and functionally correct code, helps them focus on code quality, and increases their confidence in their programming ability [4, 5].”
BackgroundCompetition and Feedback through Automated Assessment in a Data Structures and Algorithms Course
Färnqvist, T., & Heintz, F. (2016) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2899415.2899454 · Cited by 10 · cited 1×
“In general, automated feedback is viewed positively by programming students and influences them to put more effort into their work [16].”
BackgroundSoftware metrics as a programming training tool
Bowman, B. J., & Newman, W. A. (1990) · Journal of Systems and Software · doi:10.1016/0164-1212(90)90119-7 · Cited by 6 · cited 1×
“Other research has looked at the usefulness of software metrics [3, 27] as a feedback mechanism.”
Background
Cited by
Automated Grading and Feedback Tools for Programming Education: A Systematic Review
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2024) · ACM Transactions on Computing Education · doi:10.1145/3636515 · Cited by 262
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“In the case of these systems, prior work has presented multiple ways of trying to combat over-reliance such as submission penalties [7, 59] or limiting the number of submissions students can have [35, 49, 59].”
BackgroundSelf-citationChat Overflow: Artificially Intelligent Models for Computing Education - renAIssance or apocAIypse?
Denny, P., Becker, B. A., Leinonen, J., & Prather, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588773 · Cited by 32
Self-citationPrather, J., Margulieux, L., Whalley, J., Denny, P., Reeves, B. N., Becker, B. A., Singh, P., Powell, G., & Bosch, N. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543970 · Cited by 31
Self-citationImproving Student Peer Code Review Using Gamification
Indriasari, T. D., Luxton-Reilly, A., & Denny, P. C. (2021) · Australasian Computing Education Conference · doi:10.1145/3441636.3442308 · Cited by 19
“The availability of actionable automated feedback in programming tasks may also encourage students to submit their work ahead of a final deadline [8], a behavior which is strongly correlated with better outcomes.”
BackgroundSelf-citation
Metadata
- Authors
- Denny, P., Whalley, J., & Leinonen, J.
- Affiliations
- University of Auckland (NZ)
- Auckland University of Technology (NZ)
- University of Helsinki (FI)
- Venue
- Proceedings of the 23rd Australasian Computing Education Conference
- Publisher
- ACM
- Published
- 2021-02-02
- Pages
- 88-95
- OpenAlex
- W3139412227
- Cited by
- 24
- References
- 41
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 5 · 2023: 11 · 2022: 1 · 2021: 2
- FWCI
- 5.54
- Type
- conference paper
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- Australasian Computing Education Conference (publishedVersion)
- Työväentutkimus Vuosikirja (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Student Assessment and Feedback
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- programming exercises, automated feedback, early feedback, procrastination, time management, metacognition, at-risk students, introductory programming, assignment deadlines, student engagement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.