STOP THE (AUTOGRADER) INSANITY: Regression Penalties to Deter Autograder Overreliance
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Autograders are an invaluable tool for deploying assessments in large classes. However students sometimes rely on the autograder in place of careful thought for ways to improve to their solution. We sought to naturally encourage students to check their own solutions more, and hammer the grader less. To do this, we imposed a penalty each time a student's grade went down: we called these regression penalties. We assessed whether the introduction of these penalties resulted in less reliance on the autograder without hurting student performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Proceedings of the 10th Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 554 · cited 1×
“In an attempt to address these downsides, some automated assessment tools employ features such as limiting submission number or frequency, limiting feedback, and randomizing exercises [7].”
MethodA Survey of Automated Assessment Approaches for Programming Assignments
Ala-Mutka, K. M. (2005) · Computer Science Education · doi:10.1080/08993400500150747 · Cited by 546 · cited 1×
“However, students can become reliant on the automated feedback over their own tests [1], or resubmit rapidly without taking the time to think about their submission [8].”
BackgroundWeb-CAT: automatically grading programming assignments
Edwards, S. H., & Perez-Quinones, M. A. (2008) · Proceedings of the 13th annual conference on Innovation and technology in computer science education · doi:10.1145/1384271.1384371 · Cited by 256
Spacco, J., Hovemeyer, D., Pugh, W., Emad, F., Hollingsworth, J. K., & Padua-Perez, N. (2006) · ACM SIGCSE Bulletin · doi:10.1145/1140123.1140131 · Cited by 163
Web-CAT: automatically grading programming assignments
Edwards, S. H., & Perez-Quinones, M. A. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1597849.1384371 · Cited by 87 · cited 3×
“, [5, 14]) as a means to encourage self-reliance.”
BackgroundOn the use of resubmissions in automatic assessment systems
Karavirta, V., Korhonen, A., & Malmi, L. (2006) · Computer Science Education · doi:10.1080/08993400600912426 · Cited by 83 · cited 1×
“However, students can become reliant on the automated feedback over their own tests [1], or resubmit rapidly without taking the time to think about their submission [8].”
BackgroundAutomatic Grading of Programming Assignments: An Approach Based on Formal Semantics
Liu, X., Wang, S., Wang, P., & Wu, D. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet.2019.00022 · Cited by 74
Executable Examples for Programming Problem Comprehension
Wrenn, J., & Krishnamurthi, S. (2019) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3291279.3339416 · Cited by 60 · cited 1×
“Other approaches involve assessing students tests themselves against reference implementations, each with their own mutations, to identify howmany of themutants student tests identify [13].”
BackgroundBenotti, L., Aloi, F., Bulgarelli, F. L., & Gómez, M. J. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159579 · Cited by 52
Developing Software Engineering Skills using Real Tools for Automated Grading
Heckman, S. S., & King, J. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159595 · Cited by 52 · cited 1×
“Other tools employ industry-standard tools for assessment [6].”
MethodNudging student learning strategies using formative feedback in automatically graded assessments
Zamprogno, L., Holmes, R., & Baniassad, E. (2020) · doi:10.1145/3426431.3428654 · Cited by 29 · cited 1×
“, [5, 14]) as a means to encourage self-reliance.”
BackgroundSelf-citationSome aspects of grading Java code submissions in MOOCs
Király, S., Nehéz, K., & Hornyák, O. (2017) · Research in Learning Technology · doi:10.25304/rlt.v25.1945 · Cited by 15 · cited 1×
“Techniques involve running test cases against submissions and using success combined potentially with other factors such as test coverage [5, 12], style checking [11], or structural checking [4].”
BackgroundSimulating Student Mistakes to Evaluate the Fairness of Automated Grading
Clegg, B., North, S., McMinn, P., & Fraser, G. (2019) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet.2019.00021 · Cited by 11 · cited 1×
“Another allows educators to validate the fairness of their assessment framework through mutation testing [3].”
BackgroundEvaluation of a tool for Java structural specification checking
Dil, A., & Osunde, J. (2018) · International Conference on Education Technology and Computer · doi:10.1145/3290511.3290528 · Cited by 4 · cited 1×
“Techniques involve running test cases against submissions and using success combined potentially with other factors such as test coverage [5, 12], style checking [11], or structural checking [4].”
Background
Cited by
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“This is similar to earlier results that have found that students sometimes develop an over-reliance on automatically generated feedback from automated assessment systems [7].”
BackgroundResultDrop Project: An automatic assessment tool for programming assignments
Cipriano, B. P., Fachada, N., & Alves, P. (2022) · SoftwareX · doi:10.1016/j.softx.2022.101079 · Cited by 37
A Comprehensive Review on Automated Grading Systems in STEM Using AI Techniques
Tan, L. Y., Hu, S., Yeo, D. J. S., & Cheong, K. H. (2025) · Mathematics · doi:10.3390/math13172828 · Cited by 43
A Comparison of Immediate and Scheduled Feedback in Introductory Programming Projects
Leinonen, J., Denny, P., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499372 · Cited by 32
“However, immediate unlimited feedback from autograders can result in over-reliance on the feedback [3].”
BackgroundInfluential
Metadata
- Authors
- Baniassad, E., Zamprogno, L., Hall, B., & Holmes, R.
- Affiliations
- University of British Columbia (CA)
- Venue
- Proceedings of the 52nd ACM Technical Symposium on Computer Science Education
- Publisher
- ACM
- Published
- 2021-03-03
- Pages
- 1062-1068
- OpenAlex
- W3134161095
- Cited by
- 48
- References
- 14
- Citations by year
- 2026: 3 · 2025: 15 · 2024: 6 · 2023: 15 · 2022: 7 · 2021: 2
- FWCI
- 8.75
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- automated grading, student self-assessment, student performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.