The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
STOP THE (AUTOGRADER) INSANITY: Regression Penalties to Deter Autograder Overreliance
Abstract
Autograders are an invaluable tool for deploying assessments in large classes. However students sometimes rely on the autograder in place of careful thought for ways to improve to their solution. We sought to naturally encourage students to check their own solutions more, and hammer the grader less. To do this, we imposed a penalty each time a student's grade went down: we called these regression penalties. We assessed whether the introduction of these penalties resulted in less reliance on the autograder without hurting student performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Proceedings of the 10th Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 554
A Survey of Automated Assessment Approaches for Programming Assignments
Ala-Mutka, K. M. (2005) · Computer Science Education · doi:10.1080/08993400500150747 · Cited by 546
Web-CAT: automatically grading programming assignments
Edwards, S., & Pérez-Quiñones, M. A. (2008) · Annual Conference on Innovation and Technology in Computer Science Education · doi:10.1145/1597849.1384371 · Cited by 314
Web-CAT: automatically grading programming assignments
Edwards, S. H., & Perez-Quinones, M. A. (2008) · Proceedings of the 13th annual conference on Innovation and technology in computer science education · doi:10.1145/1384271.1384371 · Cited by 256
Spacco, J., Hovemeyer, D., Pugh, W., Emad, F., Hollingsworth, J. K., & Padua-Perez, N. (2006) · ACM SIGCSE Bulletin · doi:10.1145/1140123.1140131 · Cited by 163
On the use of resubmissions in automatic assessment systems
Karavirta, V., Korhonen, A., & Malmi, L. (2006) · Computer Science Education · doi:10.1080/08993400600912426 · Cited by 84
Automatic Grading of Programming Assignments: An Approach Based on Formal Semantics
Liu, X., Wang, S., Wang, P., & Wu, D. (2019) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet.2019.00022 · Cited by 75
Executable Examples for Programming Problem Comprehension
Wrenn, J., & Krishnamurthi, S. (2019) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3291279.3339416 · Cited by 60
Developing Software Engineering Skills using Real Tools for Automated Grading
Heckman, S. S., & King, J. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159595 · Cited by 46
Benotti, L., Aloi, F., Bulgarelli, F. L., & Gómez, M. J. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159579 · Cited by 45
Cited by
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“This is similar to earlier results that have found that students sometimes develop an over-reliance on automatically generated feedback from automated assessment systems [7].”
BackgroundResultDrop Project: An automatic assessment tool for programming assignments
Cipriano, B. P., Fachada, N., & Alves, P. (2022) · SoftwareX · doi:10.1016/j.softx.2022.101079 · Cited by 37
A Comprehensive Review on Automated Grading Systems in STEM Using AI Techniques
Tan, L. Y., Hu, S., Yeo, D. J. S., & Cheong, K. H. (2025) · Mathematics · doi:10.3390/math13172828 · Cited by 43
A Comparison of Immediate and Scheduled Feedback in Introductory Programming Projects
Leinonen, J., Denny, P., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499372 · Cited by 32
Metadata
- Authors
- Baniassad, E., Zamprogno, L., Hall, B., & Holmes, R.
- Affiliations
- University of British Columbia (CA)
- Venue
- Proceedings of the 52nd ACM Technical Symposium on Computer Science Education
- Publisher
- ACM
- Published
- 2021-03-03
- Pages
- 1062-1068
- OpenAlex
- W3134161095
- Cited by
- 48
- References
- 14
- Citations by year
- 2026: 3 · 2025: 15 · 2024: 6 · 2023: 15 · 2022: 7 · 2021: 2
- FWCI
- 8.75
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- automated grading, student self-assessment, student performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: STOP THE (AUTOGRADER) INSANITY: Regression Penalties to Deter Autograder Overreliance. https://app.cochat.ai/papers/10.1145/3408877.3432430
- The paper
- Baniassad, E., Zamprogno, L., Hall, B., & Holmes, R. (2021). STOP THE (AUTOGRADER) INSANITY: Regression Penalties to Deter Autograder Overreliance. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. https://doi.org/10.1145/3408877.3432430
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.