The Code Mangler: Evaluating Coding Ability Without Writing any Code
The DOI matches its Crossref record.
Abstract
Marking coding exam questions for introductory computer science courses is notoriously resource intensive and difficult to perform consistently. Students can be easily led astray by minor misunderstandings in the wording of questions, and graders often find it difficult to decide whether mistakes are attributable to simple misinterpretations, minor memory errors, or major lack of ability/understanding of the core concepts being evaluated. In this paper we detail and evaluate "Code Mangler" questions. The "Code Mangler" is a fictitious character who manipulates code; removing commenting, changing the order of lines, adding bugs, and otherwise breaking perfectly good code. The role of the student on the exam is then to use the mangled results to reverse engineer the original code. We discuss the benefits of this style of question, and perform an evaluation on a large (475 student) CS1 course, demonstrating that these questions are less resource intensive to mark than traditional coding questions, improve the confidence of the graders, and correlate strongly with student ability as assessed in traditional question styles.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Solving parsons problems versus fixing and writing code
Ericson, B. J., Margulieux, L. E., & Rick, J. (2017) · Proceedings of the 17th Koli Calling International Conference on Computing Education Research · doi:10.1145/3141880.3141895 · Cited by 121
Reeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B. A., Hellas, A., Kimmel, B., Powell, G., & Leinonen, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588805 · Cited by 56
“Cheng & Harrington studied the benefits of using Parsons problems on summative assessments with respect to grading consistency, finding that constraints imposed by the problems yielded faster grading times and higher grader confidence [2].”
BackgroundEvaluating the Effectiveness of Parsons Problems for Block-based Programming
Zhi, R., Chi, M., Barnes, T., & Price, T. W. (2019) · Proceedings of the 2019 ACM Conference on International Computing Education Research · doi:10.1145/3291279.3339419 · Cited by 42
Haynes-Magyar, C. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630898 · Cited by 9
Metadata
- Authors
- Cheng, N., & Harrington, B.
- Venue
- Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education
- Publisher
- ACM
- Published
- 2017-03-08
- Pages
- 123-128
- OpenAlex
- W2594286748
- Cited by
- 26
- References
- 0
- FWCI
- 3.29
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.