The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale
Abstract
In MOOCs, a single programming exercise may produce thousands of solutions from learners. Understanding solution variation is important for providing appropriate feedback to students at scale. The wide variation among these solutions can be a source of pedagogically valuable examples and can be used to refine the autograder for the exercise by exposing corner cases. We present OverCode, a system for visualizing and exploring thousands of programming solutions. OverCode uses both static and dynamic analysis to cluster similar solutions, and lets teachers further filter and cluster solutions based on different criteria. We evaluated OverCode against a nonclustering baseline in a within-subjects study with 24 teaching assistants and found that the OverCode interface allows teachers to more quickly develop a high-level view of students' understanding and misconceptions, and to provide feedback that is relevant to more students' solutions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Schleimer, S., S, W. D., & Aiken, A. (2003) · Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data · doi:10.1145/872757.872770 · Cited by 1,099 · cited 3×
“MOSS uses a windowing technique to select fingerprints from hashes of k-grams from a solution.”
Influential
Research works printed but not linked: 21
- 10.1162/tacl_a_00236 · DOI · Paper page
- 10.5555/850947.853341 · doi:10.5555/850947.853341
- 10.1145/2556325.2566243 · DOI
- 10.1145/2538862.2538973 · DOI
- 10.1145/2493394.2493400 · doi:10.1145/2493394.2493400
- 10.1145/2445196.2445368 · DOI · Paper page
- Jonathan Huang , Chris Piech , Andy Nguyen , and Leonidas J. Guibas . 2013. Syntactic and functional variability of a million code submissions in a machine learning MOOC . In Proceedings of the 16th International Conference on Artificial Intelligence in Education Workshop on Massive Open Online Courses (MOOCshop). Jonathan Huang, Chris Piech, Andy Nguyen, and Leonidas J. Guibas. 2013. Syntactic and functional variability of a million code submissions in a machine learning MOOC. In Proceedings of the 16th International Conference on Artificial Intelligence in Education Workshop on Massive Open Online Courses (MOOCshop).
- 10.1145/2462476.2462505 · DOI
- Ference Marton , Amy B. M. Tsui , Pakey P. M. Chik , Po Y. Ko , and Mun L . Lo . 2013 . Classroom Discourse and the Space of Learning. Taylor and Francis . Ference Marton, Amy B. M. Tsui, Pakey P. M. Chik, Po Y. Ko, and Mun L. Lo. 2013. Classroom Discourse and the Space of Learning. Taylor and Francis.
- Aditi Muralidharan and Marti Hearst . 2011 . Wordseer: Exploring language use in literary text . In Proceedings of the 5th Workshop on Human-Computer Interaction and Information Retrieval. Aditi Muralidharan and Marti Hearst. 2011. Wordseer: Exploring language use in literary text. In Proceedings of the 5th Workshop on Human-Computer Interaction and Information Retrieval.
- 10.1093/llc/fqs044 · doi:10.1093/llc/fqs044
- 10.1145/2505515.2508212 · doi:10.1145/2505515.2508212
- 10.1145/2566486.2568023 · DOI
- Kelly Rivers and Kenneth R. Koedinger . 2013. Automatic generation of programming feedback: A data-driven approach . In Proceedings of the 1st Workshop on AI-Supported Education for Computer Science (AIEDCS'13) . Kelly Rivers and Kenneth R. Koedinger. 2013. Automatic generation of programming feedback: A data-driven approach. In Proceedings of the 1st Workshop on AI-Supported Education for Computer Science (AIEDCS'13).
- 10.1145/2380116.2380125 · DOI
- 10.1109/21.286387 · DOI
- 10.1145/2491956.2462195 · DOI · Paper page
- 10.1109/TSE.1984.5010283 · DOI · Paper page
- 10.1145/2401796.2401806 · DOI
- Ahmad Taherkhani and Lauri Malmi . 2013 . Beacon-and schema-based method for recognizing algorithms from students' source code . Journal of Educational Data Mining 5 , 2, 69 -- 101 . Ahmad Taherkhani and Lauri Malmi. 2013. Beacon-and schema-based method for recognizing algorithms from students' source code. Journal of Educational Data Mining 5, 2, 69--101.
- 10.1109/TSE.2003.1191799 · DOI
Cited by
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145
Assessing student perceptions and use of instructor versus AI ‐generated feedback
Er, E., Akçapınar, G., Bayazıt, A., Noroozi, O., & Banihashem, S. K. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13558 · Cited by 114
A Survey of Machine Learning for Big Code and Naturalness
Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C. (2019) · ACM Computing Surveys · doi:10.1145/3212695 · Cited by 150
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23
“This observation motivates the need for a tool that would help programmers explore a large space of suggestions, perhaps similarly to how Overcode [Glassman et al. 2015] supports exploring a space of student solutions to a programming assignment.”
MethodSearch, align, and repair: data-driven feedback generation for introductory programming exercises
Wang, K., Singh, R., & Su, Z. (2018) · ACM SIGPLAN Notices · doi:10.1145/3296979.3192384 · Cited by 17
Self-citationToward semi-automatic misconception discovery using code embeddings
Shi, Y., Shah, K., Wang, W., Marwan, S., Penmetsa, P., & Price, T. (2021) · LAK21: 11th International Learning Analytics and Knowledge Conference · doi:10.1145/3448139.3448205 · Cited by 14
Metadata
- Authors
- Glassman, E. L., Scott, J., Singh, R., Guo, P. J., & Miller, R. C.
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2015-03-10
- Volume
- 22
- Issue
- 2
- Pages
- 1-35
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W2076771354
- Cited by
- 177
- References
- 1
- FWCI
- 24.13
- Type
- journal article
- Language
- en
- Open access
- closed
- Funders
- National Science Foundation Graduate Research Fellowship (1122374)
- Bose Foundation Fellowship
- Microsoft Research Fellowship
- Quanta Computer as part of the Qmulus Project
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.