Automated Assessment in Computer Science Education: A State-of-the-Art Review
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Practical programming competencies are critical to the success in computer science (CS) education and go-to-market of fresh graduates. Acquiring the required level of skills is a long journey of discovery, trial and error, and optimization seeking through a broad range of programming activities that learners must perform themselves. It is not reasonable to consider that teachers could evaluate all attempts that the average learner should develop multiplied by the number of students enrolled in a course, much less in a timely, deep, and fair fashion. Unsurprisingly, exploring the formal structure of programs to automate the assessment of certain features has long been a hot topic among CS education practitioners. Assessing a program is considerably more complex than asserting its functional correctness, as the proliferation of tools and techniques in the literature over the past decades indicates. Program efficiency, behavior, and readability, among many other features, assessed either statically or dynamically, are now also relevant for automatic evaluation. The outcome of an evaluation evolved from the primordial Boolean values to information about errors and tips on how to advance, possibly taking into account similar solutions. This work surveys the state of the art in the automated assessment of CS assignments, focusing on the supported types of exercises, security measures adopted, testing techniques used, type of feedback produced, and the information they offer the teacher to understand and optimize learning. A new era of automated assessment, capitalizing on static analysis techniques and containerization, has been identified. Furthermore, this review presents several other findings from the conducted review, discusses the current challenges of the field, and proposes some future research directions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Feedback in Higher and Professional Education: Understanding it and Doing it Well
Merry, S. P. (2016) · Innovative practice in higher education · Cited by 289
Luo, L., Ming, J., Wu, D., Liu, P., & Zhu, S. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635900 · Cited by 220
JPlag: finding plagiarisms among a set of programs
Prechelt, L., Malpohl, G., & Philippsen, M. (2000) · Karlsruhe · doi:10.5445/ir/542000 · Cited by 144 · cited 1×
“Plagiarism, code quality, and other software metrics are typically either handled by external tools, such as JPLAG [190] and CheckStyle [117], or left for the tool manager to set up with custom scripts.”
Method(2019) · doi:10.1145/3287324.3287374 · Cited by 104 · cited 1×
“Furthermore, as most of the assignments considered are open-answer, these tools are not commonplace to incorporate KMC feedback, even though a few recent attempts exist [142, 188, 189].”
BackgroundSupporting Students in C++ Programming Courses with Automatic Program Style Assessment
Ala-Mutka, K., Uimonen, T., & Järvinen, H.-M. (2004) · Journal of Information Technology Education Research · doi:10.28945/300 · Cited by 96
Value-based program characterization and its application to software plagiarism detection
Jhi, Y.-C., Wang, X., Jia, X., Zhu, S., Liu, P., & Wu, D. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985899 · Cited by 83
Generating Data-Driven Hints for Open-Ended Programming.
Price, T., Dong, Y., & Barnes, T. M. (2016) · Educational Data Mining · Cited by 76
A Java programming Learning Assistant System using test-driven development method
Funabiki, N., Matsushima, Y., Nakanishi, T., Watanabe, K., & Amano, N. (2013) · IAENG international journal of computer science · Cited by 75
Software plagiarism detection: a graph-based approach
Chae, D.-K., Ha, J., Kim, S.-W., Kang, B., & Im, E. G. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2507848 · Cited by 66 · cited 1×
“So, they typically encode information using graphs, such as program dependence graphs [52, 210], control-low graphs [45], and call graphs [187], rather than tokens and trees.”
BackgroundJutge.org: Characteristics and Experiences
Petit, J., Roura, S., Carmona, J., Cortadella, J., Duch, J., Gimenez, O., Mani, A., Mas, J., Rodriguez-Carbonell, E., Rubio, E., et al. (2018) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2017.2723389 · Cited by 33 · cited 3×
“org [176] Jail process YES GradeIT (PrUTOR) [62, 173] Container YES neoESPA [148, 150] User permissions NO Enki (Mooshak) [170] User permissions NO ProgEdu [50, 252] Container YES Aristotle [2] User permissions NO Submitty [180] Container YES Codeflex [40] Jail process YES Dante [70] User permissions YES Coderiu [28]…”
BackgroundMethodDochy, F. J. R. C. (2006) · Educational Research Review · doi:10.1016/j.edurev.2006.02.001 · Cited by 21
Automatic Extraction of AST Patterns for Debugging Student Programs
Lazar, T., Možina, M., & Ivan Bratko, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-61425-0_14 · Cited by 20
Author Identification of Software Source Code with Program Dependence Graphs
Chen, R., Hong, L., Chunyan Lü, C., & Deng, W. (2010) · 2010 IEEE 34th Annual Computer Software and Applications Conference Workshops · doi:10.1109/compsacw.2010.56 · Cited by 13
KNN Applied to PDG for Source Code Similarity Classification
Silva, C. D. S., da Costa, L. F., Rocha, L. S., & Viana, G. V. R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61380-8_32 · Cited by 10
Cited by
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
“One common approach for providing feedback in introductory programming courses is the use of automated assessment systems [2, 35, 60], which at the minimum provide feedback on the correctness of programming assignments submitted for evaluation.”
BackgroundAutomated Grading and Feedback Tools for Programming Education: A Systematic Review
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2024) · ACM Transactions on Computing Education · doi:10.1145/3636515 · Cited by 262
Guo, S., Zheng, Y., & Zhai, X. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12491-8 · Cited by 117
“However, studies that empirically examine the educational effectiveness of automated assessment in CS education are still not sufficient (Paiva et al., 2022).”
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
Metadata
- Authors
- Paiva, J. C., Leal, J. P., & Figueira, Á.
- Venue
- ACM Transactions on Computing Education
- Published
- 2022-06-09
- Volume
- 22
- Issue
- 3
- Pages
- 1-40
- ISSN
- 1946-6226 (ISSN-L)
- OpenAlex
- W4213421742
- Cited by
- 274
- References
- 14
- FWCI
- 42.11
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.