The effect of automatic assessment on novice programming: Strengths and limitations of existing systems
The DOI matches its Crossref record.
Abstract
Computer programming is always of high concern for students in introductory programming courses. High rates of failure occur every semester due to lack of adequate skills in programming. No student can become a programmer overnight because such learning requires proper guidance as well as consistent practice with the programming exercises. The role of instructors in the development of students' learning skills is crucial in order to provide feedback on their errors and improve their knowledge accordingly. On the other hand, due to the large number of students, instructors are also overloading themselves to focus on each individual student's errors. To address these issues, researchers have developed numerous Automatic Assessment (AA) systems that not only evaluate the students' programs but also provide instant feedback on their errors as well as abridge the workload of the instructors. Due to the large pool of existing systems, it is difficult to cover each and every system in one study. Therefore, this paper provides a comprehensive overview of some of the existing systems based on the three‐analysis approaches: dynamic, static, and hybrid. Moreover, this paper aims to discuss the strengths and limitations of these systems and suggests some potential recommendations regarding the AA specifications for novice programming, which may help in standardizing these systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Automated Grading and Feedback Tools for Programming Education: A Systematic Review
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2024) · ACM Transactions on Computing Education · doi:10.1145/3636515 · Cited by 262
Comparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195
“To help, instructors can create explanations for students as examples, although providing personalized explanations to every student in the class is time consuming [40].”
BackgroundMacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.02265 · Cited by 195
Yang, T.-C., Hsu, Y.-C., & Wu, J.-Y. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2450659 · Cited by 80
“Automated grading systems may also limit topics, use rigid criteria, and fail to distinguish between trivial and nontrivial errors (Ullah et al., 2018).”
Pereira, F. D., Rodrigues, L., Henklain, M. H. O., Freitas, H., Oliveira, D. F., Cristea, A. I., Carvalho, L., Isotani, S., Benedict, A., Dorodchi, M., et al. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2022.3224121 · Cited by 26
“It evaluates students’ code using a “dynamic analysis approach,” as defined by the taxonomy proposed by Ullah et al. [36], i.e., students submit their program to the platform, and the output is compared against a set of test cases, manually provided by the instructor or by the system maintainers.”
MethodMarquès, J. M., Calvet, L., Arguedas, M., Daradoumis, T., & Mor, E. (2022) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2021.1938400 · Cited by 14
A Survey on Feedback Types in Automated Programming Assessment Systems
Frankford, E., Antensteiner, T., Vierhauser, M., Sauerwein, C., Wallner, V., Groher, I., Plösch, R., & Breu, R. (2026) · ACM Transactions on Computing Education · doi:10.1145/3773911 · Cited by 10
“This impact is crucial for students in introductory programming courses as it provides immediate feedback and eases the teacher’s workload when the number of students is high, so that the development of essential skills can be supported [44].”
Improving Student Learning with Automated Assessment
Acuña, R., & Bansal, A. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653603 · Cited by 7
Metadata
- Authors
- Ullah, Z., Lajis, A., Jamjoom, M., Altalhi, A., Al‐Ghamdi, A., & Saleem, F.
- Venue
- Computer Applications in Engineering Education
- Publisher
- Wiley
- Published
- 2018-05-29
- Volume
- 26
- Issue
- 6
- Pages
- 2328-2341
- ISSN
- 1061-3773 (ISSN-L)
- 1099-0542
- OpenAlex
- W2807648480
- Cited by
- 79
- References
- 0
- FWCI
- 6.35
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.