The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Review of recent systems for automatic assessment of programming assignments
Abstract
This paper presents a systematic literature review of the recent (2006--2010) development of automatic assessment tools for programming exercises. We discuss the major features that the tools support and the different approaches they are using both from the pedagogical and the technical point of view. Examples of these features are ways for the teacher to define tests, resubmission policies, security issues, and so forth. We have also identified a list of novel features, like assessing web software, that are likely to get more research attention in the future. As a conclusion, we state that too many new systems are developed, but also acknowledge the current reasons for the phenomenon. As one solution we encourage opening up the existing systems and joining efforts on developing those further. Selected systems from our survey are briefly described in Appendix A.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Teaching for Quality Learning at University: What the Student Does
BIGGS, J. B., & Tang, C. S. (1999) · Cited by 3,374
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.07.009 · Cited by 2,394 · cited 1×
“This means that an explicit procedure in selecting the systems and papers was applied (see [7] for details): to be included, a paper must have presented an AA system providing summative, numerical feedback from programming assignments or described results from using such system.”
MethodThe Delphi technique as a forecasting tool: issues and analysis
Rowe, G., & Wright, G. (1999) · International Journal of Forecasting · doi:10.1016/s0169-2070(99)00018-7 · Cited by 1,789
A Survey of Automated Assessment Approaches for Programming Assignments
Ala-Mutka, K. M. (2005) · Computer Science Education · doi:10.1080/08993400500150747 · Cited by 546
Tango: a framework and system for algorithm animation
Stasko, J. T. (1990) · Computer · doi:10.1109/2.58216 · Cited by 506
Alice: a 3-D tool for introductory programming concepts
Cooper, S., Dann, W. P., & Pausch, R. F. (2000) · Journal of computing sciences in colleges · Cited by 489
Automatic test-based assessment of programming: A review
Douce, C., Livingstone, D., & Orwell, J. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163409 · Cited by 416
Karel the Robot: A Gentle Introduction to the Art of Programming
Pattis, R. E., Roberts, J., & Stehlik, M. (1994) · Cumulative Index of Computer Aided Architectural Design · Cited by 353
The boss online submission and assessment system
Joy, M., Griffiths, N., & Boyatt, R. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163407 · Cited by 282
Web-CAT: automatically grading programming assignments
Edwards, S. H., & Perez-Quinones, M. A. (2008) · Proceedings of the 13th annual conference on Innovation and technology in computer science education · doi:10.1145/1384271.1384371 · Cited by 256
Visualizing programs with Jeliot 3
Moreno, A., Myller, N., Sutinen, E., & Ben-Ari, M. (2004) · Proceedings of the working conference on Advanced visual interfaces · doi:10.1145/989863.989928 · Cited by 256
Grading student programs using ASSYST
Jackson, D., & Usher, M. (1997) · Proceedings of the twenty-eighth SIGCSE technical symposium on Computer science education · doi:10.1145/268084.268210 · Cited by 253
Spacco, J., Hovemeyer, D., Pugh, W., Emad, F., Hollingsworth, J. K., & Padua-Perez, N. (2006) · ACM SIGCSE Bulletin · doi:10.1145/1140123.1140131 · Cited by 163
Analysis of research into the teaching and learning of programming
Sheard, J., Simon, S., Hamilton, M., & Lönnberg, J. (2009) · Proceedings of the fifth international workshop on Computing education research workshop · doi:10.1145/1584322.1584334 · Cited by 161
CS educational research: a meta-analysis of SIGCSE technical symposium proceedings
Valentine, D. W. (2004) · ACM SIGCSE Bulletin · doi:10.1145/1028174.971391 · Cited by 161
ANIMAL: A System for Supporting Multiple Roles in Algorithm Animation
Rößling, G., & Freisleben, B. (2002) · Journal of Visual Languages & Computing · doi:10.1006/jvlc.2002.0239 · Cited by 157
Individualized exercises for self-assessment of programming knowledge: An evaluation of QuizPACK
Brusilovsky, P. L., & Александрович, С. С. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163411 · Cited by 135
Carter, J., Ala-Mutka, K., Fuller, U., Dick, M., English, J., Fone, W., & Sheard, J. (2003) · ACM SIGCSE Bulletin · doi:10.1145/960492.960539 · Cited by 109
Constructing a core literature for computing education research
Pears, A., Seidman, S. B., Eney, C., Kinnunen, P., & Malmi, L. (2005) · ACM SIGCSE Bulletin · doi:10.1145/1113847.1113893 · Cited by 108
CS educational research: a meta-analysis of SIGCSE technical symposium proceedings
Valentine, D. W. (2004) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/971300.971391 · Cited by 105
Affective and behavioral predictors of novice programmer achievement
Rodrigo, M. M. T., Baker, R. S., Jadud, M. C., Amarra, A. C. M., Dy, T., Espejo-Lahoz, M. B. V., Lim, S. A. L., Pascua, S. A. M. S., Sugay, J. O., & Tabanao, E. S. (2009) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1562877.1562929 · Cited by 103
Enhancing learning management systems to better support computer science education
Rößling, G., Joy, M. S., Moreno, A., Radenski, A., Malmi, L., Kerren, A., Naps, T. L., Ross, R. J., Clancy, M. J., Korhonen, A., et al. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1473195.1473239 · Cited by 95
Malmi, L., Karavirta, V., Korhonen, A., & Nikander, J. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163412 · Cited by 90
Self-citationA web-based programming learning environment to support cognitive development
Hwang, W., Wang, C., Hwang, G., Huang, Y., & Huang, S. (2008) · Interacting with Computers · doi:10.1016/j.intcom.2008.07.002 · Cited by 87
Semantic similarity-based grading of student programs
Wang, T., Su, X., Wang, Y., & Ma, P. (2006) · Information and Software Technology · doi:10.1016/j.infsof.2006.03.001 · Cited by 86
RoboCode & problem-based learning: a non-prescriptive approach to teaching programming
O'Kelly, J., & Gibson, J. P. (2006) · ACM SIGCSE Bulletin · doi:10.1145/1140123.1140182 · Cited by 79
Marking student programs using graph similarity
Naudé, K. A., Greyling, J., & Vogts, D. (2009) · Computers & Education · doi:10.1016/j.compedu.2009.09.005 · Cited by 68
Interface-based programming assignments and automatic grading of java programs
Helmick, M. T. (2007) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1268784.1268805 · Cited by 65
A new Moodle module supporting automatic verification of VHDL-based assignments
Gutiérrez, E., Trenas, M. A., Ramos, J., Corbera, F., & Romero, S. (2010) · Comput. Educ. · doi:10.1016/j.compedu.2009.09.006 · Cited by 62
An experimental study of the impact of visual semantic feedback on novice programming
Hundhausen, C. D., & Brown, J. L. (2006) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2006.09.001 · Cited by 58
Suleman, H. (2008) · doi:10.1145/1456659.1456686 · Cited by 53
EduComponents: experiences in e-assessment in computer science education
Amelung, M., Piotrowski, M., & Rösner, D. (2006) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1140124.1140150 · Cited by 50
Computer Science Education in the 21st Century
Greening, T. (2000) · Springer eBooks · doi:10.1007/978-1-4612-1298-0 · Cited by 47
A Visualization-Based Computer Science Hypertextbook Prototype
Rößling, G., & Vellaramkalayil, T. (2009) · ACM Transactions on Computing Education · doi:10.1145/1538234.1538238 · Cited by 47
A course on algorithms and data structures using on-line judging
García-Mateos, G., & Fernández‐Alemán, J. L. (2009) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1562877.1562897 · Cited by 41
Experiences using test-driven development with an automated grader
Edwards, S. H., & Pérez-Quiñones, M. A. (2007) · Journal of computing sciences in colleges · doi:10.5555/1181849.1181855 · Cited by 39
The Recent Development of Automated Programming Assessment
Liang, Y., Liu, Q., Xu, J., & Wang, D. (2009) · International Conference on Computational Intelligence and Software Engineering (CiSE) · doi:10.1109/cise.2009.5365307 · Cited by 39
RoboCode & problem-based learning: a non-prescriptive approach to teaching programming
O'Kelly, J., & Gibson, J. P. (2006) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1140124.1140182 · Cited by 38
Søndergaard, H. (2009) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1562877.1562893 · Cited by 33
Amelung, M., Piotrowski, M., & Rösner, D. (2006) · ACM SIGCSE Bulletin · doi:10.1145/1140123.1140150 · Cited by 26
Providing Accurate And Timely Feedback By Automatically Grading Student Programming Labs.
Nordquist, P. (2007) · Journal of computing sciences in colleges · doi:10.5555/1292428.1292432 · Cited by 24
de RAADT, M., Dekeyser, S., & Lee, T. Y. (2006) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1315803.1315821 · Cited by 24
Gotel, O., Scharff, C., & Wildenberg, A. (2008) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1384271.1384329 · Cited by 19
A course on algorithms and data structures using on-line judging
García-Mateos, G., & Fernández‐Alemán, J. L. (2009) · ACM SIGCSE Bulletin · doi:10.1145/1595496.1562897 · Cited by 16
Work in progress-improving feedback using an automatic assessment tool
Jiménez-González, D., Álvarez, C., López, D., et al. (2008) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2008.4720591 · Cited by 11
Sant, J. A. (2009) · doi:10.1145/1562877.1562971 · Cited by 10
Gotel, O., Scharff, C., & Wildenberg, A. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1597849.1384329 · Cited by 8
Cited by
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
“One common approach for providing feedback in introductory programming courses is the use of automated assessment systems [2, 35, 60], which at the minimum provide feedback on the correctness of programming assignments submitted for evaluation.”
BackgroundIntroductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., Paterson, J., Scott, M. J., Sheard, J., et al. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295779 · Cited by 485
Automatic feedback in online learning environments: A systematic literature review
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100027 · Cited by 440
A systematic literature review of automated feedback generation for programming exercises
Keuning, H., Jeuring, J., & Heeren, B. (2019) · ACM Transactions on Computing Education · doi:10.1145/3231711 · Cited by 370
Automated Grading and Feedback Tools for Programming Education: A Systematic Review
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2024) · ACM Transactions on Computing Education · doi:10.1145/3636515 · Cited by 262
Metadata
- Authors
- Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O.
- Venue
- Proceedings of the 10th Koli Calling International Conference on Computing Education Research
- Publisher
- ACM
- Published
- 2010-10-28
- Pages
- 86-93
- OpenAlex
- W2151828850
- Cited by
- 554
- References
- 48
- FWCI
- 14.53
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.