Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 546

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

A Survey of Automated Assessment Approaches for Programming Assignments

Ala-Mutka, Kirsti M·2005·Computer Science Education
DOI 10.1080/08993400500150747
Open this paper in CoChat — free account

Abstract

Practical programming is one of the basic skills pursued in computer science education. On programming courses, the coursework consists of programming assignments that need to be assessed from different points of view. Since the submitted assignments are executable programs with a formal structure, some features can be assessed automatically. The basic requirement for automated assessment is the numerical measurability of assessment targets, but semiautomatic approaches can overcome this restriction. Recognizing automatically assessable features can help teachers to create educational models, where automatic tools let teachers concentrate their work on the learning issues that need student-teacher interaction the most. Several automatic tools for both static and dynamic assessment of computer programs have been reported in the literature. This article promotes these issues by surveying several automatic approaches for assessing programming assignments. Not all the existing tools will be covered, simply because of the vast number of them. The article concentrates on bringing forward different assessment techniques and approaches to give an interested reader starting points for finding further information in the area. Automatic assessment tools can be used to help teachers in grading tasks as well as to support students' working process with automatic feedback. Common advantages of automation are the speed, availability, consistency and objectivity of assessment. However, automatic tools emphasize the need for careful pedagogical design of the assignment and assessment settings. To effectively share the knowledge and good assessment solutions already developed, better interoperability and portability of the tools is needed.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1080/08993400500150747

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Research works printed but not linked: 55

  1. Ala-Mutka K Järvinen H-M 2004 Assessment Process for Programming Assignments Proceedings of the 4th IEEE International Conference on Advanced Learning Technologies , Finland 181 185 · DOI
  2. 10.3923/itj.2004.245.269 · DOI
  3. 10.1016/S0164-1212(01)00061-9 · DOI
  4. Arnow D, 1999, Proceedings of the 29th ASEE/IEEE Frontiers in Education Conference, 12A9/39 – 12A9/
  5. Becker K 2003 Grading programming assignments using rubrics In Proceedings of the 8th annual conference on Innovation and technology in computer science education Greece, 253 · DOI
  6. Bettini L Crescenzi P Innocenti G Loreti M Cecchi L 2004 An Environment for Self-Assessing Java Programming Skills in Undergraduate First Programming Courses In Proceedings of the 4th IEEE International Conference on Advanced Learning Technologies Finland, 161 – 165 · doi:10.1109/icalt.2004.1357395
  7. Bloom BS (Ed.) 1956 Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook I. Cognitive Domain New York: David McKay Company, Inc
  8. 10.1145/960492.960539 · DOI
  9. 10.1016/S0360-1315(03)00030-7 · DOI
  10. 10.1109/TE.2004.825220 · doi:10.1109/te.2004.825220
  11. 10.1016/S0360-1315(97)00054-7 · doi:10.1016/s0360-1315(97)00054-7
  12. 10.1109/32.345830 · DOI · Paper page
  13. 10.1145/1029994.1029995 · DOI
  14. Ellsworth C Fenwick J Kurtz B 2004 The Quiver System In Proceedings of the 35th SIGCSE technical symposium on Computer Science Education US, 205 – 209 · doi:10.1145/971300.971374
  15. English J 2004 Automatic Assessment of GUI Programs using JEWL In Proceedings of 9th annual conference on Innovation and technology in computer science education UK, 137 – 141 · doi:10.1145/1007996.1008033
  16. English J 2002 Experience with a computer-assisted formal programming examination In Proceedings of 7th annual conference on Innovation and technology in computer science education Denmark, 51 – 54 · doi:10.1145/544414.544432
  17. Halstead MH 1977 Elements of Software Science New York: Elsevier North-Holland
  18. Hansen H Ruuska M 2003 Assessing time-efficiency in a course on data structures and algorithms In Proceedings of the 3rd Annual Finnish/Baltic Sea Conference on Computer Science Education , Finland
  19. 10.1023/A:1026364126982 · doi:10.1023/a:1026364126982
  20. Higgins C Symeonidis P Tsinsifas A 2002 Diagram-based CBA using DATsys and CourseMaster In Proceedings of the International Conference on Computers in Education 367 372
  21. 10.1145/782941.782978 · doi:10.1145/782941.782978
  22. 10.1016/j.ijhcs.2004.02.001 · DOI
  23. 10.1016/0360-1315(93)90086-X · doi:10.1016/0360-1315(93)90086-x
  24. Jackson D 2000 A semi-automated approach to online assessment Proceedings of 5th annual conference on Innovation and technology in computer science education , Finland 164 167 · DOI
  25. Jackson D Usher M 1997 Grading Student Programs using ASSYST Proceedings of the 28th SIGCSE technical symposium on Computer science education , USA 335 339 · DOI
  26. Joy M Griffiths N 2004 Online Submission of Coursework – a Technological Perspective In Proceedings of the 4th IEEE International Conference on Advanced Learning Technologies Finland, 430 – 434 · doi:10.1109/icalt.2004.1357451
  27. Kay D Scott T Isaacson P Reek K 1994 Automated grading assistance for student programs In Proceedings of the 25th SIGCSE technical symposium on Computer science education USA, 381 – 382 · doi:10.1145/191033.191184
  28. Korhonen A Malmi L 2000 Algorithm simulation with automatic assessment Proceedings of 5th annual conference on Innovation and technology in computer science education , Finland 160 163 · DOI
  29. Lister R Leaney J 2003 First Year Programming: Let All the Flowers Bloom In T. Greening & R. Lister (Eds.), Computing Education 2003 Fifth Australasian Computing Education Conference (pp. 221 – 230)
  30. 10.1002/(SICI)1097-024X(19990710)29:8<721::AID-SPE257>3.0.CO;2-0 · doi:10.1002/(sici)1097-024x(19990710)29:8<721::aid-spe257>3.0.co;2-0
  31. 10.1145/572139.572181 · DOI · Paper page
  32. 10.1076/0899-3408(200008)10:2;1-C;FT147 · DOI
  33. 10.1109/TSE.1976.233837 · DOI · Paper page
  34. Mengel SA Ulans JV 1999 A case study of the analysis of the quality of novice students programs In Proceedings of the 12th Conference on Software Engineering Education and Training 40 49 · doi:10.1109/csee.1999.755178
  35. Michaelson G 1996 Automatic Analysis of Functional Program Style In Proceedings of Australian Software Engineering Conference 38 46 · doi:10.1109/aswec.1996.534121
  36. Morris D 2003 Automatic Grading of Student's Programming Assignments: An Interactive Process and Suite of Programs In Proceedings of the 33rd ASEE/IEEE Frontiers in Education Conference S3F-1 – S3F-5 · DOI
  37. MOSS. [Computer software] Retrieved November 15, 2004, from http://www.cs.berkeley.edu/∼aiken/moss.html · doi:10.1016/b978-075067740-0/50002-0
  38. Olson DM 1988 The reliability of analytic and holistic methods in rating students' computer programs In Proceedings of the 19th SIGCSE technical symposium on Computer science education USA, 293 – 298 · doi:10.1145/52965.53037
  39. Oman PW Cook CR 1990 A taxonomy for programming style In Proceedings of the 1990 ACM Annual Conference on Cooperation 244 250 · DOI
  40. 10.1145/782941.782986 · DOI
  41. Prechelt L, 2002, Journal of Universal Computer Science, 8, 1016
  42. 10.1145/857076.857090 · doi:10.1145/857076.857090
  43. Reek KA 1989 The TRY system -or- how to avoid testing student programs In Proceedings of the 20th SIGCSE technical symposium on Computer science education USA, 112 – 116 · doi:10.1145/65294.71198
  44. 10.1145/948086.948088 · DOI
  45. Ruehr F, 2002, Journal of Computing in Small Colleges, 18, 65
  46. Saikkonen R Malmi L Korhonen A 2001 Fully automatic assessment of programming exercises Proceedings of 6th annual conference on Innovation and technology in computer science education , UK 133 136 · DOI
  47. Schorsch T 1995 CAP: an automated self-assessment tool to check Pascal programs for syntax, logic and style errors In Proceedings of the 26th SIGCSE technical symposium on Computer science education 168 172 · DOI
  48. Sitthiworachart J Joy M 2003 Effective Peer Assessment for Learning Computer Programming Proceedings of the 8th annual conference on Innovation and technology in computer science education , UK 122 126 · DOI
  49. 10.1016/S0360-1315(97)00021-3 · DOI
  50. Truong N Roe P Bancroft P 2003 A Web Based Environment for Learning to Program In Proceedings of the 25th Australasian Computer Science Conference Australia, 255 – 264
  51. Truong N Roe P Bancroft P 2004 Static Analysis of Students' Java Programs In Proceedings of the sixth Australian Computing Education Conference New Zealand, 317 – 325
  52. 10.1093/comjnl/39.9.741 · DOI
  53. Woit D Mason D 2003 Effectiveness of online assessment In Proceedings of the 34th SIGCSE technical symposium on Computer science education USA, 137 – 141 · DOI
  54. Xie Y Engler D 2002 Using Redundancies to Find Errors In Proceedings of SIGSOFT 2002/FSE-10 USA, 51 – 60 · doi:10.1145/587051.587060
  55. Zeller A 2000 Making students read and review code In Proceedings of 5th annual conference on Innovation and technology in computer science education Finland, 89 – 92 · doi:10.1145/343048.343090

Cited by

See all 546

Metadata

Authors
Ala-Mutka, K. M.
Venue
Computer Science Education
Publisher
Informa UK Limited
Published
2005-06
Volume
15
Issue
2
Pages
83-102
ISSN
  • 0899-3408 (ISSN-L)
  • 1744-5175
DOI
10.1080/08993400500150747
OpenAlex
W2070405152
Cited by
546
References
2
FWCI
18.22
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Survey of Automated Assessment Approaches for Programming Assignments. https://app.cochat.ai/papers/10.1080/08993400500150747
The paper
Ala-Mutka, K. M. (2005). A Survey of Automated Assessment Approaches for Programming Assignments. Computer Science Education. https://doi.org/10.1080/08993400500150747

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.