Synthesis and analysis of automatic assessment methods in CS1: generating intelligent MCQs
The DOI matches its Crossref record; 19 of the 20 references it cites were found in a registry or an index.
Abstract
This paper describes the use of random code generation and mutation as a method for synthesising multiple choice questions which can be used in automated assessment. Whilst using multiple choice questions has proved to be a feasible method of testing if students have suitable knowledge or comprehension of a programming concept, creating suitable multiple choice questions that accurately test the students' knowledge is time intensive.This paper proposes two methods of generating code which can then be used to closely examine the comprehension ability of students. The first method takes as input a suite of template programs, and performs slight mutations on each program and ask students to comprehend the new program. The second method performs traversals on a syntax tree of possible programs, yielding slightly erratic but compilable code, again with behaviour that students can be questioned about. As well as generating code these methods also yield alternative distracting answers to challenge the students. Finally, this paper discusses the gradual introduction of these automatically generated questions as an assessment method and discusses the relative merits of each technique.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We could not find papers that extend Traynor and Gibson's early attempt [54] to automatically generate MCQs for CS1, as it appears the focus rather shifted towards approaches based on learner-sourcing [8].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [54]
Citations
Cites
The Taxonomy of Educational Objectives
(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_6027 · Cited by 4,103
Psychology Applied to Teaching
Snowman, J., Dobozy, E., Scevak, J., Bryer, F. K., Bartlett, B. J., & Biehler, R. F. (1971) · Cited by 771
Grammatical evolution: Evolving programs for an arbitrary language
Ryan, C., Collins, J., & Neill, M. O. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0055930 · Cited by 733
Some Difficulties of Learning to Program
Du Boulay, B. (1986) · Journal of Educational Computing Research · doi:10.2190/3lfx-9rrf-67t8-uvk9 · Cited by 648
McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · Working group reports from ITiCSE on Innovation and technology in computer science education · doi:10.1145/572133.572137 · Cited by 626
Studying the Novice Programmer
(2013) · Psychology Press eBooks · doi:10.4324/9781315808321 · Cited by 489
McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · ACM SIGCSE Bulletin · doi:10.1145/572139.572181 · Cited by 424
Novice mistakes: are the folk wisdoms correct?
Spohrer, J. C., & Soloway, E. (1986) · Communications of the ACM · doi:10.1145/6138.6145 · Cited by 310
Identifying and correcting Java programming errors for introductory computer science students
Hristova, M., Misra, A., Rutter, M., & Mercuri, R. (2003) · Proceedings of the 34th SIGCSE technical symposium on Computer science education · doi:10.1145/611892.611956 · Cited by 201
Introductory programming, criterion-referencing, and bloom
Lister, R., & Leaney, J. (2003) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/611892.611954 · Cited by 121
On blooming first year programming, and its blooming assessment
Lister, R. (2000) · Proceedings of the Australasian conference on Computing education · doi:10.1145/359369.359393 · Cited by 109
A Summary of Misconceptions of High School Basic Programmers
Putnam, R. T., Sleeman, D. H., Baxter, J. A., & Kuspa, L. K. (1986) · Journal of Educational Computing Research · doi:10.2190/fgn9-dj2f-86v8-3fau · Cited by 100
Assessing the assessment of programming ability
Daly, C., & Waldron, J. (2004) · ACM SIGCSE Bulletin · doi:10.1145/1028174.971375 · Cited by 92
Fleury, A. E. (2000) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/330908.331854 · Cited by 89
Objectives and objective assessment in CS1
Lister, R. (2001) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/364447.364605 · Cited by 51
Psychology Applied To Teaching
BIEHLER, R. F. (1969) · Andalas University Repository (Andalas University) · Cited by 42
Assessing the assessment of programming ability
Daly, C., & Waldron, J. (2004) · Proceedings of the 35th SIGCSE technical symposium on Computer science education · doi:10.1145/971300.971375 · Cited by 38
Managing large class assessment
Rhodes, A., Bower, K., & Bancroft, P. (2004) · Australasian Computing Education Conference · Cited by 10
Using genetic programming for the induction of novice procedural programming solution algorithms
Pillay, N. (2002) · ACM Symposium on Applied Computing · doi:10.1145/508791.508903 · Cited by 8
Research works printed but not linked: 2
- 10.5555/646806.706289 · doi:10.5555/646806.706289
- R. C. Sprinthall , N. A. Sprinthall , and S. N. Oja . Educational Psycholgy . McGraw-Hill Education , 7 th edition, 1998 . R. C. Sprinthall, N. A. Sprinthall, and S. N. Oja. Educational Psycholgy. McGraw-Hill Education, 7th edition, 1998.
Cited by
Applying Data Mining Techniques to e-Learning Problems
Castro, F., Vellido, A., Nebot, À., & Mugica, F. (2007) · Studies in Computational Intelligence · doi:10.1007/978-3-540-71974-8_8 · Cited by 276
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
Jovanović, M., Vukičević, M., Milovanović, M., & Minović, M. (2012) · International Journal of Computational Intelligence Systems · doi:10.1080/18756891.2012.696923 · Cited by 85
Evaluation of e-learning systems based on fuzzy clustering models and statistical tools
Hogo, M. A. (2010) · Expert Systems with Applications · doi:10.1016/j.eswa.2010.03.032 · Cited by 84
Adaptive assessment of student's knowledge in programming courses
Chatzopoulou, D. I., & Economides, A. A. (2010) · Journal of Computer Assisted Learning · doi:10.1111/j.1365-2729.2010.00363.x · Cited by 57
Metadata
- Authors
- Traynor, D., & Gibson, J. P.
- Affiliations
- National University of Ireland, Maynooth (IE)
- ORCID
- Gibson, J. Paul
- Venue
- ACM SIGCSE Bulletin
- Published
- 2005-02-23
- Volume
- 37
- Issue
- 1
- Pages
- 495-499
- ISSN
- 0097-8418 (ISSN-L)
- OpenAlex
- W2014989589
- Cited by
- 21
- References
- 20
- Citations by year
- 2024: 2 · 2022: 1 · 2019: 1 · 2016: 4 · 2013: 1 · 2012: 2
- FWCI
- 2.52
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGCSE Bulletin (publishedVersion)
- Funders
- National University of Ireland, Maynooth
- Irish Research Council
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- automated assessment, CS1, multiple-choice questions, mutation testing, program comprehension, distractor generation, introductory programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.