CoChat

Synthesis and analysis of automatic assessment methods in CS1: generating intelligent MCQs

Traynor, Des, Gibson, J. Paul · 2005 · ACM SIGCSE Bulletin

The DOI matches its Crossref record; 19 of the 20 references it cites were found in a registry or an index.

Abstract

This paper describes the use of random code generation and mutation as a method for synthesising multiple choice questions which can be used in automated assessment. Whilst using multiple choice questions has proved to be a feasible method of testing if students have suitable knowledge or comprehension of a programming concept, creating suitable multiple choice questions that accurately test the students' knowledge is time intensive.This paper proposes two methods of generating code which can then be used to closely examine the comprehension ability of students. The first method takes as input a suite of template programs, and performs slight mutations on each program and ask students to comprehend the new program. The second method performs traversals on a syntax tree of possible programs, yielding slightly erratic but compilable code, again with behaviour that students can be questioned about. As well as generating code these methods also yield alternative distracting answers to challenge the students. Finally, this paper discusses the gradual introduction of these automatically generated questions as an assessment method and discusses the relative merits of each technique.

Overview

Integrity signals

Registry

Crossref · 10.1145/1047124.1047502

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 2

  1. 10.5555/646806.706289 · doi:10.5555/646806.706289
  2. R. C. Sprinthall , N. A. Sprinthall , and S. N. Oja . Educational Psycholgy . McGraw-Hill Education , 7 th edition, 1998 . R. C. Sprinthall, N. A. Sprinthall, and S. N. Oja. Educational Psycholgy. McGraw-Hill Education, 7th edition, 1998.

Cited by

See all 21

Metadata

Authors
Traynor, D., & Gibson, J. P.
Affiliations
National University of Ireland, Maynooth (IE)
Venue
ACM SIGCSE Bulletin
Published
2005-02-23
Volume
37
Issue
1
Pages
495-499
ISSN
0097-8418 (ISSN-L)
OpenAlex
W2014989589
Cited by
21
References
20
Citations by year
2024: 2 · 2022: 1 · 2019: 1 · 2016: 4 · 2013: 1 · 2012: 2
FWCI
2.52
Type
journal article
Language
en
Open access
closed
Locations
ACM SIGCSE Bulletin (publishedVersion)
Funders
  • National University of Ireland, Maynooth
  • Irish Research Council
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
automated assessment, CS1, multiple-choice questions, mutation testing, program comprehension, distractor generation, introductory programming

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.