CoChat
Sign Up
Cited by 52Academic integrity and plagiarismArtificial Intelligence in EducationOnline Learning and Analytics

Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment

Open access
Richards, Mike, Waugh, Kevin, Slaymaker, Mark, Petre, Marian, Woodthorpe, John, Gooch, Daniel·2024·ACM Transactions on Computing Education
DOI 10.1145/3633287
Open this paper in CoChat — free accountOpen PDF

Abstract

Cheating has been a long-standing issue in university assessments. However, the release of ChatGPT and other free-to-use generative AI tools has provided a new and distinct method for cheating. Students can run many assessment questions through the tool and generate a superficially compelling answer, which may or may not be accurate. We ran a dual-anonymous “quality assurance” marking exercise across four end-of-module assessments across a distance university computer science (CS) curriculum. Each marker received five ChatGPT-generated scripts alongside 10 student scripts. A total of 90 scripts were marked; every ChatGPT-generated script for the undergraduate modules received at least a passing grade (>40%), with all of the introductory module CS1 scripts receiving a distinction (>85%). None of the ChatGPT-taught postgraduate scripts received a passing grade (>50%). We also present the results of interviewing the markers and of running our sample scripts through a GPT-2 detector and the TurnItIn AI detector, which both identified every ChatGPT-generated script but differed in the number of false positives. As such, we contribute a baseline understanding of how the public release of generative AI is likely to significantly impact quality assurance processes. Our analysis demonstrates that in most cases, across a range of question formats, topics, and study levels, ChatGPT is at least capable of producing adequate answers for undergraduate assessment.

AI snapshot

Written from the paper’s full text.

TL;DR

In a dual-anonymous marking exercise across four Open University computing end-of-module assessments, ChatGPT (GPT-3.5) generated scripts that passed all undergraduate modules, with all CS1 scripts receiving distinction, but failed all postgraduate CS4 scripts. Markers often noticed suspicious material but rarely formally flagged it, while TurnItIn AI detection identified all ChatGPT scripts with 0% false positives and GPT-2 had 0%–50% false positives.

Key findings

  • Across 90 marked scripts (60 distinct: 40 student, 20 ChatGPT; 30 double-marked), every ChatGPT-generated script for undergraduate CS1, CS2, and CS3 received at least a passing grade (>40%); all CS1 ChatGPT scripts received a distinction (>85%); none of the postgraduate CS4 ChatGPT scripts received a passing grade (>50%).
  • Question-level analysis found the only question type consistently receiving failing marks (<40%) was PDP and reflection-based questions; CS4 failed due to superficial and generic answers.
  • Markers’ awareness of unusual or unexpected material was impressive, but formal flagging was extremely selective; an additional nine scripts were identified at interview as containing unusual or unexpected material, and two CS2 markers between them flagged all five ChatGPT scripts for that module.
  • GPT-2 detection identified every ChatGPT-generated script (>0.2) but had a 0%–50% false-positive rate among student scripts; TurnItIn AI detection had a mean AI-origin score of 0.25% for student scripts (SD 1.17%, max 8%) versus 74.43% for ChatGPT scripts (SD 25.8%, min 28%), with 100% detection and 0% false positives. All ChatGPT scripts had TurnItIn plagiarism scores below 0.21 (mean = 0.079).

Methods

  • Dual-anonymous quality assurance marking exercise across four modules (CS1, CS2, CS3 undergraduate; CS4 postgraduate). Each marker received 10 student scripts and 5 ChatGPT-generated scripts; CS2 and CS3 were independently double marked, yielding 90 marked scripts with 30 double marked.
  • Student scripts were randomly selected from the 21J/K cohort, which completed before ChatGPT’s public release. ChatGPT scripts were generated using GPT-3.5 from original assessment questions in a single thread, copied into Word templates, stripped of ChatGPT artefacts, and all scripts were anonymized with metadata removed. Mean generation times were CS1 23 min, CS2 15 min, CS3 32 min, CS4 25 min.
  • Analysis used descriptive statistics and graphing of marks; semi-structured interviews with five of six markers were transcribed and inductively open-coded. Scripts were also run through a GPT-2 detector and TurnItIn AI detection, with a >0.2 threshold for suspecting AI origin.

Limitations

  • Stated: four of five interviewed markers were aware of LLMs but had not used them; the study was small-scale and generated insufficient ChatGPT scripts per question to find a saturation/repetition point; scripts represent the “laziest” cheating (copy-paste without rewording or supplementing); only four modules were used, with advanced undergraduate and postgraduate levels each represented by a single module; data protection limited detection to dated compliant tools; findings are a snapshot of one LLM (GPT-3.5); replication and publication-speed challenges are noted.
  • Stated: the CS4 student sample was unrepresentative of the wider cohort, preventing broader comparisons; CS3 showed a significant consistent discrepancy between its two markers; there were no exact matches among ChatGPT-generated scripts.
  • Evident from the text: the paper acknowledges no absolute guarantee that student scripts were human-written; prompts cannot be released; detection tools cannot account for student rewriting, paraphrasing, or integrating ChatGPT text with their own material; the GPT-2 detector is older and likely less successful than a dedicated GPT-3 detector.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3633287

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 43 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    All ChatGPT-generated scripts for the introductory module CS1 received a distinction (>85%).

    “with all of the introductory module CS1 scripts receiving a distinction (>85%)”
    Quote in the paperNot a substantial claim
  • Findingown data

    Every ChatGPT-generated script for the undergraduate modules received at least a passing grade (>40%).

    “every ChatGPT-generated script for the undergraduate modules received at least a passing grade (>40%)”
    Quote in the paperSubstantial claim
  • Findingown data

    None of the ChatGPT-generated scripts for the taught postgraduate module CS4 received a passing grade (>50%).

    “None of the ChatGPT taught postgraduate scripts received a passing grade (>50%)”
    Quote in the paperSubstantial claim
  • Findingown data

    There was only one occasion on which script generation took more than an hour.

    “There was only one occasion on which script generation took more than an hour”
    Quote in the paperNot a substantial claim
  • Findingown data

    For CS3, the third part of the long-form scenario question (redesign of an interface) received passing or lower marks.

    “The third and final part, requiring a redesign of an interface building on the heuristic evaluation, received passing or lower marks.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 60
  • Classified as probably not a substantial claim. 68
  • The quote only partly states this claim. 2

  • May combine two claims.

    Findingown data

    Both the GPT-2 detector and the TurnItIn AI detector identified every ChatGPT-generated script, but differed in the number of false positives.

    “running our sample scripts through a GPT-2 detector and the TurnItIn AI detector which both identified every ChatGPT-generated script, but differed in the number of false-positives.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Markers could identify scripts containing unusual or unexpected material, but their formal flagging of scripts was extremely selective.

    “Markers could identify scripts containing unusual or unexpected material, although their formal flagging of scripts was extremely selective.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All ChatGPT-generated scripts produced TurnItIn plagiarism scores lower than 0.21, with a mean of 0.079.

    “All ChatGPT-generated scripts produced TurnItIn scores lower than 0.21 (mean = 0.079)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For CS1, ChatGPT-generated scripts outperformed both the cohort and student sample; for CS2 and CS3, ChatGPT underperformed relative to both.

    “For CS1, the ChatGPT-generated scripts outperformed both the cohort and student sample; whilst ChatGPT underperformed relative to both for CS2 and CS3.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For CS1, CS2, and CS3, the mean and standard deviations for marks of ChatGPT-generated and sample student scripts align broadly with the wider cohort mean, lying within one standard deviation.

    “Table 6 shows that for CS1, CS2 and CS3 the mean and standard deviations for marks align broadly with mean marks for the wider cohort with mean ChatGPT-generated and sample student script scores lying within one standard deviation of the cohort mean.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For CS1, questions on program development improvements and SQL usage/vulnerabilities received distinction marks.

    “For CS1, questions requiring a discussion of program development improvements and definition questions regarding the usage/vulnerabilities of SQL all received a distinction mark.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For CS1, the lowest scoring questions (passing marks) assessed programming and program development, an essay on social, legal, and ethical issues around digital literacy, and reflecting on module performance.

    “The lowest scoring questionsreceiving passing marksassessed programming and program development; an essay on social, legal, and ethical issues around digital literacy; and reflecting on module performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For CS3, all 15 short-form questions on applying HCI concepts broadly received passing or higher marks from one marker, and majority distinction or higher from the other.

    “That said, all 15 of the short-form questions requiring students to apply their understanding of a key HCI conceptfrom requirement gathering to design techniques and evaluationbroadly received passing or higher marks from one marker, and majority distinction or higher from the other maker.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Although the number of scripts flagged as suspected plagiarism was small, at interview markers' identification of ChatGPT-generated scripts was more accurate than the flags alone suggest.

    “Although the number of scripts flagged as suspected of plagiarism by the markers was small, at interview it was clear that their identification of ChatGPT-generated scripts was more accurate than the flags alone might suggest.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    CS4 is an outlier: ChatGPT performed poorly, and the student sample is unrepresentative of the wider CS4 cohort.

    “Once again, CS4 represents an outlier; not only did ChatGPT perform poorly in the assessment, but the student sample is unrepresentative of the wider CS4 cohort.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For CS2, PDP questions were mostly handled poorly: all ePortfolio responses failed, majority of self-reflection questions failed, future planning tended to receive passing marks.

    “The PDP questions were mostly handled poorly, with all the ePortoflio question responses failing; the majority of the self-reflection questions failing; and the future planning tending to receive passing marks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Markers M3 and M4 both highlighted that they tended to have an extremely high threshold for academic conduct, seeing such situations as an opportunity for teaching and developing students.

    “Markers M3 and M4 both highlighted that they tended to have an extremely high threshold for academic conduct, seeing such situations as an opportunity for teaching and developing students.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    Changes in style included differences in layout, tone or voice, technical vocabulary, proficiency of writing, and specificity of answers.

    “This could be differences in the layout of the document itself or changes in the use of language, such as: changes in tone or voice; the use of technical vocabulary; the proficiency of writing; or the specificity of the answers given”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    M1 noted that many stylistic flags were more pronounced in material requiring students to integrate external material.

    “M1 noted that many of the stylistic flags were more pronounced in material requiring students to integrate external material”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The two CS2 markers had between them flagged all five ChatGPT-generated scripts for that module; at interview, M2 increased identification from one flagged during marking to correctly identifying all five.

    “The two CS2 markers had, between them, flagged all five ChatGPTgenerated scripts for that module; at interview, M2 increased identification from one flagged during marking to correctly identifying all five.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    M6 had not flagged any scripts for plagiarism in marking feedback but immediately identified three ChatGPT-generated scripts and on reflection identified the two others.

    “M6 hadn t flagged any of the scripts for plagiarism in the marking feedback, but immediately identified three of the ChatGPT-generated scripts (1, 10, and 11) and on reflection identified the two others (5 and 15).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The most-mentioned flag was a change in style in answers, potentially indicative of different authors.

    “The most-mentioned flag was a change in style in answers, potentially indicative of different authors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

    Findingown data

    TurnItIn AI detection is far more successful at identifying ChatGPT-generated material than GPT-2.

    “Figure 3 shows that more recent detection toolssuch as TurnItIn AI detection, (trained on the GPT-3 and GPT-3.5 language models underpinning ChatGPT), are far more successful at identifying ChatGPT generated material.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    The version of ChatGPT used in this study typically generates false references; all references in ChatGPT-generated scripts were either artificial or drew on material in the question.

    “The version of ChatGPT used in this study typically generates false references [53]; the research team examined every reference provided in the ChatGPT-generated scripts and found that all of the references that were included were either artificial or drew on material in the question used in the original prompt.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [53]

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Both tools correctly identified all ChatGPT scripts, but GPT-2 produced a 0%-50% false-positive rate while TurnItIn recorded 0%.

    “While both tools correctly identified all of the ChatGPT scripts, the GPT-2 detector produced a 0%-50% false-positive rate amongst student scripts, with TurnItIn recording a 0% false-positive rate.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ChatGPT-generated answers either missed personal reflection or were written in the third person, talked about reflection in broad terms, and did not relate back to individual experience.

    “The answers provided are either missing that sense of personal reflection (as with CS1), or a ChatGPT-generated script provides an answer written in the third person, talks about reflection in broad terms, and doesn t relate back to an individual experience.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both M1 and M6 said they would follow references; M6 noted that if a link cannot be found, it is an automatic fail.

    “Both M1 and M6 said that they would follow references, with the CS4 tutor (M6) noting that based on the module material if you can't find a link to them, you can't identify them, then it's an automatic fail .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both M4 and M2 typically scanned the formatting and venue of references for correctness but did not check them for validity due to tight marking time.

    “Both M4 and M2 noted that they typically scanned the formatting and venue of the reference for correctness but didn t check them for validity as the time for marking is so tight.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Five CS1, zero CS2, four CS3, and two CS4 student scripts scored >0.2 on GPT-2 detection.

    “Five student-generated CS1 scripts scored >0.2, no student-generated CS2 scripts scored >0.2, four student-generated CS3 scripts scored >0.2 and two student-generated CS4 scripts scored >0.2.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Markers identified student behaviors that were baseline behaviors requiring study support, not flags for misconduct.

    “The markers also identified student behaviors that whilst not flags for misconduct were baseline behaviors requiring study support.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Current AL practice tends toward providing study skills support rather than activating disciplinary procedures, although this depended partially on module level.

    “Current AL practiceas noted previouslytends toward providing study skills support, rather than activating disciplinary procedures, although this did depend partially on the module level.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    M3 would follow references if they suspected plagiarism.

    “M3 was similar but noted that they would follow the references if they suspected plagiarism.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All ChatGPT-generated scripts scored greater than 0.2 on GPT-2 detection, indicating they are likely LLM-generated.

    “All of the ChatGPT-generated scripts score greater than 0.2 indicating they are likely to have been generated by an LLM.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    TurnItIn detection rate was 100% with a 0% false-positive rate.

    “The detection rate was 100%, with a 0% false-positive rate.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Four markers highlighted a general expectation that plagiarism would be detected either by central members of the module team or the university's anti-plagiarism software.

    “Four of the markers (M1; M2; M3; M4) highlighted a general expectation that plagiarism would be detected either by central members of the module team or the university s anti-plagiarism software.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Questions containing detailed background information, those listing specific points to be addressed in a script, and those providing templated answers are amenable to high-scoring ChatGPT-generated scripts.

    “Likewise, questions containing detailed background information; those listing specific points to be addressed in a script; and those providing templated answers are amenable to high-scoring ChatGPT-generated scripts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The markers' awareness of scripts containing unusual or unexpected material, and their articulation of the characteristics that prompted suspicion, were impressive, although their formal flagging of scripts was extremely selective.

    “In terms of detecting material generated by ChatGPT, our markers awareness of scripts containing unusual or unexpected material, and their articulation of the characteristics that prompted suspicion, were impressive, although their formal flagging of scripts was extremely selective.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The ChatGPT-generated scripts often had a similar structure and tone of voice to one another, but there were no exact matches.

    “Secondly, whilst the ChatGPT-generated scripts often had a similar structure and tone of voice to one another, there were no exact matches.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The software detection of LLM-generated content was successful at identifying all of the ChatGPT scripts, although the tools differed in their false-positive rate amongst student scripts.

    “The software detection of LLM-generated content was successful at identifying all of the ChatGPT scripts, although the tools differed in their false-positive rate amongst student scripts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    ChatGPT has reduced the time needed to produce scripts to a few minutes.

    “However, the deployment of ChatGPT provides a new and distinct method for cheating; it drives the financial cost to zero; greatly reduces the likelihood of being accused of plagiarizing content, and; has reduced the time needed to produce scripts to a few minutes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Across a range of question formats, topics and study levels, ChatGPT is at least capable of producing adequate answers for undergraduate assessment, and its average answers would pass most assessments used in this study.

    “Our analysis demonstrates that, in most cases, across a range of question formats, topics and study levels, ChatGPT is at least capable of producing adequate answers for undergraduate assessment.”
    • Also stated in: § 1 INTRODUCTION · hedged
      “Our results highlight that ChatGPT produces average answers that would pass most of the assessments used in this study.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The four selected modules are representative of the broader computing curriculum and assessment types.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    The selected modules use a range of assessment methods and require students to demonstrate a range of academic skills.

    “The selected modules use a range of assessment methods and require students to demonstrate a range of academic skills.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The version of ChatGPT studied (GPT-3.5) has no live Internet access and cannot provide information about recent events.

    “This version of ChatGPT has no live Internet access and cannot provide information about recent events”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2]

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The study used a dual-anonymous protocol in which markers were unaware of the source of the scripts and the purpose of the study was masked from them.

    “we designed a dual-anonymous study protocol focused on providing ChatGPT-generated and student scripts to markers who were asked to mark the work as part of a quality assurance study.The purpose of the study was masked from markers”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The GPT-2 detection software is likely less successful at identifying ChatGPT outputs than a dedicated GPT-3 detector.

    “We concede that this detection software is likely to be less successful at identifying ChatGPT outputs than a dedicated GPT-3 detector.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The study used two detection tools compliant with data protection regulations.

    “Therefore, we used two tools compliant with our data protection regulations.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Scripts were anonymized and checked by two members of the research team to remove identifying information.

    “Scripts were anonymized and checked by two members of the research team to remove any identifying information”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    CS2 and CS3 scripts were independently double marked.

    “CS1 and CS4 were marked by a single marker, with CS2 and CS3 scripts being independently double marked”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The order of scripts was randomized using a random number generator, with one pair of ChatGPT-generated scripts always given consecutive numbers.

    “Ordering was randomized using a random number generator, with the exception that, in any batch of scripts, one pair of ChatGPT-generated scripts was always given consecutive numbers”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Orenstrakh et al. conclude that while detectors achieve reasonable accuracy, they are still prone to flaws and are not yet suitable for dealing with academic integrity.

    “The authors conclude: that while detectors manage to achieve a reasonable accuracy, they are still prone to flaws and can be challenging to interpret and that they are not yet suitable for dealing with academic integrity.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [38] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Guidance to markers about feedback was deliberately vague to avoid alerting them to the research being performed.

    “Guidance to markers about this feedback was deliberately vague, to avoid alerting them to the research being performed”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The ChatGPT-generated scripts were scrutinized by a member of the research team to ensure removal of evidence that they had been created using ChatGPT.

    “The ChatGPT-generated scripts were then scrutinized by a member of the research team to ensure the removal of any evidence that they had been created using ChatGPT”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Naive ChatGPT-generated scripts are more likely to be identified as suspicious by markers and detection software alike.

    “with naïve scripts more likely to be identified as suspicious by markers and detection software alike.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    The cohort of student scripts was chosen because all grades had been finalized and appeals completed, and because it was the last full cohort to finish studies before the public release of ChatGPT.

    “This cohort was chosen not only because all grades had been finalized and any appeals processes completed; but because it was the last full cohort to finish studies before the public release of ChatGPT.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The study simulates a typical marking process rather than acting as a spot-the-AI activity.

    “therefore our study simulates a typical marking process rather than acting as a spot the I activity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The unrepresentativeness of the CS4 student sample does not affect the validity of the results for the ChatGPT-generated CS4 scripts but prevents broader comparisons against the cohort.

    “This does not affect the validity of the results for the ChatGPT-generated CS4 script documents, but it does prevent broader comparisons of the performance of the scripts against the cohort as a whole.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Markers were provided pre-existing marking guides developed by the module teams.

    “Pre-existing marking guides developed by the module teams were provided.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The study resulted in 90 marked scripts, with 30 of the scripts double marked.

    “This resulted in 90 marked scripts, with 30 of the scripts double marked.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The students generating the ChatGPT scripts were not allowed to add their own text, edit responses, or remove irrelevant material other than to remove evidence that the text had been generated by ChatGPT.

    “The students generating the ChatGPT-generated scripts were not allowed to add their own text, edit responses, or remove irrelevant material other than to remove any evidence the text had been generated by ChatGPT.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The student and ChatGPT-generated scripts for four modules were marked by ALs, with CS2 and CS3 independently double marked.

    “The student and ChatGPT-generated scripts for four modules were marked by ALs, with CS2 and CS3 being independently double marked.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Most ChatGPT-generated scripts received scores away from boundary grades, so if submitted as genuine they would not be further scrutinized before being awarded a grade.

    “However, since most ChatGPTgenerated scripts received scores away from boundary grades, had they been submitted as genuine assignment documents they would not be further scrutinized before being awarded a grade.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The low mark range may not be representative of actual cheating behavior where students may supplement or alter LLM outputs.

    “As previously stated, we did not edit the output of ChatGPT when creating the ChatGPT-generated scripts, so this low level of mark range may not be representative of actual cheating behavior where students may choose to supplement or alter the LLM s outputs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisereasoning

    Changes to module content and assessment approaches meant that older student scripts were not representative of current teaching practice.

    “Whilst student scripts from older presentations were available, changes to module content and assessment approaches potentially meant that those documents were not representative of current teaching practice.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    A script was suspected of AI origin if it scored more than 0.2 on the detection software.

    “We suspect a script of being of AI origin if it scored more than 0.2.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Four modules from across the undergraduate and taught postgraduate computing curricula were selected to explore ChatGPT's ability to generate text at various levels of academic attainment and for different audiences.

    “Four modules from across the undergraduate and taught postgraduate computing curricula were selected, to explore ChatGPT s ability to generate text at various levels of academic attainment and for different audiences.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The four selected modules are representative of the broader computing curriculum and assessment types.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    ChatGPT poses specific issues in terms of convenience and cost, and the nature of its responses renders traditional plagiarism software inadequate.

    “ChatGPT poses specific issues in terms of convenience and cost, as well as the nature of its responses rendering traditional plagiarism software inadequate.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    Identifying metadata was removed from all scripts, resetting document creation and save dates and erasing editing history.

    “Identifying metadata was removed from the student and ChatGPT-generated scripts which were saved as Microsoft Word files.This not only removed author information, but reset the document creation and save dates, as well as erasing the document editing history.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Premisereasoning

    Much of the work in computing has focused on software; further work is needed to assess empirically both the capabilities of ChatGPT to the broader computing curriculum and the risks it poses to academic quality assurance processes.

    “Much of the work in the area of computing has focused on software; further work is needed to assess empirically both the capabilities of ChatGPT to the broader computing curriculum as well as exploring the risks it poses to academic quality assurance processes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    Detection tools cannot demonstrate proof that a given piece of text has been generated by AI, only the probability that it may be machine-generated, limiting their suitability in academic conduct cases.

    “these tools cannot demonstrate proof that a given piece of text has been generated by AIonly the probability that it may be machine-generated, limiting their suitability in academic conduct cases.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Module teams at the Open University often consider a TurnItIn score of 0.25 or above worthy of investigation for further evidence of plagiarism.

    “Module teams at the Open University often consider a TurnItIn score of 0.25 or above worthy of investigation for further evidence of plagiarism.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Each marker received ten student scripts and five ChatGPT-generated scripts, a sufficient number to determine if there was any underlying pattern in the AI-generated solutions but not so great as to make it obvious to markers that the scripts had not been generated by students.

    “Each marker received ten student scripts and five ChatGPT-generated scriptsa sufficient number to determine if there was any underlying pattern in the AI-generated solutions to questions, but not so great as to make it obvious to markers that the scripts had not been generated by students.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Study design and conductown data

    Participants had the right to withdraw their interview data within two weeks, and none did.

    “We allowed participants the right to withdraw their interview data within two weeks of the interview; none of the markers did so.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The project team gained appropriate permission from institutional committees for data protection and ethics.

    “The project team engaged fully with the Open University data protection and ethics policies, gaining appropriate permission from the institutional committees.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The results represent a snapshot of what a certain LLM tool could achieve at a certain time.

    “Therefore, our results represent a snapshot of what a certain LLM tool could achieve at a certain time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The interview transcripts were subjected to line-by-line analysis by the first author, who had not interviewed any participants.

    “The transcripts were subjected to a line-by-line analysis by the first author, who had not interviewed any of the participants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    An understanding of programming is best examined through practical activities on a computer.

    “Unfortunately, many areas of computing are poorly-suited to this exam format; for example, an understanding of programming is best examined through practical activities on a computer [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Guaranteeing competencies would be challenging given the open-ended nature of authentic assessment.

    “This would be challenging to guarantee given the open-ended nature of such assessment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    This aligns with Orenstrakh et al., with eight tools providing very different levels of accuracy and false positives.

    “This aligns to the findings of Orenstrakh et al., with eight tools providing very different levels of accuracy and false positives [36].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [36] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    This helps account for the low scores received by many CS4 marked scripts and why it did not result in plagiarism flags.

    “This helps account for the low scores received by many of the CS4 marked scripts, and why it didn t result in the scripts being flagged for plagiarism.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Many areas of computing are poorly-suited to the proctored exam format.

    “Unfortunately, many areas of computing are poorly-suited to this exam format; for example, an understanding of programming is best examined through practical activities on a computer [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The data is indicative that higher-level cognitive skills expected of postgraduate students cannot be replicated with ChatGPT, but the data set is too small to base judgements on.

    “Our data is indicative that the higher-level cognitive skills we expect of postgraduate students [2] cannot be replicated with ChatGPT, but this is too small a data set on which to base judgements”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Technology alone cannot solve the problem of LLM-generated academic misconduct.

    “In short, our findings can be summarized as beware of snake oil -technology alone cannot solve this problem.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Early research indicates that GPT detectors are biased against non-native English writers, weakening the case for their use.

    “In other domains, early research is indicative that GPT detectors are biased against non-native English writers [31], significantly weakening the case for their use.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [31] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Overall, markers' awareness of scripts containing unusual or unexpected material and their articulation of characteristics prompting suspicion were impressive, although their formal flagging was extremely selective.

    “Overall, markers awareness of scripts containing unusual or unexpected material, and their articulation of the characteristics that prompted suspicion, were impressive, although their formal flagging of scripts was extremely selective.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The LLM challenge to academia strongly resembles that previously created by essay mills.

    “The LLM challenge to academia strongly resembles that previously created by essay mills.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Providing less guidance and requiring longer-form answers seriously disadvantages novice students as well as those struggling with module content or language skills.

    “Whilst it is tempting to suggest these issues can be resolved by providing less guidance and requiring longer-form answers; such an approach seriously disadvantages novice students as well as those struggling with module content or language skills.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The likelihood is that newer versions of LLMs have greater capabilities that increase the risks to academic integrity.

    “While we have no empirical data investigating these further capabilities, the likelihood is that the newer versions of LLMs have greater capabilities that increase the risks to academic integrity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    A consistent compromise exists between ecological validity (using real student data) and needing to understand the capabilities of tools and technologies that are inappropriate to use with student data.

    “A consistent compromise exists between ecological validityin using real student dataand needing to understand the capabilities of tools and technologies which are inappropriate to use with student data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Ohmann highlighted a significant problem with students distributing exam questions to peers during the five-day period required to complete every individual session.

    “Ohmann also highlighted a significant problem with students from distributing exam questions to peers during the five-day period required to complete every individual session.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The study prepared scripts for two intermediate undergraduate modules but failed to recruit any markers for them.

    “We had prepared scripts for two intermediate undergraduate modules to which we failed to recruit any markers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Performing interviews at two points during a semester-long team project eliminated the student practice of copying assignments and disguising plagiarism by making minor changes.

    “Motivated by reducing plagiarism in a system analysis and design course, Dick studied student interviews as a means of assessment [15] and found that performing interviews at two points during a semester-long team project eliminated the student practice of copying assignments and disguising plagiarism by making minor changes.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Replacing written assignments with oral examinations has been demonstrated in CS, with mixed feedback from students.

    “Replacing written assignments with oral examinations has been demonstrated in CS, with mixed feedback from students (e.g.[17] [27] [42]).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] [27] [42]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Online proctoring has been strongly contested as a gross and illegal violation of privacy and as discriminatory, by assuming students have access to a reasonably private place where they can work on an exam without disturbance or violating the rights of others.

    “Online proctoring has been strongly contested as a gross, and illegal, violation of privacy [9] as well as discriminatory, by assuming students have access to a reasonably private place where they can work on an exam without disturbance or violating the rights of others.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Students and instructors who participated in a final oral exam in a foundational CS course had positive reactions about their experience, with students demonstrating a deeper level of engagement with the material than previously noted with written assessment.

    “Ohmann discovered that the students and instructors who participated in a final oral exam in a foundational CS course had positive reactions about their experience, with students demonstrating a deeper level of engagement with the material than previously noted with written assessment [37].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    When interpreting results from peer-reviewed studies, readers must be aware that the capabilities of discussed technologies have likely improved or been entirely superseded, and that the work can only represent a snapshot in time.

    “When interpreting results from peer-reviewed studies, readers must be aware that the capabilities of discussed technologies have likely improved or been entirely superseded, and that the work can only represent a snapshot in time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Regardless of what method of detection is used, some of the concerns raised-particularly in terms of false-positives and the lack of evidence-appear insoluble.

    “Regardless of what method of detection is used, some of the concerns raisedparticularly in terms of false-positives and the lack of evidenceappear insoluble.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Written final exams are known to cause anxiety amongst some students.

    “Furthermore, written final exams are known to cause anxiety amongst some students, whilst open-book exams can reduce anxiety and stress [42].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [42]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Ohmann cited difficulties with scaling the oral exam to a larger class, especially when single instructors do not have sufficient support from tutors or teaching assistants.

    “Regarding barriers to the implementation of oral exams, Ohmann cited difficulties with scaling the exam to a larger class, especially when single instructors do not have sufficient support from tutors or teaching assistants.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Students who underwent the interview assessment received immediate feedback and had opportunities to practice and develop their communication skills.

    “Further positive findings were that students received immediate feedback and had opportunities to practice and develop their communication skills.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The study struggled to recruit markers for the four courses presented; the advanced undergraduate and postgraduate levels were each evaluated through a single module.

    “We struggled to recruit markers for the four courses presented; and the advanced undergraduate and the postgraduate levels were each evaluated through a single module.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Any edits or additions to ChatGPT outputs are almost certain to increase the difficulty of detecting cheating both through automated processes and the expertise of markers.

    “Any such edits or additions are almost certain to increase the difficulty of detecting cheating both through automated processes and the expertise of markers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The study attempted to mitigate the impact of markers' lack of LLM experience by providing broad instructions to markers to report anything unusual or unexpected found in the scripts.

    “We attempted to mitigate this impact by providing broad instructions to markers to report anything unusual or unexpected found in the scripts .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    This replication challenge exacerbates the existing acknowledged crises in replication studies within computing education research, which existing strategies such as editorial policies and a healthy environment of replication studies will not address.

    “This exacerbates the existing acknowledged crises in replication studies within computing education research [1,24], which existing strategiessuch as editorial policies and a healthy environment of replication studieswill not address [24].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [1] source

      Not checked in the free run.

    • cites [24] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Research teams are likely to be working on outdated or outmoded working hypotheses within fast moving areas.

    “Research teams are likely to be working on outdated or outmoded working hypotheses within fast moving areas.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    There are concerns that proctoring technologies employing biometric techniques (e.g., facial recognition) disproportionately disadvantage minority students.

    “As Swauger explains, there are concerns that proctoring technologies employing biometric techniques (e.g., facial recognition) disproportionately disadvantage minority students [49].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    To ensure student authorship, at-home open-book online exams require online proctoring.

    “However, to ensure student authorship, at-home open-book online exams require online proctoring.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The inertia in publications compromises the velocity of dissemination of new findings.

    “The inertia in publications compromises the velocity of dissemination of new findings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Open-book exams can reduce anxiety and stress.

    “Furthermore, written final exams are known to cause anxiety amongst some students, whilst open-book exams can reduce anxiety and stress [42].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [42]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The speed of change in LLM development compared to the slow publication schedule for much peer review work creates a tension.

    “The speed of change in LLM development compared to the slow publication schedule for much peer review work creates a tension.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Many students found the interview process stressful and expressed a strong dislike for this form of assessment.

    “On the negative side, many students found the interview process stressful and expressed a strong dislike for this form of assessment.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationnone given

    Replication of work is challenging, with some technologies rapidly improving and other technologies operating within closed boxes.

    “Replication of work is challenging, with some technologies rapidly improving, and other technologies operating within closed boxes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The ChatGPT-generated scripts produced in this experiment represent the laziest of cheats: those who copy and paste ChatGPT outputs into assignment answers without rewording or supplementing with their own words.

    “Thirdly, the ChatGPT-generated scripts produced in this experiment represent the laziest of cheatsthose who take ChatGPT s outputs and copy and paste it into their assignment answers without attempting to reword the answers or supplement the chatbot s material with their own words.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Highlighting institutional plagiarism policies and highlighting academic good practice has been associated with reduced plagiarism in CS courses.

    “In such cases, highlighting institutional plagiarism policies and highlighting academic good practice has been associated with reduced plagiarism in CS courses [32,34].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [32] source

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Unlike essay mills, the speed and availability of ChatGPT allows its use in remote examination environments.

    “Unlike essay mills, the speed and availability of ChatGPT allows its use in remote examination environments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    ChatGPT greatly reduces the likelihood of being accused of plagiarizing content.

    “However, the deployment of ChatGPT provides a new and distinct method for cheating; it drives the financial cost to zero; greatly reduces the likelihood of being accused of plagiarizing content, and; has reduced the time needed to produce scripts to a few minutes.”
    Quote in the paperNot a substantial claim
110 of 110

  • The paper relies on this without support.

    The Open University practice of genuine student scripts having minimal metadata and short editing times also occurs elsewhere.

    “There is no reason to believe that this does not happen elsewhere.”

  • The paper relies on this without support.

    The four selected modules are representative of the broader computing curriculum and assessment types.

    “Four modules from across the undergraduate and taught postgraduate computing curricula were selected, to explore ChatGPT s ability to generate text at various levels of academic attainment and for different audiences.”

  • The authors state this as an assumption.

    All 40 student scripts were written by people.

    “On the balance of probabilitygiven the relative complexities of using previous-generation LLMswe believe that all 40 student scripts were written by people but acknowledge that this may not be the case.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GenAI tools like ChatGPT can generate adequate responses for various assessment formats.

    “Richards et al. [39] address using GenAI in academic assessments, pointing out that although GenAI tools like ChatGPT can generate adequate responses for various assessment formats, they struggle with tasks requiring higher cognitive skills.”

    From Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda (2025) · cites this paper as [39]

    The Summary (§6.4) and Conclusions (§8) say ChatGPT produced adequate scripts across a range of question formats in undergraduate computer science modules. Section 6.1 says only that limited postgraduate data are indicative of difficulty with higher-level cognitive skills; it cautions against broader judgments, particularly about more advanced tools. The citing sentence overstates that second point as a general limitation of GenAI tools. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • GenAI tools struggle with tasks requiring higher cognitive skills.

    “Richards et al. [39] address using GenAI in academic assessments, pointing out that although GenAI tools like ChatGPT can generate adequate responses for various assessment formats, they struggle with tasks requiring higher cognitive skills.”

    From Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda (2025) · cites this paper as [39]

    Section 6.4 reports that ChatGPT produced adequate undergraduate scripts across a range of assessment formats. Section 6.1 suggests that the version tested could not replicate the higher-level cognitive skills expected of postgraduate students, but explicitly calls the dataset too small for broader judgments. The claim overgeneralizes that tentative finding to GenAI tools and higher-cognitive-skill tasks generally. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Traditional assessment methods may not be effective in a GenAI-enhanced learning environment.

    “As Richards et al. [39] and Tu [42] noted, that traditional assessment methods may not be effective in a GenAI-enhanced learning environment.”

    From Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda (2025) · cites this paper as [39]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Most CS1 ChatGPT scripts did not reach the stated distinction threshold

    Minor

    The paper defines a distinction as a score greater than 85%, but Table 5 gives three of the five CS1 ChatGPT scripts scores of 80%, 83%, and 84%. Only the two scoring 86% meet that threshold.

  • Table 2 lists four modules, not six

    Minor

    The text says Table 2 gives generation times for six modules, but the table contains entries only for CS1, CS2, CS3, and CS4.

  • Reported document count conflicts with authors and documents per author

    Minor

    The text says nine authors each created eight documents, which yields 72 documents, not the stated 54 examined.

    Reported 54 · recomputed 72

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. Lorin W. Anderson and David R. Krathwohl. 2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. Longman.
  2. 10.1136/bmjsem-2023-001568 · DOI · Paper page
  3. 10.1162/daed_a_01909 · DOI
  4. 10.1057/s41599-023-02269-7 · DOI · Paper page
  5. 10.1080/0260293042000264244 · DOI
  6. Laura Bergmans Nacir Bouali Marloes Luttikhuis and Arend Rensink. 2021. On the efficacy of online proctoring using proctorio. In Proceedings of the 13th International Conference on Computer Supported Education (CSEDU'21) Online Streaming April 23-25 2021 Volume 1 SCITEPRESS 279--290. DOI: 10.5220/0010399602790290 · DOI
  7. Nathan Brunelle and John R. Hott. 2020. Ask me anything: Assessing academic dishonesty. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (SIGCSE ’20). Association for Computing Machinery, New York, NY, 1375. DOI:10.1145/3328778.3372658 · DOI
  8. Charlie Daly and John Waldron. 2004. Assessing the assessment of programming ability. In Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education (SIGCSE ’04). Association for Computing Machinery, New York, NY, 210–213. DOI:10.1145/971300.971375 · DOI · Paper page
  9. Paul Denny, Sathiamoorthy Manoharan, Ulrich Speidel, Giovanni Russello, and Angela Chang. 2019. On the fairness of multiple-variant multiple-choice examinations. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 462–468. Retrieved March 24, 2023, from 10.1145/3287324.3287357 · DOI
  10. Martin Dick. 2005. Student interviews as a tool for assessment and learning in a systems analysis and design course. In Proceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE ’05). Association for Computing Machinery, New York, NY, 24–28. DOI:10.1145/1067445.1067456 · DOI · Paper page
  11. Martin Dick, Judy Sheard, Cathy Bareiss, Janet Carter, Donald Joyce, Trevor Harding, and Cary Laxer. 2002. Addressing student cheating: definitions and solutions. ACM SIGCSE Bulletin 35, 2 (2002), 172–184. · DOI
  12. 10.1145/953049.800956 · DOI
  13. James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. 2023. My AI wants to know if this will be on the exam: Testing OpenAI's codex on CS2 programming exercises. In Proceedings of the 25th Australasian Computing Education Conference (ACE ’23). Association for Computing Machinery, New York, NY, 97–104. DOI:10.1145/3576123.3576134 · DOI · Paper page
  14. Robert Fraser. 2014. Collaboration, collusion and plagiarism in computer science coursework. Informatics in Education-An International Journal 13, 2 (2014), 179–195. · DOI
  15. 10.1007/s00146-022-01397-z · DOI · Paper page
  16. Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe Charles Foster Jason Phang Horace He Anish Thite and Noa Nabeshima. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020). · arXiv
  17. 10.1145/3345328 · DOI · Paper page
  18. Arto Hellas, Juho Leinonen, and Petri Ihantola. 2017. Plagiarism in take-home exams: Help-seeking, collaboration, and systematic cheating. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE ’17). Association for Computing Machinery, New York, NY, 238–243. DOI:10.1145/3059009.3059065 · DOI · Paper page
  19. 10.2307/378403 · DOI
  20. 10.3352/jeehp.2023.20.1 · DOI · Paper page
  21. David J. Malan, Brian Yu, and Doug Lloyd. 2020. Teaching academic honesty in CS50. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (SIGCSE ’20). Association for Computing Machinery, New York, NY, 282–288. DOI:10.1145/3328778.3366940 · DOI · Paper page
  22. 10.1016/j.jss.2023.111734 · DOI · Paper page
  23. Peter Ohmann. 2019. An assessment of oral exams in introductory CS. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE ’19). Association for Computing Machinery, New York, NY, 613–619. DOI:10.1145/3287324.3287489 · DOI · Paper page
  24. Joël Porquet-Lupine, Hiroya Gojo, and Philip Breault. 2022. LupSeat: A randomized seating chart generator to prevent exam cheating. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 2 (SIGCSE ’22). Association for Computing Machinery, New York, NY, 1078. DOI:10.1145/3478432.3499139 · DOI
  25. Keith Quille, Keith Nolan, Brett A. Becker, and Seán McHugh. 2021. Developing an open-book online exam for final year students. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE ’21). Association for Computing Machinery, New York, NY, 338–344. DOI:10.1145/3430665.3456373 · DOI
  26. Gili Rusak and Lisa Yan. 2021. Unique Exams: Designing assessments for integrity and fairness. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (SIGCSE'21) Association for Computing Machinery Virtual Event 1170--1176. DOI:10.1145/3408877.3432556 · DOI
  27. 10.1007/s40593-022-00300-7 · DOI
  28. Judy Sheard, Simon, Matthew Butler, Katrina Falkner, Michael Morgan, and Amali Weerasinghe. 2017. Strategies for maintaining academic integrity in first-year computing courses. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE ’17). Association for Computing Machinery, New York, NY, 244–249. DOI:10.1145/3059009.3059064 · DOI · Paper page
  29. 10.35542/osf.io/4mec3 · DOI · Paper page

Cited by

Sorted by citations

Authors
Richards, M., Waugh, K., Slaymaker, M., Petre, M., Woodthorpe, J., & Gooch, D.
Affiliations
The Open University (GB)
ORCID
Venue
ACM Transactions on Computing Education
Publisher
Association for Computing Machinery (ACM)
Published
2024-01-14
Volume
24
Issue
1
Pages
1-32
ISSN
1946-6226 (ISSN-L)
DOI
10.1145/3633287
OpenAlex
W4388848791
Semantic Scholar
a2fe4822e7b6612b5a8bbd693492c12bbf95286a
Cited by
52
References
18
Citations by year
2026: 11 · 2025: 23 · 2024: 18
FWCI
13.04
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Transactions on Computing Education (publishedVersion, cc-by-nd)
  • Open Research Online (The Open University) (submittedVersion)
Funders
The Open University
Topics
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Keywords
ChatGPT, generative AI, academic dishonesty, university evaluation, computer science education, quality assurance, AI detection, Turnitin, undergraduate assessment, postgraduate assessment, CS1

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review