CoChat
Sign Up
Cited by 29Academic integrity and plagiarismArtificial Intelligence in EducationAuthorship Attribution and Profiling

Investigating marker accuracy in differentiating between university scripts written by students and those produced using ChatGPT

Open access
Hassoulas, Athanasios, Powell, Ned, Roberts, Lindsay, Umla-Runge, Katja, Gray, Laurence, Coffey, Marcus J·2023·Journal of Applied Learning and Teaching
DOI 10.37074/jalt.2023.6.2.13
Open this paper in CoChat — free accountOpen PDF

Abstract

The introduction of OpenAI’s ChatGPT has widely been considered a turning point for assessment in higher education. Whilst we find ourselves on the precipice of a profoundly disruptive technology, generative artificial intelligence (AI) is here to stay. At present, institutions around the world are considering how best to respond to such new and emerging tools, ranging from outright bans to re-evaluating assessment strategies. In evaluating the extent of the problem that these tools pose to the marking of assessments, a study was designed to investigate marker accuracy in differentiating between scripts prepared by students and those produced using generative AI. A survey containing undergraduate reflective writing scripts and postgraduate extended essays was administered to markers at a medical school in Wales, UK. The markers were asked to assess the scripts on writing style and content, and to indicate whether they believed the scripts to have been produced by students or ChatGPT. Of the 34 markers recruited, only 23% and 19% were able to correctly identify the ChatGPT undergraduate and postgraduate scripts, respectively. A significant effect of suspected script authorship was found for script content, X²(4, n=34) = 10.41, p<0.05, suggesting that written content holds clues as to how markers assign authorship. We recommend consideration be given to how generative AI can be responsibly integrated into assessment strategies and expanding our definition of what constitutes academic misconduct in light of this new technology.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.37074/jalt.2023.6.2.13

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The effectiveness of conventional anti-plagiarism software, such as iThenticate and Turnitin, remains questionable against AI-generated content.

    “Despite advancements in AI detection tools, the effectiveness of conventional anti-plagiarism software, such as iThenticate and Turnitin, remains questionable against AI-generated content (Perkins, 2023;Chaka, 2023Chaka, , 2024;;Hassoulas et al., 2023;Ifelebuegu, 2023;Mohammadkarimi et al., 2023;Sullivan et al., 2023).”

    From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Hassoulas et al., 2023

    The Discussion and conclusion questions the practical usefulness of AI-detection tools, saying they remain under development and provide little current support to coursework markers. It does not assess conventional anti-plagiarism software or identify iThenticate or Turnitin, so it supports the broader concern but not the claim about those products. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Athanasios Hassoulas, Ned Powell, Lindsay Roberts, Katja Umla-Runge, Laurence Gray and Marcus J Coffey. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Cited by

Sorted by citations

Related

Authors
Hassoulas, A., Powell, N., Roberts, L., Umla-Runge, K., Gray, L., & Coffey, M. J.
Affiliations
Cardiff University (GB)
ORCID
Venue
Journal of Applied Learning and Teaching
Publisher
Open Access Publishing Association
Published
2023-07-24
Volume
6
Issue
2
Pages
71-77
ISSN
2591-801X (ISSN-L)
DOI
10.37074/jalt.2023.6.2.13
OpenAlex
W4385188787
Cited by
28
References
17
Citations by year
2026: 4 · 2025: 8 · 2024: 13 · 2023: 3
FWCI
7.02
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Applied Learning & Teaching (publishedVersion, cc-by)
  • ORCA Online Research @Cardiff (Cardiff University) (submittedVersion, cc-by)
Topics
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
Keywords
ChatGPT, generative AI, academic integrity, academic misconduct, reflective writing, extended essay, medical education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review