CoChat

Are AI detection and plagiarism similarity scores worthwhile in the age of ChatGPT and other Generative AI?

Open access
Baron, Philip · 2024 · Scholarship of Teaching and Learning in the South
Cited by 35

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Abstract

Recent advancements in chatbots have provided students and academics with a new mode of how knowledge can be sourced and composed. Within a very short space of time, students and academics have flocked to use ChatGPT and other Generative Artificial Intelligence (GAI) platforms owing to their capable responses. Additionally, apart from the generative chatbots (such as ChatGPT and Gemini), AI writing tools for paraphrasing, summarising, and co-writing have also become capable and increasingly prevalent to such a degree that the public is spoilt for choice. Having conducted tests on popular chatbots and AI writing tools, it became clear that while programs like Turnitin are developing new algorithms to detect plagiarism and AI-generated content, the initial findings from this study suggest that this may be an increasingly difficult task. These tests were published on YouTube, and within a few weeks, the evidence garnered tens of thousands of views as students and educators seemed uncertain about the strengths, weaknesses, and legalities of these AI tools. What is clear is that we have passed the tipping point, and AI assistance is no longer just a grammar fixer. The implications of this are concerning and far-reaching, as plagiarism is already a significant problem in universities. This position paper reports on tests conducted using Turnitin software and AI writing tools such as ChatGPT and QuillBot. These real-world tests support the paper’s position that it is becoming increasingly difficult to determine what constitutes original work in a world of GAI. The aim of this article is to provide evidence that educators who rely on similarity checking and AI detectors in their current form may inadvertently be supporting plagiarism rather than reducing it. A new method of academic plagiarism detection is proposed, utilizing large language models to generate and track ideas, thereby serving as an idea database. The proposed method focuses on the "understanding" of the work rather than on text similarity.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.36615/sotls.v8i2.411

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

1 retracted

Show all 12 references

Cited by

Metadata

Authors
Baron, P.
Venue
Scholarship of Teaching and Learning in the South
Published
2024-09-02
Volume
8
Issue
2
Pages
151-179
ISSN
2523-1154 (ISSN-L)
OpenAlex
W4402205339
Cited by
35
References
12
FWCI
14.03
Type
journal article
Language
en
Open access
diamond

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.