Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index; 4 of the 12 sources its claims cite were checked for support.

Testing of detection tools for AI-generated text

Open access

Abstract

Abstract Recent advances in generative pre-trained transformer large language models have emphasised the potential risks of unfair use of artificial intelligence (AI) generated content in an academic environment and intensified efforts in searching for solutions to detect such content. The paper examines the general functionality of detection tools for AI-generated text and evaluates them based on accuracy and error type analysis. Specifically, the study seeks to answer research questions about whether existing detection tools can reliably differentiate between human-written text and ChatGPT-generated text, and whether machine translation and content obfuscation techniques affect the detection of AI-generated text. The research covers 12 publicly available tools and two commercial systems (Turnitin and PlagiarismCheck) that are widely used in the academic setting. The researchers conclude that the available detection tools are neither accurate nor reliable and have a main bias towards classifying the output as human-written rather than detecting AI-generated text. Furthermore, content obfuscation techniques significantly worsen the performance of tools. The study makes several significant contributions. First, it summarises up-to-date similar scientific and non-scientific efforts in the field. Second, it presents the result of one of the most comprehensive tests conducted so far, based on a rigorous research methodology, an original document set, and a broad coverage of tools. Third, it discusses the implications and drawbacks of using detection tools for AI-generated text in academic settings.

Overview

CoChat Summary

TL;DR

This study tested 14 AI-text detection tools on 54 English-language documents spanning human-written, machine-translated, ChatGPT-generated, manually edited, and machine-paraphrased text. It concludes the tools are neither accurate nor reliable, are biased toward classifying text as human-written, and perform worse under obfuscation.

Key findings

  • Across 756 tests, the highest binary-approach accuracy was Turnitin at 76%, followed by Compilatio at 74% and GPT-2 Output Detector Demo at 72%; Content at Scale was lowest at 33%. Turnitin ranked highest across all accuracy approaches, followed by Compilatio and GPT-2 Output Detector.
  • With the exception of GPT Zero, all tested tools had higher accuracy identifying human-written text than AI-generated or modified text. Overall, approximately 20% of AI-generated texts would likely be misattributed to humans; with obfuscation, approximately 50% would likely be misattributed.
  • False-positive probability ranged from 0% (Turnitin) to 50% (GPT Zero); false-negative probability ranged from 8% (GPT Zero) to 100% (Content at Scale).
  • Machine translation dropped accuracy by 20%; manually edited AI text (05-ManEd) received 42% compared with 74% for unmodified AI cases (03-AI and 04-AI); machine-paraphrased AI text (06-Para) had an overall accuracy of 26%.

Methods

  • The researchers created 54 test cases from nine researchers: human-written (01-Hum), human-written then machine-translated (02-MT), ChatGPT-generated (03-AI and 04-AI), ChatGPT-generated then manually edited (05-ManEd), and ChatGPT-generated then machine-paraphrased with Quillbot (06-Para). ChatGPT’s February 13, 2023 model was used; translation used DeepL or Google Translate.
  • Fourteen detection tools were tested: 12 publicly available tools plus Turnitin and PlagiarismCheck. Testing occurred March 7–28, 2023, with Turnitin April 14–20 and PlagiarismCheck May 2–8, for 756 total tests.
  • Tool outputs were mapped to a five-step classification (TN, PTN, UNC, PFP, FP for human-written; FN, PFN, UNC, PTP, TP for AI-generated). Accuracy was calculated using binary, binary inclusive, semi-binary, and logarithmic approaches; false-positive rate, false-negative rate, recall, and precision were also analyzed.

Limitations

  • Stated by the authors: the study focused only on English-language texts; it did not specifically test performance on computer code; it did not systematically examine replicability because results appeared to vary when the same material was tested at different times; and the document set did not test hybrid writing with iterative AI use.
  • Evident from the text: the paid tools knew the submitted documents were part of a test suite and could use the entire test document, so their results may differ from free tools; several tools showed beta-stage usability problems, server errors, inconsistent outputs, and unclear result presentation.
  • Evident from the text: only one ChatGPT model version and Quillbot’s default settings were used, limiting generalization to other generators or obfuscation settings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s40979-023-00146-z

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
From the PDF
Fig. 2 Overall accuracy for each document type (calculated as an average of all approaches discussed)
Fig. 2 Overall accuracy for each document type (calculated as an average of all approaches discussed)From the PDF
Fig. 3 Accuracy (logarithmic) for each document type by detection tool for AI-generated text
Fig. 3 Accuracy (logarithmic) for each document type by detection tool for AI-generated textFrom the PDF
Fig. 5 False accusations for machine-translated documents
Fig. 5 False accusations for machine-translated documentsFrom the PDF
Image 5
From the PDF
Image 6
From the PDF
Fig. 7 False negatives for AI-generated documents 04-AI
Fig. 7 False negatives for AI-generated documents 04-AIFrom the PDF
Fig. 8 False negatives for AI-generated documents 03-AI and 04-AI together
Fig. 8 False negatives for AI-generated documents 03-AI and 04-AI togetherFrom the PDF
Fig. 10 False negatives for machine-paraphrased documents
Fig. 10 False negatives for machine-paraphrased documentsFrom the PDF
Image 10
From the PDF
Fig. 11 Compilatio's NaN% reliability
Fig. 11 Compilatio's NaN% reliabilityFrom the PDF
Fig. 12 Turnitin's similarity report shows up first, it is not clear that the 'AI' is clickable
Fig. 12 Turnitin's similarity report shows up first, it is not clear that the 'AI' is clickableFrom the PDF
Fig. 14 AIDT23-05-AAN
Fig. 14 AIDT23-05-AANFrom the PDF
Fig. 15 AIDT23-05-DWW
Fig. 15 AIDT23-05-DWWFrom the PDF
Fig. 16 AIDT23-05-JGD
Fig. 16 AIDT23-05-JGDFrom the PDF
Fig. 17 AIDT23-05-JPK
Fig. 17 AIDT23-05-JPKFrom the PDF
Fig. 18 AIDT23-05-LLW
Fig. 18 AIDT23-05-LLWFrom the PDF
Fig. 19 AIDT23-05-OLU
Fig. 19 AIDT23-05-OLUFrom the PDF
Fig. 20 AIDT23-05-PTR
Fig. 20 AIDT23-05-PTRFrom the PDF
Fig. 21 AIDT23-05-SBB
Fig. 21 AIDT23-05-SBBFrom the PDF
Fig. 24 AIDT23-06-DWW
Fig. 24 AIDT23-06-DWWFrom the PDF
Fig. 25 AIDT23-06-JGD
Fig. 25 AIDT23-06-JGDFrom the PDF
Fig. 26 AIDT23-06-JPK
Fig. 26 AIDT23-06-JPKFrom the PDF
Fig. 27 AIDT23-06-LLW
Fig. 27 AIDT23-06-LLWFrom the PDF
Fig. 28 AIDT23-06-OLU
Fig. 28 AIDT23-06-OLUFrom the PDF
Fig. 29 AIDT23-06-PTR
Fig. 29 AIDT23-06-PTRFrom the PDF
Fig. 30 AIDT23-06-SBB
Fig. 30 AIDT23-06-SBBFrom the PDF

Verification

Claims the paper makes

2 Valid · 13 Not valid · 6 Not checked · 61 Own results and interpretations

39 more claims

Create a free account to see all 82 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 18
  • Probably not a substantial claim. 27

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

33 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 21 references

Research works printed but not linked: 2

  1. Clarke R, Lancaster T. (2006). Eliminating the successor to plagiarism? Identifying the usage of contract cheating sites. Proceedings of 2nd International Plagiarism Conference Newcastle, UK, 14
  2. ICML. (2023). ICML 2023 Call For Papers, Fortieth International Conference on Machine Learning. https://icml.cc/Conferences/2023/CallForPapers. Accessed 14 June 2023

Cited by

See all 582

Metadata

Authors
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L.
Affiliations
  • HTW Berlin - University of Applied Sciences (DE)
  • Riga Technical University (LV)
  • Uppsala University (SE)
  • Masaryk University (CZ)
  • University of Monterrey (MX)
  • Queen Mary University of London (GB)
  • University of Leeds (GB)
Corresponding author
Bjelobaba, Sonja
Venue
International Journal for Educational Integrity
Published
2023-12-25
Volume
19
Issue
1
ISSN
1833-2595 (ISSN-L)
OpenAlex
W4390175962
Cited by
582
References
21
Citations by year
2026: 265 · 2025: 194 · 2024: 111 · 2023: 11
FWCI
144.96
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal for Educational Integrity (publishedVersion, cc-by)
  • Uppsala University Publications (Uppsala University) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Uppsala University
Topics
  • Social Sciences › Education › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
AI-generated text detection, large language models, ChatGPT, machine translation, Turnitin, academic integrity, generative AI in education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Testing of detection tools for AI-generated text. https://app.cochat.ai/papers/10.1007/s40979-023-00146-z
The paper
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity. https://doi.org/10.1007/s40979-023-00146-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.