The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index; 4 of the 12 sources its claims cite were checked for support.
Testing of detection tools for AI-generated text
Open accessAbstract
Abstract Recent advances in generative pre-trained transformer large language models have emphasised the potential risks of unfair use of artificial intelligence (AI) generated content in an academic environment and intensified efforts in searching for solutions to detect such content. The paper examines the general functionality of detection tools for AI-generated text and evaluates them based on accuracy and error type analysis. Specifically, the study seeks to answer research questions about whether existing detection tools can reliably differentiate between human-written text and ChatGPT-generated text, and whether machine translation and content obfuscation techniques affect the detection of AI-generated text. The research covers 12 publicly available tools and two commercial systems (Turnitin and PlagiarismCheck) that are widely used in the academic setting. The researchers conclude that the available detection tools are neither accurate nor reliable and have a main bias towards classifying the output as human-written rather than detecting AI-generated text. Furthermore, content obfuscation techniques significantly worsen the performance of tools. The study makes several significant contributions. First, it summarises up-to-date similar scientific and non-scientific efforts in the field. Second, it presents the result of one of the most comprehensive tests conducted so far, based on a rigorous research methodology, an original document set, and a broad coverage of tools. Third, it discusses the implications and drawbacks of using detection tools for AI-generated text in academic settings.
Overview
CoChat Summary
TL;DR
This study tested 14 AI-text detection tools on 54 English-language documents spanning human-written, machine-translated, ChatGPT-generated, manually edited, and machine-paraphrased text. It concludes the tools are neither accurate nor reliable, are biased toward classifying text as human-written, and perform worse under obfuscation.
Key findings
- Across 756 tests, the highest binary-approach accuracy was Turnitin at 76%, followed by Compilatio at 74% and GPT-2 Output Detector Demo at 72%; Content at Scale was lowest at 33%. Turnitin ranked highest across all accuracy approaches, followed by Compilatio and GPT-2 Output Detector.
- With the exception of GPT Zero, all tested tools had higher accuracy identifying human-written text than AI-generated or modified text. Overall, approximately 20% of AI-generated texts would likely be misattributed to humans; with obfuscation, approximately 50% would likely be misattributed.
- False-positive probability ranged from 0% (Turnitin) to 50% (GPT Zero); false-negative probability ranged from 8% (GPT Zero) to 100% (Content at Scale).
- Machine translation dropped accuracy by 20%; manually edited AI text (05-ManEd) received 42% compared with 74% for unmodified AI cases (03-AI and 04-AI); machine-paraphrased AI text (06-Para) had an overall accuracy of 26%.
Methods
- The researchers created 54 test cases from nine researchers: human-written (01-Hum), human-written then machine-translated (02-MT), ChatGPT-generated (03-AI and 04-AI), ChatGPT-generated then manually edited (05-ManEd), and ChatGPT-generated then machine-paraphrased with Quillbot (06-Para). ChatGPT’s February 13, 2023 model was used; translation used DeepL or Google Translate.
- Fourteen detection tools were tested: 12 publicly available tools plus Turnitin and PlagiarismCheck. Testing occurred March 7–28, 2023, with Turnitin April 14–20 and PlagiarismCheck May 2–8, for 756 total tests.
- Tool outputs were mapped to a five-step classification (TN, PTN, UNC, PFP, FP for human-written; FN, PFN, UNC, PTP, TP for AI-generated). Accuracy was calculated using binary, binary inclusive, semi-binary, and logarithmic approaches; false-positive rate, false-negative rate, recall, and precision were also analyzed.
Limitations
- Stated by the authors: the study focused only on English-language texts; it did not specifically test performance on computer code; it did not systematically examine replicability because results appeared to vary when the same material was tested at different times; and the document set did not test hybrid writing with iterative AI use.
- Evident from the text: the paid tools knew the submitted documents were part of a test suite and could use the entire test document, so their results may differ from free tools; several tools showed beta-stage usability problems, server errors, inconsistent outputs, and unclear result presentation.
- Evident from the text: only one ChatGPT model version and Quillbot’s default settings were used, limiting generalization to other generators or obfuscation settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



























Verification
Claims the paper makes
2 Valid · 13 Not valid · 6 Not checked · 61 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As these models are trained on human-written texts, the content generated by these tools can be quite difficult to distinguish from human-written content.”
- Also stated in: § Large language models · p. 3 · plain
“it is capable of creating responses that are hard to distinguish from human-written text (Borji 2023; Brown et al. 2020).”
What the sources say
- Not checked yetcites Borji 2023 source
Checked before — its earlier verdict is held back until a new check runs.
- Supportedcites Brown et al. 2020 source
Section 3.9.4 (p. 26) says much of GPT-3’s generated news-article text was difficult for humans to distinguish from authentic human content. It reports human detection accuracy of about 52%, barely above chance, for articles from the 175B-parameter model. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Large language models · p. 3 · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our findings are consistent with previously published studies (Gao et al. 2022; Anderson et al. 2023; Elkhatat et al. 2023; Demers 2023; Gewirtz 2023; Krishna et al. 2023; Pegoraro et al. 2023; van Oijen 2023; Wang et al. 2023)”
What the sources say
- Supportedcites Gao et al. 2022 source
Our copy is exactly the kind of study the citing sentence invokes: a previously published investigation of AI-generated text detection. The Abstract (p. 1) reports that most ChatGPT-generated medical abstracts were flagged by the 'GPT-2 Output Detector' (median 99.98% 'fake' vs 0.02% for originals, AUROC 0.94) while blinded human reviewers correctly identified 68% of generated abstracts but wrongly flagged 14% of real ones, and the Results/Discussion (p. 2-3) conclude detectors and humans are imperfect discriminators ('AI output detectors may serve as an editorial tool'). The paper does not itself compare its findings to other detection studies, but nothing in our copy contradicts the alignment asserted; the attribution is an identification of this work as a prior detection study, which it is. (LLM, conf=0.85)
- Supportedcites Anderson et al. 2023 source
The “Potentially flawed AI detection” section reports that paraphrasing AI-generated essays sharply increased their GPT-2 Output Detector ‘Real’ scores. The paper therefore provides a published finding about AI-generated text detection of the kind the citing sentence invokes; the sentence does not specify a narrower result to check. (LLM, conf=0.86)
- Supportedcites Elkhatat et al. 2023 source
The cited paper is a previously published study of AI-generated text detection. Its Conclusion reports variable accuracy across five detection tools, so it supports its use as an example of prior research on the topic. The citing paper’s own assertion that its findings are consistent with that research cannot be assessed from this source alone. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Turnitin received the highest score using all approaches to accuracy classification, followed by Compilatio and GPT-2 Output Detector (again in all approaches).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The researchers conclude that the available detection tools are neither accurate nor reliable and have a main bias towards classifying the output as human-written rather than detecting AI-generated text.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Crossplag and Go Winston were the only other tools to achieve at least 70% accuracy.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
39 more claims
Create a free account to see all 82 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 27
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many early institutional responses to generative AI have emphasized detection tools and misconduct procedures, often more than assessment redesign (Weber-Wulff et al., 2023).”
From Designing AI-resilient assessment in higher education: a four-pillar conceptual framework (2026) · cites this paper as Weber-Wulff et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many commercially available tools fail to achieve 80% accuracy despite exceptions among a few premium platforms”
From ARE AI DETECTORS FAIR TO EFL LEARNERS? A LIBRARY RESEARCH ON DETECTION BIAS AND ACADEMIC INTEGRITY (2026) · cites this paper as Weber-Wulff et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The exponential growth of Generative Artificial Intelligence (GAI) and Large Language Models (LLMs) has significantly impacted teaching and learning at all educational levels [1].”
From Generative artificial intelligence in physiotherapy education: great potential amidst challenges- a qualitative interview study (2025) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPTZero and ZeroGPT produced false-positive rates exceeding 20% on specific samples”
From Designing AI-resilient assessment in higher education: a four-pillar conceptual framework (2026) · cites this paper as Weber-Wulff et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“William H. Walters found that most AI detectors can distinguish papers generated from the GPT3.5 model and human-written papers but fail to identify papers generated from the recent GPT-4 model [138].”
From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [138]
Experimental — accuracy not yet measured
33 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Note that the maximum value for each cell is now 9 * 16 = 864.”
Reported 864 · recomputed 144
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Abbreviations · p. 36
“04-AI AI-generated text with subsequent human manual edits”
§ Test cases · p. 9
“One set (05-ManEd) was edited manually with a human exchanging some words with synonyms or reordering sentence parts”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Accuracy · p. 15
“Turnitin received the highest score using all approaches to accuracy classification, followed by Compilatio and GPT-2 Output Detector (again in all approaches).”
§ Accuracy · p. 13
“| GPT Zero | 6 | 3 | 8 | 9 | 8 | 8 | 42 | 78% | 3 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ False accusations: harm to individual students · p. 20
“For GPT Zero, half of the positive classifications would be false accusations”
§ Precision · p. 18
“| Prec_incl | 96% | 100% | -- | 88% | 100% | 100% | 79%”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Efficient methods for the estimation of word representations in multidimensional vector spaces (Mikolov et al., 2013), together with the attention mechanism and transformer architecture (Vaswani et al., 2017) made generating human-like text not only possible, but also computationally feasible.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“This was in many ways a game-changer, as it is capable of creating responses that are hard to distinguish from human-written text (Borji, 2023; Brown et al., 2020).”
BackgroundKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“We understand LLMs as systems trained to predict the likelihood of a specific character, word, or string (called a token ) in a particular context (Bender et al., 2021).”
BackgroundGao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022) · doi:10.1101/2022.12.23.521610 · Cited by 1,099 · cited 1×
“…was able to make an excellent distinction between original and generated abstracts because the majority of the original abstracts were scored extremely low (corresponding to human-written content) while the detector found a high probability of AI-generated text in the majority (33 abstracts)…”
BackgroundGao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099
Two decades of statistical language modeling: where do we go from here?
Rosenfeld, R. (2000) · Proceedings of the IEEE · doi:10.1109/5.880083 · Cited by 735
Research works printed but not linked: 2
- Clarke R, Lancaster T. (2006). Eliminating the successor to plagiarism? Identifying the usage of contract cheating sites. Proceedings of 2nd International Plagiarism Conference Newcastle, UK, 14
- ICML. (2023). ICML 2023 Call For Papers, Fortieth International Conference on Machine Learning. https://icml.cc/Conferences/2023/CallForPapers. Accessed 14 June 2023
Cited by
Generative Artificial Intelligence: Implications and Considerations for Higher Education Practice
Farrelly, T., & Baker, N. (2023) · Education Sciences · doi:10.3390/educsci13111109 · Cited by 414
“Research on the efficacy of a wide range of GAI detection tools is now beginning to become available (although most are still in pre-print or as yet not peer-reviewed at the time of writing, e.g., [13–18]).”
BackgroundGenerative AI Literacy: Twelve Defining Competencies
Annapureddy, R., Fornaroli, A., & Gatica-Perez, D. (2025) · Digital Government: Research and Practice · doi:10.1145/3685680 · Cited by 333
“Example : Several universities have developed guidelines for the use of LLMs in assignments [26].”
Wang, H., Dang, A., Wu, Z., & Mac, S. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100326 · Cited by 272
“Last but not least, considering the current GenAI detection tools are not reliable and supported to determine plagia-rism by most universities (Elkhatat et al., 2023; Weber-Wulff et al., 2023), teachers should adopt multifaceted evaluation strategies to assess students’ work.”
BackgroundA Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Wong, D. F., & Chao, L. S. (2023) · arXiv · doi:10.48550/arxiv.2310.14724 · Cited by 212
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212
Metadata
- Authors
- Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L.
- Affiliations
- HTW Berlin - University of Applied Sciences (DE)
- Riga Technical University (LV)
- Uppsala University (SE)
- Masaryk University (CZ)
- University of Monterrey (MX)
- Queen Mary University of London (GB)
- University of Leeds (GB)
- Corresponding author
- Bjelobaba, Sonja
- ORCID
- Venue
- International Journal for Educational Integrity
- Published
- 2023-12-25
- Volume
- 19
- Issue
- 1
- ISSN
- 1833-2595 (ISSN-L)
- OpenAlex
- W4390175962
- Cited by
- 582
- References
- 21
- Citations by year
- 2026: 265 · 2025: 194 · 2024: 111 · 2023: 11
- FWCI
- 144.96
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- International Journal for Educational Integrity (publishedVersion, cc-by)
- Uppsala University Publications (Uppsala University) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Uppsala University
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- AI-generated text detection, large language models, ChatGPT, machine translation, Turnitin, academic integrity, generative AI in education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Testing of detection tools for AI-generated text. https://app.cochat.ai/papers/10.1007/s40979-023-00146-z
- The paper
- Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity. https://doi.org/10.1007/s40979-023-00146-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.