Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index; 4 of the 16 sources its claims cite were checked for support.

The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors

Open access
Walters, William H. · 2023 · Open Information Science

Abstract

This study evaluates the accuracy of 16 publicly available AI text detectors in discriminating between AI-generated and human-generated writing. The evaluated documents include 42 undergraduate essays generated by ChatGPT-3.5, 42 generated by ChatGPT-4, and 42 written by students in a first-year composition course without the use of AI. Three detectors – Copyleaks, TurnItIn, and Originality.ai – have high accuracy with all three sets of documents. Overall, the detectors that require registration and payment are only slightly more accurate than the others.

Overview

CoChat Summary

TL;DR

This study compares 16 publicly available AI text detectors on 126 undergraduate composition papers: 42 ChatGPT-3.5, 42 ChatGPT-4, and 42 human-written. Copyleaks, TurnItIn, and Originality.ai were highly accurate across all three document sets, while most other detectors struggled especially with GPT-4 papers.

Key findings

  • Copyleaks and TurnItIn correctly identified the AI- or human-generated status of all 126 documents with no incorrect or uncertain responses; Originality.ai correctly assessed all but two human-generated papers, which it could not classify with certainty.
  • Among the other 13 detectors, overall accuracy ranged from 63% to 88%; only the top three detectors correctly classified GPT-4 documents with greater than 83% accuracy, while the rest tended to classify GPT-4 papers as human or uncertain.
  • Paid/free status was only weakly related to accuracy: average accuracy was 87% among six subscription detectors and 77% among the others, with Kendall’s tau-b = 0.29; appearance on “best detector” lists had no meaningful correlation for detectors other than TurnItIn, Kendall’s tau-b = 0.08.
  • False positives were 10–14% for four detectors, while false negatives were 23–36% for Crossplag, GPT Radar, and Writer, compared with a maximum of 17% and mean of 6.5% for the other detectors; GPT Radar correctly identified 98% of GPT-3.5 documents but only 31% of GPT-4 documents.

Methods

  • The study evaluated 16 publicly available AI text detectors using 126 documents: 42 ChatGPT-3.5 papers, 42 ChatGPT-4 papers, and 42 human-written papers from Manhattan College English 110 (First Year Composition) students in 2014–2015.
  • AI papers were generated in the first week of April 2023 using prompts asking for approximately 2000-word academic papers with citations and bibliographies; human papers were selected by stratified random sampling to match broad subject representation: 25 social sciences, 9 natural sciences, and 8 humanities.
  • Each document was stripped of introductory material, tables, figures, and works cited, entered as plain text, and truncated if over detector limits; tests ran June 25–July 12, 2023, producing 2,016 responses coded as AI, human, or uncertain, then assessed for overall accuracy, accuracy by document type, decisiveness, false positives, and false negatives.

Limitations

  • The paper states that TurnItIn may be especially accurate for the particular human-generated texts because student papers could potentially have been used to train the detector, though it says this is unlikely to have had a major impact.
  • The paper states that the analysis is based on 126 undergraduate composition papers (literature reviews), so results may not generalize to other kinds of documents.
  • The paper states that the most significant limitation is that it does not account for users paraphrasing or otherwise modifying AI-generated texts rather than submitting them unaltered; it also notes this is only the second study to evaluate detectors on ChatGPT-4 documents and that additional GPT-4 analyses are needed.
  • Evident from the text: responses were coded by just one individual, though the paper says the distinctions among AI, uncertain, and human categories were generally quite clear.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1515/opis-2022-0158

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 2 Not valid · 9 Not checked · 61 Own results and interpretations

26 more claims

Create a free account to see all 74 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 37
  • It may state two separate claims as one. 21
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

5 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 22 references

Research works printed but not linked: 2

  1. Atlas, S. (2023). Chatbot prompting: A guide for students, educators, and an AI-augmented workforce. https://www.researchgate.net/publication/367464129_Chatbot_Prompting_A_guide_for_students_educators_and_an_AI-augmented_workforce.
  2. Yan, D., Fauss, M., Hao, J., & Cui, W. (2023). Detection of AI-generated essays in writing assessments. Psychological Test and Assessment Modeling, 65(1), 125–144. https://www.psychologie-aktuell.com/fileadmin/Redaktion/Journale/ptam_2023-1/PTAM__1-2023_5_kor.pdf.

Cited by

See all 114

Metadata

Authors
Walters, W. H.
Affiliations
Manhattan University (US)
Corresponding author
Walters, William H.
Venue
Open Information Science
Published
2023-01-01
Volume
7
Issue
1
ISSN
2451-1781 (ISSN-L)
OpenAlex
W4387414057
Cited by
109
References
22
Citations by year
2026: 30 · 2025: 42 · 2024: 34 · 2023: 3
FWCI
27.34
Type
journal article
Language
en
Open access
Locations
  • Open Information Science (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Education › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
AI-generated text detection, ChatGPT-3.5, ChatGPT-4, academic integrity, Turnitin, false positives, false-negative results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors. https://app.cochat.ai/papers/10.1515/opis-2022-0158
The paper
Walters, W. H. (2023). The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors. Open Information Science. https://doi.org/10.1515/opis-2022-0158

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.