The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index; 4 of the 16 sources its claims cite were checked for support.
The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors
Open accessAbstract
This study evaluates the accuracy of 16 publicly available AI text detectors in discriminating between AI-generated and human-generated writing. The evaluated documents include 42 undergraduate essays generated by ChatGPT-3.5, 42 generated by ChatGPT-4, and 42 written by students in a first-year composition course without the use of AI. Three detectors – Copyleaks, TurnItIn, and Originality.ai – have high accuracy with all three sets of documents. Overall, the detectors that require registration and payment are only slightly more accurate than the others.
Overview
CoChat Summary
TL;DR
This study compares 16 publicly available AI text detectors on 126 undergraduate composition papers: 42 ChatGPT-3.5, 42 ChatGPT-4, and 42 human-written. Copyleaks, TurnItIn, and Originality.ai were highly accurate across all three document sets, while most other detectors struggled especially with GPT-4 papers.
Key findings
- Copyleaks and TurnItIn correctly identified the AI- or human-generated status of all 126 documents with no incorrect or uncertain responses; Originality.ai correctly assessed all but two human-generated papers, which it could not classify with certainty.
- Among the other 13 detectors, overall accuracy ranged from 63% to 88%; only the top three detectors correctly classified GPT-4 documents with greater than 83% accuracy, while the rest tended to classify GPT-4 papers as human or uncertain.
- Paid/free status was only weakly related to accuracy: average accuracy was 87% among six subscription detectors and 77% among the others, with Kendall’s tau-b = 0.29; appearance on “best detector” lists had no meaningful correlation for detectors other than TurnItIn, Kendall’s tau-b = 0.08.
- False positives were 10–14% for four detectors, while false negatives were 23–36% for Crossplag, GPT Radar, and Writer, compared with a maximum of 17% and mean of 6.5% for the other detectors; GPT Radar correctly identified 98% of GPT-3.5 documents but only 31% of GPT-4 documents.
Methods
- The study evaluated 16 publicly available AI text detectors using 126 documents: 42 ChatGPT-3.5 papers, 42 ChatGPT-4 papers, and 42 human-written papers from Manhattan College English 110 (First Year Composition) students in 2014–2015.
- AI papers were generated in the first week of April 2023 using prompts asking for approximately 2000-word academic papers with citations and bibliographies; human papers were selected by stratified random sampling to match broad subject representation: 25 social sciences, 9 natural sciences, and 8 humanities.
- Each document was stripped of introductory material, tables, figures, and works cited, entered as plain text, and truncated if over detector limits; tests ran June 25–July 12, 2023, producing 2,016 responses coded as AI, human, or uncertain, then assessed for overall accuracy, accuracy by document type, decisiveness, false positives, and false negatives.
Limitations
- The paper states that TurnItIn may be especially accurate for the particular human-generated texts because student papers could potentially have been used to train the detector, though it says this is unlikely to have had a major impact.
- The paper states that the analysis is based on 126 undergraduate composition papers (literature reviews), so results may not generalize to other kinds of documents.
- The paper states that the most significant limitation is that it does not account for users paraphrasing or otherwise modifying AI-generated texts rather than submitting them unaltered; it also notes this is only the second study to evaluate detectors on ChatGPT-4 documents and that additional GPT-4 analyses are needed.
- Evident from the text: responses were coded by just one individual, though the paper says the distinctions among AI, uncertain, and human categories were generally quite clear.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 9 Not checked · 61 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the other 13 detectors, overall accuracy ranges from 63 to 88%.”
§ The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors > 3.1 Accuracy of the 16 AI Text Detectors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All the detectors except Content at Scale and ContentDetector.aiare able to identify the GPT-3.5 documents as AI generated at least 86% of the time, and seven perform flawlessly with this particular set of documents (Figure 2).”
§ The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors > 3.1 Accuracy of the 16 AI Text Detectors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The four detectors labeled many each have false positive rates of 10-14%.”
§ The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors > 3.3 Key Similarities and Differences Among the 16 AI Text Detectors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this analysis, no such difficulty can be seen (Tables 4 and 5).”
§ The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors > 4.2 Previous Research and New Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The release of GPT-4 in March 2023 may have given AI users a temporary ability to pass off AI text as human-authoredbut less than 4 months later, the three most effective AI text detectors perform just as well with GPT-4 documents as with GPT-3.5 documents.”
§ The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors > 4.1 Main Findings
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
26 more claims
Create a free account to see all 74 claims and every source verdict.
- Probably not a substantial claim. 37
- It may state two separate claims as one. 21
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Detection systems frequently produce inconsistent results depending on writing style and language proficiency (Walters, 2023)”
From Academic Integrity and Assessment Design in the Era of Generative AI: Perspectives from Vietnamese Universities (2026) · cites this paper as Walters, 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Institutional responses have often relied on AI detection tools; however, such approaches have proven inadequate.”
From Academic Integrity and Assessment Design in the Era of Generative AI: Perspectives from Vietnamese Universities (2026) · cites this paper as Walters, 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At the institutional level, early responses have largely relied on AI detection tools; however, these approaches have been widely criticised for their inconsistency, susceptibility to circumvention, and potential bias against non-native English writers (Walters, 2023;Perkins et al., 2024a;Liang et al., 2023).”
From Academic Integrity and Assessment Design in the Era of Generative AI: Perspectives from Vietnamese Universities (2026) · cites this paper as Walters, 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“vendor accuracy claims routinely exceed observed performance under controlled testing”
From ARE AI DETECTORS FAIR TO EFL LEARNERS? A LIBRARY RESEARCH ON DETECTION BIAS AND ACADEMIC INTEGRITY (2026) · cites this paper as Walters, 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Thus, false positives are structural features of current detection architectures (Caner, 2025; Elkhatat et al., 2023; Walters, 2023).”
From Harms and Ethical Risks of AI Detection Tools in Education: False Positives, Bias, Surveillance, and Student Rights (2026) · cites this paper as Walters, 2023
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors > 2.2 Selecting the 16 AI Text Detectors
“The five other AI text detectors included in the study -ContentDetector.ai(2023), Grammica (2023), IvyPanda (2023), Scribbr (2023), and SEO.ai (2023)are promoted widely online, do not require registration or payment, and do not appear on any of the five "best detector" lists.”
§ The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors > 4.4 Limitations and Further Research
“IvyPanda Free, but registration is required”
Citations
Cites
Motivational processes affecting learning.
Dweck, C. S. (1986) · American Psychologist · doi:10.1037/0003-066x.41.10.1040 · Cited by 6,971
BAİDOO-ANU, D., & OWUSU ANSAH, L. (2023) · Journal of AI · doi:10.61969/jai.1337500 · Cited by 3,939
Baidoo-Anu, D., & Owusu Ansah, L. (2023) · doi:10.2139/ssrn.4337484 · Cited by 3,004
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099
Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24750 · Cited by 919
Can AI-Generated Text be Reliably Detected?
Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11156 · Cited by 810
Rule of Law and Constitutional Democracy
(2015) · DergiPark (Istanbul University) · doi:10.2139/ssrn · Cited by 804
GPT detectors are biased against non-native English writers
Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023) · Patterns · doi:10.1016/j.patter.2023.100779 · Cited by 673
Will ChatGPT get you caught? Rethinking of Plagiarism DetectionPreprint
Khalil, M., & Er, E. (2023) · Interacción · doi:10.48550/arxiv.2302.04335 · Cited by 369
How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., & Wu, Y. (2023) · arXiv · doi:10.48550/arxiv.2301.07597 · Cited by 318
Research works printed but not linked: 2
- Atlas, S. (2023). Chatbot prompting: A guide for students, educators, and an AI-augmented workforce. https://www.researchgate.net/publication/367464129_Chatbot_Prompting_A_guide_for_students_educators_and_an_AI-augmented_workforce.
- Yan, D., Fauss, M., Hao, J., & Cui, W. (2023). Detection of AI-generated essays in writing assessments. Psychological Test and Assessment Modeling, 65(1), 125–144. https://www.psychologie-aktuell.com/fileadmin/Redaktion/Journale/ptam_2023-1/PTAM__1-2023_5_kor.pdf.
Cited by
Fabrication and errors in the bibliographic citations generated by ChatGPT
Walters, W. H., & Wilder, E. I. (2023) · Scientific Reports · doi:10.1038/s41598-023-41032-5 · Cited by 499
Self-citationMcDonald, N., Johri, A., Ali, A., & Collier, A. H. (2025) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2025.100121 · Cited by 682
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Wong, D. F., & Chao, L. S. (2023) · arXiv · doi:10.48550/arxiv.2310.14724 · Cited by 212
Generative AI in Higher Education: The ChatGPT Effect
Chan, C. K. Y., & Colloton, T. (2024) · Routledge · doi:10.4324/9781003459026 · Cited by 203
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212
Metadata
- Authors
- Walters, W. H.
- Affiliations
- Manhattan University (US)
- Corresponding author
- Walters, William H.
- ORCID
- Walters, William H.
- Venue
- Open Information Science
- Publisher
- Walter de Gruyter GmbH
- Published
- 2023-01-01
- Volume
- 7
- Issue
- 1
- ISSN
- 2451-1781 (ISSN-L)
- OpenAlex
- W4387414057
- Semantic Scholar
- 516905eb942e95b113b98d96cf3ce74f66756264
- Cited by
- 109
- References
- 22
- Citations by year
- 2026: 30 · 2025: 42 · 2024: 34 · 2023: 3
- FWCI
- 27.34
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Open Information Science (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- AI-generated text detection, ChatGPT-3.5, ChatGPT-4, academic integrity, Turnitin, false positives, false-negative results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors. https://app.cochat.ai/papers/10.1515/opis-2022-0158
- The paper
- Walters, W. H. (2023). The Effectiveness of Software Designed to Detect AI-Generated Writing: A Comparison of 16 AI Text Detectors. Open Information Science. https://doi.org/10.1515/opis-2022-0158
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.