CoChat
Sign Up
PreprintCited by 71Authorship Attribution and ProfilingTopic ModelingDigital Media Forensic Detection

On the Possibilities of AI-Generated Text Detection

Open access
Chakraborty, Souradip, Bedi, Amrit Singh, Zhu, Sicheng, An, Bang, Manocha, Dinesh, Huang, Furong·2023·arXiv
DOI 10.48550/arxiv.2304.04736arXiv 2304.04736
Open this paper in CoChat — free accountOpen PDF

Abstract

Our work addresses the critical issue of distinguishing text generated by Large Language Models (LLMs) from human-produced text, a task essential for numerous applications. Despite ongoing debate about the feasibility of such differentiation, we present evidence supporting its consistent achievability, except when human and machine text distributions are indistinguishable across their entire support. Drawing from information theory, we argue that as machine-generated text approximates human-like quality, the sample size needed for detection increases. We establish precise sample complexity bounds for detecting AI-generated text, laying groundwork for future research aimed at developing advanced, multi-sample detectors. Our empirical evaluations across multiple datasets (Xsum, Squad, IMDb, and Kaggle FakeNews) confirm the viability of enhanced detection methods. We test various state-of-the-art text generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector, GPTZero. Our findings align with OpenAI's empirical data related to sequence length, marking the first theoretical substantiation for these observations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.04736

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The underlying assumption of Chakraborty et al.'s result is that several independent samples are available to the detector from either human or text distribution, which might not be practical because sentences in a document are often correlated.

    “However, the underlying assumption of their result is that several independent samples are available to the detector from either human or text distribution, which might not be a practical assumption since sentences in a document are often correlated with each other.”

    From Can AI-Generated Text be Reliably Detected? (2023) · cites this paper as Chakraborty et al., 2023

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • A recent work (Chakraborty et al., 2023) shows a trade-off on the detection problem with respect to the availability of the number of data samples for detection.

    “A recent work (Chakraborty et al., 2023) shows a trade-off on the detection problem with respect to the availability of the number of data samples for detection.”

    From Can AI-Generated Text be Reliably Detected? (2023) · cites this paper as Chakraborty et al., 2023

  • Existing supervised classifiers frequently overfit to particular domains, perform poorly on out-of-distribution data, and lose capability on unseen datasets from other domains.

    “However, these classifiers often overfit to specific domains, leading to poor performance on out-of-distribution data [12,51], and their capabilities significantly degrade when applied to unseen datasets from different domains [32].”

    From Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement (2025) · cites this paper as [12]

    The citation to this paper is attached to the claim that classifier performance degrades on unseen datasets from different domains. Section 4.1 reports results on several datasets and a performance drop under paraphrasing, but does not report testing a classifier trained in one domain on an unseen dataset from another. Searches for cross-domain generalization, out-of-distribution testing, and unseen-dataset performance found no passage establishing that claim. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 17

  1. Mikhail J. Atallah, Victor Raskin, Michael Crogan, Christian Hempelmann, Florian Kerschbaum, Dina Mohamed, and Sanket Naik. Natural language watermarking: Design, analysis, and a proof-of-concept implementation. In Proceedings of the 4th International Workshop on Information Hiding, IHW '01, pp. 185-199, Berlin, Heidelberg, 2001. Springer-Verlag. ISBN 3540427333.
  2. Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
  3. Amit Dhurandhar. Auto-correlation dependent bounds for relational data. In Proc. of the 11th Workshop on Mining and Learning with Graphs. Chicago, 2013.
  4. Tom Fawcett. An introduction to roc analysis. Pattern recognition letters, 27(8):861-874, 2006.
  5. Nir Kshetri and Jeffrey Voas. Deep learning-based social media misinformation detection. IEEE Software, 39(1):53-59, 2022. doi: 10.1109/MS.2022.3053106. · DOI
  6. Gerhard C Langelaar, Iwan Setyawan, and Reginald L Lagendijk. Watermarking digital image and video data. a state-of-the-art overview. IEEE Signal processing magazine, 17(5):20-46, 2000.
  7. Lucien Le Cam. Asymptotic methods in statistical decision theory. Springer Science & Business Media, 2012.
  8. Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
  9. Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 379-389, 2015.
  10. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156, 2023. · arXiv
  11. Tyler Schildhauer. Fake news detection in the era of ai. In Proceedings of the 25th ACM Conference on Computer-Supported Cooperative Work and Social Computing, pp. 1-10. ACM, 2022. doi: 10.1145/1234567.1234567. · DOI
  12. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2015.
  13. Jin Dong Shin, Hyun Jae Kim, Kyung Min Lee, Seonghyeon Kim, and Seungwon Shin. Multilingual language generation and automatic writing evaluation with transformer models. arXiv preprint arXiv:2104.06399, 2021. · arXiv
  14. Salil Pravin Vadhan. A study of statistical zero-knowledge proofs. PhD thesis, Massachusetts Institute of Technology, 1999.
  15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30:5998-6008, 2017.
  16. Shweta Wadhera, Deepa Kamra, Ankit Rajpal, Aruna Jain, and Vishal Jain. A comprehensive review on digital image watermarking, 2022.
  17. Xiaofei Zou and Xu Ling. Ai-based detection of misinformation in social media. IEEE Access, 9: 112408-112418, 2021. doi: 10.1109/ACCESS.2021.3104419. · DOI

Cited by

Sorted by citations

Related

Authors
Chakraborty, S., Bedi, A. S., Zhu, S., An, B., Manocha, D., & Huang, F.
Affiliations
University of Maryland, College Park (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2304.04736
arXiv
2304.04736
OpenAlex
W4364387756
Cited by
59
References
0
Citations by year
2026: 4 · 2025: 17 · 2024: 18 · 2023: 20
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
AI-generated text detection, large language models, sample complexity, information theory, text generation quality, GPT-2, llamas, GPT-3.5-turbo, GPTZero, fake news detection, sequence length

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review