The DOI matches its DataCite record; 29 of the 31 references it cites were found in a registry or an index.
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Open accessAbstract
Recent advancements in neural language modelling make it possible to rapidly generate vast amounts of human-sounding text. The capabilities of humans and automatic discriminators to detect machine-generated text have been a large source of research interest, but humans and machines rely on different cues to make their decisions. Here, we perform careful benchmarking and analysis of three popular sampling-based decoding strategies---top-$k$, nucleus sampling, and untruncated random sampling---and show that improvements in decoding methods have primarily optimized for fooling humans. This comes at the expense of introducing statistical abnormalities that make detection easy for automatic systems. We also show that though both human and automatic detector performance improve with longer excerpt length, even multi-sentence excerpts can fool expert human raters over 30% of the time. Our findings reveal the importance of using both human and automatic detectors to assess the humanness of text generation systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The rest of the zero-shot detectors (Solaiman et al., 2019; Gehrmann et al., 2019; Ippolito et al., 2019) also perform poorly after our attack.”
From Can AI-Generated Text be Reliably Detected? (2023) · cites this paper as Ippolito et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Solaiman et al. (2019) and Ippolito et al.(2020) also both present tools which showed an encouraging ability in detecting machine created text, although these tools were not tested against the latest generation of LLMs.”
From Academic Integrity considerations of AI Large Language Models in the post-pandemic era: ChatGPT and beyond (2023) · cites this paper as Ippolito et al.(2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“Encoderonly Transformer models such as BERT (Devlin et al., 2019) have been shown to do very well at tasks requiring this understanding.”
BackgroundMethodInfluentialThe spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 2×
“It propogates quickly (Vosoughi et al., 2018), sets political agendas”
BackgroundSocial Media and Fake News in the 2016 Election
Allcott, H., & Gentzkow, M. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.211 · Cited by 6,767 · cited 2×
“It propogates quickly (Vosoughi et al., 2018), sets political agendas (Vargo et al., 2018), influences elections (Allcott and Gentzkow, 2017), and undermines user trust (Wang et al., 2012; Song et al., 2015).”
BackgroundGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · cited 2×
“textual word embeddings from a pre-trained BERT model without fine-tuning to compute a quality score for generated text (Zhang et al., 2020).”
MethodHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · cited 1×
“Top-k restricts the distribution to all but the k most likely tokens, where k is a constant (Fan et al., 2018).”
BackgroundFake It Till You Make It: Reputation, Competition, and Yelp Review Fraud
Luca, M., & Zervas, G. (2016) · Management Science · doi:10.1287/mnsc.2015.2304 · Cited by 1,263 · cited 1×
“(National Literacy Trust,2018). Aside from news, crowdturfing of fake reviews on websites such as Amazon and Yelp signifi- cantly undermines user trust (Wang et al.,2012; Song et al.,2015). According toLuca and Zervas (2016), approximately 16% of Yelp restaurant reviews were fraudulent as of 2012. Recently,Adelani et al…”
BackgroundAdversarial Learning for Neural Dialogue Generation
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., & Jurafsky, D. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1230 · Cited by 955 · cited 2×
“Finally, discriminators for whether text is machine-generated are a promising research direction in adversarial training (Lin et al., 2017; Li et al., 2017) and in”
BackgroundGenerating Wikipedia by Summarizing Long SequencesPreprint
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, Ł., & Shazeer, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10198 · Cited by 784 · cited 2×
“, 2019), and Transformer-DMCA (Liu et al., 2018) are a few examples of large, publicly available models with this ability.”
BackgroundVargo, C. J., Гуо, Л., & Amazeen, M. A. (2017) · New Media & Society · doi:10.1177/1461444817712086 · Cited by 684 · cited 2×
“It propogates quickly (Vosoughi et al., 2018), sets political agendas ∗Equal contribution, ‡Google, †University of Pennsylvania (Vargo et al., 2018), influences elections (Allcott and Gentzkow, 2017), and undermines user trust (Wang et al., 2012; Song et al., 2015).”
BackgroundGLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3019 · Cited by 470 · cited 9×
“Kreps et al. (2020), Gehrmann et al. (2019), and others have stressed the importance of humans being able to identify fake content on the web.”
BackgroundMethodInfluential(2020) · arXiv (Cornell University) · Cited by 461
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422 · cited 2×
“automatic evaluation of generative model quality (Novikova et al., 2017; Kannan and Vinyals, 2017; Lowe et al., 2017).”
BackgroundProbing Neural Network Comprehension of Natural Language Arguments
Niven, T., & Kao, H.-Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1459 · Cited by 395 · cited 2×
“It is worth noting that recent work has raised questions as to whether BERT truly builds a semantic understanding to make its predictions, or whether it merely takes advantage of spurious statistical differences between the text of different classes (Niven and Kao, 2019).”
BackgroundTowards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Lowe, R., Noseworthy, M., Serban, I. V., Angelard-Gontier, N., Bengio, Y., & Pineau, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1103 · Cited by 341 · cited 3×
“It continues to be used is chatbot evaluation (Lowe et al., 2017).”
BackgroundRelease Strategies and the Social Impacts of Language Models
Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1908.09203 · Cited by 292 · cited 1×
“Total Probability Solaiman et al. (2019) pro-pose a very simple baseline consisting of a threshold on the total probability of the text sequence.”
BackgroundAdversarial Ranking for Language GenerationPreprint
Lin, K., Li, D., He, X., Zhang, Z., & Sun, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.11001 · Cited by 287 · cited 2×
“Finally, discriminators for whether text is machine-generated are a promising research direction in adversarial training (Lin et al., 2017; Li et al., 2017) and in”
BackgroundBest practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · cited 1×
“The related (but not identical) task of asking human raters to judge the quality of machine-generated excerpts remains the gold-standard for evaluating open-domain generation systems (van der Lee et al., 2019).”
BackgroundSerf and turf: crowdturfing for fun and profit
Wang, G., Wilson, C., Zhao, X., Zhu, Y., Mohanlal, M., Zheng, H., & Zhao, B. Y. (2012) · Proceedings of the 21st international conference on World Wide Web · doi:10.1145/2187836.2187928 · Cited by 197 · cited 2×
“, 2018), influences elections (Allcott and Gentzkow, 2017), and undermines user trust (Wang et al., 2012; Song et al., 2015).”
BackgroundDo Massively Pretrained Language Models Make Better Storytellers?
See, A., Pappu, A., Saxena, R., Yerukola, A., & Manning, C. D. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1079 · Cited by 139 · cited 1×
“In fact, See et al. (2019) note that it takes setting k to 1000 to achieve about the same amount of rare word usage and fraction of non-stopword text as as human writing.3 This makes it very easy for the model to pick out machine-generated text based on these distributional differences.”
BackgroundAdelani, D. I., Mai, H., Fang, F., Nguyen, H. H., Yamagishi, J., & Echizen, I. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-44041-1_114 · Cited by 121 · cited 2×
“State-of-the-art generative language models are now capable of producing multi-paragraph excerpts that at a surface level are virtually indistinguishable from human-written content (Zellers et al., 2019; Radford et al., 2019; Adelani et al., 2020).”
BackgroundThe limitations of stylometry for detecting machine-generated fake news
Schuster, T., Schuster, R., Shah, D. J., & Barzilay, R. (2020) · Computational Linguistics · doi:10.1162/coli_a_00380 · Cited by 109 · cited 1×
“Schuster et al. (2019) argue that the language distributional features implicitly or explicitly employed by these detectors are insufficient; instead, one should look to explicit fact-verification models.”
BackgroundAdversarial Evaluation of Dialogue ModelsPreprint
Kannan, A., & Vinyals, O. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.08198 · Cited by 88 · cited 2×
“automatic evaluation of generative model quality (Novikova et al., 2017; Kannan and Vinyals, 2017; Lowe et al., 2017).”
BackgroundReal or Fake? Learning to Discriminate Machine from Human Generated Text
Bakhtin, A., Gross, S., Ott, M., Deng, Y., Ranzato, M., & Szlam, A. (2019) · arXiv · doi:10.48550/arxiv.1906.03351 · Cited by 77 · cited 1×
“Bakhtin et al. (2019) frame human-text detection as a ranking task and evaluate their models’ cross-domain and cross-model generalization, finding significant loss in quality when training on one domain and evaluating on another.”
BackgroundCrowdTarget: Target-based Detection of Crowdturfing in Online Social Networks
Song, J., Lee, S., & Kim, J. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813661 · Cited by 50 · cited 2×
“, 2018), influences elections (Allcott and Gentzkow, 2017), and undermines user trust (Wang et al., 2012; Song et al., 2015).”
BackgroundFake News and Alternative Facts: Information Literacy in a Post-Truth Era
Cooke, N. A. (2018) · Cited by 49
Are We Safe Yet? The Limitations of Distributional Features for Fake News Detection.Preprint
Schuster, T., Schuster, R., Shah, D., & Barzilay, R. (2019) · arXiv (Cornell University) · Cited by 13
Miculicich, L., Marone, M., & Hassan, H. (2019) · Workshop on Neural Generation and Translation (WNGT) · doi:10.18653/v1/d19-5633 · Cited by 7 · cited 2×
“5 (Miculicich et al., 2019)), the distributional deviations that plague top-k text will surface in nucleus sampling as well.”
BackgroundInfluential
Research works printed but not linked: 4
- Timothy Niven and Hung-Yu Kao. 2019. Probing neu- ral network comprehension of natural language ar- guments. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguis- tics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725.
- Alan Turing. 1950. Computing machinery and intelligence-am turing. Mind, 59(236):433.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. CoRR, abs/1905.12616. · arXiv
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Perkins, M. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.02.07 · Cited by 839
“Solaiman et al. (2019) and Ippolito et al.(2020) also both present tools which showed an encouraging ability in detecting machine created text, although these tools were not tested against the latest generation of LLMs. Fröhling & Zubiaga (2021) present a promising low cost detection model which is…”
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727
Human heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342
“Recent work by Ippolito et al. (35) suggests that language model–decoding methods have been optimized for fooling humans at the cost of introducing statistical anomalies easily detected by machines.”
BackgroundInfluentialA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Metadata
- Authors
- Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D.
- Affiliations
- University of Pennsylvania (US)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Pages
- 1808-1822
- ISSN
- 0736-587X (ISSN-L)
- arXiv
- 1911.00650
- OpenAlex
- W3021153741
- Cited by
- 264
- References
- 31
- Citations by year
- 2026: 39 · 2025: 64 · 2024: 38 · 2023: 63 · 2022: 18 · 2021: 29 · 2020: 13
- FWCI
- 11.75
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- AI-generated text detection, decoding strategies, top-k sampling, nucleus sampling, random sampling, human raters, text generation evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
