The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Open accessAbstract
Recent advancements in neural language modelling make it possible to rapidly generate vast amounts of human-sounding text.The capabilities of humans and automatic discriminators to detect machine-generated text have been a large source of research interest, but humans and machines rely on different cues to make their decisions.Here, we perform careful benchmarking and analysis of three popular sampling-based decoding strategies-topk, nucleus sampling, and untruncated random sampling-and show that improvements in decoding methods have primarily optimized for fooling humans.This comes at the expense of introducing statistical abnormalities that make detection easy for automatic systems.We also show that though both human and automatic detector performance improve with longer excerpt length, even multi-sentence excerpts can fool expert human raters over 30% of the time.Our findings reveal the importance of using both human and automatic detectors to assess the humanness of text generation systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs typically exhibit less diversity in word usage compared to humans [Ippolito et al., 2020;Dugan et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Pre-trained language models (PTMs), especially BERT-based architectures [Devlin et al., 2019] such as BERT [Ippolito et al., 2020;Fabien et al., 2020;Manolache et al., 2021], Sentence-BERT [Schlicht and de Paula, 2021;Rivera-Soto et al., 2021], and RoBERTa [Huertas-Tato et al., 2022], have proven effective for learning authorship representation.”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, neural networkbased models such as BERT [Ippolito et al., 2020], Radar [Hu et al., 2023], and GPT-sentinel [Chen et al., 2023a] can also be applied.”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, exposing evaluators to examples before evaluation enhances their detection capabilities, especially with longer samples.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Ippolito et al. 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Neural network-based detectors, particularly those utilizing BERT, have proven effective in distinguishing between human-written texts and those generated by GPT-2 [Ippolito et al., 2020;Liu et al., 2019].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“In contrast, detection algorithms frequently outperform the human in this task [Ippolito et al., 2020].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Existing detectors also lack robustness to various factors, such as alternative decoding strategies [Ippolito et al., 2020], input sequence length [Solaiman et al., 2019], different prompts [Kumarage et al., 2023;Lu et al., 2023], and repetition penalties [Fishchuk and Braun, 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Recent studies (Ippolito et al. 2020) indicated that human evaluators might not be as proficient as detection algorithms in recognizing LLM-generated text across various settings.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Ippolito et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Ippolito et al. (2020) suggested that text perceived as high quality by humans tends to be more easily recognizable by detectors.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Ippolito et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 2×
“It propogates quickly (Vosoughi et al., 2018), sets political agendas”
BackgroundSocial Media and Fake News in the 2016 Election
Allcott, H., & Gentzkow, M. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.211 · Cited by 6,767
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Fake It Till You Make It: Reputation, Competition, and Yelp Review Fraud
Luca, M., & Zervas, G. (2016) · Management Science · doi:10.1287/mnsc.2015.2304 · Cited by 1,263
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Adversarial Learning for Neural Dialogue Generation
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., & Jurafsky, D. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1230 · Cited by 955
Research works printed but not linked: 4
- Timothy Niven and Hung-Yu Kao. 2019. Probing neu- ral network comprehension of natural language ar- guments. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguis- tics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725.
- Alan Turing. 1950. Computing machinery and intelligence-am turing. Mind, 59(236):433.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. CoRR, abs/1905.12616. · arXiv
Cited by
Sorted by citations
Items 301–324 of 324
Ali, U., Shoukat, M. R., Rajput, M., Bahaj, S. A., Khan, M. I., & Mahmood, T. (2026) · Scientific Reports · doi:10.1038/s41598-026-56292-0
LiMO: Line-level Machine-generated Code Detection for Diverse Programming Languages
Kim, T., Jang, H., Kim, S., Jin, Z., Kim, S., & Bak, J. (2026) · ACM Symposium on Applied Computing · doi:10.1145/3748522.3779718
Gultom, I., Puspadini, R., Erwis, F., Prasiwiningrum, E., & Ridwan (2026) · JOURNAL OF ICT APLICATIONS AND SYSTEM · doi:10.56313/jictas.v5i1.531
DualCodeDetect: Zero-Shot LLM-Generated Code Detection via Dual-Channel Perturbation
Li, Z., Shang, X., Fu, Z., Guo, S., Zhang, W., Yu, N., & Chen, K. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808166
Detecting and Comparing LLM Capabilities to Human Writers Through Linguistic Analysis
Liebe, L., Baum, J. D., Cech, T., Scheibel, W., & Döllner, J. (2026) · SN Computer Science · doi:10.1007/s42979-026-05131-0
Stojković, V. (2026) · Journal of AI-Driven Communication Engineering · doi:10.31875/2979-1081.2026.02.09
Sankarlingam, S. (2026) · Journal of High School Science · doi:10.64336/001c.164666
Linguistic Fingerprints for Authorship Attribution in Human-AI Collaborative Texts
Puspitasari, D., Lestariningsih, D. N., Sukma, B. P., Karlina, Y., Sanubarianto, S. T., Handoko, M. P., & Pradita, I. (2026) · OKARA Jurnal Bahasa dan Sastra · doi:10.19105/ojbs.v20i1.22271
Tefke, T., & Staudemeyer, R. C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-32335-4_11
Modeling Cross-Paragraph Consistency for Few-Shot AI-Generated Text Detection: A Dual-Path Framework
Jiayi, X., Liu, Y., & Wang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3423-3_5
Li, Q., Weilin, H., & Zhu, C. (2026) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3828734
Rathi, S. R., Anturkar, A., Borde, A., Araj, V., & Mahajan, A. (2026) · OPJU International Technology Conference on Smart Computing for Innovation and Advancement in Industry (OTCON) · doi:10.1109/otcon69332.2026.11629638
Amjad, M., & Tanveer, H. (2026) · Journal of Arts and Linguistics Studies · doi:10.71281/jals.v4i1.627
Detecting AI-Generated Phishing Campaigns Using Linguistic Forensics and Machine Learning
Auyezov, A., Abdigali, A., Dota, Y., Zhunussov, K., & Nurzhaubaev, B. (2026) · International Conference on Cybersecurity, Digital Forensics, and AI Applications (ICCSDFAI) · doi:10.1109/iccsdfai70505.2026.11648017
Rehman, I. U., Satti, M. I., Rehman, M. Y., & Hussain, B. (2026) · Knowledge · doi:10.3390/knowledge6030022
Islait, R., & Alhawamdeh, M. M. (2026) · IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies · doi:10.1109/aeect69724.2026.11657814
Combining probabilistic features and semantic features for AI-Generated text detection
Yu, Y., & Gao, W. (2026) · International Journal of Scientific Research and Management (IJSRM) · doi:10.18535/ijsrm/v14i09.ec01
A Multi-stage Framework for Detection and Attribution of LLM-Generated Text
Long, V. M. T., Chan, R. C. B., & Truong-Huu, T. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-37274-1_5
LLM-generated scientific content detection method
Bader, A., & Alhijawi, B. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12465-6
Wang, X. (2026) · International Conference on Artificial Intelligence, Digital Media and Education · doi:10.1145/3839592.3839722
Multi-level Text Feature Analysis for Detecting AI-Generated Content
Yan, L., Ye, H., Pan, T., & Ihnaini, B. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-28457-0_9
Varquez, C. F. C., Epis, S. N., Maratas, R. P., Docena, F. V., Ram, M. L., & Malangsa, R. D. (2026) · International Conference on Smart Electronics and Communication (ICOSEC) · doi:10.1109/icosec70158.2026.11735785
AID2: a robust and interpretable neural framework for AI-generated text detection under domain shift
Alkenani, A. H., Abdulwahid, A. A., Al‐Hoorie, A. H., El-Amrani, M. Y., AlSalehy, A. S., & Alsowaiq, H. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12621-y
Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education
Zeng, Z., Sha, L., Li, Y., Yang, K., Gašević, D., & Chen, G. (2023) · arXiv · doi:10.48550/arxiv.2307.12267 · Cited by 25
Metadata
- Authors
- Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D.
- Affiliations
- University of Pennsylvania (US)
- Google (United States) (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 1808-1822
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3021153741
- Cited by
- 264
- References
- 31
- Citations by year
- 2026: 39 · 2025: 64 · 2024: 38 · 2023: 63 · 2022: 18 · 2021: 29 · 2020: 13
- FWCI
- 11.75
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- AI-generated text detection, decoding strategies, top-k sampling, nucleus sampling, random sampling, human raters, text generation evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
