Automatic Detection of Generated Text is Easiest when Humans are Fooled
Open accessAbstract
Recent advancements in neural language modelling make it possible to rapidly generate vast amounts of human-sounding text.The capabilities of humans and automatic discriminators to detect machine-generated text have been a large source of research interest, but humans and machines rely on different cues to make their decisions.Here, we perform careful benchmarking and analysis of three popular sampling-based decoding strategies-topk, nucleus sampling, and untruncated random sampling-and show that improvements in decoding methods have primarily optimized for fooling humans.This comes at the expense of introducing statistical abnormalities that make detection easy for automatic systems.We also show that though both human and automatic detector performance improve with longer excerpt length, even multi-sentence excerpts can fool expert human raters over 30% of the time.Our findings reveal the importance of using both human and automatic detectors to assess the humanness of text generation systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs typically exhibit less diversity in word usage compared to humans.
“LLMs typically exhibit less diversity in word usage compared to humans [Ippolito et al., 2020;Dugan et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
Section 7 (Automatic Detection Results; page not provided) finds that top-k samples concentrate much more probability on common tokens than human writing, consistent with less diverse word usage. But it also says random and nucleus sampling are very similar to human-written text on this measure, so the finding does not support the claim about LLMs generally. (LLM, conf=0.88)
Experimental — accuracy not yet measuredPre-trained language models, especially BERT-based architectures, have proven effective for learning authorship representation.
“Pre-trained language models (PTMs), especially BERT-based architectures [Devlin et al., 2019] such as BERT [Ippolito et al., 2020;Fabien et al., 2020;Manolache et al., 2021], Sentence-BERT [Schlicht and de Paula, 2021;Rivera-Soto et al., 2021], and RoBERTa [Huertas-Tato et al., 2022], have proven effective for learning authorship representation.”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
The Automatic Detection Method section describes fine-tuning BERT to classify text as human- or machine-generated, and Automatic Detection Results reports its accuracy. This supports the effectiveness of BERT for that detection task, but the paper does not evaluate learning representations of individual authorship. (LLM, conf=0.84)
Experimental — accuracy not yet measuredNeural network-based models such as BERT, Radar, and GPT-sentinel can also be applied to human-LLM co-authored text attribution.
“Additionally, neural networkbased models such as BERT [Ippolito et al., 2020], Radar [Hu et al., 2023], and GPT-sentinel [Chen et al., 2023a] can also be applied.”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
The Automatic Detection Method section says the authors fine-tuned BERT to label sentences as human- or machine-generated, supporting the BERT portion of the citation. It does not establish that BERT can attribute human–LLM co-authored text; Radar and GPT-sentinel have separate citations and are not judged here. (LLM, conf=0.86)
Experimental — accuracy not yet measuredExposing evaluators to examples before evaluation enhances their detection capabilities, especially with longer samples.
“However, exposing evaluators to examples before evaluation enhances their detection capabilities, especially with longer samples.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Ippolito et al. 2020
Section 6 (Human Detection Method; page unavailable) reports that university-student raters shown ten examples before evaluation exceeded 70% accuracy on the longest excerpts, while untrained Mechanical Turk raters hovered around 50%. This supports an association, especially for long samples, but the groups differed as well as their preparation; the paper does not establish that exposure to examples caused the improvement. (LLM, conf=0.85)
Experimental — accuracy not yet measuredNeural network-based detectors, particularly those utilizing BERT, have proven effective in distinguishing between human-written texts and those generated by GPT-2.
“Neural network-based detectors, particularly those utilizing BERT, have proven effective in distinguishing between human-written texts and those generated by GPT-2 [Ippolito et al., 2020;Liu et al., 2019].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
Detection algorithms frequently outperform humans in distinguishing LLM-generated from human-written text.
“In contrast, detection algorithms frequently outperform the human in this task [Ippolito et al., 2020].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
Existing detectors lack robustness to alternative decoding strategies.
“Existing detectors also lack robustness to various factors, such as alternative decoding strategies [Ippolito et al., 2020], input sequence length [Solaiman et al., 2019], different prompts [Kumarage et al., 2023;Lu et al., 2023], and repetition penalties [Fishchuk and Braun, 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Ippolito et al., 2020
Human evaluators might not be as proficient as detection algorithms in recognizing LLM-generated text across various settings.
“Recent studies (Ippolito et al. 2020) indicated that human evaluators might not be as proficient as detection algorithms in recognizing LLM-generated text across various settings.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Ippolito et al. 2020
Ippolito et al. (2020) suggested that text perceived as high quality by humans tends to be more easily recognizable by detectors.
“Ippolito et al. (2020) suggested that text perceived as high quality by humans tends to be more easily recognizable by detectors.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Ippolito et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 2× · see citing works
“It propogates quickly (Vosoughi et al., 2018), sets political agendas”
BackgroundSocial Media and Fake News in the 2016 Election
Allcott, H., & Gentzkow, M. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.211 · Cited by 6,767 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · see citing works
Fake It Till You Make It: Reputation, Competition, and Yelp Review Fraud
Luca, M., & Zervas, G. (2016) · Management Science · doi:10.1287/mnsc.2015.2304 · Cited by 1,263 · see citing works
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242 · see citing works
Adversarial Learning for Neural Dialogue Generation
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., & Jurafsky, D. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1230 · Cited by 880 · see citing works
Vargo, C. J., Гуо, Л., & Amazeen, M. A. (2017) · New Media & Society · doi:10.1177/1461444817712086 · Cited by 638 · see citing works
Generating Wikipedia by Summarizing Long SequencesPreprint
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, Ł., & Shazeer, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10198 · Cited by 578 · see citing works
GLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3019 · Cited by 470 · cited 9× · see citing works
“Kreps et al. (2020), Gehrmann et al. (2019), and others have stressed the importance of humans being able to identify fake content on the web.”
BackgroundMethodInfluential(2020) · arXiv (Cornell University) · Cited by 461 · see citing works
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422 · see citing works
Probing Neural Network Comprehension of Natural Language Arguments
Niven, T., & Kao, H.-Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1459 · Cited by 395 · see citing works
Adversarial Ranking for Language GenerationPreprint
Lin, K., Li, D., He, X.-D., Zhang, Z.-Y., & Sun, M.-T. (2017) · Neural Information Processing Systems · doi:10.48550/arxiv.1705.11001 · Cited by 354 · see citing works
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Lowe, R., Noseworthy, M., Serban, I. V., Angelard-Gontier, N., Bengio, Y., & Pineau, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1103 · Cited by 341 · see citing works
Release Strategies and the Social Impacts of Language Models
Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1908.09203 · Cited by 292 · see citing works
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · cited 1× · see citing works
“The related (but not identical) task of asking human raters to judge the quality of machine-generated excerpts remains the gold-standard for evaluating open-domain generation systems (van der Lee et al., 2019).”
BackgroundSerf and turf: crowdturfing for fun and profit
Wang, G., Wilson, C., Zhao, X., Zhu, Y., Mohanlal, M., Zheng, H., & Zhao, B. Y. (2012) · Proceedings of the 21st international conference on World Wide Web · doi:10.1145/2187836.2187928 · Cited by 197 · see citing works
Do Massively Pretrained Language Models Make Better Storytellers?
See, A., Pappu, A., Saxena, R., Yerukola, A., & Manning, C. D. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1079 · Cited by 139 · see citing works
The limitations of stylometry for detecting machine-generated fake news
Schuster, T., Schuster, R., Shah, D. J., & Barzilay, R. (2020) · Computational Linguistics · doi:10.1162/coli_a_00380 · Cited by 109 · see citing works
Adelani, D. I., Mai, H., Fang, F., Nguyen, H. H., Yamagishi, J., & Echizen, I. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-44041-1_114 · Cited by 108 · see citing works
Real or Fake? Learning to Discriminate Machine from Human Generated Text
Bakhtin, A., Gross, S., Ott, M., Deng, Y., Ranzato, M., & Szlam, A. (2019) · arXiv · doi:10.48550/arxiv.1906.03351 · Cited by 77 · see citing works
Adversarial Evaluation of Dialogue ModelsPreprint
Kannan, A., & Vinyals, O. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.08198 · Cited by 65 · see citing works
CrowdTarget: Target-based Detection of Crowdturfing in Online Social Networks
Song, J., Lee, S., & Kim, J. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813661 · Cited by 50 · see citing works
Fake News and Alternative Facts: Information Literacy in a Post-Truth Era
Cooke, N. A. (2018) · Cited by 49 · see citing works
Are We Safe Yet? The Limitations of Distributional Features for Fake News Detection.Preprint
Schuster, T., Schuster, R., Shah, D., & Barzilay, R. (2019) · arXiv (Cornell University) · Cited by 13 · see citing works
Miculicich, L., Marone, M., & Hassan, H. (2019) · Workshop on Neural Generation and Translation (WNGT) · doi:10.18653/v1/d19-5633 · Cited by 7 · see citing works
Flagged first, then by citations
1 of the 31 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Timothy Niven and Hung-Yu Kao. 2019. Probing neu- ral network comprehension of natural language ar- guments. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguis- tics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725.
- Alan Turing. 1950. Computing machinery and intelligence-am turing. Mind, 59(236):433.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. CoRR, abs/1905.12616. · arXiv
Cited by
Sorted by citations
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Perkins, M. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.02.07 · Cited by 839 · see citing works
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationTesting of detection tools for AI-generated text
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023) · International Journal for Educational Integrity · doi:10.1007/s40979-023-00146-z · Cited by 582 · see citing works
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369 · see citing works
Human heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342 · see citing works
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290 · see citing works
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
Perkins, M., Roe, J., Postma, D., McGaughran, J., & Hickerson, D. (2024) · Journal of Academic Ethics · doi:10.1007/s10805-023-09492-6 · Cited by 234 · see citing works
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212 · see citing works
Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E. N., Japkowicz, N., & Viktor, H. L. (2023) · IEEE Access · doi:10.1109/access.2023.3294090 · Cited by 196 · see citing works
TweepFake: about detecting deepfake tweets
Fagni, T., Falchi, F., Gambini, M., Martella, A., & Tesconi, M. (2021) · PLOS ONE · doi:10.1371/journal.pone.0251415 · Cited by 185 · see citing works
Can AI-Generated Text be Reliably Detected?
Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11156 · Cited by 161 · see citing works
“Non-watermarking detectors such as trained classifiers [3], retrieval-based detectors [4], and zero-shot classifiers [2, 24, 23, 22] use the presence of LLM-specific signatures in AI-generated texts for their detection.”
BackgroundMethodInfluentialAutomatic Detection of Machine Generated Text: A Critical Survey
Jawahar, G., Abdul-Mageed, M., & Lakshmanan, V. S. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.208 · Cited by 156 · see citing works
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2301.11305 · Cited by 151 · see citing works
Large language models: a survey of their development, capabilities, and applications
Yadagiri, A., & Pakray, P. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02310-4 · Cited by 135 · see citing works
A Survey on the Detection and Impacts of Deepfakes in Visual, Audio, and Textual Formats
Mubarak, R., Alsboui, T., Alshaikh, O., Inuwa-Dutse, I., Khan, S., & Parkinson, S. (2023) · IEEE Access · doi:10.1109/access.2023.3344653 · Cited by 132 · see citing works
ChatGPT for Cybersecurity: Practical Applications, Challenges, and Future Directions
Al-Hawawreh, M., Aljuhani, A., & Jararweh, Y. (2023) · Cluster Computing · doi:10.1007/s10586-023-04124-5 · Cited by 124 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
Simple techniques to bypass GenAI text detectors: implications for inclusive education
Perkins, M., Roe, J., Vu, B. H., Postma, D., Hickerson, D., McGaughran, J., & Khuat, H. Q. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00487-w · Cited by 117 · see citing works
Rethinking search: making domain experts out of dilettantes
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · ACM SIGIR Forum · doi:10.1145/3476415.3476428 · Cited by 112 · see citing works
Feature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover
Fröhling, L., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.443 · Cited by 106 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding
Abdelnabi, S., & Fritz, M. (2021) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40001.2021.00083 · Cited by 105 · see citing works
DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text
Su, J., Zhuo, T. Y., Wang, D., & Nakov, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.827 · Cited by 105 · see citing works
MGTBench: Benchmarking Machine-Generated Text Detection
He, X., Shen, X., Chen, Z., Michael Backes, A., & Zhang, Y. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3670344 · Cited by 105 · see citing works
Contrasting Linguistic Patterns in Human and LLM-Generated News Text
Muñoz-Ortiz, A., Gómez-Rodríguez, C., & Vilares, D. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10903-2 · Cited by 94 · see citing works
TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation
Uchendu, A., Ma, Z., Le, T., Zhang, R., & Lee, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.172 · Cited by 82 · see citing works
Quantifying large language model usage in scientific papers
Liang, W., Zhang, Y., Wu, Z., Lepp, H., Ji, W., Zhao, X., Cao, H., Liu, S., He, S., Huang, Z., et al. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02273-8 · Cited by 80 · see citing works
Shaaban, O. A., Yıldırım, R., & Alguttar, A. A. (2023) · IEEE Access · doi:10.1109/access.2023.3333866 · Cited by 79 · see citing works
Detecting computer-generated disinformation
Stiff, H., & Johansson, F. (2022) · International Journal of Data Science and Analytics · doi:10.1007/s41060-021-00299-5 · Cited by 79 · see citing works
Embracing ChatGPT: Implications of Emergent Language Models for Academia and Libraries
Houston, A. B., & Corrado, E. M. (2023) · Technical Services Quarterly · doi:10.1080/07317131.2023.2187110 · Cited by 77 · see citing works
Liang, W., Izzo, Z., Zhang, Y., Lepp, H., Cao, H., Zhao, X., Chen, L., Ye, H., Liu, S., Huang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.07183 · Cited by 73 · see citing works
“…risks associated with AI-generated content has sparked a debate about their effectiveness and reliability (OpenAI, 2019; Jawahar et al., 2020; Fagni et al., 2021; Ippolito et al., 2019; Mitchell et al., 2023b; Gehrmann et al., 2019; Heikkil ¨ a, 2022; Crothers et al., 2022; Solaiman et al., 2019a).”
Ibrahim, K. (2023) · Language Testing in Asia · doi:10.1186/s40468-023-00260-2 · Cited by 66 · see citing works
Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?
Bhattacharjee, A., & Liu, H. (2024) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3655103.3655106 · Cited by 65 · see citing works
CHEAT: A Large-Scale Dataset for Detecting CHatGPT-writtEn AbsTracts
Yu, P., Chen, J., Feng, X., & Xia, Z. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536929 · Cited by 65 · see citing works
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369 · see citing works
Liu, Z. Q., Yao, Z., Li, F., & Luo, B. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3670392 · Cited by 63 · see citing works
Deepfake Text Detection: Limitations and Opportunities
Pu, J., Sarwar, Z., Abdullah, S. M., Rehman, A., Kim, Y., Bhattacharya, P., Javed, M., & Viswanath, B. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179387 · Cited by 62 · see citing works
Identifying and Mitigating the Security Risks of Generative AI
Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al. (2023) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000041 · Cited by 57 · see citing works
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Karpinska, M., Akoury, N., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.97 · Cited by 56 · see citing works
Sadiq, S., Aljrees, T., & Ullah, S. (2023) · IEEE Access · doi:10.1109/access.2023.3308515 · Cited by 56 · see citing works
Koike, R., Kaneko, M., & Okazaki, N. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30120 · Cited by 55 · see citing works
Accessible Cultural Heritage through Explainable Artificial Intelligence
Díaz-Rodríguez, N., & Pisoni, G. (2020) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3386392.3399276 · Cited by 55 · see citing works
Dugan, L., Ippolito, D., Kirubarajan, A., Shi, S., & Callison-Burch, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26501 · Cited by 54 · see citing works
Self-citationNeural Deepfake Detection with Factual Structure of Text
Zhong, W., Tang, D., Xu, Z., Wang, R., Duan, N., Zhou, M., Wang, J., & Yin, J. P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.193 · Cited by 54 · see citing works
An Overview of Language Models: Recent Developments and Outlook
Wei, C., Wang, Y.-C., Wang, B., & Kuo, C. ‐. J. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000010 · Cited by 55 · see citing works
SeqXGPT: Sentence-Level AI-Generated Text Detection
Wang, P., Li, L., Ren, K., Jiang, B., Zhang, D., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.73 · Cited by 53 · see citing works
Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment
Richards, M., Waugh, K., Slaymaker, M., Petre, M., Woodthorpe, J., & Gooch, D. (2024) · ACM Transactions on Computing Education · doi:10.1145/3633287 · Cited by 52 · see citing works
Do Language Models Plagiarize?
Lee, J., Le, T., Chen, J., & Lee, D. (2023) · Proceedings of the ACM Web Conference 2023 · doi:10.1145/3543507.3583199 · Cited by 51 · see citing works
Trading Off Diversity and Quality in Natural Language GenerationPreprint
Zhang, H., Duckworth, D., Ippolito, D., & Neelakantan, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.10450 · Cited by 50 · see citing works
Self-citationRAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
Dugan, L., Hwang, A., Trhlík, F., Zhu, A., Ludan, J. M., Xu, H., Ippolito, D., & Callison-Burch, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.674 · Cited by 48 · see citing works
Self-citationShah, A., Ranka, P., Dedhia, U., Prasad, S., Muni, S., & Bhowmick, K. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.01410110 · Cited by 47 · see citing works
A Comparison of DeepSeek and other LLMs
Gao, T., Jin, J., Ke, Z. T., & Moryoussef, G. (2026) · The American Statistician · doi:10.1080/00031305.2025.2611010 · Cited by 45 · see citing works
Kovács, B. (2024) · Marketing Letters · doi:10.1007/s11002-024-09729-3 · Cited by 44 · see citing works
Liu, X., Zhang, Z., Wang, Y., Pu, H., Lan, Y., & Shen, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1005 · Cited by 44 · see citing works
ConDA: Contrastive Domain Adaptation for AI-generated Text Detection
Bhattacharjee, A., Kumarage, T., Moraffah, R., & Liu, H. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.40 · Cited by 43 · see citing works
A Survey on Detection of LLMs-Generated Content
Yang, X., Pan, L., Zhao, X., Chen, H., Petzold, L. R., Wang, W. Y., & Cheng, W. D. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.572 · Cited by 41 · see citing works
RoFT: A Tool for Evaluating Human Detection of Machine-Generated Text
Dugan, L. D., Ippolito, D., Kirubarajan, A., & Callison-Burch, C. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.25 · Cited by 41 · see citing works
Self-citationCross-domain detection of GPT-2-generated technical text
Rodriguez, J., Hay, T., Gros, D., Shamsi, Z., & Srinivasan, R. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.88 · Cited by 40 · see citing works
Three Bricks to Consolidate Watermarks for Large Language Models
Fernandez, P., Chaffin, A., Tit, K., Chappelier, V., & Furon, T. (2023) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs58808.2023.10374576 · Cited by 39 · see citing works
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38 · see citing works
Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges
Huang, B., Chen, C., & Shu, K. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3715073.3715076 · Cited by 42 · see citing works
“For instance, Kowalczyk et al. (2022) detected GPT-2-generated fake reviews using Shapley Additive Explanations (SHAP) (Lundberg and Lee, 2017).”
InfluentialChein, J., Martinez, S., & Barone, A. (2024) · doi:10.21203/rs.3.rs-4277893/v1 · Cited by 38 · see citing works
Chein, J. M., Martinez, S. A., & Barone, A. R. (2024) · Scientific Reports · doi:10.1038/s41598-024-76218-y · Cited by 38 · see citing works
Madani, M., Motameni, H., & Roshani, R. (2023) · International Journal of Information Technology & Decision Making · doi:10.1142/s0219622023500347 · Cited by 37 · see citing works
Hanley, H. W. A., & Durumeric, Z. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31333 · Cited by 36 · see citing works
How Large Language Models are Transforming Machine-Paraphrase Plagiarism
Wahle, J. P., Ruas, T., Kirstein, F., & Gipp, B. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.62 · Cited by 35 · see citing works
Does Human Collaboration Enhance the Accuracy of Identifying LLM-Generated Deepfake Texts?
Uchendu, A., Lee, J., Shen, H., Le, T., Huang, T.-H., & Lee, D. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27557 · Cited by 35 · see citing works
Frustratingly Easy Edit-based Linguistic Steganography with a Masked Language Model
Ueoka, H., Murawaki, Y., & Kurohashi, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.433 · Cited by 35 · see citing works
Artificial Text Detection via Examining the Topology of Attention Maps
Kushnareva, L., Cherniavskii, D., Mikhailov, V., Artemova, E., Barannikov, S., Bernstein, A., Piontkovskaya, I., Piontkovski, D., & Burnaev, E. V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.50 · Cited by 35 · see citing works
Adversarial Semantic Collisions
Song, C., Rush, A. M., & Shmatikov, V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.344 · Cited by 34 · see citing works
Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective
Uchendu, A., Le, T., & Lee, D. (2023) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3606274.3606276 · Cited by 34 · see citing works
Mirostat: A Neural Text Decoding Algorithm that Directly Controls PerplexityPreprint
Basu, S., Ramachandran, G. S., Keskar, N. S., & Varshney, L. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14966 · Cited by 34 · see citing works
Neural Language Generation: Formulation, Methods, and EvaluationPreprint
Gârbacea, C., & Mei, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.15780 · Cited by 34 · see citing works
Truncation Sampling as Language Model Desmoothing
Hewitt, J., Manning, C. D., & Liang, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.249 · Cited by 31 · see citing works
A Comprehensive Survey of Fake Text Detection on Misinformation and LM-Generated Texts
Kwon, S.-C., & Jang, B. (2025) · IEEE Access · doi:10.1109/access.2025.3538805 · Cited by 28 · see citing works
Preventing and Detecting Misinformation Generated by Large Language Models
Liu, A., Sheng, Q., & Hu, X. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3661377 · Cited by 27 · see citing works
Detection of AI-Generated Texts: A Bi-LSTM and Attention-Based Approach
Blake, J., Miah, A. S. M., Kredens, K., & Shin, J. (2025) · IEEE Access · doi:10.1109/access.2025.3562750 · Cited by 27 · see citing works
Synthetic Disinformation Attacks on Automated Fact Verification Systems
Du, Y., Bosselut, A., & Manning, C. D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21302 · Cited by 27 · see citing works
Tactics, Threats & Targets: Modeling Disinformation and its Mitigation
Mirza, M. S., Begum, L., Niu, L., Pardo, S., Abouzeid, A., Papotti, P., & Pöpper, C. (2023) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2023.23657 · Cited by 28 · see citing works
Unsupervised and Distributional Detection of Machine-Generated TextPreprint
Gallé, M., Rozen, J., Kruszewski, G., & Elsahar, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02878 · Cited by 27 · see citing works
Enhancing Machine-Generated Text Detection: Adversarial Fine-Tuning of Pre-Trained Language Models
Lee, D., & Jang, B. (2024) · IEEE Access · doi:10.1109/access.2024.3396820 · Cited by 26 · see citing works
Can Machines Tell Stories? A Comparative Study of Deep Neural Language Models and Metrics
Das, A., & Verma, R. M. (2020) · IEEE Access · doi:10.1109/access.2020.3023421 · Cited by 26 · see citing works
Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education
Zeng, Z., Sha, L., Li, Y., Yang, K., Gašević, D., & Chen, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30258 · Cited by 25 · see citing works
A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation
Nadeem, M., He, T., Cho, K., & Glass, J. R. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.36 · Cited by 24 · see citing works
Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmers
Shi, Y., Zhang, H., Wan, C., & Gu, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00005 · Cited by 23 · see citing works
Detection of Fake Generated Scientific Abstracts
Theocharopoulos, P. C., Anagnostou, P., Tsoukala, A., Georgakopoulos, S. V., Tasoulis, S. K., & Plagianakos, V. P. (2023) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice58306.2023.00011 · Cited by 23 · see citing works
MARKMyWORDS: Analyzing and Evaluating Language Model Watermarks
Piet, J., Sitawarin, C., Fang, V., Mu, N., & Wagner, D. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00012 · Cited by 22 · see citing works
Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian
Shamardina, T., Mikhailov, V., Chernianskii, D., Fenogenova, A., Saidov, M., Valeeva, A., Shavrina, T., Smurov, I., Tutubalina, E. V., & Artemova, E. (2022) · Computational Linguistics and Intellectual Technologies · doi:10.28995/2075-7182-2022-21-497-511 · Cited by 22 · see citing works
Residual Energy-Based Models for TextPreprint
Bakhtin, A., Deng, Y., Gross, S., Ott, M., Ranzato, M., & Szlam, A. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.10188 · Cited by 22 · see citing works
The “Code” of Ethics: A Holistic Audit of AI Code Generators
Ma, W., Song, Y., Xue, M., Wen, S., & Xiang, Y. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3367737 · Cited by 21 · see citing works
How Does the Disclosure of AI Assistance Affect the Perceptions of Writing?
Li, Z., Liang, C., Peng, J., & Yin, M. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.279 · Cited by 21 · see citing works
Reverse Engineering Configurations of Neural Text Generation Models
Tay, Y., Bahri, D., Zheng, C., Brunk, C. A., Metzler, D., & Tomkins, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.25 · Cited by 21 · see citing works
AI-generated text detection: A comprehensive review of methods, datasets, and applications
Kehkashan, T., Riaz, R. A., Al‐Shamayleh, A. S., Akhunzada, A., Ali, N., Hamza, M., & Akbar, F. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100793 · Cited by 21 · see citing works
Few-shot Controllable Style Transfer for Low-Resource Multilingual Settings
Krishna, K., Nathani, D., Garcia, X., Samanta, B., & Talukdar, P. P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.514 · Cited by 20 · see citing works
A Benchmark Dataset to Distinguish Human-Written and Machine-Generated Scientific Papers
Abdalla, M. H. I., Malberg, S., Dementieva, D., Mosca, E., & Groh, G. (2023) · Information · doi:10.3390/info14100522 · Cited by 20 · see citing works
- Authors
- Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D.
- Affiliations
- University of Pennsylvania (US)
- Google (United States) (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 1808-1822
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3021153741
- Cited by
- 264
- References
- 31
- Citations by year
- 2026: 39 · 2025: 64 · 2024: 38 · 2023: 63 · 2022: 18 · 2021: 29 · 2020: 13
- FWCI
- 11.75
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- AI-generated text detection, decoding strategies, top-k sampling, nucleus sampling, random sampling, human raters, text generation evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
