A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Open accessThe DOI matches its Crossref record; 301 of the 303 references it cites were found in a registry or an index; 71 of the 197 sources its claims cite were checked for support.
Abstract
The remarkable ability of large language models (LLMs) to comprehend, interpret, and generate complex language has rapidly integrated LLM-generated text into various aspects of daily life, where users increasingly accept it. However, the growing reliance on LLMs underscores the urgent need for effective detection mechanisms to identify LLM-generated text. Such mechanisms are critical to mitigating misuse and safeguarding domains like artistic expression and social networks from potential negative consequences. LLM-generated text detection, conceptualized as a binary classification task, seeks to determine whether an LLM produced a given text. Recent advances in this field stem from innovations in watermarking techniques, statistics-based detectors, and neural-based detectors. Human-assisted methods also play a crucial role. In this survey, we consolidate recent research breakthroughs in this field, emphasizing the urgent need to strengthen detector research. Additionally, we review existing datasets, highlighting their limitations and developmental requirements. Furthermore, we examine various LLM-generated text detection paradigms, shedding light on challenges like out-of-distribution problems, potential attacks, real-world data issues, and ineffective evaluation frameworks. Finally, we outline intriguing directions for future research in LLM-generated text detection to advance responsible artificial intelligence. This survey aims to provide a clear and comprehensive introduction for newcomers while offering seasoned researchers valuable updates in the field.1
Overview
CoChat Summary
TL;DR
This survey reviews LLM-generated text detection, organizing methods into watermarking, statistics-based, neural-based, and human-assisted approaches, and argues robust detectors are urgently needed. It synthesizes datasets, benchmarks, evaluation metrics, and open challenges, concluding with future research directions.
Key findings
- The survey’s systematic literature review screened work up to November 2023 and identified 83 pertinent pieces of literature; most relevant research appeared in 2023.
- Detection methods are categorized into watermarking (data-driven, model-driven, post-processing), statistics-based (linguistic features, white-box, black-box), neural-based (feature classifiers, pre-trained classifiers, LLMs as detectors), and human-assisted methods.
- It reports that current detectors face out-of-distribution problems (cross-domain, cross-lingual, cross-LLMs), attacks (paraphrase, adversarial, prompt, training threat models), real-world data issues (mixed/human-edited text, data ambiguity), model-size effects, and a lack of effective evaluation framework.
- It cites selected performance figures from reviewed work: Hamed and Wu’s bigram-based system identified 98 of 100 LLM-written academic papers; LLMDet achieved 98.54% classification accuracy; GHOSTBUSTER achieved an average F1 of 99.0; fine-tuned BERT-family models averaged about 95% in-domain accuracy.
Methods
- Design: a systematic literature review following Kitchenham’s SLR framework, guided by the research question of prevailing LLM-generated text detection methods and their main challenges.
- Data: searches across Google Scholar, arXiv, Scopus, Web of Science, IEEE Xplore, Springer Link, ACL Anthology, and ACM Digital Library using Boolean search terms; focused on publicly accessible articles before November 2023; after de-duplication and manual screening, 83 works were included.
- Analysis: synthesizes datasets and benchmarks, detector paradigms, evaluation metrics, important issues, and future directions; no primary experiments are conducted.
Limitations
- Stated: current datasets and benchmarks are limited in attack types, domains, tasks, LLMs, and languages, and suffer from temporal staleness; there is a lack of effective evaluation framework; no detection technique is acknowledged as infallible; data ambiguity is not formally addressed by existing studies.
- Stated: the paper argues prior reviews are insufficient in depth and comprehensiveness, but it does not state limitations of its own survey.
- Evident: as a survey, it presents no new empirical validation, and its coverage is bounded by the search terms, databases, and the November 2023 cutoff with 83 included works.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
50 Valid · 21 Not valid · 106 Not checked · 68 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“In the initial stages of generative modeling's emergence, when it had yet to pose significant societal challenges, the emphasis was on assessing the truth or falsity of the content in LLM-generated text, with less regard for its source (Schuster et al. 2020b).”
What the sources say
- Disputedcites Schuster et al. 2020b sourceExperimental — accuracy not yet measured
The Introduction (page not supplied) describes earlier approaches as focusing on stylistic differences and identifying whether text was machine-generated, then calls this paper’s veracity-based benchmark the first of its kind. That is contrary to the claim that the early emphasis was on truth or falsity rather than source. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, it is pertinent to note some limitations of the HC3 dataset, such as the lack of diversity in prompts used for data creation.”
What the sources say
- Could not verifycites Guo et al. 2023 sourceExperimental — accuracy not yet measured
Section 4.1 (Dataset; page unavailable) describes a PubMedQA-based dataset and its two sets of generation instructions. Section 6 (Limitations and Future Work; page unavailable) discusses limitations of AuthentiGPT, but neither section addresses HC3 or a lack of diversity in prompts used to create it. Searches for HC3 and prompt diversity found no passage supporting the claim. (LLM, conf=0.74)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“undermine technical expertise (Rodriguez et al. 2022a; Aliman and Kester 2021)”
What the sources say
- Partly supportedcites Rodriguez et al. 2022a sourceExperimental — accuracy not yet measured
The Abstract says convincing synthetic text can undermine genuine research, and the Introduction says synthetic technical texts could hinder the scientific process. Those passages support a threat to scientific work, but they do not specifically say that technical expertise itself is undermined. (LLM, conf=0.78)
- Not matchedcites Aliman and Kester 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“promote plagiarism (Lee et al. 2023a)”
What the sources say
- Partly supportedcites Lee et al. 2023a sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) identifies code plagiarism as a concern associated with advances in AI code generation. It lists code vulnerabilities separately, however, and does not say that those vulnerabilities promote plagiarism. (LLM, conf=0.76)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“The Human ChatGPT Comparison Corpus (HC3) (Guo et al. 2023) stands as one of the initial open-source efforts to compare ChatGPT-generated text with human-written text.”
What the sources say
- Could not verifycites Guo et al. 2023 sourceExperimental — accuracy not yet measured
The Dataset section describes a PubMedQA-based comparison of human-written and GPT-generated text, not HC3. Searches for “HC3,” “Human ChatGPT Comparison Corpus,” and the claimed early open-source comparison found no passage addressing HC3 or its status among early efforts. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
64 more claims · 20 with their cited sources checked
Create a free account to see all 245 claims and to verify the 16 references still open.
- Probably not a substantial claim. 119
- It may state two separate claims as one. 92
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Training · p. 17
“| Yelp Dataset (Asghar 2016) | 700k |”
§ 4.1 Training · p. 19
“It comprises 1.5 million review text samples.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6. Evaluation Metrics · p. 38
“False Positive (FP) refers to the result of the positive category (LLM-generate text) incorrectly classified by the model.”
§ 6. Evaluation Metrics · p. 40
“The FPR refers to the proportion of all actual human-written samples that are incorrectly detected as LLM-generated samples.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Training · p. 17
“| Climate Change Dataset (Littman and Wrubel 2019) | 4m |”
§ 4.1 Training · p. 19
“This dataset houses the tweet IDs for 39.6 million tweets related to climate change.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Training · p. 17
“| TOEFL11 (Blanchard et al. 2013) | 12k |”
§ 4.1 Training · p. 18
“it encompasses 1.1k essay samples across 11 languages: Arabic, Chinese, French, German, Hindi, Italian, Japanese, Korean, Spanish, Telugu, and Turkish.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.2 Evaluation Benchmarks · p. 21
“| MULTITuDE (Macko et al. 2023) | train test | ~4k ~3k | ~40k ~26k”
§ 4.2 Evaluation Benchmarks · p. 22
“This dataset consists of 74k machine-generated texts and 7k human-written texts across 11 languages”
4 more findings
Create a free account to see every finding of this check.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Self-citationGenerative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
WordNet: An Electronic Lexical Database
(1998) · The MIT Press eBooks · doi:10.7551/mitpress/7287.001.0001 · Cited by 4,857
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Self-citationLarge language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
Self-citationSimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851
Bender, W., Gruhl, D., Morimoto, N., & Lu, A. (1996) · IBM Systems Journal · doi:10.1147/sj.353.0313 · Cited by 2,774
SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining
Baccianella, S., Esuli, A., & Sebastiani, F. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2jktnrykui9s · Cited by 2,666
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation
Sokolova, M., Japkowicz, N., & Szpakowicz, S. (2006) · Lecture Notes in Computer Science · doi:10.1007/11941439_114 · Cited by 1,974
TinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Hide and seek: an introduction to steganography
Provos, N., & Honeyman, P. (2003) · IEEE Security & Privacy · doi:10.1109/msecp.2003.1203220 · Cited by 1,292
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
Baayen, R. H. (2001) · Text, Speech and Language Technology · doi:10.1007/978-94-010-0844-0 · Cited by 978
Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers
Gao, J., Lanchantin, J., Soffa, M. L., & Qi, Y.-J. (2018) · 2018 IEEE Security and Privacy Workshops (SPW) · doi:10.48550/arxiv.1801.04354 · Cited by 906
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830
What ChatGPT and generative AI mean for science
Stokel-Walker, C., & Van Noorden, R. (2023) · Nature · doi:10.1038/d41586-023-00340-6 · Cited by 828
Can AI-Generated Text be Reliably Detected?
Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11156 · Cited by 810
AI models collapse when trained on recursively generated data
Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024) · Nature · doi:10.1038/s41586-024-07566-y · Cited by 793
Art and the science of generative AI
Epstein, Z., Hertzmann, A., the Investigators of Human Creativity, Akten, M., Farid, H., Fjeld, J., Frank, M. R., Groh, M., Herman, L., Leach, N., et al. (2023) · Science · doi:10.1126/science.adh4451 · Cited by 788
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 714
Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers
Gao, J., Lanchantin, J., Soffa, M. L., & Qi, Y. (2018) · 2018 IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00016 · Cited by 693
Horne, B., & Adali, S. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14976 · Cited by 688
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685
GPT detectors are biased against non-native English writers
Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023) · Patterns · doi:10.1016/j.patter.2023.100779 · Cited by 673
Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency
Ren, S., Deng, Y., He, K., & Che, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1103 · Cited by 635
A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., & Allen, J. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1098 · Cited by 598
The NarrativeQA Reading Comprehension Challenge
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00023 · Cited by 596
Testing of detection tools for AI-generated text
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023) · International Journal for Educational Integrity · doi:10.1007/s40979-023-00146-z · Cited by 582
Defending Against Neural Fake News
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1905.12616 · Cited by 569
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1804.07461 · Cited by 562
TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP
Morris, J., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., & Qi, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.16 · Cited by 556
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.09687 · Cited by 556
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29720 · Cited by 556
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Xu, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.01219 · Cited by 544
AI bot ChatGPT writes smart essays — should professors worry?
Stokel-Walker, C. (2022) · Nature · doi:10.1038/d41586-022-04397-7 · Cited by 543
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Yan, Y., Li, R., Wang, S., Zhang, F., Wu, W., & Xu, W. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.393 · Cited by 474
GLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3019 · Cited by 470
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
Hill, F., Bordes, A., Chopra, S., & Weston, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02301 · Cited by 467
GLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. M. (2019) · arXiv · doi:10.48550/arxiv.1906.04043 · Cited by 460
Unified Structure Generation for Universal Information Extraction
Lu, Y., Liu, Q., Dai, D., Xiao, X., Lin, H., Han, X., Sun, L., & Wu, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.395 · Cited by 441
DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text
Su, J., Zhuo, T. Y., Wang, D., & Nakov, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.827 · Cited by 388
Wordcraft: Story Writing With Large Language Models
Yuan, A., Coenen, A., Reif, E., & Ippolito, D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511105 · Cited by 374
ChatGPT: The End of Online Exam Integrity?
Susnjak, T. (2022) · arXiv · doi:10.48550/arxiv.2212.09292 · Cited by 370
DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated TextPreprint
Su, J., Zhuo, T. Y., Wang, D., & Nakov, P. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2306.05540 · Cited by 356
Murtaza, G., Shuib, L., Abdul Wahab, A. W., Mujtaba, G., Mujtaba, G., Nweke, H. F., Al-garadi, M. A., Zulfiqar, F., Raza, G., & Azmi, N. A. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-019-09716-5 · Cited by 342
RADAR: Robust AI-Text Detection via Adversarial LearningPreprint
Hu, X., Chen, P.-Y., & Ho, T.-Y. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2307.03838 · Cited by 338
Muric, G., Wu, Y., & Ferrara, E. (2021) · JMIR Public Health and Surveillance · doi:10.2196/30642 · Cited by 320
ELI5: long form question answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1346 · Cited by 319
How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., & Wu, Y. (2023) · arXiv · doi:10.48550/arxiv.2301.07597 · Cited by 318
Yelp Dataset Challenge: Review Rating PredictionPreprint
Asghar, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.05362 · Cited by 304
Májovský, M., Černý, M., Kasal, M., Komarc, M., & Netuka, D. (2023) · Journal of Medical Internet Research · doi:10.2196/46924 · Cited by 301
Evaluation Metrics and Evaluation
Dalianis, H. (2018) · Clinical Text Mining · doi:10.1007/978-3-319-78503-5_6 · Cited by 300
A Survey of Text Watermarking in the Era of Large Language Models
Liu, A., Pan, L., Lu, Y., Li, J., Hu, X., Zhang, X., Wen, L., King, I., Xiong, H., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3691626 · Cited by 299
Release Strategies and the Social Impacts of Language Models
Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1908.09203 · Cited by 292
Self-citationAll That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIsPreprint
Liang, Y., Wu, C., Song, T., Wu, W., Xia, Y., Liu, Y., Ou, Y., Lu, S., Ji, L., Mao, S., et al. (2023) · Intelligent Computing · doi:10.48550/arxiv.2303.16434 · Cited by 290
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
On the Reliability of Watermarks for Large Language ModelsPreprint
Kirchenbauer, J., Geiping, J., Wen, Y., Shu, M., Saifullah, K., Kong, K., Fernando, K., Saha, A., Goldblum, M., & Goldstein, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.04634 · Cited by 271
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Kang, D., Ammar, W., Dalvi, B., Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1149 · Cited by 270
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.164 · Cited by 264
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2019) · arXiv · doi:10.48550/arxiv.1911.00650 · Cited by 264
Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding
Abdelnabi, S., & Fritz, M. (2021) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40001.2021.00083 · Cited by 248
Mitrović, S., Andreoletti, D., & Ayoub, O. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.13852 · Cited by 232
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y.-X., Wang, S., Xue, Y.-F., Wang, Z.-H., Shen, L., Wang, A., Li, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2303.17568 · Cited by 227
TOEFL11: A CORPUS OF NON‐NATIVE ENGLISH
Blanchard, D., Tetreault, J., Higgins, D., Cahill, A., & Chodorow, M. (2013) · ETS Research Report Series · doi:10.1002/j.2333-8504.2013.tb02331.x · Cited by 204
Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E. N., Japkowicz, N., & Viktor, H. L. (2023) · IEEE Access · doi:10.1109/access.2023.3294090 · Cited by 196
Differentiating ChatGPT-Generated and Human-Written Medical Texts: Quantitative Study
Liao, W., Liu, Z., Dai, H., Xu, S., Wu, Z., Zhang, Y., Huang, X., Zhu, D., Cai, H., Li, Q., et al. (2023) · JMIR Medical Education · doi:10.2196/48904 · Cited by 195
A Watermark for Large Language Models
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023) · arXiv · doi:10.48550/arxiv.2301.10226 · Cited by 194
AI vs. Human -- Differentiation Analysis of Scientific Content GenerationPreprint
Ma, Y., Liu, J., Fan, Y., Cheng, Q., Huang, Y., Lu, W., & Liu, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.10416 · Cited by 190
ChatGPT: The End of Online Exam Integrity?
Susnjak, T., & McIntosh, T. (2024) · Education Sciences · doi:10.3390/educsci14060656 · Cited by 189
TweepFake: about detecting deepfake tweets
Fagni, T., Falchi, F., Gambini, M., Martella, A., & Tesconi, M. (2021) · PLOS ONE · doi:10.1371/journal.pone.0251415 · Cited by 185
Robust Multi-bit Natural Language Watermarking through Invariant Features
Yoo, K., Ahn, W., Jang, J., & Kwak, N. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.117 · Cited by 170
Desaire, H., Chua, A. E., Isom, M., Jarošová, R., & Hua, D. (2023) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2023.101426 · Cited by 166
Shen, D., Zheng, M., Shen, Y., Qu, Y., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13818 · Cited by 162
Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts
Tulchinskii, E., Kuznetsov, K., Kushnareva, L., Cherniavskii, D., Nikolenko, S., Burnaev, E. V., Barannikov, S., & Piontkovskaya, I. (2023) · neural information processing systems · doi:10.52202/075280-1706 · Cited by 162
Topkara, U., Topkara, M., & Atallah, M. J. (2006) · ACM workshop on Multimedia and security · doi:10.1145/1161366.1161397 · Cited by 158
Automatic Detection of Machine Generated Text: A Critical Survey
Jawahar, G., Abdul-Mageed, M., & Lakshmanan, V. S. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.208 · Cited by 156
Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers
Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., & Wei, F. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.247 · Cited by 154
Ibrahim, H., Liu, F., Asim, R., Battu, B., Benabderrahmane, S., Alhafni, B., Adnan, W., Alhanai, T., AlShebli, B., Baghdadi, R., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-38964-3 · Cited by 154
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2301.11305 · Cited by 151
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
Krishna, K., Song, Y., Karpinska, M., Wieting, J., & Iyyer, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1195 · Cited by 150
MGTBench: Benchmarking Machine-Generated Text Detection
He, X., Shen, X., Chen, Z., Michael Backes, A., & Zhang, Y. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3670344 · Cited by 147
A Semantic Invariant Robust Watermark for Large Language Models
Liu, A., Pan, L., Hu, X., Meng, S., & Wen, L. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.06356 · Cited by 146
RARR: Researching and revising what language models say, using language models
Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.910 · Cited by 141
Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features
Lee, B. W., Jang, Y. S., & Lee, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.834 · Cited by 136
DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios
Wu, J., Zhan, R., Wong, D., Yang, S., Yang, X., Yuan, Y., & Chao, L. (2024) · neural information processing systems · doi:10.52202/079017-3186 · Cited by 135
Self-citationUniSpaCh: A text-based data hiding method using Unicode space characters
Por, L. Y., Wong, K., & Chee, K. O. (2011) · Journal of Systems and Software · doi:10.1016/j.jss.2011.12.023 · Cited by 134
TinyBERT: Distilling BERT for Natural Language UnderstandingPreprint
Jiao, X., Yin, Y., Shang, L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10351 · Cited by 133
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.2.12 · Cited by 132
Deepfake Text Detection: Limitations and Opportunities
Pu, J., Sarwar, Z., Abdullah, S. M., Rehman, A., Kim, Y., Bhattacharya, P., Javed, M., & Viswanath, B. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179387 · Cited by 127
Unified Structure Generation for Universal Information ExtractionPreprint
Lu, Y., Liu, Q., Dai, D., Xiao, X., Lin, H., Han, X., Sun, L., & Wu, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.12277 · Cited by 127
Break-It-Fix-It: Unsupervised Learning for Program RepairPreprint
Yasunaga, M., & Liang, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06600 · Cited by 127
The Science of Detecting LLM-Generated Text
Tang, R., Chuang, Y.-N., & Hu, X. (2024) · Communications of the ACM · doi:10.1145/3624725 · Cited by 125
Reversible Natural Language Watermarking Using Synonym Substitution and Arithmetic Coding
Hao, W., Xiang, L., Li, Y., Yang, P., & Shen, X. (2018) · DR-NTU (Nanyang Technological University) · doi:10.3970/cmc.2018.03510 · Cited by 124
Tracing Text Provenance via Context-Aware Lexical Substitution
Yang, X., Zhang, J., Chen, K., Zhang, W., Ma, Z., Wang, F., & Yu, N. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21415 · Cited by 124
Dual Contrastive Learning: Text Classification via Label-Aware Data AugmentationPreprint
Chen, Q., Zhang, R., Zheng, Y., & Mao, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.08702 · Cited by 122
Authorship Attribution for Neural Text Generation
Uchendu, A., Le, T., Shu, K., & Lee, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.673 · Cited by 120
Red Teaming Language Model Detectors with Language Models
Shi, Z., Wang, Y., Yin, F., Chen, X., Chang, K.-W., & Hsieh, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00639 · Cited by 120
Watermarking Text Generated by Black-Box Language ModelsPreprint
Yang, X. D., Chen, K., Zhang, W., Liu, C., Qi, Y., Zhang, J., Fang, H., & Yu, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.08883 · Cited by 119
Differentiate ChatGPT-generated and Human-written Medical Texts
Liao, W., Liu, Z., Dai, H., Xu, S., Wu, Z., Zhang, Y., Huang, X., Zhu, D., Cai, H., Liu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.11567 · Cited by 119
Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N., & Choi, Y. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.501 · Cited by 118
Liu, X., Zhang, Z., Wang, Y., Pu, H., Lan, Y., & Shen, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1005 · Cited by 113
REMARK-LLM: A Robust and Efficient Watermarking Framework for Generative Large Language ModelsPreprint
Zhang, R., Hussain, S. S., Neekhara, P., & Koushanfar, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.12362 · Cited by 111
CHEAT: A Large-Scale Dataset for Detecting CHatGPT-writtEn AbsTracts
Yu, P., Chen, J., Feng, X., & Xia, Z. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536929 · Cited by 111
The limitations of stylometry for detecting machine-generated fake news
Schuster, T., Schuster, R., Shah, D. J., & Barzilay, R. (2020) · Computational Linguistics · doi:10.1162/coli_a_00380 · Cited by 109
Walters, W. H. (2023) · Open Information Science · doi:10.1515/opis-2022-0158 · Cited by 109
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
Liang, Y., Wu, C., Song, T., Wu, W., Xia, Y., Liu, Y., Ou, Y., Lu, S., Ji, L., Mao, S., et al. (2024) · Intelligent Computing · doi:10.34133/icomputing.0063 · Cited by 108
Generative AI entails a credit–blame asymmetry
Mann, S. P., Earp, B. D., Nyholm, S. R., Danaher, J., Møller, N., Bowman‐Smart, H., Hatherley, J. J., Koplin, J. J., Plozza, M., Rodger, D., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00653-1 · Cited by 105
The Threat of Offensive AI to Organizations
Mirsky, Y., Demontis, A., Kotak, J., Shankar, R., Gelei, D., Yang, L., Zhang, X., Pintor, M., Lee, W., Elovici, Y., et al. (2023) · Computers & Security · doi:10.1016/j.cose.2022.103006 · Cited by 97
An Unforgeable Publicly Verifiable Watermark for Large Language Models
Liu, A., Pan, L., Hu, X., Li, S., Wen, L., King, I., & Yu, P. S. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2307.16230 · Cited by 97
Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT
Mindner, L., Schlippe, T., & Schaaff, K. (2023) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-99-7947-9_12 · Cited by 95
He, Z., Zhou, B., Hao, H., Liu, A., Wang, X., Tu, Z., Zhang, Z., & Wang, R. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.226 · Cited by 95
Contrasting Linguistic Patterns in Human and LLM-Generated News Text
(2024) · doi:10.21203/rs.3.rs-4077382/v1 · Cited by 94
Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture
Cui, J., Ning, M., Li, Z., Chen, B., Yan, Y., Li, H., Ling, B., Tian, Y., & Yuan, L. (2023) · arXiv · doi:10.48550/arxiv.2306.16092 · Cited by 94
Contrasting Linguistic Patterns in Human and LLM-Generated News Text
Muñoz-Ortiz, A., Gómez-Rodríguez, C., & Vilares, D. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10903-2 · Cited by 94
Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N. A., & Choi, Y. (2021) · arXiv · doi:10.48550/arxiv.2107.01294 · Cited by 90
Munyer, T., Tanvir, A. A., Das, A., & Zhong, X. R. (2024) · IEEE Access · doi:10.1109/access.2024.3376693 · Cited by 85
A machine learning approach to the automatic evaluation of machine translation
Corston-Oliver, S. H., Gamon, M., & Brockett, C. (2001) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1073012.1073032 · Cited by 83
Self-consuming generative models go MAD
Casco-Rodriguez, J., Alemohammad, S., Luzi, L., Imtiaz, A., Babaei, H., LeJeune, D., Siahkoohi, A., & Baraniuk, R. (2023) · LatinX in AI at Neural Information Processing Systems Conference 2023 · doi:10.52591/lxai202312101 · Cited by 83
TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation
Uchendu, A., Ma, Z., Le, T., Zhang, R., & Lee, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.172 · Cited by 82
Retrieval-based Language Models and Applications
Asai, A., Min, S., Zhong, Z., & Chen, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts) · doi:10.18653/v1/2023.acl-tutorials.6 · Cited by 81
GPT-who: An Information Density-based Machine-Generated Text Detector
Venkatraman, S., Uchendu, A., & Lee, D. (2024) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2024.findings-naacl.8 · Cited by 81
Detecting computer-generated disinformation
Stiff, H., & Johansson, F. (2022) · International Journal of Data Science and Analytics · doi:10.1007/s41060-021-00299-5 · Cited by 79
Detecting LLM-Generated Text in Computing Education: Comparative Study for ChatGPT Cases
Orenstrakh, M. S., Karnalim, O., Suárez, C. A., & Liut, M. (2024) · 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00027 · Cited by 79
Koike, R., Kaneko, M., & Okazaki, N. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30120 · Cited by 78
Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective
Uchendu, A., Le, T., & Lee, D. (2023) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3606274.3606276 · Cited by 77
Real or Fake? Learning to Discriminate Machine from Human Generated Text
Bakhtin, A., Gross, S., Ott, M., Deng, Y., Ranzato, M., & Szlam, A. (2019) · arXiv · doi:10.48550/arxiv.1906.03351 · Cited by 77
Neural Deepfake Detection with Factual Structure of Text
Zhong, W., Tang, D., Xu, Z., Wang, R., Duan, N., Zhou, M., Wang, J., & Yin, J. P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.193 · Cited by 74
RoFT: A Tool for Evaluating Human Detection of Machine-Generated Text
Dugan, L. D., Ippolito, D., Kirubarajan, A., & Callison-Burch, C. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.25 · Cited by 73
Classification of human- and AI-generated texts: Investigating features for ChatGPT
Mindner, L., Schlippe, T., & Schaaff, K. (2023) · arXiv · doi:10.48550/arxiv.2308.05341 · Cited by 73
Veselovsky, V., Ribeiro, M. H., & West, R. (2023) · arXiv · doi:10.48550/arxiv.2306.07899 · Cited by 73
Model evaluation for extreme risks
Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.15324 · Cited by 66
The Age of Synthetic Realities: Challenges and Opportunities
Cardenuto, J. P., Yang, J., Padilha, R., Wan, R., Moreira, D., Li, H., Wang, S., Andaló, F., Marcel, S., & Rocha, A. D. R. (2023) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000138 · Cited by 66
LLMDet: A Third Party Large Language Models Generated Text Detection Tool
Wu, K., Pang, L., Shen, H., Cheng, X., & Chua, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.139 · Cited by 66
DetectRL: Benchmarking LLM-Generated Text Detection in Real-World ScenariosPreprint
Wu, J.-C., Zhan, R., Wong, D. F., Yang, S., Yang, X.-Y., Yuan, Y., & Chao, L. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2410.23746 · Cited by 66
Self-citationMirostat: A Neural Text Decoding Algorithm that Directly Controls PerplexityPreprint
Basu, S., Ramachandran, G. S., Keskar, N. S., & Varshney, L. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14966 · Cited by 66
HC3 Plus: A Semantic-Invariant Human ChatGPT Comparison CorpusPreprint
Su, Z., Wu, X. C., Zhou, W., Ma, G., & Hu, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.02731 · Cited by 65
Does Human Collaboration Enhance the Accuracy of Identifying LLM-Generated Deepfake Texts?
Uchendu, A., Lee, J., Shen, H., Le, T., Huang, T.-H., & Lee, D. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27557 · Cited by 65
Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?
Bhattacharjee, A., & Liu, H. (2024) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3655103.3655106 · Cited by 65
RADAR: Robust AI-Text Detection via Adversarial Learning
Hu, X., Chen, P.-Y., & Ho, T.-Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-0662 · Cited by 62
Did You Train on My Dataset? Towards Public Dataset Protection with CleanLabel Backdoor Watermarking
Tang, R., Feng, Q., Liu, N., Yang, F., & Hu, X. (2023) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3606274.3606279 · Cited by 62
WhiteSteg: a new scheme in information hiding using text steganography
Por, L. Y., Ang, T. F., & Delina, B. (2008) · Cited by 61
Token Prediction as Implicit Classification to Identify LLM-Generated Text
Chen, Y., Kang, H., Zhai, V., Li, L., Singh, R., & Raj, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.810 · Cited by 60
On the Possibilities of AI-Generated Text Detection
Chakraborty, S., Bedi, A. S., Zhu, S., An, B., Manocha, D., & Huang, F. (2023) · arXiv · doi:10.48550/arxiv.2304.04736 · Cited by 59
Liu, X.-M., Zhang, Z.-H., Wang, Y., Pu, H., Lan, Y., & Shen, C. (2022) · arXiv.org · doi:10.48550/arxiv.2212.10341 · Cited by 58
Adversarial Robustness of Neural-Statistical Features in Detection of Generative Transformers
Crothers, E., Japkowicz, N., Viktor, H. L., & Branco, P. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892269 · Cited by 56
Fake News Detection with Generated Comments for News Articles
Yanagi, Y., Orihara, R., Sei, Y., Tahara, Y., & Ohsuga, A. (2020) · IEEE International Conference on Intelligent Engineering Systems (INES) · doi:10.1109/ines49302.2020.9147195 · Cited by 56
Dugan, L., Ippolito, D., Kirubarajan, A., Shi, S., & Callison-Burch, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26501 · Cited by 54
The Science of Detecting LLM-Generated TextsPreprint
Tang, R., Chuang, Y.-N., & Hu, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.07205 · Cited by 53
SeqXGPT: Sentence-Level AI-Generated Text Detection
Wang, P., Li, L., Ren, K., Jiang, B., Zhang, D., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2310.08903 · Cited by 53
SeqXGPT: Sentence-Level AI-Generated Text Detection
Wang, P., Li, L., Ren, K., Jiang, B., Zhang, D., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.73 · Cited by 53
SLM: Learning a Discourse Language Representation with Sentence Unshuffling
Lee, H., Hudson, D. A., Lee, K., & Manning, C. D. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2010.16249 · Cited by 53
Unsupervised and Distributional Detection of Machine-Generated TextPreprint
Gallé, M., Rozen, J., Kruszewski, G., & Elsahar, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02878 · Cited by 52
Implementing BERT and fine-tuned RobertA to detect AI generated news by ChatGPTPreprint
Wang, Z., Cheng, J., Cui, C., & Yu, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.07401 · Cited by 51
Do Language Models Plagiarize?
Lee, J., Le, T., Chen, J., & Lee, D. (2022) · arXiv · doi:10.48550/arxiv.2203.07618 · Cited by 51
Content-preserving Text Watermarking through Unicode Homoglyph Substitution
Rizzo, S., Bertini, F., & Montesi, D. (2016) · International Database Engineering and Applications Symposium (IDEAS) · doi:10.1145/2938503.2938510 · Cited by 47
Shah, A., Ranka, P., Dedhia, U., Prasad, S., Muni, S., & Bhowmick, K. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.01410110 · Cited by 47
Explainable AI in Industry: Practical Challenges and Lessons Learned
Gade, K., Geyik, S., Kenthapadi, K., Mithal, V., & Taly, A. (2020) · Companion Proceedings of the Web Conference 2020 · doi:10.1145/3366424.3383110 · Cited by 46
Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks
Wang, Y., Feng, S., Hou, A., Xiao, P., Shen, C., Liu, X., Tsvetkov, Y., & He, T. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.160 · Cited by 46
CHEAT: A Large-scale Dataset for Detecting ChatGPT-writtEn AbsTracts
Yu, P., Chen, J., Feng, X., & Xia, Z. (2023) · arXiv · doi:10.48550/arxiv.2304.12008 · Cited by 45
SLM: Learning a Discourse Language Representation with Sentence Unshuffling
Lee, H., Hudson, D. A., Lee, K., & Manning, C. D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.120 · Cited by 44
Self-consuming generative models go MAD
Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., & Baraniuk, R. G. (2023) · arXiv · doi:10.48550/arxiv.2307.01850 · Cited by 44
Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts
Tulchinskii, E., Kuznetsov, K., Kushnareva, L., Cherniavskii, D., Barannikov, S., Piontkovskaya, I., Nikolenko, S., & Burnaev, E. (2023) · arXiv · doi:10.48550/arxiv.2306.04723 · Cited by 43
Soni, M., & Wade, V. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.17650 · Cited by 42
How Reliable Are AI-Generated-Text Detectors? An Assessment Framework Using Evasive Soft Prompts
Kumarage, T., Sheth, P., Moraffah, R., Garland, J., & Liu, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.94 · Cited by 42
Cross-domain detection of GPT-2-generated technical text
Rodriguez, J., Hay, T., Gros, D., Shamsi, Z., & Srinivasan, R. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.88 · Cited by 40
Explainable AI in industry: practical challenges and lessons learned: implications tutorial
Gade, K., Geyik, S. C., Kenthapadi, K., Mithal, V., & Taly, A. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3375664 · Cited by 40
Multitude: Large-scale multilingual machine-generated text detection benchmark
Macko, D., Moro, R., Uchendu, A., Lucas, J., Yamashita, M., Pikuliak, M., Srba, I., Le, T., Lee, D., Simko, J., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.616 · Cited by 38
(2008) · Cited by 38
Hanley, H. W. A., & Durumeric, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.09820 · Cited by 36
On the Zero-Shot Generalization of Machine-Generated Text Detectors
Xiao, P., Zhang, J., Han, X., Tsvetkov, Y., & He, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.318 · Cited by 36
Hanley, H. W. A., & Durumeric, Z. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31333 · Cited by 36
Research works printed but not linked: 22
- Aich, 2022, Demystifying neural fake news via linguistic feature-based interpretation, 6586
- Christian, 2023, CNET secretly used AI on articles that didn’t disclose that fact, staff say, Futurism
- Cui, 2023, ChatLaw: Open-source legal large language model with integrated external knowledge bases, ArXiv preprint
- Deng, 2023, Efficient detection of LLM-generated texts with a Bayesian surrogate model, ArXiv preprint
- Dong, 2023, A survey for in-context learning, ArXiv preprint
- Giorgi, 2023, “I slept like a baby”: Using human traits to characterize deceptive ChatGPT and human text, 23
- Guo, 2020, Wiki-40B: Multilingual language model dataset, 2440
- He, 2023, Empirical analysis of beam search curse and search errors with model errors in neural machine translation, 91
- Kalinichenko, 2003, Digital libraries: Advanced methods and technologies, digital collections, D-Lib Magazine, 9, 1082
- Kitchenham, Barbara and StuartCharters. 2007. Guidelines for performing systematic literature reviews in software engineering. Technical Report, EBSE Technical Report EBSE-2007-01. pages 1–57. https://legacyfileshare.elsevier.com/promis_misc/525444systematicreviewsguide.pdf
- Kuditipudi, 2024, Robust distortion-free watermarks for language models, Transactions on Machine Learning Research
- Lambert, 2022, Illustrating reinforcement learning from human feedback (RLHF), HuggingFace Blog
- Liu, 2023, A private watermark for large language models, ArXiv preprint
- Liu, 2023, Check me if you can: Detecting ChatGPT-generated academic writing using CheckGPT, ArXiv preprint
- Ma, 2023, AI vs. human–differentiation analysis of scientific content generation, arXiv
- McCarthy, Philip M. 2005. An Assessment of the Range and Usefulness of Lexical Diversity Measures and the Potential of the Measure of Textual, Lexical Diversity (MTLD). Ph.D. thesis, The University of Memphis.
- Nicks, 2023, Language model detectors are easily optimized against
- Pagnoni, 2022, Threat scenarios and best practices to detect neural fake news, 1233
- Piccolo, 2023, Many bioinformatics programming tasks can be automated with ChatGPT, ArXiv preprint
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
- Yan, 2023, Detection of AI-generated essays in writing assessment, Psychological Testing and Assessment Modeling, 65, 125
- Yu, 2023, GPT paternity test: GPT generated text detection with GPT genetic inheritance, ArXiv preprint
Cited by
LLM Evaluators Recognize and Favor Their Own Generations
Panickssery, A., Bowman, S. R., & Feng, S. (2024) · arXiv · doi:10.48550/arxiv.2404.13076 · Cited by 559
Scalable watermarking for identifying large language model outputs
Dathathri, S., See, A., Ghaisas, S., Huang, P.-S., McAdam, R., Welbl, J., Bachani, V., Kaskasoli, A., Stanforth, R., Matejovicova, T., et al. (2024) · Nature · doi:10.1038/s41586-024-08025-4 · Cited by 132
Lyu, W., Wang, Y., Chung, T. (., Sun, Y., & Zhang, Y. (2024) · Proceedings of the Eleventh ACM Conference on Learning @ Scale · doi:10.1145/3657604.3662036 · Cited by 131
Mohammadabadi, S. M. S., Kara, B. C., Eyüpoğlu, C., Uzay, C., Tosun, M. S., & Karakuş, O. (2025) · Electronics · doi:10.3390/electronics14183580 · Cited by 89
Wu, Z., Dong, Y., Li, Y., & Shi, B. (2025) · European Journal of Operational Research · doi:10.1016/j.ejor.2025.04.032 · Cited by 42
Metadata
- Authors
- Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F.
- Affiliations
- University of Macau (MO)
- Peking University (CN)
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2025
- Volume
- 51
- Issue
- 1
- Pages
- 275-338
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W4406236959
- Cited by
- 212
- References
- 257
- Citations by year
- 2026: 106 · 2025: 91
- FWCI
- 79.02
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- United Mitochondrial Disease Foundation
- Universidade de Macau
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (FDCT/060/2022/AFJ, FDCT/0070/2022/AMJ)
- State Administration for Science, Technology and Industry for National Defense
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- LLM-generated text detection, large language models, watermarking, detection dataset, out-of-distribution detection, adversarial attacks, evaluation framework, responsible AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-26, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-26 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.