Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generated text
Open accessAbstract
As text generated by large language models proliferates, it becomes vital to understand how humans engage with such text, and whether or not they are able to detect when the text they are reading did not originate with a human writer. Prior work on human detection of generated text focuses on the case where an entire passage is either human-written or machine-generated. In this paper, we study a more realistic setting where text begins as human-written and transitions to being generated by state-of-the-art neural language models. We show that, while annotators often struggle at this task, there is substantial variance in annotator skill and that given proper incentives, annotators can improve at this task over time. Furthermore, we conduct a detailed comparison study and analyze how a variety of variables (model size, decoding strategy, fine-tuning, prompt genre, etc.) affect human detection performance. Finally, we collect error annotations from our participants and use them to show that certain textual genres influence models to make different types of errors and that certain sentence-level features correlate highly with annotator selection. We release the RoFT dataset: a collection of over 21,000 human annotations paired with error classifications to encourage future work in human detection and evaluation of generated text.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs typically exhibit less diversity in word usage compared to humans.
“LLMs typically exhibit less diversity in word usage compared to humans [Ippolito et al., 2020;Dugan et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Dugan et al., 2023
In “Are diverse generations easier to detect?” (page unavailable), the paper says language models struggle to generate high-quality text with diversity similar to human-written text. It discusses diversity as dependent on decoding strategy, but does not specifically establish that LLMs typically use a less diverse vocabulary than humans. (LLM, conf=0.81)
Experimental — accuracy not yet measuredLLM-generated texts tend to repeat terms within a paragraph.
“Humanauthored texts are generally more coherent, while LLMgenerated texts tend to repeat terms within a paragraph [Dugan et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Dugan et al., 2023
Section 6, “RoFT-Specific Insights” (page not provided), reports that participants gave “repetition” as a reason for identifying text as machine-generated. It does not establish that generated text generally repeats terms within a paragraph, or measure repetition at the paragraph level. (LLM, conf=0.82)
Experimental — accuracy not yet measuredHuman-authored texts are generally more coherent than LLM-generated texts.
“Humanauthored texts are generally more coherent, while LLMgenerated texts tend to repeat terms within a paragraph [Dugan et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Dugan et al., 2023
Section 6, “RoFT-Specific Insights” (page not available in our copy), reports that irrelevance and contradiction were among the most reliable reasons participants gave for identifying generated text. That lends some support to a coherence difference, but the paper does not establish that human-authored text is generally more coherent across LLMs, or that generated text tends to repeat terms within a paragraph; it records repetition as a participant-selected reason, not a measured paragraph-level tendency. (LLM, conf=0.78)
Experimental — accuracy not yet measuredHuman-authored texts tend to be more coherent and exhibit greater lexical diversity compared to LLM-generated texts.
“Human-authored texts tend to be more coherent and exhibit greater lexical diversity compared to LLM-generated texts [Guo et al., 2023;Dugan et al., 2023;Zhang et al., 2024].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Dugan et al., 2023
Section 6, “RoFT-Specific Insights” (page unavailable), reports that irrelevant and contradictory sentences were among the most reliable signs participants used to detect generated text. This lends support to the coherence comparison, but the paper does not establish that human-authored text has greater lexical diversity; its sentence-level analysis examines length, parts of speech, and named entities instead. (LLM, conf=0.83)
Experimental — accuracy not yet measuredDistinguishing between LLM-generated and human-written texts poses significant challenges for humans.
“However, distinguishing between LLM-generated and human-written texts poses significant challenges for humans [Dugan et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Dugan et al., 2023
Dugan et al. (2023) identified that human discernment of LLM-generated text varies across different domains; LLMs tend to generate more 'generic' text in news domain, whereas in story domains, text might be more 'irrelevant'.
“Dugan et al. (2023) identified that the human discernment of LLM-generated text varies across different domains. For instance, LLMs tend to generate more 'generic' text in the news domain, whereas, in story domains, the text might be more 'irrelevant'.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Dugan et al. 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242 · see citing works
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · see citing works
GLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3019 · Cited by 470 · see citing works
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290 · see citing works
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
WebGPT: Browser-assisted question-answering with human feedback
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09332 · Cited by 272 · see citing works
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2019) · arXiv · doi:10.48550/arxiv.1911.00650 · Cited by 264
Self-citationAutomatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.164 · Cited by 264 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 14
- Deriu, J.; Tuggener, D.; von Däniken, P.; Campos, J. A.; Ro- drigo, A.; Belkacem, T.; Soroa, A.; Agirre, E.; and Cieliebak, M. 2020. Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3971-3984. On- line: Association for Computational Linguistics.
- Dou, Y.; Forbes, M.; Koncel-Kedziorski, R.; Smith, N. A.; and Choi, Y. 2021. Scarecrow: A framework for scrutinizing machine text. arXiv preprint arXiv:2107.01294. · arXiv
- Dugan, L.; Ippolito, D.; Kirubarajan, A.; and Callison- Burch, C. 2020. RoFT: A Tool for Evaluating Human Detec- tion of Machine-Generated Text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, 189-196. Online: Associ- ation for Computational Linguistics.
- Ethayarajh, K.; and Jurafsky, D. 2022. The Authenticity Gap in Human Evaluation. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, 6056-6070. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.
- He, J.; Peng, B.; Liao, Y.; Liu, Q.; and Xiong, D. 2021. TGEA: An Error-Annotated Dataset and Benchmark Tasks for TextGeneration from Pretrained Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 6012-6025. Online: Association for Com- putational Linguistics.
- Hoffmann, J.; Borgeaud, S.; Mensch, A.; Buchatskaya, E.; Cai, T.; Rutherford, E.; Casas, D. d. L.; Hendricks, L. A.; Welbl, J.; Clark, A.; et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556. · arXiv
- Kaplan, J.; McCandlish, S.; Henighan, T.; Brown, T. B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D. 2020. Scaling Laws for Neural Language Mod- els. arXiv:2001.08361. · arXiv
- Keskar, N. S.; McCann, B.; Varshney, L. R.; Xiong, C.; and Socher, R. 2019. Ctrl: A conditional transformer lan- guage model for controllable generation. arXiv preprint arXiv:1909.05858. · arXiv
- Marin, J.; Biswas, A.; Ofli, F.; Hynes, N.; Salvador, A.; Ay- tar, Y.; Weber, I.; and Torralba, A. 2019. Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images. IEEE Trans. Pattern Anal. Mach. Intell.
- Martens, D.; and Maalej, W. 2019. Towards understanding and detecting fake reviews in app stores. Empirical Software Engineering, 24(6): 3316-3355.
- Sandhaus, E. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12): e26752.
- Weidinger, L.; Mellor, J.; Rauh, M.; Griffin, C.; Uesato, J.; Huang, P.-S.; Cheng, M.; Glaese, M.; Balle, B.; Kasirzadeh, A.; Kenton, Z.; Brown, S.; Hawkins, W.; Stepleton, T.; Biles, C.; Birhane, A.; Haas, J.; Rimell, L.; Hendricks, L. A.; Isaac, W.; Legassick, S.; Irving, G.; and Gabriel, I. 2021. Ethi- cal and social risks of harm from Language Models. arXiv preprint arXiv:2112.04359. · arXiv
- Zellers, R.; Holtzman, A.; Rashkin, H.; Bisk, Y.; Farhadi, A.; Roesner, F.; and Choi, Y. 2019. Defending Against Neural Fake News. In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d'Alché-Buc, F.; Fox, E.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volume 32.
- Zhang, H.; Duckworth, D.; Ippolito, D.; and Neelakantan, A. 2021. Trading Off Diversity and Quality in Natural Lan- guage Generation. In Proceedings of the Workshop on Hu- man Evaluation of NLP Systems (HumEval), 25-33. Online: Association for Computational Linguistics.
Cited by
Sorted by citations
Sohail, S. S., Farhat, F., Himeur, Y., Nadeem, M., Madsen, D. Ø., Singh, Y., Atalla, S., & Mansoor, W. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101675 · Cited by 192 · see citing works
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Stone, B. W. (2025) · Teaching of Psychology · doi:10.1177/00986283241305398 · Cited by 54 · see citing works
RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
Dugan, L., Hwang, A., Trhlík, F., Zhu, A., Ludan, J. M., Xu, H., Ippolito, D., & Callison-Burch, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.674 · Cited by 48 · see citing works
Self-citationAuthorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges
Huang, B., Chen, C., & Shu, K. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3715073.3715076 · Cited by 42 · see citing works
M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection
Wang, Y., Mansurov, J., Ivanov, P., Su, J., Shelmanov, A., Tsvigun, A., Mohammed Afzal, O., Mahmoud, T., Puccetti, G., Arnold, T., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.218 · Cited by 25 · see citing works
Enhancing the Robustness of AI-Generated Text Detectors: A Survey
Liu, X., Li, Y., & Li, K. (2025) · Mathematics · doi:10.3390/math13132145 · Cited by 10 · see citing works
Ilic, P., & Carr, N. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343337 · Cited by 6 · see citing works
Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education
Zeng, Z., Sha, L., Li, Y., Yang, K., Gašević, D., & Chen, G. (2023) · arXiv · doi:10.48550/arxiv.2307.12267 · Cited by 25 · see citing works
- Authors
- Dugan, L., Ippolito, D., Kirubarajan, A., Shi, S., & Callison-Burch, C.
- Affiliations
- University of Pennsylvania (US)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2023-06-26
- Volume
- 37
- Issue
- 11
- Pages
- 12763-12771
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4382202515
- Cited by
- 54
- References
- 9
- Citations by year
- 2026: 16 · 2025: 17 · 2024: 16 · 2023: 5
- FWCI
- 9.57
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1928631)
- University of Pennsylvania
- Office of the Director of National Intelligence
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750)
- Intelligence Advanced Research Projects Activity
- NIH Office of the Director
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, human-written text, error analysis, textual genres, decoding strategies, fine-tuning, model size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
