Is GPT-3 text indistinguishable from human text? Scarecrow: A framework for scrutinizing machine text
Open accessAbstract
Modern neural language models can produce remarkably fluent and grammatical text.So much, in fact, that recent work by Clark et al. (2021) has reported that conventional crowdsourcing can no longer reliably distinguish between machine-authored (GPT-3) and humanauthored writing.As errors in machine generations become ever subtler and harder to spot, it poses a new challenge to the research community for robust machine text evaluation.We propose a new framework called SCARE-CROW for scrutinizing machine text via crowd annotation.To support the broad range of real machine errors that can be identified by laypeople, the ten error categories of SCARECROWsuch as redundancy , commonsense errors , and incoherence -are identified through several rounds of crowd annotation experiments without a predefined ontology.We then use SCARECROW to collect over 41k error spans in human-written and machinegenerated paragraphs of English language news text.We isolate factors for detailed analysis, including parameter count, training data, and various decoding-time configurations.Our approach successfully quantifies measurable gaps between human authored text and generations from models of several sizes, including fourteen configurations of GPT-3.In addition, our analysis unveils new insights, with detailed rationales provided by laypeople, e.g., that the commonsense capabilities have been improving with larger models while math capabilities have not, and that the choices of simple decoding hyperparameters can make remarkable differences on the perceived quality of machine text.We release our training material, annotation toolkit and dataset at
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SCARECROW reports Manual annotation outperformed detection models on half of the error types, suggesting potential in developing efficient annotation systems despite the associated human overhead.
“The result from SCARECROW reports Manual annotation outperformed detection models on half of the error types, suggesting potential in developing efficient annotation systems despite the associated human overhead.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Dou et al. 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221 · see citing works
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448 · see citing works
(2004) · Cited by 9,718 · see citing works
(2005) · Cited by 4,152 · see citing works
ConceptNet — A Practical Commonsense Reasoning Tool-Kit
Liu, H., & Singh, P. K. (2004) · BT Technology Journal · doi:10.1023/b:bttj.0000047600.45421.6d · Cited by 1,592 · see citing works
Stanza: A Python Natural Language Processing Toolkit for Many Human Languages
Qi, P., Zhang, Y., Zhang, Y., Bolton, J., & Manning, C. D. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.14 · Cited by 1,505 · see citing works
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
Self-citationExtracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · see citing works
Entity, Relation, and Event Extraction with Contextualized Span Representations
Wadden, D., Wennberg, U., Luan, Y., & Hajishirzi, H. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1585 · Cited by 625 · see citing works
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290 · see citing works
Self-citationAll That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
Self-citationEvaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207 · see citing works
Learning to Write with Cooperative Discriminators
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., & Choi, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1152 · Cited by 192 · see citing works
Self-citationUnifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161 · see citing works
Howcroft, D. M., Belz, A., Clinciu, M., Gkatzia, D., Hasan, S. A., Mahamood, S., Mille, S., van Miltenburg, E., Santhanam, S., & Rieser, V. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.23 · Cited by 129 · see citing works
Caccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J., & Charlin, L. (2020) · International Conference on Learning Representations · Cited by 103 · see citing works
With Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84 · see citing works
RankME: Reliable Human Ratings for Natural Language Generation
Novikova, J., Dušek, O., & Rieser, V. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2012 · Cited by 75 · see citing works
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation
Guan, J., & Huang, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.736 · Cited by 50 · see citing works
Light-Weight Entailment Checking for Computational Semantics
Monz, C., & de Rijke, M. (2001) · Cited by 47 · see citing works
GENIE: A Leaderboard for Human-in-the-Loop Evaluation of Text Generation.Preprint
Khashabi, D., Stanovsky, G., Bragg, J., Lourie, N., Kasai, J., Choi, Y., Smith, N. A., & Weld, D. S. (2021) · arXiv (Cornell University) · Cited by 47 · see citing works
Self-citationRoFT: A Tool for Evaluating Human Detection of Machine-Generated Text
Dugan, L. D., Ippolito, D., Kirubarajan, A., & Callison-Burch, C. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.25 · Cited by 41 · see citing works
Clark, E., & Smith, N. A. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.279 · Cited by 23 · see citing works
Self-citationPerception Score: A Learned Metric for Open-ended Text Generation Evaluation
Gu, J., Wu, Q., & Yu, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17526 · Cited by 14 · see citing works
Stanza: A Python Natural Language Processing Toolkit for Many Human Languages
Qi, P., Zhang, Y., Zhang, Y., Bolton, J., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.07082
Flagged first, then by citations
Research works printed but not linked: 12
- Yann Mathet, Antoine Widlöcher, and Jean-Philippe Métivier. 2015. The unified and holistic method gamma (γ) for inter-annotator agreement mea- sure and alignment. Computational Linguistics, 41(3):437-479.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Lan- guage models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-7.
- Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 193- 203.
- Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2020. A survey of evalua- tion metrics used for nlg systems. arXiv preprint arXiv:2008.12009. · arXiv
- Roger C Schank and Robert P Abelson. 1977. Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology Press.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. · arXiv
- Gavin Wood, Kiel Long, Tom Feltwell, Scarlett Row- land, Phillip Brooker, Jamie Mahoney, John Vines, Julie Barnett, and Shaun Lawson. 2018. Rethink- ing engagement with online news through social and visual co-annotation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1-12.
- Rowan Zellers, Ari Holtzman, Elizabeth Clark, Lianhui Qin, Ali Farhadi, and Yejin Choi. 2021. TuringAd- vice: A generative and dynamic evaluation of lan- guage use. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4856-4880, Online. Associa- tion for Computational Linguistics.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 9054-9065.
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.09675. · arXiv
Cited by
Sorted by citations
Human heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342 · see citing works
Yan, P., Abdulkadir, A., Luley, P.-P., Rosenthal, M., Schatte, G. A., Grewe, B. F., & Stadelmann, T. (2024) · IEEE Access · doi:10.1109/access.2023.3349132 · Cited by 239 · see citing works
Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E. N., Japkowicz, N., & Viktor, H. L. (2023) · IEEE Access · doi:10.1109/access.2023.3294090 · Cited by 196 · see citing works
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212 · see citing works
Li, Z., Zhu, H., Lu, Z., & Yin, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.647 · Cited by 139 · see citing works
Large language models demonstrate the potential of statistical learning in language
Contreras Kallens, P., Kristensen‐McLachlan, R. D., & Christiansen, M. H. (2023) · Cognitive Science · doi:10.1111/cogs.13256 · Cited by 130 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Narrative transportation: How stories shape how we see ourselves and the world
Green, M. C., & Appel, M. (2024) · Advances in experimental social psychology · doi:10.1016/bs.aesp.2024.03.002 · Cited by 111 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70 · see citing works
Creating a large language model of a philosopher
Schwitzgebel, E., Schwitzgebel, D., & Strasser, A. (2024) · Mind & Language · doi:10.1111/mila.12466 · Cited by 73 · see citing works
Distinguishing Human-Written and ChatGPT-Generated Text Using Machine Learning
Alamleh, H., AlQahtani, A. A. S., & ElSaid, A. A. (2023) · Systems and Information Engineering Design Symposium · doi:10.1109/sieds58326.2023.10137767 · Cited by 64 · see citing works
Deepfake Text Detection: Limitations and Opportunities
Pu, J., Sarwar, Z., Abdullah, S. M., Rehman, A., Kim, Y., Bhattacharya, P., Javed, M., & Viswanath, B. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179387 · Cited by 62 · see citing works
How to Design Translation Prompts for ChatGPT: An Empirical Study
Gao, Y., Wang, R., & Hou, F. (2024) · MMAsia Workshops · doi:10.1145/3700410.3702123 · Cited by 95 · see citing works
Coiera, E. W., & Liu, S. (2022) · Cell Reports Medicine · doi:10.1016/j.xcrm.2022.100860 · Cited by 52 · see citing works
AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation
Zheng, C., Sabour, S., Wen, J., Zhang, Z., & Huang, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.99 · Cited by 49 · see citing works
Anderl, C., Klein, S. H., Sarigül, B., Schneider, F. M., Han, J., Fiedler, P. L., & Utz, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-67829-6 · Cited by 38 · see citing works
Does Human Collaboration Enhance the Accuracy of Identifying LLM-Generated Deepfake Texts?
Uchendu, A., Lee, J., Shen, H., Le, T., Huang, T.-H., & Lee, D. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27557 · Cited by 35 · see citing works
Jungwirth, D., & Haluza, D. (2023) · Journal of Software Engineering and Applications · doi:10.4236/jsea.2023.164006 · Cited by 32 · see citing works
Evaluating GPT-3 Generated Explanations for Hateful Content Moderation
Wang, H., Hee, M. S., Awal, M. R., Choo, K. T. W., & Lee, R. K.-W. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/694 · Cited by 32 · see citing works
Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
Fu, L., Newman, B., Jakesch, M., & Kreps, S. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581351 · Cited by 31 · see citing works
Chhun, C., Suchanek, F. M., & Clavel, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00689 · Cited by 30 · see citing works
LENS: A Learnable Evaluation Metric for Text Simplification
Maddela, M., Dou, Y., Heineman, D., & Xu, W. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.905 · Cited by 38 · see citing works
Self-citationCredible without Credit: Domain Experts Assess Generative Language Models
Peskoff, D., & Stewart, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.37 · Cited by 24 · see citing works
Pookpanich, P., & Siriborvornratanakul, T. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-023-01183-9 · Cited by 26 · see citing works
SNaC: Coherence Error Detection for Narrative Summarization
Goyal, T., Li, J. J., & Durrett, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.29 · Cited by 21 · see citing works
RuCoLA: Russian Corpus of Linguistic Acceptability
Mikhailov, V., Shamardina, T., Ryabinin, M., Pestova, A., Smurov, I., & Artemova, E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.348 · Cited by 21 · see citing works
Babaali, B., Salem, M., & Alharbe, N. (2024) · International Journal of Information Technology · doi:10.1007/s41870-024-01926-7 · Cited by 19 · see citing works
Wang, H., Wu, H., Zhu, H., Miao, Y., Wang, Q., Qiao, S., Zhao, H., Chen, C., & Zhang, J. (2022) · Agriculture · doi:10.3390/agriculture12060813 · Cited by 17 · see citing works
Can LLM-generated misinformation be detected: A study on Cyber Threat Intelligence
Huang, H., Sun, N., Tani, M., Zhang, L. Y., Jiang, J., & Jha, S. K. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.107877 · Cited by 15 · see citing works
Studying Psychosis Using Natural Language Generation: A Review of Emerging Opportunities
Palaniyappan, L. K., Benrimoh, D. A., Voppel, A., & Rocca, R. (2023) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2023.04.009 · Cited by 14 · see citing works
How well does GPT phish people? An investigation involving cognitive biases and feedback
Sharma, M., Singh, K., Aggarwal, P., & Dutt, V. (2023) · IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) · doi:10.1109/eurospw59978.2023.00055 · Cited by 15 · see citing works
A new era for stress research: supporting user performance and experience in the digital age
Matthews, G., Cumings, R., Santos, E. P. D. L., Feng, I. Y., & Mouloua, S. A. (2024) · Ergonomics · doi:10.1080/00140139.2024.2425953 · Cited by 15 · see citing works
Generative AI and Perceptual Harms: Who's Suspected of using LLMs?
Kadoma, K., Metaxa, D., & Naaman, M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713897 · Cited by 14 · see citing works
WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering
Bolotova-Baranova, V., Blinov, V., Филиппова, С. В., Scholer, F., & Sanderson, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.290 · Cited by 14 · see citing works
Dhruv, A., Saha, S., Tyagi, S., & Jain, V. (2024) · 2024 2nd International Conference on Advancement in Computation & Computer Technologies (InCACCT) · doi:10.1109/incacct61598.2024.10551108 · Cited by 12 · see citing works
Singh, A., Sharma, D., Nandy, A., & Singh, V. K. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100050 · Cited by 11 · see citing works
Ding, Z. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640544.3645241 · Cited by 10 · see citing works
Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond
Liu, S., Deng, N., Sabour, S., Jia, Y., Huang, M., & Mihalcea, R. F. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.944 · Cited by 10 · see citing works
Reliability Check: An Analysis of GPT-3’s Response to Sensitive Topics and Prompt Wording
Khatun, A., & Brown, D. G. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.8 · Cited by 12 · see citing works
Mapping the Design Space of Interactions in Human-AI Text Co-creation Tasks
Ding, Z., & Chan, J. (2023) · arXiv · doi:10.48550/arxiv.2303.06430 · Cited by 9 · see citing works
Impact of GPT on the Academic Ecosystem
Cai, Y., Deng, Q., Lv, T., Zhang, W., & Zhou, Y. (2024) · Science & Education · doi:10.1007/s11191-024-00561-9 · Cited by 8 · see citing works
de Wet, M., Silva, M. O. D., & Bohnsack, R. (2025) · Innovations in Education and Teaching International · doi:10.1080/14703297.2025.2536609 · Cited by 8 · see citing works
Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA
Heineman, D., Dou, Y., Maddela, M., & Xu, W. H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.211 · Cited by 10 · see citing works
Self-citationJungwirth, D., & Haluza, D. (2023) · Preprints.org · doi:10.20944/preprints202303.0025.v1 · Cited by 8 · see citing works
Yan, P., Abdulkadir, A., Luley, P.-P., Rosenthal, M., Schatte, G. A., Grewe, B. F., & Stadelmann, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.05638 · Cited by 9 · see citing works
Transformers Can Outperform Humans in Short Creative Writing TasksPreprint
Marco, G., Gonzalo, J. A., & Rello, L. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4673692 · Cited by 7 · see citing works
Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation
Huang, F., Ke, P., & Huang, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00582 · Cited by 6 · see citing works
Leveraging Deep Learning Models for Machine-Generated Text Detection Using Transformer-Based Models
Suthanthiradevi, P., Revathy, G., Rejini, K., & Lakshmi.V, M. (2024) · International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES) · doi:10.1109/icses63760.2024.10910837 · Cited by 5 · see citing works
Barriers and enabling factors for error analysis in NLG research
van Miltenburg, E., Clinciu, M., Dušek, O., Gkatzia, D., Inglis, S., Leppänen, L., Mahamood, S., Schoch, S., Thomson, C., & Wen, L. (2023) · Northern European Journal of Language Technology · doi:10.3384/nejlt.2000-1533.2023.4529 · Cited by 5 · see citing works
FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation
Goyal, T., Li, J. J., & Durrett, G. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.35 · Cited by 5 · see citing works
Towards Informative Open-ended Text Generation with Dynamic Knowledge Triples
Ren, Z., Zhao, Y., & Zong, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.210 · Cited by 5 · see citing works
Synthetic Text as Data: On Usefulness and Limitations
Choi, S., Sim, J., & Choi, G. (2025) · Applied Sciences · doi:10.3390/app15105460 · Cited by 4 · see citing works
DeltaScore: Fine-Grained Story Evaluation with Perturbations
Xie, Z., Li, M., Cohn, T., & Lau, J. H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.353 · Cited by 6 · see citing works
Paraphrase Detection: Human vs. Machine ContentPreprint
Wahle, J. P., Becker, J., Ruas, T., & Gipp, B. (2023) · doi:10.22541/au.168323192.23601312/v1 · Cited by 4 · see citing works
Thresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluation
Heineman, D., Dou, Y., & Xu, W. H. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.30 · Cited by 3 · see citing works
Self-citationHow Natural Language Processing Enables AIGC Recognition? --Latest Trends and Future Prospects
Wang, W., & Qiao, H. (2024) · International Conference on Software Engineering and Information Management (ICSIM) · doi:10.1145/3647722.3647738 · Cited by 3 · see citing works
Small Character Models Match Large Word Models for Autocomplete Under Memory Constraints
Jawahar, G., Mukherjee, S., Dey, D., Abdul-Mageed, M., Lakshmanan, V. S. L., Mendes, C. C. T., Rosa, G. D., & Shah, S. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.22 · Cited by 2 · see citing works
Su, Z., Shen, J., Zhou, Q., & Yong, B. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3626545 · Cited by 2 · see citing works
Linguistic profile of LLMs: an overview of linguistic performance and jailbreak vulnerability
Abdelzaher, E. M., & Rafaee, E. (2026) · Journal of Cultural Cognitive Science · doi:10.1007/s41809-025-00192-8 · Cited by 2 · see citing works
Studenikina, K. A. (2023) · RESEARCH RESULT Theoretical and Applied Linguistics · doi:10.18413/2313-8912-2023-9-1-1-2 · Cited by 2 · see citing works
Fine-Tuning GPT-3 for Synthetic Danish News Generation
Almasi, M., & Schiønning, A. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.4 · Cited by 2 · see citing works
Can Language Models Trained on Written Monologue Learn to Predict Spoken Dialogue?
Umair, M., Mertens, J. B., Warnke, L., & de Ruiter, J. P. (2024) · Cognitive Science · doi:10.1111/cogs.70013 · Cited by 3 · see citing works
Lystbæk, M. S., & Lystbæk, U. S. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-96199-1_2 · Cited by 2 · see citing works
Generative Artificial Intelligence as Creative Artificial Intelligence
Appel, M., Messingschlager, T. V., Raffloer, G., & Reed, P. A. (2025) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197689875.013.0016 · Cited by 2 · see citing works
ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Origination
Ayoobi, N., Knab, L., Cheng, W., Pantoja, D., Alikhani, H., Flamant, S., Kim, J., & Mukherjee, A. (2025) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3720553.3746665 · Cited by 4 · see citing works
Error Analysis of ToTTo Table-to-Text Neural NLG Models
Sundararajan, B., Sripada, S. G., & Reiter, E. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.43 · Cited by 2 · see citing works
Berro, A., Benatallah, B., Gaci, Y., & Benabdeslem, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70341-6_20 · Cited by 1 · see citing works
SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generation
Pham, K.-N., He, Y., Xing, Y., Chen, Q., & Chen, L. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755705 · Cited by 1 · see citing works
Evalet: Evaluating Large Language Models through Functional Fragmentation
Kim, T. S., Lee, H., Lee, Y., Seering, J., & Kim, J. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790285 · Cited by 1 · see citing works
Guan, J., & Huang, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.431 · Cited by 1 · see citing works
Профессиональное самоопределение как компонент профессионального становления личности
Николаевна, Д. Т., & Николаевна, Ю. С. (2013) · Научное обеспечение системы повышения квалификации кадров
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types
Murugesan, K., Swaminathan, S., Dan, S., Chaudhury, S., Gunasekara, C., Crouse, M., Mahajan, D., Abdelaziz, I., Fokoue, A., Kapanipathi, P., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.274
Generating Dialog Responses with Specified Grammatical Items for Second Language Learning
Okano, Y., Funakoshi, K., Nagata, R., & Okumura, M. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.16
NaturalAdversaries: Can Naturalistic Adversaries Be as Effective as Artificial Adversaries?
Gabriel, S., Palangi, H., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.413
Self-citationFor Generated Text, Is NLI-Neutral Text the Best Text?
Mersinias, M., & Mahowald, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.170
Post-hoc Utterance Refining Method by Entity Mining for Faithful Knowledge Grounded Conversations
Jang, Y., Son, S., Lee, J., Son, J., Hur, Y., Lim, J., Moon, H., Yang, K., & Lim, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.295
Detecting the Usage of Large Language Models Exploiting Generative Adversarial Networks
Aversano, L., Bernardi, M. L., & Cimitile, M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-67351-1_9
Integrity in the Age of AI: Detecting Synthetic Text Across Platforms and Purposes
Shah, R., & Mahato, M. (2025) · International Conference on Cognitive Computing in Engineering, Communications, Sciences and Biomedical Health Informatics (IC3ECSBHI) · doi:10.1109/ic3ecsbhi63591.2025.10990518
Automatic Generation of Software Prototype Data for Rapid Requirements Validation
Chang, S., Gao, J., & Wang, W. (2025) · Electronics · doi:10.3390/electronics14173497 · Cited by 1 · see citing works
MedAID-ML: A Multilingual Dataset of Biomedical Texts for Detecting AI-Generated Content
Styll, P., Llanos, L. C., Fernández-García, J., & Segura-Bédmar, I. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04354-2_5
Towards Internet-Scale Inter-Domain Path Verification
Zhao, W., Wang, X., & Guo, Y. (2025) · International Workshop on Quality of Service · doi:10.1109/iwqos65803.2025.11143274
Yang, Z., & Graham, C. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.10.14.682480
LLMs highlight the importance of interaction in human language learning
Kristensen‐McLachlan, R. D., Kallens, P. C., & Christiansen, M. H. (2025) · Linguistics Vanguard · doi:10.1515/lingvan-2024-0254
GPT Adoption Dilemma and the Impact of Disclosure Policies
Yang, C., Amariles, D. R., Allen, L. P., & Troussel, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-12795-2_7
Trust and Social Robots: How can We Prevent Social Robots from Eroding Trust?
Strasser, A. (2026) · Social and cultural studies of robots and AI · doi:10.1007/978-3-031-99290-2_9
Mapping the Collaboration between Crowdsourcing and Large Language Models: A Fine-Grained Survey
Lv, C., Shen, C., & Xie, M. (2026) · Business, management and economics · doi:10.5772/intechopen.1013999
Patil, S. K. T. A. D. A., Avasthi, A., Padma, Y., Abinaya, M. G., Shashank, G., & Sreela, B. (2025) · International Conference for Advancement in Technology (ICONAT) · doi:10.1109/iconat66879.2025.11362573
Chaffin, A. (2023) · theses.fr (ABES) · doi:10.70675/5e4cd6f0z8697z4a0cz921dzd8592affa0a9
Meta-Evaluation Methodology and Benchmark for Automatic Story Generation
Chhun, C. (2024) · theses.fr (ABES) · doi:10.70675/ec296eb8z87a1z4fdbzbbbdz12bb27449b8f
Gong, X., Lin, Y., Zhong, Y., Shi, Y., Lin, Q., Zheng, H., & Wang, P. (2026) · Applied Sciences · doi:10.3390/app16104586
Sounds like AI? Consumer suspicion of AI authorship in mixed-source online reviews
Mao, M., Guo, T., Zhang, F., & Chen, S. (2026) · Internet Research · doi:10.1108/intr-07-2025-1039
- Authors
- Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N., & Choi, Y.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2022
- Pages
- 7250-7274
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4285251400
- Semantic Scholar
- ab847321118de056ff00325d58903b71bdcc76e1
- Cited by
- 118
- References
- 23
- Citations by year
- 2026: 10 · 2025: 23 · 2024: 24 · 2023: 45 · 2022: 15 · 2013: 1
- FWCI
- 13.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1714566)
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency (N66001, N66001-19-2-4031, N66001-19)
- Naval Information Warfare Center Pacific
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Keywords
- GPT-3, AI-generated text, crowd annotation, error categories, redundancy, incoherence, model size, training data, news articles
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
