All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Open accessAbstract
Human evaluations are typically considered the gold standard in natural language generation, but as models' fluency improves, how well can evaluators detect and judge machine-generated text? We run a study assessing non-experts' ability to distinguish between human- and machine-authored text (GPT2 and GPT3) in three domains (stories, news articles, and recipes). We find that, without training, evaluators distinguished between GPT3- and human-authored text at random chance level. We explore three approaches for quickly training evaluators to better identify GPT3-authored text (detailed instructions, annotated examples, and paired examples) and find that while evaluators' accuracy improved up to 55%, it did not significantly improve across the three domains. Given the inconsistent results across text domains and the often contradictory reasons evaluators gave for their judgments, we examine the role untrained human evaluations play in NLG evaluation and provide recommendations to NLG researchers for improving human evaluations of text generated from state-of-the-art models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Concerns regarding academic honesty and student learning have dominated discourse on GenAI in higher education.
“Given the ability of GenAI tools to generate human-like content, such as language and code, concerns regarding academic honesty and student learning have dominated discourse [16, 29, 33, 36, 55, 115, 117].”
From "I'd Never Actually Realized How Big An Impact It Had Until Now": Perspectives of University Students with Disabilities on Generative Artificial Intelligence (2025) · cites this paper as [29]
The Abstract and §7 Conclusion discuss whether evaluators can distinguish human-authored from GPT-generated text. Our copy does not address whether academic honesty and student learning have dominated discourse on GenAI in higher education; searches for those concerns and for education-related terms found no supporting passage. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Computing Machinery and Intelligence (1950)
Turing, A. M. (2004) · doi:10.1093/oso/9780198250791.003.0017 · Cited by 6,120 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · see citing works
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · see citing works
Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,262 · see citing works
Soylent: a word processor with a crowd inside
(2010) · doi:10.1145/1866029.1866078 · Cited by 686 · see citing works
Harnessing the wisdom of crowds in wikipedia
Kittur, A., & Kraut, R. E. (2008) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1460563.1460572 · Cited by 680 · see citing works
GLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3019 · Cited by 470 · see citing works
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422 · see citing works
(Meta-) evaluation of machine translation
Callison-Burch, C., Fordyce, C. S., Koehn, P., Monz, C., & Schroeder, J. (2007) · Conference on Machine Translation (WMT) · doi:10.3115/1626355.1626373 · Cited by 381 · see citing works
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. X. (2020) · International Conference on Machine Learning · Cited by 325 · see citing works
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.164 · Cited by 264 · see citing works
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
Comparing automatic and human evaluation of NLG systems
Belz, A., & Reiter, E. (2006) · University of Brighton Repository (University of Brighton) · Cited by 182 · see citing works
Human evaluation of automatically generated text: Current trends and best practice guidelines
van der Lee, C., Albert Gatt, A., van Miltenburg, E., & Krahmer, E. (2020) · Computer Speech & Language · doi:10.1016/j.csl.2020.101151 · Cited by 178 · see citing works
Unifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161 · see citing works
Howcroft, D. M., Belz, A., Clinciu, M., Gkatzia, D., Hasan, S. A., Mahamood, S., Mille, S., van Miltenburg, E., Santhanam, S., & Rieser, V. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.23 · Cited by 129 · see citing works
Authorship Attribution for Neural Text Generation
Uchendu, A., Le, T., Shu, K., & Lee, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.673 · Cited by 120 · see citing works
Mitra, T., Hutto, C. J., & Gilbert, E. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702553 · Cited by 102 · see citing works
STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation
Akoury, N., Wang, S., Whiting, J., Hood, S., Peng, N., & Iyyer, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.525 · Cited by 85 · see citing works
With Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84 · see citing works
RankME: Reliable Human Ratings for Natural Language Generation
Novikova, J., Dušek, O., & Rieser, V. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2012 · Cited by 75 · see citing works
RecipeNLG: A Cooking Recipes Dataset for Semi-Structured Text Generation
Bień, M., Gilski, M., Maciejewska, M., Taisner, W., Wisniewski, D., & Ławrynowicz, A. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.4 · Cited by 58 · see citing works
Mechanical Novel: Crowdsourcing Complex Work Through Reflection and Revision
Kim, J. O., Sterman, S., Cohen, A. A. B., & Bernstein, M. S. (2017) · Understanding innovation · doi:10.1007/978-3-319-60967-6_5 · Cited by 53 · see citing works
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Anuoluwapo, A., Bosselut, A., Chandu, K. R., Clinciu, M., Das, D., Dhole, K. D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.01672 · Cited by 49 · see citing works
GENIE: A Leaderboard for Human-in-the-Loop Evaluation of Text Generation.Preprint
Khashabi, D., Stanovsky, G., Bragg, J., Lourie, N., Kasai, J., Choi, Y., Smith, N. A., & Weld, D. S. (2021) · arXiv (Cornell University) · Cited by 47 · see citing works
Self-citationBelz, A., Mille, S., & Howcroft, D. M. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.24 · Cited by 45 · see citing works
RoFT: A Tool for Evaluating Human Detection of Machine-Generated Text
Dugan, L. D., Ippolito, D., Kirubarajan, A., & Callison-Burch, C. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.25 · Cited by 41 · see citing works
Rethinking the Agreement in Human Evaluation Tasks
Amidei, J., Piwek, P., & Willis, A. (2018) · Open Research Online (The Open University) · Cited by 40 · see citing works
Towards Best Experiment Design for Evaluating Dialogue System Output
Santhanam, S., & Shaikh, S. (2019) · International Conference on Natural Language Generation · doi:10.18653/v1/w19-8610 · Cited by 27 · see citing works
Clark, E., & Smith, N. A. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.279 · Cited by 23 · see citing works
Self-citationTuringAdvice: A Generative and Dynamic Evaluation of Language Use
Zellers, R., Holtzman, A., Clark, E., Qin, L., Farhadi, A., & Choi, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.386 · Cited by 21 · see citing works
Self-citationArtificial Intelligence and Collective Intelligence
Weld, D. S., Lin, C. H., & Bragg, J. (2014) · Cited by 19 · see citing works
Flagged first, then by citations
3 of the 36 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and as- surance actions. In ACM Computing Surveys, vol- ume 51. Association for Computing Machinery. · DOI
- Stephanie Schoch, Diyi Yang, and Yangfeng Ji. 2020. "This is a problem, don't you agree?" Framing and bias in human evaluation for natural language gener- ation. In Proceedings of the 1st Workshop on Evalu- ating NLG Evaluation, pages 10-16, Online (Dublin, Ireland). Association for Computational Linguistics.
- Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019a. HellaSwag: Can a machine really finish your sentence? In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4791- 4800, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019b. Defending against neural fake news. In Advances in Neural Information Process- ing Systems, volume 32.
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099 · see citing works
Perkins, M. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.02.07 · Cited by 839 · see citing works
Human heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342 · see citing works
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331 · see citing works
On the conversational persuasiveness of GPT-4
Salvi, F., Ribeiro, M. H., Gallotti, R., & West, R. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02194-6 · Cited by 326 · see citing works
AI model GPT-3 (dis)informs us better than humans
Spitale, G., Biller-Andorno, N., & Germani, F. (2023) · Science Advances · doi:10.1126/sciadv.adh1850 · Cited by 241 · see citing works
The death of the short-form physics essay in the coming AI revolution
Yeadon, W., Inyang, O.-O., Mizouri, A., Peach, A., & Testrow, C. P. (2023) · Physics Education · doi:10.1088/1361-6552/acc5cf · Cited by 199 · see citing works
Perkins, M., Roe, J., Postma, D., McGaughran, J., & Hickerson, D. (2024) · Journal of Academic Ethics · doi:10.1007/s10805-023-09492-6 · Cited by 234 · see citing works
Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E. N., Japkowicz, N., & Viktor, H. L. (2023) · IEEE Access · doi:10.1109/access.2023.3294090 · Cited by 196 · see citing works
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212 · see citing works
Artificial intelligence chatbots in academic libraries: the rise of ChatGPT
Adetayo, A. J. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0007 · Cited by 147 · see citing works
Li, Z., Zhu, H., Lu, Z., & Yin, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.647 · Cited by 139 · see citing works
Xiao, C., Xu, S. X., Zhang, K., Wang, Y., & Xia, L. (2023) · Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) · doi:10.18653/v1/2023.bea-1.52 · Cited by 138 · see citing works
AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably
Porter, B., & Machery, E. (2024) · Scientific Reports · doi:10.1038/s41598-024-76900-1 · Cited by 133 · see citing works
Scalable watermarking for identifying large language model outputs
Dathathri, S., See, A., Ghaisas, S., Huang, P.-S., McAdam, R., Welbl, J., Bachani, V., Kaskasoli, A., Stanforth, R., Matejovicova, T., et al. (2024) · Nature · doi:10.1038/s41586-024-08025-4 · Cited by 132 · see citing works
The Science of Detecting LLM-Generated Text
Tang, R., Chuang, Y.-N., & Hu, X. (2024) · Communications of the ACM · doi:10.1145/3624725 · Cited by 125 · see citing works
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Liu, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.508 · Cited by 131 · see citing works
Self-citationChatGPT for Cybersecurity: Practical Applications, Challenges, and Future Directions
Al-Hawawreh, M., Aljuhani, A., & Jararweh, Y. (2023) · Cluster Computing · doi:10.1007/s10586-023-04124-5 · Cited by 124 · see citing works
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs
Sap, M., Le Bras, R., Fried, D., & Choi, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.248 · Cited by 118 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
Self-citationLanguage Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
When and How to Use AI in the Design Process? Implications for Human-AI Design Collaboration
Lee, S.-Y., Law, M., & Hoffman, G. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2353451 · Cited by 97 · see citing works
Cook, D. A., Overgaard, J., Pankratz, V. S., Del Fiol, G., & Aakre, C. A. (2025) · Journal of Medical Internet Research · doi:10.2196/68486 · Cited by 96 · see citing works
Narrative transportation: How stories shape how we see ourselves and the world
Green, M. C., & Appel, M. (2024) · Advances in experimental social psychology · doi:10.1016/bs.aesp.2024.03.002 · Cited by 111 · see citing works
Valencia, S., Cave, R., Kallarackal, K., Seaver, K., Terry, M., & Kane, S. K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581560 · Cited by 97 · see citing works
Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N., & Choi, Y. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.501 · Cited by 118 · see citing works
Self-citationPan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00660 · Cited by 86 · see citing works
Generative AI detection in higher education assessments
Ardito, C. G. (2025) · New Directions for Teaching and Learning · doi:10.1002/tl.20624 · Cited by 86 · see citing works
Nelson, A. S., Santamaría, P. V., Javens, J. S., & Ricaurte, M. (2025) · Education Sciences · doi:10.3390/educsci15050611 · Cited by 79 · see citing works
Creating a large language model of a philosopher
Schwitzgebel, E., Schwitzgebel, D., & Strasser, A. (2024) · Mind & Language · doi:10.1111/mila.12466 · Cited by 73 · see citing works
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
Goodman, S. M., Buehler, E., Clary, P., Coenen, A., Donsbach, A., Horne, T. N., Lahav, M., MacDonald, R., Michaels, R. B., Narayanan, A., et al. (2022) · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3517428.3544819 · Cited by 71 · see citing works
On the Risk of Misinformation Pollution with Large Language Models
Pan, Y., Pan, L., Chen, W., Nakov, P., Kan, M.-Y., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.97 · Cited by 70 · see citing works
Anatomy of an AI-powered malicious social botnet
Yang, K., & Menczer, F. (2024) · Journal of Quantitative Description: Digital Media · doi:10.51685/jqd.2024.icwsm.7 · Cited by 71 · see citing works
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning
Masry, A., Kavehzadeh, P., Long, X., Hoque, E., & Joty, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.906 · Cited by 97 · see citing works
Jiang, Y., Hao, J., Fauss, M., & Li, C. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105070 · Cited by 68 · see citing works
A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing
Gómez‐Rodríguez, C., & Williams, P. A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.966 · Cited by 81 · see citing works
Markowitz, D. M., Hancock, J. T., & Bailenson, J. N. (2023) · Journal of Language and Social Psychology · doi:10.1177/0261927x231200201 · Cited by 71 · see citing works
Zaitsu, W., & Jin, M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0288453 · Cited by 61 · see citing works
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation
Liu, Y., Fabbri, A. R., Liu, P., Zhao, Y., Nan, L., Han, R., Han, S., Joty, S., Wu, C.-S., Xiong, C., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.228 · Cited by 75 · see citing works
AI-driven disinformation: a framework for organizational preparation and response
Karinshak, E., & Jin, Y. (2023) · Journal of Communication Management · doi:10.1108/jcom-09-2022-0113 · Cited by 65 · see citing works
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Karpinska, M., Akoury, N., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.97 · Cited by 56 · see citing works
BOHM, R. M., Jörling, M., Reiter, L., & Fuchs, C. (2023) · Communications Psychology · doi:10.1038/s44271-023-00032-x · Cited by 55 · see citing works
Dugan, L., Ippolito, D., Kirubarajan, A., Shi, S., & Callison-Burch, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26501 · Cited by 54 · see citing works
Stone, B. W. (2025) · Teaching of Psychology · doi:10.1177/00986283241305398 · Cited by 54 · see citing works
A Systematic Investigation of Commonsense Knowledge in Large Language Models
Li, X. L., Kuncoro, A., Hoffmann, J., d’Autume, C. D. M., Blunsom, P., & Nematzadeh, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.812 · Cited by 59 · see citing works
Writing with AI: What College Students Learned from Utilizing ChatGPT for a Writing Assignment
Wang, C., Aguilar, S. J., Bankard, J. S., Bui, E., & Nye, B. (2024) · Education Sciences · doi:10.3390/educsci14090976 · Cited by 49 · see citing works
“Despite ongoing research on the automated detection of AI-generated texts [5], it has been difficult for teachers or plagiarism detection tools to distinguish between the texts written by students themselves and the texts generated by ChatGPT [6–8].”
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 57 · see citing works
Sahebi, S., & Formosa, P. (2025) · Synthese · doi:10.1007/s11229-025-04963-2 · Cited by 69 · see citing works
Mohammadi, E., Thelwall, M., Cai, Y., Collier, T., Tahamtan, I., & Eftekhar, A. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2025.104350 · Cited by 47 · see citing works
Economic Systems in the Metaverse: Basics, State of the Art, and Challenges
Huang, H., Zhang, Q., Li, T., Yang, Q., Yin, Z., Junhao, W., Xiong, Z., Zhu, J., Wu, J., & Zheng, Z. (2023) · ACM Computing Surveys · doi:10.1145/3626315 · Cited by 60 · see citing works
Cross-domain detection of GPT-2-generated technical text
Rodriguez, J., Hay, T., Gros, D., Shamsi, Z., & Srinivasan, R. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.88 · Cited by 40 · see citing works
Chein, J. M., Martinez, S. A., & Barone, A. R. (2024) · Scientific Reports · doi:10.1038/s41598-024-76218-y · Cited by 38 · see citing works
How Large Language Models are Transforming Machine-Paraphrase Plagiarism
Wahle, J. P., Ruas, T., Kirstein, F., & Gipp, B. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.62 · Cited by 35 · see citing works
Factors influencing trust in algorithmic decision-making: an indirect scenario-based experiment
Marmolejo‐Ramos, F., Marrone, R. L., Korolkiewicz, M. W., Gabriel, F. C., Siemens, G., Joksimovíc, S., Yamada, Y., Mori, Y., Rahwan, T., Sahakyan, M., et al. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1465605 · Cited by 41 · see citing works
Transparent Human Evaluation for Image Captioning
Kasai, J., Sakaguchi, K., Dunagan, L., Morrison, J., Bras, R. J. L., Choi, Y., & Smith, N. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.254 · Cited by 34 · see citing works
Self-citationDoes Human Collaboration Enhance the Accuracy of Identifying LLM-Generated Deepfake Texts?
Uchendu, A., Lee, J., Shen, H., Le, T., Huang, T.-H., & Lee, D. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27557 · Cited by 35 · see citing works
Dual use concerns of generative AI and large language models
Grinbaum, A., & Adomaitis, L. (2024) · Journal of Responsible Innovation · doi:10.1080/23299460.2024.2304381 · Cited by 35 · see citing works
Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective
Uchendu, A., Le, T., & Lee, D. (2023) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3606274.3606276 · Cited by 34 · see citing works
Advancing health coaching: A comparative study of large language model and health coaches
Ong, Q. C., Ang, C., Chee, D. Z. Y., Lawate, A., Sundram, F., Dalakoti, M., Pasalic, L., To, D., Fox, T. E., Bojić, I., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103004 · Cited by 27 · see citing works
The Next Chapter: A Study of Large Language Models in Storytelling
Xie, Z., Cohn, T., & Lau, J. H. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.23 · Cited by 41 · see citing works
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.121 · Cited by 29 · see citing works
Quantifying the narrative flow of imagined versus autobiographical stories
Sap, M., Jafarpour, A., Choi, Y., Smith, N. A., Pennebaker, J. W., & Horvitz, E. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2211715119 · Cited by 31 · see citing works
Self-citationStudents are using large language models and AI detectors can often detect their use
Paustian, T., & Slinger, B. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1374889 · Cited by 27 · see citing works
Can novice teachers detect AI-generated texts in EFL writing?
Wilde, V. D. (2024) · ELT Journal · doi:10.1093/elt/ccae031 · Cited by 27 · see citing works
Chhun, C., Suchanek, F. M., & Clavel, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00689 · Cited by 30 · see citing works
A Comprehensive Survey of Fake Text Detection on Misinformation and LM-Generated Texts
Kwon, S.-C., & Jang, B. (2025) · IEEE Access · doi:10.1109/access.2025.3538805 · Cited by 28 · see citing works
A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games
Akoury, N., Yang, Q., & Iyyer, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.151 · Cited by 26 · see citing works
Detection of AI-Generated Texts: A Bi-LSTM and Attention-Based Approach
Blake, J., Miah, A. S. M., Kredens, K., & Shin, J. (2025) · IEEE Access · doi:10.1109/access.2025.3562750 · Cited by 27 · see citing works
SwahBERT: Language Model of Swahili
Martin, G. L., Mswahili, M. E., Jeong, Y.-S., & Young-Seob, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.23 · Cited by 25 · see citing works
Writing without borders: AI and cross-cultural convergence in academic writing quality
Prakash, A., Aggarwal, S., Varghese, J. J., & Varghese, J. J. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05484-6 · Cited by 29 · see citing works
Larooij, M., & Törnberg, P. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11412-6 · Cited by 27 · see citing works
Synthetic Disinformation Attacks on Automated Fact Verification Systems
Du, Y., Bosselut, A., & Manning, C. D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21302 · Cited by 27 · see citing works
Mutual Information Alleviates Hallucinations in Abstractive Summarization
van der Poel, L., Cotterell, R., & Meister, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.399 · Cited by 35 · see citing works
Beyond Generating Code: Evaluating GPT on a Data Visualization Course
Chen, Z., Zhang, C., Wang, Q., Troidl, J., Warchol, S., Beyer, J., Gehlenborg, N., & Pfister, H. (2023) · IEEE VIS Workshop on Visualization Education, Literacy, and Activities (EduVis) · doi:10.1109/eduvis60792.2023.00009 · Cited by 23 · see citing works
Atcheson, A., Khan, O., Siemann, B., Jain, A., & Karahalios, K. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714121 · Cited by 22 · see citing works
Towards Automatic Generation of Messages Countering Online Hate Speech and Microaggressions
Ashida, M., & Komachi, M. (2022) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2022.woah-1.2 · Cited by 22 · see citing works
Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education
Zeng, Z., Sha, L., Li, Y., Yang, K., Gašević, D., & Chen, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30258 · Cited by 25 · see citing works
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand
Kasai, J., Sakaguchi, K., Bras, R. J. L., Dunagan, L., Morrison, J., Fabbri, A. R., Choi, Y., & Smith, N. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.259 · Cited by 19 · see citing works
Self-citationThe ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practices
Hawkins, W., & Mittelstadt, B. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593995 · Cited by 19 · see citing works
SNaC: Coherence Error Detection for Narrative Summarization
Goyal, T., Li, J. J., & Durrett, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.29 · Cited by 21 · see citing works
Markowitz, D. M. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02344-0 · Cited by 19 · see citing works
Nañola, E. L., Arroyo, R. L., Hermosura, N. J. T., Ragil, M., Sabanal, J. N. U., & Mendoza, H. (2025) · System · doi:10.1016/j.system.2025.103611 · Cited by 24 · see citing works
A User-Centric Analysis of Social Media for Stock Market Prediction
Bouadjenek, M. R., Sanner, S., & Wu, G. (2022) · ACM Transactions on the Web · doi:10.1145/3532856 · Cited by 18 · see citing works
Supporting Text Entry in Virtual Reality with Large Language Models
Chen, L., Cai, Y., Wang, R., Ding, S., Tang, Y., Hansen, P., & Sun, L. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00073 · Cited by 18 · see citing works
Zhai, Y., Xue, X., Guo, Z., Jin, T., Diao, Y., & Jeung, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714423 · Cited by 17 · see citing works
Detection Vs. Anti-detection: Is Text Generated by AI Detectable?
Zhang, Y., Ma, Y., Liu, J., Liu, X., Wang, X., & Lu, W. P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57850-2_16 · Cited by 19 · see citing works
Schuster, S., & Linzen, T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.71 · Cited by 17 · see citing works
Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches, and Future Directions
Jia, Q., Liu, Y., Ren, S., & Zhu, K. Q. (2023) · ACM Computing Surveys · doi:10.1145/3622933 · Cited by 17 · see citing works
AI or Human? Finding and Responding to Artificial Intelligence in Student Work
Fisk, G. D. (2024) · Teaching of Psychology · doi:10.1177/00986283241251855 · Cited by 16 · see citing works
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Zhou, K., Blodgett, S. L., Trischler, A., Daumé, H., Suleman, K., & Olteanu, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.24 · Cited by 16 · see citing works
Studying Psychosis Using Natural Language Generation: A Review of Emerging Opportunities
Palaniyappan, L. K., Benrimoh, D. A., Voppel, A., & Rocca, R. (2023) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2023.04.009 · Cited by 14 · see citing works
Pitfalls (and advantages) of sophisticated large language models
Strasser, A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-18851-0.00007-x · Cited by 18 · see citing works
Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian
Shamardina, T., Mikhailov, V., Chernianskii, D., Fenogenova, A., Saidov, M., Valeeva, A., Shavrina, T., Smurov, I., Tutubalina, E. V., & Artemova, E. (2022) · Computational Linguistics and Intellectual Technologies · doi:10.28995/2075-7182-2022-21-497-511 · Cited by 22 · see citing works
Generative AI and Perceptual Harms: Who's Suspected of using LLMs?
Kadoma, K., Metaxa, D., & Naaman, M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713897 · Cited by 14 · see citing works
WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering
Bolotova-Baranova, V., Blinov, V., Филиппова, С. В., Scholer, F., & Sanderson, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.290 · Cited by 14 · see citing works
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation
Khashabi, D., Stanovsky, G., Bragg, J., Lourie, N., Kasai, J., Choi, Y., Smith, N. A., & Weld, D. S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.787 · Cited by 38 · see citing works
Self-citationBeyond Lexical Boundaries: LLM-Generated Text Detection for Romanian Digital Libraries
Nițu, M., & Dascălu, M. (2024) · Future Internet · doi:10.3390/fi16020041 · Cited by 14 · see citing works
Unsupervised and Distributional Detection of Machine-Generated TextPreprint
Gallé, M., Rozen, J., Kruszewski, G., & Elsahar, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02878 · Cited by 27 · see citing works
- Authors
- Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A.
- Affiliations
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 7282-7296
- arXiv
- 2107.00061
- OpenAlex
- W3174007554
- Cited by
- 290
- References
- 36
- Citations by year
- 2026: 28 · 2025: 68 · 2024: 59 · 2023: 87 · 2022: 41 · 2021: 7
- FWCI
- 22.41
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Multidisciplinary University Research Initiative
- Office of Naval Research (N00014‐18‐1‐2670, N00014-18, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Text Readability and Simplification
- Keywords
- human evaluation of generated text, assessment reliability, interobserver agreement, human annotators, text quality assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
