The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Human evaluation of automatically generated text: Current trends and best practice guidelines
Open accessAbstract
Currently, there is little agreement as to how Natural Language Generation (NLG) systems should be evaluated, with a particularly high degree of variation in the way that human evaluation is carried out. This paper provides an overview of how (mostly intrinsic) human evaluation is currently conducted and presents a set of best practices, grounded in the literature. These best practices are also linked to the stages that researchers go through when conducting an evaluation research (planning stage; execution and release stage), and the specific steps in these stages. With this paper, we hope to contribute to the quality and consistency of human evaluations in NLG.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1956) · Psychological Review · doi:10.1037/h0043158 · Cited by 17,812
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Research works printed but not linked: 72
- Amidei, 2018, Evaluation methodologies in Automatic Question Generation 2013–2018, INLG 2018, 307
- Amidei, 2019, The use of rating and Likert scales in Natural Language Generation human evaluation tasks: a review and some recommendations, 397
- Anduiza, 2016, Answering without reading: IMCs and strong satisficing in online surveys, Int. J. Public Opin. Res., 29, 497
- Ayers, 2018, Don’t quote me: reverse identification of research participants in social media studies, NPJ Digit. Med., 1, 1, 10.1038/s41746-018-0036-2 · DOI
- Barrault, 2019, Findings of the 2019 conference on machine translation (WMT19), 1
- Bem, 1987, Writing the empirical journal, 171
- Berinsky, 2016, Can we turn shirkers into workers?, J. Exp. Soc. Psychol., 66, 20, 10.1016/j.jesp.2015.09.010 · DOI
- Bernhard, 2012, Question generation for french: collating parsers and paraphrasing questions, Dialogue Discourse, 3, 43, 10.5087/dad.2012.203 · DOI
- Brysbaert, 2019, How many participants do we have to include in properly powered experiments? A tutorial of power analysis with reference tables, J. Cogn., 2, 1
- Carr, 1994, The strengths and weaknesses of quantitative and qualitative research: what method for nursing?, J. Adv. Nurs., 20, 716, 10.1046/j.1365-2648.1994.20040716.x · DOI · Paper page
- Cicchetti, 1985, The effect of number of rating scale categories on levels of interrater reliability: a Monte Carlo investigation, Appl. Psychol. Meas., 9, 31, 10.1177/014662168500900103 · DOI · Paper page
- Claesen, 2019, Preregistration: comparing dream to reality, PsyArXiv
- Clark, 1973, The language-as-fixed-effect fallacy: a critique of language statistics in psychological research, J. Verb. Learn. Verb. Behav., 12, 335, 10.1016/S0022-5371(73)80014-3 · DOI · Paper page
- Clifford, 2015, Do attempts to improve respondent attention increase social desirability bias?, Public Opin. Q, 79, 790, 10.1093/poq/nfv027 · DOI
- Colbert, 2019, Best practice versus actual practice: an audit of survey pretesting practices reported in a sample of medical education journals, Med. Educ. Online, 24, 1, 10.1080/10872981.2019.1673596 · DOI
- Colby, 1971, Artificial paranoia, Artif. Intell., 2, 1, 10.1016/0004-3702(71)90002-6 · DOI · Paper page
- Corral, 2015, Zipf’s law for word frequencies: word forms versus lemmas in long texts, PloS One, 10, e0129031, 10.1371/journal.pone.0129031 · DOI
- Crawford, 2001, Web surveys: perceptions of burden, Soc. Sci. Comput. Rev., 19, 146, 10.1177/089443930101900202 · DOI · Paper page
- Diamantopoulos, 2012, Guidelines for choosing between multi-item and single-item scales for construct measurement: a predictive validity perspective, J. Acad. Mark. Sci., 40, 434, 10.1007/s11747-011-0300-3 · DOI · Paper page
- Ejelöv, 2020, “rarely safe to assume”: evaluating the use and interpretation of manipulation checks in experimental social psychology, J. Exp. Soc. Psychol., 87, 1, 10.1016/j.jesp.2019.103937 · DOI · Paper page
- Emmery, 2019, Towards replication in computational cognitive modeling: a machine learning perspective, Comput. Brain Behav., 2, 242, 10.1007/s42113-019-00055-w · DOI
- Fulgoni, 2014, Uses and misuses of online-survey panels in digital research, J. Advert Res., 54, 133, 10.2501/JAR-54-2-133-137 · doi:10.2501/jar-54-2-133-137
- Goegan, 2018, Accessibility in questionnaire research: integrating universal design to increase the participation of individuals with learning disabilities., Learn. Disabil.: Contemp. J., 16, 177
- Goldberg, 1994, Using natural language processing to produce weather forecasts, IEEE Expert, 2, 45, 10.1109/64.294135 · DOI · Paper page
- Green, 1970, Rating scales and information recovery: how many scales and response categories to use?, J. Mark., 34, 33
- Hastie, 2014, A comparative evaluation methodology for NLG in interactive systems, 4004, 10.63317/5e52j6frtshv · DOI
- Hauser, 2018, Are manipulation checks necessary?, Front. Psychol., 9, 1, 10.3389/fpsyg.2018.00998 · DOI · Paper page
- Koplenig, 2017, Against statistical significance testing in corpus linguistics, Corpus Linguist. Linguist. Theory
- Krawczyk, 2012, (Un) available upon request: field experiment on researchers’ willingness to share supplementary materials, Account. Res., 19, 175, 10.1080/08989621.2012.678688 · DOI
- Lacity, 1994, Understanding qualitative data: a framework of text analysis methods, J. Manag. Inf. Syst., 11, 137, 10.1080/07421222.1994.11518043 · DOI · Paper page
- Lakens, 2018, Justify your alpha, Nat. Hum. Behav., 2, 168, 10.1038/s41562-018-0311-x · DOI · Paper page
- Langsford, 2018, Quantifying sentence acceptability measures: reliability, bias, and variability, Glossa: J. Gen. Linguist., 3, 1
- Lentz, 1997, The evaluation of text quality: expert-focused and reader-focused methods compared, IEEE Trans.Prof. Commun., 40, 224, 10.1109/47.649557 · DOI
- Lindley, 2016, Operationalising design fiction for ethical computing, SIGCAS Comput. Soc., 45, 79, 10.1145/2874239.2874251 · doi:10.1145/2874239.2874251
- Manfreda, 2002, Design of web survey questionnaires: three basic experiments, J. Comput.-Mediated Commun., 7, 1
- Marshall, 2006, Informed consent in international health research, J. Empir. Res. Hum. Res. Ethics, 1, 25, 10.1525/jer.2006.1.1.25 · DOI
- Martinez, 2014, Don’t classify ratings of affect; rank them!, IEEE Trans. Affect. Comput., 5, 314, 10.1109/TAFFC.2014.2352268 · DOI
- McHugh, 2012, Interrater reliability: the Kappa statistic, Biochem. Med., 22, 276, 10.11613/BM.2012.031 · DOI · Paper page
- McShane, 2019, Abandon statistical significance, Am. Stat., 73, 235, 10.1080/00031305.2018.1527253 · DOI · Paper page
- Mellish, 1998, Evaluation in the context of natural language generation, Comput. Speech Lang., 12, 349, 10.1006/csla.1998.0106 · DOI
- Meteyard, 2020, Best practice guidance for linear mixed-effects models in psychological science, J. Mem. Lang., 112, 104092, 10.1016/j.jml.2020.104092 · DOI · Paper page
- Munafò, 2018, Robust research needs many lines of evidence, Nature, 553, 399, 10.1038/d41586-018-01023-3 · DOI · Paper page
- Murphy, 2019, HARKing: how badly can cherry-picking and question trolling produce bias in published results?, J. Bus. Psychol., 34, 1, 10.1007/s10869-017-9524-7 · DOI
- Nespor, 2000, Anonymity and place in qualitative inquiry, Qualit. Inq., 6, 546, 10.1177/107780040000600408 · DOI · Paper page
- Normand, 2016, Less is more: psychologists can learn more by studying fewer people, Front. Psychol., 7, 934, 10.3389/fpsyg.2016.00934 · DOI
- Paasche-Orlow, 2003, Readability standards for informed-consent forms as compared with actual readability, New Engl. J. mMedicine, 348, 721, 10.1056/NEJMsa021212 · DOI · Paper page
- Peytchev, 2006, Web survey design: paging versus scrolling, Int. J. Public Opin. Q., 70, 596, 10.1093/poq/nfl028 · DOI
- Popel, 2009, Improving english-czech tectogrammatical MT, Prague Bull. Math. Linguist., 92, 115, 10.2478/v10108-009-0025-3 · doi:10.2478/v10108-009-0025-3
- Portet, 2009, Automatic generation of textual summaries from neonatal intensive care data, Artif. Intell., 173, 789, 10.1016/j.artint.2008.12.002 · DOI · Paper page
- Postma, 2016, Addressing the MFS bias in WSD systems, 1695, 10.63317/2qcfqqj3yifp · DOI
- Potter, 1999, Rethinking validity and reliability in content analysis, J. Appl. Commun. Res., 27, 258, 10.1080/00909889909365539 · DOI · Paper page
- Preston, 2000, Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences, Acta Psychol., 104, 1, 10.1016/S0001-6918(99)00050-5 · DOI · Paper page
- Radziwill, 2017, Evaluating quality of chatbots and intelligent conversational agents, CoRR
- Ramos-Soto, 2015, Linguistic descriptions for automatic generation of textual short-Term weather forecasts on real prediction data, IEEE Trans. Fuzzy Syst., 23, 44, 10.1109/TFUZZ.2014.2328011 · DOI
- Reiter, 2009, An investigation into the validity of some metrics for automatically evaluating natural language generation systems, Comput. Linguist., 35, 529, 10.1162/coli.2009.35.4.35405 · DOI · Paper page
- Reiter, 2005, Choosing words in computer-generated weather forecasts, Artif. Intell., 167, 137, 10.1016/j.artint.2005.06.006 · DOI · Paper page
- Rice, 2008, How to do human-subjects research if you do not have an institutional review board, Respir. Care, 53, 1362, 10.4187/respcare.08531362 · doi:10.4187/respcare.08531362
- Rodgers, 2017, The genealogy of an image, or, what does literature (not) have to do with the history of computing?Tracing the sources and reception of gUlliver’s “ knowledge engine”, Humanities, 6, 85, 10.3390/h6040085 · DOI
- Ruggles, 2003, IPUMS redesign, Histor. Methods: J. Quant. Interdiscip. Hist., 36, 9, 10.1080/01615440309601210 · doi:10.1080/01615440309601210
- Sachs, 2003, Conducting empirical research on informed consent: challenges and questions, IRB: Ethics Hum. Res., 25, 4, 10.2307/3564116 · DOI
- Silberman, 2018, Responsible research with crowds: pay crowdworkers at least minimum wage, Commun. ACM, 61, 39, 10.1145/3180492 · DOI · Paper page
- Sterling, 2018, Investigating the complexity of consent forms in ESL research, J. Res. Desi. Stat. Linguist. Commun. Sci., 4, 156
- Stieger, 2007, Forced-response in online surveys: Bias from reactance and an increase in sex-specific dropout, J. Am. Soc. Inf. Sci. Technol., 58, 1653, 10.1002/asi.20651 · DOI
- Stunkel, 2010, Comprehension and informed consent: assessing the effect of a short consent form, IRB: Ethics Hum. Res., 32, 1
- Sulem, E., Abend, O., Rappoport, A., BLEU is not suitable for the evaluation of text simplification. http://arxiv.org/abs/1810.05995arXiv preprint arXiv:1810.05995 Accepted for publication as a short paper at EMNLP 2018. 10.18653/v1/D18-1081 · DOI
- Tukey, 1980, We need both exploratory and confirmatory, Am. Stat., 34, 23, 10.1080/00031305.1980.10482706 · DOI · Paper page
- Van Enschot, 2017, Taming our wild data: on intercoder reliability in discourse research, Unpublished Manuscript
- Van Heuven, 2014, Subtlex-UK: a new and improved word frequency database for british english, Q. J. Exp. Psychol., 67, 1176, 10.1080/17470218.2013.850521 · DOI · Paper page
- Vicente, 2010, Using questionnaire design to fight nonresponse bias in web surveys, Soc. Sci. Comput. Rev., 28, 251, 10.1177/0894439309340751 · doi:10.1177/0894439309340751
- Wagenmakers, 2011, Why psychologists must change the way they analyze their data: the case of PSI: comment on bEM (2011)., J. Pers. Soc. Psychol., 100, 426, 10.1037/a0022790 · DOI · Paper page
- Yannakakis, 2015, Ratings are overrated!, Front. ICT, 2, 13, 10.3389/fict.2015.00013 · DOI
- Yu, 2017, The effectiveness of a monetary incentive offer on survey response rates and response completeness in a longitudinal study, BMC Med. Res. Methodol., 17, 77, 10.1186/s12874-017-0353-1 · DOI
Cited by
Longo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models
Zhang, H., Song, H., Li, S., Zhou, M., & Song, D. (2024) · ACM Computing Surveys · doi:10.1145/3617680 · Cited by 317
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT
Dai, W., Lin, J., Jin, H., Li, T., Tsai, Y.-S., Gašević, D., & Chen, G. (2023) · 2023 IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt58122.2023.00100 · Cited by 395
“, readability [11], [13] to examine the quality of the machinegenerated text.”
Evaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207
Karinshak, E., Liu, S. X., Park, J. S., & Hancock, J. T. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579592 · Cited by 167
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123
Dai, W., Tsai, Y.-S., Lin, J., Aldino, A., Jin, H., Li, T., Gašević, D., & Chen, G. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100299 · Cited by 93
Assessing the quality of student-generated short answer questions using GPT-3
Moore, S., Nguyen, H. A., Bier, N., Domadia, T., & Stamper, J. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-16290-9_18 · Cited by 93
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
Truth machines: synthesizing veracity in AI language models
Munn, L., Magee, L., & Arora, V. (2024) · AI & SOCIETY · doi:10.1007/s00146-023-01756-4 · Cited by 65
“While RLHF knowledge leverages human insight, this is attended by all-too-human subjectivity. van der Lee et al (2021) worry that annotators will engage in “satisficing,” succumbing to tedium and fatigue and taking shortcuts to arrive at low-quality answers.”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Karpinska, M., Akoury, N., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.97 · Cited by 56
“…2018-2020 and find many critical details often go unreported (e.g., worker qualifications, payment, task descriptions, annotator agreement), a finding in line with prior reproducibility studies outside open-ended text generation (Card et al., 2020; Howcroft et al., 2020; van der Lee et al., 2021).”
ResultErdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
de Hond, A., Leeuwenberg, T., Bartels, R., van Buchem, M., Kant, I., Moons, K. G., & van Smeden, M. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00111-0 · Cited by 53
On Decoding Strategies for Neural Text Generators
Wiher, G., Meister, C., & Cotterell, R. (2022) · arXiv · doi:10.48550/arxiv.2203.15721 · Cited by 48
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.745 · Cited by 47
“We wrote 8 dialogue evaluation questions ( Q 1 , . . . , Q 8 ) inspired by the NLG evaluation literature (Zhou et al., 2022; van der Lee et al., 2021).”
Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPT
Bhavya, B., Xiong, J., & Zhai, C. (2022) · Proceedings of the 15th International Conference on Natural Language Generation · doi:10.18653/v1/2022.inlg-main.25 · Cited by 40
Value attributed to text-based archives generated by artificial intelligence
Darda, K., Carre, M., & Cross, E. (2023) · Royal Society Open Science · doi:10.1098/rsos.220915 · Cited by 37
“Given the increasing (and unprecedented) potential of NLG algorithms and the many ethical challenges they raise such as being a potential tool for misinformation [8], it is necessary to understand not just how these texts are generated from a technological or algorithmic point of view, but also how they are evaluated…”
BackgroundViewpoint: AI as Author – Bridging the Gap Between Machine Learning and Literary Theory
Van Heerden, I., & Bas, A. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12593 · Cited by 29
Toward Clinical-Grade Evaluation of Large Language Models
Moreno, A. C., & Bitterman, D. S. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2023.11.012 · Cited by 28
The Authenticity Gap in Human Evaluation
Ethayarajh, K., & Jurafsky, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.406 · Cited by 10
Metadata
- Authors
- van der Lee, C., Gatt, A., van Miltenburg, E., & Krahmer, E.
- Venue
- Computer Speech & Language
- Publisher
- Elsevier BV
- Published
- 2021-05
- Volume
- 67
- Pages
- 101151
- ISSN
- 0885-2308 (ISSN-L)
- OpenAlex
- W3107855336
- Cited by
- 170
- References
- 44
- FWCI
- 7.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Funders
- Nationaal Regieorgaan Praktijkgericht Onderzoek SIA (2014-01-51PRO)
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (360-89-050)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.