The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Human evaluation of automatically generated text: Current trends and best practice guidelines
Open accessAbstract
Currently, there is little agreement as to how Natural Language Generation (NLG) systems should be evaluated, with a particularly high degree of variation in the way that human evaluation is carried out. This paper provides an overview of how (mostly intrinsic) human evaluation is currently conducted and presents a set of best practices, grounded in the literature. These best practices are also linked to the stages that researchers go through when conducting an evaluation research (planning stage; execution and release stage), and the specific steps in these stages. With this paper, we hope to contribute to the quality and consistency of human evaluations in NLG.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1956) · Psychological Review · doi:10.1037/h0043158 · Cited by 17,812
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Research works printed but not linked: 74
- Amidei, 2018, Evaluation methodologies in Automatic Question Generation 2013–2018, INLG 2018, 307
- Amidei, 2019, The use of rating and Likert scales in Natural Language Generation human evaluation tasks: a review and some recommendations, 397
- Anduiza, 2016, Answering without reading: IMCs and strong satisficing in online surveys, Int. J. Public Opin. Res., 29, 497
- Ayers, 2018, Don’t quote me: reverse identification of research participants in social media studies, NPJ Digit. Med., 1, 1, 10.1038/s41746-018-0036-2 · DOI
- Barrault, 2019, Findings of the 2019 conference on machine translation (WMT19), 1
- Bem, 1987, Writing the empirical journal, 171
- Berinsky, 2016, Can we turn shirkers into workers?, J. Exp. Soc. Psychol., 66, 20, 10.1016/j.jesp.2015.09.010 · DOI
- Bernhard, 2012, Question generation for french: collating parsers and paraphrasing questions, Dialogue Discourse, 3, 43, 10.5087/dad.2012.203 · DOI
- Brysbaert, 2019, How many participants do we have to include in properly powered experiments? A tutorial of power analysis with reference tables, J. Cogn., 2, 1
- Carr, 1994, The strengths and weaknesses of quantitative and qualitative research: what method for nursing?, J. Adv. Nurs., 20, 716, 10.1046/j.1365-2648.1994.20040716.x · DOI · Paper page
- Cicchetti, 1985, The effect of number of rating scale categories on levels of interrater reliability: a Monte Carlo investigation, Appl. Psychol. Meas., 9, 31, 10.1177/014662168500900103 · DOI · Paper page
- Claesen, 2019, Preregistration: comparing dream to reality, PsyArXiv
- Clark, 1973, The language-as-fixed-effect fallacy: a critique of language statistics in psychological research, J. Verb. Learn. Verb. Behav., 12, 335, 10.1016/S0022-5371(73)80014-3 · DOI · Paper page
- Clifford, 2015, Do attempts to improve respondent attention increase social desirability bias?, Public Opin. Q, 79, 790, 10.1093/poq/nfv027 · DOI
- Colbert, 2019, Best practice versus actual practice: an audit of survey pretesting practices reported in a sample of medical education journals, Med. Educ. Online, 24, 1, 10.1080/10872981.2019.1673596 · DOI
- Colby, 1971, Artificial paranoia, Artif. Intell., 2, 1, 10.1016/0004-3702(71)90002-6 · DOI · Paper page
- Corral, 2015, Zipf’s law for word frequencies: word forms versus lemmas in long texts, PloS One, 10, e0129031, 10.1371/journal.pone.0129031 · DOI
- Crawford, 2001, Web surveys: perceptions of burden, Soc. Sci. Comput. Rev., 19, 146, 10.1177/089443930101900202 · DOI · Paper page
- Diamantopoulos, 2012, Guidelines for choosing between multi-item and single-item scales for construct measurement: a predictive validity perspective, J. Acad. Mark. Sci., 40, 434, 10.1007/s11747-011-0300-3 · DOI · Paper page
- Ejelöv, 2020, “rarely safe to assume”: evaluating the use and interpretation of manipulation checks in experimental social psychology, J. Exp. Soc. Psychol., 87, 1, 10.1016/j.jesp.2019.103937 · DOI · Paper page
- Emmery, 2019, Towards replication in computational cognitive modeling: a machine learning perspective, Comput. Brain Behav., 2, 242, 10.1007/s42113-019-00055-w · DOI
- Flory, 2004, Interventions to improve research participants’ understanding in informed consent for research: a systematic review, J. Am. Med. Assoc., 292, 1593, 10.1001/jama.292.13.1593 · DOI · Paper page
- Fulgoni, 2014, Uses and misuses of online-survey panels in digital research, J. Advert Res., 54, 133, 10.2501/JAR-54-2-133-137 · doi:10.2501/jar-54-2-133-137
- Goegan, 2018, Accessibility in questionnaire research: integrating universal design to increase the participation of individuals with learning disabilities., Learn. Disabil.: Contemp. J., 16, 177
- Goldberg, 1994, Using natural language processing to produce weather forecasts, IEEE Expert, 2, 45, 10.1109/64.294135 · DOI · Paper page
- Green, 1970, Rating scales and information recovery: how many scales and response categories to use?, J. Mark., 34, 33
- Hastie, 2014, A comparative evaluation methodology for NLG in interactive systems, 4004, 10.63317/5e52j6frtshv · DOI
- Hauser, 2018, Are manipulation checks necessary?, Front. Psychol., 9, 1, 10.3389/fpsyg.2018.00998 · DOI · Paper page
- Koplenig, 2017, Against statistical significance testing in corpus linguistics, Corpus Linguist. Linguist. Theory
- Krawczyk, 2012, (Un) available upon request: field experiment on researchers’ willingness to share supplementary materials, Account. Res., 19, 175, 10.1080/08989621.2012.678688 · DOI
- Lacity, 1994, Understanding qualitative data: a framework of text analysis methods, J. Manag. Inf. Syst., 11, 137, 10.1080/07421222.1994.11518043 · DOI · Paper page
- Lakens, 2018, Justify your alpha, Nat. Hum. Behav., 2, 168, 10.1038/s41562-018-0311-x · DOI · Paper page
- Langsford, 2018, Quantifying sentence acceptability measures: reliability, bias, and variability, Glossa: J. Gen. Linguist., 3, 1
- Lentz, 1997, The evaluation of text quality: expert-focused and reader-focused methods compared, IEEE Trans.Prof. Commun., 40, 224, 10.1109/47.649557 · DOI
- Lindley, 2016, Operationalising design fiction for ethical computing, SIGCAS Comput. Soc., 45, 79, 10.1145/2874239.2874251 · doi:10.1145/2874239.2874251
- Lissitz, 1975, Effect of the number of scale points on reliability: a Monte Carlo approach., J. Appl. Psychol., 60, 10, 10.1037/h0076268 · DOI · Paper page
- Manfreda, 2002, Design of web survey questionnaires: three basic experiments, J. Comput.-Mediated Commun., 7, 1
- Marshall, 2006, Informed consent in international health research, J. Empir. Res. Hum. Res. Ethics, 1, 25, 10.1525/jer.2006.1.1.25 · DOI
- Martinez, 2014, Don’t classify ratings of affect; rank them!, IEEE Trans. Affect. Comput., 5, 314, 10.1109/TAFFC.2014.2352268 · DOI
- McHugh, 2012, Interrater reliability: the Kappa statistic, Biochem. Med., 22, 276, 10.11613/BM.2012.031 · DOI · Paper page
- McShane, 2019, Abandon statistical significance, Am. Stat., 73, 235, 10.1080/00031305.2018.1527253 · DOI · Paper page
- Mellish, 1998, Evaluation in the context of natural language generation, Comput. Speech Lang., 12, 349, 10.1006/csla.1998.0106 · DOI
- Meteyard, 2020, Best practice guidance for linear mixed-effects models in psychological science, J. Mem. Lang., 112, 104092, 10.1016/j.jml.2020.104092 · DOI · Paper page
- Munafò, 2018, Robust research needs many lines of evidence, Nature, 553, 399, 10.1038/d41586-018-01023-3 · DOI · Paper page
- Murphy, 2019, HARKing: how badly can cherry-picking and question trolling produce bias in published results?, J. Bus. Psychol., 34, 1, 10.1007/s10869-017-9524-7 · DOI
- Nespor, 2000, Anonymity and place in qualitative inquiry, Qualit. Inq., 6, 546, 10.1177/107780040000600408 · DOI · Paper page
- Normand, 2016, Less is more: psychologists can learn more by studying fewer people, Front. Psychol., 7, 934, 10.3389/fpsyg.2016.00934 · DOI
- Paasche-Orlow, 2003, Readability standards for informed-consent forms as compared with actual readability, New Engl. J. mMedicine, 348, 721, 10.1056/NEJMsa021212 · DOI · Paper page
- Peytchev, 2006, Web survey design: paging versus scrolling, Int. J. Public Opin. Q., 70, 596, 10.1093/poq/nfl028 · DOI
- Popel, 2009, Improving english-czech tectogrammatical MT, Prague Bull. Math. Linguist., 92, 115, 10.2478/v10108-009-0025-3 · doi:10.2478/v10108-009-0025-3
- Portet, 2009, Automatic generation of textual summaries from neonatal intensive care data, Artif. Intell., 173, 789, 10.1016/j.artint.2008.12.002 · DOI · Paper page
- Postma, 2016, Addressing the MFS bias in WSD systems, 1695, 10.63317/2qcfqqj3yifp · DOI
- Potter, 1999, Rethinking validity and reliability in content analysis, J. Appl. Commun. Res., 27, 258, 10.1080/00909889909365539 · DOI · Paper page
- Preston, 2000, Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences, Acta Psychol., 104, 1, 10.1016/S0001-6918(99)00050-5 · DOI · Paper page
- Radziwill, 2017, Evaluating quality of chatbots and intelligent conversational agents, CoRR
- Ramos-Soto, 2015, Linguistic descriptions for automatic generation of textual short-Term weather forecasts on real prediction data, IEEE Trans. Fuzzy Syst., 23, 44, 10.1109/TFUZZ.2014.2328011 · DOI
- Reiter, 2009, An investigation into the validity of some metrics for automatically evaluating natural language generation systems, Comput. Linguist., 35, 529, 10.1162/coli.2009.35.4.35405 · DOI · Paper page
- Reiter, 2005, Choosing words in computer-generated weather forecasts, Artif. Intell., 167, 137, 10.1016/j.artint.2005.06.006 · DOI · Paper page
- Rice, 2008, How to do human-subjects research if you do not have an institutional review board, Respir. Care, 53, 1362, 10.4187/respcare.08531362 · doi:10.4187/respcare.08531362
- Rodgers, 2017, The genealogy of an image, or, what does literature (not) have to do with the history of computing?Tracing the sources and reception of gUlliver’s “ knowledge engine”, Humanities, 6, 85, 10.3390/h6040085 · DOI
- Ruggles, 2003, IPUMS redesign, Histor. Methods: J. Quant. Interdiscip. Hist., 36, 9, 10.1080/01615440309601210 · doi:10.1080/01615440309601210
- Sachs, 2003, Conducting empirical research on informed consent: challenges and questions, IRB: Ethics Hum. Res., 25, 4, 10.2307/3564116 · DOI
- Silberman, 2018, Responsible research with crowds: pay crowdworkers at least minimum wage, Commun. ACM, 61, 39, 10.1145/3180492 · DOI · Paper page
- Sterling, 2018, Investigating the complexity of consent forms in ESL research, J. Res. Desi. Stat. Linguist. Commun. Sci., 4, 156
- Stieger, 2007, Forced-response in online surveys: Bias from reactance and an increase in sex-specific dropout, J. Am. Soc. Inf. Sci. Technol., 58, 1653, 10.1002/asi.20651 · DOI
- Stunkel, 2010, Comprehension and informed consent: assessing the effect of a short consent form, IRB: Ethics Hum. Res., 32, 1
- Sulem, E., Abend, O., Rappoport, A., BLEU is not suitable for the evaluation of text simplification. http://arxiv.org/abs/1810.05995arXiv preprint arXiv:1810.05995 Accepted for publication as a short paper at EMNLP 2018. 10.18653/v1/D18-1081 · DOI
- Tukey, 1980, We need both exploratory and confirmatory, Am. Stat., 34, 23, 10.1080/00031305.1980.10482706 · DOI · Paper page
- Van Enschot, 2017, Taming our wild data: on intercoder reliability in discourse research, Unpublished Manuscript
- Van Heuven, 2014, Subtlex-UK: a new and improved word frequency database for british english, Q. J. Exp. Psychol., 67, 1176, 10.1080/17470218.2013.850521 · DOI · Paper page
- Vicente, 2010, Using questionnaire design to fight nonresponse bias in web surveys, Soc. Sci. Comput. Rev., 28, 251, 10.1177/0894439309340751 · doi:10.1177/0894439309340751
- Wagenmakers, 2011, Why psychologists must change the way they analyze their data: the case of PSI: comment on bEM (2011)., J. Pers. Soc. Psychol., 100, 426, 10.1037/a0022790 · DOI · Paper page
- Yannakakis, 2015, Ratings are overrated!, Front. ICT, 2, 13, 10.3389/fict.2015.00013 · DOI
- Yu, 2017, The effectiveness of a monetary incentive offer on survey response rates and response completeness in a longitudinal study, BMC Med. Res. Methodol., 17, 77, 10.1186/s12874-017-0353-1 · DOI
Cited by
Longo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models
Zhang, H., Song, H., Li, S., Zhou, M., & Song, D. (2024) · ACM Computing Surveys · doi:10.1145/3617680 · Cited by 317
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT
Dai, W., Lin, J., Jin, H., Li, T., Tsai, Y.-S., Gašević, D., & Chen, G. (2023) · 2023 IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt58122.2023.00100 · Cited by 395
“, readability [11], [13] to examine the quality of the machinegenerated text.”
Metadata
- Authors
- van der Lee, C., Gatt, A., van Miltenburg, E., & Krahmer, E.
- Venue
- Computer Speech & Language
- Publisher
- Elsevier BV
- Published
- 2021-05
- Volume
- 67
- Pages
- 101151
- ISSN
- 0885-2308 (ISSN-L)
- OpenAlex
- W3107855336
- Cited by
- 170
- References
- 38
- FWCI
- 7.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Funders
- Nationaal Regieorgaan Praktijkgericht Onderzoek SIA (2014-01-51PRO)
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (360-89-050)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Human evaluation of automatically generated text: Current trends and best practice guidelines. https://app.cochat.ai/papers/10.1016/j.csl.2020.101151
- The paper
- van der Lee, C., Gatt, A., van Miltenburg, E., & Krahmer, E. (2021). Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech & Language. https://doi.org/10.1016/j.csl.2020.101151
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.