Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 170

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.

Human evaluation of automatically generated text: Current trends and best practice guidelines

Open access
van der Lee, Chris, Gatt, Albert, van Miltenburg, Emiel, Krahmer, Emiel·2021·Computer Speech & Language

Abstract

Currently, there is little agreement as to how Natural Language Generation (NLG) systems should be evaluated, with a particularly high degree of variation in the way that human evaluation is carried out. This paper provides an overview of how (mostly intrinsic) human evaluation is currently conducted and presents a set of best practices, grounded in the literature. These best practices are also linked to the stages that researchers go through when conducting an evaluation research (planning stage; execution and release stage), and the specific steps in these stages. With this paper, we hope to contribute to the quality and consistency of human evaluations in NLG.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.csl.2020.101151

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 38 references

Research works printed but not linked: 74

  1. Amidei, 2018, Evaluation methodologies in Automatic Question Generation 2013–2018, INLG 2018, 307
  2. Amidei, 2019, The use of rating and Likert scales in Natural Language Generation human evaluation tasks: a review and some recommendations, 397
  3. Anduiza, 2016, Answering without reading: IMCs and strong satisficing in online surveys, Int. J. Public Opin. Res., 29, 497
  4. Ayers, 2018, Don’t quote me: reverse identification of research participants in social media studies, NPJ Digit. Med., 1, 1, 10.1038/s41746-018-0036-2 · DOI
  5. Barrault, 2019, Findings of the 2019 conference on machine translation (WMT19), 1
  6. Bem, 1987, Writing the empirical journal, 171
  7. Berinsky, 2016, Can we turn shirkers into workers?, J. Exp. Soc. Psychol., 66, 20, 10.1016/j.jesp.2015.09.010 · DOI
  8. Bernhard, 2012, Question generation for french: collating parsers and paraphrasing questions, Dialogue Discourse, 3, 43, 10.5087/dad.2012.203 · DOI
  9. Brysbaert, 2019, How many participants do we have to include in properly powered experiments? A tutorial of power analysis with reference tables, J. Cogn., 2, 1
  10. Carr, 1994, The strengths and weaknesses of quantitative and qualitative research: what method for nursing?, J. Adv. Nurs., 20, 716, 10.1046/j.1365-2648.1994.20040716.x · DOI · Paper page
  11. Cicchetti, 1985, The effect of number of rating scale categories on levels of interrater reliability: a Monte Carlo investigation, Appl. Psychol. Meas., 9, 31, 10.1177/014662168500900103 · DOI · Paper page
  12. Claesen, 2019, Preregistration: comparing dream to reality, PsyArXiv
  13. Clark, 1973, The language-as-fixed-effect fallacy: a critique of language statistics in psychological research, J. Verb. Learn. Verb. Behav., 12, 335, 10.1016/S0022-5371(73)80014-3 · DOI · Paper page
  14. Clifford, 2015, Do attempts to improve respondent attention increase social desirability bias?, Public Opin. Q, 79, 790, 10.1093/poq/nfv027 · DOI
  15. Colbert, 2019, Best practice versus actual practice: an audit of survey pretesting practices reported in a sample of medical education journals, Med. Educ. Online, 24, 1, 10.1080/10872981.2019.1673596 · DOI
  16. Colby, 1971, Artificial paranoia, Artif. Intell., 2, 1, 10.1016/0004-3702(71)90002-6 · DOI · Paper page
  17. Corral, 2015, Zipf’s law for word frequencies: word forms versus lemmas in long texts, PloS One, 10, e0129031, 10.1371/journal.pone.0129031 · DOI
  18. Crawford, 2001, Web surveys: perceptions of burden, Soc. Sci. Comput. Rev., 19, 146, 10.1177/089443930101900202 · DOI · Paper page
  19. Diamantopoulos, 2012, Guidelines for choosing between multi-item and single-item scales for construct measurement: a predictive validity perspective, J. Acad. Mark. Sci., 40, 434, 10.1007/s11747-011-0300-3 · DOI · Paper page
  20. Ejelöv, 2020, “rarely safe to assume”: evaluating the use and interpretation of manipulation checks in experimental social psychology, J. Exp. Soc. Psychol., 87, 1, 10.1016/j.jesp.2019.103937 · DOI · Paper page
  21. Emmery, 2019, Towards replication in computational cognitive modeling: a machine learning perspective, Comput. Brain Behav., 2, 242, 10.1007/s42113-019-00055-w · DOI
  22. Flory, 2004, Interventions to improve research participants’ understanding in informed consent for research: a systematic review, J. Am. Med. Assoc., 292, 1593, 10.1001/jama.292.13.1593 · DOI · Paper page
  23. Fulgoni, 2014, Uses and misuses of online-survey panels in digital research, J. Advert Res., 54, 133, 10.2501/JAR-54-2-133-137 · doi:10.2501/jar-54-2-133-137
  24. Goegan, 2018, Accessibility in questionnaire research: integrating universal design to increase the participation of individuals with learning disabilities., Learn. Disabil.: Contemp. J., 16, 177
  25. Goldberg, 1994, Using natural language processing to produce weather forecasts, IEEE Expert, 2, 45, 10.1109/64.294135 · DOI · Paper page
  26. Green, 1970, Rating scales and information recovery: how many scales and response categories to use?, J. Mark., 34, 33
  27. Hastie, 2014, A comparative evaluation methodology for NLG in interactive systems, 4004, 10.63317/5e52j6frtshv · DOI
  28. Hauser, 2018, Are manipulation checks necessary?, Front. Psychol., 9, 1, 10.3389/fpsyg.2018.00998 · DOI · Paper page
  29. Koplenig, 2017, Against statistical significance testing in corpus linguistics, Corpus Linguist. Linguist. Theory
  30. Krawczyk, 2012, (Un) available upon request: field experiment on researchers’ willingness to share supplementary materials, Account. Res., 19, 175, 10.1080/08989621.2012.678688 · DOI
  31. Lacity, 1994, Understanding qualitative data: a framework of text analysis methods, J. Manag. Inf. Syst., 11, 137, 10.1080/07421222.1994.11518043 · DOI · Paper page
  32. Lakens, 2018, Justify your alpha, Nat. Hum. Behav., 2, 168, 10.1038/s41562-018-0311-x · DOI · Paper page
  33. Langsford, 2018, Quantifying sentence acceptability measures: reliability, bias, and variability, Glossa: J. Gen. Linguist., 3, 1
  34. Lentz, 1997, The evaluation of text quality: expert-focused and reader-focused methods compared, IEEE Trans.Prof. Commun., 40, 224, 10.1109/47.649557 · DOI
  35. Lindley, 2016, Operationalising design fiction for ethical computing, SIGCAS Comput. Soc., 45, 79, 10.1145/2874239.2874251 · doi:10.1145/2874239.2874251
  36. Lissitz, 1975, Effect of the number of scale points on reliability: a Monte Carlo approach., J. Appl. Psychol., 60, 10, 10.1037/h0076268 · DOI · Paper page
  37. Manfreda, 2002, Design of web survey questionnaires: three basic experiments, J. Comput.-Mediated Commun., 7, 1
  38. Marshall, 2006, Informed consent in international health research, J. Empir. Res. Hum. Res. Ethics, 1, 25, 10.1525/jer.2006.1.1.25 · DOI
  39. Martinez, 2014, Don’t classify ratings of affect; rank them!, IEEE Trans. Affect. Comput., 5, 314, 10.1109/TAFFC.2014.2352268 · DOI
  40. McHugh, 2012, Interrater reliability: the Kappa statistic, Biochem. Med., 22, 276, 10.11613/BM.2012.031 · DOI · Paper page
  41. McShane, 2019, Abandon statistical significance, Am. Stat., 73, 235, 10.1080/00031305.2018.1527253 · DOI · Paper page
  42. Mellish, 1998, Evaluation in the context of natural language generation, Comput. Speech Lang., 12, 349, 10.1006/csla.1998.0106 · DOI
  43. Meteyard, 2020, Best practice guidance for linear mixed-effects models in psychological science, J. Mem. Lang., 112, 104092, 10.1016/j.jml.2020.104092 · DOI · Paper page
  44. Munafò, 2018, Robust research needs many lines of evidence, Nature, 553, 399, 10.1038/d41586-018-01023-3 · DOI · Paper page
  45. Murphy, 2019, HARKing: how badly can cherry-picking and question trolling produce bias in published results?, J. Bus. Psychol., 34, 1, 10.1007/s10869-017-9524-7 · DOI
  46. Nespor, 2000, Anonymity and place in qualitative inquiry, Qualit. Inq., 6, 546, 10.1177/107780040000600408 · DOI · Paper page
  47. Normand, 2016, Less is more: psychologists can learn more by studying fewer people, Front. Psychol., 7, 934, 10.3389/fpsyg.2016.00934 · DOI
  48. Paasche-Orlow, 2003, Readability standards for informed-consent forms as compared with actual readability, New Engl. J. mMedicine, 348, 721, 10.1056/NEJMsa021212 · DOI · Paper page
  49. Peytchev, 2006, Web survey design: paging versus scrolling, Int. J. Public Opin. Q., 70, 596, 10.1093/poq/nfl028 · DOI
  50. Popel, 2009, Improving english-czech tectogrammatical MT, Prague Bull. Math. Linguist., 92, 115, 10.2478/v10108-009-0025-3 · doi:10.2478/v10108-009-0025-3
  51. Portet, 2009, Automatic generation of textual summaries from neonatal intensive care data, Artif. Intell., 173, 789, 10.1016/j.artint.2008.12.002 · DOI · Paper page
  52. Postma, 2016, Addressing the MFS bias in WSD systems, 1695, 10.63317/2qcfqqj3yifp · DOI
  53. Potter, 1999, Rethinking validity and reliability in content analysis, J. Appl. Commun. Res., 27, 258, 10.1080/00909889909365539 · DOI · Paper page
  54. Preston, 2000, Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences, Acta Psychol., 104, 1, 10.1016/S0001-6918(99)00050-5 · DOI · Paper page
  55. Radziwill, 2017, Evaluating quality of chatbots and intelligent conversational agents, CoRR
  56. Ramos-Soto, 2015, Linguistic descriptions for automatic generation of textual short-Term weather forecasts on real prediction data, IEEE Trans. Fuzzy Syst., 23, 44, 10.1109/TFUZZ.2014.2328011 · DOI
  57. Reiter, 2009, An investigation into the validity of some metrics for automatically evaluating natural language generation systems, Comput. Linguist., 35, 529, 10.1162/coli.2009.35.4.35405 · DOI · Paper page
  58. Reiter, 2005, Choosing words in computer-generated weather forecasts, Artif. Intell., 167, 137, 10.1016/j.artint.2005.06.006 · DOI · Paper page
  59. Rice, 2008, How to do human-subjects research if you do not have an institutional review board, Respir. Care, 53, 1362, 10.4187/respcare.08531362 · doi:10.4187/respcare.08531362
  60. Rodgers, 2017, The genealogy of an image, or, what does literature (not) have to do with the history of computing?Tracing the sources and reception of gUlliver’s “ knowledge engine”, Humanities, 6, 85, 10.3390/h6040085 · DOI
  61. Ruggles, 2003, IPUMS redesign, Histor. Methods: J. Quant. Interdiscip. Hist., 36, 9, 10.1080/01615440309601210 · doi:10.1080/01615440309601210
  62. Sachs, 2003, Conducting empirical research on informed consent: challenges and questions, IRB: Ethics Hum. Res., 25, 4, 10.2307/3564116 · DOI
  63. Silberman, 2018, Responsible research with crowds: pay crowdworkers at least minimum wage, Commun. ACM, 61, 39, 10.1145/3180492 · DOI · Paper page
  64. Sterling, 2018, Investigating the complexity of consent forms in ESL research, J. Res. Desi. Stat. Linguist. Commun. Sci., 4, 156
  65. Stieger, 2007, Forced-response in online surveys: Bias from reactance and an increase in sex-specific dropout, J. Am. Soc. Inf. Sci. Technol., 58, 1653, 10.1002/asi.20651 · DOI
  66. Stunkel, 2010, Comprehension and informed consent: assessing the effect of a short consent form, IRB: Ethics Hum. Res., 32, 1
  67. Sulem, E., Abend, O., Rappoport, A., BLEU is not suitable for the evaluation of text simplification. http://arxiv.org/abs/1810.05995arXiv preprint arXiv:1810.05995 Accepted for publication as a short paper at EMNLP 2018. 10.18653/v1/D18-1081 · DOI
  68. Tukey, 1980, We need both exploratory and confirmatory, Am. Stat., 34, 23, 10.1080/00031305.1980.10482706 · DOI · Paper page
  69. Van Enschot, 2017, Taming our wild data: on intercoder reliability in discourse research, Unpublished Manuscript
  70. Van Heuven, 2014, Subtlex-UK: a new and improved word frequency database for british english, Q. J. Exp. Psychol., 67, 1176, 10.1080/17470218.2013.850521 · DOI · Paper page
  71. Vicente, 2010, Using questionnaire design to fight nonresponse bias in web surveys, Soc. Sci. Comput. Rev., 28, 251, 10.1177/0894439309340751 · doi:10.1177/0894439309340751
  72. Wagenmakers, 2011, Why psychologists must change the way they analyze their data: the case of PSI: comment on bEM (2011)., J. Pers. Soc. Psychol., 100, 426, 10.1037/a0022790 · DOI · Paper page
  73. Yannakakis, 2015, Ratings are overrated!, Front. ICT, 2, 13, 10.3389/fict.2015.00013 · DOI
  74. Yu, 2017, The effectiveness of a monetary incentive offer on survey response rates and response completeness in a longitudinal study, BMC Med. Res. Methodol., 17, 77, 10.1186/s12874-017-0353-1 · DOI

Cited by

See all 170

Metadata

Authors
van der Lee, C., Gatt, A., van Miltenburg, E., & Krahmer, E.
Venue
Computer Speech & Language
Publisher
Elsevier BV
Published
2021-05
Volume
67
Pages
101151
ISSN
0885-2308 (ISSN-L)
OpenAlex
W3107855336
Cited by
170
References
38
FWCI
7.20
Type
journal article
Language
en
Open access
hybrid
Licence
Funders
  • Nationaal Regieorgaan Praktijkgericht Onderzoek SIA (2014-01-51PRO)
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (360-89-050)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Human evaluation of automatically generated text: Current trends and best practice guidelines. https://app.cochat.ai/papers/10.1016/j.csl.2020.101151
The paper
van der Lee, C., Gatt, A., van Miltenburg, E., & Krahmer, E. (2021). Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech & Language. https://doi.org/10.1016/j.csl.2020.101151

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.