CoChat
Sign Up
Cited by 123Natural Language Processing TechniquesTopic ModelingSpeech and dialogue systems

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

Open access
Gehrmann, Sebastian, Clark, Elizabeth, Sellam, Thibault·2023·Journal of Artificial Intelligence Research
DOI 10.1613/jair.1.13715
Open this paper in CoChat — free accountOpen PDF

Abstract

Evaluation practices in natural language generation (NLG) have many known flaws, but improved evaluation approaches are rarely widely adopted. This issue has become more urgent, since neural generation models have improved to the point where their outputs can often no longer be distinguished based on the surface-level features that older metrics rely on. This paper surveys the issues with human and automatic model evaluations and with commonly used datasets in NLG that have been pointed out over the past 20 years. We summarize, categorize, and discuss how researchers have been addressing these issues and what their findings mean for the current state of model evaluations. Building on those insights, we lay out a long-term vision for evaluation research and propose concrete steps for researchers to improve their evaluation processes. Finally, we analyze 66 generation papers from recent NLP conferences in how well they already follow these suggestions and identify which areas require more drastic changes to the status quo.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1613/jair.1.13715

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Static benchmarks with clear ground-truth do not capture flexible and interactive tasks needed to navigate scenarios in the real-world.

    “Static benchmarks with clear ground-truth MMLU [29], GSM8k [11], and others like it do not capture flexible and interactive tasks needed to navigate scenarios in the real-world”

    From Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents (2024) · cites this paper as [22]

    In the “Model Interpretability” section (page unavailable), the paper says that evaluating interactive systems with static data can mischaracterize their capabilities and limitations. Section 3.3 also cautions that benchmark results are specific to the scenarios evaluated. The copy does not establish this specifically for MMLU or GSM8k, or discuss their clear ground-truth answers. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • BLEU, ROUGE, and ROUGE-L measure lexical overlap with a reference, which is useful for tracking surface similarity but can underestimate valid paraphrases and overestimate fluent yet ungrounded responses.

    “BLEU, ROUGE, and ROUGE-L measure lexical overlap with a reference, which is useful for tracking surface similarity but can underestimate valid paraphrases and overestimate fluent yet ungrounded responses (Gehrmann et al., 2023; Ai et al., 2025).”

    From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Gehrmann et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 235

  1. Akter, M., Bansal, N., & Karmaker, S. K. (2022). Revisiting automatic evaluation of extractive summarization task: Can we do better than ROUGE?. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 1547-1560, Dublin, Ireland. Association for Computational Linguistics.
  2. Ambati, B. R., Reddy, S., & Steedman, M. (2016). Assessing relative sentence complexity using an incremental CCG parser. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1051-1057, San Diego, California. Association for Computational Linguistics.
  3. Amidei, J., Piwek, P., & Willis, A. (2019a). Agreement is overrated: A plea for correlation to assess human evaluation reliability. In Proceedings of the 12th International Con- ference on Natural Language Generation, pp. 344-354, Tokyo, Japan. Association for Computational Linguistics.
  4. Amidei, J., Piwek, P., & Willis, A. (2019b). The use of rating and Likert scales in natural language generation human evaluation tasks: A review and some recommendations. In Proceedings of the 12th International Conference on Natural Language Generation, pp. 397-402, Tokyo, Japan. Association for Computational Linguistics.
  5. Amplayo, R. K., Angelidis, S., & Lapata, M. (2021). Aspect-controllable opinion summa- rization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6578-6593, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  6. Amplayo, R. K., Liu, P. J., Zhao, Y., & Narayan, S. (2022). SMART: sentences as basic units for text evaluation. CoRR, abs/2208.01030. · arXiv
  7. Aralikatte, R., Narayan, S., Maynez, J., Rothe, S., & McDonald, R. (2021). Focus atten- tion: Promoting faithfulness and diversity in summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa- pers), pp. 6078-6095, Online. Association for Computational Linguistics.
  8. Artstein, R., & Poesio, M. (2009). Bias decreases in proportion to the number of annotators. In Proceedings of FG-MoL 2005: The 10th conference on Formal Grammar and The 9th Meeting on, Vol. 139.
  9. Bangalore, S., Rambow, O., & Whittaker, S. (2000). Evaluation metrics for generation. In INLG'2000 Proceedings of the First International Conference on Natural Language Generation, pp. 1-8, Mitzpe Ramon, Israel. Association for Computational Linguis- tics.
  10. Belinkov, Y., & Glass, J. (2019). Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7, 49-72.
  11. Belz, A., & Gatt, A. (2008). Intrinsic vs. extrinsic evaluation measures for referring ex- pression generation. In Proceedings of ACL-08: HLT, Short Papers, pp. 197-200, Columbus, Ohio. Association for Computational Linguistics.
  12. Belz, A., & Kow, E. (2010). Comparing rating scales and preference judgements in language evaluation. In Proceedings of the 6th International Natural Language Generation Con- ference. Association for Computational Linguistics.
  13. Belz, A., & Kow, E. (2011). Discrete vs. continuous rating scales for language evaluation in NLP. In Proceedings of the 49th Annual Meeting of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 230-235, Portland, Oregon, USA. Association for Computational Linguistics.
  14. Belz, A., Shimorina, A., Agarwal, S., & Reiter, E. (2021). The ReproGen shared task on reproducibility of human evaluations in NLG: Overview and results. In Proceedings of the 14th International Conference on Natural Language Generation, pp. 249-258, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  15. Bender, E. M. (2009). Linguistically naïve != language independent: Why NLP needs linguistic typology. In Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious or Vacuous?, pp. 26-32, Athens, Greece. Association for Computational Linguistics.
  16. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big?. In Elish, M. C., Isaac, W., & Zemel, R. S. (Eds.), FAccT '21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021, pp. 610-623. ACM.
  17. Bhandari, M., Gour, P. N., Ashfaq, A., & Liu, P. (2020a). Metrics also disagree in the low scoring range: Revisiting summarization evaluation metrics. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 5702-5711, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  18. Bhatt, S., Dev, S., Talukdar, P., Dave, S., & Prabhakaran, V. (2022). Re-contextualizing fairness in NLP: The case of India. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Pa- pers), pp. 727-740, Online only. Association for Computational Linguistics.
  19. Bojar, O., Graham, Y., & Kamran, A. (2017). Results of the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, pp. 489-513, Copenhagen, Denmark. Association for Computational Linguistics.
  20. Bojar, O., Graham, Y., Kamran, A., & Stanojević, M. (2016). Results of the WMT16 metrics shared task. In Proceedings of the First Conference on Machine Transla- tion: Volume 2, Shared Task Papers, pp. 199-231, Berlin, Germany. Association for Computational Linguistics.
  21. Bommasani, R., & Cardie, C. (2020). Intrinsic evaluation of summarization datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 8075-8096, Online. Association for Computational Linguistics.
  22. Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015). A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, Lisbon, Portugal. Association for Computational Linguistics.
  23. Bowman, S. R., & Dahl, G. (2021). What will it take to fix benchmarking in natural language understanding?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4843-4855, Online. Association for Computational Linguistics.
  24. Braun, S., Vasilyev, O., Iskender, N., & Bohannon, J. (2022). Does summary evaluation survive translation to other languages?. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pp. 2425-2435, Seattle, United States. Association for Computational Linguistics.
  25. Briakou, E., Agrawal, S., Tetreault, J. R., & Carpuat, M. (2021). Evaluating the evaluation metrics for style transfer: A case study in multilingual formality transfer. In Moens, M., Huang, X., Specia, L., & Yih, S. W. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 1321-1336. Association for Computational Linguistics.
  26. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., ..., & Amodei, D. (2020). Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., & Lin, H. (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  27. Budzianowski, P., Wen, T., Tseng, B., Casanueva, I., Ultes, S., Ramadan, O., & Gasic, M. (2018). MultiWOZ -A large-scale multi-domain Wizard-of-Oz dataset for task- oriented dialogue modelling. In Riloff, E., Chiang, D., Hockenmaier, J., & Tsujii, J. (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium, October 31 -November 4, 2018, pp. 5016-5026. Association for Computational Linguistics.
  28. Cai, W., Encarnacion, R., Chern, B., Corbett-Davies, S., Bogen, M., Bergman, S., & Goel, S. (2022). Adaptive sampling strategies to construct equitable training datasets. In FAccT '22: 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21 -24, 2022, pp. 1467-1478. ACM.
  29. Callison-Burch, C., Osborne, M., & Koehn, P. (2006). Re-evaluating the role of Bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics, pp. 249-256, Trento, Italy. Association for Computational Linguistics.
  30. Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020). With little power comes great responsibility. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9263-9274, Online. Association for Computational Linguistics.
  31. Celikyilmaz, A., Clark, E., & Gao, J. (2020). Evaluation of text generation: A survey. CoRR, abs/2006.14799. · arXiv
  32. Chen, H., Brahman, F., Ren, X., Ji, Y., Choi, Y., & Swayamdipta, S. (2022). REV: information-theoretic evaluation of free-text rationales. CoRR, abs/2210.04982. · arXiv
  33. Chen, P., Wu, F., Wang, T., & Ding, W. (2018). A semantic QA-based approach for text summarization evaluation. In McIlraith, S. A., & Weinberger, K. Q. (Eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI- 18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 4800-4807. AAAI Press.
  34. Chen, Y., Liu, P., & Qiu, X. (2021). Are factuality checkers reliable? adversarial meta- evaluation of factuality in summarization. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, pp. 2082-2095, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  35. Choshen, L., Fox, L., Aizenbud, Z., & Abend, O. (2020). On the weaknesses of reinforcement learning for neural machine translation. In International Conference on Learning Representations.
  36. Clark, E., Celikyilmaz, A., & Smith, N. A. (2019). Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2748-2760, Florence, Italy. Association for Computational Linguistics.
  37. Colombo, P., Staerman, G., Clavel, C., & Piantanida, P. (2021). Automatic text evaluation through the lens of Wasserstein barycenters. In Moens, M., Huang, X., Specia, L., & Yih, S. W. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pp. 10450-10466. Association for Computational Linguistics.
  38. Colombo, P. J. A., Clavel, C., & Piantanida, P. (2022). InfoLM: A new metric to evaluate summarization & data2text generation. In Thirty-Sixth AAAI Conference on Artifi- cial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 -March 1, 2022, pp. 10554-10562. AAAI Press.
  39. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online. Association for Computational Linguistics.
  40. Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S., Schwenk, H., & Stoyanov, V. (2018). XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
  41. Dale, R., & Mellish, C. (1998). Towards evaluation in natural language generation. In Pro- ceedings of the First International Conference on Language Resources and Evaluation, LREC 1998, May 28-30, 1998, Granada, Spain, pp. 555-562. European Language Re- sources Association.
  42. Dang, H. T. (2006). DUC 2005: Evaluation of question-focused summarization systems. In Proceedings of the Workshop on Task-Focused Summarization and Question Answer- ing, pp. 48-55, Sydney, Australia. Association for Computational Linguistics.
  43. Demsar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal Machine Learning Research, 7, 1-30.
  44. Deng, M., Tan, B., Liu, Z., Xing, E. P., & Hu, Z. (2021). Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Moens, M., Huang, X., Specia, L., & Yih, S. W. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 7580-7605. Association for Computational Linguistics.
  45. Deutsch, D., Bedrax-Weiss, T., & Roth, D. (2021a). Towards question-answering as an automatic metric for evaluating the content quality of a summary. Trans. Assoc. Comput. Linguistics, 9, 774-789.
  46. Deutsch, D., Dror, R., & Roth, D. (2021b). A Statistical Analysis of Summarization Evalu- ation Metrics Using Resampling Methods. Transactions of the Association for Com- putational Linguistics, 9, 1132-1146.
  47. Deutsch, D., Dror, R., & Roth, D. (2022). Re-examining system-level correlations of au- tomatic summarization evaluation metrics. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pp. 6038-6052, Seattle, United States. Association for Computational Linguistics.
  48. Deutsch, D., & Roth, D. (2021). Understanding the extent to which content quality metrics measure the information quality of summaries. In Proceedings of the 25th Conference on Computational Natural Language Learning, pp. 300-309, Online. Association for Computational Linguistics.
  49. DeYoung, J., Beltagy, I., van Zuylen, M., Kuehl, B., & Wang, L. L. (2021). MSˆ2: Multi- document summarization of medical studies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7494-7513, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  50. Díaz, M., Kivlichan, I. D., Rosen, R., Baker, D. K., Amironesei, R., Prabhakaran, V., & Denton, E. (2022). CrowdWorkSheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation. In FAccT '22: 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21 -24, 2022, pp. 2342-2351. ACM.
  51. Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S. L., Hovy, D., Boureau, Y., & Rieser, V. (2021). Anticipating safety issues in E2E conversational AI: framework and tooling. CoRR, abs/2107.03451. · arXiv
  52. Djolonga, J., Lucic, M., Cuturi, M., Bachem, O., Bousquet, O., & Gelly, S. (2020). Precision- recall curves using information divergence frontiers. In Chiappa, S., & Calandra, R. (Eds.), The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], Vol. 108 of Pro- ceedings of Machine Learning Research, pp. 2550-2559. PMLR.
  53. Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019). Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2185-2194, Hong Kong, China. Association for Computational Linguistics.
  54. Dorr, B., Monz, C., President, S., Schwartz, R., & Zajic, D. (2005). A methodology for extrinsic evaluation of text summarization: Does ROUGE correlate?. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 1-8, Ann Arbor, Michigan. Association for Computational Linguistics.
  55. Dras, M. (2015). Squibs: Evaluating human pairwise preference judgments. Computational Linguistics, 41 (2), 309-317.
  56. Dušek, O., Howcroft, D. M., & Rieser, V. (2019). Semantic noise matters for neural natural language generation. In Proceedings of the 12th International Conference on Natu- ral Language Generation, pp. 421-426, Tokyo, Japan. Association for Computational Linguistics.
  57. Dušek, O., Novikova, J., & Rieser, V. (2018). Findings of the E2E NLG challenge. In Proceedings of the 11th International Conference on Natural Language Generation, pp. 322-328, Tilburg University, The Netherlands. Association for Computational Linguistics.
  58. Ethayarajh, K., & Jurafsky, D. (2020). Utility is in the eye of the user: A critique of NLP leaderboards. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4846-4853, Online. Association for Computational Linguistics.
  59. Eyal, M., Baumel, T., & Elhadad, M. (2019). Question answering as an automatic evalu- ation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3938-3948, Minneapolis, Minnesota. Association for Computational Linguistics.
  60. Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021). SummEval: Re-evaluating Summarization Evaluation. Transactions of the Association for Computational Linguistics, 9, 391-409.
  61. Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I., & Gurevych, I. (2019). Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 2214-2220, Florence, Italy. Association for Computational Linguistics.
  62. Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon Mechanical Turk: Gold Mine or Coal Mine?. Computational Linguistics, 37 (2), 413-420.
  63. Freitag, M., Grangier, D., & Caswell, I. (2020). BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 61-71, Online. Association for Computational Linguistics.
  64. Freitag, M., Rei, R., Mathur, N., Lo, C.-k., Stewart, C., Foster, G., Lavie, A., & Bojar, O. (2021). Results of the WMT21 metrics shared task: Evaluating metrics with expert- based human evaluations on TED and news domain. In Proceedings of the Sixth Con- ference on Machine Translation, pp. 733-774, Online. Association for Computational Linguistics.
  65. Gao, M., & Wan, X. (2022). DialSummEval: Revisiting summarization evaluation for di- alogues. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5693-5709, Seattle, United States. Association for Computational Linguistics.
  66. Gatt, A., Belz, A., & Kow, E. (2008). The TUNA challenge 2008: Overview and evalua- tion results. In Proceedings of the Fifth International Natural Language Generation Conference, pp. 198-206, Salt Fork, Ohio, USA. Association for Computational Lin- guistics.
  67. Gatt, A., & Krahmer, E. (2018). Survey of the state of the art in natural language gen- eration: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research, 61, 65-170.
  68. Gehrmann, S. (2020). Human-AI Collaboration for Natural Language Generation with In- terpretable Neural Networks. Ph.D. thesis, Harvard University.
  69. Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., ..., & Zhou, J. (2021). The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pp. 96-120, Online. Association for Computational Linguistics.
  70. Gehrmann, S., Deng, Y., & Rush, A. (2018). Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
  71. Gehrmann, S., Layne, S., & Dernoncourt, F. (2019). Improving human text comprehension through semi-Markov CRF-based neural section title generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1677-1688, Minneapolis, Minnesota. Association for Computational Linguistics.
  72. Gehrmann, S., Ruder, S., Nikolaev, V., Botha, J. A., Chavinda, M., Parikh, A. P., & Rivera, C. (2022). TaTa: A multilingual table-to-text dataset for African languages. CoRR, abs/2211.00142. · arXiv
  73. Ghazarian, S., Hedayatnia, B., Papangelis, A., Liu, Y., & Hakkani-Tur, D. (2022a). What is wrong with you?: Leveraging user sentiment for automatic dialog evaluation. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 4194-4204, Dublin, Ireland. Association for Computational Linguistics.
  74. Giannakopoulos, G., Kubina, J., Conroy, J., Steinberger, J., Favre, B., Kabadjov, M., Kr- uschwitz, U., & Poesio, M. (2015). MultiLing 2015: Multilingual summarization of single and multi-documents, on-line fora, and call-center conversations. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 270-274, Prague, Czech Republic. Association for Computational Linguistics.
  75. Gillick, D., & Liu, Y. (2010). Non-expert evaluation of summarization systems is risky. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pp. 148-151, Los Angeles. Association for Com- putational Linguistics.
  76. Gkatzia, D., & Mahamood, S. (2015). A snapshot of NLG evaluation practices 2005 - 2014. In Proceedings of the 15th European Workshop on Natural Language Generation (ENLG), pp. 57-60, Brighton, UK. Association for Computational Linguistics.
  77. Goel, K., Rajani, N. F., Vig, J., Taschdjian, Z., Bansal, M., & Ré, C. (2021). Robustness gym: Unifying the NLP evaluation landscape. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, pp. 42-55, Online. Association for Computational Linguistics.
  78. Goyal, T., Li, J. J., & Durrett, G. (2022). News summarization and evaluation in the era of GPT-3. CoRR, abs/2209.12356. · arXiv
  79. Graham, Y. (2015). Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 128-137, Lisbon, Portugal. Association for Computational Linguistics.
  80. Grusky, M., Naaman, M., & Artzi, Y. (2018). Newsroom: A dataset of 1.3 million sum- maries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Papers), pp. 708-719, New Orleans, Louisiana. Association for Computational Linguistics.
  81. Harman, D., & Over, P. (2004). The effects of human variation in DUC summarization evaluation. In Text Summarization Branches Out, pp. 10-17, Barcelona, Spain. As- sociation for Computational Linguistics.
  82. Harris, C., Halevy, M., Howard, A. M., Bruckman, A., & Yang, D. (2022). Exploring the role of grammar and word choice in bias toward African American English (AAE) in hate speech classification. In FAccT '22: 2022 ACM Conference on Fairness, Account- ability, and Transparency, Seoul, Republic of Korea, June 21 -24, 2022, pp. 789-798. ACM.
  83. Hasan, T., Bhattacharjee, A., Islam, M. S., Mubasshir, K., Li, Y.-F., Kang, Y.-B., Rah- man, M. S., & Shahriyar, R. (2021). XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4693-4703, Online. Association for Computa- tional Linguistics.
  84. Hermann, K. M., Kociský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015). Teaching machines to read and comprehend. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., & Garnett, R. (Eds.), Advances in Neu- ral Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1693- 1701.
  85. Higashinaka, R., Mizukami, M., Funakoshi, K., Araki, M., Tsukahara, H., & Kobayashi, Y. (2015). Fatal or not? finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2243-2248, Lisbon, Portugal. Association for Com- putational Linguistics.
  86. Hirschman, L. (1998). The evolution of evaluation: Lessons from the message understanding conferences. Comput. Speech Lang., 12 (4), 281-305.
  87. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems.
  88. Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022). TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the Second DialDoc Workshop on Document- grounded Dialogue and Conversational Question Answering, pp. 161-175, Dublin, Ire- land. Association for Computational Linguistics.
  89. Hovy, E., Lin, C.-Y., & Zhou, L. (2005). Evaluating duc 2005 using basic elements. In Proceedings of DUC, Vol. 2005. Citeseer.
  90. Howcroft, D. M., Belz, A., Clinciu, M.-A., Gkatzia, D., Hasan, S. A., Mahamood, S., Mille, S., van Miltenburg, E., Santhanam, S., & Rieser, V. (2020). Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, pp. 169-182, Dublin, Ireland. Association for Computational Linguistics.
  91. Howcroft, D. M., & Rieser, V. (2021). What happens if you treat ordinal ratings as interval data? human evaluations in NLP are even more under-powered than you think. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 8932-8939, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  92. Huynh, J., Bigham, J., & Eskénazi, M. (2021). A survey of NLP-related crowdsourcing hits: what works and what does not. CoRR, abs/2111.05241. · arXiv
  93. Iskender, N., Polzehl, T., & Möller, S. (2020). Best practices for crowd-based evaluation of German summarization: Comparing crowd, expert and automatic evaluation. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pp. 164-175, Online. Association for Computational Linguistics.
  94. Jiang, Y., Liu, T., Ma, S., Zhang, D., Yang, J., Huang, H., Sennrich, R., Cotterell, R., Sachan, M., & Zhou, M. (2022). BlonDe: An automatic evaluation metric for document-level machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1550-1565, Seattle, United States. Association for Com- putational Linguistics.
  95. Jones, K. S., & Galliers, J. R. (1995). Evaluating natural language processing systems: An analysis and review, Vol. 1083. Springer Science & Business Media.
  96. Jørgensen, A., Hovy, D., & Søgaard, A. (2015). Challenges of studying and processing dialects in social media. In Proceedings of the Workshop on Noisy User-generated Text, pp. 9-18, Beijing, China. Association for Computational Linguistics.
  97. Jørgensen, A., Hovy, D., & Søgaard, A. (2016). Learning a POS tagger for AAVE-like language. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1115-1120, San Diego, California. Association for Computational Linguistics.
  98. Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020). The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282-6293, Online. Association for Computational Linguistics.
  99. Kamal Eddine, M., Shang, G., Tixier, A., & Vazirgiannis, M. (2022). FrugalScore: Learn- ing cheaper, lighter and faster evaluation metrics for automatic text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 1305-1318, Dublin, Ireland. Association for Computational Linguistics.
  100. Kane, H., Kocyigit, M. Y., Abdalla, A., Ajanoh, P., & Coulibali, M. (2020). NUBIA: NeUral based interchangeability assessor for text generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pp. 28-37, Online (Dublin, Ireland). Association for Computational Linguistics.
  101. Kasai, J., Sakaguchi, K., Le Bras, R., Dunagan, L., Morrison, J., Fabbri, A., Choi, Y., & Smith, N. A. (2022). Bidimensional leaderboards: Generate and evaluate language hand in hand. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3540-3557, Seattle, United States. Association for Computational Linguistics.
  102. Kaster, M., Zhao, W., & Eger, S. (2021). Global explainability of bert-based evaluation metrics by disentangling along linguistic factors. In Moens, M., Huang, X., Specia, L., & Yih, S. W. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pp. 8912-8925. Association for Computational Linguistics.
  103. Ke, P., Zhou, H., Lin, Y., Li, P., Zhou, J., Zhu, X., & Huang, M. (2022). CTRLEval: An unsupervised reference-free metric for evaluating controlled text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 2306-2319, Dublin, Ireland. Association for Computational Linguistics.
  104. Kedzie, C., McKeown, K., & Daumé III, H. (2018). Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1818-1828, Brussels, Belgium. Association for Computational Linguistics.
  105. Khalid, B., & Lee, S. (2022). Explaining dialogue evaluation metrics using adversarial behav- ioral analysis. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5871-5883, Seattle, United States. Association for Computational Linguistics.
  106. Khashabi, D., Stanovsky, G., Bragg, J., Lourie, N., Kasai, J., Choi, Y., Smith, N. A., & Weld, D. S. (2021). GENIE: A leaderboard for human-in-the-loop evaluation of text generation. CoRR, abs/2101.06561. · arXiv
  107. Kim, J., Jeong, M., Choi, S., & Hwang, S.-w. (2021). Structure-augmented keyphrase generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2657-2667, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  108. Kiritchenko, S., & Mohammad, S. (2017). Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 465-470, Vancouver, Canada. Association for Computational Linguistics.
  109. Koch, B., Denton, E., Hanna, A., & Foster, J. G. (2021). Reduced, reused and recycled: The life of a dataset in machine learning research. CoRR, abs/2112.01716. · arXiv
  110. Kocmi, T., Federmann, C., Grundkiewicz, R., Junczys-Dowmunt, M., Matsushita, H., & Menezes, A. (2021). To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pp. 478-494, Online. Association for Computational Linguistics.
  111. Koehn, P., Guzmán, F., Chaudhary, V., & Pino, J. (2019). Findings of the WMT 2019 shared task on parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pp. 54-72, Florence, Italy. Association for Computational Linguistics.
  112. Koehn, P., & Monz, C. (2006). Manual and automatic evaluation of machine translation between European languages. In Proceedings on the Workshop on Statistical Machine Translation, pp. 102-121, New York City. Association for Computational Linguistics.
  113. Krishna, K., Khosla, S., Bigham, J., & Lipton, Z. C. (2021). Generating SOAP notes from doctor-patient conversations using modular summarization techniques. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4958-4972, Online. Association for Computational Linguistics.
  114. Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019). Neural text sum- marization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 540-551, Hong Kong, China. Association for Computational Linguistics.
  115. Kusner, M. J., Sun, Y., Kolkin, N. I., & Weinberger, K. Q. (2015). From word embeddings to document distances. In Bach, F. R., & Blei, D. M. (Eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6- 11 July 2015, Vol. 37 of JMLR Workshop and Conference Proceedings, pp. 957-966. JMLR.org.
  116. Ladhak, F., Durmus, E., Cardie, C., & McKeown, K. (2020). WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4034-4048, Online. Association for Computational Linguistics.
  117. Langkilde, I., & Knight, K. (1998). Generation that exploits corpus-based statistical knowl- edge. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pp. 704-710, Montreal, Quebec, Canada. Association for Computational Linguistics.
  118. Lebret, R., Grangier, D., & Auli, M. (2016). Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1203-1213, Austin, Texas. Association for Computational Linguistics.
  119. Li, H., Gao, T., Goenka, M., & Chen, D. (2022). Ditch the gold standard: Re-evaluating conversational question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8074-8085, Dublin, Ireland. Association for Computational Linguistics.
  120. Liao, T., Taori, R., Raji, I. D., & Schmidt, L. (2021). Are we learning yet? a meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  121. Liebling, D., Heller, K., Robertson, S., & Deng, W. (2022). Opportunities for human- centered evaluation of machine translation systems. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 229-240, Seattle, United States. Association for Computational Linguistics.
  122. Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. In Text Sum- marization Branches Out, pp. 74-81, Barcelona, Spain. Association for Computational Linguistics.
  123. Lin, C.-Y., & Och, F. J. (2004). ORANGE: a method for evaluating automatic evaluation metrics for machine translation. In COLING 2004: Proceedings of the 20th Interna- tional Conference on Computational Linguistics, pp. 501-507, Geneva, Switzerland.
  124. Liu, Y., & Lapata, M. (2019). Hierarchical transformers for multi-document summariza- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5070-5081, Florence, Italy. Association for Computational Linguistics.
  125. Lommel, A., Uszkoreit, H., & Burchardt, A. (2014). Multidimensional quality met- rics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica, 1 (12), 0455-463.
  126. Luccioni, A., & Viviano, J. (2021). What's in the box? an analysis of undesirable con- tent in the Common Crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pp. 182-189, Online. Association for Computational Linguistics.
  127. Lyu, X., Li, J., Gong, Z., & Zhang, M. (2021). Encouraging lexical translation consistency for document-level neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3265-3277, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  128. Ma, Q., Bojar, O., & Graham, Y. (2018). Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pp. 671-688, Belgium, Brussels. Association for Computational Linguistics.
  129. Ma, Q., Wei, J., Bojar, O., & Graham, Y. (2019). Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp. 62-90, Florence, Italy. Association for Computational Linguistics.
  130. Macháček, M., & Bojar, O. (2014). Results of the WMT14 metrics shared task. In Pro- ceedings of the Ninth Workshop on Statistical Machine Translation, pp. 293-301, Bal- timore, Maryland, USA. Association for Computational Linguistics.
  131. Mani, I., House, D., Klein, G., Hirschman, L., Firmin, T., & Sundheim, B. (1999). The TIPSTER SUMMAC text summarization evaluation. In Ninth Conference of the Eu- ropean Chapter of the Association for Computational Linguistics, pp. 77-85, Bergen, Norway. Association for Computational Linguistics.
  132. Mathur, N., Wei, J., Freitag, M., Ma, Q., & Bojar, O. (2020). Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pp. 688-725, Online. Association for Computational Linguistics.
  133. McCoy, T., Pavlick, E., & Linzen, T. (2019). Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3428-3448, Florence, Italy. Association for Computational Linguistics.
  134. McMillan-Major, A., Osei, S., Rodriguez, J. D., Ammanamanchi, P. S., Gehrmann, S., & Jernite, Y. (2021). Reusable templates and guides for documenting datasets and mod- els for natural language processing and generation: A case study of the HuggingFace and GEM data and model cards. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pp. 121-135, Online. Association for Computational Linguistics.
  135. Mille, S., Dhole, K., Mahamood, S., Perez-Beltrachini, L., Gangal, V., Kale, M., van Mil- tenburg, E., & Gehrmann, S. (2021). Automatic construction of evaluation suites for natural language generation datasets. In Thirty-fifth Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track (Round 1).
  136. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. In danah boyd, & Morgenstern, J. H. (Eds.), Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019, Atlanta, GA, USA, January 29-31, 2019, pp. 220-229. ACM.
  137. Mohankumar, A. K., & Khapra, M. (2022). Active evaluation: Efficient NLG evaluation with few pairwise comparisons. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8761-8781, Dublin, Ireland. Association for Computational Linguistics.
  138. Moradi, M., & Samwald, M. (2021). Evaluating the robustness of neural language models to input perturbations. In Moens, M., Huang, X., Specia, L., & Yih, S. W. (Eds.), Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 1558-1570. Association for Computational Linguistics.
  139. Nakayama, T. K., & Halualani, R. T. (2011). The handbook of critical intercultural com- munication. John Wiley & Sons.
  140. Nallapati, R., Zhai, F., & Zhou, B. (2017). SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents. In Singh, S. P., & Markovitch, S. (Eds.), Proceedings of the Thirty-First AAAI Conference on Artifi- cial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 3075-3081. AAAI Press.
  141. Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016). Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 280-290, Berlin, Germany. Association for Computational Linguistics.
  142. Napoles, C., Gormley, M., & Van Durme, B. (2012). Annotated Gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX), pp. 95-100, Montréal, Canada. Association for Computational Linguistics.
  143. Narayan, S., Cohen, S. B., & Lapata, M. (2018). Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
  144. Narayan, S., Zhao, Y., Maynez, J., Simoes, G., Nikolaev, V., & McDonald, R. (2021). Planning with learned entity prompts for abstractive summarization. arXiv preprint arXiv:2104.07606. · arXiv
  145. Nenkova, A. (2005). Automatic text summarization of newswire: Lessons learned from the document understanding conference. In Veloso, M. M., & Kambhampati, S. (Eds.), Proceedings, The Twentieth National Conference on Artificial Intelligence and the Seventeenth Innovative Applications of Artificial Intelligence Conference, July 9-13, 2005, Pittsburgh, Pennsylvania, USA, pp. 1436-1441. AAAI Press / The MIT Press.
  146. Nenkova, A., & Passonneau, R. (2004). Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pp. 145-152, Boston, Massachusetts, USA. Association for Com- putational Linguistics.
  147. Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020). Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, Online. Association for Computational Linguistics.
  148. Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017a). Why we need new evalu- ation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2241-2252, Copenhagen, Denmark. Association for Computational Linguistics.
  149. Novikova, J., Dušek, O., & Rieser, V. (2017b). The E2E dataset: New challenges for end- to-end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 201-206, Saarbrücken, Germany. Association for Computational Linguistics.
  150. Novikova, J., Dušek, O., & Rieser, V. (2018). RankME: Reliable human ratings for natural language generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pp. 72-78, New Orleans, Louisiana. Association for Computational Linguistics.
  151. Nye, M. I., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., & Odena, A. (2021). Show your work: Scratchpads for intermediate computation with language models. CoRR, abs/2112.00114. · arXiv
  152. Over, P., & Yen, J. (2004). An introduction to DUC-2004. In Proceedings of the Document Understanding Conference.
  153. Owczarzak, K., Rankel, P. A., Dang, H. T., & Conroy, J. M. (2012). Assessing the effect of inconsistent assessors on summarization evaluation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 359-362, Jeju Island, Korea. Association for Computational Linguistics.
  154. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  155. Parikh, A., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., & Das, D. (2020). ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1173-1186, Online. Association for Computational Linguistics.
  156. Pasunuru, R., & Bansal, M. (2018). Multi-reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 646-653, New Orleans, Louisiana. Association for Com- putational Linguistics.
  157. Paulus, R., Xiong, C., & Socher, R. (2018). A deep reinforced model for abstractive sum- marization. In International Conference on Learning Representations.
  158. Perez-Beltrachini, L., & Lapata, M. (2021). Models and datasets for cross-lingual sum- marisation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9408-9423, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  159. Peyrard, M. (2019). Studying summarization evaluation metrics in the appropriate scoring range. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5093-5100, Florence, Italy. Association for Computational Linguistics.
  160. Phang, J., Chen, A., Huang, W., & Bowman, S. R. (2021). Adversarially constructed evaluation sets are more challenging, but may not be fair. CoRR, abs/2111.08181. · arXiv
  161. Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., & Harchaoui, Z. (2021). MAUVE: measuring the gap between neural text and human text using divergence frontiers. CoRR, abs/2102.01454. · arXiv
  162. Pitler, E., Louis, A., & Nenkova, A. (2010). Automatic evaluation of linguistic quality in multi-document summarization. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 544-554, Uppsala, Sweden. Association for Computational Linguistics.
  163. Popovic, M., & Ney, H. (2011). Towards automatic error analysis of machine translation output. Comput. Linguistics, 37 (4), 657-688.
  164. Post, M. (2018). A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium. Association for Computational Linguistics.
  165. Prabhakaran, V., Mostafazadeh Davani, A., & Diaz, M. (2021). On releasing annotator- level labels and information in datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pp. 133-138, Punta Cana, Dominican Republic. Association for Computa- tional Linguistics.
  166. Pratapa, A., Anastasopoulos, A., Rijhwani, S., Chaudhary, A., Mortensen, D. R., Neubig, G., & Tsvetkov, Y. (2021). Evaluating the morphosyntactic well-formedness of gener- ated texts. In Moens, M., Huang, X., Specia, L., & Yih, S. W. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 7131-7150. Association for Computational Linguistics.
  167. Pu, A., Chung, H. W., Parikh, A., Gehrmann, S., & Sellam, T. (2021). Learning compact metrics for MT. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pp. 751-762, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  168. Puduppully, R., Dong, L., & Lapata, M. (2019). Data-to-text generation with content selection and planning. In The Thirty-Third AAAI Conference on Artificial Intelli- gence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 -February 1, 2019, pp. 6908-6915. AAAI Press.
  169. Pushkarna, M., Zaldivar, A., & Kjartansson, O. (2022). Data cards: Purposeful and trans- parent dataset documentation for responsible AI. In FAccT '22: 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21 - 24, 2022, pp. 1776-1826. ACM.
  170. Quiñonero-Candela, J., Sugiyama, M., Lawrence, N. D., & Schwaighofer, A. (2009). Dataset shift in machine learning. Mit Press.
  171. Rankel, P. A., Conroy, J. M., Dang, H. T., & Nenkova, A. (2013). A decade of automatic content evaluation of news summaries: Reassessing the state of the art. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 131-136, Sofia, Bulgaria. Association for Computational Lin- guistics.
  172. Rankel, P. A., Conroy, J. M., & Schlesinger, J. D. (2012). Better metrics to automatically predict the quality of a text summary. Algorithms, 5 (4), 398-420.
  173. Rebuffel, C., Scialom, T., Soulier, L., Piwowarski, B., Lamprier, S., Staiano, J., Scoutheeten, G., & Gallinari, P. (2021). Data-QuestEval: A referenceless metric for data-to-text se- mantic evaluation. In Moens, M., Huang, X., Specia, L., & Yih, S. W. (Eds.), Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 8029-8036. Association for Computational Linguistics.
  174. Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020). COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2685-2702, Online. Association for Computational Linguistics.
  175. Reiter, E. (2018). A structured review of the validity of BLEU. Computational Linguistics, 44 (3), 393-401.
  176. Reiter, E., & Belz, A. (2009). An investigation into the validity of some metrics for auto- matically evaluating natural language generation systems. Computational Linguistics, 35 (4), 529-558.
  177. Reiter, E., & Dale, R. (1997). Building applied natural language generation systems. Natural Language Engineering, 3 (1), 57-87.
  178. Reiter, E., & Sripada, S. (2002). Should corpora texts be gold standards for NLG?. In Proceedings of the International Natural Language Generation Conference, pp. 97- 104, Harriman, New York, USA. Association for Computational Linguistics.
  179. Reiter, E., & Thomson, C. (2020). Shared task on evaluating accuracy. In Proceedings of the 13th International Conference on Natural Language Generation, pp. 227-231, Dublin, Ireland. Association for Computational Linguistics.
  180. Ribeiro, L. F. R., Liu, M., Gurevych, I., Dreyer, M., & Bansal, M. (2022). FactGraph: Evalu- ating factuality in summarization with semantic graph representations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 3238-3253, Seattle, United States. Association for Computational Linguistics.
  181. Rogers, A., & Augenstein, I. (2020). What can we do to improve peer review in NLP?. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1256- 1262, Online. Association for Computational Linguistics.
  182. Rogers, A., Baldwin, T., & Leins, K. (2021). 'just what do you think you're doing, dave?' a checklist for responsible data use in NLP. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, pp. 4821-4833, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  183. Rony, M. R. A. H., Kovriguina, L., Chaudhuri, D., Usbeck, R., & Lehmann, J. (2022). RoMe: A robust metric for evaluating natural language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5645-5657, Dublin, Ireland. Association for Computational Linguistics.
  184. Rush, A. M., Chopra, S., & Weston, J. (2015). A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Meth- ods in Natural Language Processing, pp. 379-389, Lisbon, Portugal. Association for Computational Linguistics.
  185. Saggion, H., Torres-Moreno, J.-M., da Cunha, I., SanJuan, E., & Velázquez-Morales, P. (2010). Multilingual summarization evaluation without human models. In Coling 2010: Posters, pp. 1059-1067, Beijing, China. Coling 2010 Organizing Committee.
  186. Saha, S., Hase, P., Rajani, N., & Bansal, M. (2022). Are hard examples also harder to explain? A study with human and model-generated explanations. CoRR, abs/2211.07517. · arXiv
  187. Sai, A. B., Dixit, T., Sheth, D. Y., Mohan, S., & Khapra, M. M. (2021). Perturbation checklists for evaluating NLG evaluation metrics. In Moens, M., Huang, X., Specia, L., & Yih, S. W. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pp. 7219-7234. Association for Computational Linguistics.
  188. Same, F., Chen, G., & Van Deemter, K. (2022). Non-neural models matter: a re-evaluation of neural referring expression generation systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5554-5567, Dublin, Ireland. Association for Computational Linguistics.
  189. Scheuerman, M. K., Hanna, A., & Denton, E. (2021). Do datasets have politics? disciplinary values in computer vision dataset development. Proceedings of the Conference on ACM Human Computer Interaction, 5 (CSCW2), 1-37.
  190. Schmidt, F. A. (2013). The good, the bad and the ugly: Why crowdsourcing needs ethics. In 2013 International Conference on Cloud and Green Computing, Karlsruhe, Germany, September 30 -October 2, 2013, pp. 531-535. IEEE Computer Society.
  191. Schoch, S., Yang, D., & Ji, Y. (2020). "this is a problem, don't you agree?" framing and bias in human evaluation for natural language generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pp. 10-16, Online (Dublin, Ireland). Association for Computational Linguistics.
  192. Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., & Staiano, J. (2020). MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8051-8067, Online. Association for Computational Linguistics.
  193. Scialom, T., Dray, P., Lamprier, S., Piwowarski, B., Staiano, J., Wang, A., & Gallinari, P. (2021). Questeval: Summarization asks for fact-based evaluation. In Moens, M., Huang, X., Specia, L., & Yih, S. W. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 6594-6604. Association for Computational Linguistics.
  194. Scialom, T., Lamprier, S., Piwowarski, B., & Staiano, J. (2019). Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3246-3256, Hong Kong, China. Association for Computational Linguistics.
  195. Scott, D., & Moore, J. (2007). An NLG evaluation competition? eight reasons to be cau- tious. In Proceedings of the Workshop on Shared Tasks and Comparative Evaluation in Natural Language Generation.
  196. Sellam, T., Das, D., & Parikh, A. (2020). BLEURT: Learning robust metrics for text gener- ation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892, Online. Association for Computational Linguistics.
  197. Shapira, O., Gabay, D., Gao, Y., Ronen, H., Pasunuru, R., Bansal, M., Amsterdamer, Y., & Dagan, I. (2019). Crowdsourcing lightweight pyramids for manual summary evalu- ation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 682-687, Minneapolis, Minnesota. Association for Computational Linguistics.
  198. Sheng, V. S., Provost, F. J., & Ipeirotis, P. G. (2008). Get another label? improving data quality and data mining using multiple, noisy labelers. In Li, Y., Liu, B., & Sarawagi, S. (Eds.), Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24-27, 2008, pp. 614-622. ACM.
  199. Shimorina, A., & Belz, A. (2021). The human evaluation datasheet 1.0: A template for recording details of human evaluation experiments in NLP. CoRR, abs/2103.09710. · arXiv
  200. Shmueli, B., Fell, J., Ray, S., & Ku, L.-W. (2021). Beyond fair pay: Ethical implications of NLP crowdsourcing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pp. 3758-3769, Online. Association for Computational Linguistics.

200 of 235 shown

Cited by

Sorted by citations

Authors
Gehrmann, S., Clark, E., & Sellam, T.
Affiliations
Google (United States) (US)
ORCID
Gehrmann, Sebastian
Venue
Journal of Artificial Intelligence Research
Publisher
AI Access Foundation
Published
2023-05-29
Volume
77
Pages
103-166
ISSN
1076-9757 (ISSN-L)
DOI
10.1613/jair.1.13715
OpenAlex
W4378908626
Cited by
123
References
54
Citations by year
2026: 22 · 2025: 20 · 2024: 28 · 2023: 31 · 2022: 22
FWCI
8.71
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Artificial Intelligence Research (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
natural language generation, assessment practices, human evaluation, automatic evaluation metrics, evaluation benchmark, research reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review