CoChat
Sign Up
PreprintCited by 406Natural Language Processing TechniquesSpeech and dialogue systemsTopic Modeling

Why We Need New Evaluation Metrics for NLG

Open access
Novikova, Jekaterina, Dušek, Ondřej, Curry, Amanda Cercas, Rieser, Verena·2017·arXiv
DOI 10.48550/arxiv.1707.06875arXiv 1707.06875
Open this paper in CoChat — free accountOpen PDF

Abstract

The majority of NLG evaluation relies on automatic metrics, such as BLEU . In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1707.06875

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LoCoMo relies on string-matching-based metrics to compare model outputs with a reference answer, assuming that correctness can be determined by surface-level overlap.

    “On the output side, LoCoMo relies on stringmatching-based metrics to compare model outputs with a reference answer. These metrics assume that correctness can be determined by surface-level overlap.”

    From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Novikova et al., 2017

    Section 4.1 describes word-overlap metrics that compare generated text with human references, and Section 8.2 says such metrics can rate an utterance as correct despite grammatical errors. The paper does not establish that LoCoMo uses these metrics; its discussion concerns NLG evaluation generally, not LoCoMo specifically. Page numbers are unavailable in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Generation-based metrics such as BLEU, ROUGE, exact match, and token-level F1 poorly reflect semantic correctness and behavioral validity in open-ended generation.

    “these metrics are known to poorly reflect semantic correctness and behavioral validity in open-ended generation”

    From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Novikova et al., 2017

    Section 8.3 says word-based metrics cannot distinguish good- from medium-quality output, and the example analysis shows BLEU and ROUGE scoring a meaning-preserving utterance poorly. The paper studies NLG outputs in restaurant and hotel search; it does not establish the claim for exact match or token-level F1, or assess behavioral validity across open-ended generation. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Common evaluation practices such as task-disclosed prompting and string-matching metrics can be systematically misleading, even for factual memory, by conflating memory fidelity with prompt adaptation and generation style.

    “We further show that common evaluation practices-such as task-disclosed prompting and string-matching metrics-can be systematically misleading, even for factual memory, by conflating memory fidelity with prompt adaptation and generation style”

    From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Novikova et al., 2017

    The example-based analysis (page not specified in our copy) shows that word-overlap metrics gave a low score to an accurate, highly rated NLG output because its wording differed from the references. That supports the warning about string-matching metrics and generation style. The paper does not address task-disclosed prompting, factual memory, or a conflation of memory fidelity with prompt adaptation. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 45

  1. Anja Belz and Eric Kow. 2011. Discrete vs. con- tinuous rating scales for language evaluation in NLP. In Proceedings of the 49th Annual Meet- ing of the Association for Computational Linguis- tics: Human Language Technologies: Short Pa- pers -Volume 2. Association for Computational Linguistics, Portland, OR, USA, pages 230-235. http://aclweb.org/anthology/P11-2040.
  2. Anja Belz and Ehud Reiter. 2006. Comparing au- tomatic and human evaluation of NLG systems. In Proceedings of the 11th Conference of the Eu- ropean Chapter of the Association for Computa- tional Linguistics. Trento, Italy, pages 313-320. http://aclweb.org/anthology/E06-1040.
  3. Aoife Cahill. 2009. Correlating human and au- tomatic evaluation of a German surface realiser. In Proceedings of the ACL-IJCNLP 2009 Con- ference Short Papers. Association for Computa- tional Linguistics, Suntec, Singapore, pages 97-100. https://aclweb.org/anthology/P09-2025.
  4. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of BLEU in machine translation research. In Proceed- ings of the 11th Conference of the European Chapter of the Association for Computational Linguistics. Trento, Italy, pages 249-256. http://aclweb.org/anthology/E06-1032.
  5. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co- occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research.
  6. Ondrej Dušek, Jekaterina Novikova, and Verena Rieser. 2017. Referenceless quality estimation for natu- ral language generation. In Proceedings of the 1st Workshop on Learning to Generate Natural Lan- guage.
  7. Ondřej Dušek and Filip Jurčíček. 2015. Train- ing a natural language generator from unaligned data. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Lin- guistics and the 7th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers). Beijing, China, pages 451-461. http://aclweb.org/anthology/P15-1044.
  8. Ondřej Dušek and Filip Jurčíček. 2016. A context- aware natural language generator for dialogue systems. In Proceedings of the 17th An- nual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics, Los Angeles, CA, USA, pages 185-190. arXiv:1608.07076. http://aclweb.org/anthology/W16-3622. · arXiv
  9. Ondřej Dušek and Filip Jurčíček. 2016. Sequence- to-sequence generation for spoken dialogue via deep syntax trees and strings. In Proceed- ings of the 54th Annual Meeting of the As- sociation for Computational Linguistics. Berlin, Germany, pages 45-51. arXiv:1606.05491. http://aclweb.org/anthology/P16-2008. · arXiv
  10. Desmond Elliott and Frank Keller. 2014. Comparing automatic evaluation measures for image descrip- tion. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers). Association for Computa- tional Linguistics, Baltimore, MD, USA, pages 452- 457. http://aclweb.org/anthology/P14-2074.
  11. Dominic Espinosa, Rajakrishnan Rajkumar, Michael White, and Shoshana Berleant. 2010. Further meta- evaluation of broad-coverage surface realization. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics, pages 564-574. http://aclweb.org/anthology/D10-1055.
  12. Thomas Francois and Delphine Bernhard, editors. 2014. Recent Advances in Automatic Readability Assessment and Text Simplification, volume 165:2 of International Journal of Applied Linguistics. John Benjamins. http://doi.org/10.1075/itl.165. · DOI
  13. Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Mar- garet Mitchell, Jianfeng Gao, and Bill Dolan. 2015. deltaBLEU: A discriminative metric for generation tasks with intrinsically diverse targets. In Proceed- ings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Interna- tional Joint Conference on Natural Language Pro- cessing (Volume 2: Short Papers). Association for Computational Linguistics, Beijing, China, pages 445-450. http://aclweb.org/anthology/P15-2073.
  14. Jesús Giménez and Lluís Màrquez. 2008. A smor- gasbord of features for automatic MT evaluation. In Proceedings of the Third Workshop on Statisti- cal Machine Translation. Association for Computa- tional Linguistics, Columbus, OH, USA, pages 195- 198. http://aclweb.org/anthology/W08-0332.
  15. Dimitra Gkatzia, Oliver Lemon, and Verena Rieser. 2016. Natural language generation enhances human decision-making with uncertain informa- tion. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Berlin, Germany, pages 264-268. arXiv:1606.03254. http://aclweb.org/anthology/P16-2043. · arXiv
  16. Dimitra Gkatzia and Saad Mahamood. 2015. A snap- shot of NLG evaluation practices 2005-2014. In Proceedings of the 15th European Workshop on Natural Language Generation (ENLG). Association for Computational Linguistics, Brighton, UK, pages 57-60. https://doi.org/10.18653/v1/W15-4708. · DOI
  17. Aaron L. F. Han, Derek F. Wong, and Lidia S. Chao. 2012. LEPOR: A robust evaluation metric for ma- chine translation with augmented factors. In Pro- ceedings of COLING 2012: Posters. The COLING 2012 Organizing Committee, Mumbai, India, pages 441-450. http://aclweb.org/anthology/C12-2044.
  18. Lushan Han, Abhay Kashyap, Tim Finin, James Mayfield, and Jonathan Weese. 2013. UMBC EBIQUITY-CORE: Semantic textual simi- larity systems. In Proceedings of the Second Joint Conference on Lexical and Computational Seman- tics (*SEM). Atlanta, Georgia, volume 1, pages 44-52. http://aclweb.org/anthology/S13-1005.
  19. Helen Hastie, Heriberto Cuayahuitl, Nina Dethlefs, Si- mon Keizer, and Xingkun Liu. 2016. Why bother? Is evaluation of NLG in an end-to-end Spoken Di- alogue System worth it? In Proceedings of the In- ternational Workshop on Spoken Dialogue Systems (IWSDS). Saariselkä, Finland.
  20. Micah Hodosh and Julia Hockenmaier. 2016. Focused evaluation for image description with binary forced- choice tasks. In Proceedings of the 5th Workshop on Vision and Language. Berlin, Germany, pages 19- 28. http://aclweb.org/anthology/W16-3203.
  21. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard H. Hovy. 2013. Learning whom to trust with MACE. In Proceedings of NAACL- HLT. Atlanta, GA, USA, pages 1120-1130. http://aclweb.org/anthology/N13-1132.
  22. Min-Yen Kan, Kathleen R. McKeown, and Judith L. Klavans. 2001. Applying natural language gen- eration to indicative summarization. In Proceed- ings of the 8th European Workshop on Natural Language Generation. Association for Computa- tional Linguistics, Toulouse, France, pages 1-9. https://doi.org/10.3115/1117840.1117853. · DOI
  23. Anjuli Kannan and Oriol Vinyals. 2017. Adver- sarial evaluation of dialogue models. CoRR abs/1701.08198. https://arxiv.org/abs/1701.08198. · arXiv
  24. Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2017. Re-evaluating auto- matic metrics for image captioning. In Pro- ceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computa- tional Linguistics. Association for Computational Linguistics, Valencia, Spain. arXiv:1612.07600. https://arxiv.org/abs/1612.07600. · arXiv
  25. Gerasimos Lampouras and Andreas Vlachos. 2016. Imitation learning for language generation from un- aligned data. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, pages 1101- 1112. http://aclweb.org/anthology/C16-1105.
  26. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for cat- egorical data. Biometrics 33(1):159-174. https://doi.org/10.2307/2529310. · DOI · Paper page
  27. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Pro- ceedings of the Second Workshop on Statistical Ma- chine Translation. Association for Computational Linguistics, Prague, Czech Republic, pages 228- 231. http://aclweb.org/anthology/W07-0734.
  28. Chin-Yew Lin. 2004. ROUGE: A package for au- tomatic evaluation of summaries. In Text summa- rization branches out: Proceedings of the ACL- 04 workshop. Barcelona, Spain, pages 74-81. http://aclweb.org/anthology/W04-1013.
  29. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue sys- tem: An empirical study of unsupervised evalua- tion metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, TX, USA, pages 2122-2132. arXiv:1603.08023. http://aclweb.org/anthology/D16-1230. · arXiv
  30. Franc ¸ois Mairesse, Milica Gašić, Filip Jurčíček, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Pro- ceedings of the 48th Annual Meeting of the Associa- tion for Computational Linguistics. Association for Computational Linguistics, Uppsala, Sweden, pages 1552-1561. http://aclweb.org/anthology/P10-1157.
  31. Hongyuan Mei, Mohit Bansal, and Matthew R. Wal- ter. 2016. What to talk about and how? Se- lective generation using LSTMs with coarse-to- fine alignment. In Proceedings of NAACL-HLT 2016. San Diego, CA, USA. arXiv:1509.00838. http://aclweb.org/anthology/N16-1086. · arXiv
  32. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2016. There's no comparison: Reference-less evaluation metrics in gram- matical error correction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, TX, USA, pages 2109-2115. arXiv:1610.02124. http://aclweb.org/anthology/D16-1228. · arXiv
  33. Jekaterina Novikova, Ondrej Dušek, and Verena Rieser. 2017. The E2E dataset: New chal- lenges for end-to-end generation. In Pro- ceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dia- logue. Saarbrücken, Germany. ArXiv:1706.09254. https://arxiv.org/abs/1706.09254. · arXiv
  34. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing NLG data: Pictures elicit better data. In Proceedings of the 9th Inter- national Natural Language Generation Conference. Edinburgh, UK, pages 265-273. arXiv:1608.00339. http://aclweb.org/anthology/W16-2302. · arXiv
  35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Compu- tational Linguistics, Philadelphia, PA, USA, pages 311-318. http://aclweb.org/anthology/P02-1040.
  36. Ehud Reiter and Anja Belz. 2009. An investiga- tion into the validity of some metrics for automat- ically evaluating natural language generation sys- tems. Computational Linguistics 35(4):529-558. https://doi.org/10.1162/coli.2009.35.4.35405. · DOI
  37. Verena Rieser, Oliver Lemon, and Simon Keizer. 2014. Natural language generation as incre- mental planning under uncertainty: Adaptive information presentation for statistical dialogue systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing 22(5):979-993. https://doi.org/10.1109/TASL.2014.2315271. · DOI
  38. Shikhar Sharma, Jing He, Kaheer Suleman, Hannes Schulz, and Philip Bachman. 2016. Natural lan- guage generation in dialogue using lexicalized and delexicalized data. CoRR abs/1606.03632. http://arxiv.org/abs/1606.03632. · arXiv
  39. Matthew Snover, Bonnie Dorr, Richard Schwartz, Lin- nea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annota- tion. In Proceedings of the 7th Conference of the As- sociation for Machine Translation of the Americas. Cambridge, MA, USA, pages 223-231. http://mt- archive.info/AMTA-2006-Snover.pdf.
  40. Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010. Machine translation evaluation versus qual- ity estimation. Machine translation 24(1):39-50. https://doi.org/10.1007/s10590-010-9077-2. · DOI
  41. Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for gener- ation in the presence of variation. In Computa- tional Linguistics and Intelligent Text Processing: 6th International Conference, CICLing 2005, Mex- ico City, Mexico, February 13-19, 2005. Proceed- ings. Springer, Berlin/Heidelberg, pages 341-351. https://doi.org/10.1007/978-3-540-30586-6 · DOI
  42. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus- based image description evaluation. In Pro- ceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA, pages 4566-4575. https://doi.org/10.1109/CVPR.2015.7299087. · DOI · Paper page
  43. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina Maria Rojas-Barahona, Pei-hao Su, David Vandyke, and Steve J. Young. 2016. Multi- domain neural network language generation for spoken dialogue systems. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies. San Diego, CA, USA, pages 120-129. arXiv:1603.01232. http://aclweb.org/anthology/N16-1015. · arXiv
  44. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Processing. Lisbon, Portugal, pages 1711-1721. http://aclweb.org/anthology/D15-1199.
  45. Evan James Williams. 1959. Regression analysis. John Wiley & Sons, New York, NY, USA.

Cited by

Sorted by citations

Authors
Novikova, J., Dušek, O., Curry, A. C., & Rieser, V.
Affiliations
Heriot-Watt University (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Pages
2241-2252
DOI
10.48550/arxiv.1707.06875
arXiv
1707.06875
OpenAlex
W2737041661
Cited by
406
References
0
Citations by year
2026: 16 · 2025: 21 · 2024: 25 · 2023: 44 · 2022: 57 · 2021: 93 · 2020: 73 · 2019: 57 · 2018: 18 · 2017: 2
FWCI
18.71
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Research Explorer (The University of Manchester) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Nvidia
  • Engineering and Physical Sciences Research Council (EP/N017536/1, EP/M005429/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
natural language generation evaluation, automatic evaluation metrics, BLEU, human judgment correlation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review