Why We Need New Evaluation Metrics for NLG
Open accessAbstract
The majority of NLG evaluation relies on automatic metrics, such as BLEU.In this paper, we motivate the need for novel, system-and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG.We also show that metric performance is data-and system-specific.Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.4 https://github.com/glampouras/JLOLS_NLG 5 Note that we use lexicalised versions of SFHOTEL and SFREST and a partially lexicalised version of BAGEL, where proper names and place names are replaced by placeholders ("X"), in correspondence with the outputs generated by the MR: inform(name=X, area=X, pricerange=moderate, type=restaurant) Reference: "X is a moderately priced restaurant in X."
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Automatic n-gram metrics such as ROUGE, BLEU, and METEOR show relatively weak correlation with human judgments, especially for open-ended generation or tasks needing domain expertise.
“However, these methods have been shown to exhibit a relatively weak correlation with human judgments, particularly in the context of tasks involving open-ended generation or requiring domain-specific expertise (Novikova et al., 2017).”
From ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2023) · cites this paper as Novikova et al., 2017
Section 7.1 reports that no metric achieved even a moderate utterance-level correlation with human ratings across the datasets and systems studied; its table includes BLEU, ROUGE, and METEOR. The paper does not establish that correlations are especially weak for open-ended generation or tasks requiring domain-specific expertise. Page numbers are unavailable in our copy. (LLM, conf=0.87)
Experimental — accuracy not yet measuredConventional string-matching metrics and explicit task-type prompting are misaligned with cognitive memory scenarios.
“We further show that conventional string-matching metrics and explicit task-type prompting are misaligned with such scenarios”
From LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Novikova et al., 2017
Section 7.1 reports that automatic NLG metrics did not achieve even moderate correlations with human ratings across the datasets, and the example-based analysis shows word-overlap metrics underrating a well-rated output. This supports concern about string-matching metrics in the paper's NLG setting, but the paper does not address explicit task-type prompting or cognitive memory scenarios. (LLM, conf=0.85)
Experimental — accuracy not yet measuredComparing LLM-generated reviews to human reference reviews using lexical overlap metrics is limited, since the references are diverse and the metrics do not align well with human judgments.
“Comparing such reviews to human reference reviews using lexical overlap metrics is limited, since the references are diverse and the metrics do not align well with human judgments (Novikova et al., 2017; Reiter, 2018).”
From The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors (2025) · cites this paper as Novikova et al., 2017
Section 8.2 (page unavailable) discusses how multiple human references affect word-based metric scores, and Section 7.1 (page unavailable) finds that no metric consistently achieves even moderate correlation with human ratings. The study evaluates generated restaurant and hotel utterances, not LLM-generated reviews, so it does not establish the claim specifically for reviews. (LLM, conf=0.85)
Experimental — accuracy not yet measuredExisting intrinsic metrics such as BLEU and ROUGE inadequately capture semantic nuances, contextual relevance, long-range dependencies, changes in critical semantic ordering, and human-centric perspectives, limiting their effectiveness in evaluating healthcare chatbots.
“For this reason, they inadequately capture vital aspects like semantic nuances, contextual relevance, long-range dependencies, changes in critical semantic ordering, and human-centric perspectives11, thereby limiting their effectiveness in evaluating healthcare chatbots.”
From Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI (2024) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Draper, N. R., & Smith, H. (2005) · Technometrics · doi:10.1198/tech.2005.s303 · Cited by 17,993 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
A Study of Translation Edit Rate with Targeted Human Annotation
Snover, M., Dorr, B. J., Schwartz, R. M., Micciulla, L., & Makhoul, J. I. (2006) · Cited by 2,380 · see citing works
Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments
Lavie, A., & Agarwal, A. (2007) · Proceedings of the Second Workshop on Statistical Machine Translation - StatMT '07 · doi:10.3115/1626355.1626389 · Cited by 1,047 · see citing works
Re-evaluating the Role of Bleu in Machine Translation Research
Callison-Burch, C., Osborne, M., & Koehn, P. (2006) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 632 · see citing works
Liu, C., Lowe, R., Serban, I. V., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08023 · Cited by 603 · see citing works
The E2E Dataset: New Challenges For End-to-End Generation
Novikova, J., Dušek, O., & Rieser, V. (2017) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w17-5525 · Cited by 275 · see citing works
Self-citationWhat to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment
Mei, H., Bansal, M., & Walter, M. R. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1086 · Cited by 251 · see citing works
(1979) · Harper & Row eBooks · Cited by 224 · see citing works
Reiter, E., & Belz, A. (2009) · Computational Linguistics · doi:10.1162/coli.2009.35.4.35405 · Cited by 204 · see citing works
Machine translation evaluation versus quality estimation
Specia, L., Raj, D., & Turchi, M. (2010) · Machine Translation · doi:10.1007/s10590-010-9077-2 · Cited by 165 · see citing works
Comparing Automatic Evaluation Measures for Image Description
Elliott, D., & Keller, F. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2074 · Cited by 137 · see citing works
Evaluating Evaluation Methods for Generation in the Presence of Variation
Stent, A., Marge, M., & Singhai, M. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30586-6_38 · Cited by 126 · see citing works
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
Galley, M., Brockett, C., Sordoni, A., Ji, Y., Auli, M., Quirk, C., Mitchell, M., Gao, J., & Dolan, B. (2015) · doi:10.3115/v1/p15-2073 · Cited by 101 · see citing works
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse TargetsPreprint
Galley, M., Brockett, C., Sordoni, A., Ji, Y., Auli, M., Quirk, C., Mitchell, M., Gao, J., & Dolan, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06863 · Cited by 90 · see citing works
Adversarial Evaluation of Dialogue ModelsPreprint
Kannan, A., & Vinyals, O. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.08198 · Cited by 65 · see citing works
Applying natural language generation to indicative summarization
Kan, M., McKeown, K. R., & Klavans, J. L. (2001) · doi:10.3115/1117840.1117853 · Cited by 59 · see citing works
Natural Language Generation enhances human decision-making with uncertain information
Gkatzia, D., Lemon, O., & Rieser, V. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2043 · Cited by 59 · see citing works
Self-citationA Context-aware Natural Language Generator for Dialogue Systems
Dušek, O., & Jurčíček, F. (2016) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w16-3622 · Cited by 52 · see citing works
Self-citationA Snapshot of NLG Evaluation Practices 2005 - 2014
Gkatzia, D., & Mahamood, S. (2015) · Proceedings of the 15th European Workshop on Natural Language Generation (ENLG) · doi:10.18653/v1/w15-4708 · Cited by 48 · see citing works
Crowd-sourcing NLG Data: Pictures Elicit Better Data.
Novikova, J., Lemon, O., & Rieser, V. (2016) · International Natural Language Generation Conference (INLG) · doi:10.18653/v1/w16-6644 · Cited by 48 · see citing works
Self-citationThere's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction
Napoles, C., Sakaguchi, K., & Tetreault, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1228 · Cited by 48 · see citing works
Imitation learning for language generation from unaligned data
Λάμπουρας, Γ., & Vlachos, A. (2016) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 36 · see citing works
Rieser, V., Lemon, O., & Keizer, S. (2014) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2014.2315271 · Cited by 35 · see citing works
Self-citationTraining a Natural Language Generator From Unaligned Data
Dušek, O., & Jurčíček, F. (2015) · doi:10.3115/v1/p15-1044 · Cited by 34 · see citing works
Self-citationNatural Language Generation in Dialogue using Lexicalized and Delexicalized DataPreprint
Sharma, S., He, J., Suleman, K., Schulz, H., & Bachman, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03632 · Cited by 29 · see citing works
Correlating human and automatic evaluation of a German surface realiser
Cahill, A. (2009) · doi:10.3115/1667583.1667615 · Cited by 28 · see citing works
Referenceless Quality Estimation for Natural Language GenerationPreprint
Dušek, O., Novikova, J., & Rieser, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.01759 · Cited by 22 · see citing works
Self-citationWhat to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine AlignmentPreprint
Mei, H., Bansal, M., & Walter, M. R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.00838 · Cited by 22 · see citing works
Re-evaluating Automatic Metrics for Image CaptioningPreprint
Kilickaya, M., Erdem, A., İkizler-Cinbiş, N., & Erdem, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07600 · Cited by 14 · see citing works
Multi-domain Neural Network Language Generation for Spoken Dialogue SystemsPreprint
Wen, T.-H., Gašić, M., Mrkšić, N., Rojas-Barahona, L. M., Su, P.-H., Vandyke, D., & Young, S. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.01232 · Cited by 8 · see citing works
Crowd-sourcing NLG Data: Pictures Elicit Better DataPreprint
Novikova, J., Lemon, O., & Rieser, V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.00339 · Cited by 8 · see citing works
Self-citationThere's No Comparison: Reference-less Evaluation Metrics in Grammatical Error CorrectionPreprint
Napoles, C., Sakaguchi, K., & Tetreault, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02124 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 16
- Anja Belz and Eric Kow. 2011. Discrete vs. con- tinuous rating scales for language evaluation in NLP. In Proceedings of the 49th Annual Meet- ing of the Association for Computational Linguis- tics: Human Language Technologies: Short Pa- pers -Volume 2. Association for Computational Linguistics, Portland, OR, USA, pages 230-235. http://aclweb.org/anthology/P11-2040.
- Anja Belz and Ehud Reiter. 2006. Comparing au- tomatic and human evaluation of NLG systems. In Proceedings of the 11th Conference of the Eu- ropean Chapter of the Association for Computa- tional Linguistics. Trento, Italy, pages 313-320. http://aclweb.org/anthology/E06-1040.
- George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co- occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research.
- Ondřej Dušek and Filip Jurčíček. 2016. Sequence- to-sequence generation for spoken dialogue via deep syntax trees and strings. In Proceed- ings of the 54th Annual Meeting of the As- sociation for Computational Linguistics. Berlin, Germany, pages 45-51. arXiv:1606.05491. http://aclweb.org/anthology/P16-2008. · arXiv
- Dominic Espinosa, Rajakrishnan Rajkumar, Michael White, and Shoshana Berleant. 2010. Further meta- evaluation of broad-coverage surface realization. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics, pages 564-574. http://aclweb.org/anthology/D10-1055.
- Thomas Francois and Delphine Bernhard, editors. 2014. Recent Advances in Automatic Readability Assessment and Text Simplification, volume 165:2 of International Journal of Applied Linguistics. John Benjamins. http://doi.org/10.1075/itl.165. · DOI
- Jesús Giménez and Lluís Màrquez. 2008. A smor- gasbord of features for automatic MT evaluation. In Proceedings of the Third Workshop on Statisti- cal Machine Translation. Association for Computa- tional Linguistics, Columbus, OH, USA, pages 195- 198. http://aclweb.org/anthology/W08-0332.
- Aaron L. F. Han, Derek F. Wong, and Lidia S. Chao. 2012. LEPOR: A robust evaluation metric for ma- chine translation with augmented factors. In Pro- ceedings of COLING 2012: Posters. The COLING 2012 Organizing Committee, Mumbai, India, pages 441-450. http://aclweb.org/anthology/C12-2044.
- Lushan Han, Abhay Kashyap, Tim Finin, James Mayfield, and Jonathan Weese. 2013. UMBC EBIQUITY-CORE: Semantic textual simi- larity systems. In Proceedings of the Second Joint Conference on Lexical and Computational Seman- tics (*SEM). Atlanta, Georgia, volume 1, pages 44-52. http://aclweb.org/anthology/S13-1005.
- Helen Hastie, Heriberto Cuayahuitl, Nina Dethlefs, Si- mon Keizer, and Xingkun Liu. 2016. Why bother? Is evaluation of NLG in an end-to-end Spoken Di- alogue System worth it? In Proceedings of the In- ternational Workshop on Spoken Dialogue Systems (IWSDS). Saariselkä, Finland.
- Micah Hodosh and Julia Hockenmaier. 2016. Focused evaluation for image description with binary forced- choice tasks. In Proceedings of the 5th Workshop on Vision and Language. Berlin, Germany, pages 19- 28. http://aclweb.org/anthology/W16-3203.
- Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard H. Hovy. 2013. Learning whom to trust with MACE. In Proceedings of NAACL- HLT. Atlanta, GA, USA, pages 1120-1130. http://aclweb.org/anthology/N13-1132.
- Chin-Yew Lin. 2004. ROUGE: A package for au- tomatic evaluation of summaries. In Text summa- rization branches out: Proceedings of the ACL- 04 workshop. Barcelona, Spain, pages 74-81. http://aclweb.org/anthology/W04-1013.
- Franc ¸ois Mairesse, Milica Gašić, Filip Jurčíček, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Pro- ceedings of the 48th Annual Meeting of the Associa- tion for Computational Linguistics. Association for Computational Linguistics, Uppsala, Sweden, pages 1552-1561. http://aclweb.org/anthology/P10-1157.
- Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Processing. Lisbon, Portugal, pages 1711-1721. http://aclweb.org/anthology/D15-1199.
- Evan James Williams. 1959. Regression analysis. John Wiley & Sons, New York, NY, USA.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005 · see citing works
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · see citing works
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · see citing works
Using large language models in psychology
Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00241-5 · Cited by 369 · see citing works
Challenges in building intelligent open-domain dialog systems
Huang, M., Zhu, X., & Gao, J. (2020) · ACM Transactions on Information Systems · doi:10.1145/3383123 · Cited by 288 · see citing works
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290 · see citing works
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2019) · arXiv · doi:10.48550/arxiv.1911.00650 · Cited by 264
“automatic evaluation of generative model quality (Novikova et al., 2017; Kannan and Vinyals, 2017; Lowe et al., 2017).”
BackgroundAutomatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.164 · Cited by 264 · see citing works
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247 · see citing works
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
“In the following, we briefly outline these evaluation methods.”
MethodBest practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
“There are arguably still good reasons to use automatic metrics: they are a cheap, quick and re-peatable way to approximate text quality (Reiter and Belz, 2009), and they can be useful for error analysis and system development (Novikova et al., 2017).”
BackgroundMethodInfluentialA Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235 · see citing works
Learning to Write with Cooperative Discriminators
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., & Choi, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1152 · Cited by 192 · see citing works
Unifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161 · see citing works
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Dhingra, B., Faruqui, M., Parikh, A., Chang, M.-W., Das, D., & Cohen, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1483 · Cited by 150 · see citing works
Re-evaluating Evaluation in Text Summarization
Bhandari, M., Gour, P. N., Ashfaq, A., Liu, P., & Neubig, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.751 · Cited by 148 · see citing works
Do Massively Pretrained Language Models Make Better Storytellers?
See, A., Pappu, A., Saxena, R., Yerukola, A., & Manning, C. D. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1079 · Cited by 139 · see citing works
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., & Manning, C. D. (2018) · arXiv · doi:10.48550/arxiv.1809.09600 · Cited by 97 · see citing works
GO FIGURE: A Meta Evaluation of Factuality in Summarization
Gabriel, S., Celikyilmaz, A., Jha, R., Choi, Y., & Gao, J. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.42 · Cited by 75 · see citing works
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., & Liu, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.07201 · Cited by 55 · see citing works
“However, these methods have been shown to exhibit a relatively weak correlation with human judgments, particularly in the context of tasks involving open-ended generation or requiring domain-specific expertise (Novikova et al., 2017).”
BackgroundControlling hallucinations at word level in data-to-text generation
Rebuffel, C., Roberti, M., Soulier, L., Scoutheeten, G., Cancelliere, R., & Gallinari, P. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-021-00801-4 · Cited by 50 · see citing works
Distilled GPT for source code summarization
Su, C.-Y., & McMillan, C. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00421-4 · Cited by 49 · see citing works
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
Roller, S., Boureau, Y.-L., Weston, J., Bordes, A., Dinan, E., Fan, A., Gunning, D., Ju, D., Li, M., Poff, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.12442 · Cited by 38 · see citing works
Correlating Automated and Human Evaluation of Code Documentation Generation Quality
Hu, X., Chen, Q., Wang, H., Xia, X., Lo, D., & Zimmermann, T. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3502853 · Cited by 33 · see citing works
Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., & Hakkani-Tur, D. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8608 · Cited by 26 · see citing works
Qi, P., Zhang, Y., & Manning, C. D. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.3 · Cited by 23 · see citing works
Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation
Rebuffel, C., Scialom, T., Soulier, L., Piwowarski, B., Lamprier, S., Staiano, J., Scoutheeten, G., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.633 · Cited by 24 · see citing works
Benchmark Data Contamination of Large Language Models: A Survey
Xu, C., Guan, S., Greene, D., & Kechadi, M.-T. (2024) · arXiv · doi:10.48550/arxiv.2406.04244 · Cited by 22 · see citing works
Refocusing on Relevance: Personalization in NLG
Dudy, S., Bedrick, S., & Webber, B. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.421 · Cited by 17 · see citing works
Learning to Ask Like a Physician
Lehman, E., Lialin, V., Legaspi, K. E., Sy, A. J., Pile, P. T., Alberto, N. R., Ragasa, R. R., Puyat, C. V., Taliño, M. K., Alberto, I. R., et al. (2022) · Proceedings of the 4th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2022.clinicalnlp-1.8 · Cited by 10 · see citing works
The Good, the Bad and the Constructive: Automatically Measuring Peer Review’s Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1476 · Cited by 1 · see citing works
LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
Li, Y., Guo, W., Zhang, L., Xu, R., Huang, M., Liu, H., Xu, L., Xu, Y., & Liu, J. (2026) · arXiv · doi:10.48550/arxiv.2602.10715
“However, these metrics are known to poorly reflect semantic correctness and behavioral validity in open-ended generation (Novikova et al., 2017; Post, 2018; Bulian et al., 2022).”
PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers
Loc, N. P. P., La Viet, T. H., Khanh, T. T., Nguyen, D. A., Pham, T. A. N., Nguyen, T., Chawla, N. V., Buntine, W., Wong, K.-S., Doan, K. D., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.26730
The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · arXiv · doi:10.48550/arxiv.2509.04484
“Comparing such reviews to human reference reviews using lexical overlap metrics is limited, since the references are diverse and the metrics do not align well with human judgments (Novikova et al., 2017; Reiter, 2018).”
- Authors
- Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V.
- Affiliations
- Heriot-Watt University (GB)
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 2241-2252
- OpenAlex
- W2737041661
- Cited by
- 422
- References
- 35
- Citations by year
- 2026: 16 · 2025: 21 · 2024: 25 · 2023: 44 · 2022: 59 · 2021: 93 · 2020: 79 · 2019: 59 · 2018: 24 · 2017: 2
- FWCI
- 20.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Research Explorer (The University of Manchester) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- Engineering and Physical Sciences Research Council (EP/N017536/1, EP/M005429/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- natural language generation evaluation, automatic evaluation metrics, BLEU, human judgment correlation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
