How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
We investigate evaluation metrics for dialogue response generation systems where supervised labels, such as task completion, are not available.Recent works in response generation have adopted metrics from machine translation to compare a model's generated response to a single target response.We show that these metrics correlate very weakly with human judgements in the non-technical Twitter domain, and not at all in the technical Ubuntu domain.We provide quantitative and qualitative results highlighting specific weaknesses in existing metrics, and provide recommendations for future development of better automatic evaluation metrics for dialogue systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For high-variation domains like our generation tasks, on the other hand, wordoverlap-based machine translation metrics like BLEU (Post 2018) are expected to have low correlation with human quality judgments (Liu et al. 2016).”
From Can Large Language Models Transform Computational Social Science? (2024) · cites this paper as Liu et al. 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 1×
“, 2015), machine translation (Cho et al., 2014), and dialogue response generation systems (Sordoni et al.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
(2004) · Cited by 9,718
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · cited 2×
“2 w.r.t. the other respondents, which is a standard measure for inter-rater agreement (Cohen, 1968).”
BackgroundLandauer, T. K., & Dumais, S. T. (1997) · Psychological Review · doi:10.1037/0033-295x.104.2.211 · Cited by 6,282
(2005) · Cited by 4,152
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926
Cited by
Deep Reinforcement Learning for Dialogue Generation
Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., & Gao, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1127 · Cited by 1,204
A Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., & Gašić, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1547 · Cited by 1,028
“, 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundBLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. P. (2020) · arXiv · doi:10.48550/arxiv.2004.04696 · Cited by 1,005
“Thus, they have been repeatedly shown to correlate poorly with human judgment, in particular when all the systems to compare have a similar level of accuracy (Liu et al., 2016; Novikova et al., 2017; Chaganty et al., 2018).”
BackgroundCoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975
Adversarial Learning for Neural Dialogue Generation
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., & Jurafsky, D. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1230 · Cited by 873
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2305.03514 · Cited by 839
The design and implementation of Xiaoice, an empathetic social chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2020) · Computational Linguistics · doi:10.1162/coli_a_00368 · Cited by 777
The Design and Implementation of XiaoIce, an Empathetic Social Chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2018) · arXiv · doi:10.48550/arxiv.1812.08989 · Cited by 777
Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset
Rashkin, H., Smith, E. M., Li, M., & Boureau, Y.-L. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1534 · Cited by 771
Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
Zhou, H., Huang, M., Zhang, T., Zhu, X., & Liu, B. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11325 · Cited by 763
NewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706
ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning
Sap, M., Le Bras, R., Allaway, E., Bhagavatula, C., Lourie, N., Rashkin, H., Roof, B., Smith, N. A., & Choi, Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013027 · Cited by 740
Zhao, T., Zhao, R., & Eskenazi, M. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1061 · Cited by 736
“Automatically evaluating an open-domain generative dialog model is an open research challenge (Liu et al., 2016).”
BackgroundA Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
Serban, I., Sordoni, A., Lowe, R., Charlin, L., Pineau, J., Courville, A., & Bengio, Y. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10983 · Cited by 697
Self-citationDailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
Li, Y., Su, H., Shen, X., Li, W., Cao, Z., & Niu, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.03957 · Cited by 646
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614
“, 2002), ROUGE (Lin, 2004), or other related metrics in translation and summarization, while popular and easy to compute, have been shown to be unsuitable for dialog (Liu et al., 2016) or more broadly language gener-”
BackgroundResultInfluentialCan Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585
A Survey on Dialogue Systems: Recent Advances and New Frontiers
Chen, H., Liu, X., Yin, D., & Tang, J. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3166054.3166058 · Cited by 579
Retrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575
Challenges in Data-to-Document Generation
Wiseman, S., Shieber, S., & Rush, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1239 · Cited by 524
Plug and Play Language Models: A Simple Approach to Controlled Text Generation
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., & Liu, R. (2019) · arXiv · doi:10.48550/arxiv.1912.02164 · Cited by 508
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1053 · Cited by 490
“BLEU and similar metrics tend to reward systems that use the same wording as reference texts, causing repetitive word usage that is deemed undesirable to humans (Liu et al., 2016).”
BackgroundDas, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.121 · Cited by 445
“[26] perform a study of problems in existing evaluation protocols for free-form dialog.”
BackgroundWhy We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422
“…by various studies in NLG (Stent et al., 2005; Belz and Reiter, 2006; Reiter and Belz, 2009), as well as in related fields, such as dialogue systems (Liu et al., 2016), machine translation (MT) (Callison-Burch et al., 2006), and image captioning (Elliott and Keller, 2014; Kilickaya et al., 2017).”
BackgroundA Knowledge-Grounded Neural Conversation Model
Ghazvininejad, M., Brockett, C., Chang, M.-W., Dolan, B., Gao, J., Yih, W.-T., & Galley, M. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11977 · Cited by 422
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Curry, A. C., & Rieser, V. (2017) · arXiv · doi:10.48550/arxiv.1707.06875 · Cited by 406
Key-Value Retrieval Networks for Task-Oriented Dialogue
Eric, M., Krishnan, L., Charette, F., & Manning, C. D. (2017) · Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue · doi:10.18653/v1/w17-5506 · Cited by 388
GuessWhat?! Visual Object Discovery through Multi-modal Dialogue
De Vries, H., Strub, F., Chandar, S., Pietquin, O., Larochelle, H., & Courville, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.475 · Cited by 369
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Wang, A., Cho, K., & Lewis, M. (2020) · arXiv · doi:10.48550/arxiv.2004.04228 · Cited by 356
“We briefly describe the most common metrics in this family, and refer readers to Liu et al. (2016) for further discussion.”
BackgroundThe Second Conversational Intelligence Challenge (ConvAI2)
Dinan, E., Logacheva, V., Malykh, V., Miller, A., Shuster, K., Urbanek, J., Kiela, D., Szlam, A., Serban, I., Lowe, R., et al. (2020) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-29135-8_7 · Cited by 348
Self-citationDeal or No Deal? End-to-End Learning of Negotiation Dialogues
Lewis, M., Yarats, D., Dauphin, Y., Parikh, D., & Batra, D. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1259 · Cited by 347
Advances and challenges in conversational recommender systems: A survey
Gao, C., Lei, W., He, X., de Rijke, M., & Chua, T.-S. (2021) · AI Open · doi:10.1016/j.aiopen.2021.06.002 · Cited by 337
Social Bias Frames: Reasoning about Social and Power Implications of Language
Sap, M., Gabriel, S., Qin, L., Jurafsky, D., Smith, N. A., & Choi, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.486 · Cited by 332
An Introduction to Neural Information Retrieval
Mitra, B., & Craswell, N. (2018) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000061 · Cited by 317
Generative Adversarial Networks for face generation: A survey
Kammoun, A., Slama, R., Tabia, H., Ouni, T., & Abid, M. (2022) · ACM Computing Surveys · doi:10.1145/1122445.1122456 · Cited by 311
Plan-and-Write: Towards Better Automatic Storytelling
Yao, L., Peng, N., Weischedel, R., Knight, K., Zhao, D., & Yan, R. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017378 · Cited by 308
Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems
Madotto, A., Wu, C.-S., & Fung, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1136 · Cited by 302
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290
“Sometimes the generated text is explicitly compared to human-authored text (e.g., Liu et al., 2016; Zellers et al., 2021; Zhang et al., 2020), but even when no human-authored text is evaluated, evaluators implicitly compare the generated text to their knowledge of language and norms within specific…”
BackgroundAll That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
Challenges in building intelligent open-domain dialog systems
Huang, M., Zhu, X., & Gao, J. (2020) · ACM Transactions on Information Systems · doi:10.1145/3383123 · Cited by 288
Bisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J., Lapata, M., Lazaridou, A., May, J., Nisnevich, A., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.703 · Cited by 274
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.16594 · Cited by 271
Augmenting End-to-End Dialogue Systems With Commonsense Knowledge
Young, T., Cambria, E., Chaturvedi, I., Zhou, H., Biswas, S., & Huang, M. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11923 · Cited by 266
Recent advances in deep learning based dialogue systems: a systematic survey
Ni, J., Young, T., Pandelea, V., Xue, F., & Cambria, E. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10248-8 · Cited by 264
DailyDialog: A Manually Labelled Multi-Turn Dialogue Dataset
Li, Y. (2025) · TIB Data Manager · doi:10.57702/l9x4bl6j · Cited by 262
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247
Evaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207
A Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235
Learning to Write with Cooperative Discriminators
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., & Choi, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1152 · Cited by 192
“While many of the issues of automatic metrics (Liu et al., 2016; Novikova et al., 2017) have been alleviated by crowd-sourcing, we found it difficult to incentivize crowd workers to spend significant time on any one datum, forcing them to rely on a shallower understanding of the text.”
BackgroundReformulating Unsupervised Style Transfer as Paraphrase Generation
Krishna, K., Wieting, J., & Iyyer, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.55 · Cited by 185
Personalizing Dialogue Agents via Meta-Learning
Madotto, A., Lin, Z., Wu, C.-S., & Fung, P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1542 · Cited by 183
Retrieve and Refine: Improved Sequence Generation Models For Dialogue
Weston, J., Dinan, E., & Miller, A. (2018) · Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI · doi:10.18653/v1/w18-5713 · Cited by 165
Unifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161
Sharma, S., Asri, L. E., Schulz, H., & Zumer, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09799 · Cited by 170
“We also compute other embedding-based meth-ods which have been used as evaluation metrics for measuring human correlation in recent literature (Liu et al., 2016) for non task-oriented dialogue in Sections 3.2.2, 3.2.3, and 3.2.4.”
BackgroundMethodResultInfluentialConversational Agents: Goals, Technologies, Vision and Challenges
Allouch, M., Azaria, A., & Azoulay, R. (2021) · Sensors · doi:10.3390/s21248448 · Cited by 168
“Liu et al. [207] investigated the usage of the above translation and summarization evaluation metrics for CA.”
BackgroundPersonalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · arXiv · doi:10.48550/arxiv.1801.07243 · Cited by 165
Hallucination‐Free? Assessing the Reliability of Leading AI Legal Research Tools
Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025) · Journal of Empirical Legal Studies · doi:10.1111/jels.12413 · Cited by 182
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Dhingra, B., Faruqui, M., Parikh, A., Chang, M.-W., Das, D., & Cohen, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1483 · Cited by 150
“Over the years several studies have evaluated automatic metrics for measuring text generation performance (Callison-Burch et al., 2006; Stent et al., 2005; Belz and Reiter, 2006; Reiter, 2018; Liu et al., 2016; Kilickaya et al., 2017; Gatt and Krahmer, 2018).”
BackgroundUSR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Mehri, S., & Eskenazi, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.64 · Cited by 149
Do Massively Pretrained Language Models Make Better Storytellers?
See, A., Pappu, A., Saxena, R., Yerukola, A., & Manning, C. D. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1079 · Cited by 139
“Though the lack of human evaluation limits our ability to measure overall quality (Liu et al., 2016; Novikova et al., 2017; Hashimoto et al., 2019), we are able to produce an objectively defined, richly detailed and reproducible evaluation of the generated text.”
BackgroundOut of the BLEU: how should we assess quality of the Code Generation models?
Evtikhiev, M., Bogomolov, E., Sokolov, Y., & Bryksin, T. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111741 · Cited by 127
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
Scialom, T., Lamprier, S., Piwowarski, B., & Staiano, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1320 · Cited by 121
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113
“Standard token-matching metrics, such as BLEU (Papineni et al., 2002) or METEOR (Baner-jee and Lavie, 2005) in machine translation, or ROUGE (Lin, 2004) in summarization, were shown to have weak or no correlation with human judgements for dialogue (Liu et al., 2016; Lowe et al., 2017).”
BackgroundTowards a Better Metric for Evaluating Question Generation Systems
Nema, P., & Khapra, M. M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1429 · Cited by 106
The price of debiasing automatic metrics in natural language evalaution
Chaganty, A., Mussmann, S., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1060 · Cited by 96
“Similarly, Liu et al. (2016b) find correlations less than 0 .”
BackgroundMethodInfluentialCook, D. A., Overgaard, J., Pankratz, V. S., Del Fiol, G., & Aakre, C. A. (2025) · Journal of Medical Internet Research · doi:10.2196/68486 · Cited by 96
Importance of Search and Evaluation Strategies in Neural Dialogue Modeling
Kulikov, I., Miller, A., Cho, K., & Weston, J. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8609 · Cited by 83
Caccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J., & Charlin, L. (2018) · arXiv · doi:10.48550/arxiv.1811.02549 · Cited by 83
Self-citationA Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation
Juraska, J., Karagiannis, P., Bowden, K., & Walker, M. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1014 · Cited by 75
Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features
Rashkin, H., Reitter, D., Tomar, G. S., & Das, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.58 · Cited by 74
“While BLEU gives a general sense of the fluency, there are drawbacks to word-overlap metrics for evaluating open-ended generations like dialogue (Liu et al., 2016).”
BackgroundTowards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation
Pang, B., Nijkamp, E., Han, W., Zhou, L., Liu, Y., & Tu, K. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.333 · Cited by 70
“Indeed, Liu et al. (2016) conducts extensive empirical studies on these metrics (e.g., BLEU, METEOR, and ROUGE) to test their effectiveness on evaluating dialogue generation and find limited relation between these automatic metrics and human judgments.”
BackgroundLiang, Y., Meng, F., Zhang, Y., Chen, Y., Xu, J., & Zhou, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17575 · Cited by 63
Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models
Wang, Q., Fu, Y., Cao, Y., Wang, S., Tian, Z., & Ding, L. (2023) · arXiv · doi:10.48550/arxiv.2308.15022 · Cited by 55
Shen, J., Yang, B., Dudley, J. J., & Kristensson, P. O. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511145 · Cited by 48
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.745 · Cited by 47
“Yet manual evaluation is expensive, time-consuming, and not without its own quality and reliability issues (Hosking et al., 2023; Liu et al., 2016; Smith et al., 2022).”
ViLBERTScore: Evaluating Image Caption Using Vision-and-Language BERT
Lee, H., Yoon, S., Dernoncourt, F., Kim, D. S., Bui, T., & Jung, K. (2020) · Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems · doi:10.18653/v1/2020.eval4nlp-1.4 · Cited by 43
“These approaches bear limitations in dealing with the text’s diverse nature, similarly found in other text generation tasks (e.g., abstractive summarization and dialog) (Kryscinski et al., 2019; Liu et al., 2016).”
BackgroundEvalAI: Towards Better Evaluation Systems for AI Agents
Yadav, D., Jain, R., Agrawal, H., Chattopadhyay, P., Singh, T., Jain, A., Singh, S. B., Lee, S., & Batra, D. (2019) · arXiv · doi:10.48550/arxiv.1902.03570 · Cited by 42
Smith, E., Hsu, O., Qian, R., Roller, S., Boureau, Y.-L., & Weston, J. (2022) · Proceedings of the 4th Workshop on NLP for Conversational AI · doi:10.18653/v1/2022.nlp4convai-1.8 · Cited by 40
Recursively summarizing enables long-term dialogue memory in large language models
Wang, Q., Fu, Y., Cao, Y., Wang, S., Tian, Z., & Ding, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130193 · Cited by 55
A Distributional Approach to Controlled Text Generation
Khalifa, M., Elsahar, H., & Dymetman, M. (2020) · arXiv · doi:10.48550/arxiv.2012.11635 · Cited by 39
“…Controlled Text Generation When using such approaches, one needs to take care of not forgetting too much of the original LM policy (“degeneration”): Liu et al. (2016a) noted that such optimization may produce adversarial examples that improve the average reward without an actual increase in…”
BackgroundMethodInfluentialOpen-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
Roller, S., Boureau, Y.-L., Weston, J., Bordes, A., Dinan, E., Fan, A., Gunning, D., Ju, D., Li, M., Poff, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.12442 · Cited by 38
Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions
Luo, M., Warren, C. J., Cheng, L., Abdul-Muhsin, H. M., & Banerjee, I. (2024) · 2024 IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825307 · Cited by 36
“However, that is only an "average."”
Large Language Models for Automated Open-domain Scientific Hypotheses Discovery
Yang, Z., Du, X., Li, J., Zheng, J., Poria, S., & Cambria, E. (2023) · arXiv · doi:10.48550/arxiv.2309.02726 · Cited by 34
Correlating Automated and Human Evaluation of Code Documentation Generation Quality
Hu, X., Chen, Q., Wang, H., Xia, X., Lo, D., & Zimmermann, T. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3502853 · Cited by 33
Improvised Theatre Alongside Artificial Intelligences
Mathewson, K., & Mirowski, P. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v13i1.12926 · Cited by 31
CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models
Zhao, J., Fang, M., Shi, Z., Li, Y., Chen, L., & Pechenizkiy, M. (2023) · arXiv · doi:10.48550/arxiv.2305.11262 · Cited by 31
Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., & Hakkani-Tur, D. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8608 · Cited by 26
On the Generation of Medical Question-Answer Pairs
Shen, S., Li, Y., Du, N., Wu, X., Xie, Y., Ge, S., Yang, T., Wang, K., Liang, X., & Fan, W. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6410 · Cited by 25
Sequence to Sequence Modeling for User Simulation in Dialog Systems
Crook, P., & Marin, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-161 · Cited by 25
“[21] discuss the ineffectiveness of other unsupervised metrics, such as BLEU [22], METEOR [23], and ROUGE [24], in particular in the context of evaluating the dialog system responses which may or may not have strong surface form overlap with the ground truth.”
BackgroundD-Score: Holistic Dialogue Evaluation Without Reference
Zhang, C., Lee, G., D'Haro, L. F., & Li, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3074012 · Cited by 21
Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity
Shuster, K., Urbanek, J., Szlam, A., & Weston, J. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.182 · Cited by 20
Abu-Jeyyab, M., Alrosan, S., & Alkhawaldeh, I. (2023) · High Yield Medical Reviews · doi:10.59707/hymrfbya5348 · Cited by 17
Spurious Correlations in Reference-Free Evaluation of Text Generation
Durmus, E., Ladhak, F., & Hashimoto, T. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.102 · Cited by 16
CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models
Zhao, J., Fang, M., Shi, Z., Li, Y., Chen, L., & Pechenizkiy, M. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.757 · Cited by 14
Metadata
- Authors
- Liu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J.
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 2122-2132
- OpenAlex
- W2963903950
- Cited by
- 1,228
- References
- 47
- FWCI
- 80.59
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.