USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Open accessAbstract
The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research.Standard language generation metrics have been shown to be ineffective for evaluating dialog models.To this end, this paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog.USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog.USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42, systemlevel: 1.0) and PersonaChat (turn-level: 0.48 and system-level: 1.0).USR additionally produces interpretable measures for several desirable properties of dialog.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
USR is an open-domain dialogue dataset where the task is dialogue response generation, with outputs being continuations of dialogue contexts and expert evaluators providing ratings across six dimensions of dialogue quality.
“USR (Mehri and Eskenazi, 2020) is an opendomain dialogue dataset, where the task is dialogue response generation.The outputs are continuations of dialogue contexts, and expert evaluators provide ratings across six dimensions of dialogue quality.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Mehri and Eskenazi, 2020
Section 3 (Human Quality Annotation; page unavailable) describes responses to open-domain dialogue contexts rated by human annotators on six quality dimensions, and Section 3.4 identifies the annotators as dialogue researchers. But the Introduction and Conclusions identify USR as an evaluation metric, not the name of a dialogue dataset; the annotations were collected using Topical-Chat and PersonaChat. (LLM, conf=0.88)
Experimental — accuracy not yet measuredMehri and Eskenazi (2020) and Pang et al. (2020) suggested using large pretrained language models to develop reference-response-free metrics for dialogue evaluation.
“Recently, Mehri and Eskenazi (2020) and Pang et al. (2020) suggested to use large pretrained language models (Liu et al., 2019;Radford et al., 2019) to develop reference-response-free metrics for dialogue evaluation.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Mehri and Eskenazi (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
(2004) · Cited by 9,718 · see citing works
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293 · see citing works
Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,262 · see citing works
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177 · see citing works
Adversarial Learning for Neural Dialogue Generation
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., & Jurafsky, D. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1230 · Cited by 880 · see citing works
Zhao, T., Zhao, R., & Eskenazi, M. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1061 · Cited by 736 · see citing works
Liu, C., Lowe, R., Serban, I. V., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08023 · Cited by 603 · see citing works
Walker, M. A., Litman, D. J., Kamm, C., & Abella, A. (1997) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/976909.979652 · Cited by 548 · see citing works
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational AgentsPreprint
Wolf, T., Sanh, V., Chaumond, J., & Delangue, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08149 · Cited by 435 · see citing works
The Second Conversational Intelligence Challenge (ConvAI2)
Dinan, E., Logacheva, V., Malykh, V., Miller, A., Shuster, K., Urbanek, J., Kiela, D., Szlam, A. D., Serban, I. V., Lowe, R., et al. (2019) · The Springer series on challenges in machine learning · doi:10.1007/978-3-030-29135-8_7 · Cited by 362 · see citing works
The Second Conversational Intelligence Challenge (ConvAI2)
Dinan, E., Logacheva, V., Malykh, V., Miller, A., Shuster, K., Urbanek, J., Kiela, D., Szlam, A., Serban, I., Lowe, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1902.00098 · Cited by 355 · see citing works
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Lowe, R., Noseworthy, M., Serban, I. V., Angelard-Gontier, N., Bengio, Y., & Pineau, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1103 · Cited by 341 · see citing works
Survey on evaluation methods for dialogue systems
Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., & Cieliebak, M. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09866-x · Cited by 333 · see citing works
Towards Universal Paraphrastic Sentence EmbeddingsPreprint
Wieting, J., Bansal, M., Gimpel, K., & Livescu, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08198 · Cited by 324 · see citing works
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems
Lowe, R., Pow, N., Serban, I. V., & Pineau, J. (2015) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w15-4640 · Cited by 282 · see citing works
Sharma, S., Asri, L. E., Schulz, H., & Zumer, J. (2017) · arXiv.org · doi:10.48550/arxiv.1706.09799 · Cited by 250 · see citing works
Adversarial Learning for Neural Dialogue GenerationPreprint
Li, J., Monroe, W., Shi, T., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.06547 · Cited by 211 · see citing works
Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations
Gopalakrishnan, K., Hedayatnia, B., Chen, Q., Gottardi, A., Kwatra, S., Venkatesh, A., Gabriel, R., & Hakkani-Tür, D. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-3079 · Cited by 207 · see citing works
PARADISE: A Framework for Evaluating Spoken Dialogue AgentsPreprint
Walker, M. A., Litman, D. J., Kamm, C., & Abella, A. (1997) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9704004 · Cited by 175 · see citing works
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · arXiv · doi:10.48550/arxiv.1801.07243 · Cited by 165 · see citing works
Zhao, T., Zhao, R., & Eskénazi, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.10960 · Cited by 164 · see citing works
Rus, V., & Lintean, M. C. (2012) · Cited by 117 · see citing works
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
Galley, M., Brockett, C., Sordoni, A., Ji, Y., Auli, M., Quirk, C., Mitchell, M., Gao, J., & Dolan, B. (2015) · doi:10.3115/v1/p15-2073 · Cited by 101 · see citing works
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse TargetsPreprint
Galley, M., Brockett, C., Sordoni, A., Ji, Y., Auli, M., Quirk, C., Mitchell, M., Gao, J., & Dolan, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06863 · Cited by 90 · see citing works
Pretraining Methods for Dialog Context Representation Learning
Mehri, S., Razumovskaia, E., Zhao, T., & Eskénazi, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1373 · Cited by 79 · see citing works
Self-citationInvestigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References
Gupta, P., Mehri, S., Zhao, T., Pavel, A., Eskénazi, M., & Bigham, J. P. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5944 · Cited by 72 · see citing works
Self-citationOn Evaluating and Comparing Open Domain Dialog SystemsPreprint
Venkatesh, A., Khatri, C., Ram, A., Guo, F., Gabriel, R., Nagar, A., Prasad, R., Cheng, M., Hedayatnia, B., Metallinou, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.03625 · Cited by 34 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Survey on Evaluation Methods for Dialogue Systems
Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., & Cieliebak, M. (2019) · arXiv · doi:10.48550/arxiv.1905.04071
Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References
Gupta, P., Mehri, S., Zhao, T., Pavel, A., Eskenazi, M., & Bigham, J. P. (2019) · arXiv · doi:10.48550/arxiv.1907.10568
Self-citationThe Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems
Lowe, R., Pow, N., Serban, I., & Pineau, J. (2015) · arXiv · doi:10.48550/arxiv.1506.08909
Pretraining Methods for Dialog Context Representation Learning
Mehri, S., Razumovskaia, E., Zhao, T., & Eskenazi, M. (2019) · arXiv · doi:10.48550/arxiv.1906.00414
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Gabriel Forgues, Joelle Pineau, Jean-Marie Larchevêque, and Réal Tremblay. 2014. Boot- strapping dialog systems with word embeddings. In Nips, modern machine learning and natural language processing workshop, volume 2.
- Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and William B. Dolan. 2015. A neural network approach to context- sensitive generation of conversational responses. In HLT-NAACL.
Cited by
Sorted by citations
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.595 · Cited by 1,232 · see citing works
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948 · see citing works
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948 · see citing works
Retrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Recent advances in deep learning based dialogue systems: a systematic survey
Ni, J., Young, T., Pandelea, V., Xue, F., & Cambria, E. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10248-8 · Cited by 279 · see citing works
CLIPScore: A Reference-free Evaluation Metric for Image CaptioningPreprint
Hessel, J., Holtzman, A., Forbes, M., Bras, R. J. L., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08718 · Cited by 174 · see citing works
Retrieval Augmentation Reduces Hallucination in ConversationPreprint
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07567 · Cited by 172 · see citing works
Lin, Y., & Chen, Y.-N. (2023) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2023.nlp4convai-1.5 · Cited by 140 · see citing works
Towards a Unified Multi-Dimensional Evaluator for Text Generation
Zhong, M., Liu, Y., Yin, D., Mao, Y., Jiao, Y., Liu, P., Zhu, C., Ji, H., & Han, J. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.131 · Cited by 117 · see citing works
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113 · see citing works
Unsupervised Evaluation of Interactive Dialog with DialoGPT
Mehri, S., & Eskenazi, M. (2020) · Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2020.sigdial-1.28 · Cited by 111 · see citing works
Self-citationChen, Y., Wang, R., Jiang, H., Shi, S., & Xu, R. (2023) · doi:10.18653/v1/2023.findings-ijcnlp.32 · Cited by 93 · see citing works
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00486 · Cited by 71 · see citing works
A Comprehensive Assessment of Dialog Evaluation Metrics
Yeh, Y.-T., Eskénazi, M., & Mehri, S. (2021) · doi:10.18653/v1/2021.eancs-1.3 · Cited by 71 · see citing works
Self-citationOverview of the Ninth Dialog System Technology Challenge: DSTC9
Gunasekara, C., Kim, S., D’Haro, L. F., Rastogi, A., Chen, Y.-N., Eric, M., Hedayatnia, B., Gopalakrishnan, K., Liu, Y., Huang, C.-W., et al. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3426331 · Cited by 65 · see citing works
Self-citationDeepfake Text Detection: Limitations and Opportunities
Pu, J., Sarwar, Z., Abdullah, S. M., Rehman, A., Kim, Y., Bhattacharya, P., Javed, M., & Viswanath, B. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179387 · Cited by 62 · see citing works
Achieving Reliable Human Assessment of Open-Domain Dialogue Systems
Ji, T., Graham, Y., Jones, G. J. F., Lyu, C., & Liu, Q. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.445 · Cited by 60 · see citing works
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., & Liu, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.07201 · Cited by 55 · see citing works
“We then take the human annotation results from Mehri & Eskenazi (2020) where they carry out the annotations on 60 dialogue contexts with each response generated by 6 different systems.”
MethodDynaEval: Unifying Turn and Dialogue Level Evaluation
Zhang, C., Chen, Y., D’Haro, L. F., Zhang, Y., Friedrichs, T., Lee, G., & Li, H. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.441 · Cited by 52 · see citing works
Deng, M., Tan, B., Liu, Z., Xing, E. P., & Hu, Z. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.599 · Cited by 49 · see citing works
Meta-evaluation of Conversational Search Evaluation Metrics
Liu, Z., Zhou, K., & Wilson, M. L. (2021) · ACM Transactions on Information Systems · doi:10.1145/3445029 · Cited by 48 · see citing works
Learning towards conversational AI: A survey
Fu, T., Gao, S., Zhao, X., Wen, J.-R., & Yan, R. (2022) · AI Open · doi:10.1016/j.aiopen.2022.02.001 · Cited by 46 · see citing works
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning
Gupta, P., Jiao, C., Yeh, Y.-T., Mehri, S., Eskénazi, M., & Bigham, J. P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.33 · Cited by 46 · see citing works
Self-citationMultitask Learning for Crash Analysis: A Fine-Tuned LLM Framework Using Twitter Data
Jaradat, S., Nayak, R., Paz, A., Ashqar, H. I., & Elhenawy, M. M. Z. (2024) · Smart Cities · doi:10.3390/smartcities7050095 · Cited by 45 · see citing works
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2021) · arXiv · doi:10.48550/arxiv.2107.03451 · Cited by 44 · see citing works
Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems
Phy, V., Zhao, Y., & Aizawa, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.368 · Cited by 39 · see citing works
Commonsense-Focused Dialogues for Response Generation: An Empirical Study
Zhou, P., Gopalakrishnan, K., Hedayatnia, B., Kim, S., Pujara, J., Ren, X., Liu, Y., & Hakkani-Tür, D. Z. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.13 · Cited by 38 · see citing works
Controlling Dialogue Generation with Semantic Exemplars
Gupta, P., Bigham, J. P., Tsvetkov, Y., & Pavel, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.240 · Cited by 36 · see citing works
On the Limitations of Reference-Free Evaluations of Generated Text
Deutsch, D., Dror, R., & Roth, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.753 · Cited by 32 · see citing works
Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems
Deriu, J., Tuggener, D., von Däniken, P., Campos, J. A., Rodrigo, Á., Belkacem, T., Soroa, A., Agirre, E., & Cieliebak, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.326 · Cited by 30 · see citing works
Shades of BLEU, Flavours of Success: The Case of MultiWOZ
Nekvinda, T., & Dušek, O. (2021) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2021.gem-1.4 · Cited by 29 · see citing works
Opportunities and Challenges in Neural Dialog Tutoring
Mačina, J., Daheim, N., Wang, L., Sinha, T., Kapur, M., Gurevych, I., & Sachan, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.173 · Cited by 29 · see citing works
Understanding and Predicting User Satisfaction with Conversational Recommender Systems
Siro, C., Aliannejadi, M., & de Rijke, M. (2023) · ACM Transactions on Information Systems · doi:10.1145/3624989 · Cited by 28 · see citing works
Principles and Guidelines for the Use of LLM Judges
Dietz, L., Zendel, O., Bailey, P., Clarke, C. L. A., Cotterill, E., Dalton, J., Hasibi, F., Sanderson, M., & Craswell, N. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744588 · Cited by 28 · see citing works
Joko, H., Chatterjee, S., Ramsay, A., de Vries, A. P., Dalton, J., & Hasibi, F. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657815 · Cited by 24 · see citing works
Unstructured Text Enhanced Open-Domain Dialogue System: A Systematic Survey
Ma, L., Li, M., Zhang, W., Li, J., & Liu, T. (2021) · ACM Transactions on Information Systems · doi:10.1145/3464377 · Cited by 23 · see citing works
Automatic Evaluation and Moderation of Open-domain Dialogue SystemsPreprint
Zhang, C., Sedoc, J., D’Haro, L. F., Banchs, R. E., & Rudnicky, A. I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02110 · Cited by 23 · see citing works
D-Score: Holistic Dialogue Evaluation Without Reference
Zhang, C., Lee, G., D'Haro, L. F., & Li, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3074012 · Cited by 21 · see citing works
Dialogue-adaptive language model pre-training from quality estimation☆
Li, J., Zhang, Z., & Zhao, H. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.10.036 · Cited by 21 · see citing works
Understanding User Satisfaction with Task-oriented Dialogue Systems
Siro, C., Aliannejadi, M., & de Rijke, M. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531798 · Cited by 20 · see citing works
Zhang, C., D’Haro, L. F., Chen, Y., Zhang, M., & Li, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29923 · Cited by 20 · see citing works
On the Blind Spots of Model-Based Evaluation Metrics for Text Generation
He, T., Zhang, J., Wang, T., Kumar, S., Cho, K., Glass, J. R., & Tsvetkov, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.674 · Cited by 19 · see citing works
Towards Standard Criteria for human evaluation of Chatbots: A SurveyPreprint
Liang, H., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11197 · Cited by 19 · see citing works
Ditch the Gold Standard: Re-evaluating Conversational Question Answering
Li, H., Gao, T., Goenka, M., & Chen, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.555 · Cited by 18 · see citing works
Assessing Dialogue Systems with Distribution Distances
Xiang, J., Liu, Y., Cai, D., Li, H., Lian, D., & Liu, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.193 · Cited by 17 · see citing works
Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation
Gupta, P., Tsvetkov, Y., & Bigham, J. P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.338 · Cited by 16 · see citing works
The Glass Ceiling of Automatic Evaluation in Natural Language Generation
Colombo, P., Peyrard, M., Noiry, N., West, R., & Piantanida, P. (2023) · doi:10.18653/v1/2023.findings-ijcnlp.16 · Cited by 16 · see citing works
Ye, G., Zhao, H., Zhang, Z., & Jiang, Z. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104035 · Cited by 16 · see citing works
Better than Average: Paired Evaluation of NLP systems
Peyrard, M., Zhao, W., Eger, S., & West, R. (2021) · doi:10.18653/v1/2021.acl-long.179 · Cited by 16 · see citing works
MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation
Zhang, C., D’Haro, L. F., Friedrichs, T., & Li, H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21420 · Cited by 16 · see citing works
Deep Learning for Dialogue Systems: Chit-Chat and Beyond
Yan, R., Li, J., & Zhou, Y. (2022) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000083 · Cited by 16 · see citing works
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V. A., Lamm, M., Aroyo, L. M., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00490 · Cited by 16 · see citing works
Asking Clarification Questions to Handle Ambiguity in Open-Domain QA
Lee, D., Kim, S., Lee, M., Lee, H., Park, J., Lee, S., & Jung, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.772 · Cited by 15 · see citing works
Filtering Noisy Dialogue Corpora by Connectivity and Content Relatedness
Akama, R., Yokoi, S., Suzuki, J., & Inui, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.68 · Cited by 14 · see citing works
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation
Zhang, C., D’Haro, L. F., Zhang, Q., Friedrichs, T., & Li, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.220 · Cited by 14 · see citing works
PoE: A Panel of Experts for Generalized Automatic Dialogue Assessment
Zhang, C., D’Haro, L. F., Zhang, Q., Friedrichs, T., & Li, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3250825 · Cited by 12 · see citing works
A Multi-Agent Chatbot Architecture for AI-Driven Language Learning
Aleedy, M., Atwell, E., & Meshoul, S. (2025) · Applied Sciences · doi:10.3390/app151910634 · Cited by 11 · see citing works
Weakly Supervised Turn-level Engagingness Evaluator for Dialogues
Jiang, S., Vakulenko, S., & de Rijke, M. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578319 · Cited by 10 · see citing works
Don’t Forget Your ABC’s: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems
Finch, S. E., Finch, J. D., & Choi, J. D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.839 · Cited by 10 · see citing works
NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist
Ni’mah, I., Fang, M., Menkovski, V., & Pechenizkiy, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.69 · Cited by 10 · see citing works
Park, C., Jang, E., Yang, W., & Park, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.120 · Cited by 9 · see citing works
Belainine, B., Sadat, F., & Boukadoum, M. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3151347 · Cited by 9 · see citing works
Ferron, A., Shore, A., Mitra, E., & Agrawal, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.137 · Cited by 8 · see citing works
Neural Dialogue Generation Methods in Open Domain: A Survey
Sun, B., & Li, K. (2021) · Natural Language Processing Research · doi:10.2991/nlpr.d.210223.001 · Cited by 8 · see citing works
Tan, T. F., Elangovan, K., Ong, J. C. L., Lee, A., Shah, N. H., Sung, J. J. �., Wong, T. Y., Lan, X., Liu, N., Wang, H., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5029562 · Cited by 8 · see citing works
RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue
Shi, Z., Sun, W., Zhang, S., Zhang, Z., Ren, P., & Ren, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.719 · Cited by 7 · see citing works
Advancing Faithfulness of Large Language Models in Goal-Oriented Dialogue Question Answering
Sticha, A., Braunschweiler, N., Doddipatla, R., & Knill, K. (2024) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3640794.3665573 · Cited by 7 · see citing works
Probing the Robustness of Trained Metrics for Conversational Dialogue Systems
Deriu, J., Tuggener, D., von Däniken, P., & Cieliebak, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.85 · Cited by 7 · see citing works
ChatMatch: Evaluating Chatbots by Autonomous Chat Tournaments
Yang, R., Li, Z., Tang, H., & Zhu, K. Q. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.522 · Cited by 7 · see citing works
Jiang, Z., Ye, G., Rao, D., Wang, D., & Miao, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.762 · Cited by 7 · see citing works
Investigating the Impact of Pre-trained Language Models on Dialog Evaluation
Zhang, C., D’Haro, L. F., Chen, Y., Friedrichs, T., & Li, H. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-5538-9_21 · Cited by 7 · see citing works
Commonsense-Focused Dialogues for Response Generation: An Empirical StudyPreprint
Zhou, P., Gopalakrishnan, K., Hedayatnia, B., Kim, S., Pujara, J., Ren, X., Liu, Y., & Hakkani-Tür, D. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06427 · Cited by 7 · see citing works
C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation
Ren, L., Sidhu, M., Zeng, Q., Reddy, R. G., Ji, H., & Zhai, C. (2023) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2023.dialdoc-1.9 · Cited by 6 · see citing works
FEEL: A Framework for Evaluating Emotional Support Capability with Large Language Models
Zhang, H., Chen, Y., Wang, M., & Feng, S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5618-6_9 · Cited by 6 · see citing works
Kim, J.-C., Goh, J., Hwang, I., Cho, J., & Ok, J. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.481 · Cited by 6 · see citing works
Language Model Augmented Relevance Score
Liu, R., Wei, J., & Vosoughi, S. (2021) · doi:10.18653/v1/2021.acl-long.521 · Cited by 6 · see citing works
Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sources
Hwang, Y., Kim, Y., Bae, H., Lee, H., Bang, J., & Jung, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.545 · Cited by 6 · see citing works
Wang, H., Guo, B., Zeng, Y., Chen, M., Ding, Y., Zhang, Y., Yao, L., & Yu, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3715098 · Cited by 6 · see citing works
Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentation
Gangal, V., Jhamtani, H., Hovy, E. H., & Berg-Kirkpatrick, T. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.357 · Cited by 6 · see citing works
Assessing Reference-Free Peer Evaluation for Machine Translation
Agrawal, S., Foster, G. F., Freitag, M., & Cherry, C. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.91 · Cited by 6 · see citing works
How to Evaluate Your Dialogue Models: A Review of ApproachesPreprint
Li, X., Wu, W., Qin, L., & Yin, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01369 · Cited by 6 · see citing works
Identifying Untrustworthy Samples
Shen, L., Zhan, H., Shen, X., Chen, H., Zhao, X., & Zhu, X. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482352 · Cited by 2 · see citing works
Large language models robustness against perturbation
Alahmari, S. S., Hall, L., Mouton, P. R., & Goldgof, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-29770-0 · Cited by 5 · see citing works
RetriEVAL: Evaluating Text Generation with Contextualized Lexical Match
Li, Z., Li, X., Tao, C., Feng, J., Shen, T., Xu, C., Wang, H., Zhao, D., & Ma, S. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703581 · Cited by 5 · see citing works
Empowering Indian Farmers with Multilingual AI: A Voice-Enabled ChatBot Using Dhenu2
Mehra, B., & Anitha, J. (2025) · International Conference on Cognitive Robotics and Intelligent Systems (ICC-ROBINS) · doi:10.1109/icc-robins64345.2025.11086147 · Cited by 5 · see citing works
The Illusion of Success: Procedure-Aware Evaluation is Critical for LLM Agents
Cao, H., Driouich, I., & Thomas, E. M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37682-4_29 · Cited by 5 · see citing works
Rethinking Response Evaluation from Interlocutor’s Eye for Open-Domain Dialogue Systems
Tsuta, Y., Yoshinaga, N., Sato, S., & Toyoda, M. (2023) · doi:10.18653/v1/2023.ijcnlp-srw.8 · Cited by 5 · see citing works
Legaleval-q: a benchmark for quality evaluation of LLM-generated Chinese legal text
Li, Y., & Wu, G. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02703-7 · Cited by 5 · see citing works
Dialogues Are Not Just Text: Modeling Cognition for Dialogue Coherence Evaluation
Li, X., Su, J., Yang, Y., Gao, Z., Duan, X., & Guan, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29819 · Cited by 5 · see citing works
Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks
Himmi, A., Irurozki, E., Noiry, N., Clémençon, S., & Colombo, P. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.688 · Cited by 5 · see citing works
Berlot-Attwell, I., & Rudzicz, F. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.14 · Cited by 4 · see citing works
QualityAdapt: an Automatic Dialogue Quality Estimation Framework
Mendonca, J., Lavie, A., & Trancoso, I. M. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.9 · Cited by 4 · see citing works
Explaining Dialogue Evaluation Metrics using Adversarial Behavioral Analysis
Khalid, B., & Lee, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.430 · Cited by 4 · see citing works
xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark
Zhang, C., D’Haro, L. F., Tang, C., Shi, K., Tang, G., & Li, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.371 · Cited by 4 · see citing works
FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows
Zhao, J., Li, Y., Du, W., Ji, Y., Yu, D., Lyu, M. R.-T., & Wang, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.715 · Cited by 4 · see citing works
Goal Alignment in LLM-Based User Simulators for Conversational AI
Mehri, S., Yang, X., Kim, T., Tür, G., Mehri, S., & Hakkani-Tur, D. Z. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.687 · Cited by 3 · see citing works
Self-citationRethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs
Siro, C., Aliannejadi, M., & de Rijke, M. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657712 · Cited by 3 · see citing works
MP2D: An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphs
Hwang, Y., Kim, Y., Jang, Y., Bang, J., Bae, H., & Jung, K. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.979 · Cited by 3 · see citing works
Chatbot Evaluation Frameworks: From BLEU and F1 to Multi-Dimensional Real-World Benchmarks
GUDIPATI, S. K. (2025) · International Conference on Management Science and Computer Engineering (MSCE) · doi:10.1145/3760023.3760060 · Cited by 3 · see citing works
- Authors
- Mehri, S., & Eskenazi, M.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 681-707
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3023366413
- Cited by
- 149
- References
- 27
- Citations by year
- 2026: 15 · 2025: 12 · 2024: 16 · 2023: 40 · 2022: 26 · 2021: 37 · 2020: 3
- FWCI
- 10.12
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- dialogue generation, automatic evaluation metrics, reference-free evaluation, unsupervised learning, open-domain dialogue systems, dialogue quality, human judgment correlation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
