Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation
Open accessAbstract
Open-domain dialogue generation has gained increasing attention in Natural Language Processing.Its evaluation requires a holistic means.Human ratings are deemed as the gold standard.As human evaluation is inefficient and costly, an automated substitute is highly desirable.In this paper, we propose holistic evaluation metrics that capture different aspects of open-domain dialogues.Our metrics consist of (1) GPT-2 based context coherence between sentences in a dialogue, (2) GPT-2 based fluency in phrasing, (3) n-gram based diversity in responses to augmented queries, and (4) textual-entailment-inference based logical self-consistency.The empirical validity of our metrics is demonstrated by strong correlations with human judgments.We open source the code and relevant materials.1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Mehri and Eskenazi (2020) and Pang et al. (2020) suggested using large pretrained language models to develop reference-response-free metrics for dialogue evaluation.
“Recently, Mehri and Eskenazi (2020) and Pang et al. (2020) suggested to use large pretrained language models (Liu et al., 2019;Radford et al., 2019) to develop reference-response-free metrics for dialogue evaluation.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Pang et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Kilgarriff, A. (2000) · Language · doi:10.2307/417141 · Cited by 11,595 · see citing works
(2004) · Cited by 9,718 · see citing works
(2005) · Cited by 4,152 · see citing works
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,451 · see citing works
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
OpenNMT: Open-Source Toolkit for Neural Machine Translation
Klein, G., Kim, Y., Deng, Y., Sénellart, J., & Rush, A. M. (2017) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p17-4012 · Cited by 1,771 · see citing works
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Serban, I. V., Sordoni, A., Bengio, Y., Courville, A. C., & Pineau, J. (2016) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v30i1.9883 · Cited by 1,639 · see citing works
Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400 · see citing works
Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,262 · see citing works
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177 · see citing works
Neural Responding Machine for Short-Text Conversation
Shang, L., Lu, Z., & Li, H. (2015) · doi:10.3115/v1/p15-1152 · Cited by 1,009 · see citing works
Enriching the knowledge sources used in a maximum entropy part-of-speech tagger
Toutanova, K., & Manning, C. D. (2000) · doi:10.3115/1117794.1117802 · Cited by 947 · see citing works
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J., Gao, J., & Dolan, B. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1020 · Cited by 768 · see citing works
DailyDialog: A Manually Labelled Multi-turn Dialogue DatasetPreprint
Li, Y., Su, H., Shen, X., Li, W., Cao, Z., & Niu, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.03957 · Cited by 757 · see citing works
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
Serban, I. V., Sordoni, A., Lowe, R., Charlin, L., Pineau, J., Courville, A. C., & Bengio, Y. (2017) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v31i1.10983 · Cited by 734 · see citing works
Data-Driven Response Generation in Social Media
Ritter, A., Cherry, C. A., & Dolan, W. B. (2011) · Cited by 566 · see citing works
A Knowledge-Grounded Neural Conversation Model
Ghazvininejad, M., Brockett, C., Chang, M.-W., Dolan, B., Gao, J., Yih, W.-T., & Galley, M. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.11977 · Cited by 457 · see citing works
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational AgentsPreprint
Wolf, T., Sanh, V., Chaumond, J., & Delangue, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08149 · Cited by 435 · see citing works
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Lowe, R., Noseworthy, M., Serban, I. V., Angelard-Gontier, N., Bengio, Y., & Pineau, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1103 · Cited by 341 · see citing works
DailyDialog: A Manually Labelled Multi-Turn Dialogue Dataset
Li, Y. (2025) · TIB Data Manager · doi:10.57702/l9x4bl6j · Cited by 262 · see citing works
Dialogue Natural Language Inference
Welleck, S., Weston, J., Szlam, A., & Cho, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1363 · Cited by 249 · see citing works
Mou, L., Song, Y., Yan, R., Li, G., Zhang, L., & Jin, Z. G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.00970 · Cited by 214 · see citing works
RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
Tao, C., Mou, L., Zhao, D., & Yan, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11321 · Cited by 213 · see citing works
Learning to Write with Cooperative Discriminators
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., & Choi, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1152 · Cited by 192 · see citing works
Unifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161 · see citing works
Caccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J., & Charlin, L. (2020) · International Conference on Learning Representations · Cited by 103 · see citing works
Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings
Ghazarian, S., Wei, J. T.-Z., Galstyan, A., & Peng, N. (2019) · doi:10.18653/v1/w19-2310 · Cited by 84 · see citing works
Caccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J., & Charlin, L. (2018) · arXiv · doi:10.48550/arxiv.1811.02549 · Cited by 83 · see citing works
Xu, J., Ren, X., Lin, J., & Sun, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.01345 · Cited by 53 · see citing works
Dialog System Using Real-Time Crowdsourcing and Twitter Large-Scale Corpus
Bessho, F., Harada, T., & Kuniyoshi, Y. (2012) · Cited by 52 · see citing works
Su, H., Shen, X., Hu, P., Li, W., & Chen, Y. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12158 · Cited by 23 · see citing works
Flagged first, then by citations
4 of the 38 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- George A Miller. 1998. WordNet: An electronic lexical database. MIT press.
- Alan Turing. 1950. Computing machinery and intelligence-am turing. Mind, 59(236):433.
Cited by
Sorted by citations
Recent advances in deep learning based dialogue systems: a systematic survey
Ni, J., Young, T., Pandelea, V., Xue, F., & Cambria, E. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10248-8 · Cited by 279 · see citing works
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113 · see citing works
A Comprehensive Assessment of Dialog Evaluation Metrics
Yeh, Y.-T., Eskénazi, M., & Mehri, S. (2021) · doi:10.18653/v1/2021.eancs-1.3 · Cited by 71 · see citing works
DynaEval: Unifying Turn and Dialogue Level Evaluation
Zhang, C., Chen, Y., D’Haro, L. F., Zhang, Y., Friedrichs, T., Lee, G., & Li, H. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.441 · Cited by 52 · see citing works
Adversarial Attack and Defense of Structured Prediction Models
Han, W., Zhang, L., Jiang, Y., & Tu, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.182 · Cited by 33 · see citing works
Self-citationLee, U., Jeon, M., Lee, Y., Byun, G., Son, Y., Shin, J., Ko, H., & Kim, H. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100297 · Cited by 22 · see citing works
D-Score: Holistic Dialogue Evaluation Without Reference
Zhang, C., Lee, G., D'Haro, L. F., & Li, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3074012 · Cited by 21 · see citing works
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation
Ghazarian, S., Hedayatnia, B., Papangelis, A., Liu, Y., & Hakkani-Tür, D. Z. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.331 · Cited by 18 · see citing works
Self-citationYe, G., Zhao, H., Zhang, Z., & Jiang, Z. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104035 · Cited by 16 · see citing works
Dialogue-adaptive language model pre-training from quality estimation☆
Li, J., Zhang, Z., & Zhao, H. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.10.036 · Cited by 21 · see citing works
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation
Zhang, C., D’Haro, L. F., Zhang, Q., Friedrichs, T., & Li, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.220 · Cited by 14 · see citing works
Zhao, Y., Cheng, B. H., Huang, Y., & Wan, Z. (2023) · International Journal of Intelligent Systems · doi:10.1155/2023/9267487 · Cited by 14 · see citing works
PoE: A Panel of Experts for Generalized Automatic Dialogue Assessment
Zhang, C., D’Haro, L. F., Zhang, Q., Friedrichs, T., & Li, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3250825 · Cited by 12 · see citing works
A Review of Quality Assurance Research of Dialogue Systems
Li, X., Tao, C., Gao, J. Z., & Guo, H. (2022) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest55621.2022.00021 · Cited by 12 · see citing works
Weakly Supervised Turn-level Engagingness Evaluator for Dialogues
Jiang, S., Vakulenko, S., & de Rijke, M. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578319 · Cited by 10 · see citing works
A Semantic, Syntactic, and Context-Aware Natural Language Adversarial Example Generator
Asl, J. R., Rafiei, M. H., Alohaly, M., & Takabi, D. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3359817 · Cited by 12 · see citing works
Park, C., Jang, E., Yang, W., & Park, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.120 · Cited by 9 · see citing works
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks
Kreutzer, J., Riezler, S., & Lawrence, C. (2021) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2021.spnlp-1.4 · Cited by 10 · see citing works
Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator
Cheng, Q., Li, L., Quan, G., Gao, F., Mou, X., & Qiu, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.90 · Cited by 10 · see citing works
Towards Standard Criteria for human evaluation of Chatbots: A SurveyPreprint
Liang, H., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11197 · Cited by 19 · see citing works
Belainine, B., Sadat, F., & Boukadoum, M. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3151347 · Cited by 9 · see citing works
Approximating Online Human Evaluation of Social Chatbots with Prompting
Svikhnushina, E., & Pu, P. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.25 · Cited by 10 · see citing works
Neural Dialogue Generation Methods in Open Domain: A Survey
Sun, B., & Li, K. (2021) · Natural Language Processing Research · doi:10.2991/nlpr.d.210223.001 · Cited by 8 · see citing works
C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation
Ren, L., Sidhu, M., Zeng, Q., Reddy, R. G., Ji, H., & Zhai, C. (2023) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2023.dialdoc-1.9 · Cited by 6 · see citing works
ChatMatch: Evaluating Chatbots by Autonomous Chat Tournaments
Yang, R., Li, Z., Tang, H., & Zhu, K. Q. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.522 · Cited by 7 · see citing works
Jiang, Z., Ye, G., Rao, D., Wang, D., & Miao, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.762 · Cited by 7 · see citing works
Automatic Scoring for Translations Based on Language Models
Wu, D., Wang, M., & Li, X. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/2171206 · Cited by 6 · see citing works
Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
Jiang, H., Dai, B., Yang, M., Zhao, T., & Wei, W. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.589 · Cited by 6 · see citing works
Jurassic is (Almost) All You Need: Few-Shot Meaning-to-Text Generation for Open-Domain Dialogue
Reed, L., Li, C., Ramirez, A. M., Wu, L., & Walker, M. A. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-5538-9_7 · Cited by 5 · see citing works
Learning from Human Feedback: Challenges for Real-World Reinforcement Learning in NLP.Preprint
Kreutzer, J., Riezler, S., & Lawrence, C. (2020) · arXiv (Cornell University) · Cited by 5 · see citing works
Emily: Developing An Emotion-affective Open-Domain Chatbot with Knowledge Graph-based PersonaPreprint
Wang, W., Cai, X., Huang, C. H., Wang, H., Lu, H., Liu, X., & Peng, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08875 · Cited by 5 · see citing works
Automatic Coding of Contingency in Child-Caregiver ConversationsPreprint
Agrawal, A., Nikolaus, M., Favre, B., & Fourtassi, A. (2024) · doi:10.31234/osf.io/hwnms · Cited by 5 · see citing works
Explaining Dialogue Evaluation Metrics using Adversarial Behavioral Analysis
Khalid, B., & Lee, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.430 · Cited by 4 · see citing works
How About Kind of Generating Hedges using End-to-End Neural Models?
Abulimiti, A., Clavel, C., & Cassell, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.50 · Cited by 7 · see citing works
Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference
Zouhar, V., Dhuliawala, S., Zhou, W., Daheim, N., Kocmi, T., Jiang, Y. E., & Sachan, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.95 · Cited by 4 · see citing works
FluGCF: A Fluent Dialogue Generation Model With Coherent Concept Entity Flow
Zhao, Y., Cheng, B. H., Huang, Y., & Wan, Z. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3340610 · Cited by 4 · see citing works
xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark
Zhang, C., D’Haro, L. F., Tang, C., Shi, K., Tang, G., & Li, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.371 · Cited by 4 · see citing works
Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sources
Hwang, Y., Kim, Y., Bae, H., Lee, H., Bang, J., & Jung, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.545 · Cited by 6 · see citing works
FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows
Zhao, J., Li, Y., Du, W., Ji, Y., Yu, D., Lyu, M. R.-T., & Wang, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.715 · Cited by 4 · see citing works
Berlot-Attwell, I., & Rudzicz, F. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.14 · Cited by 4 · see citing works
Overview of the Tenth Dialog System Technology Challenge: DSTC10
Yoshino, K., Chen, Y.-N., Crook, P., Kottur, S., Li, J., Hedayatnia, B., Moon, S., Fei, Z., Li, Z., Zhang, J., et al. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3293030 · Cited by 3 · see citing works
Self-citationSemantic Ambiguity Detection in Sentence Classification using Task-Specific Embeddings
Kim, J. M., Lee, Y., Jung, S., & Choi, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.41 · Cited by 3 · see citing works
ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems
Ghazarian, S., Shao, Y., Han, R., Galstyan, A., & Peng, N. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.241 · Cited by 3 · see citing works
Yerukola, A., Zhou, X., Clark, E., & Sap, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.701 · Cited by 3 · see citing works
User Response and Sentiment Prediction for Automatic Dialogue EvaluationPreprint
Ghazarian, S., Hedayatnia, B., Papangelis, A., Liu, Y., & Hakkani-Tür, D. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08808 · Cited by 3 · see citing works
Self-citationIdentifying Untrustworthy Samples
Shen, L., Zhan, H., Shen, X., Chen, H., Zhao, X., & Zhu, X. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482352 · Cited by 2 · see citing works
Pu, J., Cheng, L., Fan, L., Lv, T., & Zhang, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.590 · Cited by 2 · see citing works
Dial-M: A Masking-based Framework for Dialogue Evaluation
Dey, S., & Desarkar, M. S. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.7 · Cited by 2 · see citing works
Kim, Y., Hwang, Y., Bae, H., Kang, T., & Jung, K. (2024) · IEEE Access · doi:10.1109/access.2024.3409377 · Cited by 2 · see citing works
DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimation
Park, C., Lee, S., Rim, D. J., & Choo, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.896 · Cited by 2 · see citing works
A Study on Automatic Open-Domain Dialogue Evaluation Metrics
Kim, M., & Kim, J. (2024) · International Conference on Consumer Electronics · doi:10.1109/icce59016.2024.10444444 · Cited by 2 · see citing works
CueBot: Cue-Controlled Response Generation for Assistive Interaction Usages
Kumar, S. H., Su, H., Manuvinakurike, R., Pinaroc, M., Prasad, S., Sahay, S., & Nachman, L. (2022) · Workshop on Speech and Language Processing for Assistive Technologies · doi:10.18653/v1/2022.slpat-1.9 · Cited by 1 · see citing works
Modeling What-to-ask and How-to-ask for Answer-unaware Conversational Question Generation
Long, X., Zou, B., Joty, S., Tai, T., Pan, L., Chen, N. F., & Aw, A. T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.603 · Cited by 3 · see citing works
MERCY: Multiple Response Ranking Concurrently in Realistic Open-Domain Conversational Systems
Ghazarian, S., Hedayatnia, B., Jin, D., Liu, S., Peng, N., Liu, Y., & Hakkani-Tür, D. Z. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.58 · Cited by 1 · see citing works
Self-citationAutomatic Evaluate Dialogue Appropriateness by Using Dialogue Act
Chen, B., Wang, Y., Liu, Z., & Guo, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.492 · Cited by 1 · see citing works
Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation ApproachPreprint
Jiang, H., Dai, B., Yang, M., Zhao, T., & Wei, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.10242 · Cited by 2 · see citing works
SHARP: Search-Based Adversarial Attack for Structured Prediction
Zhang, L., Jia, Z., Han, W., Zheng, Z., & Tu, K. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.71
Self-citationHAUSER: Towards Holistic and Automatic Evaluation of Simile Generation
He, Q., Zhang, Y., Liang, J., Huang, Y., Xiao, Y., & Chen, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.702 · Cited by 1 · see citing works
Correction of Errors in Preference Ratings from Automated Metrics for Text Generation
Deriu, J., von Däniken, P., Tuggener, D., & Cieliebak, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.404
KiProL: A Knowledge-Injected Prompt Learning Framework for Language Generation
Zhao, Y., Huang, Y., & Cheng, B. H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2266-2_6
Is Our Chatbot Telling Lies? Assessing Correctness of an Llm-Based Dutch Support ChatbotPreprint
Lassche, H., Overeem, M., & Rastogi, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5362625
Lee, U., Son, Y., Shin, J., Byun, G., Lee, Y., Koh, J., Jeon, M., & Kim, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98284-2_17
Is it Enough to Ask Questions? Dialogue Evaluation through Question Answering and Generation
Vilaça, L., & Viana, P. (2025) · doi:10.1145/3746274.3760399
An exhaustive evaluation method for open-domain LLM dialogue by constructing recursive CoT
Zhao, S., & Xie, Z. (2026) · Computer Speech & Language · doi:10.1016/j.csl.2026.101957
Brown, C. S., Chandler, C., Sawaya, S., & D’Mello, S. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29770-9_63
- Authors
- Pang, B., Nijkamp, E., Han, W., Zhou, L., Liu, Y., & Tu, K.
- Affiliations
- University of California, Los Angeles (US)
- National University of Singapore (SG)
- ShanghaiTech University (CN)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 3619-3629
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2990563493
- Cited by
- 70
- References
- 38
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 8 · 2023: 23 · 2022: 17 · 2021: 15 · 2020: 2
- FWCI
- 5.44
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Natural Science Foundation of China (61976139)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- open-domain dialogue generation, automatic evaluation metrics, holistic assessment, fluency, response diversity, natural language inference, GPT-2, human evaluation correlation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
