USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation

Attention Is All You Need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325

BLEU: a method for automatic evaluation of machine translation

Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515

Untitled work

(2004) · Cited by 9,718

HuggingFace's Transformers: State-of-the-art Natural Language Processing

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347

BERTScore: Evaluating Text Generation with BERT

Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865

Meteor Universal: Language Specific Translation Evaluation for Any Target Language

Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,377

CLIPScore: A Reference-free Evaluation Metric for Image Captioning

Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.595 · Cited by 1,232

G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment

Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948

G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment

Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948

Retrieval Augmentation Reduces Hallucination in Conversation

Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569

Recent advances in deep learning based dialogue systems: a systematic survey

Ni, J., Young, T., Pandelea, V., Xue, F., & Cambria, E. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10248-8 · Cited by 313

CLIPScore: A Reference-free Evaluation Metric for Image CaptioningPreprint

Hessel, J., Holtzman, A., Forbes, M., Bras, R. J. L., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08718 · Cited by 1,051

Retrieval Augmentation Reduces Hallucination in ConversationPreprint

Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07567 · Cited by 403

LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models

Lin, Y., & Chen, Y.-N. (2023) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2023.nlp4convai-1.5 · Cited by 225

Towards a Unified Multi-Dimensional Evaluator for Text Generation

Zhong, M., Liu, Y., Yin, D., Mao, Y., Jiao, Y., Liu, P., Zhu, C., Ji, H., & Han, J. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.131 · Cited by 117

Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering

Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113

Unsupervised Evaluation of Interactive Dialog with DialoGPT

Mehri, S., & Eskenazi, M. (2020) · Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2020.sigdial-1.28 · Cited by 111

Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study

Chen, Y., Wang, R., Jiang, H., Shi, S., & Xu, R. (2023) · doi:10.18653/v1/2023.findings-ijcnlp.32 · Cited by 142

Measuring Attribution in Natural Language Generation Models

Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00486 · Cited by 71

A Comprehensive Assessment of Dialog Evaluation Metrics

Yeh, Y.-T., Eskénazi, M., & Mehri, S. (2021) · doi:10.18653/v1/2021.eancs-1.3 · Cited by 82

Overview of the Ninth Dialog System Technology Challenge: DSTC9

Gunasekara, C., Kim, S., D’Haro, L. F., Rastogi, A., Chen, Y.-N., Eric, M., Hedayatnia, B., Gopalakrishnan, K., Liu, Y., Huang, C.-W., et al. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3426331 · Cited by 80

Deepfake Text Detection: Limitations and Opportunities

Pu, J., Sarwar, Z., Abdullah, S. M., Rehman, A., Kim, Y., Bhattacharya, P., Javed, M., & Viswanath, B. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179387 · Cited by 75

Achieving Reliable Human Assessment of Open-Domain Dialogue Systems

Ji, T., Graham, Y., Jones, G. J. F., Lyu, C., & Liu, Q. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.445 · Cited by 65

ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., & Liu, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.07201 · Cited by 725

DynaEval: Unifying Turn and Dialogue Level Evaluation

Zhang, C., Chen, Y., D’Haro, L. F., Zhang, Y., Friedrichs, T., Lee, G., & Li, H. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.441 · Cited by 62

Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation

Deng, M., Tan, B., Liu, Z., Xing, E. P., & Hu, Z. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.599 · Cited by 60

Meta-evaluation of Conversational Search Evaluation Metrics

Liu, Z., Zhou, K., & Wilson, M. L. (2021) · ACM Transactions on Information Systems · doi:10.1145/3445029 · Cited by 50

Learning towards conversational AI: A survey

Fu, T., Gao, S., Zhao, X., Wen, J.-R., & Yan, R. (2022) · AI Open · doi:10.1016/j.aiopen.2022.02.001 · Cited by 46

InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning

Gupta, P., Jiao, C., Yeh, Y.-T., Mehri, S., Eskénazi, M., & Bigham, J. P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.33 · Cited by 75

Multitask Learning for Crash Analysis: A Fine-Tuned LLM Framework Using Twitter Data

Jaradat, S., Nayak, R., Paz, A., Ashqar, H. I., & Elhenawy, M. M. Z. (2024) · Smart Cities · doi:10.3390/smartcities7050095 · Cited by 51

Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling

Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2021) · arXiv · doi:10.48550/arxiv.2107.03451 · Cited by 44

Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems

Phy, V., Zhao, Y., & Aizawa, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.368 · Cited by 50

Commonsense-Focused Dialogues for Response Generation: An Empirical Study

Zhou, P., Gopalakrishnan, K., Hedayatnia, B., Kim, S., Pujara, J., Ren, X., Liu, Y., & Hakkani-Tür, D. Z. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.13 · Cited by 40

Controlling Dialogue Generation with Semantic Exemplars

Gupta, P., Bigham, J. P., Tsvetkov, Y., & Pavel, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.240 · Cited by 42

On the Limitations of Reference-Free Evaluations of Generated Text

Deutsch, D., Dror, R., & Roth, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.753 · Cited by 45

Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems

Deriu, J., Tuggener, D., von Däniken, P., Campos, J. A., Rodrigo, Á., Belkacem, T., Soroa, A., Agirre, E., & Cieliebak, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.326 · Cited by 30

Shades of BLEU, Flavours of Success: The Case of MultiWOZ

Nekvinda, T., & Dušek, O. (2021) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2021.gem-1.4 · Cited by 29

Opportunities and Challenges in Neural Dialog Tutoring

Mačina, J., Daheim, N., Wang, L., Sinha, T., Kapur, M., Gurevych, I., & Sachan, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.173 · Cited by 37

Understanding and Predicting User Satisfaction with Conversational Recommender Systems

Siro, C., Aliannejadi, M., & de Rijke, M. (2023) · ACM Transactions on Information Systems · doi:10.1145/3624989 · Cited by 32

Principles and Guidelines for the Use of LLM Judges

Dietz, L., Zendel, O., Bailey, P., Clarke, C. L. A., Cotterill, E., Dalton, J., Hasibi, F., Sanderson, M., & Craswell, N. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744588 · Cited by 50

Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search

Joko, H., Chatterjee, S., Ramsay, A., de Vries, A. P., Dalton, J., & Hasibi, F. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657815 · Cited by 33

Unstructured Text Enhanced Open-Domain Dialogue System: A Systematic Survey

Ma, L., Li, M., Zhang, W., Li, J., & Liu, T. (2021) · ACM Transactions on Information Systems · doi:10.1145/3464377 · Cited by 24

Automatic Evaluation and Moderation of Open-domain Dialogue SystemsPreprint

Zhang, C., Sedoc, J., D’Haro, L. F., Banchs, R. E., & Rudnicky, A. I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02110 · Cited by 29

D-Score: Holistic Dialogue Evaluation Without Reference

Zhang, C., Lee, G., D'Haro, L. F., & Li, H. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3074012 · Cited by 21

Dialogue-adaptive language model pre-training from quality estimation☆

Li, J., Zhang, Z., & Zhao, H. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.10.036 · Cited by 26

Understanding User Satisfaction with Task-oriented Dialogue Systems

Siro, C., Aliannejadi, M., & de Rijke, M. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531798 · Cited by 20

A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators

Zhang, C., D’Haro, L. F., Chen, Y., Zhang, M., & Li, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29923 · Cited by 31

On the Blind Spots of Model-Based Evaluation Metrics for Text Generation

He, T., Zhang, J., Wang, T., Kumar, S., Cho, K., Glass, J. R., & Tsvetkov, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.674 · Cited by 37

Towards Standard Criteria for human evaluation of Chatbots: A SurveyPreprint

Liang, H., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11197 · Cited by 24

Ditch the Gold Standard: Re-evaluating Conversational Question Answering

Li, H., Gao, T., Goenka, M., & Chen, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.555 · Cited by 22

Assessing Dialogue Systems with Distribution Distances

Xiang, J., Liu, Y., Cai, D., Li, H., Lian, D., & Liu, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.193 · Cited by 23

Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation

Gupta, P., Tsvetkov, Y., & Bigham, J. P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.338 · Cited by 18

The Glass Ceiling of Automatic Evaluation in Natural Language Generation

Colombo, P., Peyrard, M., Noiry, N., West, R., & Piantanida, P. (2023) · doi:10.18653/v1/2023.findings-ijcnlp.16 · Cited by 19

UniDE: A multi-level and low-resource framework for automatic dialogue evaluation via LLM-based data augmentation and multitask learning

Ye, G., Zhao, H., Zhang, Z., & Jiang, Z. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104035 · Cited by 16

Better than Average: Paired Evaluation of NLP systems

Peyrard, M., Zhao, W., Eger, S., & West, R. (2021) · doi:10.18653/v1/2021.acl-long.179 · Cited by 27

MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation

Zhang, C., D’Haro, L. F., Friedrichs, T., & Li, H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21420 · Cited by 18

Deep Learning for Dialogue Systems: Chit-Chat and Beyond

Yan, R., Li, J., & Zhou, Y. (2022) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000083 · Cited by 17

Measuring Attribution in Natural Language Generation Models

Rashkin, H., Nikolaev, V. A., Lamm, M., Aroyo, L. M., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00490 · Cited by 16

Asking Clarification Questions to Handle Ambiguity in Open-Domain QA

Lee, D., Kim, S., Lee, M., Lee, H., Park, J., Lee, S., & Jung, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.772 · Cited by 32

Filtering Noisy Dialogue Corpora by Connectivity and Content Relatedness

Akama, R., Yokoi, S., Suzuki, J., & Inui, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.68 · Cited by 14

FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation

Zhang, C., D’Haro, L. F., Zhang, Q., Friedrichs, T., & Li, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.220 · Cited by 16

PoE: A Panel of Experts for Generalized Automatic Dialogue Assessment

Zhang, C., D’Haro, L. F., Zhang, Q., Friedrichs, T., & Li, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3250825 · Cited by 13

A Multi-Agent Chatbot Architecture for AI-Driven Language Learning

Aleedy, M., Atwell, E., & Meshoul, S. (2025) · Applied Sciences · doi:10.3390/app151910634 · Cited by 11

Weakly Supervised Turn-level Engagingness Evaluator for Dialogues

Jiang, S., Vakulenko, S., & de Rijke, M. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578319 · Cited by 11

Don’t Forget Your ABC’s: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems

Finch, S. E., Finch, J. D., & Choi, J. D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.839 · Cited by 20

NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist

Ni’mah, I., Fang, M., Menkovski, V., & Pechenizkiy, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.69 · Cited by 15

Generating Negative Samples by Manipulating Golden Responses for Unsupervised Learning of a Response Evaluation Model

Park, C., Jang, E., Yang, W., & Park, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.120 · Cited by 9

End-to-End Dialogue Generation Using a Single Encoder and a Decoder Cascade With a Multidimension Attention Mechanism

Belainine, B., Sadat, F., & Boukadoum, M. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3151347 · Cited by 10

MEEP: Is this Engaging? Prompting Large Language Models for Dialogue Evaluation in Multilingual Settings

Ferron, A., Shore, A., Mitra, E., & Agrawal, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.137 · Cited by 10

Neural Dialogue Generation Methods in Open Domain: A Survey

Sun, B., & Li, K. (2021) · Natural Language Processing Research · doi:10.2991/nlpr.d.210223.001 · Cited by 8

A Proposed S.C.O.R.E. Evaluation Framework for Large Language Models – Safety, Consensus & Context, Objectivity, Reproducibility and ExplainabilityPreprint

Tan, T. F., Elangovan, K., Ong, J. C. L., Lee, A., Shah, N. H., Sung, J. J. �., Wong, T. Y., Lan, X., Liu, N., Wang, H., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5029562 · Cited by 8

RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue

Shi, Z., Sun, W., Zhang, S., Zhang, Z., Ren, P., & Ren, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.719 · Cited by 7

Advancing Faithfulness of Large Language Models in Goal-Oriented Dialogue Question Answering

Sticha, A., Braunschweiler, N., Doddipatla, R., & Knill, K. (2024) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3640794.3665573 · Cited by 8

Probing the Robustness of Trained Metrics for Conversational Dialogue Systems

Deriu, J., Tuggener, D., von Däniken, P., & Cieliebak, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.85 · Cited by 7

ChatMatch: Evaluating Chatbots by Autonomous Chat Tournaments

Yang, R., Li, Z., Tang, H., & Zhu, K. Q. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.522 · Cited by 7

IM2: an Interpretable and Multi-category Integrated Metric Framework for Automatic Dialogue Evaluation

Jiang, Z., Ye, G., Rao, D., Wang, D., & Miao, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.762 · Cited by 7

Investigating the Impact of Pre-trained Language Models on Dialog Evaluation

Zhang, C., D’Haro, L. F., Chen, Y., Friedrichs, T., & Li, H. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-5538-9_21 · Cited by 7

Commonsense-Focused Dialogues for Response Generation: An Empirical StudyPreprint

Zhou, P., Gopalakrishnan, K., Hedayatnia, B., Kim, S., Pujara, J., Ren, X., Liu, Y., & Hakkani-Tür, D. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06427 · Cited by 9

C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation

Ren, L., Sidhu, M., Zeng, Q., Reddy, R. G., Ji, H., & Zhai, C. (2023) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2023.dialdoc-1.9 · Cited by 6

FEEL: A Framework for Evaluating Emotional Support Capability with Large Language Models

Zhang, H., Chen, Y., Wang, M., & Feng, S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5618-6_9 · Cited by 7

Bridging the Gap between Expert and Language Models: Concept-guided Chess Commentary Generation and Evaluation

Kim, J.-C., Goh, J., Hwang, I., Cho, J., & Ok, J. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.481 · Cited by 6

Language Model Augmented Relevance Score

Liu, R., Wei, J., & Vosoughi, S. (2021) · doi:10.18653/v1/2021.acl-long.521 · Cited by 10

Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sources

Hwang, Y., Kim, Y., Bae, H., Lee, H., Bang, J., & Jung, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.545 · Cited by 7

Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review

Wang, H., Guo, B., Zeng, Y., Chen, M., Ding, Y., Zhang, Y., Yao, L., & Yu, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3715098 · Cited by 7

Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentation

Gangal, V., Jhamtani, H., Hovy, E. H., & Berg-Kirkpatrick, T. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.357 · Cited by 7

Assessing Reference-Free Peer Evaluation for Machine Translation

Agrawal, S., Foster, G. F., Freitag, M., & Cherry, C. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.91 · Cited by 7

How to Evaluate Your Dialogue Models: A Review of ApproachesPreprint

Li, X., Wu, W., Qin, L., & Yin, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01369 · Cited by 6

Identifying Untrustworthy Samples

Shen, L., Zhan, H., Shen, X., Chen, H., Zhao, X., & Zhu, X. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482352 · Cited by 2

Large language models robustness against perturbation

Alahmari, S. S., Hall, L., Mouton, P. R., & Goldgof, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-29770-0 · Cited by 5

RetriEVAL: Evaluating Text Generation with Contextualized Lexical Match

Li, Z., Li, X., Tao, C., Feng, J., Shen, T., Xu, C., Wang, H., Zhao, D., & Ma, S. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703581 · Cited by 5

Empowering Indian Farmers with Multilingual AI: A Voice-Enabled ChatBot Using Dhenu2

Mehra, B., & Anitha, J. (2025) · International Conference on Cognitive Robotics and Intelligent Systems (ICC-ROBINS) · doi:10.1109/icc-robins64345.2025.11086147 · Cited by 5

The Illusion of Success: Procedure-Aware Evaluation is Critical for LLM Agents

Cao, H., Driouich, I., & Thomas, E. M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37682-4_29 · Cited by 33

Rethinking Response Evaluation from Interlocutor’s Eye for Open-Domain Dialogue Systems

Tsuta, Y., Yoshinaga, N., Sato, S., & Toyoda, M. (2023) · doi:10.18653/v1/2023.ijcnlp-srw.8 · Cited by 5

Legaleval-q: a benchmark for quality evaluation of LLM-generated Chinese legal text

Li, Y., & Wu, G. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02703-7 · Cited by 5

Dialogues Are Not Just Text: Modeling Cognition for Dialogue Coherence Evaluation

Li, X., Su, J., Yang, Y., Gao, Z., Duan, X., & Guan, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29819 · Cited by 5

Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks

Himmi, A., Irurozki, E., Noiry, N., Clémençon, S., & Colombo, P. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.688 · Cited by 9

Relevance in Dialogue: Is Less More? An Empirical Comparison of Existing Metrics, and a Novel Simple Metric

Berlot-Attwell, I., & Rudzicz, F. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.14 · Cited by 4

QualityAdapt: an Automatic Dialogue Quality Estimation Framework

Mendonca, J., Lavie, A., & Trancoso, I. M. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.9 · Cited by 4

Explaining Dialogue Evaluation Metrics using Adversarial Behavioral Analysis

Khalid, B., & Lee, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.430 · Cited by 5

xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark

Zhang, C., D’Haro, L. F., Tang, C., Shi, K., Tang, G., & Li, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.371 · Cited by 6

FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows

Zhao, J., Li, Y., Du, W., Ji, Y., Yu, D., Lyu, M. R.-T., & Wang, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.715 · Cited by 5

Goal Alignment in LLM-Based User Simulators for Conversational AI

Mehri, S., Yang, X., Kim, T., Tür, G., Mehri, S., & Hakkani-Tur, D. Z. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.687 · Cited by 3

Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs

Siro, C., Aliannejadi, M., & de Rijke, M. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657712 · Cited by 3

MP2D: An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphs

Hwang, Y., Kim, Y., Jang, Y., Bang, J., Bae, H., & Jung, K. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.979 · Cited by 3

Chatbot Evaluation Frameworks: From BLEU and F1 to Multi-Dimensional Real-World Benchmarks

GUDIPATI, S. K. (2025) · International Conference on Management Science and Computer Engineering (MSCE) · doi:10.1145/3760023.3760060 · Cited by 3

Unsupervised Evaluation of Interactive Dialog with DialoGPT

Mehri, S., & Eskenazi, M. (2020) · Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2020.sigdial-1.28 · Cited by 111

RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems

Tao, C., Mou, L., Zhao, D., & Yan, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11321 · Cited by 213

Learning an Unreferenced Metric for Online Dialogue Evaluation

Sinha, K., Parthasarathi, P., Wang, J., Lowe, R., Hamilton, W. L., & Pineau, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.220 · Cited by 73

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,377

Meta-Prompting Follow-Ups for Unsupervised Dialogue Evaluation Using Open-Source Large Language Models

Cimino, G., Li, C., Carenini, G., & Deufemia, V. (2026) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4i8vxn9qi57r