The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Open accessAbstract
The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research.Standard language generation metrics have been shown to be ineffective for evaluating dialog models.To this end, this paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog.USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog.USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42, systemlevel: 1.0) and PersonaChat (turn-level: 0.48 and system-level: 1.0).USR additionally produces interpretable measures for several desirable properties of dialog.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“USR (Mehri and Eskenazi, 2020) is an opendomain dialogue dataset, where the task is dialogue response generation.The outputs are continuations of dialogue contexts, and expert evaluators provide ratings across six dimensions of dialogue quality.”
From Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026) · cites this paper as Mehri and Eskenazi, 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recently, Mehri and Eskenazi (2020) and Pang et al. (2020) suggested to use large pretrained language models (Liu et al., 2019;Radford et al., 2019) to develop reference-response-free metrics for dialogue evaluation.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Mehri and Eskenazi (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
(2004) · Cited by 9,718
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108
Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,377
Flagged first, then by citations
Research works printed but not linked: 2
- Gabriel Forgues, Joelle Pineau, Jean-Marie Larchevêque, and Réal Tremblay. 2014. Boot- strapping dialog systems with word embeddings. In Nips, modern machine learning and natural language processing workshop, volume 2.
- Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and William B. Dolan. 2015. A neural network approach to context- sensitive generation of conversational responses. In HLT-NAACL.
Cited by
Sorted by citations
Items 101–165 of 165
Considerations on creating conversational agents for multiple environments and users
Cebrián, J., Martínez, R., Rodríguez, N., & D’Haro, L. F. (2021) · AI Magazine · doi:10.1609/aimag.v42i2.7484 · Cited by 4
On the Effectiveness of Automated Metrics for Text Generation Systems
von Däniken, P., Deriu, J., Tuggener, D., & Cieliebak, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.108 · Cited by 3
Dialogue Evaluation with Offline Reinforcement Learning
Lubis, N., Geishauser, C., Lin, H.-C., van Niekerk, C., Heck, M. C., Feng, S., & Gašić, M. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.46 · Cited by 3
Raposo, G., Coheur, L., & Martins, B. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.37 · Cited by 4
Measuring the ‘I don’t know’ Problem through the Lens of Gricean Quantity
Khayrallah, H., & Sedoc, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.450 · Cited by 4
FFAEval: Evaluating Dialogue System via Free-For-All Ranking
Ma, Z., Yao, Z., Zhang, J., Yu, J., Zhang, X., Li, J., & Tang, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1049 · Cited by 2
Pu, J., Cheng, L., Fan, L., Lv, T., & Zhang, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.590 · Cited by 2
KPIs for AI Agents and Generative AI: A Rigorous Framework for Evaluation and Accountability
Sunkara, V. L. B. (2024) · International Journal of Scientific Research and Modern Technology. · doi:10.38124/ijsrmt.v3i4.572 · Cited by 2
LMUNIT: Fine-grained Evaluation with Natural Language Unit Tests
Saad-Falcon, J., Vivek, R., Berrios, W., Naik, N. S., Franklin, M., Vidgen, B., Singh, A., Kiela, D., & Mehri, S. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.176 · Cited by 6
Self-citationDial-M: A Masking-based Framework for Dialogue Evaluation
Dey, S., & Desarkar, M. S. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.7 · Cited by 3
Towards Credible Human Evaluation of Open-Domain Dialog Systems Using Interactive Setup
Liu, S., Lange, P. L., Hedayatnia, B., Papangelis, A., Jin, D., Wirth, A., Liu, Y., & Hakkani-Tür, D. Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26557 · Cited by 2
Evaluation of Response Generation Models: Shouldn’t It Be Shareable and Replicable?
Mousavi, S. M., Roccabruna, G., Lorandi, M., Caldarella, S., & Riccardi, G. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.12 · Cited by 2
DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering
Ke, P., Huang, F., Mi, F., Wang, Y., Liu, Q., Zhu, X., & Huang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.539 · Cited by 2
Towards Multilingual Automatic Open-Domain Dialogue Evaluation
Mendonca, J., Lavie, A., & Trancoso, I. M. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.11 · Cited by 2
Kim, Y., Hwang, Y., Bae, H., Kang, T., & Jung, K. (2024) · IEEE Access · doi:10.1109/access.2024.3409377 · Cited by 3
DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimation
Park, C., Lee, S., Rim, D. J., & Choo, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.896 · Cited by 5
Developing and Evaluating a Model-Based Metric for Legal Question Answering Systems
Bakır, D., Yildiz, B., & Aktaş, M. S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386689 · Cited by 2
A Study on Automatic Open-Domain Dialogue Evaluation Metrics
Kim, M., & Kim, J. (2024) · International Conference on Consumer Electronics · doi:10.1109/icce59016.2024.10444444 · Cited by 2
Better than Random: Reliable NLG Human Evaluation with Constrained Active Sampling
Ruan, J., Pu, X., Gao, M., Wan, X., & Zhu, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29857 · Cited by 4
Data Augmentation for Conversational AI
Soudani, H., Petcu, R., Kanoulas, E., & Hasibi, F. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3641238 · Cited by 2
Gao, J. Y., Bi, W., Xu, R., & Shi, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.220 · Cited by 2
Evaluating Large Language Models for Conversational AI Agents
Kohli, G. S. (2025) · Journal of Computational Analysis and Applications · doi:10.48047/jocaaa.2025.34.10.10 · Cited by 1
Towards Effective Automatic Evaluation of Generated Reflections for Motivational Interviewing
Wu, Z., Helaoui, R., Recupero, D. R., & Riboni, D. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3610661.3616127 · Cited by 2
Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluation
Zheng, S., Zhang, X., de Melo, G., Wang, X., & Wang, L. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34803 · Cited by 1
Automatic Generation of Conversational Skills from Dialog Datasets
Kapelyushnik, D., Baymurzina, D., Кузнецов, Д., & Burtsev, M. (2022) · Studies in computational intelligence · doi:10.1007/978-3-031-19032-2_4 · Cited by 1
MERCY: Multiple Response Ranking Concurrently in Realistic Open-Domain Conversational Systems
Ghazarian, S., Hedayatnia, B., Jin, D., Liu, S., Peng, N., Liu, Y., & Hakkani-Tür, D. Z. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.58 · Cited by 1
Automatic Evaluate Dialogue Appropriateness by Using Dialogue Act
Chen, B., Wang, Y., Liu, Z., & Guo, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.492 · Cited by 4
WeChat AI's Submission for DSTC9 Interactive Dialogue Evaluation TrackPreprint
Li, Z., Li, Z., Zhang, J., Feng, Y., & Zhou, J. (2021) · arXiv (Cornell University) · Cited by 1
Studies in conversational AI : multilingual capabilities, world knowledge, and evaluation strategies
Bruyn, M. D. (2024) · doi:10.63028/10067/2023030151162165141 · Cited by 1
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Liu, Y., James, J., Wang, Y., Zhao, K., Xiao, C., & Lin, C. (2026) · arXiv · doi:10.48550/arxiv.2610.07109
Chen, Y., Zhang, C., Luo, D., D'Haro, L. F., Tan, R. T., & Li, H. (2024) · arXiv · doi:10.48550/arxiv.2405.14646
Utterance Pair Scoring for Noisy Dialogue Data FilteringPreprint
Akama, R., Yokoi, S., Suzuki, J., & Inui, K. (2020) · arXiv (Cornell University)
Nyberg, J., Manuvinakurike, R., & Paetzel, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05224
Liu, Z., Zhou, K., Mao, J., & Wilson, M. L. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482463
Proxy Indicators for the Quality of Open-domain Dialogues
Nedelchev, R., Lehmann, J., & Usbeck, R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.618
Sato, S., Akama, R., Ouchi, H., Suzuki, J., & Inui, K. (2022) · Journal of Natural Language Processing · doi:10.5715/jnlp.29.53
Uncertainty-aware Automatic Evaluation Method for Open-domain Dialogue Systems
Tsuta, Y., Yoshinaga, N., & Toyoda, M. (2023) · Journal of Natural Language Processing · doi:10.5715/jnlp.30.531
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types
Murugesan, K., Swaminathan, S., Dan, S., Chaudhury, S., Gunasekara, C., Crouse, M., Mahajan, D., Abdelaziz, I., Fokoue, A., Kapanipathi, P., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.274 · Cited by 1
Correction of Errors in Preference Ratings from Automated Metrics for Text Generation
Deriu, J., von Däniken, P., Tuggener, D., & Cieliebak, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.404 · Cited by 1
Adversarial Training and Model Ensemble for User Feedback Prediciton in Conversation System
Wang, J., Leng, Y., Zhai, X., Zong, L., Lin, H., & Xu, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44699-3_33
DiQAD: A Benchmark Dataset for Open-domain Dialogue Quality Assessment
Zhao, Y., Yan, L., Sun, W., Meng, C., Wang, S., Cheng, Z., Ren, Z., & Yin, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1010
CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogs
Aksu, T., Hazarika, D., Mehri, S., Kim, S., Hakkani-Tür, D. Z., Liu, Y., & Namazifar, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.717 · Cited by 1
Self-citationInvestigating Effects of Variation in Evaluating Dialogue Quality Using Large Language Models
Irum, A., & Beg, M. O. (2024) · International Multi-Topic Conference · doi:10.1109/inmic64792.2024.11004416
Ye, G., Zhao, H., Li, B., Chen, H., Zhao, Z., Jiang, Z., & Li, K. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3580272
Liu, X., Xie, Z., Zhou, J., & Jiang, S. (2025) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194025500433
Is it Enough to Ask Questions? Dialogue Evaluation through Question Answering and Generation
Vilaça, L., & Viana, P. (2025) · doi:10.1145/3746274.3760399
Co-Eval: Augmenting LLM-based Evaluation with Machine Metrics
Wu, L., Wu, W., Chen, M., Xue, J., & Li, G. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1307
Matsuo, K., Ishii, Y., Otsuka, A., Ishii, R., Sugiyama, H., Mizukami, M., Arimoto, T., Nomoto, N., Sato, Y., & Yamaguchi, T. (2025) · Journal of Information Processing · doi:10.2197/ipsjjip.33.1201
Afzal, A., Mercier, A., & Matthes, F. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-0887-7_27
Greek Automatic Text Summarization: From General Language Models to Legal Domain Challenges
Αθανασίου, Π., Karvelis, P. S., & Stylios, C. D. (2025) · South-East Europe Design Automation, Computer Engineering, Computer Networks and Social Media Conference (SEEDA-CECNSM) · doi:10.1109/seeda-cecnsm68644.2025.11329930
An exhaustive evaluation method for open-domain LLM dialogue by constructing recursive CoT
Zhao, S., & Xie, Z. (2026) · Computer Speech & Language · doi:10.1016/j.csl.2026.101957
Pareto-Aware Dual-Preference Optimization for Task-Oriented Dialogue
Bao, S., & Abisado, M. (2026) · Symmetry · doi:10.3390/sym18020372
Human-Machine Dialogue in the Medical Field. Using Dialog to Collect Important Patient Information
Liednikova, A. (2022) · theses.fr (ABES) · doi:10.70675/b572f8cbz5876z4683z828bzfb09085ddd78
Accessible Six-DOF Clay-Printing Platform Using a 3D-Printed Robotic Arm
Čabaravdić, M., & Sarajlić, A. (2026) · AI Computer Science and Robotics Technology · doi:10.5772/acrt.20250181
Engagingness in Open-Domain Dialogue Systems: A Systematic Survey of Datasets, Metrics and Methods
Mitra, E., Ferron, A., & Agrawal, A. (2026) · AI Computer Science and Robotics Technology · doi:10.5772/acrt20250081
Rehumanizer: Enhancing LLM Recommendations for Humanlike Engagement and Intriguing Interactions
Galitsky, B. (2026) · Springer eBooks · doi:10.1007/978-3-032-16524-4_5
The Medium is the Prompt: Prompts as Design Science Artifacts
Schulte, K., Herath, S., Møller, F., & Legner, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-28316-0_18
MPCEval: A Benchmark for Multi-Party Conversation Generation
Zhang, M., Yang, Y., Jia, Z., Yang, X., Pei, J., Zang, Y., Deng, X., & Chen, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817574
SE4SS for Intelligent Customer Services
Li, X., & Wang, F.-Y. (2026) · SpringerBriefs in service science · doi:10.1007/978-3-032-24152-8_3
Quality Score: A Behavioural Metric for Deliberation Quality in LLM-MAS Systems
Kapusta, J., Bauer, W., & Baranowski, J. (2026) · International Conference on Methods and Models in Automation and Robotics · doi:10.1109/mmar70562.2026.11667893
ayub, A. K. A. K. S. K. (2026) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2026.8.9.1
A Survey of Evaluation Methods for the Security and Trustworthiness of Large Language Models
Zhao, L., Xu, Z., Zhao, R., Li, C., Yang, B., & Feng, D. (2026) · International Conference on Machine Learning Technologies (ICMLT) · doi:10.1109/icmlt69916.2026.11689272
Full State-Based CS-QTM: An Architecture for Question Intervention Control in LLM-Based Chatbots
김근형, & Kim, H. S. (2026) · Applied Sciences · doi:10.3390/app162010217
Metadata
- Authors
- Mehri, S., & Eskenazi, M.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 681-707
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3023366413
- Cited by
- 149
- References
- 27
- Citations by year
- 2026: 15 · 2025: 12 · 2024: 16 · 2023: 40 · 2022: 26 · 2021: 37 · 2020: 3
- FWCI
- 10.12
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- dialogue generation, automatic evaluation metrics, reference-free evaluation, unsupervised learning, open-domain dialogue systems, dialogue quality, human judgment correlation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
