Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Open accessAbstract
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, Joelle Pineau. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Supervised assessment methods learn to predict human-like evaluation scores but require significant annotation effort for training data.
“Supervised assessment methods learn to predict human-like evaluation scores (Lowe et al., 2017), but they require a significant annotation effort for achieving training data.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Lowe et al., 2017
Standard token-matching metrics such as BLEU, METEOR, and ROUGE were shown to have weak or no correlation with human judgements for dialogue.
“Standard token-matching metrics, such as BLEU (Papineni et al., 2002) or METEOR (Banerjee and Lavie, 2005) in machine translation, or ROUGE (Lin, 2004) in summarization, were shown to have weak or no correlation with human judgements for dialogue (Liu et al., 2016;Lowe et al., 2017).”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Lowe et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Research works printed but not linked: 45
- Joshua Albrecht and Rebecca Hwa. 2007. Regression for sentence-level mt evaluation with pseudo refer- ences. In ACL.
- Ron Artstein, Sudeep Gandhe, Jillian Gerten, Anton Leuski, and David Traum. 2009. Semi-formal eval- uation of conversational characters. In Languages: From Formal to Natural, Springer, pages 22-35.
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . · arXiv
- Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradi- ent descent is difficult. IEEE transactions on neural networks 5(2):157-166.
- Samuel R Bowman, Luke Vilnis, Oriol Vinyals, An- drew M Dai, Rafal Jozefowicz, and Samy Ben- gio. 2016. Generating sentences from a continuous space. COLING .
- Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Pro- ceedings of the Sixth Workshop on Statistical Ma- chine Translation. Association for Computational Linguistics, pages 22-64.
- Tim Cooijmans, Nicolas Ballas, César Laurent, and Aaron Courville. 2016. Recurrent batch normaliza- tion. arXiv preprint arXiv:1603.09025 . · arXiv
- David DeVault, Anton Leuski, and Kenji Sagae. 2011. Toward learning and evaluation of dialogue policies with text examples. In Proceedings of the SIG- DIAL 2011 Conference. Association for Computa- tional Linguistics, pages 39-48.
- Bhuwan Dhingra, Zhong Zhou, Dylan Fitzpatrick, Michael Muehl, and William W Cohen. 2016. Tweet2vec: Character-based distributed repre- sentations for social media. arXiv preprint arXiv:1605.03481 . · arXiv
- Harrison Edwards and Amos Storkey. 2016. Censoring representations with an adversary. ICLR .
- Salah El Hihi and Yoshua Bengio. 1995. Hierarchical recurrent neural networks for long-term dependen- cies. In NIPS. Citeseer, volume 400, page 409.
- Philip Gage. 1994. A new algorithm for data compres- sion. The C Users Journal 12(2):23-38.
- Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Mar- garet Mitchell, Jianfeng Gao, and Bill Dolan. 2015. deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. arXiv preprint arXiv:1506.06863 . · arXiv
- Sudeep Gandhe and David Traum. 2016. A semi- automated evaluation metric for dialogue model coherence. In Situated Dialog in Speech-Based Human-Computer Interaction, Springer, pages 217- 225.
- Rohit Gupta, Constantin Orasan, and Josef van Gen- abith. 2015. Reval: A simple and effective machine translation evaluation metric based on recurrent neu- ral networks. In Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP).
- Ryuichiro Higashinaka, Toyomi Meguro, Kenji Ima- mura, Hiroaki Sugiyama, Toshiro Makino, and Yoshihiro Matsuo. 2014. Evaluating coherence in open domain conversational systems. In INTER- SPEECH. pages 130-134.
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735- 1780.
- Sergey Ioffe and Christian Szegedy. 2015. Batch nor- malization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 . · arXiv
- Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, László Lukács, Marina Ganea, Peter Young, et al. 2016. Smart reply: Automated re- sponse suggestion for email. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). volume 36, pages 495- 503.
- Anjuli Kannan and Oriol Vinyals. 2017. Adversar- ial evaluation of dialogue models. arXiv preprint arXiv:1701.08198 . · arXiv
- Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . · arXiv
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objec- tive function for neural conversation models. arXiv preprint arXiv:1510.03055 . · arXiv
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A persona-based neural con- versation model. arXiv preprint arXiv:1603.06155 . · arXiv
- Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Learn- ing to decode for future success. arXiv preprint arXiv:1701.06549 . · arXiv
- Jiwei Li, Will Monroe, Alan Ritter, and Dan Jurafsky. 2016b. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541 . · arXiv
- Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation met- rics for dialogue response generation. arXiv preprint arXiv:1603.08023 . · arXiv
- Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dia- logue systems. arXiv preprint arXiv:1506.08909 . · arXiv
- Matouš Machácek and Ondrej Bojar. 2014. Results of the wmt14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Transla- tion. Citeseer, pages 293-301.
- Sebastian Möller, Roman Englert, Klaus-Peter Engel- brecht, Verena Vanessa Hafner, Anthony Jameson, Antti Oulasvirta, Alexander Raake, and Norbert Re- ithinger. 2006. Memo: towards automatic usability evaluation of spoken dialogue services by user error simulations. In INTERSPEECH.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th annual meeting on association for compu- tational linguistics. Association for Computational Linguistics, pages 311-318.
- Karl Pearson. 1901. Principal components analysis. The London, Edinburgh and Dublin Philosophical Magazine and Journal 6(2):566.
- Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In Proceedings of the conference on empirical meth- ods in natural language processing. Association for Computational Linguistics, pages 583-593.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 . · arXiv
- Iulian Vlad Serban, Alessandro Sordoni, Yoshua Ben- gio, Aaron Courville, and Joelle Pineau. 2016a. Building end-to-end dialogue systems using gener- ative hierarchical neural network models. In AAAI. pages 3776-3784.
- Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016b. A hierarchical latent variable encoder-decoder model for generating dia- logues. arXiv preprint arXiv:1605.06069 . · arXiv
- Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neu- ral responding machine for short-text conversation. arXiv preprint arXiv:1503.02364 . · arXiv
- Lifeng Shang, Tetsuya Sakai, Zhengdong Lu, Hang Li, Ryuichiro Higashinaka, and Yusuke Miyao. 2016. Overview of the ntcir-12 short text conversation task. Proceedings of NTCIR-12 pages 473-484.
- Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian- Yun Nie. 2015a. A hierarchical recurrent encoder- decoder for generative context-aware query sugges- tion. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Man- agement. ACM, pages 553-562.
- Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015b. A neural network approach to context-sensitive gen- eration of conversational responses. arXiv preprint arXiv:1506.06714 . · arXiv
- Miloš Stanojevic, Amir Kamran, Philipp Koehn, and Ondrej Bojar. 2015. Results of the wmt15 metrics shared task. In Proceedings of the Tenth Workshop on Statistical Machine Translation. pages 256-273.
- Alan M Turing. 1950. Computing machinery and intel- ligence. Mind 59(236):433-460.
- Oriol Vinyals and Quoc Le. 2015. A neural conversa- tional model. arXiv preprint arXiv:1506.05869 . · arXiv
- Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. Paradise: A framework for evaluating spoken dialogue agents. In Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics. Associa- tion for Computational Linguistics, pages 271-280.
- J. Weizenbaum. 1966. ELIZAa computer program for the study of natural language communication be- tween man and machine. Communications of the ACM 9(1):36-45.
- Yang Xiang, Yaoyun Zhang, Xiaoqiang Zhou, Xiao- long Wang, and Yang Qin. 2014. Problematic situa- tion analysis and automatic recognition for chi-nese online conversational system. Proc. CLP pages 43- 51.
Cited by
Sorted by citations
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Adversarial Learning for Neural Dialogue Generation
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., & Jurafsky, D. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1230 · Cited by 880 · see citing works
The design and implementation of Xiaoice, an empathetic social chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2020) · Computational Linguistics · doi:10.1162/coli_a_00368 · Cited by 777 · see citing works
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · see citing works
A Survey on Dialogue Systems: Recent Advances and New Frontiers
Chen, H., Liu, X., Yin, D., & Tang, J. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3166054.3166058 · Cited by 579 · see citing works
Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network
Zhou, X., Lü, L., Dong, D., Liu, Y., Chen, Y., Zhao, W. X., Yu, D., & Wu, H. R. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1103 · Cited by 339 · see citing works
Survey on evaluation methods for dialogue systems
Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., & Cieliebak, M. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09866-x · Cited by 333 · see citing works
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models
Zhang, H., Song, H., Li, S., Zhou, M., & Song, D. (2024) · ACM Computing Surveys · doi:10.1145/3617680 · Cited by 317 · see citing works
Challenges in building intelligent open-domain dialog systems
Huang, M., Zhu, X., & Gao, J. (2020) · ACM Transactions on Information Systems · doi:10.1145/3383123 · Cited by 288 · see citing works
Recent advances in deep learning based dialogue systems: a systematic survey
Ni, J., Young, T., Pandelea, V., Xue, F., & Cambria, E. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10248-8 · Cited by 279 · see citing works
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2019) · arXiv · doi:10.48550/arxiv.1911.00650 · Cited by 264
“It continues to be used is chatbot evaluation (Lowe et al., 2017).”
BackgroundAutomatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.164 · Cited by 264 · see citing works
Users' experiences with chatbots: findings from a questionnaire study
Følstad, A., & Brandtzæg, P. B. (2020) · Quality and User Experience · doi:10.1007/s41233-020-00033-2 · Cited by 280 · see citing works
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247 · see citing works
RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
Tao, C., Mou, L., Zhao, D., & Yan, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11321 · Cited by 213 · see citing works
Evaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207 · see citing works
A Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235 · see citing works
A Deep Reinforcement Learning ChatbotPreprint
Serban, I. V., Sankar, C., Germain, M., Zhang, S., Lin, Z., Subramanian, S., Kim, T., Pieper, M., Chandar, S., Ke, N. R., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.02349 · Cited by 195 · see citing works
Self-citationSharma, S., Asri, L. E., Schulz, H., & Zumer, J. (2017) · arXiv.org · doi:10.48550/arxiv.1706.09799 · Cited by 250 · see citing works
Transformer models used for text-based question answering systems
Nassiri, K., & Akhloufi, M. A. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04052-8 · Cited by 177 · see citing works
Conversational Agents: Goals, Technologies, Vision and Challenges
Allouch, M., Azaria, A., & Azoulay, R. (2021) · Sensors · doi:10.3390/s21248448 · Cited by 168 · see citing works
Unifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161 · see citing works
No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
Wang, X. E., Chen, W., Wang, Y.-F., & Wang, W. Y. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1083 · Cited by 161 · see citing works
Polite Dialogue Generation Without Parallel Data
Niu, T., & Bansal, M. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00027 · Cited by 158 · see citing works
A Survey of Natural Language Generation
Dong, C., Li, Y., Gong, H., Chen, M., Li, J., Shen, Y., & Yang, M. (2022) · ACM Computing Surveys · doi:10.1145/3554727 · Cited by 169 · see citing works
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Mehri, S., & Eskenazi, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.64 · Cited by 149 · see citing works
Learning to Evaluate Image Captioning
Cui, Y., Yang, G., Veit, A., Huang, X., & Belongie, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00608 · Cited by 140 · see citing works
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113 · see citing works
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn ComparisonsPreprint
Li, M., Weston, J., & Roller, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03087 · Cited by 111 · see citing works
Neural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2019) · now publishers, Inc. eBooks · doi:10.1561/9781680835533 · Cited by 124 · see citing works
Towards a Better Metric for Evaluating Question Generation Systems
Nema, P., & Khapra, M. M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1429 · Cited by 106 · see citing works
Counterfactual Story Reasoning and Generation
Qin, L., Bosselut, A., Holtzman, A., Bhagavatula, C., Clark, E., & Choi, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1509 · Cited by 98 · see citing works
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.29 · Cited by 117 · see citing works
The price of debiasing automatic metrics in natural language evalaution
Chaganty, A., Mussmann, S., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1060 · Cited by 96 · see citing works
Evaluating Coherence in Dialogue Systems using Entailment
Dziri, N., Kamalloo, E., Mathewson, K. W., & Zai͏̈ane, O. R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1381 · Cited by 91 · see citing works
Survey on Intelligent Chatbots: State-of-the-Art and Future Research Directions
Almansor, E. H., & Hussain, F. K. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-22354-0_47 · Cited by 96 · see citing works
#MeToo Alexa: How Conversational Systems Respond to Sexual Harassment
Curry, A. C., & Rieser, V. (2018) · doi:10.18653/v1/w18-0802 · Cited by 87 · see citing works
Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints
Baheti, A., Ritter, A., Li, J., & Dolan, B. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1431 · Cited by 87 · see citing works
Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings
Ghazarian, S., Wei, J. T.-Z., Galstyan, A., & Peng, N. (2019) · doi:10.18653/v1/w19-2310 · Cited by 84 · see citing works
"Chitty-Chitty-Chat Bot": Deep Learning for Conversational AI
Yan, R. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/778 · Cited by 88 · see citing works
Augmenting Neural Response Generation with Context-Aware Topical Attention
Dziri, N., Kamalloo, E., Mathewson, K. W., & Zai͏̈ane, O. R. (2019) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/w19-4103 · Cited by 77 · see citing works
GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems
Huang, L., Ye, Z., Qin, J., Lin, L., & Liang, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.742 · Cited by 81 · see citing works
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog SystemsPreprint
Ghandeharioun, A., Shen, J. H., Jaques, N., Ferguson, C., Jones, N. C., Lapedriza, À., & Picard, R. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09308 · Cited by 77 · see citing works
“I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset
Smith, E. M., Hall, M., Kambadur, M., Presani, E., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.625 · Cited by 97 · see citing works
RankME: Reliable Human Ratings for Natural Language Generation
Novikova, J., Dušek, O., & Rieser, V. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2012 · Cited by 75 · see citing works
Discourse-Aware Neural Rewards for Coherent Text Generation
Bosselut, A., Çelikyılmaz, A., He, X., Gao, J., Huang, P.-S., & Choi, Y. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1016 · Cited by 73 · see citing works
Chandra, Y. W., & Suyanto, S. (2019) · Procedia Computer Science · doi:10.1016/j.procs.2019.08.179 · Cited by 73 · see citing works
Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References
Gupta, P., Mehri, S., Zhao, T., Pavel, A., Eskénazi, M., & Bigham, J. P. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5944 · Cited by 72 · see citing works
Cui, F., Cui, Q., & Song, Y. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2985588 · Cited by 72 · see citing works
Irving, G., Christiano, P., & Amodei, D. (2018) · arXiv · doi:10.48550/arxiv.1805.00899 · Cited by 71 · see citing works
Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation
Pang, B., Nijkamp, E., Han, W., Zhou, L., Liu, Y., & Tu, K. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.333 · Cited by 70 · see citing works
Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions
Aliannejadi, M., Kiseleva, J., Chuklin, A., Dalton, J., & Burtsev, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.367 · Cited by 77 · see citing works
Learning an Unreferenced Metric for Online Dialogue Evaluation
Sinha, K., Parthasarathi, P., Wang, J., Lowe, R., Hamilton, W. L., & Pineau, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.220 · Cited by 64 · see citing works
Self-citationA Comprehensive Assessment of Dialog Evaluation Metrics
Yeh, Y.-T., Eskénazi, M., & Mehri, S. (2021) · doi:10.18653/v1/2021.eancs-1.3 · Cited by 71 · see citing works
Improving Neural Conversational Models with Entropy-Based Data Filtering
Csáky, R., Purgai, P., & Recski, G. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1567 · Cited by 60 · see citing works
TIGEr: Text-to-Image Grounding for Image Caption Evaluation
Jiang, M., Huang, Q., Zhang, L., Wang, X., Zhang, P., Gan, Z., Diesner, J., & Gao, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1220 · Cited by 59 · see citing works
Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining
Sai, A. B., Mohankumar, A. K., Arora, S., & Khapra, M. M. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00347 · Cited by 58 · see citing works
Deepfake Text Detection: Limitations and Opportunities
Pu, J., Sarwar, Z., Abdullah, S. M., Rehman, A., Kim, Y., Bhattacharya, P., Javed, M., & Viswanath, B. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179387 · Cited by 62 · see citing works
This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation
Zhang, R., & Tetreault, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1043 · Cited by 55 · see citing works
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation
Guan, J., & Huang, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.736 · Cited by 50 · see citing works
Shen, J., Yang, B., Dudley, J. J., & Kristensson, P. O. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511145 · Cited by 48 · see citing works
Improving Access to Justice with Legal Chatbots
Queudot, M., Charton, É., & Meurs, M. (2020) · Stats · doi:10.3390/stats3030023 · Cited by 59 · see citing works
Deng, M., Tan, B., Liu, Z., Xing, E. P., & Hu, Z. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.599 · Cited by 49 · see citing works
The First Conversational Intelligence Challenge
Burtsev, M., Logacheva, V., Malykh, V. A., Serban, I. V., Lowe, R., Prabhumoye, S., Black, A. W., Rudnicky, A. I., & Bengio, Y. (2018) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-94042-7_2 · Cited by 46 · see citing works
Self-citationLearning towards conversational AI: A survey
Fu, T., Gao, S., Zhao, X., Wen, J.-R., & Yan, R. (2022) · AI Open · doi:10.1016/j.aiopen.2022.02.001 · Cited by 46 · see citing works
Meta-evaluation of Conversational Search Evaluation Metrics
Liu, Z., Zhou, K., & Wilson, M. L. (2021) · ACM Transactions on Information Systems · doi:10.1145/3445029 · Cited by 48 · see citing works
Learning to Collaborate for Question Answering and Asking
Tang, D., Duan, N., Yan, Z., Zhang, Z., Sun, Y., Liu, S., Lv, Y., & Zhou, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1141 · Cited by 45 · see citing works
Predictive Engagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue Systems
Ghazarian, S., Weischedel, R., Galstyan, A., & Peng, N. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6283 · Cited by 44 · see citing works
Gao, J., Galley, M., & Li, L. C. (2019) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000074 · Cited by 44 · see citing works
Deep Learning Based Chatbot ModelsPreprint
Csaky, R. (2019) · arXiv.org · doi:10.48550/arxiv.1908.08835 · Cited by 53 · see citing works
Smith, E., Hsu, O., Qian, R., Roller, S., Boureau, Y.-L., & Weston, J. (2022) · Proceedings of the 4th Workshop on NLP for Conversational AI · doi:10.18653/v1/2022.nlp4convai-1.8 · Cited by 40 · see citing works
Joint Learning of Question Answering and Question Generation
Sun, Y., Tang, D., Duan, N., Qin, T., Liu, S., Yan, Z., Zhou, M., Lv, Y., Yin, W., Feng, X., et al. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2897773 · Cited by 40 · see citing works
Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems
Phy, V., Zhao, Y., & Aizawa, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.368 · Cited by 39 · see citing works
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
Roller, S., Boureau, Y.-L., Weston, J., Bordes, A., Dinan, E., Fan, A., Gunning, D., Ju, D., Li, M., Poff, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.12442 · Cited by 38 · see citing works
Commonsense-Focused Dialogues for Response Generation: An Empirical Study
Zhou, P., Gopalakrishnan, K., Hedayatnia, B., Kim, S., Pujara, J., Ren, X., Liu, Y., & Hakkani-Tür, D. Z. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.13 · Cited by 38 · see citing works
Designing Precise and Robust Dialogue Response Evaluators
Zhao, T., Lala, D., & Kawahara, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.4 · Cited by 38 · see citing works
Topic-based Evaluation for Conversational BotsPreprint
Guo, F., Metallinou, A., Khatri, C., Raju, A., Venkatesh, A., & Ram, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.03622 · Cited by 38 · see citing works
Hierarchical Reinforcement Learning for Open-Domain Dialog
Saleh, A., Jaques, N., Ghandeharioun, A., Shen, J. H., & Picard, R. W. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6400 · Cited by 16 · see citing works
Razumovskaia, E., Glavaš, G., Majewska, O., Ponti, E. M., Korhonen, A., & Vulić, I. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13083 · Cited by 57 · see citing works
The price of debiasing automatic metrics in natural language evaluationPreprint
Chaganty, A. T., Mussmann, S., & Liang, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02202 · Cited by 36 · see citing works
Towards Automated Customer Support
Hardalov, M., Koychev, I., & Nakov, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99344-7_5 · Cited by 35 · see citing works
Perturbation CheckLists for Evaluating NLG Evaluation Metrics
Sai, A. B., Dixit, T., Sheth, D. Y., Mohan, S., & Khapra, M. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.575 · Cited by 34 · see citing works
On Evaluating and Comparing Open Domain Dialog SystemsPreprint
Venkatesh, A., Khatri, C., Ram, A., Guo, F., Gabriel, R., Nagar, A., Prasad, R., Cheng, M., Hedayatnia, B., Metallinou, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.03625 · Cited by 34 · see citing works
What makes a good conversation? How controllable attributes affect human judgmentsPreprint
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08654 · Cited by 34 · see citing works
Survey on Factuality in Large Language Models
Wang, C., Liu, X., Yue, Y., Guo, Q., Hu, X., Tang, X., Zhang, T., Cheng, J., Yao, Y., Hu, X., et al. (2025) · ACM Computing Surveys · doi:10.1145/3742420 · Cited by 221 · see citing works
Re-Evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
Sai, A. B., Gupta, M. D., Khapra, M. M., & Srinivasan, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016220 · Cited by 33 · see citing works
Unifying Human and Statistical Evaluation for Natural Language GenerationPreprint
Hashimoto, T., Zhang, H., & Liang, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.02792 · Cited by 33 · see citing works
Zhou, W., & Xu, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6521 · Cited by 31 · see citing works
Arora, K., Asri, L. E., Bahuleyan, H., & Cheung, J. C. K. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.58 · Cited by 44 · see citing works
Learning to Converse with Noisy Data: Generation with Calibration
Shang, M., Fu, Z., Peng, N., Feng, Y., Zhao, D., & Yan, R. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/603 · Cited by 30 · see citing works
Alana: Social Dialogue using an Ensemble Model and a Ranker trained on User Feedback
Papaioannou, I. A., Curry, A. C., Part, J. L., Shalyminov, I., Xu, X., Yu, Y., Dušek, O., Rieser, V., & Lemon, O. (2017) · Edinburgh Napier Research Repository (Edinburgh Napier University) · Cited by 30 · see citing works
Santhanam, S., & Shaikh, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.00500 · Cited by 43 · see citing works
Lan, T., Mao, X.-L., Wei, W., Gao, X., & Huang, H. (2020) · ACM Transactions on Information Systems · doi:10.1145/3423168 · Cited by 40 · see citing works
Towards Best Experiment Design for Evaluating Dialogue System Output
Santhanam, S., & Shaikh, S. (2019) · International Conference on Natural Language Generation · doi:10.18653/v1/w19-8610 · Cited by 27 · see citing works
OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics
Guan, J., Zhang, Z., Feng, Z., Liu, Z., Ding, W., Mao, X., Fan, C., & Huang, M. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.500 · Cited by 33 · see citing works
Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., & Hakkani-Tur, D. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8608 · Cited by 26 · see citing works
Towards Designing Conversational Agent Systems
Jadhav, K. P., & Thorat, S. A. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-981-32-9515-5_51 · Cited by 26 · see citing works
Mrinalini, K., Vijayalakshmi, P., & Nagarajan, T. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3161160 · Cited by 27 · see citing works
Automatic Article Commenting: the Task and Dataset
Qin, L., Liu, L., Bi, W., Wang, Y., Liu, X., Hu, Z., Zhao, H., & Shi, S. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2025 · Cited by 25 · see citing works
Results of the Multi-Domain Task-Completion Dialog Challenge
Li, J., Peng, B., Lee, S., Gao, J., Takanobu, R., Zhu, Q., Huang, M., Schulz, H., Atkinson, A., & Adada, M. (2020) · National Conference on Artificial Intelligence · Cited by 23 · see citing works
- Authors
- Lowe, R., Noseworthy, M., Serban, I. V., Angelard-Gontier, N., Bengio, Y., & Pineau, J.
- Affiliations
- McGill University (CA)
- Canadian Institute for Advanced Research (CA)
- ORCID
- Bengio, Yoshua
- Venue
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2017
- Pages
- 1116-1126
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2741333084
- Cited by
- 341
- References
- 0
- Citations by year
- 2026: 6 · 2025: 11 · 2024: 10 · 2023: 35 · 2022: 34 · 2021: 61 · 2020: 69 · 2019: 58 · 2018: 47 · 2017: 10
- FWCI
- 22.95
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Funders
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- Turing test, automated assessment, chatbots, human-computer dialogue, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
