Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
Open accessThe DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
Sanghyun Yi, Rahul Goel, Chandra Khatri, Alessandra Cervone, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, Dilek Hakkani-Tur. Proceedings of the 12th International Conference on Natural Language Generation. 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · cited 2×
“Our word embeddings are initialized with FastText (Bojanowski et al., 2016) to capture the sub-word features and then fine-tuned.”
MethodEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · cited 2×
“Furthermore, since the class distribution of the dataset is highly imbalanced, we also calculate Matthews correlation co-efficient (MCC), which takes into account true and false positives and negatives.”
(2005) · Cited by 4,152
A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts
Pang, B., & Lee, L. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1218990 · Cited by 3,301
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Serban, I., Sordoni, A., Bengio, Y., Courville, A., & Pineau, J. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.9883 · Cited by 1,615
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073465 · Cited by 1,578
Liu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1230 · Cited by 1,228
Deep Reinforcement Learning for Dialogue Generation
Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., & Gao, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1127 · Cited by 1,204
A Persona-Based Neural Conversation Model
Li, J., Galley, M., Brockett, C., Spithourakis, G., Gao, J., & Dolan, B. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1094 · Cited by 898
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J.-Y., Gao, J., & Dolan, B. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1020 · Cited by 764
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems
Lowe, R., Pow, N., Serban, I., & Pineau, J. (2015) · Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w15-4640 · Cited by 748
Data-Driven Response Generation in Social Media
Ritter, A., Cherry, C., & Dolan, W. B. (2011) · NPARC · Cited by 742
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
Serban, I., Sordoni, A., Lowe, R., Charlin, L., Pineau, J., Courville, A., & Bengio, Y. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10983 · Cited by 697
Modeling Local Coherence: An Entity-Based Approach
Barzilay, R., & Lapata, M. (2008) · Computational Linguistics · doi:10.1162/coli.2008.34.1.1 · Cited by 671
Effective Approaches to Attention-based Neural Machine Translation
Luong, M.-T., Pham, H., & Manning, C. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.04025 · Cited by 655
A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts
Pang, B., & Lee, L. (2004) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0409058 · Cited by 652
A SICK cure for the evaluation of compositional distributional semantic models
Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Raffaella Bernardi, B., & Zamparelli, R. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/39qdhuevzbqa · Cited by 625
Liu, C., Lowe, R., Serban, I. V., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08023 · Cited by 603
PARADISE: a framework for evaluating spoken dialogue agents
Walker, M. A., Litman, D. J., Kamm, C. A., & Abella, A. (1997) · Proceedings of the 35th annual meeting on Association for Computational Linguistics - · doi:10.3115/976909.979652 · Cited by 537
Topic Aware Neural Response Generation
Xing, C., Wu, W., Wu, Y., Liu, J., Huang, Y., Zhou, M., & Ma, W.-Y. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10981 · Cited by 438
An efficient framework for learning sentence representations
Logeswaran, L., & Lee, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02893 · Cited by 427
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422 · cited 1×
“, 2010), the problem is aggravated for dialog generation evaluation because of the much larger output space (Liu et al., 2016; Novikova et al., 2017).”
BackgroundWhy We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Curry, A. C., & Rieser, V. (2017) · arXiv · doi:10.48550/arxiv.1707.06875 · Cited by 406
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, É., Joulin, A., & Mikolov, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.04606 · Cited by 384
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Lowe, R., Noseworthy, M., Serban, I. V., Angelard-Gontier, N., Bengio, Y., & Pineau, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1103 · Cited by 341 · cited 3×
“Lowe et al. (2017) train a model to predict the score of a system response given dialog context.”
BackgroundParlAI: A Dialog Research Software Platform
Miller, A., Feng, W., Batra, D., Bordes, A., Fisch, A., Lu, J., Parikh, D., & Weston, J. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d17-2014 · Cited by 315
Topic Aware Neural Response GenerationPreprint
Chen, X., Wu, W., Wu, Y., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.08340 · Cited by 313
Danescu-Niculescu-Mizil, C., & Lee, L. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1106.3077 · Cited by 312
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.03055 · Cited by 241
RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog SystemsPreprint
Tao, C., Mou, L.-L., Zhao, D., & Yan, R. (2017) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1701.03079 · Cited by 230
RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
Tao, C., Mou, L., Zhao, D., & Yan, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11321 · Cited by 211
Learning to Write with Cooperative Discriminators
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., & Choi, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1152 · Cited by 192
A Deep Reinforcement Learning Chatbot
Serban, I. V., Sankar, C., Germain, M., Zhang, S., Lin, Z., Subramanian, S., Kim, T., Pieper, M., Chandar, S., Ke, N. R., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.02349 · Cited by 185
Conversational AI: The Science Behind the Alexa Prize
Ram, A., Prasad, R., Khatri, C., Venkatesh, A., Gabriel, R., Liu, Q., Nunn, J., Hedayatnia, B., Cheng, M., Nagar, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.03604 · Cited by 185
Self-citationGenerating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models
Shao, Y., Gouws, S., Britz, D., Goldie, A., Strope, B., & Kurzweil, R. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1235 · Cited by 179
PARADISE: A Framework for Evaluating Spoken Dialogue Agents
Walker, M. A., Litman, D. J., Kamm, C., & Abella, A. (1997) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9704004 · Cited by 166
A Persona-Based Neural Conversation Model
Li, J., Galley, M., Brockett, C., Spithourakis, G., Gao, J., & Dolan, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06155 · Cited by 148
Accurate Evaluation of Segment-level Machine Translation Metrics
Graham, Y., Baldwin, T., & Mathur, N. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1124 · Cited by 92
Strategy and Policy Learning for Non-Task-Oriented Conversational Systems
Yu, Z., Xu, Z., Black, A. W., & Rudnicky, A. (2016) · Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w16-3649 · Cited by 88
Mechanism-Aware Neural Machine for Dialogue Response Generation
Zhou, G., Luo, P., Cao, R., Lin, F., Chen, B., & He, Q. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10976 · Cited by 76
Learning to Control the Specificity in Neural Response Generation
Zhang, R., Guo, J., Fan, Y., Lan, Y., Xu, J., & Cheng, X. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1102 · Cited by 74
Dialog act modelling for conversational speech
Stolcke, A., Shriberg, E. E., Bates, R. A., Coccaro, N., Jurafsky, D. S., Martin, R. W., Meteer, M. W., Ries, K., Taylor, P., & Ess-Dykema, C. V. (1998) · ERA · Cited by 68
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J., Gao, J., & Dolan, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06714 · Cited by 62
Topic-based Evaluation for Conversational BotsPreprint
Guo, F., Metallinou, A., Khatri, C., Raju, A., Venkatesh, A., & Ram, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.03622 · Cited by 61
Self-citationGenerating High-Quality and Informative Conversation Responses with Sequence-to-Sequence ModelsPreprint
Shao, L. Y., Gouws, S., Britz, D., Goldie, A., Strope, B. P., & Kurzweil, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.03185 · Cited by 54
An Attentional Neural Conversation Model with Improved SpecificityPreprint
Yao, K., Peng, B., Zweig, G., & Wong, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01292 · Cited by 35
Cervone, A., Stepanov, E. A., & Riccardi, G. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2446 · Cited by 31
Self-citationAlana: Social Dialogue using an Ensemble Model and a Ranker trained on User Feedback
Papaioannou, I. A., Curry, A. C., Part, J. L., Shalyminov, I., Xu, X., Yu, Y., Dušek, O., Rieser, V., & Lemon, O. (2017) · Edinburgh Napier Research Repository (Edinburgh Napier University) · Cited by 30
Further Meta-Evaluation of Broad-Coverage Surface Realization
Espinosa, D., Rajkumar, R. P., White, M., & Berleant, S. (2010) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 29
Data Distillation for Controlling Specificity in Dialogue GenerationPreprint
Li, J., Monroe, W., & Jurafsky, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.06703 · Cited by 24
On Evaluating and Comparing Open Domain Dialog Systems
Venkatesh, A., Khatri, C., Ram, A., Guo, F., Gabriel, R., Nagar, A., Prasad, R., Cheng, M., Hedayatnia, B., Metallinou, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.03625 · Cited by 23
Self-citationContextual Topic Modeling For Dialog Systems
Khatri, C., Goel, R., Hedayatnia, B., Metanillou, A., Venkatesh, A., Gabriel, R., & Mandal, A. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639552 · Cited by 21
Self-citation
Research works printed but not linked: 3
- ACL.
- Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016b. Build- ing end-to-end dialogue systems using generative hi- erarchical neural network models. In AAAI.
- Ellen M Voorhees and Hoa Trang Dang. 2003. Overview of the trec 2003 question answering track. In TREC, volume 2003, pages 54-68.
Cited by
Opitz, J. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00675 · Cited by 301
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92
“Sparse rewards can lead to sample-inefficient learning due to extensive exploration requirements [17].”
Belz, A., Mille, S., & Howcroft, D. M. (2020) · Proceedings of the 13th International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.24 · Cited by 45
DynaEval: Unifying Turn and Dialogue Level Evaluation
Zhang, C., Chen, Y., D’Haro, L. F., Zhang, Y., Friedrichs, T., Lee, G., & Li, H.-Z. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.441 · Cited by 89
Predictive Engagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue Systems
Ghazarian, S., Weischedel, R., Galstyan, A., & Peng, N. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6283 · Cited by 75
Metadata
- Authors
- Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., & Hakkani-Tur, D.
- Affiliations
- California Institute of Technology (US)
- Google (United States) (US)
- Uber AI (United States) (US)
- University of Trento (IT)
- Amazon (United States) (US)
- Venue
- Proceedings of the 12th International Conference on Natural Language Generation
- Published
- 2019
- Pages
- 65-75
- OpenAlex
- W2996044589
- Cited by
- 26
- References
- 53
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 5 · 2023: 6 · 2022: 2 · 2021: 4 · 2020: 6
- FWCI
- 1.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- International Conference on Natural Language Generation (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- dialogue coherence, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.