CoChat
Sign Up
Cited by 341Speech and dialogue systemsNatural Language Processing TechniquesTopic Modeling

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

Open access
Lowe, Ryan, Noseworthy, Michael, Serban, Iulian Vlad, Angelard-Gontier, Nicolas, Bengio, Yoshua, Pineau, Joelle·2017·Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/p17-1103
Open this paper in CoChat — free accountOpen PDF

Abstract

Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, Joelle Pineau. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/p17-1103

Cites

0 citation errors · 0 not checkable

Statistics

Figures

Figure 2 The ADEM model, which uses a hierarchical encoder to produce the context embedding c .
Figure 2 The ADEM model, which uses a hierarchical encoder to produce the context embedding c .CC-BY · From the PDF
Figure 3 The VHRED model used for pre-training. The hierarchical structure of the RNN encoder is shown in the red box around the bottom half of the figure. After training using the VHRED procedure, the last hidden state of the context-level encoder is used as a vector representation of the input text.
Figure 3 The VHRED model used for pre-training. The hierarchical structure of the RNN encoder is shown in the red box around the bottom half of the figure. After training using the VHRED procedure, the last hidden state of the context-level encoder is used as a vector representation of the input text.CC-BY · From the PDF
Figure 4 Scatter plot showing model against human scores, for BLEU-2 and ROUGE on the full dataset, and ADEM on the test set. We add Gaussian noise drawn from N (0 , 0 . 3) to the integer human scores to better visualize the density of points, at the expense of appearing less correlated.
Figure 4 Scatter plot showing model against human scores, for BLEU-2 and ROUGE on the full dataset, and ADEM on the test set. We add Gaussian noise drawn from N (0 , 0 . 3) to the integer human scores to better visualize the density of points, at the expense of appearing less correlated.CC-BY · From the PDF
Figure 5 Scatterplots depicting the system-level correlation results for ADEM, BLEU-2, BLEU-4,and ROUGE on the test set. Each point represents the average scores for the responses from a dialogue model (TFIDF, DE, HRED, human). Human scores are shown on the horizontal axis, with normalized metric scores on the vertical axis. The ideal metric has a perfectly linear relationship.
Figure 5 Scatterplots depicting the system-level correlation results for ADEM, BLEU-2, BLEU-4,and ROUGE on the test set. Each point represents the average scores for the responses from a dialogue model (TFIDF, DE, HRED, human). Human scores are shown on the horizontal axis, with normalized metric scores on the vertical axis. The ideal metric has a perfectly linear relationship.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 42

  1. Joshua Albrecht and Rebecca Hwa. 2007. Regression for sentence-level mt evaluation with pseudo refer- ences. In ACL.
  2. Ron Artstein, Sudeep Gandhe, Jillian Gerten, Anton Leuski, and David Traum. 2009. Semi-formal eval- uation of conversational characters. In Languages: From Formal to Natural, Springer, pages 22-35.
  3. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . · arXiv
  4. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradi- ent descent is difficult. IEEE transactions on neural networks 5(2):157-166.
  5. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Pro- ceedings of the Sixth Workshop on Statistical Ma- chine Translation. Association for Computational Linguistics, pages 22-64.
  6. Tim Cooijmans, Nicolas Ballas, César Laurent, and Aaron Courville. 2016. Recurrent batch normaliza- tion. arXiv preprint arXiv:1603.09025 . · arXiv
  7. David DeVault, Anton Leuski, and Kenji Sagae. 2011. Toward learning and evaluation of dialogue policies with text examples. In Proceedings of the SIG- DIAL 2011 Conference. Association for Computa- tional Linguistics, pages 39-48.
  8. Bhuwan Dhingra, Zhong Zhou, Dylan Fitzpatrick, Michael Muehl, and William W Cohen. 2016. Tweet2vec: Character-based distributed repre- sentations for social media. arXiv preprint arXiv:1605.03481 . · arXiv
  9. Harrison Edwards and Amos Storkey. 2016. Censoring representations with an adversary. ICLR .
  10. Salah El Hihi and Yoshua Bengio. 1995. Hierarchical recurrent neural networks for long-term dependen- cies. In NIPS. Citeseer, volume 400, page 409.
  11. Philip Gage. 1994. A new algorithm for data compres- sion. The C Users Journal 12(2):23-38.
  12. Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Mar- garet Mitchell, Jianfeng Gao, and Bill Dolan. 2015. deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. arXiv preprint arXiv:1506.06863 . · arXiv
  13. Sudeep Gandhe and David Traum. 2016. A semi- automated evaluation metric for dialogue model coherence. In Situated Dialog in Speech-Based Human-Computer Interaction, Springer, pages 217- 225.
  14. Rohit Gupta, Constantin Orasan, and Josef van Gen- abith. 2015. Reval: A simple and effective machine translation evaluation metric based on recurrent neu- ral networks. In Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP).
  15. Ryuichiro Higashinaka, Toyomi Meguro, Kenji Ima- mura, Hiroaki Sugiyama, Toshiro Makino, and Yoshihiro Matsuo. 2014. Evaluating coherence in open domain conversational systems. In INTER- SPEECH. pages 130-134.
  16. Sergey Ioffe and Christian Szegedy. 2015. Batch nor- malization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 . · arXiv
  17. Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, László Lukács, Marina Ganea, Peter Young, et al. 2016. Smart reply: Automated re- sponse suggestion for email. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). volume 36, pages 495- 503.
  18. Anjuli Kannan and Oriol Vinyals. 2017. Adversar- ial evaluation of dialogue models. arXiv preprint arXiv:1701.08198 . · arXiv
  19. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . · arXiv
  20. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objec- tive function for neural conversation models. arXiv preprint arXiv:1510.03055 . · arXiv
  21. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A persona-based neural con- versation model. arXiv preprint arXiv:1603.06155 . · arXiv
  22. Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Learn- ing to decode for future success. arXiv preprint arXiv:1701.06549 . · arXiv
  23. Jiwei Li, Will Monroe, Alan Ritter, and Dan Jurafsky. 2016b. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541 . · arXiv
  24. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation met- rics for dialogue response generation. arXiv preprint arXiv:1603.08023 . · arXiv
  25. Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dia- logue systems. arXiv preprint arXiv:1506.08909 . · arXiv
  26. Matouš Machácek and Ondrej Bojar. 2014. Results of the wmt14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Transla- tion. Citeseer, pages 293-301.
  27. Sebastian Möller, Roman Englert, Klaus-Peter Engel- brecht, Verena Vanessa Hafner, Anthony Jameson, Antti Oulasvirta, Alexander Raake, and Norbert Re- ithinger. 2006. Memo: towards automatic usability evaluation of spoken dialogue services by user error simulations. In INTERSPEECH.
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th annual meeting on association for compu- tational linguistics. Association for Computational Linguistics, pages 311-318.
  29. Karl Pearson. 1901. Principal components analysis. The London, Edinburgh and Dublin Philosophical Magazine and Journal 6(2):566.
  30. Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In Proceedings of the conference on empirical meth- ods in natural language processing. Association for Computational Linguistics, pages 583-593.
  31. Iulian Vlad Serban, Alessandro Sordoni, Yoshua Ben- gio, Aaron Courville, and Joelle Pineau. 2016a. Building end-to-end dialogue systems using gener- ative hierarchical neural network models. In AAAI. pages 3776-3784.
  32. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016b. A hierarchical latent variable encoder-decoder model for generating dia- logues. arXiv preprint arXiv:1605.06069 . · arXiv
  33. Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neu- ral responding machine for short-text conversation. arXiv preprint arXiv:1503.02364 . · arXiv
  34. Lifeng Shang, Tetsuya Sakai, Zhengdong Lu, Hang Li, Ryuichiro Higashinaka, and Yusuke Miyao. 2016. Overview of the ntcir-12 short text conversation task. Proceedings of NTCIR-12 pages 473-484.
  35. Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian- Yun Nie. 2015a. A hierarchical recurrent encoder- decoder for generative context-aware query sugges- tion. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Man- agement. ACM, pages 553-562.
  36. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015b. A neural network approach to context-sensitive gen- eration of conversational responses. arXiv preprint arXiv:1506.06714 . · arXiv
  37. Miloš Stanojevic, Amir Kamran, Philipp Koehn, and Ondrej Bojar. 2015. Results of the wmt15 metrics shared task. In Proceedings of the Tenth Workshop on Statistical Machine Translation. pages 256-273.
  38. Alan M Turing. 1950. Computing machinery and intel- ligence. Mind 59(236):433-460.
  39. Oriol Vinyals and Quoc Le. 2015. A neural conversa- tional model. arXiv preprint arXiv:1506.05869 . · arXiv
  40. Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. Paradise: A framework for evaluating spoken dialogue agents. In Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics. Associa- tion for Computational Linguistics, pages 271-280.
  41. J. Weizenbaum. 1966. ELIZAa computer program for the study of natural language communication be- tween man and machine. Communications of the ACM 9(1):36-45.
  42. Yang Xiang, Yaoyun Zhang, Xiaoqiang Zhou, Xiao- long Wang, and Yang Qin. 2014. Problematic situa- tion analysis and automatic recognition for chi-nese online conversational system. Proc. CLP pages 43- 51.

Cited by

Sorted by citations

Items 101–200 of 341

Related

See all

Metadata

Authors
Lowe, R., Noseworthy, M., Serban, I. V., Angelard-Gontier, N., Bengio, Y., & Pineau, J.
Affiliations
  • McGill University (CA)
  • Canadian Institute for Advanced Research (CA)
ORCID
Bengio, Yoshua
Venue
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2017
Pages
1116-1126
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/p17-1103
OpenAlex
W2741333084
Cited by
341
References
0
Citations by year
2026: 6 · 2025: 11 · 2024: 10 · 2023: 35 · 2022: 34 · 2021: 61 · 2020: 69 · 2019: 58 · 2018: 47 · 2017: 10
FWCI
22.95
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Learning Representations
Funders
Samsung
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
Turing test, automated assessment, chatbots, human-computer dialogue, natural language understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review