CoChat
Sign Up
Cited by 40Speech and dialogue systemsEmotion and Mood RecognitionMobile Crowdsensing and Crowdsourcing

Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents

Open access
Smith, Eric, Hsu, Orion, Qian, Rebecca, Roller, Stephen, Boureau, Y-Lan, Weston, Jason·2022·Proceedings of the 4th Workshop on NLP for Conversational AI
DOI 10.18653/v1/2022.nlp4convai-1.8
Open this paper in CoChat — free accountOpen PDF

Abstract

At the heart of improving conversational AI is the open problem of how to evaluate conversations.Issues with automatic metrics are well known (Liu et al., 2016), with human evaluations still considered the gold standard.Unfortunately, how to perform human evaluations is also an open problem: differing data collection methods have varying levels of human agreement and statistical sensitivity, resulting in differing amounts of human annotation hours and labor costs.In this work we compare five different crowdworker-based human evaluation methods and find that different methods are best depending on the types of models compared, with no clear winner across the board.While this highlights the open problems in the area, our analysis leads to advice of when to use which one, and possible future directions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.nlp4convai-1.8

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Human ratings are more consistent within a comparison than globally, giving a less noisy estimate of how models perform relative to each other.

    “Because human ratings are more consistent within a comparison than globally this gives us a less noisy estimate of how the models perform relative to each other [29].”

    From LLM Critics Help Catch LLM Bugs (2024) · cites this paper as [29]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 53 · Cited by 40

Research works printed but not linked: 44

  1. Joshua Albrecht and Rebecca Hwa. 2007. A re- examination of machine learning approaches for sentence-level mt evaluation. In Proceedings of the 45th Annual Meeting of the Association of Computa- tional Linguistics, pages 880-887.
  2. Norman H Anderson. 1965. Primacy effects in person- ality impression formation using a generalized order effect paradigm. Journal of personality and social psychology, 2(1):1.
  3. SE Asch. 1946. Forming impressions of personality. The Journal of Abnormal and Social Psychology, 41(3):258-290.
  4. Ram Ashwin, Prasad Rohit, Khatri Chandra, Venkatesh Anu, Gabriel Raefer, Liu Qing, Nunn Jeff, Hedayat- nia Behnam, Cheng Ming, Nagar Ashish, King Eric, Bland Kate, Wartick Amanda, Pan Yi, Song Han, Jayadevan Sk, Hwang Gene, and Pettigrue Art. 2017. Conversational AI: The science behind the Adlexa Prize. In Proceedings of Workshop on Conversa- tional AI.
  5. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. Proceedings of the Interna- tional AAAI Conference on Web and Social Media, 14(1):830-839.
  6. Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. In Proceedings of the International Conference on Learning Representations.
  7. Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Evaluating question answering evaluation. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 119- 124.
  8. Elizabeth Clark and Noah A Smith. 2021. Choose your own adventure: Paired suggestions in collaborative writing for evaluating story generation models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3566-3575.
  9. Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2021. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review, 54(1):755-810.
  10. Jan Milan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Álvaro Rodrigo, Thiziri Belka- cem, Aitor Soroa, Eneko Agirre, and Mark Cieliebak. 2020. Spot the bot: A robust and efficient framework for the evaluation of conversational dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3971-3984.
  11. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019b. The second conversational intelligence challenge (convai2). arXiv preprint arXiv:1902.00098. · arXiv
  12. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241. · arXiv
  13. Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: a corpus for adding memory to goal-oriented dialogue systems. In Proceedings of the 18th Annual SIGDIAL Meeting on Discourse and Dialogue, pages 207-219. ACL.
  14. Sarah E Finch and Jinho D Choi. 2020. Towards unified dialogue system evaluation: A comprehensive anal- ysis of current evaluation protocols. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 236-245.
  15. Raefer Gabriel, Yang Liu, Anna Gottardi, Mihail Eric, Anju Khatri, Anjali Chadha, Qinlang Chen, Behnam Hedayatnia, Pankaj Rajan, Ali Binici, et al. 2020. Further advances in open domain dialog systems in the third alexa prize socialbot grand challenge. Alexa Prize Proceedings.
  16. Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Àgata Lapedriza, and Rosalind W. Picard. 2019. Approx- imating interactive human evaluation with self-play for open-domain dialog systems. Advances in Neural Information Processing Systems.
  17. Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 379-391.
  18. Helen Hastie. 2012. Metrics and evaluation of spo- ken dialogue systems. In Oliver Lemon and Olivier Pietquin, editors, Data-Driven Methods for Adaptive Spoken Dialogue Systems, pages 131-150. Springer.
  19. Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meet- ing of the Special Interest Group on Discourse and Dialogue, pages 263-272.
  20. Stephen J Hoch. 1984. Availability and interference in predictive judgment. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10(4):649.
  21. Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020. Challenges in building intelligent open-domain di- alog systems. ACM Transactions on Information Systems (TOIS), 38(3):1-32.
  22. Jessica Huynh, Jeffrey Bigham, and Maxine Eskenazi. 2021. A survey of nlp-related crowdsourcing hits: what works and what does not. arXiv preprint arXiv:2111.05241. · arXiv
  23. Ilia Kulikov, Alexander Miller, Kyunghyun Cho, and Jason Weston. 2019. Importance of search and eval- uation strategies in neural dialogue modeling. In Proceedings of the 12th International Conference on Natural Language Generation, pages 76-87, Tokyo, Japan. Association for Computational Linguistics. · DOI · Paper page
  24. Seolhwa Lee, Heuiseok Lim, and João Sedoc. 2020. An evaluation protocol for generative conversational systems. arXiv preprint arXiv:2010.12741. · arXiv
  25. Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. In Pro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics, pages 994-1003.
  26. ACL.
  27. Margaret Li, Jason Weston, and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation with opti- mized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087. · arXiv
  28. Weixin Liang, James Zou, and Zhou Yu. 2020. Beyond user self-reported likert scale ratings: A comparison model for automatic dialog evaluation. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1363-1374.
  29. Chia-Wei Liu, Ryan Lowe, Iulian Vlad Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation met- rics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 2122-2132.
  30. Norman Miller and Donald T Campbell. 1959. Recency and primacy in persuasion as a function of the tim- ing of speeches and measurements. The Journal of Abnormal and Social Psychology, 59(1):1.
  31. Leo Postman and Laura W Phillips. 1965. Short-term temporal changes in free recall. Quarterly journal of experimental psychology, 17(2):132-138.
  32. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 5370-5381, Florence, Italy. Association for Computational Linguistics. · DOI
  33. Alexander Schmitt and Stefan Ultes. 2015. Interaction quality: Assessing the quality of ongoing spoken dialog interaction by experts-and how it relates to user satisfaction. Speech Communication, 74:12-36. · DOI
  34. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics, pages 1702-1723. ACL.
  35. Kurt Shuster, Jack Urbanek, Emily Dinan, Arthur Szlam, and Jason Weston. 2020. Deploying life- long open-domain dialogue learning. arXiv preprint arXiv:2008.08076. · arXiv
  36. Neil Stewart, Gordon DA Brown, and Nick Chater. 2005. Absolute identification by relative judgment. Psycho- logical review, 112(4):881.
  37. Alan M Turing and J Haugeland. 1950. Computing machinery and intelligence. MIT Press Cambridge, MA.
  38. Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metalli- nou, et al. 2018. On evaluating and comparing con- versational agents. arXiv preprint arXiv:1801.03625, 4:60-68. · arXiv
  39. Oriol Vinyals and Quoc Le. 2015. A neural conver- sational model. In Proceedings of the 31st Interna- tional Conference on Machine Learning, Deep Learn- ing Workshop, Lille, France.
  40. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. 2020. Neu- ral text generation with unlikelihood training. In International Conference on Learning Representa- tions.
  41. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Mil- ica Gasic, Lina M. Rojas Barahona, Pei-Hao Su, Ste- fan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 438-449. ACL.
  42. Jason D Williams, Antoine Raux, and Matthew Hender- son. 2016. The dialog state tracking challenge series: A review. Dialogue & Discourse, 7(3):4-33.
  43. Yu Wu, Furu Wei, Shaohan Huang, Yunli Wang, Zhou- jun Li, and Ming Zhou. 2019. Response generation by context-aware prototype editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7281-7288.
  44. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Ja- son Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079. · arXiv

Authors
Smith, E., Hsu, O., Qian, R., Roller, S., Boureau, Y.-L., & Weston, J.
Affiliations
  • Meta (United States) (US)
  • Duke University (US)
ORCID
Boureau, Y-Lan
Venue
Proceedings of the 4th Workshop on NLP for Conversational AI
Publisher
Association for Computational Linguistics
Published
2022
Pages
77-97
DOI
10.18653/v1/2022.nlp4convai-1.8
OpenAlex
W4285188834
Cited by
40
References
53
Citations by year
2026: 4 · 2025: 6 · 2024: 5 · 2023: 18 · 2022: 7
FWCI
4.98
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Workshop on NLP for Conversational AI (NLP4ConvAI) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
Keywords
human evaluation, comparison of assessment methods, conversational AI, interobserver agreement, automatic evaluation metrics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review