CoChat
Sign Up
Cited by 21Speech and dialogue systemsNatural Language Processing TechniquesAI in Service Interactions

D-Score: Holistic Dialogue Evaluation Without Reference

Open access
Zhang, Chen, Lee, Grandee, D'Haro, Luis Fernando, Li, Haizhou·2021·IEEE/ACM Transactions on Audio, Speech, and Language Processing
DOI 10.1109/taslp.2021.3074012
Open this paper in CoChat — free accountOpen PDF

Abstract

In artistic gymnastics, difficulty score or D-score is used for judging performance. Starting from zero, an athlete earns points from different aspects such as composition requirement, difficulty, and connection between moves. The final score is a composition of the quality of various performance indicators. Similarly, when evaluating dialogue responses, human judges generally follow a number of criteria, among which language fluency, context coherence, logical consistency, and semantic appropriateness are on top of the agenda. In this paper, we propose an automatic dialogue evaluation framework called D-score that resembles the way gymnastics is evaluated. Following the four human judging criteria above, we devise a range of evaluation tasks and model them under a multi-task learning framework. The proposed framework, without relying on any human-written reference, learns to appreciate the overall quality of human-human conversations through a representation that is shared by all tasks without over-fitting to individual task domain. We evaluate D-score by performing comprehensive correlation analyses with human judgement on three dialogue evaluation datasets, among which two are from past DSTC series, and benchmark against state-of-the-art baselines. D-score not only outperforms the best baseline by a large margin in terms of system-level Spearman correlation but also represents an important step towards explainable dialogue scoring.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/taslp.2021.3074012

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Several works have proposed evaluation metrics for measuring consistency, such as using natural language inference, training learnable evaluation metrics, or releasing an additional test set for coherence.

    “Recently, several works have proposed evaluation metrics for measuring consistency, such as using natural language inference (NLI) [58, 278], training learnable evaluation metrics [309], or releasing an additional test set for coherence [14].”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [309]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 61

  1. C.-W. Liu, R. Lowe, I. Serban, M. Noseworthy, L. Charlin, and J. Pineau, "How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation," in Proc. Conf. Empirical Methods Natural Lang. Process., Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 2122-2132.
  2. J. Deriu et al., "Survey on evaluation methods for dialogue systems," Artif. Intell. Rev., vol. 54, no. 1, pp. 755-810, 2021.
  3. S. E. Finch and J. D. Choi, "Towards unified dialogue system evaluation: A comprehensive analysis of current evaluation protocols," in Proc. 21th Ann. Meeting Special Interest Group Discourse Dialogue, 2020, pp. 236-245.
  4. T. Zhao, R. Zhao, and M. Eskenazi, "Learning discourse-level diversity for neural dialog models using conditional variational autoencoders," in Proc. 55th Ann. Meeting Assoc. Comput. Linguistics, (Volume 1: Long Papers). Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 654-664.
  5. P. Gupta, S. Mehri, T. Zhao, A. Pavel, M. Eskenazi, and J. Bigham, "Investigating evaluation of open-domain dialogue systems with human generated multiple references," in Proc. 20th Annu. SIGDIAL Meeting Discourse Dialogue. Stockholm, Sweden: Association for Computational Linguistics, Sep. 2019, pp. 379-391.
  6. S. Ghazarian, J. Wei, A. Galstyan, and N. Peng, "Better automatic evaluation of open-domain dialogue systems with contextualized em- beddings," in Proc. Workshop Methods Optim. Evaluating Neural Lang. Gener.. Minneapolis, Minnesota: Association for Computational Linguis- tics, Jun. 2019, pp. 82-89.
  7. C. Tao, L. Mou, D. Zhao, and R. Yan, "Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems," in Proc. AAAI Conf. Artif. Intell., vol. 32, pp. 722-729, 2018.
  8. K. Sinha, P. Parthasarathi, J. Wang, R. Lowe, W. L. Hamilton, and J. Pineau, "Learning an unreferenced metric for online dialogue evaluation," in Proc. 58th Annu. Meeting Assoc. Comput. Linguistics. Online: Association for Computational Linguistics, Jul. 2020, pp. 2430-2441.
  9. S. Ghazarian, R. Weischedel, A. Galstyan, and N. Peng, "Predictive En- gagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue Systems," Proc. AAAI Conf. Artif. Intell., vol. 34, no. 5, pp. 7789-7796, 2020.
  10. R. Caruana, "Multitask learning," Mach. Learn., vol. 28, no. 1, pp. 41-75, 1997.
  11. A. See, S. Roller, D. Kiela, and J. Weston, " What makes a good con- versation? how controllable attributes affect human judgments," in Proc. 2019 Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technol., vol. 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 1702-1723.
  12. N. Dziri, E. Kamalloo, K. Mathewson, and O. Zaiane, "Evaluating Co- herence in Dialogue Systems Using Entailment," in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technol., vol. 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Com- putational Linguistics, Jun. 2019, pp. 3806-3812.
  13. A. Venkatesh et al., "On evaluating and comparing open domain dialog systems," 2018, arXiv:1801.03625. · arXiv
  14. C. Zhang, L. F. D'Haro, R. E. Banchs, T. Friedrichs, and H. Li, "Deep AM-FM: Toolkit for automatic dialogue evaluation," in Conversational Dialogue Syst. Decade. Singapore, Springer, 2020, pp. 53-69, doi: 10.1007/978-981-15-8395-7_5. · DOI
  15. L. F. D'Haro, R. E. Banchs, C. Hori, and H. Li, "Automatic evaluation of end-to-end dialog systems with adequacy-fluency metrics," Comput. Speech Lang., vol. 55, pp. 200-215, 2019, doi: 10.1016/j.csl.2018.12.004. · DOI
  16. P. Xu et al., "A cross-domain transferable neural coherence model," in Proc. 57th Annu. Meeting Assoc. Comput. Linguistics. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 678-687.
  17. J. Li, M. Galley, C. Brockett, G. Spithourakis, J. Gao, and B. Dolan, "A persona-based neural conversation model," in Proc. 54th Ann. Meeting Assoc. Comput. Linguistics (Volume 1: Long Papers), 2016, pp. 994-1003.
  18. S. Zhang, E. Dinan, J. Urbanek, A. Szlam, D. Kiela, and J. Weston, "Personalizing dialogue agents: I have a dog, do you have pets too?," in Proc. 56th Annual Meeting Assoc. Comput. Linguistics (Volume 1: Long Papers), 2018, pp. 2204-2213.
  19. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "Bleu: A method for automatic evaluation of machine translation," in Proc. 40th Annu. Meeting Assoc. Comput. Linguistics. Philadelphia, Pennsylvania, USA: Associa- tion for Computational Linguistics, Jul. 2002, pp. 311-318.
  20. M. Denkowski and A. Lavie, "Meteor universal: Language specific trans- lation evaluation for any target language," in Proc. 9th Workshop Statist. Mach. Transl.. Baltimore, Maryland, USA: Association for Computational Linguistics, Jun. 2014, pp. 376-380.
  21. C.-Y. Lin, "ROUGE: A Package for Automatic Evaluation of Summaries," in Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74-81.
  22. E. Dinan et al., "The second conversational intelligence challenge (ConvAI2)," The Springer Ser. Challenges Mach. Learn., pp. 187-208, Nov. 2019.
  23. V. Rus and M. Lintean, "An optimal assessment of natural language student input using word-to-word similarity metrics," in Int. Conf. Intell. Tutoring Syst.. Springer, 2012, pp. 675-676.
  24. J. Mitchell and M. Lapata, "Vector-based models of semantic composi- tion," in Proc. ACL-08: HLT, 2008, pp. 236-244.
  25. G. Forgues, J. Pineau, J.-M. Larchevêque, and R. Tremblay, "Bootstrap- ping dialog systems with word embeddings," in Nips, Modern Mach. Learn. Natural Lang. Process. Workshop, vol. 2, 2014.
  26. J. Pennington, R. Socher, and C. D. Manning, "Glove: Global vectors for word representation," in Proc. 2014 Conf. Empirical Methods Natural Lang. Process. (EMNLP), 2014, pp. 1532-1543.
  27. P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, "Enriching word vectors with subword information," Trans. Assoc. Comput. Linguistics, vol. 5, pp. 135-146, 2017, doi: 10.1162/tacl_a_00051. · DOI · Paper page
  28. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, "Bertscore: Evaluating text generation with bert," in 8th Int. Conf. Learn. Represent., 2020.
  29. T. Sellam, D. Das, and A. Parikh, "BLEURT: Learning robust metrics for text generation," in Proc. 58th Annu. Meeting Assoc. Comput. Linguistics. Online: Association for Computational Linguistics, Jul. 2020, pp. 7881- 7892.
  30. M. Li, J. Weston, and S. Roller, "Acute-eval: Improved dialogue eval- uation with optimized questions and multi-turn comparisons," 2019, arXiv:1909.03087. · arXiv
  31. Y. Liu et al., "A robustly optimized bert pretraining approach," 2019, arXiv:1907.11692. · arXiv
  32. A. Halliday Michael and H. Ruqaiya, "Cohesion in English," vol. 9, 2014.
  33. P. W. Foltz, W. Kintsch, and T. K. Landauer, "The measurement of textual coherence with latent semantic analysis," Discourse Processes, vol. 25, no. 2-3, pp. 285-307, 1998.
  34. J. Li, R. Li, and E. Hovy, "Recursive deep models for discourse parsing," in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), 2014, pp. 2061-2069.
  35. Z. Li, C. Niu, F. Meng, Y. Feng, Q. Li, and J. Zhou, "Incremental trans- former with deliberation decoder for document grounded conversations," in Proc. 57th Annu. Meeting Assoc. Comput. Linguistics. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 12-21.
  36. Q. Chen, X. Zhu, Z.-H. Ling, S. Wei, H. Jiang, and D. Inkpen, "Enhanced LSTM for Natural Language Inference," in Proc. 55th Annu. Meeting Assoc. Comput. Linguistics (Volume 1: Long Papers). Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 1657-1668.
  37. A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes, "Super- vised learning of universal sentence representations from natural language inference data," in Proc. Conf. Empirical Methods Natural Lang. Pro- cess.. Copenhagen, Denmark: Association for Computational Linguistics, Sep.. 2017, pp. 670-680.
  38. A. Ghandeharioun et al., "Approximating interactive human evaluation with self-play for open-domain dialog systems," in Proc. Adv. Neural Inf. Process. Syst., 2019, pp. 13658-13669.
  39. R. E. Banchs, L. F. D'Haro, and H. Li, "Adequacy-fluency metrics: Eval- uating MT in the continuous space model framework," IEEE/ACM Trans. Audio, Speech Lang. Process., vol. 23, no. 3, pp. 472-482, Mar. 2015.
  40. T. Evgeniou and M. Pontil, "Regularized multi-task learning," in Proc. Tenth ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, ser. KDD '04. New York, NY, USA: Association for Computing Machinery, 2004, p. 109-117.
  41. R. K. Ando and T. Zhang, "A framework for learning predictive structures from multiple tasks and unlabeled data," J. Mach. Learn. Res., vol. 6, pp. 1817-1853, Dec. 2005.
  42. A. Kumar and H. Daume III, "Learning task grouping and overlap in multi-task learning," 2012, pp. 1723-1730.
  43. X. Liu, J. Gao, X. He, L. Deng, K. Duh, and Y.-y. Wang, "Representation learning using multi-task deep neural networks for semantic classification and information retrieval," in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Hum. Lang. Technol.. Denver, Colorado: Association for Computational Linguistics, May-Jun. 2015, pp. 912-921.
  44. M.-T. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and L. Kaiser, "Multi-task sequence to sequence learning," 2015, arXiv:1511.06114. · arXiv
  45. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "Albert: A lite BERT for self-supervised learning of language representa- tions," in 8th Int. Conf. Learn. Representations, 2020.
  46. R. Collobert and J. Weston, "A unified architecture for natural language processing: Deep neural networks with multitask learning," in Proc. 25th Int. Conf. Mach. Learn., 2008, pp. 160-167.
  47. R. Sennrich, B. Haddow, and A. Birch, "Neural machine translation of rare words with subword units," in Proc. 54th Annual Meeting Assoc. Comput. Linguistics (Volume 1: Long Papers), 2016, pp. 1715-1725.
  48. A. Vaswani et al., "Attention is all you need," in Adv. Neural Inf. Process. Syst., 2017, pp. 5998-6008.
  49. D. Hendrycks and K. Gimpel, "Gaussian error linear units (gelus)," 2016, arXiv:1606.08415. · arXiv
  50. S. Sato, R. Akama, H. Ouchi, J. Suzuki, and K. Inui, "Evaluating di- alogue generation systems via response selection," in Proc. 58th Annu. Meeting Assoc. Comput. Linguistics. Online: Assoc. Comput. Linguistics, Jul. 2020, pp. 593-599.
  51. O. Vinyals and Q. Le, "A neural conversational model," 2015, arXiv:1506.05869. · arXiv
  52. E. M. Smith, M. Williamson, K. Shuster, J. Weston, and Y.-L. Boureau, "Can you put it all together: Evaluating conversational agents" ability to blend skills," in Proc. 58th Annu. Meeting Assoc. Comput. Linguistics. Online: Assoc. Comput. Linguistics, Jul. 2020, pp. 2021-2030.
  53. C. Hori et al., "Overview of the sixth dialog system technology chal- lenge: DSTC6," Comput. Speech Lang., vol. 55, pp. 1-25, 2019, doi: 10.1016/j.csl.2018.09.004. · DOI
  54. M. Galley, C. Brockett, X. Gao, J. Gao, and B. Dolan, "Grounded response generation task at DSTC7," in Proc. AAAI Dialog Syst. Tech. Challenges Workshop, 2019.
  55. M. Ghazvininejad et al., "A knowledge-grounded neural conversation model," in Proc. AAAI Conf. Artif. Intell., vol. 32, pp. 5110-5117, 2018.
  56. A. H. Miller et al., "Parlai: A dialog research software platform," in Proc. Conf. Empirical Methods Nat. Lang. Process.: Syst. Demonstrations, 2017, pp. 79-84, 2017.
  57. K. Sinha, P. Parthasarathi, J. Wang, R. Lowe, W. L. Hamilton, and J. Pineau, "Learning an unreferenced metric for online dialogue evaluation," in Proc. 58th Ann. Meeting Assoc. Comput. Linguistics, 2020, pp. 2430-2441.
  58. P. Gupta, S. Mehri, T. Zhao, A. Pavel, M. Eskenazi, and J. P. Bigham, "Investigating evaluation of open-domain dialogue systems with human generated multiple references," in Proc. 20th Annu. SIGDIAL Meeting Discourse Dialogue, 2019, pp. 379-391.
  59. I. Kulikov, A. Miller, K. Cho, and J. Weston, "Importance of search and evaluation strategies in neural dialogue modeling," in Proc. 12th Int. Conf. Natural Lang. Gener., Tokyo, Japan: Association for Computational Linguistics, Oct.-Nov. 2019, pp. 76-87.
  60. Chen Zhang (Student Member, IEEE) is currently a third year Ph.D. candidate with Electrical & Com- puter Engineering Department of National University of Singapore (NUS) under the Bosch-NUS Indus- trial Ph.D. Program. His research interests include automatic dialogue evaluation, machine learning and natural language processing.
  61. Grandee Lee (Student Member, IEEE) received the B.Eng. (Highest Distinction) degree in electrical en- gineering from the National University of Singapore (NUS), Singapore in 2017. He is currently a research scholar working towards the Ph.D. degree with NUS. His current research interests include cross-lingual language modeling and representation, and dialogue system evaluation.

Cited by

Sorted by citations

  • Survey of Hallucination in Natural Language Generation

    Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works

    “Recently, several works have proposed evaluation metrics for measuring consistency, such as using natural language inference (NLI) [40, 202], training learnable evaluation metrics [225], or releasing an additional test set for coherence [10].”
    Background

Authors
Zhang, C., Lee, G., D'Haro, L. F., & Li, H.
Affiliations
National University of Singapore (SG)
ORCID
Venue
IEEE/ACM Transactions on Audio, Speech, and Language Processing
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2021
Volume
29
Pages
2502-2516
ISSN
  • 2329-9290 (ISSN-L)
  • 2329-9304
DOI
10.1109/taslp.2021.3074012
OpenAlex
W3155737070
Cited by
21
References
10
Citations by year
2026: 4 · 2025: 1 · 2023: 9 · 2022: 4 · 2021: 3
FWCI
1.41
Type
journal article
Language
en
Open access
Licence
Locations
IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion, cc-by)
Funders
  • National Research Foundation Singapore
  • AI Singapore Programme (AISG-GC-2019-002)
  • Human-Robot Interaction Phase 1 (19225 00054)
  • National Research Foundation
  • National Robotics Programme
  • Human Robot Collaborative AI for AME (A18A2b0046)
  • NRF Singapore
  • Robert Bosch
  • EDB.s Industrial Postgraduate Programme II
  • AMIC (TIN2017-85854-C4-4-R)
  • CAVIAR (TEC2017-84593-C2-1-R)
  • European Commission
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
Keywords
dialogue evaluation, reference-free evaluation, multi-task learning, language fluency, logical consistency, chatbots

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review