Predicting Interaction Quality Aspects Using Level-Based Scores for Conversational Agents
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
In order to improve human-agent interaction, it is essential to have good measures of interaction quality. We define interaction quality based on multiple aspects, including usability, likability and perceived conversation quality as subjective measures, and interaction length, completion rate and frequency of unrecognized utterances as objective measures. Determining necessary improvements to a conversational agent is a non-trivial task, because it is difficult to infer from an evaluation of the agent as a whole, which aspects of the agent need to be improved to raise the interaction quality. In this paper, we propose a scoring system for task-oriented conversational agents to predict aspects of interaction quality and to guide an iterative improvement process. Our scoring system does not provide a single score, but leverages structural features of the dialogue management approach and assigns a score on three levels: the utterance, dialogue move, and genre level. Using the agent's scores on separate levels to predict the interaction quality allows making targeted improvements to the conversational agent. In order to evaluate our scoring system, we apply it over the course of multiple crowdsourcing pilot studies, using a recipe recommendation agent. We evaluate the obtained scores in regard to their ability to predict selected objective and subjective interaction quality aspects, as well as their suitability for making informed decisions about necessary improvements.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ashfaq, M., Yun, J., Yu, S., & Loureiro, S. M. C. (2020) · Telematics and Informatics · doi:10.1016/j.tele.2020.101473 · Cited by 987 · cited 1×
“Most metrics proposed in the literature for evaluating the quality of conversational agents focus on the general quality of a dialogue and response generation [2, 19, 21].”
BackgroundTowards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · cited 2×
“End-to-end neural network approaches offer the simplicity of a single learned model, whereas other approaches typically rely on more complex frameworks, expert knowledge, and manual effort [1].”
BackgroundLiu, C., Lowe, R., Serban, I. V., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08023 · Cited by 603 · cited 1×
“However, there is still an overall lack of guidance regarding the evaluation of the quality of conversational agents [21], while unsupervised evaluation methods, have shown to be only weakly correlated with human judgement [18].”
BackgroundA Survey on Dialogue Systems: Recent Advances and New Frontiers
Chen, H., Liu, X., Yin, D., & Tang, J. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3166054.3166058 · Cited by 579 · cited 1×
“Conversational agents range from task-oriented and closed-domain to chit-chat and open-domain systems [9].”
BackgroundPARADISE: a framework for evaluating spoken dialogue agents
Walker, M. A., Litman, D. J., Kamm, C. A., & Abella, A. (1997) · Proceedings of the 35th annual meeting on Association for Computational Linguistics - · doi:10.3115/976909.979652 · Cited by 537
Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset
Rastogi, A., Zang, X., Sunkara, S., Gupta, R., & Khaitan, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6394 · Cited by 450
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction
Larson, S., Mahendran, A., Peper, J. J., Clarke, C., Lee, A., Hill, P., Kummerfeld, J. K., Leach, K., Laurenzano, M. A., Tang, L., et al. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1131 · Cited by 392 · cited 1×
“Although it may not always be easy to delineate what is in and out of scope, for our purposes we assume that such a binary decision can be made [17].”
BackgroundThe Information State Approach to Dialogue Management
Traum, D. R., & Larsson, S. (2003) · Text, Speech and Language Technology · doi:10.1007/978-94-010-0019-2_15 · Cited by 342 · cited 1×
“The notion of dialogue act or intent is seen in a diversity of task-based approaches to conversational AI and connected to different types of dialogue managers, both probabilistic and handcrafted (see e.g.: [5, 22, 25, 26]).”
BackgroundTowards a tool for the Subjective Assessment of Speech System Interfaces (SASSI)
HONE, K. S., & GRAHAM, R. (2000) · Natural Language Engineering · doi:10.1017/s1351324900002497 · Cited by 305 · cited 1×
“5) from the SASSI questionnaire [12].”
Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems
Thomson, B., & Young, S. (2010) · Computer Speech & Language · doi:10.1016/j.csl.2009.07.003 · Cited by 303
Survey on evaluation methods for dialogue systems
Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., & Cieliebak, M. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-020-09866-x · Cited by 215 · cited 2×
“For task-oriented agents, metrics such as task completion and user satisfaction can be used to evaluate overall dialogue quality [10].”
BackgroundEvaluating Quality of Chatbots and Intelligent Conversational Agents
Radziwill, N., & Benton, M. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.04579 · Cited by 181 · cited 2×
“Most metrics proposed in the literature for evaluating the quality of conversational agents focus on the general quality of a dialogue and response generation [2, 19, 21].”
BackgroundProceedings of the Sixth International Joint Conference on Natural Language Processing
Mitkov, R., & Park, J. C. (2013) · International Joint Conference on Natural Language Processing · Cited by 147
Rasa: Open Source Language Understanding and Dialogue Management
Bocklisch, T., Faulkner, J., Pawlowski, N., & Nichol, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.05181 · Cited by 137 · cited 2×
“The notion of dialogue act or intent is seen in a diversity of task-based approaches to conversational AI and connected to different types of dialogue managers, both probabilistic and handcrafted (see e.g.: [5, 22, 25, 26]).”
BackgroundMethodUnsupervised Evaluation of Interactive Dialog with DialoGPT
Mehri, S., & Eskenazi, M. (2020) · Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2020.sigdial-1.28 · Cited by 111
Dialogue Management in Conversational Systems: A Review of Approaches, Challenges, and Opportunities
Brabra, H., Báez, M., Benatallah, B., Gaaloul, W., Bouguelia, S., & Zamanirad, S. (2021) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3086565 · Cited by 94 · cited 1×
“There exist multiple approaches to dialogue management (DM) [7, 11].”
BackgroundApproaches for Dialog Management in Conversational Agents
Harms, J.-G., Kucherbaev, P., Bozzon, A., & Houben, G.-J. (2019) · IEEE Internet Computing · doi:10.1109/mic.2018.2881519 · Cited by 78 · cited 1×
“There exist multiple approaches to dialogue management (DM) [7, 11].”
BackgroundDialogue Act Annotation with the ISO 24617-2 Standard
Bunt, H., Petukhova, V. V., Traum, D. R., & Alexandersson, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-42816-1_6 · Cited by 62 · cited 1×
“This is a common approach to represent conversations, and used in both a standardized dialogue act annotation scheme [8] and intent schemes catered to specific tasks (see Table 1 for the scheme we used).”
MethodSmith, E., Hsu, O., Qian, R., Roller, S., Boureau, Y.-L., & Weston, J. (2022) · Proceedings of the 4th Workshop on NLP for Conversational AI · doi:10.18653/v1/2022.nlp4convai-1.8 · Cited by 40
HHAI2022: Augmenting Human Intellect
Amsterdam, I. C. O. H. H.-A. I. 2., Stefan Schlobach, A., Pérez‐Ortiz, M., & Tielman, M. L. (2022) · Frontiers in artificial intelligence and applications · doi:10.3233/faia354 · Cited by 20
R-cube: A dialogue agent for restaurant recommendation and reservation
Kim, S., & Banchs, R. E. (2014) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2014.7041732 · Cited by 18
Revisiting Habitability in Conversational Systems
Kocaballi, A. B., Coiera, E. W., & Berkovsky, S. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3383014 · Cited by 8
A cloud-based middleware for multi-modal interaction services and applications
Avenoğlu, B., Koeman, V. J., & Hindriks, K. V. (2022) · Journal of Ambient Intelligence and Smart Environments · doi:10.3233/ais-220161 · Cited by 5
Self-citationSUPPLE: A Dialogue Management Approach Based on Conversation Patterns
Kunneman, F., & Hindriks, K. V. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-5538-9_14 · Cited by 1 · cited 2×
“We assume the Intent-Entity-Context-Recognition (IECR) framework [20] and conversational patterns as a framework for managing dialogue [15, 16] for developing conversational agents in our approach.”
BackgroundSelf-citationReconstructive Genres of Everyday Communication
Bergmann, J. R., & Luckmann, T. (1995) · Aspects of Oral Communication · doi:10.1515/9783110879032.289 · cited 1×
“Conversational genres can be defined as “routinized communicative solutions to recurrent communicative problems of a speech community” [4].”
Background
Research works printed but not linked: 2
- Simone Borsci, Alessio Malizia, Martin Schmettow, Frank Van Der Velde, Gunay Tariverdiyeva, Divyaa Balaji, and Alan Chamberlain. 2022. The Chatbot usability scale: The design and pilot of a usability scale for interaction with AI-based conversational agents. Personal and ubiquitous computing 26, 1 (2022), 95--119.
- Yin Jiang Zhao, Yan Ling Li, and Min Lin. 2019. A review of the research on dialogue management of task-oriented systems. In Journal of Physics: Conference Series, Vol. 1267. IOP Publishing, 012025.
Cited by
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74
Evaluating LLM-based Agents for Multi-turn Conversations: A Survey
Guan, S., Wang, J., 渡辺, 将., Zhu, B., Lou, J., & Xiong, H. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3793671 · Cited by 87
The Illusion of Success: Procedure-Aware Evaluation is Critical for LLM Agents
Cao, H., Driouich, I., & Thomas, E. M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37682-4_29 · Cited by 32
林, 雅. (2025) · Advances in Psychology · doi:10.12677/ap.2025.152060 · Cited by 1
Metadata
- Authors
- Reimann, M. M., Oertel, C., Kunneman, F. A., & Hindriks, K. V.
- Affiliations
- Vrije Universiteit Amsterdam (NL)
- Delft University of Technology (NL)
- Venue
- Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents
- Publisher
- ACM
- Published
- 2023-09-19
- Pages
- 1-8
- OpenAlex
- W4390094929
- Cited by
- 4
- References
- 16
- Citations by year
- 2026: 2 · 2025: 2
- FWCI
- 0.71
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Intelligent Virtual Agents (IVA) (publishedVersion, cc-by)
- VU Research Portal (submittedVersion, cc-by)
- Funders
- Hybrid Intelligence Centre (024.004.022)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- chatbots, interaction quality, scoring system, dialogue management, task-oriented dialogue, usability, likeability, recipe recommendation, crowdsourcing evaluation, iterative improvement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

