Automatic Online Evaluation of Intelligent Assistants
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Voice-activated intelligent assistants, such as Siri, Google Now, and Cortana, are prevalent on mobile devices. However, it is challenging to evaluate them due to the varied and evolving number of tasks supported, e.g., voice command, web search, and chat. Since each task may have its own procedure and a unique form of correct answers, it is expensive to evaluate each task individually. This paper is the first attempt to solve this challenge. We develop consistent and automatic approaches that can evaluate different tasks in voice-activated intelligent assistants. We use implicit feedback from users to predict whether users are satisfied with the intelligent assistant as well as its components, i.e., speech recognition and intent classification. Using this approach, we can potentially evaluate and compare different tasks within and across intelligent assistants ac-cording to the predicted user satisfaction rates. Our approach is characterized by an automatic scheme of categorizing user-system interaction into task-independent dialog actions, e.g., the user is commanding, selecting, or confirming an action. We use the action sequence in a session to predict user satisfaction and the quality of speech recognition and intent classification. We also incorporate other features to further improve our approach, including features derived from previous work on web search satisfaction prediction, and those utilizing acoustic characteristics of voice requests. We evaluate our approach using data collected from a user study. Results show our approach can accurately identify satisfactory and unsatisfactory sessions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Friedman, J., Hastie, T., & Tibshirani, R. (2000) · The Annals of Statistics · doi:10.1214/aos/1016218223 · Cited by 6,884
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841 · cited 2×
“Traditionally search system have been evaluated using retrieval metrics such as MAP and nDCG [14], where a collection of document s, queries and human labeled relevance judgments are used to evaluate search system performance.”
Method
Cited by
A Theoretical Framework for Conversational Search
Radlinski, F., & Craswell, N. (2017) · Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval · doi:10.1145/3020165.3020183 · Cited by 439
Evaluating and Informing the Design of Chatbots
Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196735 · Cited by 376
Talk to me: Exploring user interactions with the Amazon Alexa
Lopatovska, I., Rink, K., Knight, I., Raines, K., Cosenza, K., Williams, H., Sorsche, P., Hirsch, D., Li, Q., & Martinez, A. (2019) · Journal of Librarianship and Information Science · doi:10.1177/0961000618759414 · Cited by 298
Brill, T. M., Munoz, L., & Miller, R. J. (2019) · Journal of Marketing Management · doi:10.1080/0267257x.2019.1687571 · Cited by 291
Understanding adoption of intelligent personal assistants: A parasocial relationship perspective
Han, S., & Yang, H. (2018) · Industrial Management & Data Systems · doi:10.1108/imds-05-2017-0214 · Cited by 289
Understanding User Satisfaction with Intelligent Assistants
Kiseleva, J., Williams, K., Jiang, J., Hassan Awadallah, A., Crook, A. C., Zitouni, I., & Anastasakos, T. (2016) · Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval · doi:10.1145/2854946.2854961 · Cited by 150
Self-citationSearching by Talking: Analysis of Voice Queries on Mobile Web Search
Guy, I. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911525 · Cited by 140
FarmChat: A Conversational Agent to Answer Farmer Queries
Jain, M., Kumar, P., Bhansali, I., Liao, Q. V., Truong, K., & Patel, S. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3287048 · Cited by 117
Liao, Q. V., Mas-ud Hussain, M., Chandar, P., Davis, M., Khazaeni, Y., Crasso, M. P., Wang, D., Muller, M., Shami, N. S., & Geyer, W. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173577 · Cited by 116
Predicting User Satisfaction with Intelligent Assistants
Kiseleva, J., Williams, K., Hassan Awadallah, A., Crook, A. C., Zitouni, I., & Anastasakos, T. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911521 · Cited by 108
Self-citationThe Characteristics of Voice Search: Comparing Spoken with Typed-in Mobile Web Search Queries
Guy, I. (2018) · ACM Transactions on Information Systems · doi:10.1145/3182163 · Cited by 71
Health and Fitness Apps for Hands-Free Voice-Activated Assistants: Content Analysis
Chung, A. E., Griffin, A. C., Selezneva, D., & Gotz, D. (2018) · JMIR mHealth and uHealth · doi:10.2196/mhealth.9705 · Cited by 70
“Previous VAA studies have focused only on research-grade VAAs that were not commercially available, were not hands-free, and were primarily focused on usability and design of the devices rather than the apps that could be used with VAAs [1,2,6,7,10,26,27].”
BackgroundUnderstanding the Adoption of Voice Activated Personal Assistants
Coskun-Setirek, A., & Mardikyan, S. (2017) · International Journal of E-Services and Mobile Applications · doi:10.4018/ijesma.2017070101 · Cited by 44
Mavrina, L., Szczuka, J., Strathmann, C., Bohnenkamp, L. M., Krämer, N., & Kopp, S. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.791704 · Cited by 43
User interactions with “Alexa” in public academic space
Lopatovska, I., & Oropeza, H. (2018) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2018.14505501034 · Cited by 41
Barbosa, N. M., Hayes, J., Kaushik, S., & Wang, Y. (2022) · ACM Transactions on Accessible Computing · doi:10.1145/3519032 · Cited by 28
“, [32]), and as a direction worth further exploration (e.”
BackgroundUsing LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction
Kim, H., Choi, Y., Yang, T., Lee, H., Park, C., Lee, Y., Kim, J. Y., & Kim, J. (2024) · arXiv · doi:10.48550/arxiv.2407.13166
Metadata
- Authors
- Jiang, J., Hassan Awadallah, A., Jones, R., Ozertem, U., Zitouni, I., Gurunath Kulkarni, R., & Khan, O. Z.
- Venue
- Proceedings of the 24th International Conference on World Wide Web
- Published
- 2015-05-18
- Pages
- 506-516
- OpenAlex
- W2189217339
- Cited by
- 148
- References
- 2
- FWCI
- 12.12
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.