CoChat
Sign Up
Cited by 150Usability and User Interface DesignSpeech and dialogue systemsAI in Service Interactions

Understanding User Satisfaction with Intelligent Assistants

Kiseleva, Julia, Williams, Kyle, Jiang, Jiepu, Hassan Awadallah, Ahmed, Crook, Aidan C., Zitouni, Imed, Anastasakos, Tasos·2016·Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval
DOI 10.1145/2854946.2854961
Open this paper in CoChat — free account

Abstract

Voice-controlled intelligent personal assistants, such as Cortana, Google Now, Siri and Alexa, are increasingly becoming a part of users' daily lives, especially on mobile devices. They introduce a significant change in information access, not only by introducing voice control and touch gestures but also by enabling dialogues where the context is preserved. This raises the need for evaluation of their effectiveness in assisting users with their tasks. However, in order to understand which type of user interactions reflect different degrees of user satisfaction we need explicit judgements. In this paper, we describe a user study that was designed to measure user satisfaction over a range of typical scenarios of use: controlling a device, web search, and structured search dialogue. Using this data, we study how user satisfaction varied with different usage scenarios and what signals can be used for modeling satisfaction in the different scenarios. We find that the notion of satisfaction varies across different scenarios, and show that, in some scenarios (e.g. making a phone call), task completion is very important while for others (e.g. planning a night out), the amount of effort spent is key. We also study how the nature and complexity of the task at hand affects user satisfaction, and find that preserving the conversation context is essential and that overall task-level satisfaction cannot be reduced to query-level satisfaction alone. Finally, we shed light on the relative ef…

Integrity signals

Registry

Crossref · 10.1145/2854946.2854961

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Kiseleva, J., Williams, K., Jiang, J., Hassan Awadallah, A., Crook, A. C., Zitouni, I., & Anastasakos, T.
Affiliations
  • Eindhoven University of Technology (NL)
  • Pennsylvania State University (US)
  • University of Massachusetts Amherst (US)
  • Microsoft, Redmond, WA, USA
  • Microsoft (United States) (US)
ORCID
Venue
Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval
Publisher
ACM
Published
2016-03-13
Pages
121-130
DOI
10.1145/2854946.2854961
OpenAlex
W2331728731
Cited by
150
References
2
Citations by year
2026: 3 · 2025: 7 · 2024: 5 · 2023: 16 · 2022: 15 · 2021: 24 · 2020: 25 · 2019: 15 · 2018: 17 · 2017: 19 · 2016: 4
FWCI
12.51
Type
conference paper
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
Conference on Human Information Interaction and Retrieval (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
Keywords
user satisfaction, virtual assistants, voice user interface, chatbots, task completion, effort, conversation context, task complexity, mobile devices, web search

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review