CoChat

The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index; 19 of the 68 sources its claims cite were checked for support.

A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures

Open access
Baughan, Amanda, Wang, Xuezhi, Liu, Ariel, Mercurio, Allison, Chen, Jilin, Ma, Xiao · 2023 · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Abstract

Despite huge gains in performance in natural language understanding via large language models in recent years, voice assistants still often fail to meet user expectations. In this study, we conducted a mixed-methods analysis of how voice assistant failures affect users’ trust in their voice assistants. To illustrate how users have experienced these failures, we contribute a crowdsourced dataset of 199 voice assistant failures, categorized across 12 failure sources. Relying on interview and survey data, we find that certain failures, such as those due to overcapturing users’ input, derail user trust more than others. We additionally examine how failures impact users’ willingness to rely on voice assistants for future tasks. Users often stop using their voice assistants for specific tasks that result in failures for a short period of time before resuming similar usage. We demonstrate the importance of low stakes tasks, such as playing music, towards building trust after failures.

Overview

CoChat Summary

## TL;DR

This mixed-methods study shows that not all voice assistant failures damage trust equally: response failures and some perception failures hurt trust more, while ambiguity- and spurious-trigger-related failures are less damaging. The authors also find that people often rebuild trust by returning first to low-stakes tasks like playing music, and they release a dataset of 199 real-world failures.

## Key findings

  • The authors crowdsourced a dataset of 199 voice assistant failures from 107 unique crowd workers, spanning 12 failure categories.
  • In the survey, failure type significantly affected trust in all three dimensions: ability (F(3, 2656.78) = 17.17, p < .001), benevolence (F(3, 2711.23) = 8.87, p < .001), and integrity (F(3, 2772.09) = 20.56, p < .001).
  • Response failures were most damaging overall; compared with attention failures, they reduced trust in ability (m = 43.6), benevolence (m = 52.5), and integrity (m = 57.3). Within response failures, incorrect action was the reference and was described as the most damaging to integrity.
  • Some failures were less harmful: ambiguity maintained the highest trust, with higher confidence in ability (m = 62.6) and benevolence (m = 67.9); spurious triggers were also relatively benign, with higher confidence in ability (m = 57.9) and benevolence (m = 64.2) than most other failure types.
  • Users often paused use for the failed task, then resumed later; trust was often rebuilt through simple tasks such as playing music, and future-task trust was highest for playing a song (m = 3.29), lower for texting a coworker (m = 2.34), and lowest for transferring money (m = 1.56).

## Methods

  • Mixed-methods design: the paper combined 12 remote interviews, a crowdsourced failure dataset, and a survey to study how voice assistant failures affect trust and future intended use.
  • Dataset collection: 107 U.S.-based Mechanical Turk workers each submitted up to three failures; after cleaning and re-coding, the finalized dataset contained 199 failures across 12 categories. The authors adapted a taxonomy from Hong et al. and had two raters iteratively code 101 submitted failures with 70% interrater agreement.
  • Survey and analysis: the survey used 268 Mechanical Turk participants and presented 12 failure scenarios drawn from the dataset. The main analysis used linear mixed models / mixed-linear regressions and ordinal mixed models, with participant ID as a random effect and general trust tendency as a control.

## Limitations

  • The paper states that the study relied on retrospective recall for the dataset and interviews rather than in-situ observation, which introduces recall bias.
  • The survey used hypothetical scenarios, not observed interactions; the authors note that future work could use ESM or diary studies to capture effects more directly.
  • The sample consisted of frequent voice assistant users, who may forgive errors more easily than other populations; this may limit generalizability.
  • The authors did not study embodied conversational agents or text-based conversational agents, so the findings may not generalize to those interfaces.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3544548.3581152

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 To analyze the impact of voice assistant failures on user trust, we used a mixed-methods approach, including interviews and a survey. As part of the materials for our survey, we crowdsourced 199 failures from 107 voice assistant users, and include this dataset as part of our contributions.
Figure 1 To analyze the impact of voice assistant failures on user trust, we used a mixed-methods approach, including interviews and a survey. As part of the materials for our survey, we crowdsourced 199 failures from 107 voice assistant users, and include this dataset as part of our contributions.From the PDF
Figure 2 Average scores across the three dimensions of trust (ability, benevolence, and integrity) by failure type. Participants expressed higher confidence across three trust dimensions after encountering attention and understanding failures, compared to perception and response. Error bars display the confidence interval.
Figure 2 Average scores across the three dimensions of trust (ability, benevolence, and integrity) by failure type. Participants expressed higher confidence across three trust dimensions after encountering attention and understanding failures, compared to perception and response. Error bars display the confidence interval.From the PDF
Image 3
From the PDF

Verification

Claims the paper makes

7 Valid · 1 Not valid · 30 Not checked · 130 Own results and interpretations

111 more claims

Create a free account to see all 168 claims and to verify the 13 references still open.

See all claims — free account
  • Probably not a substantial claim. 30
  • It may state two separate claims as one. 27
  • The quote only partly states this claim. 5

Create a free account to see these claims.

Citations

Cites

Show all 74 references

Research works printed but not linked: 2

  1. 10.18653/v1 · doi:10.18653/v1
  2. 10.5555/3524938.3525579 · doi:10.5555/3524938.3525579

Cited by

See all 32

Metadata

Authors
Baughan, A., Wang, X., Liu, A., Mercurio, A., Chen, J., & Ma, X.
Affiliations
  • University of Washington (US)
  • Google (United States) (US)
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-16
OpenAlex
W4322831937
Cited by
32
References
67
Citations by year
2026: 11 · 2025: 14 · 2024: 7
FWCI
5.67
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
voice assistants, user trust, task abandonment, music playback, large language models, natural language understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures. https://app.cochat.ai/papers/10.1145/3544548.3581152
The paper
Baughan, A., Wang, X., Liu, A., Mercurio, A., Chen, J., & Ma, X. (2023). A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3544548.3581152

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.