A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures
Open accessThe DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index; 19 of the 68 sources its claims cite were checked for support.
Abstract
Despite huge gains in performance in natural language understanding via large language models in recent years, voice assistants still often fail to meet user expectations. In this study, we conducted a mixed-methods analysis of how voice assistant failures affect users’ trust in their voice assistants. To illustrate how users have experienced these failures, we contribute a crowdsourced dataset of 199 voice assistant failures, categorized across 12 failure sources. Relying on interview and survey data, we find that certain failures, such as those due to overcapturing users’ input, derail user trust more than others. We additionally examine how failures impact users’ willingness to rely on voice assistants for future tasks. Users often stop using their voice assistants for specific tasks that result in failures for a short period of time before resuming similar usage. We demonstrate the importance of low stakes tasks, such as playing music, towards building trust after failures.
Overview
CoChat Summary
## TL;DR
This mixed-methods study shows that not all voice assistant failures damage trust equally: response failures and some perception failures hurt trust more, while ambiguity- and spurious-trigger-related failures are less damaging. The authors also find that people often rebuild trust by returning first to low-stakes tasks like playing music, and they release a dataset of 199 real-world failures.
## Key findings
- The authors crowdsourced a dataset of 199 voice assistant failures from 107 unique crowd workers, spanning 12 failure categories.
- In the survey, failure type significantly affected trust in all three dimensions: ability (F(3, 2656.78) = 17.17, p < .001), benevolence (F(3, 2711.23) = 8.87, p < .001), and integrity (F(3, 2772.09) = 20.56, p < .001).
- Response failures were most damaging overall; compared with attention failures, they reduced trust in ability (m = 43.6), benevolence (m = 52.5), and integrity (m = 57.3). Within response failures, incorrect action was the reference and was described as the most damaging to integrity.
- Some failures were less harmful: ambiguity maintained the highest trust, with higher confidence in ability (m = 62.6) and benevolence (m = 67.9); spurious triggers were also relatively benign, with higher confidence in ability (m = 57.9) and benevolence (m = 64.2) than most other failure types.
- Users often paused use for the failed task, then resumed later; trust was often rebuilt through simple tasks such as playing music, and future-task trust was highest for playing a song (m = 3.29), lower for texting a coworker (m = 2.34), and lowest for transferring money (m = 1.56).
## Methods
- Mixed-methods design: the paper combined 12 remote interviews, a crowdsourced failure dataset, and a survey to study how voice assistant failures affect trust and future intended use.
- Dataset collection: 107 U.S.-based Mechanical Turk workers each submitted up to three failures; after cleaning and re-coding, the finalized dataset contained 199 failures across 12 categories. The authors adapted a taxonomy from Hong et al. and had two raters iteratively code 101 submitted failures with 70% interrater agreement.
- Survey and analysis: the survey used 268 Mechanical Turk participants and presented 12 failure scenarios drawn from the dataset. The main analysis used linear mixed models / mixed-linear regressions and ordinal mixed models, with participant ID as a random effect and general trust tendency as a control.
## Limitations
- The paper states that the study relied on retrospective recall for the dataset and interviews rather than in-situ observation, which introduces recall bias.
- The survey used hypothetical scenarios, not observed interactions; the authors note that future work could use ESM or diary studies to capture effects more directly.
- The sample consisted of frequent voice assistant users, who may forgive errors more easily than other populations; this may limit generalizability.
- The authors did not study embodied conversational agents or text-based conversational agents, so the findings may not generalize to those interfaces.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
7 Valid · 1 Not valid · 30 Not checked · 130 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“People generally expect to have functional conversations with voice assistants [11].”
What the sources say
- Supportedcites [11] [11] Leigh Clark, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin Edwards, Brendan Spillane, Emer Gilmartin, Christine Murad, Cosmin Munteanu, Vincent Wade, and Benjamin R. Cowan. 2019. What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19) . Association for Computing Machinery, New York, NY, USA, 1-12. https://doi.org/10.1145/3290605.3300705 - source
The Discussion (p. 8) says participants conceptualised conversation with agents in almost purely transactional terms and described its qualities in highly functional terms. This supports the paraphrase that people generally expect functional conversations with voice assistants. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Standard NLP models are often evaluated on large datasets of coherent text-based questions and answers [48, 49] or paired written dialogue [68].”
What the sources say
- Supportedcites [48] [48] Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822 (2018). - source
The Introduction describes a large dataset of questions paired with text paragraphs, and §5.1 describes evaluating three model architectures on it. This citation is attached to the question-and-answer dataset portion of the sentence; the paired-dialogue portion has a separate citation. (LLM, conf=0.90)
- Supportedcites [49] [49] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016). - source
The citation is attached to the question-and-answer dataset portion of the sentence, not to paired written dialogue. The Introduction (p. 1) describes SQuAD as 107,785 question-answer pairs based on Wikipedia passages, and the Introduction (p. 2) reports model performance on it. (LLM, conf=0.90)
- Supportedcites [68] [68] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243 (2018). source
Citation [68] is attached to the paired-written-dialogue portion of the sentence, not the questions-and-answers portion. The Introduction describes a dataset of 164,356 utterances from paired crowdworkers, and §3.4 Evaluation calls predicting the next utterance from dialogue history a standard dialogue task. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We found that existing NLP datasets do not cover the wide breadth of possible conversational failure cases due to their emphasis on correct data for training.”
What the sources say
- Partly supportedcites [48] [48] Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822 (2018). - sourceExperimental — accuracy not yet measured
Section 3.1 (page unavailable) reports that some reading-comprehension datasets exclude unanswerable examples and that other negative examples can be easy to identify. Section 3.3 (page unavailable) says multiple-choice datasets may be less suited to training user-facing systems. These passages support a narrower gap in question-answering datasets, but not the claim about the breadth of conversational failure cases across NLP datasets or its stated cause of emphasizing correct training data. (LLM, conf=0.82)
- Could not verifycites [49] [49] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016). - sourceExperimental — accuracy not yet measured
Section 2 (pp. 2–3) surveys limitations of existing reading-comprehension and question-answering datasets, but does not discuss conversational failure cases. Section 3 (p. 4) describes collecting answerable question–answer pairs; it does not say that an emphasis on correct training data causes existing NLP datasets to miss a broad range of conversational failures. Searches for conversational failures, incorrect answers, and training-data limitations found no passage making that claim. (LLM, conf=0.74)
- Partly supportedcites [50] [50] Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics 7 (2019), 249-266. - sourceExperimental — accuracy not yet measured
The Abstract (p. 1) says existing reading-comprehension datasets lack conversational phenomena such as coreference and pragmatic reasoning, and the Introduction (p. 2) says large-scale datasets lack questions dependent on conversation history. These passages support a narrower gap in dataset coverage, but they do not establish a wide breadth of conversational failure cases or attribute the gap to an emphasis on correct training data. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [25] [25] Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019. Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In Proc. Interspeech 2019 . 1891-1895. https://doi.org/10.21437/Interspeech.2019-3079 - source
- Could not verifycites [57] [57] Yueming Sun and Yi Zhang. 2018. Conversational Recommender System. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR '18) . Association for Computing Machinery, [New York, NY, USA, 235-244. https://doi.org/10.1145/3209978.3210002](https://doi.org/10.1145/3209978.3210002) - sourceExperimental — accuracy not yet measured
Our copy of Sun, Zhang & Chen's CRM paper never addresses the coverage of conversational failure cases in existing NLP datasets. The nearest passages are §4.4 (p. 6) — the Yelp dataset has no dialogue utterances and 'to the best of our knowledge, there is no such available dataset,' so they crowd-sourced utterances assuming 'users are cooperative and always inform the facet values of the target' — and §4.2 (p. 5), which notes that launching an untrained dialogue system to real users 'is likely to fail' and that simulated users are the common bootstrap. §2.1 (p. 2) only observes that prior work 'focus on NLP challenges instead of commercial success metrics.' Searched for: conversational failure cases, error/noisy utterance coverage, dataset limitations, related-work dataset coverage — no passage states the attributed claim, and none contradicts it. (LLM, conf=0.70)
- Partly supportedcites [68] [68] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243 (2018). sourceExperimental — accuracy not yet measured
The Introduction and Related Work say existing dialogue datasets have limitations that contribute to problems such as inconsistent personality and weak long-term coherence. They do not say that NLP datasets fail to cover the wide breadth of possible conversational failure cases, or attribute such a gap to an emphasis on correct training data. Page numbers are unavailable in our copy. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Relying on interview and survey data, we find that certain failures, such as those due to overcapturing users' input, derail user trust more than others.”
- Also stated in: § 1 INTRODUCTION · plain
“We found that different types of voice assistant failures have a differential impact on trust.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To illustrate how users have experienced these failures, we contribute a crowdsourced dataset of 199 voice assistant failures, categorized across 12 failure sources.”
- Also stated in: § 1 INTRODUCTION · plain
“C1: Iterating on the existing taxonomy of NLP failures, we crowdsource a dataset of 199 failures users have experienced across 12 different sources of failure.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
111 more claims
Create a free account to see all 168 claims and to verify the 13 references still open.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 27
- The quote only partly states this claim. 5
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Qualitative Inquiry and Research Design: Choosing Among Five Approaches
Lewis, S. (2015) · Health Promotion Practice · doi:10.1177/1524839915580941 · Cited by 22,105
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 3×
“In the case of question answering, state-of-art models perform nearly at human-level for reading comprehension on standard benchmarks collected from Wikipedia [49].”
BackgroundClark, H. H. (1996) · doi:10.1017/cbo9780511620539 · Cited by 5,123
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · cited 1×
“in terms of people’s confdence in a machine’s ability to perform as expected, benevolence (well-meaning), and integrity to adhere to ethical standards [36] Broadly, past research has evaluated how various factors such as accuracy and errors afect people’s trust in algorithms [19, 20, 67].”
BackgroundKnow What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · cited 3×
“We first explored benchmark datasets in NLP, which contain a large number of either questions and answers [48–50], or conversational dialogue [25, 57, 68].”
BackgroundProceedings of the 2017 CHI Conference on Human Factors in Computing Systems
(2017) · doi:10.1145/3025453 · Cited by 2,207
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176 · cited 9×
“Sometimes after a failure, users will try to reformulate, simplify, or hyper-enunciate their commands as a way to continue using the device [31, 35, 43, 61].”
BackgroundResultInfluentialMariani, J. (2018) · HAL · Cited by 1,151
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 1×
“Within the past fve years, advancements in NLP have achieved huge gains in accuracy when tested against standard datasets [8, 18, 34, 56, 58, 60, 63], with state-ofthe-art accuracy in natural language processing models as high as 99% for certain tasks [8, 34].”
BackgroundCoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975
HotFlip: White-Box Adversarial Examples for Text Classification
Ebrahimi, J., Rao, A., Lowd, D., & Dou, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2006 · Cited by 922 · cited 1×
“For example, noisy channel and transcription failures can be modeled as small perturbations to the input, which is well researched [4, 21].”
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 1×
“Within the past fve years, advancements in NLP have achieved huge gains in accuracy when tested against standard datasets [8, 18, 34, 56, 58, 60, 63], with state-ofthe-art accuracy in natural language processing models as high as 99% for certain tasks [8, 34].”
BackgroundSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · arXiv · doi:10.48550/arxiv.1606.05250 · Cited by 678
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1×
“in terms of people’s confidence in a machine’s ability to perform as expected, benevolence (well-meaning), and integrity to adhere to ethical standards [36] Broadly, past research has evaluated how various factors such as accuracy and errors affect people’s trust in algorithms [19, 20, 67].”
BackgroundSalem, M., Lakatos, G., Amirabdollahian, F., & Dautenhahn, K. (2015) · Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2696454.2696497 · Cited by 494 · cited 2×
“[53] found that if a robot had faulty performance, this did not influence participants’ decisions to comply with its requests, but it did significantly affect their perceptions of the robot’s reliability and trustworthiness.”
BackgroundRelational agents: a model and implementation of building user trust
Bickmore, T., & Cassell, J. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365304 · Cited by 479 · cited 2×
“Indeed, past research has shown that users trust embodied conversational agents more when they engage in small talk [5], although this varies by user personality type and level of embodiment of the agent [6].”
BackgroundWhat Makes a Good Conversation?: Challenges in Designing Truly Conversational Agents
Clark, L., Pantidi, N., Cooney, O., Doyle, P., Garaialde, D., Edwards, J., Spillane, B., Gilmartin, E., Murad, C., Munteanu, C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300705 · Cited by 461 · cited 3×
“These social conversations can help build trust, rapport, and common ground [11].”
BackgroundThe Perceived Utility of Human and Automated Aids in a Visual Detection Task
Dzindolet, M. T., Pierce, L. G., Beck, H. P., & Dawe, L. A. (2002) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/0018720024494856 · Cited by 386 · cited 1×
“in terms of people’s confdence in a machine’s ability to perform as expected, benevolence (well-meaning), and integrity to adhere to ethical standards [36] Broadly, past research has evaluated how various factors such as accuracy and errors afect people’s trust in algorithms [19, 20, 67].”
BackgroundSynthetic and Natural Noise Both Break Neural Machine Translation
Belinkov, Y., & Bisk, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02173 · Cited by 386 · cited 1×
“For example, noisy channel and transcription failures can be modeled as small perturbations to the input, which is well researched [4, 21].”
BackgroundAmericas conference on information systems
Berghout, E. W., Schuurman, P., & van Wingerden, D. (2009) · Cited by 385
Conversational Recommender System
Sun, Y., & Zhang, Y. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210002 · Cited by 378
Algorithm Aversion: People Erroneously Avoid Algorithms after Seeing Them ErrPreprint
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2466040 · Cited by 356
Social Dialongue with Embodied Conversational Agents
Bickmore, T., & Cassell, J. (2005) · Text, Speech and Language Technology · doi:10.1007/1-4020-3933-6_2 · Cited by 324 · cited 2×
“Indeed, past research has shown that users trust embodied conversational agents more when they engage in small talk [5], although this varies by user personality type and level of embodiment of the agent [6].”
BackgroundStress Test Evaluation for Natural Language InferencePreprint
Naik, A., Ravichander, A., Sadeh, N., Rosé, C. P., & Neubig, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00692 · Cited by 319
Low-resource Languages: A Review of Past Work and Future ChallengesPreprint
Magueresse, A., Carles, V., & Heetderks, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07264 · Cited by 311
Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.11990 · Cited by 286 · cited 1×
“Within the past fve years, advancements in NLP have achieved huge gains in accuracy when tested against standard datasets [8, 18, 34, 56, 58, 60, 63], with state-ofthe-art accuracy in natural language processing models as high as 99% for certain tasks [8, 34].”
BackgroundPatterns for How Users Overcome Obstacles in Voice User Interfaces
Myers, C., Furqan, A., Nebolsky, J., Caro, K., & Zhu, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173580 · Cited by 213 · cited 1×
“Sometimes after a failure, users will try to reformulate, simplify, or hyper-enunciate their commands as a way to continue using the device [31, 35, 43, 61].”
BackgroundTopical-Chat: Towards Knowledge-Grounded Open-Domain Conversations
Gopalakrishnan, K., Hedayatnia, B., Chen, Q., Gottardi, A., Kwatra, S., Venkatesh, A., Gabriel, R., & Hakkani-Tür, D. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-3079 · Cited by 207 · cited 1×
“We frst explored benchmark datasets in NLP, which contain a large number of either questions and answers [48–50], or conversational dialogue [25, 57, 68].”
BackgroundSelf-Disclosure and Perceived Trustworthiness of Airbnb Host Profiles
Ma, X., Hancock, J. T., Lim Mingjie, K., & Naaman, M. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998269 · Cited by 199 · cited 3×
“These were selected because prior work on trust has determined these elements explain a large portion of trustworthiness [36, 40].”
BackgroundMethodSelf-citationAt Your Service: Designing Voice Assistant Personalities to Improve Automotive User Interfaces
Braun, M., Mainz, A., Chadowitz, R., Pfleging, B., & Alt, F. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300270 · Cited by 182 · cited 1×
“In addition, customizing a voice assistant’s personality to the user can lead to higher trust [7], while gender does not impact users’ trust in a voice assistant [59].”
BackgroundPersonalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · arXiv · doi:10.48550/arxiv.1801.07243 · Cited by 165
Is Trust Important in Technology Adoption? A Policy Capturing Approach
Bahmanziari, T., Pearson, J. M., & Crosby, L. (2003) · Journal of Computer Information Systems · doi:10.1080/08874417.2003.11647533 · Cited by 162 · cited 1×
“User trust is pivotal to user adoption of various technologies [3], and in this case, low user trust results in reluctance to try voice assistants’ novel capabilities.”
BackgroundEkinci, Y., & Dawes, P. L. (2009) · The Service Industries Journal · doi:10.1080/02642060802283113 · Cited by 160
Readiness for voice assistants to support healthcare delivery during a health crisis and pandemic
Sezgin, E., Huang, Y., Ramtekkar, U., & Lin, S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00332-0 · Cited by 155 · cited 4×
“It is especially important to support trust-building between users and voice assistants as researchers continue to imagine and develop new capabilities for them, including complex tasks such as supporting healthcare tasks [41, 55], giving mental health advice [52, 66], and other high stakes decision-making [17].”
BackgroundInfluentialConversation as Action Under Uncertainty
Paek, T., & Horvitz, E. J. (2013) · arXiv · doi:10.48550/arxiv.1301.3883 · Cited by 145 · cited 3×
“Other research aims to provide a broad categorization of voice assistant failures, drawing from theoretical frameworks of communication between humans [10, 28, 46].”
BackgroundAdvances in Natural Multimodal Dialogue Systems
(2005) · Text, Speech and Language Technology · doi:10.1007/1-4020-3933-6 · Cited by 99
Seymour, W., & Van Kleek, M. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479515 · Cited by 94 · cited 1×
“Concretely, trust can be conceptualized as a combination of confidence in a system as well as willingness to act on its provided recommendations [37, 54].”
BackgroundMeasure and Improve Robustness in NLP Models: A Survey
Wang, X., Wang, H., & Yang, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.339 · Cited by 90 · cited 1×
“NLP robustness can be defned as understanding how model performance changes when testing on a new dataset, which has a diferent distribution from the dataset the model is trained on [62].”
BackgroundSelf-citation(2017) · Journal of the Association for Information Systems · Cited by 89
Progressivity for voice interface design
Fischer, J. E., Reeves, S., Porcheron, M., & Sikveland, R. O. (2019) · Proceedings of the 1st International Conference on Conversational User Interfaces · doi:10.1145/3342775.3342788 · Cited by 88 · cited 1×
“Fischer et al. [23] encourages voice assistant responses to support progres-sivity of the conversation, especially when the response does not help the user.”
BackgroundTolmeijer, S., Zierau, N., Janson, A., Wahdatehagh, J. S., Leimeister, J. M. M., & Bernstein, A. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451623 · Cited by 85 · cited 1×
“In addition, customizing a voice assistant’s personality to the user can lead to higher trust [7], while gender does not impact users’ trust in a voice assistant [59].”
BackgroundReliable and Safe Use of Machine Translation in Medical Settings
Mehandru, N., Robertson, S., & Salehi, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533244 · Cited by 76 · cited 4×
“As the underlying technology for voice assistants continues to improve in accuracy and ability, and its applications become increasingly high stakes to human health and well-being [17, 41, 55, 66], we discuss our fndings with the goal of improving user trust and long-term engagement in voice assistants.”
BackgroundInfluentialThe Effect of Natural Distribution Shift on Question Answering Models
Miller, J., Krauth, K., Recht, B., & Schmidt, L. (2020) · arXiv · doi:10.48550/arxiv.2004.14444 · Cited by 74 · cited 2×
“[42] found that model performance drops when the question answering model is evaluated on different topic domains, such as”
BackgroundYang, S., Lee, J., Sezgin, E., Bridge, J., & Lin, S. (2021) · JMIR mHealth and uHealth · doi:10.2196/24045 · Cited by 68 · cited 4×
“As the underlying technology for voice assistants continues to improve in accuracy and ability, and its applications become increasingly high stakes to human health and well-being [17, 41, 55, 66], we discuss our fndings with the goal of improving user trust and long-term engagement in voice assistants.”
BackgroundInfluentialFitzGerald, J., Ananthakrishnan, S., Arkoudas, K., Bernardi, D., Bhagia, A., Delli Bovi, C., Cao, J., Chada, R., Chauhan, A., Chen, L., et al. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539173 · Cited by 68
Seymour, W., & Kleek, M. G. V. (2019) · doi:10.17605/osf.io/53q6j · Cited by 65
Reducing Cognitive Load and Improving Warfighter Problem Solving With Intelligent Virtual Assistants
de Melo, C. M., Kim, K., Norouzi, N., Bruder, G., & Welch, G. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.554706 · Cited by 57 · cited 4×
“It is especially important to support trust-building between users and voice assistants as researchers continue to imagine and develop new capabilities for them, including complex tasks such as supporting healthcare tasks [41, 55], giving mental health advice [52, 66], and other high stakes decision-making [17].”
BackgroundInfluentialPlanning for Natural Language Failures with the AI Playbook
Hong, M. K., Fourney, A., DeBellis, D., & Amershi, S. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445735 · Cited by 54 · cited 9×
“We determined they were not sufficient for our purposes, so we proceeded to crowdsource a dataset of failures, adapting a taxonomy from Hong et al. [28] to guide our collection.”
BackgroundMethodInfluentialMy Bad! Repairing Intelligent Voice Assistant Errors Improves Interaction
Cuadra, A., Li, S., Lee, H., Cho, J., & Ju, W. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449101 · Cited by 52 · cited 4×
“[16] found that when voice assistants make mistakes, voice assistant self-repair greatly improves people’s assessment of an intelligent voice assistant, but it can have the opposite impact if no correction is needed.”
BackgroundMethodInfluentialOwning Mistakes Sincerely: Strategies for Mitigating AI Errors
Mahmood, A., Fung, J. W., Won, I., & Huang, C.-M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517565 · Cited by 52 · cited 3×
“[39] has found that failure mitigation strategies such as apologies were efective in restoring perceptions of likability and intelligence of a voice assistant after a failure.”
BackgroundMethodInfluentialWinter, P., & Carusi, A. (2022) · Science & Technology Studies · doi:10.23987/sts.102198 · Cited by 49
Lee, C.-H., Wu, S.-L., Liu, C.-L., & Lee, H.-Y. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1714 · Cited by 45 · cited 4×
“The NLP community has also examined voice assistant failures from a slightly diferent angle, focusing on the robustness of diferent NLP components underlying voice assistants, such as models for tasks in natural language inference [44], question answering [26, 42], and speech recognition [32].”
BackgroundMethodInfluentialRole of interaction quality and trust in use of AI-based voice-assistant systems
Lee, O.-K. D., Ayyagari, R., Nasirian, F., & Ahmadian, M. (2021) · Journal of Systems and Information Technology · doi:10.1108/jsit-07-2020-0132 · Cited by 44 · cited 2×
“Prior work has illustrated the importance of trust for continued voice assistant use [31, 35], as trust is pivotal to user adoption of voice assistants [33, 45] and willingness to broaden the scope of voice assistant tasks [27].”
BackgroundA Shoulder to Cry on: Towards A Motivational Virtual Assistant for Assuaging Mental Agony
Saha, T., Reddy, S. M., Das, A., Saha, S., & Bhattacharyya, P. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.174 · Cited by 37 · cited 2×
“It is especially important to support trust-building between users and voice assistants as researchers continue to imagine and develop new capabilities for them, including complex tasks such as supporting healthcare tasks [41, 55], giving mental health advice [52, 66], and other high stakes decision-making [17].”
BackgroundThe Effect of Audiences on the User Experience with Conversational Interfaces in Physical Spaces
Candello, H., Pinhanez, C., Pichiliani, M., Cavalin, P., Figueiredo, F., Vasconcelos, M., & Do Carmo, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300320 · Cited by 33 · cited 1×
“[9] defned failure as occasions in which someone asked a question that could not be understood or was out of scope of the voice assistants’ knowledge, in which case it would divert the conversation to ask an unrelated question.”
BackgroundRobertson, S., & Díaz, M. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3534638 · Cited by 31 · cited 1×
“As voice assistants continue to perform increasingly complex and high stakes tasks across various industries [17, 41, 51, 55, 66], we hope that this research will help Baughan et al.”
BackgroundWhen Users Assist the Voice Assistants: From Supervision to Failure Resolution
Lahoual, D., & Frejus, M. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3299053 · Cited by 28 · cited 6×
“Sometimes after a failure, users will try to reformulate, simplify, or hyper-enunciate their commands as a way to continue using the device [31, 35, 43, 61].”
BackgroundInfluentialLet Me Ask You This: How Can a Voice Assistant Elicit Explicit User Feedback?
Xiao, Z., Mennicken, S., Huber, B., Shonkoff, A., & Thom, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479532 · Cited by 23 · cited 1×
“[65] demonstrated that situating the voice assistant as a learner, and helping users understand when to give feedback to the voice assistant, improved users’ perceptions of the voice assistant.”
BackgroundDisfl-QA: A Benchmark Dataset for Understanding Disfluencies in Question Answering
Gupta, A., Xu, J., Upadhyay, S., Yang, D., & Faruqui, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.293 · Cited by 22 · cited 4×
“In addition to these benchmark datasets, we investigated datasets that incorporated spoken word speech patterns, such as the Spoken SQuAD dataset [32] and Disflu-QA dataset [26], as well as humanagent interaction datasets, such as the ACE dataset [1], the Niki and Julie corpus [2], and a video dataset of voice assista…”
BackgroundMethodInfluentialConversational Error Analysis in Human-Agent Interaction
Aneja, D., McDuff, D., & Czerwinski, M. (2020) · Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents · doi:10.1145/3383652.3423901 · Cited by 15 · cited 2×
“In addition to these benchmark datasets, we investigated datasets that incorporated spoken word speech patterns, such as the Spoken SQuAD dataset [32] and Disflu-QA dataset [26], as well as humanagent interaction datasets, such as the ACE dataset [1], the Niki and Julie corpus [2], and a video dataset of voice assista…”
BackgroundMethodFPI: Failure Point Isolation in Large-scale Conversational Assistants
Khaziev, R., Shahid, U., Röding, T., Chada, R., Kapanci, E., & Natarajan, P. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-industry.17 · Cited by 8 · cited 1×
“By understanding how diferent types of failures afect trust in voice assistants overall, we can then try to pinpoint the underlying NLP components that are the root cause of the most critical failures that erode trust [30].”
BackgroundArtstein, R., Boberg, J., Gainer, A., Gratch, J., Johnson, E., Leuski, A., Lucas, G., & Traum, D. (2018) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/5m3sw35yfkrx · Cited by 5 · cited 2×
“In these cases, we found that the datasets were still restricted to only failures at the understanding and response level [26, 32] or the context for the failures was very specifc and did not necessarily capture the breadth of possible failures users experience [1, 2].”
BackgroundMethodLook at Me When I Talk to You: A Video Dataset to Enable Voice Assistants to Recognize Errors
Cuadra, A., Lee, H., Cho, J., & Ju, W. (2021) · arXiv · doi:10.48550/arxiv.2104.07153 · Cited by 5 · cited 2×
“[15]’s video dataset was the closest available ft for our needs, but we still found the use case of in-lab question-answering too narrow for our purposes.”
BackgroundMethodMeasure and Improve Robustness in NLP Models: A SurveyPreprint
Wang, X., Wang, H., & Yang, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.08313
Self-citation
Research works printed but not linked: 1
- 10.18653/v1 · doi:10.18653/v1
Cited by
Trust in AI chatbots: A systematic review
Ng, S. W. T., & Zhang, R. (2025) · Telematics and Informatics · doi:10.1016/j.tele.2025.102240 · Cited by 143
The Trust Recovery Journey. The Effect of Timing of Errors on the Willingness to Follow AI Advice.
Kahr, P. K., Rooks, G., Snijders, C., & Willemsen, M. C. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645167 · Cited by 21
User acceptance of AI voice assistants in Jordan’s telecom industry
Al-kfairy, M., Mustafa, D., Al-Adaileh, A., Zriqat, S., & Sendaba, O. (2024) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2024.100521 · Cited by 24
LLM-mediated domain-specific voice agents: the case of TextileBot
Zhong, S., Gatti, E., Hardwick, J., Ribul, M., Cho, Y., & Obrist, M. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2456667 · Cited by 22
“The human-computer interaction (HCI) community is seeing a rising emphasis on the development of voice-based conversational agents [95, 9, 94, 40].”
BackgroundNLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMs
Antoniak, M., Naik, A., Alvarado, C. S., Wang, L. L., & Chen, I. Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658982 · Cited by 51
Metadata
- Authors
- Baughan, A., Wang, X., Liu, A., Mercurio, A., Chen, J., & Ma, X.
- Affiliations
- University of Washington (US)
- Google (United States) (US)
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-16
- OpenAlex
- W4322831937
- Cited by
- 32
- References
- 67
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 7
- FWCI
- 5.67
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- voice assistants, user trust, task abandonment, music playback, large language models, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.