Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record, with a small difference in title or year; 78 of the 80 references it cites were found in a registry or an index; 32 of the 75 sources its claims cite were checked for support.

Situated Understanding of Errors in Older Adults' Interactions with Voice Assistants: A Month-Long, In-Home Study

Open access
Mahmood, Amama, Wang, Junxiang, Huang, Chien-Ming·2026·ACM Transactions on Accessible Computing

Abstract

Our work addresses the challenges older adults face with commercial Voice Assistants (VAs), notably in conversation breakdowns and error handling. Traditional methods of collecting user experiences—usage logs and post hoc interviews—do not fully capture the intricacies of older adults’ interactions with VAs, particularly regarding their reactions to errors. To bridge this gap, we equipped 15 older adults’ homes with smart speakers integrated with custom audio recorders to collect “in-the-wild” audio interaction data for detailed error analysis. Recognizing the growing use of Large Language Models (LLMs) to enhance the capabilities of VAs, our study also explored how this integration of LLMs changes older adults’ interaction dynamics, specifically during errors. Midway through our study, we deployed ChatGPT-powered VA to investigate its efficacy for older adults. Our research suggests that while technical improvements—such as leveraging vocal and verbal responses combined with LLMs’ contextual capabilities—can enhance error prevention and management in VAs, interaction-level challenges still remain, particularly those unique to older adults. We propose design considerations to better align future VAs with older adults’ expectations and lived experiences.

Overview

CoChat Summary

TL;DR

This four-week in-home study recorded older adults’ audio interactions with Amazon Alexa and found a true error rate of 24.76%, with most errors becoming conversational breakdowns and few resolved on the immediate retry. A ChatGPT-powered Alexa skill showed lower error rates and better resolution in limited use, but adoption was hindered by access complexity and unfamiliarity.

Key findings

  • Across 2,552 user query–VA response pairs from 15 older adults, 632 queries (24.76%) had errors; 98.10% of these manifested as conversational breakdowns, and only 25.47% (160) were resolved on the immediate next attempt.
  • The most frequent error type was intent recognition (n = 204, 32.3% of all errors), followed by VA limitation errors (n = 130, 20.6%); VA errors had the lowest resolution rate (20.44%), while human errors had the highest (38.9%).
  • Participants recognized 79.84% of conversational breakdowns through reactions or corrective actions, but they reacted verbally or vocally to only 13.77% of errors (87 of 632); common reactions included negation (27), interruption (21), laughter (14), and negative comments (10).
  • The ChatGPT+Alexa skill had a lower percentage of errors and a higher resolution rate of 14.1% versus 8.7% for standard information-seeking interactions, but only a few participants (P1, P3, P11, P15) interacted with it, mainly around deployment day.

Methods

  • Four-week field deployment of a 5th-generation Amazon Echo Dot in 15 older adults’ homes, with a custom Arduino-based audio recorder that captured each interaction plus 10 seconds after Alexa’s blue light turned off.
  • Data included usage logs, recorded audio, pre-study and exit semi-structured interviews, and a ChatGPT-3.5-powered Alexa skill deployed as a technology probe in Weeks 3–4.
  • Audio was transcribed and matched to Alexa logs; 2,552 query–response pairs were coded for interaction purpose, error presence/type, resolution, recovery method, and whether errors led to breakdowns, using an inductive-then-deductive codebook.

Limitations

  • The paper states the four-week duration may be too short for longer-term trends, the analysis is qualitative, the LLM-powered VA was tested only for generic information retrieval, and the sample was small and limited in location, cognitive ability, and ethnicity.
  • The text also makes evident that audio data for participant P6 was missing due to a recorder malfunction, and that ChatGPT-skill findings are based on very limited use by only a few participants.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3796236

Cites

78 of 80 references verified0 citation errors · 2 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

27 Valid · 1 Not valid · 21 Not checked · 171 Own results and interpretations

133 more claims · 10 with their cited sources checked

Create a free account to see all 220 claims and to verify the 8 references still open.

See all claims — free account
  • Probably not a substantial claim. 31
  • It may state two separate claims as one. 58

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 52 references

Research works printed but not linked: 2

  1. Jiaan Li, Yulu Hu, Shihan Fu, Zhiqing Wu, Pan Hui, Hailiang Wang, and Mingming Fan. 2024. ‘It’s like having a friend’: Exploring the opportunity of large language models (LLMs) in assisting retirees in their daily lives. In Proceedings of the 12th International Symposium of Chinese CHI, 463–475.
  2. Cathy Pearl. 2016. Designing Voice User Interfaces: Principles of Conversational Experiences. O’Reilly Media, Inc.

Cited by

See all 6

Metadata

Authors
Mahmood, A., Wang, J., & Huang, C.-M.
Affiliations
  • Johns Hopkins University (US)
  • Carnegie Mellon University (US)
Venue
ACM Transactions on Accessible Computing
Published
2026-03-18
Volume
19
Issue
1
Pages
1-36
ISSN
  • 1936-7228 (ISSN-L)
  • 1936-7236
OpenAlex
W7128159089
Cited by
6
References
52
Citations by year
2026: 2
FWCI
29.72
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • ACM Transactions on Accessible Computing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (1840088)
  • Johns Hopkins University Institute for Assured Autonomy, and Malone Center for Engineering in Healthcare.
Topics
  • Health Sciences › Medicine › Geriatrics and Gerontology › Technology Use by Older Adults
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
voice assistants, older adults, error handling, smart speakers, large language models, ChatGPT, user experience, situated understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.