The DOI matches its Crossref record, with a small difference in title or year; 78 of the 80 references it cites were found in a registry or an index; 32 of the 75 sources its claims cite were checked for support.
Situated Understanding of Errors in Older Adults' Interactions with Voice Assistants: A Month-Long, In-Home Study
Open accessAbstract
Our work addresses the challenges older adults face with commercial Voice Assistants (VAs), notably in conversation breakdowns and error handling. Traditional methods of collecting user experiences—usage logs and post hoc interviews—do not fully capture the intricacies of older adults’ interactions with VAs, particularly regarding their reactions to errors. To bridge this gap, we equipped 15 older adults’ homes with smart speakers integrated with custom audio recorders to collect “in-the-wild” audio interaction data for detailed error analysis. Recognizing the growing use of Large Language Models (LLMs) to enhance the capabilities of VAs, our study also explored how this integration of LLMs changes older adults’ interaction dynamics, specifically during errors. Midway through our study, we deployed ChatGPT-powered VA to investigate its efficacy for older adults. Our research suggests that while technical improvements—such as leveraging vocal and verbal responses combined with LLMs’ contextual capabilities—can enhance error prevention and management in VAs, interaction-level challenges still remain, particularly those unique to older adults. We propose design considerations to better align future VAs with older adults’ expectations and lived experiences.
Overview
CoChat Summary
TL;DR
This four-week in-home study recorded older adults’ audio interactions with Amazon Alexa and found a true error rate of 24.76%, with most errors becoming conversational breakdowns and few resolved on the immediate retry. A ChatGPT-powered Alexa skill showed lower error rates and better resolution in limited use, but adoption was hindered by access complexity and unfamiliarity.
Key findings
- Across 2,552 user query–VA response pairs from 15 older adults, 632 queries (24.76%) had errors; 98.10% of these manifested as conversational breakdowns, and only 25.47% (160) were resolved on the immediate next attempt.
- The most frequent error type was intent recognition (n = 204, 32.3% of all errors), followed by VA limitation errors (n = 130, 20.6%); VA errors had the lowest resolution rate (20.44%), while human errors had the highest (38.9%).
- Participants recognized 79.84% of conversational breakdowns through reactions or corrective actions, but they reacted verbally or vocally to only 13.77% of errors (87 of 632); common reactions included negation (27), interruption (21), laughter (14), and negative comments (10).
- The ChatGPT+Alexa skill had a lower percentage of errors and a higher resolution rate of 14.1% versus 8.7% for standard information-seeking interactions, but only a few participants (P1, P3, P11, P15) interacted with it, mainly around deployment day.
Methods
- Four-week field deployment of a 5th-generation Amazon Echo Dot in 15 older adults’ homes, with a custom Arduino-based audio recorder that captured each interaction plus 10 seconds after Alexa’s blue light turned off.
- Data included usage logs, recorded audio, pre-study and exit semi-structured interviews, and a ChatGPT-3.5-powered Alexa skill deployed as a technology probe in Weeks 3–4.
- Audio was transcribed and matched to Alexa logs; 2,552 query–response pairs were coded for interaction purpose, error presence/type, resolution, recovery method, and whether errors led to breakdowns, using an inductive-then-deductive codebook.
Limitations
- The paper states the four-week duration may be too short for longer-term trends, the analysis is qualitative, the LLM-powered VA was tested only for generic information retrieval, and the sample was small and limited in location, cognitive ability, and ethnicity.
- The text also makes evident that audio data for participant P6 was missing due to a recorder malfunction, and that ChatGPT-skill findings are based on very limited use by only a few participants.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
27 Valid · 1 Not valid · 21 Not checked · 171 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“they often find it challenging to remember commands associated with VA features and skills [49, 57]-even those as simple as a wake word [1, 15, 49].”
What the sources say
- Partly supportedcites [49] [49] Alisha Pradhan, Amanda Lazar, and Leah Findlater. 2020. Use of intelligent voice assistants by older adults with low technology use. ACM Transactions on Computer-Human Interaction (TOCHI) 27, 4 (2020), 1-27. - sourceExperimental — accuracy not yet measured
Section 4.3, “Challenges and Barriers to Use,” reports that five participants had difficulty remembering keywords for voice-assistant features and skills. It mentions problems after using the wake word, but does not report difficulty remembering the wake word itself. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [57] [57] Brodrick Stigall, Jenny Waycott, Steven Baker, and Kelly Caine. 2019. Older adults' perception and use of voice user interfaces: a preliminary review of the computing literature. In Proceedings of the 31st Australian Conference on Human-Computer-Interaction . 423-427. - source
- Not checked — no full text of the sourcecites [1] [1] Leonardo Angelini, Maurizio Caon, Emmanuel Michielan, Omar Abou Khaled, and Elena Mugellini. 2021. Seniors' perception of smart speakers: challenges and opportunities elicited in the Silver&Home Living Lab. In Congress of the International Ergonomics Association . Springer, 137-144. - source
- Partly supportedcites [15] [15] Andrea Cuadra, Hyein Baek, Deborah Estrin, Malte Jung, and Nicola Dell. 2022. On Inclusion: Video Analysis of Older Adult Interactions with a Multi-Modal Voice Assistant in a Public Setting. In Proceedings of the 2022 International Conference on Information and Communication Technologies and Development . 1-17. - sourceExperimental — accuracy not yet measured
Section 4.1.1, “Overlooked by the machine” (page unavailable), reports that older participants frequently mispronounced, omitted, or misplaced the wake word. The paper supports difficulty using the wake word, but does not establish that the cause was difficulty remembering it or show a broader difficulty remembering commands for features and skills. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite older adults' desires to have conversational interactions with VAs, current commercial VA capabilities are far more limited [48].”
What the sources say
- Partly supportedcites [48] [48] Alisha Pradhan, Leah Findlater, and Amanda Lazar. 2019. " Phantom Friend" or" Just a Box with Information" Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-21. - sourceExperimental — accuracy not yet measured
Section 4.4 reports that some older participants desired social contact through Alexa. Section 3.2.1 reports that Alexa sometimes failed to respond conversationally, particularly in its chat feature. The paper does not establish the broader claim that current commercial voice assistants’ capabilities are far more limited than older adults’ conversational desires; it also reports that brief exchanges with Alexa met some participants’ social needs. The indexed passages provide no page number. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“While researchers have investigated older adults' long-term interactions with VAs at home-focusing on their adoption of the technology, its benefits and challenges, and their perceptions of the experience [28, 48, 49, 51]-a significant research gap remains in the area of comprehensive error analysis.”
What the sources say
- Not checked — no full text of the sourcecites [28] [28] Sunyoung Kim and Abhishek Choudhury. 2021. Exploring older adults' perception and use of smart speaker-based voice assistants: A longitudinal study. Computers in Human Behavior 124 (2021), 106914. - source
- Supportedcites [48] [48] Alisha Pradhan, Leah Findlater, and Amanda Lazar. 2019. " Phantom Friend" or" Just a Box with Information" Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-21. - source
The citation is attached to the statement that researchers have studied older adults’ in-home voice-assistant use, not to the citing authors’ conclusion about an error-analysis gap. The Methods section describes a three-week Echo Dot deployment in seven older adults’ homes, and §3.2 says the interviews examined their perceptions and use. (LLM, conf=0.87)
- Supportedcites [49] [49] Alisha Pradhan, Amanda Lazar, and Leah Findlater. 2020. Use of intelligent voice assistants by older adults with low technology use. ACM Transactions on Computer-Human Interaction (TOCHI) 27, 4 (2020), 1-27. - source
The Methods section describes a three-week deployment of a voice assistant with seven older adults, using interviews, diaries, and usage logs to examine their perceptions and use. The Discussion reports benefits and challenges of using it at home. The claim that comprehensive error analysis remains a research gap is the citing authors’ own conclusion, not a finding attributed to this source. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [51] [51] Sandeep Purao, Haijing Hao, and Chenhang Meng. 2021. The use of smart home speakers by the elderly: exploratory analyses and potential for big data. Big Data Research 25 (2021), 100224. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“LLMs, on the other hand, have shown enhanced conversational capabilities, yielding robust and fluid interactions with users [9, 37].”
What the sources say
- Partly supportedcites [9] [9] Szeyi Chan, Jiachen Li, Bingsheng Yao, Amama Mahmood, Chien-Ming Huang, Holly Jimison, Elizabeth D Mynatt, and Dakuo Wang. 2023. " Mango Mango, How to Let The Lettuce Dry Without A Spinner?': Exploring User Perceptions of Using An LLM-Based Conversational Assistant Toward Cooking Partner. arXiv preprint arXiv:2310.05853 (2023). - sourceExperimental — accuracy not yet measured
The Introduction (p. 2) says LLMs can understand lengthy input and accommodate multi-turn dialogue, and the findings describe conversations becoming more natural (section 4.2.2, p. 16). But that section also reports misunderstandings, wake-word constraints, and increased cognitive load, so the claim of robust and fluid user interactions overstates what the paper supports. (LLM, conf=0.86)
- Partly supportedcites [37] [37] Amama Mahmood, Junxiang Wang, Bingsheng Yao, Dakuo Wang, and Chien-Ming Huang. 2023. LLM-Powered Conversational Voice Assistants: Interaction Patterns, Opportunities, Challenges, and Design Guidelines. arXiv preprint arXiv:2309.13879 (2023). - sourceExperimental — accuracy not yet measured
The Conclusion (p. 24) says LLMs are better than traditional voice assistants at understanding context and handling breakdowns, and reports that ChatGPT absorbed 81% of intent-recognition failures. But it describes more fluid interactions as a goal; the Abstract (p. 1) likewise calls resilient and fluid interactions a potential benefit, rather than an established result. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Recent efforts to incorporate LLMs into voice-based interactions for older adults have shown improvement in conversational aspects [26, 63]”
What the sources say
- Not checked yetcites [26] [26] Eunkyung Jo, Daniel A. Epstein, Hyunhoon Jung, and Young-Ho Kim. 2023. Understanding the Benefits and Challenges of Deploying Conversational AI Leveraging Large Language Models for Public Health Intervention. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23) . Association for Computing Machinery, New York, NY, USA, Article 18, 16 pages. https://doi.org/10.1145/ 3544548.3581503 -
- Supportedcites [63] [63] Ziqi Yang, Xuhai Xu, Bingsheng Yao, Ethan Rogers, Shao Zhang, Stephen Intille, Nawar Shara, Guodong Gordon Gao, and Dakuo Wang. 2024. Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8, 2 (2024), 1-35. - source
In Section 5.2.1 (page unavailable), older adults compared Talk2Care’s LLM-powered voice assistant with leaving a message and rated it as better supporting communication with providers. The paper attributes that benefit to its natural, interactive conversation, supporting the claim of improvement in conversational aspects. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
133 more claims · 10 with their cited sources checked
Create a free account to see all 220 claims and to verify the 8 references still open.
- Probably not a substantial claim. 31
- It may state two separate claims as one. 58
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“in 238 instances they made at least one attempt to correct the error; of these, 107 errors (44 . 96%) were resolved after the first retry, 43 instances (15 . 19%) led to participants giving up”
Reported 15.19 · recomputed 18.067227
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Percentage of Errors and Their Resolution by Interaction Category · p. 14
“compared to human errors (6 . 7%), as shown in Table 2.”
§ 4 Findings · p. 9
“| Human errors | User made a mistake | 72 | 38.9% | 6.9% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.5 Data and Analysis · p. 8
“| 13 | M | 81 | Caucasian | Vision issues | Good”
§ For each erroneous interaction · p. 30
“| Vision issues | Very good”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Analysis of Erroneous Interactions · p. 11
“health-related queries (52 . 38%)”
§ 4.1 Analysis of Erroneous Interactions · p. 10
“| Health | 43 | 55.81%”
Citations
Cites
Czaja, S. J., Charness, N., Fisk, A. D., Hertzog, C., Nair, S. N., Rogers, W. A., & Sharit, J. (2006) · Psychology and Aging · doi:10.1037/0882-7974.21.2.333 · Cited by 2,113 · cited 1×
“Design Consideration 4: VAs should employ context-aware responses with dynamic tutorials and real-time guidance in daily interactions to better align older adults’ mental models with evolving voice assistance technology.”
Technology probes: inspiring design for and with families
Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B. B., Druin, A., Plaisant, C., Beaudouin-Lafon, M., Conversy, S., Evans, H., Hansen, H., et al. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642616 · Cited by 1,471
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Vaportzis, E., Giatsi Clausen, M., & Gow, A. J. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01687 · Cited by 916 · cited 1×
“Design Consideration 4: VAs should employ context-aware responses with dynamic tutorials and real-time guidance in daily interactions to better align older adults’ mental models with evolving voice assistance technology.”
Voice Interfaces in Everyday Life
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174214 · Cited by 717 · cited 4×
“These shifts to transactional dynamics have led researchers to debate whether such interactions should truly be considered “conversations” in the first place [59].”
InfluentialPradhan, A., Mehta, K., & Findlater, L. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174033 · Cited by 448 · cited 1×
“Older adults find rigid keyword requirements frustrating [61, 62].”
"What can i help you with?": infrequent users' experiences of intelligent personal assistants
Cowan, B. R., Pantidi, N., Coyle, D., Morrissey, K., Clarke, P., Al-Shehri, S., Earley, D., & Bandeira, N. (2017) · Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services · doi:10.1145/3098279.3098539 · Cited by 351
Communication Breakdowns Between Families and Alexa
Beneteau, E., Richards, O. K., Zhang, M., Kientz, J. A., Yip, J., & Hiniker, A. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300473 · Cited by 313 · cited 1×
“As users grow more accustomed to VA technology, they begin to value not just the systems’ operational ease, but also the sense of digital companionship they offer [34].”
Talk to me: Exploring user interactions with the Amazon Alexa
Lopatovska, I., Rink, K., Knight, I., Raines, K., Cosenza, K., Williams, H., Sorsche, P., Hirsch, D., Li, Q., & Martinez, A. (2019) · Journal of Librarianship and Information Science · doi:10.1177/0961000618759414 · Cited by 298 · cited 1×
“Overall, we observed a variety of interactions—from setting reminders to seeking information about various topics—as shown in Tables 3 and 4 and observed decline in interaction frequency over time, consistent with prior work [43, 61].”
Use of Intelligent Voice Assistants by Older Adults with Low Technology Use
Pradhan, A., Lazar, A., & Findlater, L. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3373759 · Cited by 285 · cited 8×
“While previous research has identified error types, estimated error rates, and examined recovery strategies in older adults’ VA use [34, 61], these studies do not address the sequential nature of how errors compound when users attempt to correct them.”
Influential
Research works printed but not linked: 2
- Jiaan Li, Yulu Hu, Shihan Fu, Zhiqing Wu, Pan Hui, Hailiang Wang, and Mingming Fan. 2024. ‘It’s like having a friend’: Exploring the opportunity of large language models (LLMs) in assisting retirees in their daily lives. In Proceedings of the 12th International Symposium of Chinese CHI, 463–475.
- Cathy Pearl. 2016. Designing Voice User Interfaces: Principles of Conversational Experiences. O’Reilly Media, Inc.
Cited by
Public opinion on types of voice systems for older adults
Qu, N. Z., Li, J., Kongmanee, J., & Chignell, M. (2024) · Journal of Rehabilitation and Assistive Technologies Engineering · doi:10.1177/20556683241287414 · Cited by 1
Elderly Centric Mobile Assistance Platform with Voice and GPS Integration
N, P. K., S, A. H., Chouraddi, B. R., et al. (2026) · International Conference on Intelligent and Innovative Technologies in Computing, Electrical and Electronics (IITCEE) · doi:10.1109/iitcee67948.2026.11394103
Summoogum, K., Lah, J. Y., Ansari, M., & Das, D. (2026) · doi:10.1109/iccms69991.2026.11642652
Metadata
- Authors
- Mahmood, A., Wang, J., & Huang, C.-M.
- Affiliations
- Johns Hopkins University (US)
- Carnegie Mellon University (US)
- Venue
- ACM Transactions on Accessible Computing
- Published
- 2026-03-18
- Volume
- 19
- Issue
- 1
- Pages
- 1-36
- ISSN
- 1936-7228 (ISSN-L)
- 1936-7236
- OpenAlex
- W7128159089
- Semantic Scholar
- 663b9fec2dbf8e29192d40503063e6dcd462f7bd
- Cited by
- 6
- References
- 52
- Citations by year
- 2026: 2
- FWCI
- 29.72
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- ACM Transactions on Accessible Computing (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1840088)
- Johns Hopkins University Institute for Assured Autonomy, and Malone Center for Engineering in Healthcare.
- Topics
- Health Sciences › Medicine › Geriatrics and Gerontology › Technology Use by Older Adults
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- voice assistants, older adults, error handling, smart speakers, large language models, ChatGPT, user experience, situated understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.