CoChat
Sign Up
Cited by 16Technology Use by Older AdultsSocial Robot Interaction and HRIInnovative Human-Technology Interaction

On Inclusion: Video Analysis of Older Adult Interactions with a Multi-Modal Voice Assistant in a Public Setting

Open access
Cuadra, Andrea, Baek, Hyein, Estrin, Deborah, Jung, Malte, Dell, Nicola·2022·International Conference on Information & Communication Technologies and Development 2022
DOI 10.1145/3572334.3572371
Open this paper in CoChat — free accountOpen PDF

Abstract

Older adults around the world lack access to a wide range of potentially life-changing digital applications, services, and information that could be provided by voice assistants (such as Amazon’s Alexa, Google’s Assistant, or Apple’s Siri). However, older adults’ needs are underrepresented in the design of voice assistants. Because of this, we are missing opportunities for digital inclusion, and increasing risks of excluding older adults as these devices permeate public settings. In this work, we video record older adults (n=26) interacting with a multi-modal voice assistants while waiting in line at food pantries, and use Interaction Analysis to draw insights from these recordings. We find that by being agnostic to body language, audio-prosodic features, and other contextual factors, voice assistants fail to capture and react to some important aspects of interactions. We discuss design (e.g, interpreting users’ posture as a cue to wake the device when they are leaning towards the device) and research (e.g., surveillance trade-offs) implications, and argue for the use of multi-modal inputs with attention to privacy. Designing and training voice assistants to take in and appropriately respond to non-verbal cues may increase their inclusivity, helping them fulfill important needs of our aging population.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3572334.3572371

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Older adults often find it challenging to remember commands associated with voice assistant features and skills, even simple wake words.

    “they often find it challenging to remember commands associated with VA features and skills [49, 57]-even those as simple as a wake word [1, 15, 49].”

    From Situated Understanding of Errors in Older Adults' Interactions with Voice Assistants: A Month-Long, In-Home Study (2026) · cites this paper as [15]

    Section 4.1.1, “Overlooked by the machine” (page unavailable), reports that older participants frequently mispronounced, omitted, or misplaced the wake word. The paper supports difficulty using the wake word, but does not establish that the cause was difficulty remembering it or show a broader difficulty remembering commands for features and skills. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Being agnostic to user state (e.g., body language, audio-prosodic features, and other contextual information) results in a failure to appropriately respond or react to older adult's queries.

    “Being agnostic to user state ( e.g., body language, audio-prosodic features, and other contextual information) results in a failure to appropriately respond or react to older adult's queries [15].”

    From Situated Understanding of Errors in Older Adults' Interactions with Voice Assistants: A Month-Long, In-Home Study (2026) · cites this paper as [15]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 57

  1. Tawfiq Ammari, Jofish Kaye, Janice Y Tsai, and Frank Bentley. 2019. Music, Search, and IoT: How People (Really) Use Voice Assistants. ACM Trans. Comput. Hum. Interact. 26, 3 (2019), 17-1.
  2. Beverly Beisgen and Marilyn Kraitchman. 2003. Senior centers: Opportunities for successful aging. Springer Publishing Company.
  3. Genevieve Bell and Paul Dourish. 2007. Yesterday's tomorrows: notes on ubiq- uitous computing's dominant vision. Personal and ubiquitous computing 11, 2 (2007), 133-143.
  4. Erin Beneteau, Olivia K Richards, Mingrui Zhang, Julie A Kientz, Jason Yip, and Alexis Hiniker. 2019. Communication breakdowns between families and Alexa. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-13.
  5. Frank Bentley, Chris Luvogt, Max Silverman, Rushani Wirasinghe, Brooke White, and Danielle Lottridge. 2018. Understanding the long-term use of smart speaker assistants. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1-24.
  6. Clara Berridge and Terrie Fox Wetle. 2020. Why older adults and their children disagree about in-home surveillance technology, sensors, and tracking. The Gerontologist 60, 5 (2020), 926-934.
  7. Karen Bonilla and Aqueasha Martin-Hammond. 2020. Older adults' perceptions of intelligent voice assistant privacy, transparency, and online privacy guidelines. In Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020).
  8. Robin Brewer. 2016. Connecting Older Adults through Voice-Based Interfaces. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion (San Francisco, California, USA) (CSCW '16 Companion). Association for Computing Machinery, New York, NY, USA, 131-134. https://doi.org/10.1145/2818052.2874350 · DOI
  9. Robin Brewer. 2016. Connecting older adults through voice-based interfaces. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion. 131-134.
  10. Alberto Brunete, Ernesto Gambao, Miguel Hernando, and Raquel Cedazo. 2021. Smart Assistive Architecture for the Integration of IoT Devices, Robotic Systems, and Multimodal Interfaces in Healthcare Environments. Sensors 21, 6 (2021), 2212.
  11. Justine Cassell, Yukiko I Nakano, Timothy W Bickmore, Candace L Sidner, and Charles Rich. 2001. Non-verbal cues for discourse structure. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics. 114-123.
  12. Yuan Cheng, Yuchao Yang, Hai-Bao Chen, Ngai Wong, and Hao Yu. 2021. S3-Net: A Fast and Lightweight Video Scene Understanding Network by Single-Shot Segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 3329-3337.
  13. Jia-Luen Chua, Yoong Choon Chang, and Wee Keong Lim. 2015. A simple vision- based fall detection technique for indoor video surveillance. Signal, Image and Video Processing 9, 3 (2015), 623-633.
  14. Kyungjin Chung, Young Hoon Oh, and Da Young Ju. 2019. Elderly Users' Interac- tion with Conversational Agent. In Proceedings of the 7th International Conference on Human-Agent Interaction. 277-279.
  15. Benjamin R Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017. "What can i help you with?" infrequent users' experiences of intelligent personal assistants. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1-12.
  16. Andy Crabtree, Steve Benford, Chris Greenhalgh, Paul Tennent, Matthew Chalmers, and Barry Brown. 2006. Supporting ethnographic studies of ubiquitous computing in the wild. In Proceedings of the 6th conference on Designing Interactive systems. 60-69.
  17. Andrea Cuadra, Shuran Li, Hansol Lee, Jason Cho, and Wendy Ju. 2021. My Bad! Repairing Intelligent Voice Assistant Errors Improves Interaction. In Proceedings of the 2021 conference on Computer supported cooperative work. ACM.
  18. Sara J Czaja, Walter R Boot, Neil Charness, and Wendy A Rogers. 2019. Designing for older adults: Principles and creative human factors approaches. CRC press.
  19. Philip R Doyle, Leigh Clark, and Benjamin R Cowan. 2021. What do we see in them? identifying dimensions of partner models for speech interfaces using a psycholexical approach. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-14.
  20. Sidney K D'mello and Arthur Graesser. 2010. Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modeling and User-Adapted Interaction 20, 2 (2010), 147-187.
  21. Paul Ekman and Wallace V. Friesen. 2010. The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding:. De Gruyter Mouton, 57-106. https: //doi.org/doi:10.1515/9783110880021.57 · DOI
  22. Longlong Jing, Toufiq Parag, Zhe Wu, Yingli Tian, and Hongcheng Wang. 2021. VideoSSL: Semi-Supervised Learning for Video Classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 1110-1119.
  23. Brigitte Jordan and Austin Henderson. 1995. Interaction analysis: Foundations and practice. The journal of the learning sciences 4, 1 (1995), 39-103.
  24. Malte F Jung, Jin Joo Lee, Nick DePalma, Sigurdur O Adalgeirsson, Pamela J Hinds, and Cynthia Breazeal. 2013. Engaging robots: easing complex human- robot teamwork using backchanneling. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 1555-1566.
  25. Veton Kepuska and Gamal Bohouta. 2018. Next-generation of virtual personal assistants (microsoft cortana, apple siri, amazon alexa and google home). In 2018 IEEE 8th annual computing and communication workshop and conference (CCWC). IEEE, 99-103.
  26. Dimosthenis Kontogiorgos, Andre Pereira, Boran Sahindal, Sanne van Waveren, and Joakim Gustafson. 2020. Behavioural responses to robot conversational failures. In Proceedings of the 2020 ACM/IEEE International Conference on Human- Robot Interaction. 53-62.
  27. Małgorzata Kowalska, Aleksandra Gładyś, Barbara Kalańska-Łukasik, Monika Gruz-Kwapisz, Wojciech Wojakowski, and Tomasz Jadczyk. 2020. Readiness for Voice Technology in Patients With Cardiovascular Diseases: Cross-Sectional Study. Journal of medical Internet research 22, 12 (2020), e20456.
  28. Ewelina Lacka and Alain Chong. 2016. Usability perspective on social media sites' adoption in the B2B context. Industrial Marketing Management 54 (2016), 80-91.
  29. Min Kyung Lee, Sara Kiesler, and Jodi Forlizzi. 2010. Receptionist or information kiosk: how do people talk with a robot?. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. 31-40.
  30. Noah Liebman and Darren Gergle. 2016. It's (Not) Simply a Matter of Time: The Relationship Between CMC Cues and Interpersonal Affinity. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Comput- ing (San Francisco, California, USA) (CSCW '16). Association for Computing Ma- chinery, New York, NY, USA, 570-581. https://doi.org/10.1145/2818048.2819945 · DOI
  31. Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-23.
  32. Donald McMillan, Barry Brown, Ikkaku Kawaguchi, Razan Jaber, Jordi Solsona Be- lenguer, and Hideaki Kuzuoka. 2019. Designing with Gaze: Tama-a Gaze Acti- vated Smart-Speaker. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-26.
  33. Katashi Nagao and Akikazu Takeuchi. 1994. Speech dialogue with facial displays: Multimodal human-computer conversation. arXiv preprint cmp-lg/9406002 (1994). · arXiv
  34. Clifford Nass, Jonathan Steuer, and Ellen R Tauber. 1994. Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems. 72-78.
  35. Clifford Ivar Nass and Scott Brave. 2005. Wired for speech: How voice activates and advances the human-computer relationship. MIT press Cambridge, MA.
  36. Liqiang Nie, Mengzhao Jia, Xuemeng Song, Ganglu Wu, Harry Cheng, and Jian Gu. 2021. Multimodal Activation: Awakening Dialog Robots without Wake Words. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 491-500.
  37. Young Hoon Oh, Kyungjin Chung, Da Young Ju, et al. 2020. Differences in Interactions with a Conversational Agent. International Journal of Environmental Research and Public Health 17, 9 (2020), 3189.
  38. Martin Porcheron, Joel E Fischer, Stuart Reeves, and Sarah Sharples. 2018. Voice interfaces in everyday life. In proceedings of the 2018 CHI conference on human factors in computing systems. 1-12.
  39. Martin Porcheron, Joel E Fischer, and Sarah Sharples. 2017. "Do Animals Have Accents?" Talking with Agents in Multi-Party Conversation. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing. 207-219.
  40. Alisha Pradhan, Leah Findlater, and Amanda Lazar. 2019. "Phantom Friend" or "Just a Box with Information" Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-21.
  41. Byron Reeves and Clifford Ivar Nass. 1996. The media equation: How people treat computers, television, and new media like real people and places. Cambridge university press.
  42. Sergio Sayago, Barbara Barbosa Neves, and Benjamin R Cowan. 2019. Voice as- sistants and older people: some open issues. In Proceedings of the 1st International Conference on Conversational User Interfaces. 1-3.
  43. Alex Sciuto, Arnita Saini, Jodi Forlizzi, and Jason I Hong. 2018. "Hey Alexa, What's Up?" A Mixed-Methods Studies of In-Home Conversational Agent Usage. In Proceedings of the 2018 Designing Interactive Systems Conference. 857-868.
  44. Katie Seaborn, Norihisa P Miyake, Peter Pennefather, and Mihoko Otake- Matsuura. 2021. Voice in Human-Agent Interaction: A Survey. ACM Computing Surveys (CSUR) 54, 4 (2021), 1-43.
  45. Jaisie Sin, Rachel L. Franz, Cosmin Munteanu, and Barbara Barbosa Neves. 2021. Digital Design Marginalization: New Perspectives on Designing Inclusive Inter- faces. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 380, 11 pages. https://doi.org/10.1145/3411764.3445180 · DOI
  46. George M Slavich, Sara Taylor, and Rosalind W Picard. 2019. Stress measurement using speech: Recent advancements, validation issues, and ethical and privacy considerations. Stress 22, 4 (2019), 408-413.
  47. Brodrick Stigall, Jenny Waycott, Steven Baker, and Kelly Caine. 2019. Older Adults' Perception and Use of Voice User Interfaces: A Preliminary Review of the Computing Literature. In Proceedings of the 31st Australian Conference on Human-Computer-Interaction. 423-427.
  48. Erik Stone, Marjorie Skubic, Marilyn Rantz, Carmen Abbott, and Steve Miller. 2015. Average in-home gait speed: Investigation of a new metric for mobility and fall risk assessment of elders. Gait & posture 41, 1 (2015), 57-62.
  49. Sarah Strohkorb Sebo, Margaret Traeger, Malte Jung, and Brian Scassellati. 2018. The ripple effects of vulnerability: The effects of a robot's vulnerable behavior on trust in human-robot teams. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. 178-186.
  50. Lucy Suchman and Lucy A Suchman. 2007. Human-machine reconfigurations: Plans and situated actions. Cambridge university press.
  51. John C Tang. 1991. Findings from observational studies of collaborative work. International Journal of Man-machine studies 34, 2 (1991), 143-160.
  52. Milka Trajkova and Aqueasha Martin-Hammond. 2020. "Alexa is a Toy": Exploring Older Adults' Reasons for Using, Limiting, and Abandoning Echo. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1-13.
  53. Joseph Turow. 2021. 3 An Operating System for Your Life. In The Voice Catchers. Yale University Press, 110-150.
  54. Qifei Wang, Junjie Ke, Joshua Greaves, Grace Chu, Gabriel Bender, Luciano Sbaiz, Alec Go, Andrew Howard, Ming-Hsuan Yang, Jeff Gilbert, Peyman Milanfar, and Feng Yang. 2021. Multi-Path Neural Networks for On-Device Multi-Domain Visual Classification. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision (WACV). 3019-3028.
  55. Laurie Weingart, Philip Smith, and Mara Olekalns. 2004. Quantitative coding of negotiation behavior. International negotiation 9, 3 (2004), 441-456.
  56. Samuel Yang, Jennifer Lee, Emre Sezgin, Jeffrey Bridge, and Simon Lin. 2021. Clinical Advice by Voice Assistants on Postpartum Depression: Cross-Sectional Investigation Using Apple Siri, Amazon Alexa, Google Assistant, and Microsoft Cortana. JMIR mHealth and uHealth 9, 1 (2021), e24045.
  57. Po-Yin Yen and Suzanne Bakken. 2012. Review of health information technology usability study methodologies. Journal of the American Medical Informatics Association 19, 3 (2012), 413-422.

Cited by

Sorted by citations

Authors
Cuadra, A., Baek, H., Estrin, D., Jung, M., & Dell, N.
Affiliations
  • Stanford University (US)
  • Cornell Tech (US)
  • Cornell University (US)
ORCID
Venue
International Conference on Information & Communication Technologies and Development 2022
Publisher
ACM
Published
2022-06-27
Pages
1-17
DOI
10.1145/3572334.3572371
OpenAlex
W4372215795
Cited by
16
References
0
Citations by year
2026: 5 · 2025: 4 · 2024: 3 · 2023: 4
FWCI
9.54
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
International Conference on Information and Communication Technologies and Development (ICTD) (publishedVersion)
Funders
National Science Foundation (#2026577 and #1700832)
Topics
  • Social Sciences › Demography › Technology Use by Older Adults
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
older adults, voice assistants, digital inclusion, multimodal interaction, nonverbal cues, body language, prosody, privacy, food pantries, public settings, population aging, assistive technology design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review