CoChat

Understanding older people's voice interactions with smart voice assistants: a new modified rule-based natural language processing model with human input

Open access
Yan, Zhengxu, Dube, Victoria, Heselton, Judith, Johnson, Kate, Yan, Changmin, Jones, Valerie, Blaskewicz Boron, Julie, Shade, Marcia · 2024 · Frontiers in Digital Health

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 10 of the 19 sources its claims cite were checked for support.

Abstract

The COVID-19 pandemic has expedited the integration of Smart Voice Assistants (SVA) among older people. The qualitative data derived from user commands on SVA is pivotal for elucidating the engagement patterns of older individuals with such systems. However, the sheer volume of user-generated voice interaction data presents a formidable challenge for manual coding. Compounding this issue, age-related cognitive decline and alterations in speech patterns further complicate the interpretation of older users’ SVA voice interactions. Conventional dictionary-based textual analysis tools, which count word frequencies, are inadequate in capturing the evolving and communicative essence of these interactions that unfold over a series of dialogues and modify with time. To address these challenges, our study introduces a novel, modified rule-based Natural Language Processing (MR-NLP) model augmented with human input. This reproducible approach capitalizes on human-derived insights to establish a lexicon of critical keywords and to formulate rules for the iterative refinement of the NLP model. English speakers, aged 50 or older and residing alone, were enlisted to engage with Amazon Alexa™ via predefined daily routines for a minimum of 30 min daily spanning three months ( N = 35, mean age = 77). We amassed time-stamped, textual data comprising participants’ user commands and responses from Alexa™. Initially, a subset constituting 20% of the data (1,020 instances) underwent manual coding by human coder, predicated on keywords and commands. Separately, a rule-based Natural Language Processing (NLP) methodology was employed to code the identical subset. Discrepancies arising between human coder and the NLP model programmer were deliberated upon and reconciled to refine the rule-based NLP coding framework for the entire dataset. The modified rule-based NLP approach demonstrated notable enhancements in efficiency and scalability and reduced susceptibility to inadvertent errors in comparison to manual coding. Furthermore, human input was instrumental in augmenting the NLP model, yielding insights germane to the aging adult demographic, such as recurring speech patterns or ambiguities. By disseminating this innovative software solution to the scientific community, we endeavor to advance research and innovation in NLP model formulation, subsequently contributing to the understanding of older people's interactions with SVA and other AI-powered systems.

Overview

CoChat Summary

TL;DR

This study introduces a modified rule-based NLP (MR-NLP) approach, refined with human input, to code older adults’ smart voice assistant interactions more efficiently and with fewer errors than manual coding. Using data from 35 older adults over 12 weeks, the authors report large time savings and mostly strong agreement between human and MR-NLP coding, though human coding produced a different factor structure.

Key findings

  • Manual coding took an average of 3 h per participant (range 2–5 h), whereas the rule-based NLP method processed 35 participants in 9 h; the NLP method used only 9% of the time needed for manual coding.
  • On the initial 20% subset (n = 7; 1,020 cases), manual vs. MR-NLP coding showed 85.49% agreement (872 cases) and Cohen’s kappa = 0.82.
  • Of the 148 discrepancies (14.51%), 0.29% (3 cases) were due to program errors, 7.68% (67 cases) to human errors, and 6.54% (78 cases) to coding-definition differences.
  • In factor analysis of the 12 interaction categories, MR-NLP coding and initial NLP coding each yielded four dimensions explaining 92% and 91.69% of variance, respectively, while manual coding yielded five dimensions; the authors say this supported H3.
  • After expanding comparison, the paper reports 99% agreement and 0.99 Cohen’s kappa between human coding and NLP coding, with the remaining 1% disagreement attributed to human fatigue.

Methods

  • The study recruited 35 English-speaking adults aged 50+ living alone in independent living facilities, with mean age 77; participants used Amazon Alexa for at least 30 min daily over 12 weeks.
  • Participants’ Alexa commands and responses were extracted from time-stamped account data; a 20% subset (n = 7; 1,020 cases) was manually coded and then coded with a rule-based NLP system, with discrepancies discussed and used to refine the model.
  • Analysis compared efficiency (coding time), error/consistency (agreement, Cohen’s kappa), and interpretive impact via principal-components factor analysis with varimax rotation on the 20% subset.

Limitations

  • The authors state that only 20% of the data was manually coded initially, which could miss less frequent but meaningful interaction patterns and nuances in daily routines.
  • They note limited generalizability because the sample was predominantly White, English-speaking, and from independent living facilities in one region.
  • They also state that the study compared MR-NLP mainly with manual coding, not with other automated NLP models, so broader comparative performance and scalability remain untested.
  • Evident from the text: the study’s factor-analysis comparison and reported agreement are based on a subset, so claims about the full dataset depend on extrapolation from that subset.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/fdgth.2024.1329910

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

16 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Figures

Figure 1 Text normalization by sorting the data by weeks.
Figure 1 Text normalization by sorting the data by weeks.CC BY · PubMed Central
Figure 2 Data sense-making by rule-based NLP or modified rule-based NLP (MR-NLP).
Figure 2 Data sense-making by rule-based NLP or modified rule-based NLP (MR-NLP).CC BY · PubMed Central

Verification

Claims the paper makes

8 Valid · 0 Not valid · 6 Not checked · 72 Own results and interpretations

33 more claims

Create a free account to see all 86 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 31
  • It may state two separate claims as one. 25
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upt(33) = -0.17reported p = .88 · recomputed p = .866

    “t(33)=−0.17, p = .88”

  • Consistentt(33) = 0.46reported p = .65 · recomputed p = .649

    “t(33) = 0.46, p = .65”

  • Consistentt(33) = -0.69reported p = .49 · recomputed p = .495

    “t(33)=−0.69, p = .49”

  • Consistentt(33) = 0reported p = 1 · recomputed p = 1.000

    “t(33) = 0, p = 1.00”

  • Consistentt(33) = 1.12reported p = .27 · recomputed p = .271

    “t(33) = 1.12, p = .27”

5 of 16 · the rest aren’t loaded

11 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Citations

Cites

Show all 37 references

Research works printed but not linked: 2

  1. 2021, 2021 Tech Trends and the 50-Plus
  2. Saldaña, 2016, The Coding Manual for Qualitative Researchers

Cited by

See all 14

Metadata

Authors
Yan, Z., Dube, V., Heselton, J., Johnson, K., Yan, C., Jones, V., Blaskewicz Boron, J., & Shade, M.
Affiliations
  • University of California, Berkeley (US)
  • University of Nebraska at Omaha (US)
  • University of Nebraska–Lincoln (US)
  • University of Nebraska Medical Center (US)
Corresponding author
Yan, Changmin
Venue
Frontiers in Digital Health
Published
2024-05-14
Volume
6
Pages
1329910-1329910
ISSN
2673-253X (ISSN-L)
OpenAlex
W4396900669
Cited by
14
References
37
Citations by year
2026: 7 · 2025: 6 · 2024: 1
FWCI
11.98
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Digital Health (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Lincoln (University of Nebraska) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Geriatrics and Gerontology › Technology Use by Older Adults
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
Keywords
older adults, voice interaction, rule-based natural language processing, human input, Amazon Alexa, user commands, age-related cognitive decline, speech patterns, COVID-19 pandemic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Understanding older people's voice interactions with smart voice assistants: a new modified rule-based natural language processing model with human input. https://app.cochat.ai/papers/10.3389/fdgth.2024.1329910
The paper
Yan, Z., Dube, V., Heselton, J., Johnson, K., Yan, C., Jones, V., Blaskewicz Boron, J., & Shade, M. (2024). Understanding older people's voice interactions with smart voice assistants: a new modified rule-based natural language processing model with human input. Frontiers in Digital Health. https://doi.org/10.3389/fdgth.2024.1329910

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.