The DOI matches its Crossref record; 31 of the 33 references it cites were found in a registry or an index; 2 of the 31 sources its claims cite were checked for support.
Field evaluation with cognitively-impaired older adults of attention management in the Embodied Conversational Agent Louise
Open accessAbstract
We present the first experiment we conducted to evaluate the attention monitoring performance of Louise, following a Wizard of Oz method, during the interactions with a cohort of 8 elderly users in a day hospital environment. Louise is a new, semi-automatic prototype of an Embodied Conversational Agent (ECA), a virtual character interacting with users through social-like communication, adapted to the special needs of older adults with cognitive impairment; it is intended to ultimately provide assistance in their activities of daily living. We recorded and analyzed both videos of the conversation-like interactions and Louise's tracking data. In our experiment, Louise's attention estimation algorithm achieved about 80% accuracy; moreover, in almost all cases, the user's attention was successfully recaptured by Louise after a planned, experimenter-induced distraction. These results are in line with what was observed in previous experiments involving only younger adults, thus suggesting that attention measurement tools embedded in cognitive prostheses will not need to be adapted to elderly patients. Finally, to gain further insights on conversation management and provide evidence-based suggestions for future work, we performed an anthropological analysis of the whole experiment.
Overview
CoChat Summary
TL;DR
The paper evaluates attention monitoring and attention-recapture in the Embodied Conversational Agent Louise with 8 cognitively impaired and healthy older adults in a day hospital, using a Wizard of Oz setup. Louise’s attention estimator reached about 80% accuracy, with no significant difference from younger adults, and attention was successfully recaptured after almost all experimenter-induced distractions.
Key findings
- Louise’s attention estimation algorithm achieved about 80% accuracy; overall performance was 84% on all data (AURC = 0.68), or 86% (AURC = 0.74) after removing skeleton tracking errors (3.3% of samples). Table I reports 76% on all samples, 80% with no tracking errors, 72% for participants with cognitive impairment only, and 77% for that group with no tracking errors.
- Group comparison with 14 healthy younger adults showed no significant differences in medians (p > 0.05), suggesting attention measurement tools embedded in cognitive prostheses may not need adaptation for elderly patients.
- Attention recapture was effective: across 19 distractions, the experimenter had to ask the participant to look at the screen only once; prompting phrases were triggered only 6 times in 19 distractions.
- 6 of 8 participants successfully interacted with Louise; one had poor hearing, and one with the highest cognitive impairment (MMSE = 17) refused to continue after the first distraction. Anthropological indicators showed PWD vs control subjects averaged word count 60.7/26.5, new information 6.0/2.5, and silence length 1.32/0.83 seconds.
Methods
- Wizard of Oz study with the semi-automated Louise ECA, using a Microsoft Kinect; the attention estimator combined shoulder orientation and head yaw/pitch into a weighted attention level (0–10) with hysteresis thresholds: engaged if >8, disengaged if <6.
- Participants were 8 older adults (2 males, 6 females; ages 63–91, mean 78 ± 1 years); 6 had cognitive impairment (3 MCI, 3 AD; MMSE 17–29, mean 23 ± 3) and 2 did not. The interaction included 3 introduction utterances, 7 questions, 5 acknowledgments, 5 prompting utterances, a transition, and 2 conclusion utterances, with two experimenter-induced distractions at fixed moments.
- Videos were annotated by three experts; algorithm decisions were compared with annotations using correct detection rates, ROC/AURC, point-biserial correlations, and a Wilcoxon rank sum test against 14 younger adults. An anthropological analysis used ELAN coding, expert interviews, and a data session.
Limitations
- Stated: the sample was small, and the authors say findings should be confirmed with a larger sample; the anthropological silence findings also need further research due to the small number of participants.
- Stated: the day hospital setting and experimenter presence likely produced a “white-blouse effect” and a three-party interaction, limiting ecological validity; future experiments in a more casual context with more participants are warranted.
- Stated/evident: tracking failures occurred (one complete Kinect failure, brief body/face tracking loss, and major failures in the cognitive impairment group), and participant factors such as poor hearing and one refusal reduced usable data. Evident: no self-induced distractions due to losing track of the task were observed, so attention recapture for that scenario remains untested.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 15 Not checked · 73 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“It combines the measures of orientation of the user's shoulders [15] [16] and head pose [17], seen here as proxies for his or her intensity of attention towards Louise.”
What the sources say
- Not checked — no full text of the sourcecites [15] [15] W. Benkaouar and D. Vaufreydaz, 'Multi-sensors engagement detection with a robot companion in a home environment,' in Workshop on Assistance and Service robotics in a human environment at IEEE International Conference on Intelligent Robots and Systems (IROS2012) , Oct 2012, p. 45-52. [Online]. Available: http://hal.inria.fr/hal-00735150/ -
- Not checked — no full text of the sourcecites [16] [16] L. Li, Q. Xu, and Y. K. Tan, 'Attention-based addressee selection for service and social robots to interact with multiple persons,' in Proceedings of the Workshop at SIGGRAPH Asia , ser. WASA '12. New York, NY, USA: ACM, Nov 2012, pp. 131-136. - source
- Supportedcites [17] [17] R. Stricker, S. Hommel, C. Martin, and H.-M. Gross, 'Realtime user attention and emotion estimation on a mobile robot,' in Proceedings of 55th Ilmenau International Scientific Colloquium , Sep 2010, p. 13-17. -
Section 4.2.2, “Measuring Visual Attention,” says that head pose over time can be used to measure a user's visual focus and attention. This source is cited for head pose; the shoulder-orientation citations are separate, and “towards Louise” describes the citing authors' application. (LLM, conf=0.91)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In previous work [5], this attention estimation algorithm was evaluated with younger adults, who are assistive technology experts. The method showed over 80% accuracy with this group.”
What the sources say
- Partly supportedcites [5] [5] P. Wargnier, A. Malaise, J. Jacquemot, S. Benveniste, P. Jouvelot, M. Pino, and A.-S. Rigaud, 'Towards attention monitoring of older adults with cognitive impairment during interaction with an embodied conversational agent,' in Virtual and Augmented Assistive Technology (VAAT), 2015 3rd IEEE VR International Workshop on , March 2015, pp. 23-28. - sourceExperimental — accuracy not yet measured
The only text our copy holds for this paper is the HAL record/abstract, which states 'We evaluated the performance of Louise with a group of experts in assistive technologies' and 'Louise's simple attention estimator is more than 80% accurate' (Résumé, abstract section) — so the assistive-technology-expert group and the >80% accuracy are supported. The additional qualifier 'younger adults' is not stated anywhere in our copy: repeated searches (evaluation with experts / young participants / mean age / user trial participants) returned only the abstract and bibliographic metadata, never a description of the evaluators' age. Our copy therefore supports the accuracy and expert-group parts but not the age descriptor, and the paper's body where participant demographics would appear is not available to us. (LLM, conf=0.68)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“moreover, in almost all cases, the user's attention was successfully recaptured by Louise after a planned, experimenter-induced distraction.”
- Also stated in: § I. INTRODUCTION · p. 2 · plain
“moreover, in almost all cases, the user's attention was successfully recaptured by Louise after a planned, experimenter-induced distraction.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § I. INTRODUCTION · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“6 participants had cognitive impairment (3 MCI and 3 AD).”
plainn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“word count (H1, 60.7/26.5)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
48 more claims
Create a free account to see all 92 claims and to verify the 1 reference still open.
- The quote only partly states this claim. 1
- It may state two separate claims as one. 16
- Probably not a substantial claim. 29
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“CAs have shown to provide useful and usable support for older adults with MCI across a variety of functions, from information searching to calendaring, especially when sufcient training is provided.”
From A Collaborative Approach to Support Medication Management in Older Adults with Mild Cognitive Impairment Using Conversational Assistants (CAs) (2022) · cites this paper as [46]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“More recently, Wargnier et al. (2016) have evaluated a low-level attention monitoring and management module with a small sample of older adults with mild cognitive impairment; their system performed as well as with the control group.”
From Enabling robust and fluid spoken dialogue with cognitively impaired users (2017) · cites this paper as Wargnier et al., 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The two latter teams also mentioned that interactions were unsuccessful for only their respective participant with the most overtly noticeable impairments.”
From Enabling robust and fluid spoken dialogue with cognitively impaired users (2017) · cites this paper as Wargnier et al., 2016
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unfortunately, our study confirms that the quality of automatic speech recognition and synthesis is still a technical issue and has room for improvement [7-11], whereas touch screen modality is almost stable and positively used by people living with dementia [3,25,26].”
From Usability and Acceptance of the Embodied Conversational Agent Anne by People with Dementia and their Caregivers: an exploratory study in home environment settings (Preprint) (2020) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While most of the existing work focuses on older adults and their interactions with CAs, the challenge of designing for compensatory cognitive support is underexplored, with the exception of a few studies that address cognitive behaviors as a result of MCI [20,45,46].”
From A Collaborative Approach to Support Medication Management in Older Adults with Mild Cognitive Impairment Using Conversational Assistants (CAs) (2022) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ IV. EXPERIMENTS WITH COGNITIVELY-IMPAIRED OLDER ADULTS · p. 5
“The average age of the subjects was 78 ± 1 years old ( mean ± standard deviation ).”
§ IV. EXPERIMENTS WITH COGNITIVELY-IMPAIRED OLDER ADULTS · p. 5
“All participants were older adults from 63 to 91 years old.”
Citations
Cites
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Biometrics Bulletin · doi:10.2307/3001968 · Cited by 15,354
Foundations in Sociolinguistics: An Ethnographic Approach
Kauffman, L. E., & Hymes, D. (1976) · Anthropological Quarterly · doi:10.2307/3316843 · Cited by 3,195
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_16 · Cited by 2,915
Polinsky, M., & Lambrecht, K. (1999) · Language · doi:10.2307/417062 · Cited by 1,616
2 On the Preferences for Agreement and Contiguity in Sequences in Conversation
Sacks, H. (1987) · Talk and Social Organisation · doi:10.21832/9781800418226-004 · Cited by 861
Fisk, A. D., Rogers, W. A., Charness, N., Czaja, S. J., & Sharit, J. (2004) · doi:10.1201/9781420023862 · Cited by 781
Foundations of Sociolinguistics: An Ethnographic Approach
Thomaneck, J. K. A., & Hymes, D. (1980) · The British Journal of Sociology · doi:10.2307/590085 · Cited by 516
Annotation by Category: ELAN and ISO DCR
Sloetjes, H., & Wittenburg, P. (2008) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/454o8uo6f7z8 · Cited by 469
O'Reilly, K. (2012) · Routledge eBooks · doi:10.4324/9780203864722 · Cited by 467
Designing for Older Adults: Principles and Creative Human Factors Approaches, Second Edition
Fisk, A. D., Czaja, S. J., Rogers, W. A., Charness, N., Czaja, S. J., & Sharit, J. (2018) · doi:10.1201/9781420080681 · Cited by 396
Research works printed but not linked: 3
- wargnier, 0, List of utterances
- pino, 2014, User-driven innovation for dementia care in France: The LUSAGE living lab case study, J Interdisciplinary Studies, 3, 251
- 2012, Dementia A Public Health Priority
Cited by
Survey of conversational agents in health
Montenegro, J. L. Z., da Costa, C. A., & da Rosa Righi, R. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.03.054 · Cited by 322
Face Value? Exploring the Effects of Embodiment for a Group Facilitation Agent
Shamekhi, A., Liao, Q., Wang, D., Bellamy, R., & Erickson, T. D. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173965 · Cited by 161
“Wizard-of-oz is a commonly used approach to study user interactions with conversational agents [4, 60, 8].”
Security, privacy, and healthcare-related conversational agents: a scoping review
May, R., & Denecke, K. (2021) · Informatics for Health and Social Care · doi:10.1080/17538157.2021.1983578 · Cited by 104
Huq, S. M., Maskeliūnas, R., & Damaševičius, R. (2022) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2022.2146768 · Cited by 106
“Louse was used for field evaluation study of attention management with cognitively-impaired older adults [48].”
MethodKuttal, S. K., Ong, B., Kwasny, K., & Robe, P. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445659 · Cited by 44
“unique qualities of human-agent interactions [41], and study user interactions with conversational agents [17, 18, 173].”
Background
Metadata
- Authors
- Wargnier, P., Carletti, G., Laurent-Corniquet, Y., Benveniste, S., Jouvelot, P., & Rigaud, A.-S.
- Affiliations
- École Nationale Supérieure des Mines de Paris (FR)
- Assistance Publique – Hôpitaux de Paris (FR)
- Centre d’Expertise National en Stimulation Cognitive (FR)
- Université Paris Descartes (FR)
- Venue
- 2016 IEEE International Conference on Serious Games and Applications for Health (SeGAH)
- Publisher
- IEEE
- Published
- 2016-05
- Pages
- 1-8
- ISSN
- 2330-5649 (ISSN-L)
- OpenAlex
- W2255044263
- Cited by
- 28
- References
- 33
- Citations by year
- 2025: 1 · 2024: 3 · 2023: 1 · 2022: 6 · 2021: 5 · 2020: 4 · 2019: 1 · 2018: 4 · 2017: 2 · 2016: 1
- FWCI
- 4.50
- Type
- conference paper
- Language
- en
- Open access
- Licence
- other-oa
- Locations
- SeGAH ... IEEE ... International Conference on Serious Games and Applications for Health/IEEE International Conference on Serious Games and Applications for Health (publishedVersion)
- HAL (submittedVersion, other-oa)
- HAL (submittedVersion)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- embodied conversational agents, attention monitoring, attention estimation, cognitive impairment, older adults, Wizard of Oz, day hospital, activities of daily living, cognitive prostheses, conversation management, anthropological research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Field evaluation with cognitively-impaired older adults of attention management in the Embodied Conversational Agent Louise. https://app.cochat.ai/papers/10.1109/segah.2016.7586282
- The paper
- Wargnier, P., Carletti, G., Laurent-Corniquet, Y., Benveniste, S., Jouvelot, P., & Rigaud, A.-S. (2016). Field evaluation with cognitively-impaired older adults of attention management in the Embodied Conversational Agent Louise. 2016 IEEE International Conference on Serious Games and Applications for Health (SeGAH). https://doi.org/10.1109/segah.2016.7586282
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.