CoChat
Sign Up
Cited by 64Social Robot Interaction and HRIEmotion and Mood RecognitionTelemedicine and Telehealth Implementation

Robotic Telemedicine for Mental Health: A Multimodal Approach to Improve Human-Robot Engagement

Open access
Lima, Maria R., Wairagkar, Maitreyee, Natarajan, Nirupama, Vaitheswaran, Sridhar, Vaidyanathan, Ravi·2021·Frontiers in Robotics and AI
DOI 10.3389/frobt.2021.618866PMC PMC8014955
Open this paper in CoChat — free accountOpen PDF

Abstract

COVID-19 has severely impacted mental health in vulnerable demographics, in particular older adults, who face unprecedented isolation. Consequences, while globally severe, are acutely pronounced in low- and middle-income countries (LMICs) confronting pronounced gaps in resources and clinician accessibility. Social robots are well-recognized for their potential to support mental health, yet user compliance (i.e., trust) demands seamless affective human-robot interactions; natural ‘human-like’ conversations are required in simple, inexpensive, deployable platforms. We present the design, development, and pilot testing of a multimodal robotic framework fusing verbal (contextual speech) and nonverbal (facial expressions) social cues, aimed to improve engagement in human-robot interaction and ultimately facilitate mental health telemedicine during and beyond the COVID-19 pandemic. We report the design optimization of a hybrid face robot, which combines digital facial expressions based on mathematical affect space mapping with static 3D facial features. We further introduce a contextual virtual assistant with integrated cloud-based AI coupled to the robot’s facial representation of emotions, such that the robot adapts its emotional response to users’ speech in real-time. Experiments with healthy participants demonstrate emotion recognition exceeding 90% for happy, tired, sad, angry, surprised and stern/disgusted robotic emotions. When separated, stern and disgusted are occasionally transposed (70%+ accuracy overall) but are easily distinguishable from other emotions. A qualitative user experience analysis indicates overall enthusiastic and engaging reception to human-robot multimodal interaction with the new framework. The robot has been modified to enable clinical telemedicine for cognitive engagement with older adults and people with dementia (PwD) in LMICs. The mechanically simple and low-cost social robot has been deployed in pilot tests to support older individuals and PwD at the Schizophrenia Research Foundation (SCARF) in Chennai, India. A procedure for deployment addressing challenges in cultural acceptance, end-user acclimatization and resource allocation is further introduced. Results indicate strong promise to stimulate human-robot psychosocial interaction through the hybrid-face robotic system. Future work is targeting deployment for telemedicine to mitigate the mental health impact of COVID-19 on older adults and PwD in both LMICs and higher income regions.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper presents a low-cost hybrid-face social robot with digital facial expressions and cloud-based conversational AI, tested with healthy UK participants and piloted as a telemedicine cognitive-engagement tool for older adults and people with dementia in India. It reports high emotion recognition in UK healthy participants, strong preference for multimodal interaction, and feasibility/cultural acceptability in South India, while noting lower robot-expression recognition in Indian older/PwD cohorts and technical and sample-size limitations.

Key findings

  • In UK healthy participants (N = 15, 23–49 years), optimized robot expressions were all recognized above 70%: happy and tired 100%, angry/surprised/sad 93.3%, afraid 86.7%, and stern and disgusted 73.3% each, with stern/disgusted often confused with one another; disgusted improved from 18.4% to 73.3% and afraid from 44% to 86.7% versus prior work.
  • In the UEQ with N = 10 healthy volunteers, all 10 preferred the multimodal system over pure voice communication, 90% found the robotic platform easy to use, and 70% found voice interaction extremely useful; novelty and stimulation received the most positive ratings, while efficiency was least positive and 20% had difficulty identifying emotion from combined speech and facial expression.
  • In South India, with N = 14 people with dementia (PwD) and N = 26 healthy older adults, recognition of the robot’s simulated expressions was 14–62%, lower than recognition of validated Indian human pictures; afraid, angry, and disgusted were lowest (14–20%), and happy recognition was double for healthy controls compared with PwD.
  • In the telemedicine pilot with one PwD (67 years, male, vascular dementia, CDR 1), three 30-min remote sessions showed engagement duration increasing from 9 min 35 s to 18 min 1 s; “talking to robot” received the highest possible score of 3 in all sessions, “disruptive” received 0 in all sessions, and mood was rated “very happy” pre- and post-session.

Methods

  • The hybrid face robot used 13 degrees of freedom, mathematical affect-space emotion mapping, optional 3D-printed faceplate, and an intelligent virtual assistant integrating IBM Watson Assistant, IBM Tone Analyzer, Google Speech-to-Text, and Google Text-to-Speech via a Python orchestrator.
  • UK evaluation included a forced-choice emotion recognition task with N = 15 healthy participants and a user experience questionnaire (UEQ) with N = 10 healthy volunteers, plus qualitative interviews.
  • India evaluation included user-centred cultural validation via focus groups/in-depth interviews, emotion recognition tasks with N = 14 PwD and N = 26 healthy older adults using Indian validated pictures and robot expressions, and a Wizard-of-Oz telemedicine pilot with one PwD over three 30-min sessions using pre/post face scale and modified observational measure of engagement (OME).

Limitations

  • Stated: the AI conversational system needs more natural and unstructured dialogue and adaptation for mental health applications; future work should include more training data, different voices/speaking styles, and machine learning for personalization over time.
  • Stated: the India pilot included only one person with dementia; the remote setup required PwD and clinician to use the same internet network; lags and voice distortion were main technical issues; future studies need more participants, different dementia types/stages, and stricter controls for age, experience with interactive technologies, cognitive ability, and education.
  • Evident from the text: the UK emotion recognition sample was small and mostly male (12 male, 3 female), participants did not interact with the robot using different modalities in a controlled comparison, robot-expression recognition in India was much lower than in the UK, and 13 of 15 participants disliked the added 3D faceplate, suggesting an uncanny valley effect.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/frobt.2021.618866

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 38 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    A qualitative user experience analysis indicates overall enthusiastic and engaging reception to human-robot multimodal interaction with the new framework.

    “A qualitative user experience analysis indicates overall enthusiastic and engaging reception to human-robot multimodal interaction with the new framework.”
    Quote in the paperSubstantial claim
  • Findingown data

    Experiments with healthy participants demonstrate emotion recognition exceeding 90% for happy, tired, sad, angry, surprised and stern/disgusted robotic emotions.

    “Experiments with healthy participants demonstrate emotion recognition exceeding 90% for happy, tired, sad, angry, surprised and stern/disgusted robotic emotions.”
    Quote in the paperSubstantial claim
  • Findingown data

    On average, the novelty and stimulation aspects of the UEQ received the most positive ratings.

    “On average, the novelty and stimulation aspects have received the most positive ratings.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Participants perfectly identified the emotions for happy and tired.

    “Participants perfectly identified the emotions for happy and tired.”
    Quote in the paperNot a substantial claim
  • Findingown data

    All facial expressions of the hybrid face robot showed high recognition rates above 70% in the healthy-participant emotion recognition task.

    “All facial expressions showed high recognition rates above 70%.”
    Quote in the paperSubstantial claim
  • May combine two claims. 42
  • Classified as probably not a substantial claim. 43

  • May combine two claims.

    Findingown data

    When separated, stern and disgusted robotic emotions are occasionally transposed, with 70%+ accuracy overall, but are easily distinguishable from other emotions.

    “When separated, stern and disgusted are occasionally transposed (70%+ accuracy overall) but are easily distinguishable from other emotions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Participants' feedback indicated that the addition of the faceplate was perceived unfavourably and the physical depth was poorly perceived.

    “Despite the added human features, participants’ feedback indicated the addition of the faceplate was perceived unfavourably and the physical depth was poorly perceived.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Although the tired expression showed 100% recognition, participants reported some confusion with stern and disgusted.

    “Though the tired expression showed 100% recognition, participants reported some confusion with stern and disgusted, which can be due to their similarity of partially-closed eyelids and low eyebrows.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All respondents consistently considered the robot as a creative and inventive robotic platform and showed enthusiasm in having new interactions with the robot.

    “Notably, all respondents consistently considered the robot as a creative and inventive robotic platform (n1 and n2) and showed enthusiasm in having new interactions with the robot (s3).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Disgusted and stern showed the lowest recognition rates (73.3%) and were often confused between one another.

    “Disgusted and stern showed the lowest recognition rates (73.3%) and were often confused between one another, which is in line with our past results.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Accuracies for all of the robot's simulated facial expressions were lower than expected (between 14 and 62%), with similar values for both people with dementia and healthy controls.

    “We found accuracies for all robot’s simulated facial expressions were lower than expected (between 14 and 62%), with similar values for both PwD and HC.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The participant enjoyed interacting with the robot in all three sessions, and his mood was rated "very happy" on the pre and post measures of the modified face scale before and after each session.

    “The participant enjoyed interacting with the robot in all three sessions and his mood was rated “very happy” on the pre and post measures of the modified face scale before and after each session, respectively.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    The caregiver reported a positive impression of using the hybrid face robot as a telemedicine tool for cognitive engagement and perceived it as a technology to help with her husband's condition.

    “The caregiver reported a positive impression of using the hybrid face robot as a telemedicine tool for cognitive engagement and perceived it as a technology to help with her husband’s condition.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A discrepancy was observed between cultural recognition of human emotions and robotic simulated expressions, but both people with dementia and healthy older participants perceived the robot's facial representation of emotion comparably.

    “Hence, a discrepancy was observed between cultural recognition of human emotions vs. robotic simulated expressions, however both PwD and healthy older participants perceived the robot’s facial representation of emotion comparably.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The participant had no difficulty talking to the robot as assessed by the OME "Talking to robot" measure, which received the highest possible score of engagement in all three sessions.

    “The participant had no difficulty talking to the robot as assessed by OME “Talking to robot” measure, which received the highest possible score of engagement in all three sessions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    The clinicians who conducted the cognitive engagement sessions commented that the use of affective robotic platforms for engaging persons with dementia holds significant promise.

    “The clinicians who conducted cognitive engagement sessions commented that the use of affective robotic platforms for engaging persons with dementia, who otherwise are unable to participate in many activities due to the restrictions imposed by pandemic scenarios, holds significant promise.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The robot's ability to adapt its animated facial representation of emotion in response to user speech in real-time stood out.

    “Particularly, the robot’s ability to adapt its animated facial representation of emotion in response to user speech in real-time stood out.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The robot was considered a viable, low-cost and culturally appropriate tool to assist in clinical cognitive engagement with healthy older adults and people with dementia in India.

    “The preliminary indications are promising as the robot was considered a viable, low-cost and culturally appropriate tool to assist in clinical cognitive engagement with healthy older adults (healthy control—“HC”) and PwD in India.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    There were problems in streaming music.

    “3) problems in streaming music;”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    As observed by the nursing assistant, the technical issues often resulted in distraction, impacting the participant's engagement.

    “As observed by the nursing assistant, these technical issues often resulted in distraction, impacting participant’s engagement, yet the advantages of robot-assisted cognitive engagement with PwD were acknowledged.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Recognition rates of human pictures were similarly high for people with dementia and healthy controls for all emotions except afraid, which was well recognized by healthy controls but showed an accuracy below 50% for people with dementia.

    “Recognition rates of human pictures were similarly high for PwD and HC for all emotions except for afraid, which was well recognized by HC but showed an accuracy below 50% for PwD.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Stakeholders concurred that the possibility of using a hybrid face robot for remote cognitive engagement can potentially help in meaningful engagement of people with dementia at home and alleviate the perils of social isolation.

    “Stakeholders concurred that the possibility of using a hybrid face robot for remote cognitive engagement can potentially help in meaningful engagement of people with dementia at home and also alleviate the perils of social isolation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There was an overall easiness in interacting and getting familiar with the robot, and 90% of participants found the robotic platform easy to use.

    “Regarding questions about perspicuity, there was an overall easiness in interacting and getting familiar with the robot and 90% of participants found the robotic platform easy to use (p3).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The efficiency aspect showed the less positive answers, and 20% of participants claimed difficulty identifying the emotion conveyed by the combination of robotic speech and facial expression.

    “The efficiency aspect showed the less positive answers (e1, e2) and 20% of participants claimed difficulty identifying the emotion conveyed by the combination of robotic speech and facial expression.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    There was a strong user preference for multimodal over pure voice communication, and benefits in user engagement of multimodal vs. pure voice communication were qualitatively identified.

    “Although participants did not interact with the robot with different modalities (i.e., speech with and without integration of the robotic expressive face), there was a strong user preference for multimodal over pure voice communication.”
    • Also stated in: § Conclusion · plain
      “We qualitatively identified the benefits in user engagement of multimodal vs. pure voice communication.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    The clinician indicated strong promise in using social robotic platforms as a means of telemedicine for dementia support; the caregiver perceived the robot as a technological tool to help with her husband's condition.

    “Particularly, the clinician indicated strong promise in using social robotic platforms as a means of telemedicine for dementia support; the caregiver also perceived the robot as a technological tool to help with her husband’s condition.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Differences between healthy older adults and PwD in India were marginal except for the happy expression, for which double recognition was obtained for the healthy control group.

    “In our experiments, we further observed differences between healthy older adults and PwD in India were marginal except for the happy expression, for which double recognition was obtained for the healthy control group.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The nursing assistant indicated distraction of PwD when technical issues occurred.

    “Although the nursing assistant indicated distraction of PwD when technical issues occurred, the participant was overall attentive and enjoyed interacting with the robot.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The participant was overall attentive and enjoyed interacting with the robot.

    “Although the nursing assistant indicated distraction of PwD when technical issues occurred, the participant was overall attentive and enjoyed interacting with the robot.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Robotic telemedicine using the hybrid face robot is feasible, culturally appropriate, and low-cost for mental health support in India.

    “Our study demonstrates feasibility and cultural appropriateness of robotic telemedicine for mental health support in India.”
    • Also stated in: § Robotic Telemedicine Beyond COVID-19 · plain
      “In our set of user-centred validation studies, including focus group discussions with stakeholders and emotion recognition experiments with 14 PwD and 26 healthy older adults, we identified the hybrid face robot as a feasible, culturally appropriate, and low-cost telemedicine system to support mental health in India.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A trend of increased duration of engagement with the robot from the first to last session was observed, and a trend of longer duration of engagement with the robot was observed through the protocol to introduce the robot to people with dementia in the cultural context of South India.

    “We observed a trend of increased duration of engagement with the robot from the first to last session, and no alterations in PwD’s mood before and after each session.”
    • Also stated in: § Conclusion · plain
      “Results revealed robot-assisted cognitive engagement sessions are feasible in India (and more broadly LMICs), and a trend of longer duration of engagement with the robot was observed through our protocol to introduce the robot to people with dementia in the cultural context of South India.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Lower recognition rates for all facial expressions simulated by the robot were observed in India compared to increased recognition accuracies obtained from younger participants in the United Kingdom.

    “Despite the increased recognition accuracies obtained from younger participants in the United Kingdom, we observed lower recognition rates for all facial expressions simulated by the robot in India.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Stern and disgusted robotic simulated facial expressions were occasionally mistaken for one another (70%+ accuracy overall) but were easily distinguishable from all other simulated emotions.

    “When separated, stern and disgusted were occasionally mistaken for one another (70%+ accuracy overall) but were easily distinguishable from all other simulated emotions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Lags and distortion of voice during remote clinical sessions were the main technical issue to solve for next trials.

    “We identified lags and distortion of voice during remote clinical sessions as the main technical issue to solve for next trials, in order to ensure maximum engagement.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    COVID-19 has severely impacted mental health in vulnerable demographics, particularly older adults, who face unprecedented isolation.

    “COVID-19 has severely impacted mental health in vulnerable demographics, in particular older adults, who face unprecedented isolation.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Mehra and Grover, 2020

      Not checked in the free run.

    • cites Wang et al., 2020 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    63% of people with dementia already live in LMICs, where 70% of new cases occur.

    “Prevalence of dementia is further skyrocketing in LMICs; 63% of PwD already live in LMICs, where 70% of new cases occur (Prince et al., 2007; Prince et al., 2013).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Prince et al., 2007

      Not checked in the free run.

    • cites Prince et al., 2013

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In India, the treatment gap for dementia is 90%.

    “In India, the treatment gap today is a staggering 90% (Dias and Patel, 2009).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Dias and Patel, 2009

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Social robots are well-recognized for their potential to support mental health.

    “Social robots are well-recognized for their potential to support mental health”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Tapus et al., 2007 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    People with dementia questioned the acceptability of a voice system without a face.

    “One interesting observation that deserves more introspection is that people with dementia questioned the acceptability of a voice system without a face.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wolters et al., 2016 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The need for mental health and psychosocial support of people with dementia and their carers worldwide is well-documented both during and prior to the COVID-19 pandemic.

    “The need for mental health and psychosocial support of PwD and their carers worldwide is well-documented (Wang et al., 2020) both during and prior to the COVID-19 pandemic.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wang et al., 2020 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Ethics clearance for the study with healthy participants was obtained from the Imperial College London Science, Engineering and Technology Research Ethics Committee (SETREC).

    “Ethics clearance was obtained by Imperial College London Science, Engineering and Technology Research Ethics Committee (SETREC).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Realism features such as constant motion of the face, random blinking of the eyes, and pupil dilation can be controlled and may lead to more dynamic human-robot interactions.

    “Realism features, such as constant motion of the face, random blinking of the eyes, and pupil dilation (Bazo et al., 2010; Craig et al., 2010; Wairagkar et al., 2020) can be controlled and may lead to more dynamic HRIs.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Bazo et al., 2010

      Not checked in the free run.

    • cites Craig et al., 2010

      Not checked in the free run.

    • cites Wairagkar et al., 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A recent study conducted with caregivers in South India highlights a clear need for more services and support of people with dementia and caregivers for the post-pandemic, including stronger adoption of technology.

    “A recent study conducted with caregivers in South India (Vaitheswaran et al., 2020) highlights a clear need for more services and support of PwD and caregivers for the post-pandemic, including stronger adoption of technology.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Vaitheswaran et al., 2020 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Written informed consent was obtained from participants in the healthy-participant study.

    “Written informed consent was obtained from participants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The implementation of expressive robotic faces on LCD screens allows easy customization, adaptability to users' preferences and culture, higher accessibility and scalability.

    “The implementation of expressive robotic faces on LCD screens has recently been applied in different SAR platforms, which allows easy customization, adaptability to users’ preferences and culture, higher accessibility and scalability (Abdollahi et al., 2017; Portugal et al., 2019);”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Abdollahi et al., 2017

      Not checked in the free run.

    • cites Portugal et al., 2019 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The level of anxiety and exhaustion among staff in care residence facilities has increased during COVID-19.

    “Furthermore, the level of anxiety and exhaustion among staff in care residence facilities has increased during COVID-19 (Wang et al., 2020).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wang et al., 2020 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Understanding the context of conversations is of utmost importance to create natural, believable interactions between intelligent virtual assistants and humans.

    “To create natural, believable interactions between intelligent virtual assistants and humans, understanding the context of conversations is of utmost importance (Harms et al., 2018).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Harms et al., 2018 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The most effective communication mode in human-robot interaction is largely considered to be speech.

    “the most effective communication mode in human-robot interaction (HRI) is largely considered to be speech (Fong et al., 2003).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Fong et al., 2003 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The user experience questionnaire (UEQ) used in this study matched those deployed for measuring the user experience of interactive products.

    “The UEQ matched those deployed for measuring the user experience of interactive products (Schrepp, 2015; Memeti and Pllana, 2018).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Schrepp, 2015 source

      Not checked in the free run.

    • cites Memeti and Pllana, 2018

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study protocol for the LMIC pilot was reviewed and approved by the Institutional Ethics Committee (IEC) of the Schizophrenia Research Foundation (SCARF) in Chennai, India.

    “This study protocol: “Use of a Hybrid Face Humanoid Robot in Dementia Care: A preliminary study of feasibility and acceptability” was reviewed and approved by the Institutional Ethics Committee (IEC) of the Schizophrenia Research Foundation (SCARF) in Chennai, India.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The user experience study was conducted independently with a different set of users to obtain entirely unbiased user experience feedback.

    “It was conducted independently with a different set of users to obtain entirely unbiased user experience feedback.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    All participants in the LMIC pilot were required to provide informed consent before recruitment.

    “All participants were required to provide informed consent before recruitment.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Repeated sessions were used to acclimatize the participant to the robotic system and develop familiarity, enabling a comprehensive comparison of user behavior and engagement across repeated human-robot sessions.

    “Repeated sessions were used to acclimatize the participant to the robotic system and develop a level of familiarity. Together with the pre- and post-measures, this enabled a comprehensive comparison of user behavior and engagement in repeated human-robot sessions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The use of this mechanically simple robotic platform for remote cognitive engagement may enhance mental health care and mitigate the impact of COVID-19 on people with dementia.

    “Ultimately, these findings suggest the use of this mechanically simple robotic platform for remote cognitive engagement may enhance mental health care and mitigate the impact of COVID-19 on people with dementia.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Recognition may show differences due to age, cultural context, and cognitive state.

    “Recognition may show differences due to age, cultural context, and cognitive state.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    The robotic system may be used as an alternative platform to assist clinicians and support dementia care, which may be especially useful in times when social and medical support of people with dementia is limited, such as during and beyond the COVID-19 pandemic.

    “The robotic system may be used as an alternative platform to assist clinicians and support dementia care, which may be especially useful in times when social and medical support of PwD is limited, such as during and beyond the COVID-19 pandemic.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The additional 3D faceplate was not used in the user experience study because it was not perceived favourably in previous human-robot interaction experiments, suggesting a potential uncanny valley effect.

    “For instance, the additional 3D faceplate was not used in this study as it was not perceived favourably in previous human-robot interaction experiments, suggesting a potential uncanny valley effect, induced when the faceplate was added to the digital face (see Section Emotion Recognition).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Expressions of the hybrid face robot should be adapted accordingly for use in South India to maximize long-term user engagement and compliance.

    “Hence, expressions of the hybrid face robot should be adapted accordingly for use in South India to maximize long-term user engagement and compliance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The experimental approach is applicable to other scenarios where in-person meetings with clinicians are not feasible, necessary or desirable.

    “However, our experimental approach is applicable to other scenarios where in-person meetings with clinicians are not feasible, necessary or desirable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Lack of a synthetically generated robot's voice instead of a recognizable human one may interfere with participants' acceptability of the robot.

    “4) lack of a synthetically generated robot’s voice instead of a recognizable human one, which may interfere with participants’ acceptability of the robot.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    This is a common drawback of existing dialogue systems.

    “This is a common drawback of existing dialogue systems (Fitzpatrick et al., 2017; Harms et al., 2018).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Fitzpatrick et al., 2017 source

      Not checked in the free run.

    • cites Harms et al., 2018 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The robotic affective communication approach is novel in its mechanically simple, low-cost and multimodal design.

    “Our approach for robotic affective communication offers novelty in its mechanically simple, low-cost and multimodal design.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Young participants might be more familiar with robotic faces and digital characters, such as emoticons, than older adults tested in India, which is a potential explanation for the lower recognition rates in India.

    “One potential explanation is the fact that young participants might be more familiar with robotic faces and digital characters, such as emoticons, than older adults tested in India.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The hybrid face robot design is flexible and adaptable, which is a key feature for cultural usefulness in India.

    “This highlights the flexibility and adaptability in design of the hybrid face robot, which we have identified as a key feature for cultural usefulness in India.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    The robotic platform is clinically useful and culturally appropriate technology to deliver cognitive engagement for dementia support in LMICs, particularly in India.

    “We propose it as clinically useful and culturally appropriate technology to deliver cognitive engagement for dementia support in LMICs, particularly in India.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The expressiveness of social robots must be adapted to the culture they will be deployed to following a user-centred and iterative approach to ensure effectiveness, user acceptability and compliance.

    “We argue the expressiveness of social robots must be adapted to the culture they will be deployed to following a user-centred and iterative approach to ensure effectiveness, user acceptability and compliance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    The AI voice system would need to support regional languages for fully autonomous use in India.

    “While our investigation of human-robot multimodal interactions with healthy participants in the United Kingdom yielded promising results, the AI voice system would need to support regional languages for fully autonomous use in India.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Problems with acceptability of the robot are unlikely due to cultural influences but rather due to the effects of cognitive impairment in dementia.

    “As PwD are from the same culture, this allows us to infer that problems with acceptability are unlikely due to the cultural influences but rather due to the effects of cognitive impairment in dementia.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Great efforts are being made in this promising research field to create natural 'human-like' conversations, including the exploration of conversational robots and voice-based systems for supporting cognitive impaired individuals.

    “Nevertheless, great efforts are being made in this promising research field to create natural ‘human-like’ conversations (Harms et al., 2018; Griol et al., 2019) 6 , including the exploration of conversational robots and voice-based systems for supporting cognitive impaired individuals (Cruz-Sandoval et al., 2020; Pou-Prom et al., 2020; Salichs et al., 2020).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Harms et al., 2018 source

      Not checked in the free run.

    • cites Griol et al., 2019 source

      Not checked in the free run.

    • cites Cruz-Sandoval et al., 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    This work may provide useful insights into testing and adjusting a hybrid social robot for cognitive engagement with PwD in the cultural context of India and lay the foundation for future telemedicine deployment.

    “As no similar study has been previously conducted in the literature, this work may provide useful insights into testing and adjusting a hybrid social robot for cognitive engagement with PwD in the cultural context of India and lay the foundation for future telemedicine deployment.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: No similar study has been previously conducted in the literature.

  • May combine two claims.

    Interpretationown data

    Emotion recognition of affective robots and the overall effectiveness of HRI are influenced by culture, age, cognitive ability and familiarity with similar technologies.

    “Although further investigation is needed, these studies suggest that emotion recognition of affective robots and the overall effectiveness of HRI are influenced by culture, age, cognitive ability and familiarity with similar technologies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    This technology may be of special utility, but not limited to dementia support in the COVID-19 era.

    “This technology may be of special utility, but not limited to dementia support in the COVID-19 era.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The robotic platform is an alternative or complementary technological solution to deliver cognitive engagement and enhanced mental health support to older citizens or PwD, during and beyond COVID-19.

    “Hence, we propose it as an alternative or complementary technological solution to deliver cognitive engagement and enhanced mental health support to older citizens or PwD, during and beyond COVID-19.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Even in its current form, the robotic framework provides a more accessible tool to deliver cognitive engagement in LMICs, with potential for positive impact in mental health and dementia care, during and beyond the COVID-19 pandemic.

    “Even in its current form, our robotic framework provides a more accessible tool to deliver cognitive engagement in LMICs, with potential for positive impact in mental health and dementia care, during and beyond the COVID-19 pandemic.”
    Quote in the paperNot a substantial claim
69 of 69

  • The paper relies on this without support.

    No research to date has developed and pilot tested mechanically simple and low-cost robotic solutions to support dementia care in challenging scenarios imposed by the COVID-19 outbreak in LMICs, particularly in India.

    “While SAR have shown great potential to support PwD in clinical and residential environments, no research to date has developed and pilot tested mechanically simple and low-cost robotic solutions to support dementia care in challenging scenarios imposed by the COVID-19 outbreak in LMICs, particularly in India.”

  • The paper relies on this without support.

    User compliance (trust) demands seamless affective human-robot interactions.

    “yet user compliance (i.e., trust) demands seamless affective human-robot interactions”

  • The paper relies on this without support.

    The high cost of complex interactive systems is oftentimes a barrier for deployment, especially in LMICs.

    “The high cost of such complex interactive systems is oftentimes a barrier for deployment, especially in LMICs.”

  • The paper relies on this without support.

    The integration of affective communicative modalities and implementation in social robots can ultimately lead to improved engagement in human-robot interaction in demanding healthcare contexts, such as dementia care.

    “The integration of affective communicative modalities and implementation in social robots can ultimately lead to improved engagement in HRI in demanding healthcare contexts, such as dementia care, during and beyond the COVID-19 pandemic.”

  • The paper relies on this without support.

    Virtual assistants and voice-enabled smart speakers alone have limited flexibility to adapt for mental health care applications, particularly to support people with dementia.

    “Despite benefits in feasibility of implementation and commercial availability, virtual assistants and voice-enabled smart speakers (e.g., Amazon Alexa, Google Home) alone have limited flexibility to adapt for mental health care applications, particularly to support PwD.”

  • The paper relies on this without support.

    The mechatronic complexity in the development of embodied, fully actuated robots with the desired expressive ability is associated with high costs.

    “However, the mechatronic complexity in the development of embodied, fully actuated robots with the desired expressive ability is associated with high costs.”

  • The paper relies on this without support.

    The high cost of mechatronic complexity constitutes a barrier for real-world deployment beyond academic research, especially in LMICs.

    “This in turn constitutes a barrier for real-world deployment beyond academic research, especially in LMICs.”

  • The paper relies on this without support.

    The pilot study described is the first of its nature to explore the feasibility and cultural acceptability of robotic telemedicine for mental health and dementia support in India.

    “The pilot study here described is the first of its nature to explore the feasibility and cultural acceptability of robotic telemedicine for mental health and dementia support in India.”

  • The paper relies on this without support.

    The regional language in Chennai, India (Tamil) is not yet supported by the IVA.

    “as the regional language in Chennai, India (Tamil) is not yet supported by the IVA.”

  • The paper relies on this without support.

    Participants assumed the voice came from the robot when the mobile phone was placed out of sight behind the hybrid face robot.

    “The mobile phone in PwD’s room was placed out of sight behind the hybrid face robot as shown in Figure 7 such that participants assumed the voice came from the robot (Wizard of Oz approach).”

  • The paper relies on this without support.

    Internet connectivity was unreliable for smooth voice over IP during the remote cognitive engagement sessions.

    “The two-way verbal communication between PwD and clinician during cognitive engagement sessions took place via an additional phone call connection due to unreliable internet connectivity for smooth voice over IP.”

  • The paper relies on this without support.

    The robot is assumed to be the one speaking due to synchronization of the robot’s mouth animation and the audio signal.

    “This allowed a better synchronization of the robot’s mouth animation and the audio signal (dB), in such a way that the robot is assumed to be the one speaking.”

  • The paper relies on this without support.

    Social robots hold significant promise to support mental health and may provide end-users with complementary assistance to stimulate interaction, alleviate anxiety and loneliness, in addition to reducing the caregiver burden, which is a critical need during and after the COVID-19 context.

    “Social robots hold significant promise to support mental health and may provide end-users with complementary assistance to stimulate interaction, alleviate anxiety and loneliness, in addition to reducing the caregiver burden, which is a critical need during and after the COVID-19 context.”

  • The paper relies on this without support.

    There is a larger gap of both resources and targeted research in LMICs for socially assistive robots.

    “While user trust, complexity and expense of socially assistive robots is a challenge in any setting, we believe there is a larger gap of both resources and targeted research in LMICs.”

  • The paper relies on this without support.

    Cultural differences influence compliance.

    “Cultural differences—which influence compliance—as well as technical challenges and cost need to be addressed.”

  • The paper relies on this without support.

    No similar study has been previously conducted in the literature.

    “As no similar study has been previously conducted in the literature, this work may provide useful insights into testing and adjusting a hybrid social robot for cognitive engagement with PwD in the cultural context of India and lay the foundation for future telemedicine deployment.”

  • The paper relies on this without support.

    User trust, complexity and expense of socially assistive robots is a challenge in any setting.

    “While user trust, complexity and expense of socially assistive robots is a challenge in any setting, we believe there is a larger gap of both resources and targeted research in LMICs.”
17 of 17

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent progress has been made to integrate virtual and embodied robots with meaningful facial expressions for enhanced engagement in HRI.

    “Recent progress has particularly been made to integrate virtual and embodied robots with meaningful facial expressions for enhanced engagement in HRI [7], [74].”

    From Conversational Affective Social Robots for Ageing and Dementia Support (2022) · cites this paper as [7]

  • AI-powered companion robots can sense emotions in residents through facial expressions and gestures.

    “These robots can sense emotions in residents through facial expressions and gestures [130].”

    From Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond (2024) · cites this paper as [130]

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Inoue, 2012, Effective application of Paro: seal type robots for disabled people in according to ideas of occupational therapists, 1321
  2. Prince, 2013, Policy brief: the global impact of dementia 2013–2050, Alzheimer’s Dis. Int., 28, 728
  3. Shaji, 2010, The dementia India report: prevalence, impact, costs and services for dementia, Alzheimer’s Relat. Disord. Soc. India, 60, 44
  4. Valenti Soler, 2015, Social robots in advanced dementia, Front. Aging Neurosci., 7, 133, 10.3389/fnagi.2015.00133 · DOI

Cited by

Sorted by citations

Related

Authors
Lima, M. R., Wairagkar, M., Natarajan, N., Vaitheswaran, S., & Vaidyanathan, R.
Affiliations
  • UK Dementia Research Institute (GB)
  • Schizophrenia Research Foundation (IN)
Corresponding author
Vaidyanathan, Ravi
ORCID
Venue
Frontiers in Robotics and AI
Publisher
Frontiers Media SA
Published
2021-03-18
Volume
8
Pages
618866-618866
ISSN
2296-9144 (ISSN-L)
DOI
10.3389/frobt.2021.618866
PMC
PMC8014955
OpenAlex
W3138299337
Semantic Scholar
a36c8bff615487d380ede19fac6db4197c415ec5
Cited by
64
References
66
Citations by year
2026: 13 · 2025: 12 · 2024: 14 · 2023: 9 · 2022: 9 · 2021: 7
FWCI
9.55
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Robotics and AI (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Spiral (Imperial College London) (submittedVersion, other-oa)
  • Greater South Information System
Funders
UK Dementia Research Institute
Topics
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
Keywords
social robots, human-robot interaction, multimodal interaction, facial expressions, emotion recognition, affective computing, COVID-19 pandemic mental health, older adults, dementia, low- and middle-income countries, virtual assistants, user engagement, cultural acceptance, psychosocial interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review