Cited by 24Social Robot Interaction and HRIEmotion and Mood RecognitionFace Recognition and Perception

Emotive response to a hybrid-face robot and translation to consumer social robots

Open access
Wairagkar, Maitreyee, Lima, Maria R., Bazo, Daniel, Craig, Richard, Weissbart, Hugo, Etoundi, Appolinaire C., Reichenbach, Tobias, Iyengar, Prashant +5 more·2022·IEEE Internet of Things Journal

Abstract

We present the conceptual formulation, design, fabrication, control and commercial translation of an IoT enabled social robot as mapped through validation of human emotional response to its affective interactions. The robot design centres on a humanoid hybrid-face that integrates a rigid faceplate with a digital display to simplify conveyance of complex facial movements while providing the impression of three-dimensional depth. We map the emotions of the robot to specific facial feature parameters, characterise recognisability of archetypical facial expressions, and introduce pupil dilation as an additional degree of freedom for emotion conveyance. Human interaction experiments demonstrate the ability to effectively convey emotion from the hybrid-robot face to humans. Conveyance is quantified by studying neurophysiological electroencephalography (EEG) response to perceived emotional information as well as through qualitative interviews. Results demonstrate core hybrid-face robotic expressions can be discriminated by humans (80%+ recognition) and invoke face-sensitive neurophysiological event-related potentials such as N170 and Vertex Positive Potentials in EEG. The hybrid-face robot concept has been modified, implemented, and released by Emotix Inc in the commercial IoT robotic platform Miko (‘My Companion’), an affective robot currently in use for human-robot interaction with children. We demonstrate that human EEG responses to Miko emotions are comparative to that of the hybrid-face robot validating design modifications implemented for large scale distribution. Finally, interviews show above 90% expression recognition rates in our commercial robot. We conclude that simplified hybrid-face abstraction conveys emotions effectively and enhances human-robot interaction.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/jiot.2021.3097592

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Fully teleoperated robots lacking autonomy may face additional challenges for fully autonomous, effective, two-way conversations between social robots and target users, especially with cognitive impairment or dementia.

    “This may suggest additional challenges when it comes to fully autonomous, effective, two-way conversations between social robots and target users, especially with cognitive impairment or dementia.”

    From Conversational Affective Social Robots for Ageing and Dementia Support (2022) · cites this paper as [63]

    The Introduction (p. 1) says smooth human–robot interaction remains challenging, and the IoT discussion (p. 11) identifies support for dementia patients as future work. The paper does not specifically say that fully autonomous, effective two-way conversations with people with cognitive impairment or dementia pose additional challenges; its reference to a teleoperated robot is in a cited study’s title (References, p. 13). (LLM, conf=0.82)

    Experimental — accuracy not yet measured
  • Recent progress has been made to integrate virtual and embodied robots with meaningful facial expressions for enhanced engagement in HRI.

    “Recent progress has particularly been made to integrate virtual and embodied robots with meaningful facial expressions for enhanced engagement in HRI [7], [74].”
  • To achieve effective speech-based assistance with people with dementia, it is important to mimic caregiver communication techniques such as slower rate of speech and sentences with reduced syntactic complexity.

    “To achieve effective speech-based assistance with PwD, the authors highlight the importance of mimicking some of the communication techniques employed by the caregiver, such as slower rate of speech and sentences with reduced syntactic complexity.”

    From Conversational Affective Social Robots for Ageing and Dementia Support (2022) · cites this paper as [63]

    The discussion of IoT impact (p. 11) mentions speech delivered through a cloud interface and future work supporting people with dementia, but gives no guidance on how to speak to them. Searches for caregiver communication, slower speech, and reduced syntactic complexity found no passage addressing those techniques. (LLM, conf=0.76)

    Experimental — accuracy not yet measured
  • Speech recognition often fails in human-robot dialogs with older adults and people with dementia, making conversation a difficult research task.

    “This study revealed that speech recognition remains a major open challenge for HRI with PwD.”

    From Conversational Affective Social Robots for Ageing and Dementia Support (2022) · cites this paper as [63]

    The Introduction (p. 2) mentions speech as a capability social robots need, but does not discuss speech-recognition failures. The study describes emotion-recognition tests with healthy participants (Participants, pp. 4 and 7), not conversational HRI with people with dementia. Searches for speech recognition and dementia-related dialogue found no passage establishing the claimed challenge. (LLM, conf=0.76)

    Experimental — accuracy not yet measured
  • Conversation repair features are needed to overcome user confusion, lack of interest, and conversation breakdowns in human-robot interaction with people with dementia.

    “Conversation repair features are needed to overcome user confusion, lack of interest, and conversation breakdowns.”

    From Conversational Affective Social Robots for Ageing and Dementia Support (2022) · cites this paper as [63]

    The Introduction (pp. 1–2) discusses adaptive, emotionally responsive social robots, but does not address conversation repair, user confusion, lack of interest, or breakdowns. The discussion of Miko I (p. 11) says its conversational ability was not investigated; the paper mentions support for dementia patients only as future work. Searches for conversation repair and dementia-related breakdowns found no supporting passage. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Research works printed but not linked: 65

  1. [1][1] C. L. Breazeal, Designing Sociable Robots (Intelligent Robots and Autonomous Agents). Cambridge, MA, USA: MIT Press, 2002, p. 263. -
  2. [2][2] K. Dautenhahn, 'The art of designing socially intelligent agents: Science, fiction, and the human in the loop,' Appl. Artif. Intell. , vol. 12, nos. 7-8, pp. 573-617, 1998. -
  3. [3][3] A. G. Pipe et al. , 'Affective robotics: Human motion and behavioural inspiration for safe cooperation between humans and humanoid assistive robots,' in Biomimetics: Nature-Based Innovation , Y. Bar-Cohen Ed. Boca Raton, FL, USA: CRC Press, 2012. -
  4. [4][4] C. Breazeal and B. Scassellati, 'Robots that imitate humans,' Trends Cogn. Sci. , vol. 6, no. 11, pp. 481-487, 2002. -
  5. [6][6] K. Kuwamura, S. Nishio, and S. Sato, 'Can we talk through a robot as if face-to-face? Long-term fieldwork using teleoperated robot for seniors with Alzheimer's disease,' Front. Psychol , vol. 7, p. 1066, Jul. 2016, doi: 10.3389/fpsyg.2016.01066. - · DOI
  6. [8][8] O. Nocentini, L. Fiorini, G. Acerbi, A. Sorrentino, G. Mancioppi, and F. Cavallo, 'A survey of behavioral models for social robots,' Robotics , vol. 8, no. 3, p. 54, 2019. -
  7. [9][9] M. Chita-Tegmark and M. Scheutz, 'Assistive robots for the social management of health: A framework for robot design and human-robot interaction research,' Int. J. Social Robot. , vol. 13, no. 8, pp. 197-217, 2021, doi: 10.1007/s12369-020-00634-z. - · DOI
  8. [10][10] P. Caleb-Solly, S. Dogramadzi, C. A. Huijnen, and H. V. D. Heuvel, 'Exploiting ability for human adaptation to facilitate improved humanrobot interaction and acceptance,' Inf. Soc. , vol. 34, no. 3, pp. 153-165, 2018. -
  9. [11][11] L. Bishop, A. Van Maris, S. Dogramadzi, and N. Zook, 'Social robots: The influence of human and robot characteristics on acceptance,' Paladyn J. Behav. Robot. , vol. 10, no. 1, pp. 346-358, 2019. -
  10. [12][12] C. Breazeal, 'Emotion and sociable humanoid robots,' Int. J. HumanComput. Stud. , vol. 59, nos. 1-2, pp. 119-155, 2003. -
  11. [13][13] H. Ishiguro, 'Understanding humans by building androids,' in Proc. 11th Annu. Meeting Spec. Interest Group Discourse Dialogue , 2010, p. 175. -
  12. [14][14] H. Ishiguro, 'Android science: Towards a new cross-interdisciplinary framework,' in Proc. CogSci Workshop Towards Social Mechanisms Android Sci., , 2005, pp. 1-6. -
  13. [16][16] A. M. Rosenthal-von der Pütten, N. Kramer, and J. Herrmann, 'The effects of humanlike and robot-specific affective nonverbal,' Int. J. Social Robot. , vol. 10, pp. 569-582, Jan. 2018, doi: 10.1007/s12369-018-0466-7. - · DOI
  14. [18][18] A. Lambert, N. Norouzi, G. Bruder, and G. Welch, 'A systematic review of ten years of research on human interaction with social robots,' Int. J. Human Comput. Interact. , vol. 36, no. 19, pp. 1804-1817, 2020. -
  15. [19][19] P. Simoens et al. , 'Internet of Robotic Things: Context-aware and personalized interventions of assistive social robots (short paper),' in Proc. 5th IEEE Int. Conf. Cloud Netw. (Cloudnet) , 2016, pp. 204-207. -
  16. [20][20] C. Turcu and C. Turcu, 'The social Internet of Things and the RFIDbased robots,' in Proc. 4th Int. Congr. Ultra Modern Telecommun. Control Syst. , 2012, pp. 77-83. -
  17. [21][21] A. Vulpe et al. , 'IoT security approaches in social robots for ambient assisted living scenarios,' in Proc. 22nd Int. Symp. Wireless Pers. Multimedia Commun. (WPMC) , 2019, pp. 1-6. -
  18. [22][22] Y.-W. Cheng, Y. Wang, Y.-F. Yang, Z.-K. Yang, and N.-S. Chen, 'Designing an authoring system of robots and IoT-based toys for EFL teaching and learning,' Comput. Assist. Lang. Learn. , vol. 34, nos. 1-2, pp. 6-34, 2020. -
  19. [23][23] A. Amanatiadis, V. G. Kaburlasos, C. Dardani, and S. A. Chatzichristofis, 'Interactive social robots in special education,' in Proc. IEEE 7th Int. Conf. Consum. Electron. (ICCE) , 2017, pp. 126-129. -
  20. [25][25] K. Hornfeck, Y. Zhang, and K. Lee, 'Philos: A sociable robot for human robot interactions and wireless health monitoring,' in Proc. 27th Annu. ACM Symp. Appl. Comput. , 2012, pp. 293-294. -
  21. [26][26] P. Ekman, 'Universals and cultural differences in facial expressions of emotion,' in Nebraska Symposium on Motivation , J. Cole, Ed. Lincoln, NE, USA: Univ. Nebraska Press, pp. 207-283, 1972. -
  22. [27][27] P. Ekman and W. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement , Consulting Psychol. Press, Palo Alto, CA, USA, 1978. -
  23. [28][28] M. Mori, K. F. MacDorman, and N. Kageki, 'The uncanny valley [from the field],' IEEE Robot. Autom. Mag. , vol. 19, no. 2, pp. 98-100, Jun. 2012. -
  24. [29][29] I.-H. Kuo, C. Jayawardena, E. Broadbent, and B. A. MacDonald, 'Multidisciplinary design approach for implementation of interactive services,' Int. J. Social Robot. , vol. 3, no. 4, pp. 443-456, 2011. -
  25. [31][31] B. Johansson, T. A. Tjøstheim, and C. Balkenius, 'EPI: An open humanoid platform for developmental robotics,' Int. J. Adv. Robot. Syst. , vol. 17, no. 2, 2020, Art. no. 1729881420911498. -
  26. [32][32] C. Breazeal and J. Velasquez, 'Robot in society: Friend or appliance,' in Proc. Auton. Agents Workshop Emotion-Based Agent Archit. , 1999, pp. 18-26. -
  27. [33][33] R. Arkin and L. Moshkina, 'Affect in human-robot interaction,' in The Oxford Handbook of Affective Computing , S. D. M. R. Calvo, J. Gratch, A. Kappas, Eds. New York, NY, USA: Oxford Univ. Press, 2015, pp. 483-493. -
  28. [34][34] I. M. Menne and F. Schwab, 'Faces of emotion: Investigating emotional facial expressions towards,' Int. J. Social Robots , vol. 10, no. 2, pp. 199209, 2008, doi: 10.1007/s12369-017-0447-2. - · DOI
  29. [35][35] Y. Wang and S. Quadflieg, 'In our own image? Emotional and neural processing differences,' Social Cogn. Affect. Neurosci. , vol. 10, no. 11, pp. 1515-1524, 2015. -
  30. [36][36] R. Adolphs, 'Recognizing emotion from facial expressions: Psychological and neurological mechanisms,' Behav. Cogn. Neurosci. Rev. , vol. 1, no. 1, pp. 21-62, 2002. -
  31. [37][37] M. C. Pastor, M. M. Bradley, A. Löw, F. Versace, J. Moltó, and P. J. Lang, 'Affective picture perception: Emotion, context, and the late positive potential,' Brain Res. , vol. 1189, pp. 145-151, Jan. 2008. -
  32. [38][38] M. L. Kesler et al. , 'Neural substrates of facial emotion processing using fMRI,' Cogn. Brain Res. , vol. 11, no. 2, pp. 213-226, 2001. -
  33. [39][39] M. Eimer, A. Holmes, and F. P. McGlone, 'The role of spatial attention in the processing of facial expression: An ERP study of rapid brain responses to six basic emotions,' Cogn. Affect. Behav. Neurosci. , vol. 3, no. 2, pp. 97-110, 2003. -
  34. [40][40] G. Pfurtscheller and F. L. Da Silva, 'Event-related EEG/MEG synchronization and desynchronization: Basic principles,' Clin. Neurophysiol. , vol. 110, no. 11, pp. 1842-1857, 1999. -
  35. [41][41] S. Bentin, T. Allison, A. Puce, E. Perez, and G. McCarthy, 'Electrophysiological studies of face perception in humans,' J. Cogn. Neurosci. , vol. 8, no. 6, pp. 551-565, 1996. -
  36. [42][42] B. Sadeh, A. Zhdanov, I. Podlipsky, T. Hendler, and G. Yovel, 'The validity of the face-selective ERP N170 component during simultaneous recording with functional MRI,' Neuroimage , vol. 42, no. 2, pp. 778-786, 2008. -
  37. [43][43] D. Jeffreys, 'A face-responsive potential recorded from the human scalp,' Exp. Brain Res. , vol. 78, no. 1, pp. 193-202, 1989. -
  38. [44][44] D. Jeffreys and E. Tukmachi, 'The vertex-positive scalp potential evoked by faces and by objects,' Exp. Brain Res. , vol. 91, no. 2, pp. 340-350, 1992. -
  39. [45][45] C. Joyce and B. Rossion, 'The face-sensitive N170 and VPP components manifest the same brain processes: The effect of reference electrode site,' Clin. Neurophysiol. , vol. 116, no. 11, pp. 2613-2631, 2005. -
  40. [46][46] F. Haist, K. Lee, and J. Stiles, 'Individuating faces and common objects produces equal responses in putative face-processing areas in the ventral occipitotemporal cortex,' Front. Human Neurosci. , vol. 4, p. 181, Oct. 2010. -
  41. [47][47] V. C. Blau, U. Maurer, N. Tottenham, and B. D. McCandliss, 'The facespecific N170 component is modulated by emotional facial expression,' Behav. Brain Funct. , vol. 3, no. 1, pp. 1-13, 2007. -
  42. [48][48] A. Krombholz, F. Schaefer, and W. Boucsein, 'Modification of N170 by different emotional expression of schematic faces,' Biol. Psychol. , vol. 76, no. 3, pp. 156-162, 2007. -
  43. [49][49] R. Craig, R. Vaidyanathan, C. James, and C. Melhuish, 'Assessment of human response to robot facial expressions through visual evoked potentials,' in Proc. 10th IEEE-RAS Int. Conf. Humanoid Robots , Dec. 2010, pp. 647-652, doi: 10.1109/ICHR.2010.5686272. - · DOI
  44. [50][50] L. Boer and H. Bewley, 'Reconfiguring the appearance and expression of social robots by acknowledging their otherness,' in Proc. Conf. Designing Interact. Syst. , 2018, pp. 667-677. -
  45. [51][51] D. Bazo, R. Vaidyanathan, A. Lentz, and C. Melhuish, 'Design and testing of a hybrid expressive face for a humanoid robot,' in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. , Oct. 2010, pp. 5317-5322, doi: 10.1109/IROS.2010.5651469. - · DOI
  46. [52][52] M. A. Al-Taee, W. Al-Nuaimy, Z. J. Muhsin, and A. Al-Ataby, 'Robot assistant in management of diabetes in children based on the Internet of Things,' IEEE Internet Things J. , vol. 4, no. 2, pp. 437-445, Apr. 2017. -
  47. [53][53] M. Lindberg, H. Sandberg, M. Liljenberg, M. Eriksson, B. Johansson, and C. Balkenius, 'The expression of mental states in a humanoid robot,' in Proc. Int. Conf. Intell. Virtual Agents , 2017, pp. 247-250. -
  48. [54][54] C. Breazeal, 'Affective interaction between humans and robots,' in Proc. Eur. Conf. Artif. Life , 2001, pp. 582-591. -
  49. [55][55] P. Fitzpatrick, G. Metta, and L. Natale, 'Towards long-lived robot genes,' Robot. Auton. Syst. , vol. 56, no. 1, pp. 29-45, 2008. -
  50. [56][56] A. Bruce, I. Nourbakhsh, and R. Simmons, 'The role of expressiveness and attention in human-robot interaction,' in Proc. IEEE Int. Conf. Robot. Autom. , vol. 4, 2002, pp. 4138-4142. -
  51. [57][57] D. J. Schiano, S. M. Ehrlich, K. Rahardja, and K. Sheridan, 'Face to interface: Facial affect in (hu) man and machine,' in Proc. SIGCHI Conf. Human Factors Comput. Syst. , 2000, pp. 193-200. -
  52. [58][58] J. A. Russell and J. M. F. Dols, The Psychology of Facial Expression . Cambridge, U.K.: Cambridge Univ. Press, 1997. -
  53. [59][59] A. Kleinsmith, P. R. De Silva, and N. Bianchi-Berthouze, 'Grounding affective dimensions into posture features,' in Proc. Int. Conf. Affect. Comput. Intell. Interact. , 2005, pp. 263-270. -
  54. [60][60] L. Cañamero and J. Fredslund, 'I show you how I like you-Can you read it in my face? [robotics],' IEEE Trans. Syst., Man, Cybern. A, Syst., Humans , vol. 31, no. 5, pp. 454-459, Sep. 2001. -
  55. [61][61] C. Bartneck, J. Reichenbach, and A. Van Breemen, 'In your face, robot! The influence of a character's embodiment on how users perceive its emotional expressions,' in Proc. Design Emotion , 2004, pp. 1-19. -
  56. [62][62] T. Partala and V. Surakka, 'Pupil size variation as an indication of affective processing,' Int. J. Human-Comput. Stud. , vol. 59, nos. 1-2, pp. 185-198, 2003. -
  57. [63][63] E. H. Hess and J. M. Polt, 'Pupil size in relation to mental activity during simple problem-solving,' Science , vol. 143, no. 3611, pp. 1190-1192, 1964. -
  58. [65][65] M. J. Lyons, S. Akamatsu, M. Kamachi, J. Gyoba, and J. Budynek, 'The Japanese female facial expression (JAFFE) database,' in Proc. 3rd Int. Conf. Autom. Face Gesture Recognit. , 1998, pp. 14-16. -
  59. [66][66] H.-C. Lo and R. Chung, 'Facial expression recognition approach for performance animation,' in Proc. 2nd Int. Workshop Digit. Comput. Video , 2001, pp. 132-139. -
  60. [67][67] J. Seyama and R. S. Nagayama, 'Probing the uncanny valley with the eye size aftereffect,' Presence Teleoper. Virtual Environ. , vol. 18, no. 5, pp. 321-339, 2009. -
  61. [68][68] A. Delorme and S. Makeig, 'EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis,' J. Neurosci. Methods , vol. 134, no. 1, pp. 9-21, 2004. -
  62. [71][71] S. You and L. Robert, 'Emotional attachment, performance, and viability in teams collaborating with embodied physical action (EPA) robots,' J. Assoc. Inf. Syst. , vol. 19, no. 5, pp. 377-407, 2017. -
  63. [72][72] M. Kwon, M. F. Jung, and R. A. Knepper, 'Human expectations of social robots,' in Proc. 11th ACM/IEEE Int. Conf. Human-Robot Interact. (HRI) , 2016, pp. 463-464. -
  64. [73][73] R. J. Itier and M. J. Taylor, 'N170 or N1? Spatiotemporal differences between object and face processing using ERPs,' Cerebr. Cortex , vol. 14, no. 2, pp. 132-142, 2004, doi: 10.1093/cercor/bhg111. - · DOI
  65. [74][74] T. Chaminade and G. Cheng, 'Social cognitive neuroscience and humanoid robotics,' J. Physiol. Paris , vol. 103, nos. 3-5, pp. 286-295, 2009. -

Cited by

Sorted by citations

Authors
Wairagkar, M., Lima, M. R., Bazo, D., Craig, R., Weissbart, H., Etoundi, A. C., Reichenbach, T., Iyengar, P., Vaswani, S., James, C., Barnaghi, P., Melhuish, C., & Vaidyanathan, R.
Affiliations
  • UK Dementia Research Institute (GB)
  • University of California, Santa Barbara (US)
  • Radboud University Nijmegen (NL)
  • University of the West of England (GB)
  • Friedrich-Alexander-Universität Erlangen-Nürnberg (DE)
  • University of Warwick (GB)
Venue
IEEE Internet of Things Journal
Published
2022-03-01
Volume
9
Issue
5
Pages
3174-3188
ISSN
  • 2327-4662 (ISSN-L)
  • 2372-2541
OpenAlex
W3111793797
Cited by
24
References
6
Citations by year
2026: 6 · 2025: 5 · 2024: 5 · 2023: 6 · 2022: 1 · 2021: 1
FWCI
3.19
Type
journal article
Language
en
Open access
Locations
  • IEEE Internet of Things Journal (publishedVersion, cc-by)
  • Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Spiral (Imperial College London) (submittedVersion)
  • UWE Research Repository (UWE Bristol) (submittedVersion, cc-by)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • RN Chidakashi Technologies Pvt Ltd
  • European Commission (CHRIS FP7-215805)
  • U.K. Dementia Research Institute Care Research and Technology Centre (EP/F01869X/1)
  • U.K. Research and Innovation Global Challenges Research Fund
Topics
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
Keywords
social robots, facial expressions, pupil dilation, human-robot interaction, affective robots, N170, EEG responses

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review