Equipping social robots with culturally-sensitive facial expressions of emotion using data-driven methods
Open accessThe DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
Social robots must be able to generate realistic and recognizable facial expressions to engage their human users. Many social robots are equipped with standardized facial expressions of emotion that are widely considered to be universally recognized across all cultures. However, mounting evidence shows that these facial expressions are not universally recognized - for example, they elicit significantly lower recognition accuracy in East Asian cultures than they do in Western cultures. Therefore, without culturally sensitive facial expressions, state-of-the-art social robots are restricted in their ability to engage a culturally diverse range of human users, which in turn limits their global marketability. To develop culturally sensitive facial expressions, novel data-driven methods are used to model the dynamic face movement patterns that convey basic emotions (e.g., happy, sad, anger) in a given culture using cultural perception. Here, we tested whether such dynamic facial expression models, derived in an East Asian culture and transferred to a popular social robot, improved the social signalling generation capabilities of the social robot with East Asian participants. Results showed that, compared to the social robot's existing set of facial `universal' expressions, the culturally-sensitive facial expression models are recognized with generally higher accuracy and judged as more human-like by East Asian participants. We also detail the specific dynamic face movements (Action Units) that are associated with high recognition accuracy and judgments of human-likeness, including those that further boost performance. Our results therefore demonstrate the utility of using data-driven methods that employ human cultural perception to derive culturally-sensitive facial expressions that improve the social face signal generation capabilities of social robots. We anticipate that these methods will continue to inform the design of social robots and broaden their usability and global marketability.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Nonparametric permutation tests for functional neuroimaging: A primer with examples
Nichols, T. E., & Holmes, A. P. (2002) · Human Brain Mapping · doi:10.1002/hbm.1058 · Cited by 6,256
Hall, E. T. (1973) · Leonardo · doi:10.2307/1572461 · Cited by 4,859
A survey of socially interactive robots
Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(02)00372-x · Cited by 3,162
Elements of Information Theory
Nalewajski, R. F. (2011) · Perspectives in Electronic Structure Theory · doi:10.1007/978-3-642-20180-6_8 · Cited by 2,309
Universals and cultural differences in facial expressions of emotion.
Ekman, P. (1972) · Medical Entomology and Zoology · Cited by 2,273
On the universality and cultural specificity of emotion recognition: a meta-analysis
Elfenbein, H. A., & Ambady, N. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.2.203 · Cited by 2,049 · cited 2×
“These consistent cultural differences therefore suggest that the facial expressions widely considered to be universal are instead more representative of Western culture [see also reviews of 18, 34].”
BackgroundAutomatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778
Pan-cultural elements in facial displays of emotion
Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969) · Science · doi:10.1126/science.164.3875.86 · Cited by 1,775
Universals and cultural differences in the judgments of facial expressions of emotion.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.53.4.712 · Cited by 1,763
Russell, J. A. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.115.1.102 · Cited by 1,640 · cited 1×
“Whereas all six facial expressions are recognized with comparably high accuracy in Western cultures, facial expressions such as fear, disgust, and anger elicit significantly lower accuracy in a number of other cultures [17, 18, see also reviews of 19, 20].”
BackgroundCultural variations in emotions: A review.
Mesquita, B., & Frijda, N. H. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.112.2.179 · Cited by 1,084
Facial expressions of emotion are not culturally universal
Jack, R. E., Garrod, O. G. B., Yu, H., Caldara, R., & Schyns, P. G. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1200155109 · Cited by 940 · cited 7×
“2 in [17]).”
BackgroundMethodInfluentialSelf-citationCompound facial expressions of emotion
Du, S., Tao, Y., & Martinez, A. M. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1322355111 · Cited by 806
A new pan-cultural facial expression of emotion
Ekman, P., & Friesen, W. V. (1986) · Motivation and Emotion · doi:10.1007/bf00992253 · Cited by 681
The Expression Of The Emotions In Man And Animals
Darwin, C. (1998) · The Expression Of The Emotions In Man And Animals · doi:10.1093/oso/9780195112719.002.0002 · Cited by 650
Cultural Confusions Show that Facial Expressions Are Not Universal
Jack, R. E., Blais, C., Scheepers, C., Schyns, P. G., & Caldara, R. (2009) · Current Biology · doi:10.1016/j.cub.2009.07.051 · Cited by 602
Self-citationBiehl, M., Matsumoto, D., Ekman, P., Hearn, V., Heider, K., Kudoh, T., & Ton, V. (1997) · Journal of Nonverbal Behavior · doi:10.1023/a:1024902500935 · Cited by 494
American-Japanese cultural differences in intensity ratings of facial expressions of emotion
Matsumoto, D., & Ekman, P. (1989) · Motivation and Emotion · doi:10.1007/bf00992959 · Cited by 425
Facial dynamics as indicators of trustworthiness and cooperative behavior.
Krumhuber, E., Manstead, A. S. R., Cosker, D., Marshall, D., Rosin, P. L., & Kappas, A. (2007) · Emotion · doi:10.1037/1528-3542.7.4.730 · Cited by 407 · cited 1×
“A further limitation of these standardized facial expressions of emotion is that they are static and therefore do not represent the naturalistic dynamics of human facial expressions [39].”
BackgroundUniversals and Cultural Differences in Recognizing Emotions
Elfenbein, H. A., & Ambady, N. (2003) · Current Directions in Psychological Science · doi:10.1111/1467-8721.01252 · Cited by 333
Furhat: A Back-Projected Human-Like Robot Head for Multiparty Human-Machine Interaction
Al Moubayed, S., Beskow, J., Skantze, G., & Granström, B. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-34584-5_9 · Cited by 297
Self-citationAmerican-Japanese Cultural Differences in the Recognition of Universal Facial Expressions
Matsumoto, D. (1992) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022192231005 · Cited by 289
The expression of the emotions in man and animals, 3rd ed.
Darwin, C. D., & Ekman, P. (1872) · Cited by 226
Functional Smiles: Tools for Love, Sympathy, and War
Rychlowska, M., Jack, R. E., Garrod, O. G. B., Schyns, P. G., Martin, J. D., & Niedenthal, P. M. (2017) · Psychological Science · doi:10.1177/0956797617706082 · Cited by 223
Self-citationFour not six: Revealing culturally common facial expressions of emotion.
Jack, R. E., Sun, W., Delis, I., Garrod, O. G. B., & Schyns, P. G. (2016) · Journal of Experimental Psychology: General · doi:10.1037/xge0000162 · Cited by 212
Self-citationNelson, N. L., & Russell, J. A. (2013) · Emotion Review · doi:10.1177/1754073912457227 · Cited by 209
Development of the Face Robot SAYA for Rich Facial Expressions
Hashimoto, T., Hitramatsu, S., Tsuji, T., & Kobayashi, H. (2006) · 2006 SICE-ICASE International Joint Conference · doi:10.1109/sice.2006.315537 · Cited by 168
Perception-driven facial expression synthesis
Yu, H. L., Garrod, O., & Schyns, P. G. (2011) · Computers & Graphics · doi:10.1016/j.cag.2011.12.002 · Cited by 145
Self-citationThe fear gasping face as a threat display in a Melanesian society
Crivelli, C., Russell, J. A., Jarillo, S., & Fernández-Dols, J.-M. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611622113 · Cited by 144
Recognition of Facial Expression in Three Cultures
Boucher, J. D., & Carlson, G. E. (1980) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022180113003 · Cited by 131
Distinct facial expressions represent pain and pleasure across cultures
Chen, C., Crivelli, C., Garrod, O., Schyns, P. G., Dols, J. M. F., & Jack, R. E. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1807862115 · Cited by 128
Self-citationEmotion perception from a componential perspective
Shuman, V., Clark‐Polner, E., Meuleman, B., Sander, D., & Scherer, K. R. (2015) · Cognition & Emotion · doi:10.1080/02699931.2015.1075964 · Cited by 117
Accuracy and Latency of Judgment of Facial Expressions of Emotions
Kirouac, G., & Doré, F. Y. (1983) · Perceptual and Motor Skills · doi:10.2466/pms.1983.57.3.683 · Cited by 116
Kirouac, G., & Doré, F. Y. (1985) · Journal of Nonverbal Behavior · doi:10.1007/bf00987555 · Cited by 112
I show you how I like you - can you read it in my face? [robotics]
Canamero, L., & Fredslund, J. (2001) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.952719 · Cited by 107
Data-Driven Methods to Diversify Knowledge of Human Psychology
Jack, R. E., Crivelli, C., & Wheatley, T. P. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.10.002 · Cited by 96
Self-citationA Cross-Cultural Study of Recognition Thresholds for Facial Expressions of Emotion
McAndrew, F. T. (1986) · Journal of Cross-Cultural Psychology · doi:10.1177/0022002186017002005 · Cited by 84
Culture and facial expressions of emotion
Jack, R. E. (2013) · Visual Cognition · doi:10.1080/13506285.2013.835367 · Cited by 79
Self-citationDeriving Minimal Features for Human-Like Facial Expressions in Robotic Faces
Bennett, C. C., & Šabanović, S. (2014) · International Journal of Social Robotics · doi:10.1007/s12369-014-0237-z · Cited by 78
Facial Movements Strategically Camouflage Involuntary Social Signals of Face Morphology
Gill, D., Garrod, O., Jack, R. E., & Schyns, P. G. (2014) · Psychological Science · doi:10.1177/0956797614522274 · Cited by 73
Self-citationShimoda, K., Argyle, M., & Bitti, P. R. (1978) · European Journal of Social Psychology · doi:10.1002/ejsp.2420080203 · Cited by 65
Misinterpretation of facial expression: A cross‐cultural study
Shioiri, T., Someya, T., Helmeste, D., & Tang, S. W. (1999) · Psychiatry and Clinical Neurosciences · doi:10.1046/j.1440-1819.1999.00469.x · Cited by 62
Hemifacial differences in the in‐group advantage in emotion recognition
Anger Elfenbein, H., Mandal, M., Ambady, N., Harizuka, S., & Kumar, S. (2004) · Cognition & Emotion · doi:10.1080/02699930341000257 · Cited by 57
Ducci, L., Arcuri, L., Georgis, T. W., & Sineshaw, T. (1982) · Journal of Cross-Cultural Psychology · doi:10.1177/0022002182013003005 · Cited by 49
Differential judgement of static facial expressions of emotions in three cultures
Huang, Y., Tang, S., Helmeste, D., Shioiri, T., & Someya, T. (2001) · Psychiatry and Clinical Neurosciences · doi:10.1046/j.1440-1819.2001.00893.x · Cited by 44
Can a Humanoid Face be Expressive? A Psychophysiological Investigation
Lazzeri, N., Mazzei, D., Greco, A., Rotesi, A., Lanatà, A., & Rossi, D. E. D. (2015) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2015.00064 · Cited by 43
Reverse Engineering Psychologically Valid Facial Expressions of Emotion into Social Robots
Chen, C., Garrod, O., Zhan, J., Beskow, J., Schyns, P. G., & Jack, R. E. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00072 · Cited by 36
Self-citationTrovato, G., Kishi, T., Endo, N., Hashimoto, K., & Takanishi, A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34103-8_4 · Cited by 25
The Look of Fear from the Eyes Varies with the Dynamic Sequence of Facial Actions
Krumhuber, E. G., & Scherer, K. R. (2016) · Swiss Journal of Psychology · doi:10.1024/1421-0185/a000166 · Cited by 19
Petričević, N., Čelebić, A., Ibrahimagić-Šeper, L., & Selimović, E. (2006) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 13
The Face is the Mirror of the Cultural Mind
Chen, C., Garrod, O., Schyns, P. G., & Jack, R. E. (2015) · Journal of Vision · doi:10.1167/15.12.928 · Cited by 11
Self-citation
Research works printed but not linked: 2
- 10.1037/0022-3514.53.4.712 · DOI · Paper page
- ekman, 1978, Manual for the Facial Action Coding System
Cited by
Conversational Affective Social Robots for Ageing and Dementia Support
Lima, M. R., Wairagkar, M., Gupta, M., Rodriguez y Baena, F., Barnaghi, P., Sharp, D. J., & Vaidyanathan, R. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3115228 · Cited by 75
Zhan, J., Liu, M., Garrod, O., Daube, C., Ince, R. A. A., Jack, R. E., & Schyns, P. G. (2021) · Current Biology · doi:10.1016/j.cub.2021.03.013 · Cited by 56
Self-citationHuman-robot facial coexpression
Hu, Y., Chen, B., Lin, J., Wang, Y., Wang, Y., Mehlman, C., & Lipson, H. (2024) · Science Robotics · doi:10.1126/scirobotics.adi4724 · Cited by 73
Liu, M., Duan, Y., Ince, R. A. A., Chen, C., Garrod, O., Schyns, P. G., & Jack, R. E. (2021) · Current Biology · doi:10.1016/j.cub.2021.10.035 · Cited by 48
Self-citationCultural facial expressions dynamically convey emotion category and intensity information
Chen, C., Messinger, D. S., Cheng, C., Yan, H., Duan, Y., Ince, R. A. A., Garrod, O., Schyns, P. G., & Jack, R. E. (2023) · Current Biology · doi:10.1016/j.cub.2023.12.001 · Cited by 37
Self-citation
Metadata
- Authors
- Chen, C., Hensel, L. B., Duan, Y., Ince, R. A. A., Garrod, O. G. B., Beskow, J., Jack, R. E., & Schyns, P. G.
- Affiliations
- University of Glasgow (GB)
- ORCID
- Venue
- 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 1-8
- ISSN
- 1541-5058 (ISSN-L)
- OpenAlex
- W2922943851
- Cited by
- 41
- References
- 52
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 5 · 2023: 6 · 2022: 3 · 2021: 12 · 2020: 8 · 2019: 1
- FWCI
- 9.90
- Type
- conference paper
- Language
- en
- Open access
- green
- eprints.gla.ac.uk
- Licence
- Locations
- International Conference on Automatic Face and Gesture Recognition (publishedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion, cc-by-sa)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/N019261/1)
- Economic and Social Research Council (1944569)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Keywords
- social robots, emotional facial expressions, cultural perception, East Asian culture, facial action units, universal facial expressions, human-likeness, social signaling, robot expressiveness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.