Equipping social robots with culturally-sensitive facial expressions of emotion using data-driven methods
Open accessThe DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
Social robots must be able to generate realistic and recognizable facial expressions to engage their human users. Many social robots are equipped with standardized facial expressions of emotion that are widely considered to be universally recognized across all cultures. However, mounting evidence shows that these facial expressions are not universally recognized - for example, they elicit significantly lower recognition accuracy in East Asian cultures than they do in Western cultures. Therefore, without culturally sensitive facial expressions, state-of-the-art social robots are restricted in their ability to engage a culturally diverse range of human users, which in turn limits their global marketability. To develop culturally sensitive facial expressions, novel data-driven methods are used to model the dynamic face movement patterns that convey basic emotions (e.g., happy, sad, anger) in a given culture using cultural perception. Here, we tested whether such dynamic facial expression models, derived in an East Asian culture and transferred to a popular social robot, improved the social signalling generation capabilities of the social robot with East Asian participants. Results showed that, compared to the social robot's existing set of facial `universal' expressions, the culturally-sensitive facial expression models are recognized with generally higher accuracy and judged as more human-like by East Asian participants. We also detail the specific dynamic face movements (Action Units) that are associated with high recognition accuracy and judgments of human-likeness, including those that further boost performance. Our results therefore demonstrate the utility of using data-driven methods that employ human cultural perception to derive culturally-sensitive facial expressions that improve the social face signal generation capabilities of social robots. We anticipate that these methods will continue to inform the design of social robots and broaden their usability and global marketability.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Nonparametric permutation tests for functional neuroimaging: A primer with examples
Nichols, T. E., & Holmes, A. P. (2002) · Human Brain Mapping · doi:10.1002/hbm.1058 · Cited by 6,256
Hall, E. T. (1973) · Leonardo · doi:10.2307/1572461 · Cited by 4,859
A survey of socially interactive robots
Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(02)00372-x · Cited by 3,162
Elements of Information Theory
Nalewajski, R. F. (2011) · Perspectives in Electronic Structure Theory · doi:10.1007/978-3-642-20180-6_8 · Cited by 2,309
Universals and cultural differences in facial expressions of emotion.
Ekman, P. (1972) · Medical Entomology and Zoology · Cited by 2,273
On the universality and cultural specificity of emotion recognition: a meta-analysis
Elfenbein, H. A., & Ambady, N. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.2.203 · Cited by 2,049 · cited 2×
“These consistent cultural differences therefore suggest that the facial expressions widely considered to be universal are instead more representative of Western culture [see also reviews of 18, 34].”
BackgroundAutomatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778
Pan-cultural elements in facial displays of emotion
Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969) · Science · doi:10.1126/science.164.3875.86 · Cited by 1,775
Universals and cultural differences in the judgments of facial expressions of emotion.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.53.4.712 · Cited by 1,763
Research works printed but not linked: 2
- 10.1037/0022-3514.53.4.712 · DOI · Paper page
- ekman, 1978, Manual for the Facial Action Coding System
Cited by
Conversational Affective Social Robots for Ageing and Dementia Support
Lima, M. R., Wairagkar, M., Gupta, M., Rodriguez y Baena, F., Barnaghi, P., Sharp, D. J., & Vaidyanathan, R. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3115228 · Cited by 75
Zhan, J., Liu, M., Garrod, O., Daube, C., Ince, R. A. A., Jack, R. E., & Schyns, P. G. (2021) · Current Biology · doi:10.1016/j.cub.2021.03.013 · Cited by 56
Self-citationHuman-robot facial coexpression
Hu, Y., Chen, B., Lin, J., Wang, Y., Wang, Y., Mehlman, C., & Lipson, H. (2024) · Science Robotics · doi:10.1126/scirobotics.adi4724 · Cited by 73
Liu, M., Duan, Y., Ince, R. A. A., Chen, C., Garrod, O., Schyns, P. G., & Jack, R. E. (2021) · Current Biology · doi:10.1016/j.cub.2021.10.035 · Cited by 48
Self-citationCultural facial expressions dynamically convey emotion category and intensity information
Chen, C., Messinger, D. S., Cheng, C., Yan, H., Duan, Y., Ince, R. A. A., Garrod, O., Schyns, P. G., & Jack, R. E. (2023) · Current Biology · doi:10.1016/j.cub.2023.12.001 · Cited by 37
Self-citationLearning Bodily Expression of Emotion for Social Robots Through Human Interaction
Tuyen, N. T. V., Elibol, A., & Chong, N. Y. (2020) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2020.3005907 · Cited by 30
A Feasibility Study of Culture-Aware Cloud Services for Conversational Robots
Recchiuto, C. T., & Sgorbissa, A. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3015461 · Cited by 29
Wang, H., Gaddy, V., Beveridge, J. R., & Ortega, F. R. (2021) · Multimodal Technologies and Interaction · doi:10.3390/mti5030013 · Cited by 34
Emotive response to a hybrid-face robot and translation to consumer social robots
Wairagkar, M., Lima, M. R., Bazo, D., Craig, R., Weissbart, H., Etoundi, A. C., Reichenbach, T., Iyengar, P., Vaswani, S., James, C., et al. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3097592 · Cited by 24
Facial Expression Modeling and Synthesis for Patient Simulator Systems: Past, Present, and Future
Pourebadi, M., & Riek, L. D. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3483598 · Cited by 25
Chen, P.-H. A., & Qu, Y. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.703999 · Cited by 19
Deep learning enhanced attributes conditional random forest for robust facial expression recognition
Liao, H., Wang, D., Fan, P., & Ding, L. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10951-8 · Cited by 11
Gingras, F., Fiset, D., Plouffe‐Demers, M., Deschênes, A., Cormier, S., Forget, H., & Blais, C. (2023) · British Journal of Psychology · doi:10.1111/bjop.12641 · Cited by 13
Chen, C., Garrod, O., Ince, R. A. A., Foster, M. E., Schyns, P. G., & Jack, R. E. (2020) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3383652.3423913 · Cited by 8
Self-citationSocial appropriateness in HMI: The Five Factors of Social Appropriateness (FASA) Model
Wullenkord, R., Bellon, J., Gransche, B., Nähr-Wagener, S., & Eyssel, F. (2022) · Interaction Studies Social Behaviour and Communication in Biological and Artificial Systems · doi:10.1075/is.22017.wul · Cited by 7
Teaching robots the art of human social synchrony
Jack, R. E. (2024) · Science Robotics · doi:10.1126/scirobotics.ado5755 · Cited by 6
Self-citationFacial expression generation of 3D avatar based on semantic analysis
Mukashev, D., Kairgaliyev, M., Alibekov, U., Oralbayeva, N., & Sandygulova, A. (2021) · IEEE RO-MAN · doi:10.1109/ro-man50785.2021.9515463 · Cited by 6
Exploring AI's Role in Supporting Diversity and Inclusion Initiatives in Multicultural Marketplaces
Mariyono, D., & Hidayatullah, A. N. A. (2024) · International Journal of Religion · doi:10.61707/apmwg008 · Cited by 16
Expressive Cognitive Architecture for a Curious Social Robot
Rosenberg, M., Park, H. W., Rosenberg‐Kima, R. B., Ali, S., Ostrowski, A. K., Breazeal, C., & Gordon, G. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3451531 · Cited by 5
Data-driven Communicative Behaviour Generation: A Survey
Oralbayeva, N., Aly, A., Sandygulova, A., & Belpaeme, T. (2024) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3609235 · Cited by 12
Investigating the Role of Culture on Negative Emotion Expressions in the Wild
Hughson, E., Javadi, R., Thompson, J., & Lim, A. (2021) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2021.699667 · Cited by 4
Interpreting and Generating Gestures with Embodied Human Computer InteractionsPreprint
Pustejovksy, J., Krishnaswamy, N., Beveridge, R., Ortega, F. R., Patil, D., Wang, H., & McNeely-White, D. G. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4088624 · Cited by 4
A preliminary study on Reading the Mind in the Eyes of the Robot
Kang, J. C., & Park, Y. E. (2021) · IEEE RO-MAN · doi:10.1109/ro-man50785.2021.9515409 · Cited by 5
The Effect of Conceptual Embodiment on Human-Robot Trust During a Youth Emotion Classification Task
Bryant, D. G., Xu, J. C., Rogers, K., & Howard, A. M. (2021) · IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO) · doi:10.1109/arso51874.2021.9542839 · Cited by 3
Beauty is in the Eye of the Cultural BeholderPreprint
Zhan, J., Liu, M., Garrod, O., Daube, C., Ince, R. A. A., Jack, R. E., & Schyns, P. G. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/34vsj · Cited by 3
Self-citationJo, H., & Park, D.-H. (2024) · IEEE Access · doi:10.1109/access.2024.3434544 · Cited by 4
Improving Emotional Expression Recognition of Robots Using Regions of Interest from Human Data
Bloem, A. C., Barakova, E. I., Hootsmans, I. M., Opheij, L. M., Toebosch, R., Kerzel, M., & Barros, P. V. A. (2020) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3371382.3378359 · Cited by 2
Virtual and real: Symbolic and natural experiences with social robots
Reeves, B. B. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x22001522 · Cited by 1
From Music to Motion: Learning Robotic Emotional Body Expressions from Musical Instruments Playing
Ma, K., Zhang, B., He, J., & Tan, J. (2025) · International Conference on Control, Automation and Systems (ICCAS) · doi:10.23919/iccas66577.2025.11301280 · Cited by 1
A Review of The Evolution of Cross-Cultural Design Research
Yu, D., Liu, Y., Lei, X., & Rau, P. P. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2607560 · Cited by 4
Facial expressions dynamically cascade emotion category and intensity information over timePreprint
Chen, C., Messinger, D. S., Cheng, C., Yan, H., Duan, Y., Ince, R. A. A., Garrod, O., Schyns, P. G., & Jack, R. E. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4gpev · Cited by 1
Self-citationDesigning Social Signals for Artificial Agents Using Psychological Science
Jack, R. E. (2019) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3308532.3337711
Self-citationBuilding a Heterogeneous, Large Scale Morphable Face Model.Preprint
Ferrari, C., Berretti, S., Pala, P., & Bimbo, A. D. (2020) · arXiv (Cornell University)
DIỄN HOẠT BIỂU CẢM NHÂN VẬT 3D THEO NGỮ NGHĨA CÂU TIẾNG VIỆT
Chi, Đ. T., & Thai, L. S. (2022) · TNU Journal of Science and Technology · doi:10.34238/tnu-jst.6361
Modelling Culturally Diverse Smiles Using Data-Driven Methods
Chen, C., Garrod, O., Schyns, P. G., & Jack, R. E. (2023) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg57933.2023.10042621 · Cited by 1
Self-citationUsing Data-Driven Methods to Advance Knowledge of Social Face Perception
Gosetti, V., & Jack, R. E. (2023) · Cambridge University Press eBooks · doi:10.1017/9781009209656.013
Self-citationFAVS: 3D Facial Animation According to Vietnamese Semantic Analysis
Chi, D. T., Thai, L. S., & Huân, N. V. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-50818-9_29
Emotion Perception in Synthetic Aging: Divergent Insights from Human Perception and Algorithms
Huang, H., & Chen, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07448-5_32
Self-citationData-driven psychophysical methods to diversify SIAs and address bias
Gosetti, V., & Jack, R. E. (2026) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-026-00472-9
Self-citationIshizu, N., Taguchi, R., Yeoh, W. L., & Fukuda, O. (2026) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii64115.2026.11404444
Towards Hyper-realistic Robot Heads: A Review
Itatani, N., & Zavaglia, M. (2026) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-026-02417-6
Metadata
- Authors
- Chen, C., Hensel, L. B., Duan, Y., Ince, R. A. A., Garrod, O. G. B., Beskow, J., Jack, R. E., & Schyns, P. G.
- Affiliations
- University of Glasgow (GB)
- ORCID
- Venue
- 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 1-8
- ISSN
- 1541-5058 (ISSN-L)
- OpenAlex
- W2922943851
- Cited by
- 41
- References
- 52
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 5 · 2023: 6 · 2022: 3 · 2021: 12 · 2020: 8 · 2019: 1
- FWCI
- 9.90
- Type
- conference paper
- Language
- en
- Open access
- green
- eprints.gla.ac.uk
- Licence
- Locations
- International Conference on Automatic Face and Gesture Recognition (publishedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion, cc-by-sa)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/N019261/1)
- Economic and Social Research Council (1944569)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Keywords
- social robots, emotional facial expressions, cultural perception, East Asian culture, facial action units, universal facial expressions, human-likeness, social signaling, robot expressiveness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.