The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Multimodal Integration of Emotional Signals from Voice, Body, and Context: Effects of (In)Congruence on Emotion Recognition and Attitudes Towards Robots
Open accessAbstract
Humanoid social robots have an increasingly prominent place in today’s world. Their acceptance in social and emotional human–robot interaction (HRI) scenarios depends on their ability to convey well recognized and believable emotional expressions to their human users. In this article, we incorporate recent findings from psychology, neuroscience, human–computer interaction, and HRI, to examine how people recognize and respond to emotions displayed by the body and voice of humanoid robots, with a particular emphasis on the effects of incongruence. In a social HRI laboratory experiment, we investigated contextual incongruence (i.e., the conflict situation where a robot’s reaction is incongrous with the socio-emotional context of the interaction) and cross-modal incongruence (i.e., the conflict situation where an observer receives incongruous emotional information across the auditory (vocal prosody) and visual (whole-body expressions) modalities). Results showed that both contextual incongruence and cross-modal incongruence confused observers and decreased the likelihood that they accurately recognized the emotional expressions of the robot. This, in turn, gives the impression that the robot is unintelligent or unable to express “empathic” behaviour and leads to profoundly harmful effects on likability and believability. Our findings reinforce the need of proper design of emotional expressions for robots that use several channels to communicate their emotional states in a clear and effective way. We offer recommendations regarding design choices and discuss future research areas in the direction of multimodal HRI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334 · cited 3×
“, positivity/negativity of the emotion), according to the two dimensional categorical model of emotion proposed by Russel [50] (see Fig.”
MethodInfluentialAn argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646 · cited 1×
“Future studies should include more emotions, such as Ekman’s basic emotions [70], or more complex emotions (e.”
BackgroundBartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009) · International Journal of Social Robotics · doi:10.1007/s12369-008-0001-3 · Cited by 3,193 · cited 1×
“LikabilityParticipants rated the robot on the Perceived Intelligence and Likability dimensions of the Godspeed questionnaire [63].”
MethodAutonomic nervous system activity in emotion: a review
Kreibig, S. D. (2010) · Biological Psychology · doi:10.1016/j.biopsycho.2010.03.010 · Cited by 3,188 · cited 1×
“Emotion elicitation using movie clips is a common experimental manipulation used in psychology studies of emotions [48], and has been successfully used in HRI studies to elicit emotional responses in healthy individuals in the laboratory (e.g. [47,49]).”
BackgroundA survey of socially interactive robots
Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(02)00372-x · Cited by 3,162 · cited 1×
“emotional roles, such as household assistants, companions for children and elderly, partners in industries, guides in public spaces, educational tutors at school and so on [1].”
BackgroundEmotion elicitation using films
Gross, J. J., & Levenson, R. W. (1995) · Cognition & Emotion · doi:10.1080/02699939508408966 · Cited by 2,179
On the universality and cultural specificity of emotion recognition: a meta-analysis
Elfenbein, H. A., & Ambady, N. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.2.203 · Cited by 2,049
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · cited 1×
“A modality is a sense, used to perceive signals from the outside world (i.e., sight, hearing) [9].”
BackgroundDapretto, M., Davies, M. S., Pfeifer, J. H., Scott, A. A., Sigman, M., Bookheimer, S. Y., & Iacoboni, M. (2006) · Nature Neuroscience · doi:10.1038/nn1611 · Cited by 1,694
Body expressions influence recognition of emotions in the face and voice.
Van den Stock, J., Righart, R., & de Gelder, B. (2007) · Emotion · doi:10.1037/1528-3542.7.3.487 · Cited by 457 · cited 4×
“behavioural experiments, functional neuroimaging (fMRI) and electroencephalography (EEG)measurementswith faces and voices [34–36], faces and bodies [37,38], body expres-”
BackgroundMethodInfluential
Cited by
Advancements in Humanoid Robots: A Comprehensive Review and Future Prospects
Tong, Y., Liu, H., & Zhang, Z. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.124140 · Cited by 288
Voice in Human–Agent Interaction: A Survey
Seaborn, K., Miyake, N. P., Pennefather, P., & Otake-Matsuura, M. (2022) · ACM Computing Surveys · doi:10.1145/3386867 · Cited by 249
Metadata
- Authors
- Tsiourti, C., Weiss, A., Wac, K., & Vincze, M.
- Venue
- International Journal of Social Robotics
- Published
- 2019-02-04
- Volume
- 11
- Issue
- 4
- Pages
- 555-573
- ISSN
- 1875-4791 (ISSN-L)
- 1875-4805
- OpenAlex
- W2912159753
- Cited by
- 182
- References
- 13
- FWCI
- 23.58
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Integration of Emotional Signals from Voice, Body, and Context: Effects of (In)Congruence on Emotion Recognition and Attitudes Towards Robots. https://app.cochat.ai/papers/10.1007/s12369-019-00524-z
- The paper
- Tsiourti, C., Weiss, A., Wac, K., & Vincze, M. (2019). Multimodal Integration of Emotional Signals from Voice, Body, and Context: Effects of (In)Congruence on Emotion Recognition and Attitudes Towards Robots. International Journal of Social Robotics. https://doi.org/10.1007/s12369-019-00524-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.