Decision level combination of multiple modalities for recognition and analysis of emotional expression
Abstract
Emotion is expressed and perceived through multiple modalities. In this work, we model face, voice and head movement cues for emotion recognition and we fuse classifiers using a Bayesian framework. The facial classifier is the best performing followed by the voice and head classifiers and the multiple modalities seem to carry complementary information, especially for happiness. Decision fusion significantly increases the average total unweighted accuracy, from 55% to about 62%. Overall, we achieve average accuracy on the order of 65-75% for emotional states and 30-40% for neutral state using a large multi-speaker, multimodal database. Performance analysis for the case of anger and neutrality suggests a positive correlation between the number of classifiers that performed well and the perceptual salience of the expressed emotion.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Considering multiple modalities may give a more complete description of the expressed emotion and generally tends to lead to more accurate results than unimodal techniques.
“considering multiple modalities may give a more complete description of the expressed emotion and generally tends to lead to more accurate results than unimodal techniques [3].”
From Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling (2010) · cites this paper as [3]
Cites
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · cited 1× · see citing works
“Index Terms— Multimodal Emotion Recognition, Hidden Markov Model, Bayesian Information Fusion, Perceptual Salience”
BackgroundSelf-citation
Flagged first, then by citations
Cited by
Sorted by citations
Deep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401 · see citing works
Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-646 · Cited by 201 · see citing works
Self-citationError weighted semi-coupled hidden Markov model for audio-visual emotion recognition
Lin, J.-C., Wu, C.-H., & Wei, W.-L. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2011.2171334 · Cited by 121 · see citing works
Deep Multimodal Emotion Recognition on Human Speech: A Review
Koromilas, P., & Giannakopoulos, T. (2021) · Applied Sciences · doi:10.3390/app11177962 · Cited by 58 · see citing works
- Authors
- Metallinou, A., Lee, S., & Narayanan, S.
- Affiliations
- University of Southern California (US)
- Venue
- 2010 IEEE International Conference on Acoustics, Speech and Signal Processing
- Publisher
- IEEE
- Published
- 2010
- Pages
- 2462-2465
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W1995794789
- Cited by
- 89
- References
- 1
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 3 · 2022: 3 · 2021: 8 · 2020: 5 · 2019: 10 · 2018: 4 · 2017: 7 · 2016: 7 · 2015: 9 · 2014: 4 · 2013: 13 · 2012: 3
- FWCI
- 15.66
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal emotion recognition, decision fusion, facial expressions, vocal emotion, happiness, anger, neutral emotion, perceptual salience
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
