A multimodal fuzzy inference system using a continuous facial expression representation for emotion detection
Abstract
This paper presents a multimodal fuzzy inference system for emotion detection. The system extracts and merges visual, acoustic and context relevant features. The experiments have been performed as part of the AVEC 2012 challenge. Facial expressions play an important role in emotion detection. However, having an automatic system to detect facial emotional expressions on unknown subjects is still a challenging problem. Here, we propose a method that adapts to the morphology of the subject and that is based on an invariant representation of facial expressions. Our method relies on 8 key expressions of emotions of the subject. In our system, each image of a video sequence is defined by its relative position to these 8 expressions. These 8 expressions are synthesized for each subject from plausible distortions learnt on other subjects and transferred on the neutral face of the subject. Expression recognition in a video sequence is performed in this space with a basic intensity-area detector. The emotion is described in the 4 dimensions: valence, arousal, power and expectancy. The results show that the duration of high intensity smile is an expression that is meaningful for continuous valence detection and can also be used to improve arousal detection. The main variations in power and expectancy are given by context data.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
An Experiment in Linguistic Synthesis with a Fuzzy Logic Controller
Mamdani, E. H., & ASSILIAN, S. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1973.0303 · Cited by 5,627 · see citing works
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409 · see citing works
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797 · see citing works
Comprehensive database for facial expression analysis
Takeo Kanade, A., Cohn, J. F., & Tian, Y. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840611 · Cited by 2,612 · see citing works
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603 · see citing works
Automatic facial expression analysis: a survey
Fasel, B., & Luettin, J. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(02)00052-3 · Cited by 1,810 · see citing works
(1972) · Elsevier eBooks · doi:10.1016/c2013-0-02458-9 · Cited by 1,455 · see citing works
The world of emotions is not two-dimensional
Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.02024.x · Cited by 1,179 · see citing works
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · see citing works
FEELTRACE: an instrument for recording perceived emotion in real time
Cowie, R., Douglas‐Cowie, E., Savvidou, S. M., & McMahon, E. R. (2000) · Research Portal (Queen's University Belfast) · Cited by 539 · see citing works
Emotion recognition in human–computer interaction
Fragopanagos, N., & Taylor, J. G. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.006 · Cited by 414 · see citing works
The first facial expression recognition and analysis challenge
Valstar, M. F., Jiang, B., Mehu, M., Pantic, M., & Scherer, K. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771374 · Cited by 385 · see citing works
The painful face – Pain expression recognition using active appearance models
Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar, Z., Prkachin, K. M., & Solomon, P. E. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2009.05.007 · Cited by 355 · see citing works
The SEMAINE corpus of emotionally coloured character interactions
McKeown, G. J., Valstar, M., Cowie, R., & Pantić, M. (2010) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2010.5583006 · Cited by 282 · see citing works
AVEC 2011–The First International Audio/Visual Emotion Challenge
Schuller, B. W., Valstar, M., Eyben, F., McKeown, G. J., Cowie, R., & Pantić, M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_53 · Cited by 270 · see citing works
(2009) · Springer eBooks · doi:10.1007/978-0-387-73003-5_2171 · Cited by 269 · see citing works
Ekman, P. 1.-2. (1982) · Cited by 237 · see citing works
Schuller, B. W., Valstar, M., Eyben, F., Cowie, R., & Pantić, M. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388776 · Cited by 225 · see citing works
Emotion Recognition Based on Joint Visual and Audio Cues
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.489 · Cited by 202 · see citing works
Littlewort, G. C., Bartlett, M. S., & Lee, K. (2007) · International Conference on Multimodal Interfaces · doi:10.1145/1322192.1322198 · Cited by 152 · see citing works
Natural facial expression recognition using differential-AAM and manifold learning
Cheon, Y., & Kim, D. (2008) · Pattern Recognition · doi:10.1016/j.patcog.2008.10.010 · Cited by 131 · see citing works
Audio-visual based emotion recognition-a new approach
Song, M., Bu, J., Chen, C., & Li, N. (2004) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2004.1315276 · Cited by 126 · see citing works
Chang, Y., Hu, C., & Turk, M. (2004) · Proceedings - IEEE International SOI Conference/Proceedings · doi:10.1109/amfg.2003.1240820 · Cited by 122 · see citing works
Facial expression recognition using emotion avatar image
Yang, S., & Bhanu, B. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771364 · Cited by 96 · see citing works
Bimodal Fusion of Emotional Data in an Automotive Environment
Hoch, S., Althoff, F., McGlaun, G., & Rigoll, G. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1415597 · Cited by 87 · see citing works
Appearance Manifold of Facial Expression
Shan, C., Gong, S., & McOwan, P. W. (2005) · Lecture notes in computer science · doi:10.1007/11573425_22 · Cited by 86 · see citing works
Audiovisual discrimination between laughter and speech
Petridis, S., & Pantić, M. (2008) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2008.4518810 · Cited by 83 · see citing works
Schuller, B. W., Valstar, M., Cowie, R., & Pantić, M. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388758 · Cited by 76 · see citing works
Real-Time Facial Expression Recognition Using a Fuzzy Emotion Model
Esau, N., Wetzel, E., Kleinjohann, L., et al. (2007) · Proceedings of ... IEEE International Conference on Fuzzy Systems · doi:10.1109/fuzzy.2007.4295451 · Cited by 75 · see citing works
Modeling Naturalistic Affective States Via Facial, Vocal, and Bodily Expressions Recognition
Karpouzis, K. C., Caridakis, G., Kessous, L., Amir, N., Raouzaiou, A. T., Malatesta, L., & Kollias, S. (2007) · Springer eBooks · doi:10.1007/978-3-540-72348-6_5 · Cited by 72 · see citing works
Sénéchal, T., Rapp, V., Salam, H., et al. (2011) · doi:10.1109/fg.2011.5771363 · Cited by 48 · see citing works
Self-citation2018 IEEE International Workshop on
(2018) · doi:10.1109/metroi4.2018.8428339 · Cited by 38 · see citing works
Automatic design of a control interface for a synthetic face
Stoiber, N., Séguier, R., & Breton, G. (2009) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1502650.1502681 · Cited by 29 · see citing works
Self-citationSoladié, C., Stoiber, N., & Séguier, R. (2012) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2012.6467435 · Cited by 8 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
AFEW-VA database for valence and arousal estimation in-the-wild
Kossaifi, J., Tzimiropoulos, G., Todorovic, S., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.02.001 · Cited by 236 · see citing works
Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities
Savran, A., Cao, H., Nenkova, A., & Verma, R. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2362101 · Cited by 53 · see citing works
- Authors
- Soladié, C., Salam, H., Pelachaud, C., Stoiber, N., & Séguier, R.
- Affiliations
- Supélec (FR)
- CNRS - Telecom ParisTech, Paris, France
- Dynamixyz, Rennes, France
- Venue
- Proceedings of the 14th ACM international conference on Multimodal interaction
- Publisher
- ACM
- Published
- 2012-10-22
- Pages
- 493-500
- OpenAlex
- W2109624827
- Cited by
- 33
- References
- 34
- Citations by year
- 2024: 2 · 2023: 3 · 2022: 1 · 2020: 1 · 2019: 3 · 2017: 5 · 2016: 2 · 2015: 5 · 2014: 6 · 2013: 5
- FWCI
- 9.89
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- HAL (submittedVersion)
- Funders
- Agence Nationale de la Recherche
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- emotion recognition, multimodal fusion, fuzzy inference system, facial expression recognition, facial expression synthesis, arousal detection, contextual features, video-based emotion recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
