CoChat
Sign Up
Cited by 20Emotion and Mood RecognitionSpeech and Audio ProcessingMusic and Audio Processing

Video emotion recognition in the wild based on fusion of multimodal features

Chen, Shizhe, Li, Xinrui, Jin, Qin, Zhang, Shilei, Qin, Yong·2016·Proceedings of the 18th ACM International Conference on Multimodal Interaction
DOI 10.1145/2993148.2997629
Open this paper in CoChat — free account

Abstract

In this paper, we present our methods to the Audio-Video Based Emotion Recognition subtask in the 2016 Emotion Recognition in the Wild (EmotiW) Challenge. The task is to predict one of the seven basic emotions for the characters in the video clips extracted from movies or TV shows. In our approach, we explore various multimodal features from audio, facial image and video motion modalities. The audio features contain statistical acoustic features, MFCC Bag-of-Audio-Words and MFCC Fisher Vectors. For image related features, we extract hand-crafted features (LBP-TOP and SPM Dense SIFT) and learned features (CNN features). The improved Dense Trajectory is used as the motion related features. We train SVM, Random Forest and Logistic Regression classifiers for each kind of feature. Among them, MFCC fisher vector is the best acoustic features and the facial CNN feature is the most discriminative feature for emotion recognition. We utilize late fusion to combine different modality features and achieve a 50.76% accuracy on the testing set, which significantly outperforms the baseline test accuracy of 40.47%.

Integrity signals

Registry

Crossref · 10.1145/2993148.2997629

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Improvements in accuracy are reported when following the approach of acquiring large models pre-trained for object classification, freezing lower-level layers, and fine-tuning higher-level layers for affect recognition (e.g., from 39% to 42% in [89]).

    “Improvements in accuracy are reported when following this approach (e.g., from 39% to 42% in [89]; see also [99], [92]).”

    From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [89]

  • Since the number of labeled examples for FER remains relatively limited, large models are likely to overfit the data.

    “Since the number of labeled examples for FER remains relatively limited, large 11 models are likely to overfit the data [106], [107], [89], [108].”

    From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [89]

  • In benchmarking experiments across various datasets, many studies found that features extracted with CNNs lead to higher recognition accuracies than handcrafted features.

    “In benchmarking experiments across various datasets, many studies found that features extracted with CNNs lead to higher recognition accuracies than handcrafted features (e.g., [87], [88], [89], [90], [91], [64], [92], [93], [94]).”

    From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [89]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Chen, S., Li, X., Jin, Q., Zhang, S., & Qin, Y.
Affiliations
  • Renmin University of China (CN)
  • IBM Research (China) (CN)
ORCID
Venue
Proceedings of the 18th ACM International Conference on Multimodal Interaction
Publisher
ACM
Published
2016-10-31
Pages
494-500
DOI
10.1145/2993148.2997629
OpenAlex
W2548844710
Cited by
20
References
29
Citations by year
2025: 3 · 2024: 2 · 2023: 3 · 2022: 1 · 2021: 2 · 2020: 2 · 2019: 3 · 2018: 3 · 2017: 1
FWCI
2.70
Type
conference paper
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
Keywords
video-based emotion recognition, multimodal fusion, basic emotions, audio features, facial image features, CNN features, motion features, decision fusion, random forest, logistic regression

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review