Cited by 346Emotion and Mood RecognitionSpeech and Audio ProcessingFace and Expression Recognition

Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition

Zhang, Shiqing, Zhang, Shiliang, Huang, Tiejun, Gao, Wen, Tian, Qi·2018·IEEE Transactions on Circuits and Systems for Video Technology

Abstract

Emotion recognition is challenging due to the emotional gap between emotions and audio-visual features. Motivated by the powerful feature learning ability of deep neural networks, this paper proposes to bridge the emotional gap by using a hybrid deep model, which first produces audio-visual segment features with Convolutional Neural Networks (CNNs) and 3D-CNN, then fuses audio-visual segment features in a Deep Belief Networks (DBNs). The proposed method is trained in two stages. First, CNN and 3D-CNN models pre-trained on corresponding large-scale image and video classification tasks are fine-tuned on emotion recognition tasks to learn audio and visual segment features, respectively. Second, the outputs of CNN and 3D-CNN models are combined into a fusion network built with a DBN model. The fusion network is trained to jointly learn a discriminative audio-visual segment feature representation. After average-pooling segment features learned by DBN to form a fixed-length global video feature, a linear Support Vector Machine is used for video emotion classification. Experimental results on three public audio-visual emotional databases, including the acted RML database, the acted eNTERFACE05 database, and the spontaneous BAUM-1s database, demonstrate the promising performance of the proposed method. To the best of our knowledge, this is an early work fusing audio and visual cues with CNN, 3D-CNN, and DBN for audio-visual emotion recognition.

Integrity signals

Registry

Crossref · 10.1109/tcsvt.2017.2719043

Cites

69 of 69 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Knowledge transfer from domains like music and visual object recognition is also possible.

    “Knowledge transfer from domains like music [141] and visual object recognition [132], [85] is also possible.”
  • Some studies considered using known architectures from object recognition and regarded it as necessary to pre-train these larger models on ImageNet to avoid overfitting the relatively few labeled examples.

    “Some considered using known architectures from object recognition [132], [85], [127]. Here, the authors regarded it as necessary to pre-train these larger models on the ImageNet database to avoid overfitting the relatively few labeled examples.”

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q.
Affiliations
  • Peking University (CN)
  • The University of Texas at San Antonio (US)
Venue
IEEE Transactions on Circuits and Systems for Video Technology
Published
2018-10
Volume
28
Issue
10
Pages
3030-3043
ISSN
  • 1051-8215 (ISSN-L)
  • 1558-2205
OpenAlex
W2703895418
Cited by
346
References
69
Citations by year
2026: 12 · 2025: 31 · 2024: 39 · 2023: 45 · 2022: 57 · 2021: 55 · 2020: 43 · 2019: 43 · 2018: 20 · 2017: 1
FWCI
38.12
Type
journal article
Language
en
Open access
closed
Locations
IEEE Transactions on Circuits and Systems for Video Technology (publishedVersion)
Funders
  • National Natural Science Foundation of China (61572050, LY16F020011, 91538111, 61620106009, 61429201)
  • ARO (W911NF-15-1-0290)
  • Faculty Research Gift Awards by NEC Laboratories of America and Blippar
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
audio-visual emotion recognition, affective features, 3D convolutional neural networks, deep belief network, audio-visual feature fusion, support vector machine classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review