Cited by 211Face recognition and analysisEmotion and Mood RecognitionFace and Expression Recognition

Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network

Li, Huibin, Sun, Jian, Xu, Zongben, Chen, Liming·2017·IEEE Transactions on Multimedia

Abstract

This paper presents a novel and efficient deep fusion convolutional neural network (DF-CNN) for multimodal 2D+3D facial expression recognition (FER). DF-CNN comprises a feature extraction subnet, a feature fusion subnet, and a softmax layer. In particular, each textured three-dimensional (3D) face scan is represented as six types of 2D facial attribute maps (i.e., geometry map, three normal maps, curvature map, and texture map), all of which are jointly fed into DF-CNN for feature learning and fusion learning, resulting in a highly concentrated facial representation (32-dimensional). Expression prediction is performed by two ways: 1) learning linear support vector machine classifiers using the 32-dimensional fused deep features, or 2) directly performing softmax prediction using the six-dimensional expression probability vectors. Different from existing 3D FER methods, DF-CNN combines feature learning and fusion learning into a single end-to-end training framework. To demonstrate the effectiveness of DF-CNN, we conducted comprehensive experiments to compare the performance of DFCNN with handcrafted features, pre-trained deep features, finetuned deep features, and state-of-the-art methods on three 3D face datasets (i.e., BU-3DFE Subset I, BU-3DFE Subset II, and Bosphorus Subset). In all cases, DF-CNN consistently achieved the best results. To the best of our knowledge, this is the first work of introducing deep CNN to 3D FER and deep learning-based featurelevel fusion for multimodal 2D+3D FER.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tmm.2017.2713408

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Improvements in accuracy are reported when following the approach of acquiring large models pre-trained for object classification, freezing lower-level layers, and fine-tuning higher-level layers for affect recognition (e.g., from 39% to 42% in [89]).

    “Improvements in accuracy are reported when following this approach (e.g., from 39% to 42% in [89]; see also [99], [92]).”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

8 of the 76 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Li, H., Sun, J., Xu, Z., & Chen, L.
Affiliations
  • Xi'an Jiaotong University (CN)
  • École Centrale de Lyon (FR)
Venue
IEEE Transactions on Multimedia
Published
2017-12
Volume
19
Issue
12
Pages
2816-2831
ISSN
  • 1520-9210 (ISSN-L)
  • 1941-0077
OpenAlex
W2621864722
Cited by
211
References
76
Citations by year
2026: 5 · 2025: 9 · 2024: 13 · 2023: 16 · 2022: 27 · 2021: 46 · 2020: 35 · 2019: 49 · 2018: 11
FWCI
10.78
Type
journal article
Language
en
Open access
closed
Locations
IEEE Transactions on Multimedia (publishedVersion)
Funders
  • NSFC (11401464, 61472313, 11622106)
  • Chinese Postdoctoral Science Foundation (2014M560785)
  • International Exchange Foundation of China NSFC
  • United Kingdom RS (61711530242)
  • French Research Agency
  • l’Agence Nationale de Recherche (ANR-13-CORD-0004-02, ANR-13-INSE-0004-02)
  • Partner University Foundation
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
facial expression recognition, 3D facial scans, multimodal fusion, feature fusion, normal maps, curvature maps, end-to-end learning, BU-3DFE

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review