The DOI matches its Crossref record; 68 of the 76 references it cites were found in a registry or an index.
Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network
Abstract
This paper presents a novel and efficient deep fusion convolutional neural network (DF-CNN) for multimodal 2D+3D facial expression recognition (FER). DF-CNN comprises a feature extraction subnet, a feature fusion subnet, and a softmax layer. In particular, each textured three-dimensional (3D) face scan is represented as six types of 2D facial attribute maps (i.e., geometry map, three normal maps, curvature map, and texture map), all of which are jointly fed into DF-CNN for feature learning and fusion learning, resulting in a highly concentrated facial representation (32-dimensional). Expression prediction is performed by two ways: 1) learning linear support vector machine classifiers using the 32-dimensional fused deep features, or 2) directly performing softmax prediction using the six-dimensional expression probability vectors. Different from existing 3D FER methods, DF-CNN combines feature learning and fusion learning into a single end-to-end training framework. To demonstrate the effectiveness of DF-CNN, we conducted comprehensive experiments to compare the performance of DFCNN with handcrafted features, pre-trained deep features, finetuned deep features, and state-of-the-art methods on three 3D face datasets (i.e., BU-3DFE Subset I, BU-3DFE Subset II, and Bosphorus Subset). In all cases, DF-CNN consistently achieved the best results. To the best of our knowledge, this is the first work of introducing deep CNN to 3D FER and deep learning-based featurelevel fusion for multimodal 2D+3D FER.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Improvements in accuracy are reported when following this approach (e.g., from 39% to 42% in [89]; see also [99], [92]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [92]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846
Vedaldi, A., & Fulkerson, B. (2010) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1873951.1874249 · Cited by 2,691
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · cited 4×
“, [5], [6], [8], [18], [37], [49], [55], [56], [58], [61], [62]).”
BackgroundMethodInfluentialA 3D Facial Expression Database For Facial Behavior Research
Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.6 · Cited by 1,171
Bosphorus Database for 3D Face Analysis
Savran, A., Alyüz, N., Dibeklioğlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., & Akarun, L. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-89991-4_6 · Cited by 812
Facial Expression Recognition via a Boosted Deep Belief Network
Liu, P., Han, S., Meng, Z., & Tong, Y. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.233 · Cited by 667
Research works printed but not linked: 1
- li, 0, 3D facial expression recognition via multiple kernel learning of multi-scale local normal patterns, Proc 21st Int Conf Pattern Recog, 2577
Cited by
Sorted by citations
Items 201–211 of 211
Allaert, B., Bilasco, I. M., & Djéraba, C. (2022) · doi:10.1002/9781394173853.ch4
Deep Learning Taxonomy on Human Face Expression Recognition for Communication Applications
Bhargava, R., Arivazhagan, N., & Sureshbabu, K. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-27622-4_8
A Systematic Assessment on 3D-Based Deep Learning Models and Challenges in FER
Singh, R., & Vohra, A. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-2271-0_34
Real-Time Recording and Analysis of Facial Expressions of Video Viewers
Kanjalkar, P. M., Patil, S. S., Chinchole, P. J., Chitre, A. A., & Kanjalkar, J. P. (2023) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-1301-5.ch009
Low Rank Graph Regularization Embedding for 2D+3D Facial Expression Recognition
(2022) · Journal of Ubiquitous Systems and Pervasive Networks · doi:10.5383/juspn.17.02.003
Saad, N. (2026) · Information systems engineering and management · doi:10.1007/978-3-032-14778-3_6
Behzad, M. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3675961
Tajmiri, S., Kim, B., Gharti, H. N., & Jeong, C. (2026) · Journal of Engineering Mechanics · doi:10.1061/jenmdt.emeng-8742
Modified Convolutional Neural Networks for Facial Emotion Classification
Yousaf, S., Anjum, S., Ibrar, N., Bibi, R., & Asghar, M. A. (2024) · International Journal of Innovations in Science and Technology · doi:10.33411/ijist/1110
Hariri, W. (2026) · Soft Computing · doi:10.1007/s00500-026-11527-0
Metadata
- Authors
- Li, H., Sun, J., Xu, Z., & Chen, L.
- Affiliations
- Xi'an Jiaotong University (CN)
- École Centrale de Lyon (FR)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2017-12
- Volume
- 19
- Issue
- 12
- Pages
- 2816-2831
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2621864722
- Cited by
- 211
- References
- 76
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 13 · 2023: 16 · 2022: 27 · 2021: 46 · 2020: 35 · 2019: 49 · 2018: 11
- FWCI
- 10.78
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Funders
- NSFC (11401464, 61472313, 11622106)
- Chinese Postdoctoral Science Foundation (2014M560785)
- International Exchange Foundation of China NSFC
- United Kingdom RS (61711530242)
- French Research Agency
- l’Agence Nationale de Recherche (ANR-13-CORD-0004-02, ANR-13-INSE-0004-02)
- Partner University Foundation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- facial expression recognition, 3D facial scans, multimodal fusion, feature fusion, normal maps, curvature maps, end-to-end learning, BU-3DFE
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
