The DOI matches its Crossref record; 68 of the 76 references it cites were found in a registry or an index.
Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network
Abstract
This paper presents a novel and efficient deep fusion convolutional neural network (DF-CNN) for multimodal 2D+3D facial expression recognition (FER). DF-CNN comprises a feature extraction subnet, a feature fusion subnet, and a softmax layer. In particular, each textured three-dimensional (3D) face scan is represented as six types of 2D facial attribute maps (i.e., geometry map, three normal maps, curvature map, and texture map), all of which are jointly fed into DF-CNN for feature learning and fusion learning, resulting in a highly concentrated facial representation (32-dimensional). Expression prediction is performed by two ways: 1) learning linear support vector machine classifiers using the 32-dimensional fused deep features, or 2) directly performing softmax prediction using the six-dimensional expression probability vectors. Different from existing 3D FER methods, DF-CNN combines feature learning and fusion learning into a single end-to-end training framework. To demonstrate the effectiveness of DF-CNN, we conducted comprehensive experiments to compare the performance of DFCNN with handcrafted features, pre-trained deep features, finetuned deep features, and state-of-the-art methods on three 3D face datasets (i.e., BU-3DFE Subset I, BU-3DFE Subset II, and Bosphorus Subset). In all cases, DF-CNN consistently achieved the best results. To the best of our knowledge, this is the first work of introducing deep CNN to 3D FER and deep learning-based featurelevel fusion for multimodal 2D+3D FER.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Improvements in accuracy are reported when following this approach (e.g., from 39% to 42% in [89]; see also [99], [92]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [92]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846
Vedaldi, A., & Fulkerson, B. (2010) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1873951.1874249 · Cited by 2,691
Automatic analysis of facial expressions: the state of the art
Pantie, M., & Rothkrantz, L. J. M. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895976 · Cited by 1,778 · cited 4×
“, [5], [6], [8], [18], [37], [49], [55], [56], [58], [61], [62]).”
BackgroundMethodInfluentialA 3D Facial Expression Database For Facial Behavior Research
Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.6 · Cited by 1,171
Bosphorus Database for 3D Face Analysis
Savran, A., Alyüz, N., Dibeklioğlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., & Akarun, L. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-89991-4_6 · Cited by 812
Facial Expression Recognition via a Boosted Deep Belief Network
Liu, P., Han, S., Meng, Z., & Tong, Y. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.233 · Cited by 667
Collecting Large, Richly Annotated Facial-Expression Databases from Movies
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2012) · IEEE Multimedia · doi:10.1109/mmul.2012.26 · Cited by 658
Zhang, Z., Lyons, M., Schuster, M., & Akamatsu, S. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670990 · Cited by 613
Image based Static Facial Expression Recognition with Multiple Deep Network Learning
Yu, Z., & Zhang, C. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2830595 · Cited by 602
Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2011) · 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) · doi:10.1109/iccvw.2011.6130508 · Cited by 541 · cited 2×
“, Acted Facial Expressions in the Wild (AFEW) [10] and Static Facial Expressions in the Wild (SFEW) [9]), leaning facial representations using deep learning is becoming the mainstream in”
BackgroundCorneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527
Segmentation and Classification of Range Images
Hoffman, R. L., & Jain, A. K. (1987) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1987.4767955 · Cited by 418
EmoNets: Multimodal deep learning approaches for emotion recognition in video
Kahou, S. E., Bouthillier, X., Lamblin, P., Gülçehre, Ç., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., et al. (2015) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0195-2 · Cited by 413
Wang, S., Liu, Z., Lv, S., Lv, Y., Wu, G., Peng, P., Chen, F., & Wang, X. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2060716 · Cited by 397
Static and dynamic 3D facial expression recognition: A comprehensive survey
Sandbach, G., Zafeiriou, S., Pantic, M., & Yin, L. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.005 · Cited by 357
Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis
Liu, M., Li, S., Shan, S., Wang, R., & Chen, X. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16817-3_10 · Cited by 333
3D Facial Expression Recognition Based on Primitive Surface Feature Distribution
Wang, J., Yin, L., Wei, X., & Sun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.14 · Cited by 307
Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?
Khorrami, P., Paine, T. L., & Huang, T. S. (2015) · 2015 IEEE International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2015.12 · Cited by 285
A novel cubic-order algorithm for approximating principal direction vectors
Goldfeather, J., & Interrante, V. (2004) · ACM Transactions on Graphics · doi:10.1145/966131.966134 · Cited by 278
Recognition of facial expressions and measurement of levels of interest from video
Yeasin, M., Bullot, B., & Sharma, R. (2006) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.870737 · Cited by 234
AU-inspired Deep Networks for Facial Expression Feature Learning
Liu, M., Li, S., Shan, S., & Chen, X. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.02.011 · Cited by 223
Learning Multiscale Active Facial Patches for Expression Analysis
Zhong, L., Liu, Q., Yang, P., Huang, J., & Metaxas, D. (2014) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2354351 · Cited by 217
Disentangling Factors of Variation for Facial Expression Recognition
Rifai, S., Bengio, Y., Courville, A., Vincent, P., & Mirza, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_58 · Cited by 215
Selective Transfer Machine for Personalized Facial Expression Analysis
Chu, W.-S., la Torre, F. D., & Cohn, J. F. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2547397 · Cited by 190
Bilinear Models for 3-D Face and Facial Expression Recognition
Mpiperis, I., Malassiotis, S., & Strintzis, M. G. (2008) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2008.924598 · Cited by 185
Deep Learning using Support Vector MachinesPreprint
Tang, Y. (2013) · arXiv (Cornell University) · Cited by 184
DeXpression: Deep Convolutional Neural Network for Expression RecognitionPreprint
Burkert, P., Trier, F., Afzal, M. Z., Dengel, A., & Liwicki, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.05371 · Cited by 164
A Set of Selected SIFT Features for 3D Facial Expression Recognition
Berretti, S., Bimbo, A. D., Pala, P., Amor, B. B., & Daoudi, M. (2010) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2010.1002 · Cited by 156
Emotion recognition using dynamic grid-based HoG features
Dahmane, M., & Meunier, J. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771368 · Cited by 154
Facial Expression Recognition Using 3D Facial Feature Distances
Soyel, H., & Demirel, H. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74260-9_74 · Cited by 146
3D facial expression recognition based on automatically selected features
Tang, H., & Huang, T. S. (2008) · 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4563052 · Cited by 140
Kim, B.-K., Lee, H., Roh, J., & Lee, S.-Y. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2830590 · Cited by 126
Real-time 2D+3D facial action and expression recognition
Tsalakanidou, F., & Malassiotis, S. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2009.12.009 · Cited by 125
Automatic facial expression recognition on a single 3D face by exploring shape deformation
Gong, B., Wang, Y., Liu, J., & Tang, X. (2009) · Proceedings of the 17th ACM international conference on Multimedia · doi:10.1145/1631272.1631358 · Cited by 116
Tang, H., & Huang, T. S. (2008) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2008.4813304 · Cited by 108
An efficient multimodal 2D + 3D feature-based approach to automatic facial expression recognition
Li, H., Ding, H., Huang, D., Wang, Y., Zhao, X., Morvan, J., & Chen, L. (2015) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2015.07.005 · Cited by 99
Self-citationMuscular Movement Model-Based Automatic 3D/4D Facial Expression Recognition
Zhen, Q., Huang, D., Wang, Y., & Chen, L. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2557063 · Cited by 97
Self-citationAutomatic 3D facial expression recognition using geometric scattering representation
Yang, X., Huang, D., Wang, Y., & Chen, L. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163090 · Cited by 92
Self-citation3D facial expression recognition: A perspective on promises and challenges
Fang, T., Zhao, X., Ocegueda, O., Shah, S. K., & Kakadiaris, I. A. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771466 · Cited by 90
Learning Personalized Models for Facial Expression Analysis and Gesture Recognition
Zen, G., Porzi, L., Sangineto, E., Ricci, E., & Sebe, N. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2523421 · Cited by 89
Shape analysis of local facial patches for 3D facial expression recognition
Maalej, A., Amor, B. B., Daoudi, M., Srivastava, A., & Berretti, S. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2011.02.012 · Cited by 83
3D/4D facial expression analysis: An advanced annotated face model approach
Fang, T., Zhao, X., Ocegueda, O., Shah, S. K., & Kakadiaris, I. A. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2012.02.004 · Cited by 81
Analyses of a Multimodal Spontaneous Facial Expression Database
Wang, S., Liu, Z., Wang, Z., Wu, G., Shen, P., He, S., & Wang, X. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.32 · Cited by 76
Automatic 3D Face Detection, Normalization and Recognition
Mian, A. S., Bennamoun, M., & Owens, R. A. (2006) · International Symposium on 3D Data Processing, Visualization and Transmission · doi:10.1109/3dpvt.2006.32 · Cited by 76
Lemaire, P., Ardabilian, M., Chen, L., & Daoudi, M. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553821 · Cited by 69
Self-citationZhao, X., Huang, D., Dellandrea, E., & Chen, L. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.907 · Cited by 66
Self-citation3D facial expression recognition with geometrically localized facial features
Soyel, H., & Demirel, H. (2008) · International Symposium on Computer and Information Sciences · doi:10.1109/iscis.2008.4717898 · Cited by 56
Prototype-Based Modeling for Facial Expression Analysis
Dahmane, M., & Meunier, J. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2321113 · Cited by 56
Liu, P., Zhou, J. T., Tsang, I. W.-H., Meng, Z., Han, S., & Tong, Y. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10593-2_11 · Cited by 55
Face Expression Recognition by Cross Modal Data Association
Tawari, A., & Trivedi, M. M. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2266635 · Cited by 53
Emotion Recognition in Text for 3-D Facial Expression Rendering
Calix, R. A., Mallepudi, S. A., Chen, B., & Knapp, G. M. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2052026 · Cited by 53
Discriminant Graph Structures for Facial Expression Recognition
Zafeiriou, S. P., & Pitas, I. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.2007292 · Cited by 51
An automatic 3D expression recognition framework based on sparse representation of conformal images
Zeng, W., Li, H., Chen, L., Morvan, J.-M., & Gu, X. D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553749 · Cited by 48
Self-citationSpeaking Effect Removal on Emotion Recognition From Facial Expressions Based on Eigenface Conversion
Wu, C., Wei, W.-L., Lin, J., & Lee, W. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2272917 · Cited by 46
Local 3D Shape Analysis for Facial Expression Recognition
Maalej, A., Amor, B. B., Daoudi, M., Srivastava, A., & Berretti, S. (2010) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2010.1003 · Cited by 46
An Automatic Framework for Textured 3D Video-Based Facial Expression Recognition
Hayat, M., & Bennamoun, M. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2330580 · Cited by 41
A Generic Framework for Efficient 2-D and 3-D Facial Expression Analogy
Song, M., Dong, Z., Theobalt, C., Wang, H., Liu, Z., & Seidel, H. (2007) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2007.906591 · Cited by 41
3D Facial Expression Recognition Based on Histograms of Surface Differential Quantities
Li, H., Morvan, J., & Chen, L. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23687-7_44 · Cited by 40
Self-citationA Data-Driven Approach for Facial Expression Retargeting in Video
Li, K., Dai, Q., Wang, R., Liu, Y., Xu, F., & Wang, J. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2293064 · Cited by 36
Facial action unit detection: 3D versus 2D modality
Savran, A., Sankur, B., & Bilge, M. T. (2010) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2010.5543263 · Cited by 32
Huang, Y., Li, Y., & Fan, N. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2052792 · Cited by 30
Muscular Movement Model Based Automatic 3D Facial Expression Recognition
Zhen, Q., Huang, D., Wang, Y., & Chen, L. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-14445-0_45 · Cited by 27
Self-citationExpressive Maps for 3D Facial Expression Recognition
Ocegueda, O., Fang, T., Shah, S. K., & Kakadiaris, I. A. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130397 · Cited by 22
A group of facial normal descriptors for recognizing 3D identical twins
Li, H., Huang, D., Chen, L., Wang, Y., & Morvan, J. (2012) · IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS) · doi:10.1109/btas.2012.6374588 · Cited by 12
Self-citation
Research works printed but not linked: 1
- li, 0, 3D facial expression recognition via multiple kernel learning of multi-scale local normal patterns, Proc 21st Int Conf Pattern Recog, 2577
Cited by
Sorted by citations
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 2,175
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 713
A Survey on Human Face Expression Recognition Techniques
Revina, I. M., & Emmanuel, W. R. S. (2018) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2018.09.002 · Cited by 327
Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expressions
Xia, Z., Hong, X., Gao, X., Feng, X., & Zhao, G. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2931351 · Cited by 296
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243
Metadata
- Authors
- Li, H., Sun, J., Xu, Z., & Chen, L.
- Affiliations
- Xi'an Jiaotong University (CN)
- École Centrale de Lyon (FR)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2017-12
- Volume
- 19
- Issue
- 12
- Pages
- 2816-2831
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2621864722
- Cited by
- 211
- References
- 76
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 13 · 2023: 16 · 2022: 27 · 2021: 46 · 2020: 35 · 2019: 49 · 2018: 11
- FWCI
- 10.78
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Funders
- NSFC (11401464, 61472313, 11622106)
- Chinese Postdoctoral Science Foundation (2014M560785)
- International Exchange Foundation of China NSFC
- United Kingdom RS (61711530242)
- French Research Agency
- l’Agence Nationale de Recherche (ANR-13-CORD-0004-02, ANR-13-INSE-0004-02)
- Partner University Foundation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- facial expression recognition, 3D facial scans, multimodal fusion, feature fusion, normal maps, curvature maps, end-to-end learning, BU-3DFE
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
