Facial Expression Recognition with Identity and Emotion Joint Learning
The DOI matches its Crossref record; 46 of the 47 references it cites were found in a registry or an index.
Abstract
Different subjects may express a specific expression in different ways due to inter-subject variabilities. In this work, besides training deep-learned facial expression feature (emotional feature), we also consider the influence of latent face identity feature such as the shape or appearance of face. We propose an identity and emotion joint learning approach with deep convolutional neural networks (CNNs) to enhance the performance of facial expression recognition (FER) tasks. First, we learn the emotion and identity features separately using two different CNNs with their corresponding training data. Second, we concatenate these two features together as a deep-learned Tandem Facial Expression (TFE) Feature and feed it to the subsequent fully connected layers to form a new model. Finally, we perform joint learning on the newly merged network using only the facial expression training data. Experimental results show that our proposed approach achieves 99.31 and 84.29 percent accuracy on the CK+ and the FER+ database, respectively, which outperforms the residual network baseline as well as many other state-of-the-art methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 3×
“ResNet can converge faster and perform more accurately due to its residual learning mechanism, shortcut connection [22] and batch normalization [32].”
BackgroundMethodInfluential(2010) · International Conference on Machine Learning · Cited by 13,995
A training algorithm for optimal margin classifiers
Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130401 · Cited by 11,992
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikainen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,584
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636 · cited 1×
“They further combined the residual network architecture with Inception-v4 as Inception-ResNet [34] and achieve better performance on the ImageNet dataset [35].”
Method(2013) · Cited by 3,740
(1989) · neural information processing systems · Cited by 3,736
Comprehensive database for facial expression analysis
Kanade, T., Cohn, J. F., & Tian, Y. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840611 · Cited by 2,591
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300
Deep Learning Face Representation from Predicting 10,000 Classes
Sun, Y., Wang, X., & Tang, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.244 · Cited by 1,997
Deep Learning Face Representation by Joint Identification-Verification
Sun, Y., Wang, X., & Tang, X. (2014) · arXiv · doi:10.48550/arxiv.1406.4773 · Cited by 1,906
Liu, C., & Wechsler, H. (2002) · IEEE Transactions on Image Processing · doi:10.1109/tip.2002.999679 · Cited by 1,886
Challenges in Representation Learning: A Report on Three Machine Learning Contests
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-42051-1_16 · Cited by 1,719
Learning Face Representation from Scratch
Yi, D., Lei, Z., Liao, S., & Li, S. Z. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.7923 · Cited by 1,521
Deep Convolutional Network Cascade for Facial Point Detection
Sun, Y., Wang, X., & Tang, X. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.446 · Cited by 1,382
Training deep networks for facial expression recognition with crowd-sourced label distribution
Barsoum, E., Zhang, C., Ferrer, C. C., & Zhang, Z. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2993165 · Cited by 926
DeepID3: Face Recognition with Very Deep Neural Networks
Sun, Y. J., Liang, D., Wang, X., & Tang, X. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.00873 · Cited by 842
Lopes, A. T., de Aguiar, E., De Souza, A. F., & Oliveira-Santos, T. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.07.026 · Cited by 799
Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition
Jung, H., Lee, S., Yim, J., Park, S., & Kim, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.341 · Cited by 781 · cited 5×
“[1], we use feature-level concatenation of deeplearned identity and emotion features to form a new network with”
BackgroundMethodInfluentialChallenges in representation learning: A report on three machine learning contests
Goodfellow, I. J., Erhan, D., Luc Carrier, P., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.005 · Cited by 697
Facial Expression Recognition via a Boosted Deep Belief Network
Liu, P., Han, S., Meng, Z., & Tong, Y. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.233 · Cited by 650
Image based Static Facial Expression Recognition with Multiple Deep Network Learning
Yu, Z., & Zhang, C. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830595 · Cited by 598
Speaker adaptation of neural network acoustic models using i-vectors
Saon, G., Soltau, H., Nahamoo, D., & Picheny, M. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707705 · Cited by 577
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527
Learning to Align from Scratch
Huang, G. B., Mattar, M., Lee, H., & Miller, E. G. (2012) · neural information processing systems · Cited by 351
Peak-Piloted Deep Network for Facial Expression Recognition
Zhao, X., Liang, X., Liu, L., Li, T., Han, Y., Vasconcelos, N., & Yan, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_27 · Cited by 304
Surpassing Human-Level Face Verification Performance on LFW with GaussianFace
Lu, C., & Tang, X. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9797 · Cited by 303
Disentangling Factors of Variation for Facial Expression Recognition
Rifai, S., Bengio, Y., Courville, A., Vincent, P., & Mirza, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_58 · Cited by 215 · cited 1×
“Second, different subjects express the same specific facial expression in different ways due to the inter-subject variability and their facial biometric shapes [4], [5], [6], etc.”
BackgroundDeep Learning using Support Vector MachinesPreprint
Tang, Y. (2013) · arXiv (Cornell University) · Cited by 202
Wang, S.-J., Chen, H.-L., Yan, W.-J., Chen, Y.-H., & Fu, X. (2014) · Neural Processing Letters · doi:10.1007/s11063-013-9288-7 · Cited by 178
Automatic Pain Assessment with Facial Activity Descriptors
Werner, P., Al-Hamadi, A., Limbrecht-Ecklundt, K., Walter, S., Gruss, S., & Traue, H. C. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2537327 · Cited by 166
Facial Expression Recognition Using Constructive Feedforward Neural Networks
Ma, L., & Khorasani, K. (2004) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2004.825930 · Cited by 160
I-vector-based speaker adaptation of deep neural networks for French broadcast audio transcription
Gupta, V., Kenny, P., Ouellet, P., & Stafylakis, T. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854823 · Cited by 114
Facial expression recognition based on local binary patterns
Feng, X., Pietikäinen, M., & Hadid, A. (2007) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661807040190 · Cited by 101
Hybrid Deep Learning for Face Verification
Sun, Y. J., Wang, X., & Tang, X. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2505293 · Cited by 96
Facial expression recognition based on transfer learning from deep convolutional networks
Xu, M., Cheng, W., Zhao, Q., Ma, L., & Xu, F. (2015) · International Conference on Natural Computation (ICNC) · doi:10.1109/icnc.2015.7378076 · Cited by 85
Task-dependent enhancement of facial expression and identity representations in human cortex
Dobs, K., Schultz, J., Bülthoff, I., & Gardner, J. L. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.02.013 · Cited by 47
Extreme Learning Machine Ensemble Using Bagging for Facial Expression Recognition
Ghimire, D., & Lee, J. (2014) · Journal of Information Processing Systems · doi:10.3745/jips.02.0004 · Cited by 46
Incorporating prior knowledge from the new person into recognition of facial expression
Mohammadian, A., Aghaeinia, H., & Towhidkhah, F. (2014) · Signal Image and Video Processing · doi:10.1007/s11760-014-0732-6 · Cited by 7
Research works printed but not linked: 1
- torre, 2015, Intraface, Proc IEEE Int Conf Workshops Autom Face Gesture Recognit, 1
Cited by
A-MobileNet: An approach of facial expression recognition
Nan, Y., Ju, J., Hua, Q., Zhang, H., & Wang, B. (2022) · Alexandria Engineering Journal · doi:10.1016/j.aej.2021.09.066 · Cited by 229
Vision Transformer With Attentive Pooling for Robust Facial Expression Recognition
Xue, F., Wang, Q., Tan, Z., Ma, Z., & Guo, G. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3226473 · Cited by 162
Karnati, M., Seal, A., Bhattacharjee, D., Yazidi, A., & Krejcar, O. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3243661 · Cited by 161
Efficient Facial Feature Learning with Wide Ensemble-Based Convolutional Neural Networks
Siqueira, H. L., Magg, S., & Wermter, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6037 · Cited by 175
Recognizing Facial Expressions Using a Shallow Convolutional Neural Network
Miao, S., Xu, H., Han, Z., & Zhu, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2921220 · Cited by 110
Self-citation
Metadata
- Authors
- Li, M., Xu, H., Huang, X., Song, Z., Liu, X., & Li, X.
- Affiliations
- Duke Kunshan University (CN)
- Sun Yat-sen University (CN)
- Shandong Yingcai University (CN)
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2021-04-01
- Volume
- 12
- Issue
- 2
- Pages
- 544-550
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W2900164843
- Cited by
- 128
- References
- 47
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 25 · 2023: 25 · 2022: 26 · 2021: 18
- FWCI
- 10.81
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Affective Computing (publishedVersion)
- Funders
- National Natural Science Foundation of China (61773413)
- Natural Science Foundation of Guangzhou City (201707010363)
- Guangzhou Municipal Science and Technology Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- facial expression recognition, emotion recognition, joint learning, interindividual variability, facial shape, facial appearance, deep convolutional neural networks, CK+ database
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.