Facial Expression Recognition with Identity and Emotion Joint Learning
The DOI matches its Crossref record; 46 of the 47 references it cites were found in a registry or an index.
Abstract
Different subjects may express a specific expression in different ways due to inter-subject variabilities. In this work, besides training deep-learned facial expression feature (emotional feature), we also consider the influence of latent face identity feature such as the shape or appearance of face. We propose an identity and emotion joint learning approach with deep convolutional neural networks (CNNs) to enhance the performance of facial expression recognition (FER) tasks. First, we learn the emotion and identity features separately using two different CNNs with their corresponding training data. Second, we concatenate these two features together as a deep-learned Tandem Facial Expression (TFE) Feature and feed it to the subsequent fully connected layers to form a new model. Finally, we perform joint learning on the newly merged network using only the facial expression training data. Experimental results show that our proposed approach achieves 99.31 and 84.29 percent accuracy on the CK+ and the FER+ database, respectively, which outperforms the residual network baseline as well as many other state-of-the-art methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 3×
“ResNet can converge faster and perform more accurately due to its residual learning mechanism, shortcut connection [22] and batch normalization [32].”
BackgroundMethodInfluential(2010) · International Conference on Machine Learning · Cited by 13,995
A training algorithm for optimal margin classifiers
Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130401 · Cited by 11,992
Research works printed but not linked: 1
- torre, 2015, Intraface, Proc IEEE Int Conf Workshops Autom Face Gesture Recognit, 1
Cited by
Items 101–130 of 130
Automatic Facial Emotion Recognition using Convolutional Neural Networks
Kumar, S., & Yadav, R. K. (2021) · International Conference on Signal Processing and Integrated Networks (SPIN) · doi:10.1109/spin52536.2021.9566134 · Cited by 1
Facial Expression Analysis using Convolutional Neural Networks
Gopadi, A. S., Deepak, S., Kiran, R., Naveena, A. M., Srividya, M. S., & Anala, M. R. (2020) · International Journal of Engineering and Advanced Technology · doi:10.35940/ijeat.d8335.049420 · Cited by 1
Deep Learning Based Intelligent Screening Mechanism
Hussain, G. K. J., Rithic, K. C., Shyam, S., Prithiviraj, M., & Rajendran, R. (2022) · International Conference on Communication, Computing and Internet of Things (IC3IoT) · doi:10.1109/ic3iot53935.2022.9767865 · Cited by 1
FERA-net: An emotion classifier from facial expressions using FER-net with attention mechanism✱
Sreedivya, R. S., VIJITHA, P. R., & Ganapathy, S. (2022) · Indian Conference on Computer Vision, Graphics and Image Processing · doi:10.1145/3571600.3571616 · Cited by 1
Grey Wolf Optimization for Facial Emotion Recognition: Survey
Marzouk, H. F., & Mosa, Q. O. (2023) · Journal of Al-Qadisiyah for Computer Science and Mathematics · doi:10.29304/jqcm.2023.15.2.1229 · Cited by 1
Research on Facial Expression Recognition Method Based on Improved VGG16 Network
Luo, X., Xu, Z., & Feng, F. (2023) · IEEE Information Technology and Mechatronics Engineering Conference (ITOEC) · doi:10.1109/itoec57671.2023.10292030 · Cited by 1
Kaur, T., & Gandhi, T. K. (2023) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt56998.2023.10308188 · Cited by 1
Jabbar, S. F., Abdul-Rahaim, L. A., & Lilo, M. A. (2024) · International Congress on Human-Computer Interaction, Optimization and Robotic Applications (ICHORA) · doi:10.1109/hora61326.2024.10550840 · Cited by 1
Real Time Emotion Recognition and Classification for Diverse Suggestions using Deep Learning
Sanika, A. G., Rachana, N. P., Ritika, A. P., Sharwari, H. K., & Rohini, K. (2024) · ITEGAM- Journal of Engineering and Technology for Industrial Applications (ITEGAM-JETIA) · doi:10.5935/jetia.v10i48.1007 · Cited by 1
Binthu Kumari, M., & Sivagami, B. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20079-0 · Cited by 1
Occlusion-Aware Facial Expression Recognition Based Region Re-weight Network
Zhang, X., Zhang, X., Zhou, J., & Lin, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-89370-5_16
Kumari, M. B., & Sivagami, B. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4096801
Kumari, M. B., & Sivagami, B. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4170111
Memristive Circuit Design of Sequencer Network for Human Emotion Classification
Ji, X., Dong, Z., Wang, H., Lai, C. S., & Qi, D. (2022) · IEEE International Conference on Recent Advances in Systems Science and Engineering (RASSE) · doi:10.1109/rasse54974.2022.9989605
Kumari, M. B., & Sivagami, B. (2023) · Research Square · doi:10.21203/rs.3.rs-2703867/v1
An Intelligent Teaching Evaluation System Integrating Emotional Computing and Cloud Platform
Dian-kui, L., Chen, Y., & Chen, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-99-9640-7_39
A Proposal for Deep Learning Based Emotion Identification
Nayak, S., Pandey, M., & Rautaray, S. S. (2024) · International Conference on System Modeling & Advancement in Research Trends (SMART) · doi:10.1109/smart63812.2024.10882469
Harakannanavar, S. S., Somesh, S., Sughosh, V., & Sumeet, P. (2024) · International Conference on Distributed Systems, Computer Networks and Cybersecurity (ICDSCNC) · doi:10.1109/icdscnc62492.2024.10939593
Saini, S., & Bollipo, L. M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5387191
Saini, S., & Bollipo, L. M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5387194
Sentiment and Emotion Analysis in Video Interviews Using Deep Learning Techniques
Jain, R. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-6516-9.ch006
Khayyiba, K., Amine, A., & Nassih, B. (2025) · Springer eBooks · doi:10.1007/978-3-031-86837-5_9
Joice, C. S., Shanthini, S. H., & Jenisha, C. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-06253-6_30
Emotion Recognition by the Facial Expressions using Machine Learning
Moravčík, E. (2021) · DSpace VŠB-TUO (VŠB-TUO)
Decoupled Multimodal Fusion Network Based on Peripheral Physiological Signals
Fan, T., Qiu, S., Wang, Y., Wang, Z., Zhao, H., Jiang, J., Xu, J., Sun, T., & Tian, F. (2026) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2026.3668039
A generative multimodal network for facial expression recognition
Zhao, Y., Song, M., Zhang, Q., Yang, J., Yoshigoe, K., & Tian, C. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113518
A Hybrid CNN-BiLSTM Architecture for Emotion Recognition from Audio-Visual Inputs
Kirar, A., Gill, S., Jangra, V., & Sharma, B. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20114-0_37
Gümüş, S. (2026) · IEEE International Conference on Computing and Machine Intelligence (ICMI) · doi:10.1109/icmi68585.2026.11539822
Liu, J., Xie, Y., Zhang, D., & Lee, D.-J. (2026) · Electronics · doi:10.3390/electronics15153279
MD-GLFNet: multi-directional global–local feature fusion network for facial expression recognition
Yin, Z., Li, H., Quan, W., Han, C., & Cai, Y. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01263-8
Metadata
- Authors
- Li, M., Xu, H., Huang, X., Song, Z., Liu, X., & Li, X.
- Affiliations
- Duke Kunshan University (CN)
- Sun Yat-sen University (CN)
- Shandong Yingcai University (CN)
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2021-04-01
- Volume
- 12
- Issue
- 2
- Pages
- 544-550
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W2900164843
- Cited by
- 128
- References
- 47
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 25 · 2023: 25 · 2022: 26 · 2021: 18
- FWCI
- 10.81
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Affective Computing (publishedVersion)
- Funders
- National Natural Science Foundation of China (61773413)
- Natural Science Foundation of Guangzhou City (201707010363)
- Guangzhou Municipal Science and Technology Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- facial expression recognition, emotion recognition, joint learning, interindividual variability, facial shape, facial appearance, deep convolutional neural networks, CK+ database
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.