FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition
Open accessAbstract
Relatively small data sets available for expression recognition research make the training of deep networks very challenging. Although fine-tuning can partially alleviate the issue, the performance is still below acceptable levels as the deep features probably contain redundant information from the pretrained domain. In this paper, we present FaceNet2ExpNet, a novel idea to train an expression recognition network based on static images. We first propose a new distribution function to model the high-level neurons of the expression network. Based on this, a two-stage training algorithm is carefully designed. In the pre-training stage, we train the convolutional layers of the expression net, regularized by the face net; In the refining stage, we append fully-connected layers to the pre-trained convolutional layers and train the whole network jointly. Visualization results show that the model trained with our method captures improved high-level expression semantics. Evaluations on four public expression databases, CK+, Oulu- CASIA, TFD, and SFEW demonstrate that our method achieves better results than state-of-the-art.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
(2014) · Cited by 33,419 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479 · see citing works
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · see citing works
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335 · see citing works
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714 · see citing works
(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534 · see citing works
A Spatio-Temporal Descriptor Based on 3D-Gradients
Klaeser, A., Marszałek, M., & Schmid, C. (2008) · British Machine Vision Conference (BMVC) · doi:10.5244/c.22.99 · Cited by 1,842 · see citing works
Challenges in Representation Learning: A Report on Three Machine Learning Contests
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-42051-1_16 · Cited by 1,719 · see citing works
Learning Face Representation from ScratchPreprint
Yi, D., Lei, Z., Liao, S., & Li, S. Z. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.7923 · Cited by 1,629 · see citing works
Going deeper in facial expression recognition using deep neural networks
Mollahosseini, A., Chan, D., & Mahoor, M. H. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477450 · Cited by 1,117 · see citing works
The MegaFace Benchmark: 1 Million Faces for Recognition at Scale
Kemelmacher-Shlizerman, I., Seitz, S. M., Miller, D., & Brossard, E. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.527 · Cited by 912 · see citing works
A large-scale car dataset for fine-grained categorization and verification
Yang, L., Luo, P., Loy, C. C., & Tang, X. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299023 · Cited by 867 · see citing works
Facial expression recognition from near-infrared videos
Zhao, G., Huang, X., Taini, M., Li, S. Z., & Pietikäinen, M. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.07.002 · Cited by 771 · see citing works
Training Deep Neural Networks on Noisy Labels with BootstrappingPreprint
Reed, S., Lee, H., Anguelov, D., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6596 · Cited by 762 · see citing works
Facial Expression Recognition via a Boosted Deep Belief Network
Liu, P., Han, S., Meng, Z., & Tong, Y. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.233 · Cited by 666 · see citing works
Return of the Devil in the Details: Delving Deep into Convolutional NetsPreprint
Chatfield, K., Simonyan, K., Vedaldi, A., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.3531 · Cited by 663 · see citing works
Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning
Ng, H.-W., Nguyen, V. D., Vonikakis, V., & Winkler, S. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830593 · Cited by 657 · cited 3× · see citing works
“Compared with networks adopted in previous works [9], [23], [25], AlexNet [47] or VGG-M [48], the size of our network is fairly small.”
MethodResultRecognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior
Bartlett, M. S., Littlewort, G. C., Frank, M. G., Lainscsek, C. S. M., Fasel, I., & Movellan, J. R. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.297 · Cited by 632 · see citing works
Image based Static Facial Expression Recognition with Multiple Deep Network Learning
Yu, Z., & Zhang, C. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2830595 · Cited by 602 · see citing works
Learning Expressionlets on Spatio-temporal Manifold for Dynamic Facial Expression Recognition
Liu, M., Shan, S., Wang, R., & Chen, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.226 · Cited by 392 · see citing works
Learning active facial patches for expression analysis
Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J., & Metaxas, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247974 · Cited by 380 · see citing works
Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis
Liu, M., Li, S., Shan, S., Wang, R., & Chen, X. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16817-3_10 · Cited by 336 · see citing works
Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns
Levi, G., & Hassner, T. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830587 · Cited by 336 · see citing works
Peak-Piloted Deep Network for Facial Expression Recognition
Zhao, X., Liang, X., Liu, L., Li, T., Han, Y., Vasconcelos, N. M., & Yan, S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_27 · Cited by 319 · see citing works
Video and Image based Emotion Recognition Challenges in the Wild: EmotiW 2015
Dhall, A., Ramana Murthy, O. V., Goecke, R., Joshi, J., & Gedeon, T. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2829994 · Cited by 305 · see citing works
Kim, B.-K., Roh, J., Dong, S.-Y., & Lee, S.-Y. (2016) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0209-0 · Cited by 244 · see citing works
On deep generative models with applications to recognition
Ranzato, M., Susskind, J., Mnih, V., & Hinton, G. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995710 · Cited by 237 · see citing works
EMPATH: A Neural Network that Categorizes Facial Expressions
Dailey, M. N., Cottrell, G. W., Padgett, C. W., & Adolphs, R. (2002) · Journal of Cognitive Neuroscience · doi:10.1162/089892902760807177 · Cited by 235 · see citing works
AU-aware Deep Networks for facial expression recognition
Liu, M., Li, S., Shan, S., & Chen, X. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553734 · Cited by 229 · see citing works
Disentangling Factors of Variation for Facial Expression Recognition
Rifai, S., Bengio, Y., Courville, A., Vincent, P., & Mirza, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_58 · Cited by 215 · see citing works
Learning to Disentangle Factors of Variation with Manifold Interaction
Reed, S., Sohn, K., Zhang, Y., & Lee, H. (2014) · Cited by 201 · see citing works
la Torre, F. D., Chu, W.-S., Xiong, X., Vicente, F., Ding, X., & Cohn, J. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163082 · Cited by 125 · see citing works
Facial expression recognition based on local binary patterns
Feng, X., Pietikäinen, M., & Hadid, A. (2007) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661807040190 · Cited by 101 · see citing works
Dynamic Facial Expression Recognition Using Longitudinal Facial Expression Atlases
Guo, Y., Zhao, G., & Pietikäinen, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33709-3_45 · Cited by 96 · see citing works
LOMo: Latent Ordinal Model for Facial Analysis in Videos
Sikka, K., Sharma, G., & Bartlett, M. S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.602 · Cited by 94 · see citing works
Liu, P., Zhou, J. T., Tsang, I. W.-H., Meng, Z., Han, S., & Tong, Y. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10593-2_11 · Cited by 55 · see citing works
Automatic Attribute Discovery with Neural Activations
Vittayakorn, S., Umeda, T., Murasaki, K., Sudo, K., Okatani, T., & Yamaguchi, K. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46493-0_16 · Cited by 52 · see citing works
InterActive: Inter-Layer Activeness Propagation
Xie, L., Zheng, L., Wang, J., Yuille, A., & Tian, Q. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.36 · Cited by 47 · see citing works
Deep Temporal Appearance-Geometry Network for Facial Expression RecognitionPreprint
Jung, H., Lee, S., Park, S., Lee, I., Ahn, C., & Kim, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.01532 · Cited by 43 · see citing works
Peak-Piloted Deep Network for Facial Expression RecognitionPreprint
Zhao, X., Liang, X., Liu, L., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.06997 · Cited by 23 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Spatio-temporal convolutional features with nested LSTM for facial expression recognition
Yu, Z., Liu, G., Liu, Q., & Deng, J. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.07.028 · Cited by 162 · see citing works
Facial Sentiment Analysis Using AI Techniques: State-of-the-Art, Taxonomies, and Challenges
Patel, K., Mehta, D., Mistry, C., Gupta, R., Tanwar, S., Kumar, N., & Alazab, M. (2020) · IEEE Access · doi:10.1109/access.2020.2993803 · Cited by 157 · see citing works
- Authors
- Ding, H., Zhou, S. K., & Chellappa, R.
- Affiliations
- University of Maryland, College Park (US)
- Siemens (United States) (US)
- Venue
- 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 118-126
- ISSN
- 1541-5058 (ISSN-L)
- OpenAlex
- W2963112684
- Cited by
- 416
- References
- 50
- Citations by year
- 2026: 6 · 2025: 14 · 2024: 27 · 2023: 58 · 2022: 50 · 2021: 73 · 2020: 60 · 2019: 70 · 2018: 41 · 2017: 17
- FWCI
- 59.04
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference on Automatic Face and Gesture Recognition (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Office of the Director of National Intelligence (2014-14071600012)
- Intelligence Advanced Research Projects Activity (2014-14071600012)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- facial expression recognition, face recognition, transfer learning, fine-tuning, distribution function, two-stage training, convolutional layers, limited training data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
