Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition
Abstract
Temporal information has useful features for recognizing facial expressions. However, to manually design useful features requires a lot of effort. In this paper, to reduce this effort, a deep learning technique, which is regarded as a tool to automatically extract useful features from raw data, is adopted. Our deep network is based on two different models. The first deep network extracts temporal appearance features from image sequences, while the other deep network extracts temporal geometry features from temporal facial landmark points. These two models are combined using a new integration method in order to boost the performance of the facial expression recognition. Through several experiments, we show that the two models cooperate with each other. As a result, we achieve superior performance to other state-of-the-art methods in the CK+ and Oulu-CASIA databases. Furthermore, we show that our new integration method gives more accurate results than traditional methods, such as a weighted summation and a feature concatenation method.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In benchmarking experiments across various datasets, many studies found that features extracted with CNNs lead to higher recognition accuracies than handcrafted features.
“In benchmarking experiments across various datasets, many studies found that features extracted with CNNs lead to higher recognition accuracies than handcrafted features (e.g., [87], [88], [89], [90], [91], [64], [92], [93], [94]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [88]
Data augmentation artificially increases the number of available examples or training epochs by a factor typically between 10 and 30, and up to 300.
“This way, researchers artificially increase the number of available examples or training epochs by a factor typically between 10 and 30 [88], [113], [114], and up to 300 [115].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [88]
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097 · see citing works
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382 · see citing works
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · see citing works
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846 · see citing works
Face detection, pose estimation, and landmark localization in the wild
Zhu, X., & Ramanan, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248014 · Cited by 2,171 · see citing works
Supervised Descent Method and Its Applications to Face Alignment
Xiong, X., & la Torre, F. D. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.75 · Cited by 1,917 · see citing works
A Spatio-Temporal Descriptor Based on 3D-Gradients
Klaeser, A., Marszałek, M., & Schmid, C. (2008) · British Machine Vision Conference (BMVC) · doi:10.5244/c.22.99 · Cited by 1,842 · see citing works
A 3-dimensional sift descriptor and its application to action recognition
Scovanner, P., Ali, S., & Shah, M. A. (2007) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1291233.1291311 · Cited by 1,605 · see citing works
Facial expression recognition from near-infrared videos
Zhao, G., Huang, X., Taini, M., Li, S. Z., & Pietikäinen, M. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.07.002 · Cited by 771 · see citing works
A data-driven approach to cleaning large face datasets
Ng, H., & WINKLER, S. S. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7025068 · Cited by 668 · see citing works
Learning Expressionlets on Spatio-temporal Manifold for Dynamic Facial Expression Recognition
Liu, M., Shan, S., Wang, R., & Chen, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.226 · Cited by 392 · see citing works
Learning active facial patches for expression analysis
Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J., & Metaxas, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247974 · Cited by 380 · see citing works
Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis
Liu, M., Li, S., Shan, S., Wang, R., & Chen, X. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16817-3_10 · Cited by 332 · see citing works
Ghimire, D., & Lee, J. (2013) · Sensors · doi:10.3390/s130607714 · Cited by 295 · see citing works
Dense 3D face alignment from 2D videos in real-time
Jeni, L. A., Cohn, J. F., & Takeo Kanade, A. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163142 · Cited by 179 · see citing works
Towards a practical lipreading system
Zhou, Z., Zhao, G., & Pietikäinen, M. (2011) · doi:10.1109/cvpr.2011.5995345 · Cited by 161 · see citing works
Capturing Complex Spatio-temporal Relations among Facial Muscles for Facial Expression Recognition
Wang, Z., Wang, S., & Ji, Q. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.439 · Cited by 154 · see citing works
Spatio-temporal covariance descriptors for action and gesture recognition
Sanin, A., Sanderson, C., Harandi, M. T., & Lovell, B. C. (2013) · Proceedings · doi:10.1109/wacv.2013.6475006 · Cited by 154 · see citing works
Facial expression recognition with temporal modeling of shapes
Jain, S. D., Hu, C., & Aggarwal, J. K. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130446 · Cited by 140 · see citing works
Spatio-temporal covariance descriptors for action and gesture recognition
Sanin, A., Sanderson, C., Harandi, M. T., & Lovell, B. C. (2013) · Cited by 123 · see citing works
Dynamic Facial Expression Recognition Using Longitudinal Facial Expression Atlases
Guo, Y., Zhao, G., & Pietikäinen, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33709-3_45 · Cited by 96 · see citing works
Exploring Bag of Words Architectures in the Facial Expression Domain
Sikka, K., Wu, T., Susskind, J., & Bartlett, M. S. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33868-7_25 · Cited by 94 · see citing works
Facial Expression Recognition in Perceptual Color Space
Lajevardi, S. M., & Wu, H. R. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2197628 · Cited by 91 · see citing works
Automatic Facial Expression Recognition System Based on Geometric and Appearance Features
Youssif, A. A. A., & Asker, W. A. A. (2011) · Computer and Information Science · doi:10.5539/cis.v4n2p115 · Cited by 55 · see citing works
Manifold based Sparse Representation for robust expression recognition without neutral subtraction
Ptucha, R., Tsagkatakis, G., & Savakis, A. E. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130512 · Cited by 48 · see citing works
Spatio-temporal Event Classification Using Time-Series Kernel Based Structured Sparsity
Jeni, L. A., Lörincz, A., Szabó, Z., Cohn, J. F., & Takeo Kanade, A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10593-2_10 · Cited by 30 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 2,043 · see citing works
“In static-based methods [12], [13], [14], the feature representation is encoded with only spatial information from the current single image, whereas dynamicbased methods [15], [16], [17] consider the temporal relation among contiguous frames in the input facial expression sequence.”
MethodInfluentialRegion Attention Networks for Pose and Occlusion Robust Facial Expression Recognition
Wang, K., Peng, X., Yang, J., Meng, D., & Qiao, Y. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2956143 · Cited by 957 · see citing works
“Due to its potential applications in various fields, such as intelligent tutoring systems, service robots, driver fatigue monitoring, Facial Expression Recognition (FER) has attracted increasing attention in the computer vision community recently [28], [5],”
BackgroundA Brief Review of Facial Emotion Recognition Based on Visual Information
Ko, B. C. (2018) · Sensors · doi:10.3390/s18020401 · Cited by 747 · see citing works
“• Joint fine-tuning in deep neural networks [48] Sequence 70.”
BackgroundMethodInfluentialLi, S., & Deng, W. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2868382 · Cited by 670 · see citing works
“In DTAGN [30], two different models were combined to extract temporal appearance and geometry features simultaneously.”
MethodA systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 657 · see citing works
Spatial–Temporal Recurrent Neural Network for Emotion Recognition
Zhang, T., Zheng, W., Cui, Z., Zong, Y., & Li, Y. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2788081 · Cited by 595 · see citing works
Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks
Zhang, K., Huang, Y., Du, Y., & Wang, L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2689999 · Cited by 486 · see citing works
“of previous state-of-the-art methods [7], [6], [24] on this database is lower than the results on CK+.”
BackgroundMethodResultInfluentialFacial Expression Recognition by De-expression Residue Learning
Yang, H., Çiftçi, U. A., & Yin, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00231 · Cited by 461 · see citing works
“Our DeRL method achieves the highest accuracy compared to those of state-of-the-art methods, including CNN-based methods (DTAGN-Joint [12], FN2EN [6], and PPDN [35]) and handcrafted features based methods (LBP-TOP [34], HOG 3D [14] and STM-Explet [18]).”
BackgroundMethodFacial emotion recognition using convolutional neural networks (FERC)
Mehendale, N. D. (2020) · SN Applied Sciences · doi:10.1007/s42452-020-2234-1 · Cited by 428 · see citing works
“Jung et al. [38] tried to work with fixed frames which make the system not so efficient with video input.”
BackgroundDeep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
Kollias, D., Tzirakis, P., Nicolaou, M. A., Papaioannou, A., Zhao, G., Schuller, B. W., Kotsia, I., & Zafeiriou, S. P. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01158-4 · Cited by 388 · see citing works
“still used in many recent works [27], are the Cohn-Kanade database [34,54], MMI database [40,56], Multi-PIE database [22] and the BU-3D and BU-4D databases [61, 62].”
MethodFacial Expression Recognition with Inconsistently Annotated Datasets
Zeng, J., Shan, S., & Chen, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_14 · Cited by 354 · see citing works
“Based on these datasets, lots of FER approaches are proposed and achieve the state-of-the-art performance[27, 25, 30, 4, 14, 34, 18].”
BackgroundIsland Loss for Learning Discriminative Features in Facial Expression Recognition
Cai, J., Meng, Z., Khan, A. S., Li, Z., O’Reilly, J. D., & Tong, Y. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00051 · Cited by 337 · see citing works
“Benefiting from the advance in feature learning, features can be learned either unsupervised by sparse coding [25], [59], [35], [29] or supervised by deep learning [36], [44], [24], [5], [8], [30], [21], [15], [22], [58], [9], [51], [3], [49],”
BackgroundMethodInfluentialIdentity-Aware Convolutional Neural Network for Facial Expression Recognition
Meng, Z., Liu, P., Cai, J., Han, S., & Tong, Y. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.140 · Cited by 335 · see citing works
“deep learning [34], [42], [23], [29], [20], [16], [49], [30], [13], [21], [53].”
MethodPeak-Piloted Deep Network for Facial Expression Recognition
Zhao, X., Liang, X., Liu, L., Li, T., Han, Y., Vasconcelos, N. M., & Yan, S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_27 · Cited by 317 · see citing works
“59% DTAGN(Joint) [7] sequence-based 81.”
BackgroundInfluentialA Deep Neural Network-Driven Feature Learning Method for Multi-view Facial Expression Recognition
Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., & Yan, K. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2598092 · Cited by 314 · see citing works
“Table II shows the accuracies of the comparison between 2D JFDNN transformed from [18] and our neural network corresponding to the seven yaw head poses (0◦, 15◦, 30◦, 45◦, 60◦, 75◦ and 90◦).”
BackgroundMethodInfluentialKim, J., Kim, B., Roy, P. P., & Jeong, D.-M. (2019) · IEEE Access · doi:10.1109/access.2019.2907327 · Cited by 309 · see citing works
Self-citationAff-Wild: Valence and Arousal ‘In-the-Wild’ Challenge
Zafeiriou, S. P., Kollias, D., Nicolaou, M. A., Papaioannou, A., Zhao, G., & Kotsia, I. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.248 · Cited by 297 · see citing works
“Some representative datasets, which are still used in many recent works [18], include the Cohn-Kanade database [33, 24], the MMI database [29, 35], the Multi-PIE database [17] and the BU-3D and BU-4D databases [41, 40].”
MethodFrom Facial Expression Recognition to Interpersonal Relation Prediction
Zhang, Z., Luo, P., Loy, C. C., & Tang, X. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1055-1 · Cited by 305 · see citing works
“Social signal processing mainly involves facial expression recognition (Zhao and Pietikainen 2007; Tian et al. 2011; Liu et al. 2014b; Ruiz et al. 2015; Wu and Ji 2016; Fabian Benitez-Quiroz et al. 2016; Liu et al. 2014a, c; Jung et al. 2015;Mollahosseini et al. 2016; Fabian Benitez-Quiroz et al. 2016; Zhao et al. 201…”
BackgroundJoint Pose and Expression Modeling for Facial Expression Recognition
Zhang, F., Zhang, T., Mao, Q., & Xu, C. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00354 · Cited by 266 · see citing works
“, Support Vector Machines (SVMs) [16], softmax [18], and logistic regression [36], to train a facial expression recognizer for a target expression.”
BackgroundKim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2695999 · Cited by 257 · see citing works
“In [32], a temporal geometric feature was jointly learned in order to reduce the effect of the identity on the learned spatio-temporal appearance features.”
BackgroundMethodInfluentialZhang, S., Yang, Y., Chen, C., Zhang, X., Leng, Q., & Zhao, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121692 · Cited by 273 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Jiang, Y., Li, W., Hossain, M. S., Chen, M., Alelaiwi, A. A., & Al-Hammadi, M. H. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.06.019 · Cited by 219 · see citing works
Xie, S., & Hu, H. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2844085 · Cited by 216 · see citing works
“In this paper, we compare the recognition accuracy of our model with several competitive approaches, including conventional methods (LAP [44], MPSD [45] and MCSPL [30]) and deep-based methods (DTAN [37], DTGN [37] and 3DCNN-DAP [46]).”
BackgroundMethodInfluentialFrame Attention Networks for Facial Expression Recognition in Videos
Meng, D., Peng, X., Wang, K., & Qiao, Y. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803603 · Cited by 216 · see citing works
“ST network [23] S: the last frame T: all frames S: the last frame T: all frames 98.50 DTAGN [12] Fixed length Fixed length 97.25 CNN+Island loss[24] The last three frames and the first frame The last three frames and the first frame”
BackgroundMethodInfluentialAdaptive Deep Metric Learning for Identity-Aware Facial Expression Recognition
Liu, X., Kumar, B. V. K. V., You, J., & Jia, P. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.79 · Cited by 209 · see citing works
“12% DTAGN [16] 97.”
Xie, S., Hu, H., & Wu, Y. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.03.019 · Cited by 207 · see citing works
Huang, X., Wang, S., Liu, X., Zhao, G., Feng, X., & Pietikäinen, M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2713359 · Cited by 195 · see citing works
“sion recognition [18], [19], [20], [21], [22].”
Karnati, M., Seal, A., Krejcar, O., & Yazidi, A. (2020) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2020.3031835 · Cited by 196 · see citing works
“In recent years, it is observed from the literature that deep-learning-based methods are superior to handcrafted features-based methods for FER tasks [17], [18], [43], [44].”
BackgroundFacial Expression Recognition: A Survey
Huang, Y., Chen, F., Lv, S., & Wang, X. (2019) · Symmetry · doi:10.3390/sym11101189 · Cited by 195 · see citing works
“In order to take advantage of temporal features for recognising facial expression, a new integration method named Deep Temporal Appearance-Geometry Network (DTAGN) is proposed in [86].”
MethodAU-assisted Graph Attention Convolutional Network for Micro-Expression Recognition
Xie, H., Lo, L., Shuai, H.-H., & Cheng, W.-H. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3414012 · Cited by 191 · see citing works
“, cropping [45], flipping [40], rotating [15, 44].”
Three convolutional neural network models for facial expression recognition in the wild
Shao, J., & Qian, Y. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.05.005 · Cited by 192 · see citing works
Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network
Minaee, S., Minaei, M., & Abdolrashidi, A. (2021) · Sensors · doi:10.3390/s21093046 · Cited by 232 · see citing works
Automatic depression recognition using CNN with attention mechanism from videos
He, L., Chan, J. C.-W., & Wang, Z. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.015 · Cited by 179 · see citing works
Automated emotion recognition: Current trends and future perspectives
Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Barua, P. D., Murugappan, M., Chakole, Y., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106646 · Cited by 172 · see citing works
A Compact Deep Learning Model for Robust Facial Expression Recognition
Kuo, C.-M., Lai, S., & Sarkis, M. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00286 · Cited by 170 · see citing works
“The joint-fine tuning method [16] adopted data augmentation strategy with 7 different rotation angles to obtain 14 times more data.”
MethodInfluentialMicro-expression recognition based on 3D flow convolutional neural network
Li, J., Wang, Y., See, J., & Liu, W. (2018) · Pattern Analysis and Applications · doi:10.1007/s10044-018-0757-5 · Cited by 173 · see citing works
“Subsequent works by [15] and [13] proposed models that jointly learn from two types of data, i.”
BackgroundAnalysing Affective Behavior in the second ABAW2 Competition
Kollias, D., & Zafeiriou, S. P. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00408 · Cited by 168 · see citing works
POSTER: A Pyramid Cross-Fusion Transformer Network for Facial Expression Recognition
Zheng, C., Mendieta, M., & Chen, C. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00339 · Cited by 170 · see citing works
“[16] proposed two networks, where one receives the image as input and another receives facial landmark as input.”
BackgroundMethodInfluentialKhan, A. R. (2022) · Information · doi:10.3390/info13060268 · Cited by 179 · see citing works
Spatio-temporal convolutional features with nested LSTM for facial expression recognition
Yu, Z., Liu, G., Liu, Q., & Deng, J. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.07.028 · Cited by 162 · see citing works
Web-based fully automated cephalometric analysis by deep learning
Kim, H. H., Shim, E., Park, J., Kim, Y., Lee, U., & Kim, Y. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105513 · Cited by 161 · see citing works
Learning Affective Video Features for Facial Expression Recognition via Hybrid Deep Learning
Zhang, S., Pan, X., Cui, Y., Zhao, X., & Liu, L. (2019) · IEEE Access · doi:10.1109/access.2019.2901521 · Cited by 151 · see citing works
“[10] propose a small temporal CNN to extract temporal geometric features from facial landmark points.”
BackgroundModeling Stylized Character Expressions via Deep Learning
Aneja, D., Colburn, A., Faigin, G., Shapiro, L. G., & Mones, B. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-54184-6_9 · Cited by 149 · see citing works
“Joint Fine-Tuning in Deep Neural Networks [33] have also been used to combine temporal appearance features from image sequences and temporal geometry features from temporal facial landmark points to enhance the performance of the facial expression recognition.”
MethodDetection of Cattle Using Drones and Convolutional Neural Networks
Rivas, A., Chamoso, P., González‐Briones, A., & Corchado, J. M. (2018) · Sensors · doi:10.3390/s18072048 · Cited by 141 · see citing works
“For this purpose, softmax [45] is the selected activation function, which is defined in Equation (1).”
BackgroundAutomatic Recognition of Student Engagement Using Deep Learning and Facial Expression
Nezami, O. M., Dras, M., Hamey, L., Richards, D., Wan, S., & Paris, C. L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-46133-1_17 · Cited by 138 · see citing works
“Deep learning models have been successful in automatically recognizing facial expressions in images [15,23,24,30,37,38,39].”
BackgroundResNet-Based Vehicle Classification and Localization in Traffic Surveillance Systems
Jung, H., Choi, M.-K., Jung, J., Lee, J.-H., Kwon, S., & Jung, W. Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.129 · Cited by 137 · see citing works
Self-citationAdvances in Facial Expression Recognition: A Survey of Methods, Benchmarks, Models, and Datasets
Kopalidis, T., Solachidis, V., Vretos, N., & Daras, P. (2024) · Information · doi:10.3390/info15030135 · Cited by 174 · see citing works
Can Micro-Expression be Recognized Based on Single Apex Frame?
Li, Y., Huang, X., & Zhao, G. (2018) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2018.8451376 · Cited by 132 · see citing works
“Deep learning has been shown efficient on facial expression recognition [14], we present to use deep learning to recognize ME.”
MethodLi, S., & Deng, W. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1131-1 · Cited by 132 · see citing works
Geometry Guided Pose-Invariant Facial Expression Recognition
Zhang, F., Zhang, T., Mao, Q., & Xu, C. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2972114 · Cited by 131 · see citing works
“[57], [58] and learning-based features [8], [9], [59], [60].”
BackgroundFacial Expression Recognition with Identity and Emotion Joint Learning
Li, M., Xu, H., Huang, X., Song, Z., Liu, X., & Li, X. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2880201 · Cited by 128 · see citing works
“[1], we use feature-level concatenation of deeplearned identity and emotion features to form a new network with”
BackgroundMethodInfluentialRecognition of Affect in the Wild Using Deep Neural Networks
Kollias, D., Nicolaou, M. A., Kotsia, I., Zhao, G., & Zafeiriou, S. P. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.247 · Cited by 124 · see citing works
“In general, the procedure of freezing a part of the network and fine-tuning [11] the rest can be deemed very useful, in particular when the given dataset is incremented with more videos.”
BackgroundDeep spatial-temporal feature fusion for facial expression recognition in static images
Sun, N., Li, Q., Huan, R., Liu, J., & Han, G. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.10.022 · Cited by 121 · see citing works
Facial Expression Recognition in Videos Using Dynamic Kernels
Perveen, N., Roy, D., & Mohan, C. K. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3011846 · Cited by 116 · see citing works
“networks [24], recurrent neural network [13], etc are explored in past few decades for efficient facial expression recognition”
BackgroundMukhiddinov, M., Djuraev, O., Akhmedov, F., Mukhamadiyev, A., & Cho, J. (2023) · Sensors · doi:10.3390/s23031080 · Cited by 121 · see citing works
“Other studies [40,41] have introduced a variety of landmark and image fusion processes.”
Yang, H., Zhang, Z., & Yin, L. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00050 · Cited by 111 · see citing works
“DTAGN-Joint [29] sequence-based 81.”
Supervised Committee of Convolutional Neural Networks in Automated Facial Expression Analysis
Pons, G., & Masip, D. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2753235 · Cited by 109 · see citing works
“In [31],”
Transformer-based Multimodal Information Fusion for Facial Expression Analysis
Zhang, W., Qiu, F., Wang, S., Zeng, H., Zhang, Z., An, R., Ma, B., & Ding, Y. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00271 · Cited by 109 · see citing works
“Some works [3, 15] employ CNN to extract the spatial and temporal features and improve expression recognition.”
MethodFacial Expression Recognition via Deep Learning
Fathallah, A., Abdi, L., & Douik, A. (2017) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa.2017.124 · Cited by 108 · see citing works
“Performance comparison on the CK+ database, A1[20] , A2 [19], A3 [21].”
MethodLai, Y.-H., & Lai, S. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00046 · Cited by 103 · see citing works
“However, the proposed model still can compete with another deep learning based approach [7] that utilized image feature and landmarks to jointly train a DNN discriminator.”
BackgroundMethodInfluentialFacial Motion Prior Networks for Facial Expression Recognition
Chen, Y., Wang, J., Chen, S., Shi, Z., & Cai, J. (2019) · IEEE International Conference on Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip47243.2019.8965826 · Cited by 107 · see citing works
“facial landmarks [3], or optical flow [4].”
CogEmoNet: A Cognitive-Feature-Augmented Driver Emotion Recognition Model for Smart Cockpit
Li, W., Zeng, G., Zhang, J., Xu, Y. H., Xing, Y., Zhou, R., Guo, G., Shen, Y., Cao, D., & Wang, F. (2021) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2021.3127935 · Cited by 104 · see citing works
“[33] used a dual-stream network to extract temporal appearance features and temporal geometry features to improve facial expression recognition ability.”
Aff-Wild2: Extending the Aff-Wild Database for Affect RecognitionPreprint
Kollias, D., & Zafeiriou, S. P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.07770 · Cited by 142 · see citing works
“Some representative datasets developed in labs, which are still used in many recent works [12], are the Cohn-Kanade database [43], [29], the MMI database”
MethodDeep Joint Spatiotemporal Network (DJSTN) for Efficient Facial Expression Recognition
Jeong, D.-M., Kim, B., & Dong, S.-Y. (2020) · Sensors · doi:10.3390/s20071936 · Cited by 102 · see citing works
“[33], who used convolution and pooling layer, respectively.”
BackgroundMethodInfluentialOAENet: Oriented attention ensemble for accurate facial expression recognition
Wang, Z., Zeng, F., Liu, S., & Zeng, B. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107694 · Cited by 104 · see citing works
Deep convolutional BiLSTM fusion network for facial expression recognition
Liang, D., Liang, H., Yu, Z., & Zhang, Y. (2019) · The Visual Computer · doi:10.1007/s00371-019-01636-3 · Cited by 103 · see citing works
“The recent focus has been more on algorithms capturing spatial–temporal features [5, 8, 13, 14, 20, 21].”
BackgroundMethodInfluentialAdaptive Feature Mapping for Customizing Deep Learning Based Facial Expression Recognition Model
Wu, B., & Lin, C.-H. (2018) · IEEE Access · doi:10.1109/access.2018.2805861 · Cited by 99 · see citing works
“[17], and Byeon and Kwak [18] predict the expressions by passing a video, which seemsmore reasonable in practice; nevertheless, labeling video data is more labor intensive.”
BackgroundMulti-cue fusion for emotion recognition in the wild
Yan, J., Zheng, W., Cui, Z., Tang, C., Zhang, T., & Zong, Y. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.03.068 · Cited by 98 · see citing works
Facial expression recognition in videos using hybrid CNN & ConvLSTM
Singh, R., Saurav, S., Kumar, T., Saini, R., Vohra, A., & Singh, S. K. (2023) · International Journal of Information Technology · doi:10.1007/s41870-023-01183-0 · Cited by 105 · see citing works
Hard negative generation for identity-disentangled facial expression recognition
Liu, X., Kumar, B. V. K. V., Jia, P., & You, J. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.11.001 · Cited by 95 · see citing works
A comprehensive review of facial expression recognition techniques
Rashmi Adyapady, R., & Annappa, B. (2022) · Multimedia Systems · doi:10.1007/s00530-022-00984-w · Cited by 98 · see citing works
Incremental Boosting Convolutional Neural Network for Facial Action Unit RecognitionPreprint
Han, S., Meng, Z., Khan, A.-S., & Tong, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05395 · Cited by 97 · see citing works
“Specifically, a set of IB-3 CNNs, with the number of input neurons of 64, 128, 256, 512, 1042, and 2048, were trained and tested on the SEMAINE database.”
BackgroundInfluentialFacial expression recognition with FRR‐CNN
Xie, S., & Hu, H. (2017) · Electronics Letters · doi:10.1049/el.2016.4328 · Cited by 95 · see citing works
Exploring Deep Physiological Models for Nociceptive Pain Recognition
Thiam, P., Bellmann, P., Kestler, H. A., & Schwenker, F. (2019) · Sensors · doi:10.3390/s19204503 · Cited by 94 · see citing works
“…(in order to include the temporal aspect of the physiological signals in the inference model), could also potentially improve the performance of the current system, since such approaches have been successfully applied in other domains of application such as facial expression recognition [51–53].”
Conditional convolution neural network enhanced random forest for facial expression recognition
Liu, Y., Yuan, X., Gong, X., Xie, Z., Fang, F., & Luo, Z. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.07.016 · Cited by 94 · see citing works
k-Same-Net: k-Anonymity with Generative Deep Neural Networks for Face Deidentification
Meden, B., Emeršič, Ž., Štruc, V., & Peer, P. (2018) · Entropy · doi:10.3390/e20010060 · Cited by 90 · see citing works
Multimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification
Lee, S., Han, D. K., & Ko, H. (2021) · IEEE Access · doi:10.1109/access.2021.3092735 · Cited by 91 · see citing works
“[28]–[30] built an emotion recognition model by fusing handcrafted features and deep learning based features from facial images.”
BackgroundSelf-citationEfficient facial expression recognition using histogram of oriented gradients in wavelet domain
Nigam, S., Singh, R. K., & Misra, A. K. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-6040-3 · Cited by 91 · see citing works
“Recognition accuracy of the proposed method is shown 3DCNN-DAP [32] DTDN [26] DTGN [26] DTAGN (WS) [26] DTAGN (Joint) [26] DNN [25] CNN [25] TOP-1 [58] TOP-2 [58] CNN [43] Exp-Net [43] IACNN [43] AUDN [31] BDBN [34] FP+SAE [41] CNN [29] Proposed in the form of confusion matrices.”
MethodInfluentialZhang, H., Huang, B., & Tian, G. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.12.013 · Cited by 88 · see citing works
Facial Expression Recognition With Deeply-Supervised Attention Network
Fan, Y., Li, V. O. K., & Lam, J. C. K. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2988264 · Cited by 88 · see citing works
Facial Expression Recognition Using Frequency Neural Network
Tang, Y. D., Zhang, X., Hu, X., Wang, S., & Wang, H. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3037467 · Cited by 89 · see citing works
“For another example, some researches [6], [14] combined geometric feature and learned feature.”
BackgroundMethodInfluentialJi, Y., Hu, Y., Yang, Y., Shen, F., & Shen, H. T. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.12.037 · Cited by 86 · see citing works
Multiple Attention Network for Facial Expression Recognition
Gan, Y., Chen, J., Yang, Z., & Xu, L. (2020) · IEEE Access · doi:10.1109/access.2020.2963913 · Cited by 85 · see citing works
“[18] presented a joint fine-tuning strategy with”
Chang, W.-Y., Hsu, S.-H., & Chien, J.-H. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.246 · Cited by 84 · see citing works
“Moreover, CNNs also have been applied to face-related tasks [22, 29, 41] and facial expression recognition [17, 9].”
MethodFilntisis, P. P., Efthymiou, N., Koutras, P., Potamianos, G., & Maragos, P. A. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2930434 · Cited by 84 · see citing works
“However, to date, most research has focused on automatic visual recognition of emotion from facial expressions [15], [16], with only few including emotional body expressions into the recognition loop [8].”
BackgroundSAANet: Siamese action-units attention network for improving dynamic facial expression recognition
Liu, D., Ouyang, X., Xu, S., Zhou, P., He, K., & Wen, S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.06.062 · Cited by 82 · see citing works
Clip-aware expressive feature learning for video-based facial expression recognition
Liu, Y., Feng, C., Yuan, X., Zhou, L., Wang, W., Qin, J., & Luo, Z. (2022) · Information Sciences · doi:10.1016/j.ins.2022.03.062 · Cited by 83 · see citing works
“In [21], a temporal geometric feature was proposed to improve the discriminative capacity of the learned spatio-temporal appearance features.”
BackgroundMethodFacial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM
An, F., & Liu, Z. (2019) · The Visual Computer · doi:10.1007/s00371-019-01635-4 · Cited by 81 · see citing works
Co-attentive multi-task convolutional neural network for facial expression recognition
Yu, W., & Xu, H. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108401 · Cited by 82 · see citing works
Chen, Y., Li, J., Shan, S., Wang, M., & Hong, R. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3453443 · Cited by 89 · see citing works
“Performance comparison of dynamic facial expression recognition on the DFEW [4] testing set.”
Deeper cascaded peak-piloted network for weak expression recognition
Yu, Z., Liu, Q., & Liu, G. (2017) · The Visual Computer · doi:10.1007/s00371-017-1443-0 · Cited by 79 · see citing works
“Deep learning algorithms have shown excellent performance of facial expression recognition in latest significant conferences [10,14,15,36] and competitions [2,7,8,30].”
BackgroundMethodInfluentialBrain-Machine Coupled Learning Method for Facial Emotion Recognition
Liu, D., Dai, W., Zhang, H., Jin, X., Cao, J., & Kong, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3257846 · Cited by 81 · see citing works
“As shown in Table XI, compared with the state-of-the-art methods [94], [95], [96], [97], [98] on the CK+ dataset, the proposed method achieves competitive”
ResultLearning Bases of Activity for Facial Expression Recognition
Sarıyanidi, E., Güneş, H., & Cavallaro, A. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2662237 · Cited by 78 · see citing works
Facial expression recognition based on a multi-task global-local network
Yu, M., Zheng, H., Peng, Z., Dong, J., & Du, H. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.01.016 · Cited by 78 · see citing works
Deep Disturbance-Disentangled Learning for Facial Expression Recognition
Ruan, D., Yan, Y., Chen, S., Xue, J., & Wang, H. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413907 · Cited by 77 · see citing works
Facial Micro-Expression Recognition Using Two-Dimensional Landmark Feature Maps
Choi, D. Y., & Song, B. C. (2020) · IEEE Access · doi:10.1109/access.2020.3006958 · Cited by 80 · see citing works
“Also, CLFM shows about 15% higher performance than DTAGN even for mixed dataset.”
BackgroundMethodInfluentialFacial Expression Recognition With Two-Branch Disentangled Generative Adversarial Network
Xie, S., Hu, H., & Chen, Y. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3024201 · Cited by 76 · see citing works
“Note that for CK+, DTAGN [3] and DeRF [4] both extract temporal expressional information from”
BackgroundMethodInfluentialFacial Expression Recognition of Instructor Using Deep Features and Extreme Learning Machine
Bhatti, Y. K., Jamil, A., Nida, N., Yousaf, M. H., Viriri, S., & Velastín, S. A. (2021) · Computational Intelligence and Neuroscience · doi:10.1155/2021/5570870 · Cited by 78 · see citing works
Facial Landmark-Based Emotion Recognition via Directed Graph Neural Network
Ngoc, Q. T., Lee, S., & Song, B. C. (2020) · Electronics · doi:10.3390/electronics9050764 · Cited by 73 · see citing works
“For example, a study [5] demonstrated performance improvement by associating the landmark modality with other modalities.”
BackgroundMethodResultInfluentialSelf-citation
- Authors
- Jung, H., Lee, S., Yim, J., Park, S., & Kim, J.
- Affiliations
- Korea Advanced Institute of Science and Technology (KR)
- Venue
- 2015 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2015-12
- Pages
- 2983-2991
- OpenAlex
- W2217426128
- Cited by
- 781
- References
- 27
- Citations by year
- 2026: 9 · 2025: 29 · 2024: 45 · 2023: 82 · 2022: 87 · 2021: 127 · 2020: 115 · 2019: 125 · 2018: 92 · 2017: 55 · 2016: 15
- FWCI
- 72.14
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- KAIST Institutional Repository (KAIST) (submittedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- facial expression recognition, joint fine-tuning, facial landmarks, feature integration, image sequences, deep neural network, CK+ database
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
