Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning
Abstract
This paper presents the techniques employed in our team's submissions to the 2015 Emotion Recognition in the Wild contest, for the sub-challenge of Static Facial Expression Recognition in the Wild. The objective of this sub-challenge is to classify the emotions expressed by the primary human subject in static images extracted from movies. We follow a transfer learning approach for deep Convolutional Neural Network (CNN) architectures. Starting from a network pre-trained on the generic ImageNet dataset, we perform supervised fine-tuning on the network in a two-stage process, first on datasets relevant to facial expressions, followed by the contest's dataset. Experimental results show that this cascading fine-tuning approach achieves better results, compared to a single stage fine-tuning with the combined datasets. Our best submission exhibited an overall accuracy of 48.5% in the validation set and 55.6% in the test set, which compares favorably to the respective 35.96% and 39.13% of the challenge baseline.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Since the number of labeled examples for FER remains relatively limited, large models are likely to overfit the data.
“Since the number of labeled examples for FER remains relatively limited, large 11 models are likely to overfit the data [106], [107], [89], [108].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [107]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097 · see citing works
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653 · see citing works
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · see citing works
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · see citing works
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335 · see citing works
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745 · see citing works
Challenges in Representation Learning: A Report on Three Machine Learning Contests
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-42051-1_16 · Cited by 1,719 · see citing works
Raina, R., Battle, A. J., Lee, H., Packer, B., & Ng, A. Y. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273592 · Cited by 1,514 · see citing works
Challenges in representation learning: A report on three machine learning contests
Goodfellow, I. J., Erhan, D., Luc Carrier, P., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.005 · Cited by 697 · see citing works
Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition
Sarıyanidi, E., Güneş, H., & Cavallaro, A. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2366127 · Cited by 689 · see citing works
Collecting Large, Richly Annotated Facial-Expression Databases from Movies
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2012) · IEEE Multimedia · doi:10.1109/mmul.2012.26 · Cited by 664 · see citing works
Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2011) · 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) · doi:10.1109/iccvw.2011.6130508 · Cited by 541 · cited 1× · see citing works
“Images are selected from movies, in a semi-automated way, via a system based on subtitles [5, 6].”
MethodCombining modality specific deep neural networks for emotion recognition in video
Kahou, S. E., Pal, C., Bouthillier, X., Froumenty, P., Gülçehre, Ç., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R. C., et al. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531745 · Cited by 391 · see citing works
Video and Image based Emotion Recognition Challenges in the Wild: EmotiW 2015
Dhall, A., Ramana Murthy, O. V., Goecke, R., Joshi, J., & Gedeon, T. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2829994 · Cited by 305 · see citing works
Combining Multiple Kernel Methods on Riemannian Manifold for Emotion Recognition in the Wild
Liu, M., Wang, R., Li, S., Shan, S., Huang, Z., & Chen, X. (2014) · Proceedings of the 16th International Conference on Multimodal Interaction · doi:10.1145/2663204.2666274 · Cited by 202 · see citing works
Proceedings of the 38th Annual Conference of the Cognitive Science Society, CogSci 2016
Räsänen, O. J., Nagamine, T., & Mesgarani, N. (2016) · Cited by 175 · see citing works
la Torre, F. D., Chu, W.-S., Xiong, X., Vicente, F., Ding, X., & Cohn, J. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163082 · Cited by 125 · see citing works
Multiple kernel learning for emotion recognition in the wild
Sikka, K., Dykstra, K., Sathyanarayana, S., Littlewort, G., & Bartlett, M. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531741 · Cited by 119 · see citing works
Context-Sensitive Dynamic Ordinal Regression for Intensity Estimation of Facial Action Units
Rudovic, O. O., Pavlović, V., & Pantić, M. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2356192 · Cited by 111 · see citing works
Emotion Recognition in the Wild with Feature Fusion and Multiple Kernel Learning
Chen, J., Chen, Z., Chi, Z., & Fu, H. C. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2666277 · Cited by 104 · see citing works
Sun, B., Li, L., Zuo, T., Chen, Y., Zhou, G., & Wu, X. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2666272 · Cited by 71 · see citing works
Partial least squares regression on grassmannian manifold for emotion recognition
Liu, M., Wang, R., Huang, Z., Shan, S., & Chen, X. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531738 · Cited by 62 · see citing works
Recognition of facial emotion in low vision: A flexible usage of facial features
Boucart, M., DINON, J.-F., Despretz, P., Desmettre, T. J., Hladiuk, K., & Oliva, A. (2008) · Visual Neuroscience · doi:10.1017/s0952523808080656 · Cited by 55 · see citing works
Learning realistic facial expressions from web images
Yu, K., Wang, Z., Li, Z., Wang, J., Chi, Z., & Feng, D. D. (2013) · Pattern Recognition · doi:10.1016/j.patcog.2013.01.032 · Cited by 36 · see citing works
Zhang, L., Tjondronegoro, D., & Chandran, V. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.09.005 · Cited by 27 · see citing works
Keeping it real: Recognizing expressions in real compared to schematic faces
Prazak, E. R., & Burgund, E. D. (2014) · Visual Cognition · doi:10.1080/13506285.2014.914991 · Cited by 15 · see citing works
Is Facial Expression Processing Holistic?
Omigbodun, A., & Cottrell, G. W. (2013) · eScholarship (California Digital Library) · Cited by 3 · see citing works
Flagged first, then by citations
1 of the 32 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 2,041 · see citing works
“Instead of directly using the pretrained or fine-tuned models to extract features on the target dataset, a multistage fine-tuning strategy [67] can achieve better performance; after the first-stage fine-tuning using FER2013 on pretrained”
MethodReliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild
Li, S., Deng, W., & Du, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.277 · Cited by 1,470 · see citing works
“In EmotiW 2015 [8], more competitors have tried deep learning meth-ods: transfer learning was used to solve the problem of small database in [32], hierarchical committee of multi-column DCNNs in [20] gained the best result on SFEW database, LBP features combined with DCNNs structure were proposed…”
BackgroundDeep Learning vs. Traditional Computer Vision
O’Mahony, N., Campbell, S., Carvalho, A. L. D., Harapanahalli, S., Velasco-Hernandez, G., Krpálková, L., Riordan, D., & Walsh, J. L. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-17795-9_10 · Cited by 1,104 · see citing works
A Brief Review of Facial Emotion Recognition Based on Visual Information
Ko, B. C. (2018) · Sensors · doi:10.3390/s18020401 · Cited by 726 · see citing works
“Therefore, this sub-section briefly introduces some popular databases related to FER consisting of 2D and 3D video sequences and still images:”
BackgroundLi, S., & Deng, W. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2868382 · Cited by 669 · see citing works
““SFEW best [33]”, “SFEW second [77]” and “SFEW third [54]” indicate the best single model result of the winner, the runner-up and the second runner-up in EmotiW 2015, respectively.”
BackgroundMethodInfluentialApplication of Deep Learning in Food: A Review
Zhou, L., Zhang, C., Liu, F., Qiu, Z., & He, Y. (2019) · Comprehensive Reviews in Food Science and Food Safety · doi:10.1111/1541-4337.12492 · Cited by 566 · see citing works
Generalisation in humans and deep neural networksPreprint
Geirhos, R., Temme, C. R. M., Rauber, J., Schütt, H. H., Bethge, M., & Wichmann, F. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08750 · Cited by 498 · see citing works
“1 Secondly, features learned on one task often transfer to only loosely related tasks, such as from classification to saliency prediction [33], emotion recognition [34], medical imaging [35] and a large number of other transfer learning tasks [36].”
BackgroundEmotion recognition using deep learning approach from audio–visual emotional big data
Hossain, M. S., & Muhammad, G. (2018) · Information Fusion · doi:10.1016/j.inffus.2018.09.008 · Cited by 454 · see citing works
Deep learning-based facial emotion recognition for human–computer interaction applications
Chowdary, M. K., Nguyen, T. N., & Hemanth, D. J. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06012-8 · Cited by 417 · see citing works
“learning is one of the methods in deep learning; in this, a model trained for a particular task is reused for another task by the transfer of knowledge [23].”
MethodFaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition
Ding, H., Zhou, S. K., & Chellappa, R. (2017) · 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) · doi:10.1109/fg.2017.23 · Cited by 416 · see citing works
“Compared with networks adopted in previous works [9], [23], [25], AlexNet [47] or VGG-M [48], the size of our network is fairly small.”
MethodResultImage-Based malware classification using ensemble of CNN architectures (IMCEC)
Vasan, D., Alazab, M., Wassan, S., Safaei, B., & Zheng, Q. (2020) · Computers & security · doi:10.1016/j.cose.2020.101748 · Cited by 374 · see citing works
Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
Kollias, D., Tzirakis, P., Nicolaou, M. A., Papaioannou, A., Zhao, G., Schuller, B. W., Kotsia, I., & Zafeiriou, S. P. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01158-4 · Cited by 388 · see citing works
“For the second approach we used transfer learning [38], especially of the convolutional and pooling part of the pre-trained networks.”
MethodAutomatic Analysis of Facial Actions: A Survey
Martínez, B., Valstar, M., Jiang, B., & Pantić, M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2731763 · Cited by 354 · see citing works
“An alternative to dealing with a low quantity of labelled examples is the use of transfer learning techniques [122].”
BackgroundA survey on facial emotion recognition techniques: A state-of-the-art literature review
Canal, F. Z., Müller, T. R., Matias, J. C., Scotton, G. G., de Sá, A. R., Pozzebon, E., & Sobieranski, A. C. (2021) · Information Sciences · doi:10.1016/j.ins.2021.10.005 · Cited by 358 · see citing works
Island Loss for Learning Discriminative Features in Facial Expression Recognition
Cai, J., Meng, Z., Khan, A. S., Li, Z., O’Reilly, J. D., & Tong, Y. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00051 · Cited by 337 · see citing works
Identity-Aware Convolutional Neural Network for Facial Expression Recognition
Meng, Z., Liu, P., Cai, J., Han, S., & Tong, Y. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.140 · Cited by 334 · see citing works
“deep learning [34], [42], [23], [29], [20], [16], [49], [30], [13], [21], [53].”
BackgroundFrom Facial Expression Recognition to Interpersonal Relation Prediction
Zhang, Z., Luo, P., Loy, C. C., & Tang, X. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1055-1 · Cited by 305 · see citing works
“DCN of [54] - AlexNet [37] pretrained with ImageNet [65] and FER [20] datasets, and finetune on the SFEW training dataset.”
BackgroundMethodRICE PLANT DISEASE CLASSIFICATION USING TRANSFER LEARNING OF DEEP CONVOLUTION NEURAL NETWORK
Shrivastava, V. K., Pradhan, M. K., Minz, S., & Thakur, M. (2019) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-archives-xlii-3-w6-631-2019 · Cited by 251 · see citing works
“Transfer learning is a technique, where the initialization of CNN weights occurs from pre-trained network (Oquab et al., 2014; Hu et al., 2015; Ng et al., 2015; Hoo-Chang et al., 2016).”
MethodKnowledge Transfer for Rotary Machine Fault Diagnosis
Yan, R., Shen, F., Sun, C., & Chen, X. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2949057 · Cited by 245 · see citing works
“In addition, abstract CNN parameters have better learning ability but higher complexity than SVM parameters [55].”
BackgroundDeep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Surface defect classification of steels with a new semi-supervised learning method
Di, H., Ke, X., Peng, Z., & Dongdong, Z. (2019) · Optics and Lasers in Engineering · doi:10.1016/j.optlaseng.2019.01.011 · Cited by 241 · see citing works
Machine learning enables design automation of microfluidic flow-focusing droplet generation
Lashkaripour, A., Rodriguez, C., Mehdipour, N., Mardian, R., McIntyre, D., Ortiz, L., Campbell, J., & Densmore, D. M. (2021) · Nature Communications · doi:10.1038/s41467-020-20284-z · Cited by 239 · see citing works
“Consequently, an immediate drawback of training neural network models on small-scale data-sets is over-fi tting to the train-set, resulting in models that poorly generalize to the test-set 43 .”
BackgroundVideo-based emotion recognition in the wild using deep transfer learning and score fusion
Kaya, H., Gürpınar, F., & Salah, A. A. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.012 · Cited by 235 · see citing works
“In [36] the authors exploited pre-trained models based on the ImageNet dataset, followed by two stages of fine-tuning, on FER 2013 and Static Facial Expression Sub-Challenge (SFEW) Dataset from the EmotiW 2015 Challenge, respectively [8].”
MethodFrom individual to group-level emotion recognition: EmotiW 5.0
Dhall, A., Goecke, R., Ghosh, S., Joshi, J., Hoey, J., & Gedeon, T. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143004 · Cited by 218 · see citing works
“Researchers also proposed transfer learning based methods for dealing with smaller sized training data [17].”
BackgroundJiang, Y., Li, W., Hossain, M. S., Chen, M., Alelaiwi, A. A., & Al-Hammadi, M. H. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.06.019 · Cited by 219 · see citing works
Adaptive Deep Metric Learning for Identity-Aware Facial Expression Recognition
Liu, X., Kumar, B. V. K. V., You, J., & Jia, P. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.79 · Cited by 209 · see citing works
“[31] 48.”
Towards Efficient and Privacy-Preserving Federated Deep Learning
Hao, M., Li, H., Xu, G., Liu, S., & Yang, H. (2019) · International Conference on Communications · doi:10.1109/icc.2019.8761267 · Cited by 213 · see citing works
“Deep neural network has been applied in various application scenarios such as computer vision [1], emotion analysis [2] and natural language processing [3] due to its superior accuracy of prediction for training enormous amount of data.”
MethodEmotion Classification with Data Augmentation Using Generative Adversarial Networks
Zhu, X., Liu, Y., Li, J., Wan, T., & Qin, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93040-4_28 · Cited by 197 · see citing works
“On the other hand, some classes of emotion images (such as disgust ) have few samples in the training set from the real world than other classes, also some images can be very nuanced, making it difficult even for humans to agree on their correct labeling [22].”
BackgroundA comparison between shallow and deep architecture classifiers on small dataset
Pasupa, K., & Sunhem, W. (2016) · doi:10.1109/iciteed.2016.7863293 · Cited by 191 · see citing works
“An example of usages of transfer learning technique can be found in a recent work by [9].”
MethodFacial Expression Recognition Using Computer Vision: A Systematic Review
Canedo, D., & Neves, A. J. R. (2019) · Applied Sciences · doi:10.3390/app9214678 · Cited by 192 · see citing works
Kollias, D., & Zafeiriou, S. P. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3014171 · Cited by 23 · see citing works
Entity recognition from clinical texts via recurrent neural network
Liu, Z., Yang, M. Y., Wang, X., Chen, Q., Tang, B., Wang, Z., & Xu, H. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0468-7 · Cited by 176 · see citing works
“In the case of NLP, deep learning has been deployed to tackle machine translation [29], relation extraction [30], entity recognition [31–35], word sense disambiguation [36], syntax parsing [37, 38], emotion classification [39], etc.”
MethodAnalysing Affective Behavior in the second ABAW2 Competition
Kollias, D., & Zafeiriou, S. P. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00408 · Cited by 168 · see citing works
“The ABAW2 Competition contains three Challenges, which are based on the same database; these target (i) dimensional affect recognition (in terms of valence and arousal) [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], (ii) categorical affect classification (in terms of the seven basic emotions) [20, 21, 22, 23, 24, 25, 26, 2…”
BackgroundWeb-based fully automated cephalometric analysis by deep learning
Kim, H. H., Shim, E., Park, J., Kim, Y., Lee, U., & Kim, Y. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105513 · Cited by 161 · see citing works
Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion
Siriwardhana, S., Kaluarachchi, T. I., Billinghurst, M., & Nanayakkara, S. C. (2020) · IEEE Access · doi:10.1109/access.2020.3026823 · Cited by 164 · see citing works
“techniques [8]–[10] to extract features from pre-trained DL models.”
MethodA survey on big data-driven digital phenotyping of mental health
Liang, Y., Zheng, X., & Zeng, D. D. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.04.001 · Cited by 154 · see citing works
Multi-region Ensemble Convolutional Neural Network for Facial Expression Recognition
Fan, Y., Lam, J. C. K., & Li, V. O. K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01418-6_9 · Cited by 155 · see citing works
Deep neural architectures for prediction in healthcare
Kollias, D., Tagaris, A., Stafylopatis, A., Kollias, S., & Tagaris, G. L. (2017) · Complex & Intelligent Systems · doi:10.1007/s40747-017-0064-6 · Cited by 147 · see citing works
Selective deep features for micro-expression recognition
Patel, D., Hong, X., & Zhao, G. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7899972 · Cited by 146 · see citing works
“al [13] have used cascaded fine tuning of CNN trained on ImageNet dataset with more labelled”
MethodApplications of deep-learning approaches in horticultural research: a review
Yang, B., & Xu, Y. (2021) · Horticulture Research · doi:10.1038/s41438-021-00560-9 · Cited by 147 · see citing works
“As a subset of the machine-learning approaches, deep learning has also been widely employed, and has attracted more attention from various domains, such as agricultural production 15 – 17 , food science 7,18 , robotics 19,20 , and human action and speech recognition 21,22 .”
BackgroundPan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142 · see citing works
Facial Expression Recognition Using a Hybrid CNN–SIFT Aggregator
Connie, T., Al-Shabi, M., Cheah, W. P., & Goh, M. K. O. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-69456-6_12 · Cited by 140 · see citing works
“Instead of minimization of cross-entropy loss like vanilla CNN, he minimized a margin-based loss with standard hinge loss.”
MethodMorales, F. J. O., & Roggen, D. (2016) · Proceedings of the 2016 ACM International Symposium on Wearable Computers · doi:10.1145/2971763.2971764 · Cited by 136 · see citing works
Investigating the use of recurrent motion modelling for speech gesture generation
Ferstl, Y., & McDonnell, R. (2018) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3267851.3267898 · Cited by 138 · see citing works
“Transfer learning has shown success in a variety of domains, such as machine translation [38], image classification [31], and visual emotion recognition [30].”
BackgroundLi, S., & Deng, W. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1131-1 · Cited by 132 · see citing works
“In Table 6(b), the “SFEW best (Kim et al. 2016)”, “SFEW second (Yu and Zhang 2015)” and “SFEW third (Ng et al. 2015)” indicate the best single model result of the 1st, the 2nd and the 3rd participator in EmotiW 2015, respectively.”
BackgroundInfluentialDiscriminative Joint Probability Maximum Mean Discrepancy (DJP-MMD) for Domain Adaptation
Zhang, W., & Wu, D. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207365 · Cited by 126 · see citing works
“It has been widely used in image classification [9], [10], emotion recognition [11], brain-computer interfaces [12], [13], and so on.”
An Active Learning Paradigm for Online Audio-Visual Emotion Recognition
Kansizoglou, I., Bampis, L., & Gasteratos, A. C. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2961089 · Cited by 125 · see citing works
Recognition of Affect in the Wild Using Deep Neural Networks
Kollias, D., Nicolaou, M. A., Kotsia, I., Zhao, G., & Zafeiriou, S. P. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.247 · Cited by 124 · see citing works
“For the second approach we used transfer learning [17], especially of the convolutional and pooling part of the pre-trained networks.”
MethodProgressive Neural Networks for Transfer Learning in Emotion Recognition
Gideon, J., Khorram, S., Aldeneh, Z., Dimitriadis, D., & Provost, E. M. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1637 · Cited by 129 · see citing works
“[10] studied transfer learning in the context of facial expression recognition.”
BackgroundZheng, Z., Zhang, Z., & Pan, W. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103387 · Cited by 124 · see citing works
Rather, I. H., Kumar, S., & Gandomi, A. H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10859-3 · Cited by 131 · see citing works
Tabbaa, L., Searle, R., Bafti, S. M., Hossain, M. M., Intarasisrisawat, J., Glancy, M., & Ang, C. S. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3495002 · Cited by 115 · see citing works
“This technique has already successfully been used for emotion recognition [55].”
MethodOn-Road Driver Emotion Recognition Using Facial Expression
Xiao, H., Li, W., Zeng, G., Wu, Y., Xue, J., Zhang, J., Li, C., & Guo, G. (2022) · Applied Sciences · doi:10.3390/app12020807 · Cited by 111 · see citing works
“With the lack of large-scale datasets, fine-tuning has been widely investigated for facial expression recognition [44,45].”
BackgroundMethodCNN-SVM for Microvascular Morphological Type Recognition with Data Augmentation
Xue, D.-X., Zhang, R., Feng, H., & Wang, Y. (2016) · Journal of Medical and Biological Engineering · doi:10.1007/s40846-016-0182-4 · Cited by 106 · see citing works
“By reusing the knowledge gained from past related tasks [10–12], it is now much easier to tackle more challenging tasks such as image retrieval [5], semantic segmentation [13], fine grained recognition [14], and emotion recognition [15].”
MethodA Fully Convolutional Neural Network for Wood Defect Location and Identification
He, T., Liu, Y., Xu, C., Zhou, X., Hu, Z., & Fan, J. (2019) · IEEE Access · doi:10.1109/access.2019.2937461 · Cited by 106 · see citing works
“Finally, we use the transfer learning in our approach to maximize the collected data and avoid overfitting as much as possible [28]–[30].”
MethodTransfer Learning using CNN for Handwritten Devanagari Character Recognition
Aneja, N., & Aneja, S. (2019) · doi:10.1109/icait47043.2019.8987286 · Cited by 102 · see citing works
“[9] classified human emotions in images extractedfrom movies.”
BackgroundIlyas, N., Shahzad, A., & Kim, K. (2019) · Sensors · doi:10.3390/s20010043 · Cited by 101 · see citing works
Conversational transfer learning for emotion recognition
Hazarika, D., Poria, S., Zimmermann, R., & Mihalcea, R. F. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.005 · Cited by 106 · see citing works
OAENet: Oriented attention ensemble for accurate facial expression recognition
Wang, Z., Zeng, F., Liu, S., & Zeng, B. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107694 · Cited by 104 · see citing works
Facial Emotion Detection to Assess Learner's State of Mind in an Online Learning System
Mukhopadhyay, M., Pal, S., Nayyar, A., Pramanik, P. K. D., Dasgupta, N., & Choudhury, P. (2020) · International Conference on Intelligent Information Technology (ICIIT) · doi:10.1145/3385209.3385231 · Cited by 99 · see citing works
“Similarly, in [17], CNN has achieved better accuracy even by using the small-sized dataset (EMotiW).”
MethodEmotion Recognition for Cognitive Edge Computing Using Deep Learning
Muhammad, G., & Hossain, M. S. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3058587 · Cited by 107 · see citing works
Melinte, O., & Vlădăreanu, L. (2020) · Sensors · doi:10.3390/s20082393 · Cited by 104 · see citing works
“In [6] a two stage fine-tuning is presented.”
BackgroundAutomatic dementia screening and scoring by applying deep learning on clock-drawing tests
Chen, S., Stromer, D., Alabdalrahim, H. A., Schwab, S., Weih, M., & Maier, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-74710-9 · Cited by 96 · see citing works
COVID-19 Detection in Chest X-ray Images using a Deep Learning Approach.
Saiz, F. A., & Barandiarán, Í. (2020) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2020.04.003 · Cited by 94 · see citing works
Fast facial expression recognition using local binary features and shallow neural networks
Gogić, I., Manhart, M., Pandzic, I. S., & Ahlberg, J. (2018) · The Visual Computer · doi:10.1007/s00371-018-1585-8 · Cited by 96 · see citing works
“While all of the previously mentioned methods use handcrafted and heuristically determined features, experiments with deep learning using convolutional neural networks (CNN) [27] on the FER problem were recently conducted as well [4,9,26,30,33,35,39,40,45,54,57].”
MethodCogan, T., Cogan, M., & Tamil, L. S. (2019) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2019.103351 · Cited by 93 · see citing works
Audio-visual emotion recognition using deep transfer learning and multiple temporal models
Ouyang, X., Kawaai, S., Goh, E. G. H., Shen, S., Ding, W., Ming, H., & Huang, D. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3143012 · Cited by 92 · see citing works
“[16] finetuned 5-layer AlexNet for emotion classification.”
Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources
Ngai, W. K., Xie, H., Zou, D., & Chou, K. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.007 · Cited by 92 · see citing works
An Efficient Approach Based on Privacy-Preserving Deep Learning for Satellite Image Classification
Alkhelaiwi, M., Boulila, W., Ahmad, J. E., Koubâa, A., & Driss, M. (2021) · Remote Sensing · doi:10.3390/rs13112221 · Cited by 92 · see citing works
“TL is commonly used when training a small dataset where the CNN’s weights are initialized before being fine-tuned with the new dataset [19].”
MethodContemplating Visual Emotions: Understanding and Overcoming Dataset Bias
Panda, R., Zhang, J., Li, H., Lee, J., Lu, X., & Roy–Chowdhury, A. K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_36 · Cited by 85 · see citing works
Facial emotion recognition on a dataset using convolutional neural network
Tümen, V., Söylemez, Ö. F., & Ergen, B. (2017) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap.2017.8090281 · Cited by 83 · see citing works
“Yine HongWei ve arkada lar n n yapm oldu u çal mas nda [9] bir CNN modeli ile ayn veriler üzerinde uygulanm ve 10% test verisinde 55,6%, do rulama da ise 48,5% bir ba ar oran elde edilmi tir.”
BackgroundA novel database of children's spontaneous facial expressions (LIRIS-CSE)
Khan, R. A., Crenn, A., Meyer, A., & Bouakaz, S. (2019) · Image and Vision Computing · doi:10.1016/j.imavis.2019.02.004 · Cited by 82 · see citing works
“Recently, most of successful models are based on deep learning approach [29, 30, 31], specifically on Convolutional Neural Network (CNN) architecture [15].”
Balan, O. C., Moise, G., Moldoveanu, A. D. B., Leordeanu, M., & Moldoveanu, F. (2020) · Sensors · doi:10.3390/s20020496 · Cited by 82 · see citing works
“A Deep Convolutional Neural Network-based approach for expression classification on the EmotiW (The Emotion Recognition in the Wild contest) dataset is presented in [92].”
MethodA LSTM based deep learning network for recognizing emotions using wireless brainwave driven system
Sakalle, A., Tomar, P., Bhardwaj, H., Acharya, D., & Bhardwaj, A. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114516 · Cited by 82 · see citing works
Exploring the role of deep neural networks for post-disaster decision support
Chaudhuri, N., & Bose, I. (2020) · Decision Support Systems · doi:10.1016/j.dss.2019.113234 · Cited by 86 · see citing works
A Review of Generalizable Transfer Learning in Automatic Emotion Recognition
Feng, K., & Chaspari, T. (2020) · Frontiers in Computer Science · doi:10.3389/fcomp.2020.00009 · Cited by 82 · see citing works
In search of a robust facial expressions recognition model: A large-scale visual cross-corpus study
Рюмина, Е. В., Dresvyanskiy, D., & Karpov, A. A. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.10.013 · Cited by 81 · see citing works
Personalized Human Activity Recognition Based on Integrated Wearable Sensor and Transfer Learning
Fu, Z., He, X., Wang, E., Huo, J., Huang, J., & Wu, D. (2021) · Sensors · doi:10.3390/s21030885 · Cited by 79 · see citing works
“Transfer learning is widely used in image classification [42], emotion recognition [43], brain–computer interface [44,45].”
Wan, J., Escalera, S., Anbarjafari, G., Escalante, H. J., Baró, X., Guyon, I., Madadi, M., Allïk, J., Gorbova, J., Lin, C., et al. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.377 · Cited by 77 · see citing works
“TUBITAK UZAY-METU face detection pretrained CNN emotion model [32] high-level emotional features (conv5) SVM”
MethodFacial Expression Recognition Using Visual Saliency and Deep Learning
Mavani, V., Raman, S., & Miyapuram, K. P. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.327 · Cited by 78 · see citing works
Transfer‐learning‐based Raman spectra identification
Zhang, R., Xie, H., Cai, S., Hu, Y., Liu, G., Hong, W., & Tian, Z. (2019) · Journal of Raman Spectroscopy · doi:10.1002/jrs.5750 · Cited by 75 · see citing works
Data Augmentation in Emotion Classification Using Generative Adversarial NetworksPreprint
Zhu, X., Liu, Y., Qin, Z., & Li, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00648 · Cited by 75 · see citing works
“On the other hand, some classes of emotion images (such as disgust ) have few samples in the training set from the real world than other classes, also some images can be very nuanced, making it difficult even for humans to agree on their correct labeling [22].”
BackgroundComputer-Aided Diagnosis (CAD) of Pulmonary Nodule of Thoracic CT Image Using Transfer Learning
Zhang, S., Sun, F., Wang, N., Zhang, C., Yu, Q., Zhang, M., Babyn, P., & Zhong, H. (2019) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-019-00204-4 · Cited by 71 · see citing works
Unified deep learning approach for prediction of Parkinson's disease
Wingate, J., Kollia, I., Bidaut, L. M., & Kollias, S. (2020) · IET Image Processing · doi:10.1049/iet-ipr.2019.1526 · Cited by 7 · see citing works
Joint Expression Synthesis and Representation Learning for Facial Expression Recognition
Zhang, X., Zhang, F., & Xu, C. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3056098 · Cited by 74 · see citing works
Ocean oil spill detection from SAR images based on multi-channel deep learning semantic segmentation
Hasimoto-Beltrán, R., Canul‐Ku, M., Méndez, G. M. D., Ocampo-Torres, F. J., & Esquivel–Trava, B. (2023) · Marine Pollution Bulletin · doi:10.1016/j.marpolbul.2023.114651 · Cited by 74 · see citing works
Facial emotion recognition using Handcrafted features and CNN
Gautam, C., & Seeja, K. R. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.01.108 · Cited by 74 · see citing works
Polarimetric Imaging via Deep Learning: A Review
Li, X., Yan, L., Qi, P., Zhang, L., Goudail, F., Liu, T., Zhai, J., & Hu, H. (2023) · Remote Sensing · doi:10.3390/rs15061540 · Cited by 76 · see citing works
Facial expression recognition in the wild based on multimodal texture features
Sun, B., Li, L., Zhou, G., & He, J. (2016) · Journal of Electronic Imaging · doi:10.1117/1.jei.25.6.061407 · Cited by 74 · see citing works
“Ng et al.(20) explored transfer learning for deep models including VGG and AlexNet.”
BackgroundFacial expression recognition boosted by soft label with a diverse ensemble
Gan, Y., Chen, J., & Xu, L. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.04.002 · Cited by 72 · see citing works
Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition
Han, S., Meng, Z., Li, Z., O’Reilly, J. D., Cai, J., Wang, X., & Tong, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00532 · Cited by 71 · see citing works
O’Mahony, N., Campbell, S., Carvalho, A., Harapanahalli, S., Velasco-Hernandez, G., Krpálková, L., Riordan, D., & Walsh, J. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-17795-9 · Cited by 72 · see citing works
“The frame reduction techniques discussed previously such as using SIFT features [26, 32] or optical flow for moving objects [27] to first identify a region of interest are useful with respect to image resolution and also with respect to reducing the time and data required for training.”
MethodFacial Expression Recognition Using Hybrid Features of Pixel and Geometry
Liu, C., Hirota, K., Ma, J., Jia, Z., & Dai, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3054332 · Cited by 74 · see citing works
“Due to the ability of extracting deep-level semantic features and the outstanding recognition performance, deep learning methods are applied to facial expression analysis [12], [21]–[23].”
BackgroundFace-Based Attention Recognition Model for Children with Autism Spectrum Disorder
Banire, B., Al‐Thani, D., Qaraqe, M., & Mansoor, B. (2021) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-021-00101-y · Cited by 71 · see citing works
“However, other studies reported lower recognition accuracy of 55.60% [47] and 54.56% [48] due to small data size.”
BackgroundInfluentialDeep Emotion Transfer Network for Cross-database Facial Expression Recognition
Li, S., & Deng, W. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8545284 · Cited by 67 · see citing works
Kim, H., & Youn, B. D. (2019) · IEEE Access · doi:10.1109/access.2019.2906273 · Cited by 65 · see citing works
“[36] and also used parameter freezing method to”
MethodSiddiqui, M. F. H., & Javaid, A. Y. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4030046 · Cited by 62 · see citing works
“Sensitive Artificial Listener (SAL) [67,71] Face and Speech The University of Southern California (USC) CreativeIT [72,73] Speech and Body movements Japanese Female Facial Expressions (JAFFE) [74–76] Face Cohn-Kanade (CK/CK+) [77,78] Face Berlin [79] Speech Acted Facial Expression in Wild (AFEW) [80] Face Facial Expre…”
BackgroundBriechle, S., Krzystek, P., & George Vosselman, A. (2021) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2020.102292 · Cited by 61 · see citing works
“Besides relatively quick convergence, effective fine-tuning of DNNs typically requires much less samples compared to training from scratch (Ng et al., 2015).”
BackgroundRole of Zoning in Facial Expression Using Deep Learning
Shahzad, T., Iqbal, K., Khan, M. A., Imran, I., & Iqbal, N. (2023) · IEEE Access · doi:10.1109/access.2023.3243850 · Cited by 64 · see citing works
PRATIT: a CNN-based emotion recognition system using histogram equalization and data augmentation
Mungra, D., Agrawal, A., Sharma, P., Tanwar, S., & Obaidat, M. S. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08397-0 · Cited by 62 · see citing works
“The authors of [32] used a cascaded combination for fine-tuning the CNN architecture for the purpose of facial expression recognition into seven emotions.”
Method
- Authors
- Ng, H.-W., Nguyen, V. D., Vonikakis, V., & Winkler, S.
- Affiliations
- University of Illinois at Urbana-Champaign (Singapore), Singapore, Singapore
- University of Illinois Urbana-Champaign (US)
- Venue
- Proceedings of the 2015 ACM on International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2015-11-09
- Pages
- 443-449
- OpenAlex
- W2253728219
- Cited by
- 657
- References
- 32
- Citations by year
- 2026: 13 · 2025: 29 · 2024: 37 · 2023: 70 · 2022: 69 · 2021: 106 · 2020: 106 · 2019: 106 · 2018: 63 · 2017: 44 · 2016: 13 · 2015: 1
- FWCI
- 53.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Funders
- Agency for Science Technology and Research
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- emotion recognition, facial expression recognition, still images, transfer learning, convolutional neural networks, fine-tuning, ImageNet pre-training, small datasets
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
