The DOI matches its Crossref record; 31 of the 32 references it cites were found in a registry or an index.
Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning
Abstract
This paper presents the techniques employed in our team's submissions to the 2015 Emotion Recognition in the Wild contest, for the sub-challenge of Static Facial Expression Recognition in the Wild. The objective of this sub-challenge is to classify the emotions expressed by the primary human subject in static images extracted from movies. We follow a transfer learning approach for deep Convolutional Neural Network (CNN) architectures. Starting from a network pre-trained on the generic ImageNet dataset, we perform supervised fine-tuning on the network in a two-stage process, first on datasets relevant to facial expressions, followed by the contest's dataset. Experimental results show that this cascading fine-tuning approach achieves better results, compared to a single stage fine-tuning with the combined datasets. Our best submission exhibited an overall accuracy of 48.5% in the validation set and 55.6% in the test set, which compares favorably to the respective 35.96% and 39.13% of the challenge baseline.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Since the number of labeled examples for FER remains relatively limited, large 11 models are likely to overfit the data [106], [107], [89], [108].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [107]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745
Flagged first, then by citations
1 of the 32 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- J. R. S. Widen and A. Brooks . Anger and disgust: Discrete or overlapping categories ? In Proceedings of the 2004 APS Annual Convention , 2004 . J. R. S. Widen and A. Brooks. Anger and disgust: Discrete or overlapping categories? In Proceedings of the 2004 APS Annual Convention, 2004.
Cited by
Sorted by citations
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 2,175
“Instead of directly using the pretrained or fine-tuned models to extract features on the target dataset, a multistage fine-tuning strategy [67] can achieve better performance; after the first-stage fine-tuning using FER2013 on pretrained”
MethodReliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild
Li, S., Deng, W., & Du, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.277 · Cited by 1,470
“In EmotiW 2015 [8], more competitors have tried deep learning meth-ods: transfer learning was used to solve the problem of small database in [32], hierarchical committee of multi-column DCNNs in [20] gained the best result on SFEW database, LBP features combined with DCNNs structure were proposed…”
BackgroundDeep Learning vs. Traditional Computer Vision
O’Mahony, N., Campbell, S., Carvalho, A. L. D., Harapanahalli, S., Velasco-Hernandez, G., Krpálková, L., Riordan, D., & Walsh, J. L. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-17795-9_10 · Cited by 1,173
A Brief Review of Facial Emotion Recognition Based on Visual Information
Ko, B. C. (2018) · Sensors · doi:10.3390/s18020401 · Cited by 790
“Therefore, this sub-section briefly introduces some popular databases related to FER consisting of 2D and 3D video sequences and still images:”
BackgroundLi, S., & Deng, W. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2868382 · Cited by 688
““SFEW best [33]”, “SFEW second [77]” and “SFEW third [54]” indicate the best single model result of the winner, the runner-up and the second runner-up in EmotiW 2015, respectively.”
BackgroundMethodInfluentialApplication of Deep Learning in Food: A Review
Zhou, L., Zhang, C., Liu, F., Qiu, Z., & He, Y. (2019) · Comprehensive Reviews in Food Science and Food Safety · doi:10.1111/1541-4337.12492 · Cited by 595
Generalisation in humans and deep neural networksPreprint
Geirhos, R., Temme, C. R. M., Rauber, J., Schütt, H. H., Bethge, M., & Wichmann, F. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08750 · Cited by 522
“1 Secondly, features learned on one task often transfer to only loosely related tasks, such as from classification to saliency prediction [33], emotion recognition [34], medical imaging [35] and a large number of other transfer learning tasks [36].”
BackgroundEmotion recognition using deep learning approach from audio–visual emotional big data
Hossain, M. S., & Muhammad, G. (2018) · Information Fusion · doi:10.1016/j.inffus.2018.09.008 · Cited by 460
Deep learning-based facial emotion recognition for human–computer interaction applications
Chowdary, M. K., Nguyen, T. N., & Hemanth, D. J. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06012-8 · Cited by 417
“learning is one of the methods in deep learning; in this, a model trained for a particular task is reused for another task by the transfer of knowledge [23].”
MethodFaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition
Ding, H., Zhou, S. K., & Chellappa, R. (2017) · 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) · doi:10.1109/fg.2017.23 · Cited by 416
“Compared with networks adopted in previous works [9], [23], [25], AlexNet [47] or VGG-M [48], the size of our network is fairly small.”
MethodResultImage-Based malware classification using ensemble of CNN architectures (IMCEC)
Vasan, D., Alazab, M., Wassan, S., Safaei, B., & Qin, Z. (2020) · Computers & Security · doi:10.1016/j.cose.2020.101748 · Cited by 421
Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
Kollias, D., Tzirakis, P., Nicolaou, M. A., Papaioannou, A., Zhao, G., Schuller, B. W., Kotsia, I., & Zafeiriou, S. P. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01158-4 · Cited by 406
“For the second approach we used transfer learning [38], especially of the convolutional and pooling part of the pre-trained networks.”
MethodAutomatic Analysis of Facial Actions: A Survey
Martínez, B., Valstar, M., Jiang, B., & Pantić, M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2731763 · Cited by 370
“An alternative to dealing with a low quantity of labelled examples is the use of transfer learning techniques [122].”
BackgroundA survey on facial emotion recognition techniques: A state-of-the-art literature review
Canal, F. Z., Müller, T. R., Matias, J. C., Scotton, G. G., de Sá, A. R., Pozzebon, E., & Sobieranski, A. C. (2021) · Information Sciences · doi:10.1016/j.ins.2021.10.005 · Cited by 358
Island Loss for Learning Discriminative Features in Facial Expression Recognition
Cai, J., Meng, Z., Khan, A. S., Li, Z., O’Reilly, J. D., & Tong, Y. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00051 · Cited by 339
Identity-Aware Convolutional Neural Network for Facial Expression Recognition
Meng, Z., Liu, P., Cai, J., Han, S., & Tong, Y. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.140 · Cited by 335
“deep learning [34], [42], [23], [29], [20], [16], [49], [30], [13], [21], [53].”
BackgroundFrom Facial Expression Recognition to Interpersonal Relation Prediction
Zhang, Z., Luo, P., Loy, C. C., & Tang, X. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1055-1 · Cited by 315
“DCN of [54] - AlexNet [37] pretrained with ImageNet [65] and FER [20] datasets, and finetune on the SFEW training dataset.”
BackgroundMethodRICE PLANT DISEASE CLASSIFICATION USING TRANSFER LEARNING OF DEEP CONVOLUTION NEURAL NETWORK
Shrivastava, V. K., Pradhan, M. K., Minz, S., & Thakur, M. (2019) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-archives-xlii-3-w6-631-2019 · Cited by 261
“Transfer learning is a technique, where the initialization of CNN weights occurs from pre-trained network (Oquab et al., 2014; Hu et al., 2015; Ng et al., 2015; Hoo-Chang et al., 2016).”
MethodKnowledge Transfer for Rotary Machine Fault Diagnosis
Yan, R., Shen, F., Sun, C., & Chen, X. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2019.2949057 · Cited by 245
“In addition, abstract CNN parameters have better learning ability but higher complexity than SVM parameters [55].”
BackgroundDeep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243
Surface defect classification of steels with a new semi-supervised learning method
Di, H., Ke, X., Peng, Z., & Dongdong, Z. (2019) · Optics and Lasers in Engineering · doi:10.1016/j.optlaseng.2019.01.011 · Cited by 241
Machine learning enables design automation of microfluidic flow-focusing droplet generation
Lashkaripour, A., Rodriguez, C., Mehdipour, N., Mardian, R., McIntyre, D., Ortiz, L., Campbell, J., & Densmore, D. M. (2021) · Nature Communications · doi:10.1038/s41467-020-20284-z · Cited by 246
“Consequently, an immediate drawback of training neural network models on small-scale data-sets is over-fi tting to the train-set, resulting in models that poorly generalize to the test-set 43 .”
BackgroundVideo-based emotion recognition in the wild using deep transfer learning and score fusion
Kaya, H., Gürpınar, F., & Salah, A. A. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.012 · Cited by 235
“In [36] the authors exploited pre-trained models based on the ImageNet dataset, followed by two stages of fine-tuning, on FER 2013 and Static Facial Expression Sub-Challenge (SFEW) Dataset from the EmotiW 2015 Challenge, respectively [8].”
MethodFrom individual to group-level emotion recognition: EmotiW 5.0
Dhall, A., Goecke, R., Ghosh, S., Joshi, J., Hoey, J., & Gedeon, T. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143004 · Cited by 218
“Researchers also proposed transfer learning based methods for dealing with smaller sized training data [17].”
BackgroundJiang, Y., Li, W., Hossain, M. S., Chen, M., Alelaiwi, A. A., & Al-Hammadi, M. H. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.06.019 · Cited by 224
Adaptive Deep Metric Learning for Identity-Aware Facial Expression Recognition
Liu, X., Kumar, B. V. K. V., You, J., & Jia, P. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.79 · Cited by 209
“[31] 48.”
Towards Efficient and Privacy-Preserving Federated Deep Learning
Hao, M., Li, H., Xu, G., Liu, S., & Yang, H. (2019) · International Conference on Communications · doi:10.1109/icc.2019.8761267 · Cited by 214
“Deep neural network has been applied in various application scenarios such as computer vision [1], emotion analysis [2] and natural language processing [3] due to its superior accuracy of prediction for training enormous amount of data.”
MethodEmotion Classification with Data Augmentation Using Generative Adversarial Networks
Zhu, X., Liu, Y., Li, J., Wan, T., & Qin, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93040-4_28 · Cited by 203
“On the other hand, some classes of emotion images (such as disgust ) have few samples in the training set from the real world than other classes, also some images can be very nuanced, making it difficult even for humans to agree on their correct labeling [22].”
BackgroundA comparison between shallow and deep architecture classifiers on small dataset
Pasupa, K., & Sunhem, W. (2016) · doi:10.1109/iciteed.2016.7863293 · Cited by 196
“An example of usages of transfer learning technique can be found in a recent work by [9].”
MethodFacial Expression Recognition Using Computer Vision: A Systematic Review
Canedo, D., & Neves, A. J. R. (2019) · Applied Sciences · doi:10.3390/app9214678 · Cited by 200
Kollias, D., & Zafeiriou, S. P. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3014171 · Cited by 206
Entity recognition from clinical texts via recurrent neural network
Liu, Z., Yang, M. Y., Wang, X., Chen, Q., Tang, B., Wang, Z., & Xu, H. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0468-7 · Cited by 184
“In the case of NLP, deep learning has been deployed to tackle machine translation [29], relation extraction [30], entity recognition [31–35], word sense disambiguation [36], syntax parsing [37, 38], emotion classification [39], etc.”
MethodAnalysing Affective Behavior in the second ABAW2 Competition
Kollias, D., & Zafeiriou, S. P. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00408 · Cited by 168
“The ABAW2 Competition contains three Challenges, which are based on the same database; these target (i) dimensional affect recognition (in terms of valence and arousal) [10, 11, 12, 13, 14, 15, 16, 17, 18, 19], (ii) categorical affect classification (in terms of the seven basic emotions) [20, 21, 22, 23, 24, 25, 26, 2…”
BackgroundWeb-based fully automated cephalometric analysis by deep learning
Kim, H. H., Shim, E., Park, J., Kim, Y., Lee, U., & Kim, Y. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105513 · Cited by 161
Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion
Siriwardhana, S., Kaluarachchi, T. I., Billinghurst, M., & Nanayakkara, S. C. (2020) · IEEE Access · doi:10.1109/access.2020.3026823 · Cited by 171
“techniques [8]–[10] to extract features from pre-trained DL models.”
MethodA survey on big data-driven digital phenotyping of mental health
Liang, Y., Zheng, X., & Zeng, D. D. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.04.001 · Cited by 164
Multi-region Ensemble Convolutional Neural Network for Facial Expression Recognition
Fan, Y., Lam, J. C. K., & Li, V. O. K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01418-6_9 · Cited by 157
Deep neural architectures for prediction in healthcare
Kollias, D., Tagaris, A., Stafylopatis, A., Kollias, S., & Tagaris, G. L. (2017) · Complex & Intelligent Systems · doi:10.1007/s40747-017-0064-6 · Cited by 148
Selective deep features for micro-expression recognition
Patel, D., Hong, X., & Zhao, G. (2016) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2016.7899972 · Cited by 146
“al [13] have used cascaded fine tuning of CNN trained on ImageNet dataset with more labelled”
MethodApplications of deep-learning approaches in horticultural research: a review
Yang, B., & Xu, Y. (2021) · Horticulture Research · doi:10.1038/s41438-021-00560-9 · Cited by 152
“As a subset of the machine-learning approaches, deep learning has also been widely employed, and has attracted more attention from various domains, such as agricultural production 15 – 17 , food science 7,18 , robotics 19,20 , and human action and speech recognition 21,22 .”
BackgroundPan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142
Facial Expression Recognition Using a Hybrid CNN–SIFT Aggregator
Connie, T., Al-Shabi, M., Cheah, W. P., & Goh, M. K. O. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-69456-6_12 · Cited by 157
“Instead of minimization of cross-entropy loss like vanilla CNN, he minimized a margin-based loss with standard hinge loss.”
MethodMorales, F. J. O., & Roggen, D. (2016) · Proceedings of the 2016 ACM International Symposium on Wearable Computers · doi:10.1145/2971763.2971764 · Cited by 136
Investigating the use of recurrent motion modelling for speech gesture generation
Ferstl, Y., & McDonnell, R. (2018) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3267851.3267898 · Cited by 138
“Transfer learning has shown success in a variety of domains, such as machine translation [38], image classification [31], and visual emotion recognition [30].”
BackgroundLi, S., & Deng, W. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1131-1 · Cited by 134
“In Table 6(b), the “SFEW best (Kim et al. 2016)”, “SFEW second (Yu and Zhang 2015)” and “SFEW third (Ng et al. 2015)” indicate the best single model result of the 1st, the 2nd and the 3rd participator in EmotiW 2015, respectively.”
BackgroundInfluentialDiscriminative Joint Probability Maximum Mean Discrepancy (DJP-MMD) for Domain Adaptation
Zhang, W., & Wu, D. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207365 · Cited by 126
“It has been widely used in image classification [9], [10], emotion recognition [11], brain-computer interfaces [12], [13], and so on.”
An Active Learning Paradigm for Online Audio-Visual Emotion Recognition
Kansizoglou, I., Bampis, L., & Gasteratos, A. C. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2961089 · Cited by 126
Recognition of Affect in the Wild Using Deep Neural Networks
Kollias, D., Nicolaou, M. A., Kotsia, I., Zhao, G., & Zafeiriou, S. P. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.247 · Cited by 123
“For the second approach we used transfer learning [17], especially of the convolutional and pooling part of the pre-trained networks.”
MethodProgressive Neural Networks for Transfer Learning in Emotion Recognition
Gideon, J., Khorram, S., Aldeneh, Z., Dimitriadis, D., & Provost, E. M. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1637 · Cited by 129
“[10] studied transfer learning in the context of facial expression recognition.”
BackgroundZheng, Z., Zhang, Z., & Pan, W. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103387 · Cited by 126
Rather, I. H., Kumar, S., & Gandomi, A. H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10859-3 · Cited by 141
Tabbaa, L., Searle, R., Bafti, S. M., Hossain, M. M., Intarasisrisawat, J., Glancy, M., & Ang, C. S. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3495002 · Cited by 115
“This technique has already successfully been used for emotion recognition [55].”
MethodOn-Road Driver Emotion Recognition Using Facial Expression
Xiao, H., Li, W., Zeng, G., Wu, Y., Xue, J., Zhang, J., Li, C., & Guo, G. (2022) · Applied Sciences · doi:10.3390/app12020807 · Cited by 112
“With the lack of large-scale datasets, fine-tuning has been widely investigated for facial expression recognition [44,45].”
BackgroundMethodCNN-SVM for Microvascular Morphological Type Recognition with Data Augmentation
Xue, D.-X., Zhang, R., Feng, H., & Wang, Y. (2016) · Journal of Medical and Biological Engineering · doi:10.1007/s40846-016-0182-4 · Cited by 108
“By reusing the knowledge gained from past related tasks [10–12], it is now much easier to tackle more challenging tasks such as image retrieval [5], semantic segmentation [13], fine grained recognition [14], and emotion recognition [15].”
MethodA Fully Convolutional Neural Network for Wood Defect Location and Identification
He, T., Liu, Y., Xu, C., Zhou, X., Hu, Z., & Fan, J. (2019) · IEEE Access · doi:10.1109/access.2019.2937461 · Cited by 106
“Finally, we use the transfer learning in our approach to maximize the collected data and avoid overfitting as much as possible [28]–[30].”
MethodTransfer Learning using CNN for Handwritten Devanagari Character Recognition
Aneja, N., & Aneja, S. (2019) · doi:10.1109/icait47043.2019.8987286 · Cited by 102
“[9] classified human emotions in images extractedfrom movies.”
BackgroundIlyas, N., Shahzad, A., & Kim, K. (2019) · Sensors · doi:10.3390/s20010043 · Cited by 105
Conversational transfer learning for emotion recognition
Hazarika, D., Poria, S., Zimmermann, R., & Mihalcea, R. F. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.005 · Cited by 115
OAENet: Oriented attention ensemble for accurate facial expression recognition
Wang, Z., Zeng, F., Liu, S., & Zeng, B. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107694 · Cited by 105
Facial Emotion Detection to Assess Learner's State of Mind in an Online Learning System
Mukhopadhyay, M., Pal, S., Nayyar, A., Pramanik, P. K. D., Dasgupta, N., & Choudhury, P. (2020) · International Conference on Intelligent Information Technology (ICIIT) · doi:10.1145/3385209.3385231 · Cited by 99
“Similarly, in [17], CNN has achieved better accuracy even by using the small-sized dataset (EMotiW).”
MethodEmotion Recognition for Cognitive Edge Computing Using Deep Learning
Muhammad, G., & Hossain, M. S. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3058587 · Cited by 107
Melinte, O., & Vlădăreanu, L. (2020) · Sensors · doi:10.3390/s20082393 · Cited by 104
“In [6] a two stage fine-tuning is presented.”
BackgroundAutomatic dementia screening and scoring by applying deep learning on clock-drawing tests
Chen, S., Stromer, D., Alabdalrahim, H. A., Schwab, S., Weih, M., & Maier, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-74710-9 · Cited by 99
COVID-19 Detection in Chest X-ray Images using a Deep Learning Approach.
Saiz, F. A., & Barandiarán, Í. (2020) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2020.04.003 · Cited by 94
Fast facial expression recognition using local binary features and shallow neural networks
Gogić, I., Manhart, M., Pandzic, I. S., & Ahlberg, J. (2018) · The Visual Computer · doi:10.1007/s00371-018-1585-8 · Cited by 98
“While all of the previously mentioned methods use handcrafted and heuristically determined features, experiments with deep learning using convolutional neural networks (CNN) [27] on the FER problem were recently conducted as well [4,9,26,30,33,35,39,40,45,54,57].”
MethodCogan, T., Cogan, M., & Tamil, L. S. (2019) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2019.103351 · Cited by 93
Audio-visual emotion recognition using deep transfer learning and multiple temporal models
Ouyang, X., Kawaai, S., Goh, E. G. H., Shen, S., Ding, W., Ming, H., & Huang, D. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3143012 · Cited by 92
“[16] finetuned 5-layer AlexNet for emotion classification.”
Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources
Ngai, W. K., Xie, H., Zou, D., & Chou, K. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.007 · Cited by 96
An Efficient Approach Based on Privacy-Preserving Deep Learning for Satellite Image Classification
Alkhelaiwi, M., Boulila, W., Ahmad, J. E., Koubâa, A., & Driss, M. (2021) · Remote Sensing · doi:10.3390/rs13112221 · Cited by 92
“TL is commonly used when training a small dataset where the CNN’s weights are initialized before being fine-tuned with the new dataset [19].”
MethodContemplating Visual Emotions: Understanding and Overcoming Dataset Bias
Panda, R., Zhang, J., Li, H., Lee, J., Lu, X., & Roy–Chowdhury, A. K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_36 · Cited by 85
Facial emotion recognition on a dataset using convolutional neural network
Tümen, V., Söylemez, Ö. F., & Ergen, B. (2017) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap.2017.8090281 · Cited by 83
“Yine HongWei ve arkada lar n n yapm oldu u çal mas nda [9] bir CNN modeli ile ayn veriler üzerinde uygulanm ve 10% test verisinde 55,6%, do rulama da ise 48,5% bir ba ar oran elde edilmi tir.”
BackgroundA novel database of children's spontaneous facial expressions (LIRIS-CSE)
Khan, R. A., Crenn, A., Meyer, A., & Bouakaz, S. (2019) · Image and Vision Computing · doi:10.1016/j.imavis.2019.02.004 · Cited by 82
“Recently, most of successful models are based on deep learning approach [29, 30, 31], specifically on Convolutional Neural Network (CNN) architecture [15].”
Balan, O. C., Moise, G., Moldoveanu, A. D. B., Leordeanu, M., & Moldoveanu, F. (2020) · Sensors · doi:10.3390/s20020496 · Cited by 83
“A Deep Convolutional Neural Network-based approach for expression classification on the EmotiW (The Emotion Recognition in the Wild contest) dataset is presented in [92].”
MethodA LSTM based deep learning network for recognizing emotions using wireless brainwave driven system
Sakalle, A., Tomar, P., Bhardwaj, H., Acharya, D., & Bhardwaj, A. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114516 · Cited by 82
Exploring the role of deep neural networks for post-disaster decision support
Chaudhuri, N., & Bose, I. (2020) · Decision Support Systems · doi:10.1016/j.dss.2019.113234 · Cited by 86
A Review of Generalizable Transfer Learning in Automatic Emotion Recognition
Feng, K., & Chaspari, T. (2020) · Frontiers in Computer Science · doi:10.3389/fcomp.2020.00009 · Cited by 82
In search of a robust facial expressions recognition model: A large-scale visual cross-corpus study
Рюмина, Е. В., Dresvyanskiy, D., & Karpov, A. A. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.10.013 · Cited by 91
Personalized Human Activity Recognition Based on Integrated Wearable Sensor and Transfer Learning
Fu, Z., He, X., Wang, E., Huo, J., Huang, J., & Wu, D. (2021) · Sensors · doi:10.3390/s21030885 · Cited by 79
“Transfer learning is widely used in image classification [42], emotion recognition [43], brain–computer interface [44,45].”
Wan, J., Escalera, S., Anbarjafari, G., Escalante, H. J., Baró, X., Guyon, I., Madadi, M., Allïk, J., Gorbova, J., Lin, C., et al. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.377 · Cited by 77
“TUBITAK UZAY-METU face detection pretrained CNN emotion model [32] high-level emotional features (conv5) SVM”
MethodFacial Expression Recognition Using Visual Saliency and Deep Learning
Mavani, V., Raman, S., & Miyapuram, K. P. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.327 · Cited by 78
Transfer‐learning‐based Raman spectra identification
Zhang, R., Xie, H., Cai, S., Hu, Y., Liu, G., Hong, W., & Tian, Z. (2019) · Journal of Raman Spectroscopy · doi:10.1002/jrs.5750 · Cited by 75
Data Augmentation in Emotion Classification Using Generative Adversarial NetworksPreprint
Zhu, X., Liu, Y., Qin, Z., & Li, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00648 · Cited by 75
“On the other hand, some classes of emotion images (such as disgust ) have few samples in the training set from the real world than other classes, also some images can be very nuanced, making it difficult even for humans to agree on their correct labeling [22].”
BackgroundComputer-Aided Diagnosis (CAD) of Pulmonary Nodule of Thoracic CT Image Using Transfer Learning
Zhang, S., Sun, F., Wang, N., Zhang, C., Yu, Q., Zhang, M., Babyn, P., & Zhong, H. (2019) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-019-00204-4 · Cited by 72
Unified deep learning approach for prediction of Parkinson's disease
Wingate, J., Kollia, I., Bidaut, L. M., & Kollias, S. (2020) · IET Image Processing · doi:10.1049/iet-ipr.2019.1526 · Cited by 7
Joint Expression Synthesis and Representation Learning for Facial Expression Recognition
Zhang, X., Zhang, F., & Xu, C. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3056098 · Cited by 74
Ocean oil spill detection from SAR images based on multi-channel deep learning semantic segmentation
Hasimoto-Beltrán, R., Canul‐Ku, M., Méndez, G. M. D., Ocampo-Torres, F. J., & Esquivel–Trava, B. (2023) · Marine Pollution Bulletin · doi:10.1016/j.marpolbul.2023.114651 · Cited by 81
Facial emotion recognition using Handcrafted features and CNN
Gautam, C., & Seeja, K. R. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.01.108 · Cited by 74
Polarimetric Imaging via Deep Learning: A Review
Li, X., Yan, L., Qi, P., Zhang, L., Goudail, F., Liu, T., Zhai, J., & Hu, H. (2023) · Remote Sensing · doi:10.3390/rs15061540 · Cited by 81
Facial expression recognition in the wild based on multimodal texture features
Sun, B., Li, L., Zhou, G., & He, J. (2016) · Journal of Electronic Imaging · doi:10.1117/1.jei.25.6.061407 · Cited by 74
“Ng et al.(20) explored transfer learning for deep models including VGG and AlexNet.”
BackgroundFacial expression recognition boosted by soft label with a diverse ensemble
Gan, Y., Chen, J., & Xu, L. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.04.002 · Cited by 73
Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition
Han, S., Meng, Z., Li, Z., O’Reilly, J. D., Cai, J., Wang, X., & Tong, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00532 · Cited by 71
O’Mahony, N., Campbell, S., Carvalho, A., Harapanahalli, S., Velasco-Hernandez, G., Krpálková, L., Riordan, D., & Walsh, J. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-17795-9 · Cited by 72
“The frame reduction techniques discussed previously such as using SIFT features [26, 32] or optical flow for moving objects [27] to first identify a region of interest are useful with respect to image resolution and also with respect to reducing the time and data required for training.”
MethodFacial Expression Recognition Using Hybrid Features of Pixel and Geometry
Liu, C., Hirota, K., Ma, J., Jia, Z., & Dai, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3054332 · Cited by 79
“Due to the ability of extracting deep-level semantic features and the outstanding recognition performance, deep learning methods are applied to facial expression analysis [12], [21]–[23].”
BackgroundFace-Based Attention Recognition Model for Children with Autism Spectrum Disorder
Banire, B., Al‐Thani, D., Qaraqe, M., & Mansoor, B. (2021) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-021-00101-y · Cited by 71
“However, other studies reported lower recognition accuracy of 55.60% [47] and 54.56% [48] due to small data size.”
BackgroundInfluentialDeep Emotion Transfer Network for Cross-database Facial Expression Recognition
Li, S., & Deng, W. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8545284 · Cited by 66
Kim, H., & Youn, B. D. (2019) · IEEE Access · doi:10.1109/access.2019.2906273 · Cited by 65
“[36] and also used parameter freezing method to”
MethodSiddiqui, M. F. H., & Javaid, A. Y. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4030046 · Cited by 63
“Sensitive Artificial Listener (SAL) [67,71] Face and Speech The University of Southern California (USC) CreativeIT [72,73] Speech and Body movements Japanese Female Facial Expressions (JAFFE) [74–76] Face Cohn-Kanade (CK/CK+) [77,78] Face Berlin [79] Speech Acted Facial Expression in Wild (AFEW) [80] Face Facial Expre…”
BackgroundBriechle, S., Krzystek, P., & George Vosselman, A. (2021) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2020.102292 · Cited by 62
“Besides relatively quick convergence, effective fine-tuning of DNNs typically requires much less samples compared to training from scratch (Ng et al., 2015).”
BackgroundRole of Zoning in Facial Expression Using Deep Learning
Shahzad, T., Iqbal, K., Khan, M. A., Imran, I., & Iqbal, N. (2023) · IEEE Access · doi:10.1109/access.2023.3243850 · Cited by 64
PRATIT: a CNN-based emotion recognition system using histogram equalization and data augmentation
Mungra, D., Agrawal, A., Sharma, P., Tanwar, S., & Obaidat, M. S. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08397-0 · Cited by 62
“The authors of [32] used a cascaded combination for fine-tuning the CNN architecture for the purpose of facial expression recognition into seven emotions.”
Method
Metadata
- Authors
- Ng, H.-W., Nguyen, V. D., Vonikakis, V., & Winkler, S.
- Affiliations
- University of Illinois at Urbana-Champaign (Singapore), Singapore, Singapore
- University of Illinois Urbana-Champaign (US)
- Venue
- Proceedings of the 2015 ACM on International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2015-11-09
- Pages
- 443-449
- OpenAlex
- W2253728219
- Cited by
- 657
- References
- 32
- Citations by year
- 2026: 13 · 2025: 29 · 2024: 37 · 2023: 70 · 2022: 69 · 2021: 106 · 2020: 106 · 2019: 106 · 2018: 63 · 2017: 44 · 2016: 13 · 2015: 1
- FWCI
- 53.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Funders
- Agency for Science Technology and Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- emotion recognition, facial expression recognition, still images, transfer learning, convolutional neural networks, fine-tuning, ImageNet pre-training, small datasets
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
