The DOI matches its Crossref record; 31 of the 32 references it cites were found in a registry or an index.
Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning
Abstract
This paper presents the techniques employed in our team's submissions to the 2015 Emotion Recognition in the Wild contest, for the sub-challenge of Static Facial Expression Recognition in the Wild. The objective of this sub-challenge is to classify the emotions expressed by the primary human subject in static images extracted from movies. We follow a transfer learning approach for deep Convolutional Neural Network (CNN) architectures. Starting from a network pre-trained on the generic ImageNet dataset, we perform supervised fine-tuning on the network in a two-stage process, first on datasets relevant to facial expressions, followed by the contest's dataset. Experimental results show that this cascading fine-tuning approach achieves better results, compared to a single stage fine-tuning with the combined datasets. Our best submission exhibited an overall accuracy of 48.5% in the validation set and 55.6% in the test set, which compares favorably to the respective 35.96% and 39.13% of the challenge baseline.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Since the number of labeled examples for FER remains relatively limited, large 11 models are likely to overfit the data [106], [107], [89], [108].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [107]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745
Challenges in Representation Learning: A Report on Three Machine Learning Contests
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-42051-1_16 · Cited by 1,719
Raina, R., Battle, A. J., Lee, H., Packer, B., & Ng, A. Y. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273592 · Cited by 1,534
Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition
Sarıyanidi, E., Güneş, H., & Cavallaro, A. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2366127 · Cited by 702
Challenges in representation learning: A report on three machine learning contests
Goodfellow, I. J., Erhan, D., Luc Carrier, P., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.005 · Cited by 697
Collecting Large, Richly Annotated Facial-Expression Databases from Movies
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2012) · IEEE Multimedia · doi:10.1109/mmul.2012.26 · Cited by 658
Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2011) · 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) · doi:10.1109/iccvw.2011.6130508 · Cited by 541 · cited 1×
“Images are selected from movies, in a semi-automated way, via a system based on subtitles [5, 6].”
MethodCombining modality specific deep neural networks for emotion recognition in video
Kahou, S. E., Pal, C., Bouthillier, X., Froumenty, P., Gülçehre, Ç., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R. C., et al. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531745 · Cited by 391
Video and Image based Emotion Recognition Challenges in the Wild: EmotiW 2015
Dhall, A., Ramana Murthy, O. V., Goecke, R., Joshi, J., & Gedeon, T. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2829994 · Cited by 305
Combining Multiple Kernel Methods on Riemannian Manifold for Emotion Recognition in the Wild
Liu, M., Wang, R., Li, S., Shan, S., Huang, Z., & Chen, X. (2014) · Proceedings of the 16th International Conference on Multimodal Interaction · doi:10.1145/2663204.2666274 · Cited by 202
Proceedings of the 38th Annual Conference of the Cognitive Science Society, CogSci 2016
Räsänen, O. J., Nagamine, T., & Mesgarani, N. (2016) · Cited by 175
la Torre, F. D., Chu, W.-S., Xiong, X., Vicente, F., Ding, X., & Cohn, J. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163082 · Cited by 125
Multiple kernel learning for emotion recognition in the wild
Sikka, K., Dykstra, K., Sathyanarayana, S., Littlewort, G., & Bartlett, M. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531741 · Cited by 119
Context-Sensitive Dynamic Ordinal Regression for Intensity Estimation of Facial Action Units
Rudovic, O. O., Pavlović, V., & Pantić, M. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2356192 · Cited by 111
Emotion Recognition in the Wild with Feature Fusion and Multiple Kernel Learning
Chen, J., Chen, Z., Chi, Z., & Fu, H. C. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2666277 · Cited by 104
Sun, B., Li, L., Zuo, T., Chen, Y., Zhou, G., & Wu, X. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2666272 · Cited by 71
Partial least squares regression on grassmannian manifold for emotion recognition
Liu, M., Wang, R., Huang, Z., Shan, S., & Chen, X. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531738 · Cited by 62
Recognition of facial emotion in low vision: A flexible usage of facial features
Boucart, M., DINON, J.-F., Despretz, P., Desmettre, T. J., Hladiuk, K., & Oliva, A. (2008) · Visual Neuroscience · doi:10.1017/s0952523808080656 · Cited by 55
Learning realistic facial expressions from web images
Yu, K., Wang, Z., Li, Z., Wang, J., Chi, Z., & Feng, D. D. (2013) · Pattern Recognition · doi:10.1016/j.patcog.2013.01.032 · Cited by 36
Zhang, L., Tjondronegoro, D., & Chandran, V. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.09.005 · Cited by 27
Keeping it real: Recognizing expressions in real compared to schematic faces
Prazak, E. R., & Burgund, E. D. (2014) · Visual Cognition · doi:10.1080/13506285.2014.914991 · Cited by 15
Is Facial Expression Processing Holistic?
Omigbodun, A., & Cottrell, G. W. (2013) · eScholarship (California Digital Library) · Cited by 3
Flagged first, then by citations
1 of the 32 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- J. R. S. Widen and A. Brooks . Anger and disgust: Discrete or overlapping categories ? In Proceedings of the 2004 APS Annual Convention , 2004 . J. R. S. Widen and A. Brooks. Anger and disgust: Discrete or overlapping categories? In Proceedings of the 2004 APS Annual Convention, 2004.
Cited by
Sorted by citations
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 2,175
“Instead of directly using the pretrained or fine-tuned models to extract features on the target dataset, a multistage fine-tuning strategy [67] can achieve better performance; after the first-stage fine-tuning using FER2013 on pretrained”
MethodReliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild
Li, S., Deng, W., & Du, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.277 · Cited by 1,470
“In EmotiW 2015 [8], more competitors have tried deep learning meth-ods: transfer learning was used to solve the problem of small database in [32], hierarchical committee of multi-column DCNNs in [20] gained the best result on SFEW database, LBP features combined with DCNNs structure were proposed…”
BackgroundDeep Learning vs. Traditional Computer Vision
O’Mahony, N., Campbell, S., Carvalho, A. L. D., Harapanahalli, S., Velasco-Hernandez, G., Krpálková, L., Riordan, D., & Walsh, J. L. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-17795-9_10 · Cited by 1,173
A Brief Review of Facial Emotion Recognition Based on Visual Information
Ko, B. C. (2018) · Sensors · doi:10.3390/s18020401 · Cited by 790
“Therefore, this sub-section briefly introduces some popular databases related to FER consisting of 2D and 3D video sequences and still images:”
BackgroundLi, S., & Deng, W. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2868382 · Cited by 688
““SFEW best [33]”, “SFEW second [77]” and “SFEW third [54]” indicate the best single model result of the winner, the runner-up and the second runner-up in EmotiW 2015, respectively.”
BackgroundMethodInfluential
Metadata
- Authors
- Ng, H.-W., Nguyen, V. D., Vonikakis, V., & Winkler, S.
- Affiliations
- University of Illinois at Urbana-Champaign (Singapore), Singapore, Singapore
- University of Illinois Urbana-Champaign (US)
- Venue
- Proceedings of the 2015 ACM on International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2015-11-09
- Pages
- 443-449
- OpenAlex
- W2253728219
- Cited by
- 657
- References
- 32
- Citations by year
- 2026: 13 · 2025: 29 · 2024: 37 · 2023: 70 · 2022: 69 · 2021: 106 · 2020: 106 · 2019: 106 · 2018: 63 · 2017: 44 · 2016: 13 · 2015: 1
- FWCI
- 53.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Funders
- Agency for Science Technology and Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- emotion recognition, facial expression recognition, still images, transfer learning, convolutional neural networks, fine-tuning, ImageNet pre-training, small datasets
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
