Audio and face video emotion recognition in the wild using deep neural networks and small datasets
Abstract
This paper presents the techniques used in our contribution to Emotion Recognition in the Wild 2016’s video based sub-challenge. The purpose of the sub-challenge is to classify the six basic emotions (angry, sad, happy, surprise, fear & disgust) and neutral. Compared to earlier years’ movie based datasets, this year’s test dataset introduced reality TV videos containing more spontaneous emotion. Our proposed solution is the fusion of facial expression recognition and audio emotion recognition subsystems at score level. For facial emotion recognition, starting from a network pre-trained on ImageNet training data, a deep Convolutional Neural Network is fine-tuned on FER2013 training data for feature extraction. The classifiers, i.e., kernel SVM, logistic regression and partial least squares are studied for comparison. An optimal fusion of classifiers learned from different kernels is carried out at the score level to improve system performance. For audio emotion recognition, a deep Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) is trained directly using the challenge dataset. Experimental results show that both subsystems individually and as a whole can achieve state-of-the art performance. The overall accuracy of the proposed approach on the challenge test dataset is 53.9%, which is better than the challenge baseline of 40.47% .
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310 · see citing works
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · see citing works
Zeiler, M. D., Krishnan, D., Taylor, G. W., & Fergus, R. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539957 · Cited by 1,753 · see citing works
Recurrent neural networks and robust time series prediction
Connor, J. T., Martin, R. D., & Atlas, L. E. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279188 · Cited by 1,316 · see citing works
Recent developments in openSMILE, the munich open-source multimedia feature extractor
Eyben, F., Weninger, F. J., Gross, F., & Schuller, B. W. (2013) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2502081.2502224 · Cited by 1,265 · see citing works
Collecting Large, Richly Annotated Facial-Expression Databases from Movies
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2012) · IEEE Multimedia · doi:10.1109/mmul.2012.26 · Cited by 664 · see citing works
Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning
Ng, H.-W., Nguyen, V. D., Vonikakis, V., & Winkler, S. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830593 · Cited by 657 · cited 2× · see citing works
“First, we applied the CNN fine-tuning on small dataset strategy [19] and found that the extracted CNN features are also effective for video facial emotion recognition.”
MethodEffective face frontalization in unconstrained images
Hassner, T., Harel, S., Paz, E., & Enbar, R. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299058 · Cited by 613 · see citing works
Grassmann discriminant analysis
Hamm, J., & Lee, D. D. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390204 · Cited by 593 · see citing works
Valstar, M., Gratch, J. S., Schuller, B. W., Ringeval, F., Lalanne, D., Torres, M. T., Scherer, S. J., Stratou, G., Cowie, R., & Pantić, M. (2016) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2988257.2988258 · Cited by 516 · see citing works
Covariance discriminative learning: A natural and efficient approach to image set classification
Wang, R., Guo, H., Davis, L. S., et al. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247965 · Cited by 450 · see citing works
Computer Vision Using Local Binary Patterns
Pietikäinen, M., Hadid, A., Zhao, G., et al. (2011) · Computational imaging and vision · doi:10.1007/978-0-85729-748-8_14 · Cited by 446 · see citing works
Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011
Anagnostopoulos, C.-N., Iliou, T., & Giannoukos, I. (2015) · Artificial Intelligence Review · doi:10.1007/s10462-012-9368-5 · Cited by 428 · see citing works
Recurrent Neural Networks for Emotion Recognition in Video
Kahou, S. E., Michalski, V., Konda, K., Memisevic, R., & Pal, C. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2830596 · Cited by 386 · see citing works
The first facial expression recognition and analysis challenge
Valstar, M. F., Jiang, B., Mehu, M., Pantic, M., & Scherer, K. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771374 · Cited by 385 · see citing works
Video and Image based Emotion Recognition Challenges in the Wild: EmotiW 2015
Dhall, A., Ramana Murthy, O. V., Goecke, R., Joshi, J., & Gedeon, T. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2829994 · Cited by 305 · see citing works
FERA 2015 - second Facial Expression Recognition and Analysis challenge
Valstar, M. F., Almaev, T., Girard, J. M., McKeown, G., Mehu, M., Yin, L., Pantic, M., & Cohn, J. F. (2015) · 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2015.7284874 · Cited by 252 · see citing works
Emotion recognition using PHOG and LPQ features
Dhall, A., Asthana, A., Goecke, R., & Gedeon, T. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771366 · Cited by 221 · see citing works
Emotion Recognition In The Wild Challenge 2014
Dhall, A., Goecke, R., Joshi, J., Sikka, K., & Gedeon, T. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2666275 · Cited by 216 · see citing works
Combining Multiple Kernel Methods on Riemannian Manifold for Emotion Recognition in the Wild
Liu, M., Wang, R., Li, S., Shan, S., Huang, Z., & Chen, X. (2014) · Proceedings of the 16th International Conference on Multimodal Interaction · doi:10.1145/2663204.2666274 · Cited by 202 · see citing works
Introducing CURRENNT: the Munich open-source CUDA recurrent neural network toolkit
Weninger, F. J., Bergmann, J. M., & Schuller, B. W. (2015) · Spiral (Imperial College London) · Cited by 161 · see citing works
He, L., Jiang, D., Yang, L., Pei, E., Wu, P., & Hichem Sahli, B. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811641 · Cited by 149 · see citing works
EmotiW 2016: video and group-level emotion recognition challenges
Dhall, A., Goecke, R., Joshi, J., Hoey, J., & Gedeon, T. (2016) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2993148.2997638 · Cited by 147 · see citing works
Zazo, R., Lozano-Díez, A., Gónzalez-Domínguez, J., Toledano, D. T., & González-Rodríguez, J. (2016) · PLoS ONE · doi:10.1371/journal.pone.0146917 · Cited by 144 · see citing works
Kim, B.-K., Lee, H., Roh, J., & Lee, S.-Y. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2830590 · Cited by 126 · see citing works
Challenges in Representation Learning: A report on three machine learning contestsPreprint
Goodfellow, I., Erhan, D., Carrier, P. L., Courville, A. C., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D., et al. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.0414 · Cited by 114 · see citing works
Finding Celebrities in Billions of Web Images
Zhang, X., Zhang, L., Wang, X., & Shum, H. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2186121 · Cited by 101 · see citing works
Kernel Learning for Extrinsic Classification of Manifold Features
Vemulapalli, R., Pillai, J. K., & Chellappa, R. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.233 · Cited by 101 · see citing works
Capturing AU-Aware Facial Features and Their Latent Relations for Emotion Recognition in the Wild
Yao, A., Shao, J., Ma, N., & Chen, Y. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2830585 · Cited by 96 · see citing works
A Novel Earth Mover's Distance Methodology for Image Matching with Gaussian Mixture Models
Li, P., Wang, Q., & Zhang, L. (2013) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.212 · Cited by 90 · see citing works
Robust Unsupervised Arousal Rating:A Rule-Based Framework withKnowledge-Inspired Vocal Features
Bone, D., Lee, C.-C., & Narayanan, S. S. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2326393 · Cited by 67 · see citing works
Context dependent phone models for LSTM RNN acoustic modelling
Senior, A., Sak, H., & Shafran, I. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178839 · Cited by 55 · see citing works
Emotion Recognition in the Wild
Ringeval, F., Amiriparian, S., Eyben, F., Scherer, K. R., & Schuller, B. W. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2666271 · Cited by 30 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Modeling multimodal cues in a deep learning-based framework for emotion recognition in the wild
Pini, S., Ahmed, O. B., Cornia, M., Baraldi, L., Cucchiara, R., & Huet, B. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143006 · Cited by 44 · see citing works
- Authors
- Ding, W., Xu, M., Huang, D., Lin, W., Dong, M., Yu, X., & Li, H.
- Affiliations
- Central China Normal University (CN)
- University of British Columbia (CA)
- Agency for Science, Technology and Research (SG)
- Nanyang Technological University (SG)
- Venue
- Proceedings of the 18th ACM International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2016-10-31
- Pages
- 506-513
- OpenAlex
- W2548128734
- Cited by
- 51
- References
- 38
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 4 · 2023: 5 · 2022: 3 · 2021: 10 · 2020: 6 · 2019: 11 · 2018: 1 · 2017: 5
- FWCI
- 6.56
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Funders
- National Natural Science Foundation of China (61272206)
- Open Projects Program of National Laboratory of Pattern Recognition (201600008)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- audio emotion recognition, facial expression recognition, multimodal fusion, score-level fusion, basic emotions, spontaneous emotions, convolutional neural networks, recurrent neural networks, kernel SVM, small datasets
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
