Modeling multimodal cues in a deep learning-based framework for emotion recognition in the wild
Open accessAbstract
In this paper, we propose a multimodal deep learning architecture for emotion recognition in video regarding our participation to the audio-video based sub-challenge of the Emotion Recognition in the Wild 2017 challenge. Our model combines cues from multiple video modalities, including static facial features, motion patterns related to the evolution of the human expression over time, and audio information. Specifically, it is composed of three sub-networks trained separately: the first and second ones extract static visual features and dynamic patterns through 2D and 3D Convolutional Neural Networks (CNN), while the third one consists in a pretrained audio network which is used to extract useful deep acoustic signals from video. In the audio branch, we also apply Long Short Term Memory (LSTM) networks in order to capture the temporal evolution of the audio features. To identify and exploit possible relationships among different modalities, we propose a fusion network that merges cues from the different modalities in one representation. The proposed architecture outperforms the challenge baselines (38.81 % and 40.47 %): we achieve an accuracy of 50.39 % and 49.92 % respectively on the validation and the testing data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SoundNet, a 1D CNN trained with unlabeled video, has been shown to perform well in SER even without fine-tuning, and was featured in a challenge-winning submission.
“SoundNet [138], a 1D CNN trained with unlabeled video, has been shown to perform well in SER even without fine-tuning [139], and was featured in a challenge-winning submission [17].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [139]
In end-to-end training for spatial and temporal feature learning, the problem of limited labeled data becomes especially noticeable.
“however in this setting the problem of limited labeled data becomes especially noticeable [63], [139].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [139]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
(2014) · Cited by 33,419 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · see citing works
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382 · see citing works
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214 · see citing works
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479 · see citing works
NetVLAD: CNN Architecture for Weakly Supervised Place Recognition
Arandjelović, R., Gronát, P., Torii, A., Pajdla, T., & Šivic, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.572 · Cited by 3,441 · see citing works
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv.org · doi:10.48550/arxiv.1409.2329 · Cited by 3,086 · see citing works
Aggregating local descriptors into a compact image representation
Jeǵou, H., Douze, M., Schmid, C., & Pérez, P. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540039 · Cited by 2,679 · see citing works
NetVLAD: CNN architecture for weakly supervised place recognitionPreprint
Arandjelović, R., Gronát, P., Torii, A., Pajdla, T., & Šivic, J. (2015) · arXiv (Cornell University) · Cited by 1,790 · see citing works
Speech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-57 · Cited by 824 · see citing works
Collecting Large, Richly Annotated Facial-Expression Databases from Movies
Dhall, A., Goecke, R., Lucey, S., & Gedeon, T. (2012) · IEEE Multimedia · doi:10.1109/mmul.2012.26 · Cited by 664 · see citing works
SoundNet: Learning Sound Representations from Unlabeled VideoPreprint
Aytar, Y., Vondrick, C., & Torralba, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09001 · Cited by 658 · see citing works
The eNTERFACE’05 Audio-Visual Emotion Database
Martin, O., Kotsia, I., Macq, B., & Pitas, I. (2006) · 22nd International Conference on Data Engineering Workshops (ICDEW'06) · doi:10.1109/icdew.2006.145 · Cited by 615 · see citing works
Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568 · see citing works
Video-based emotion recognition using CNN-RNN and C3D hybrid networks
Fan, Y., Lu, X., Li, D., & Liu, Y. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2997632 · Cited by 556 · see citing works
Anticipating Visual Representations from Unlabeled Video
Vondrick, C., Pirsiavash, H., & Torralba, A. B. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.18 · Cited by 538 · see citing works
Speech emotion recognition using convolutional and Recurrent Neural Networks
Lım, W., Jang, D., & Lee, T. (2016) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2016.7820699 · Cited by 421 · see citing works
Speech Emotion Recognition Using CNN
Huang, Z.-W., Dong, M., Mao, Q., & Zhan, Y. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654984 · Cited by 416 · see citing works
Deep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401 · see citing works
Combining modality specific deep neural networks for emotion recognition in video
Kahou, S. E., Pal, C., Bouthillier, X., Froumenty, P., Gülçehre, Ç., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R. C., et al. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531745 · Cited by 391 · see citing works
Recurrent Neural Networks for Emotion Recognition in Video
Kahou, S. E., Michalski, V., Konda, K., Memisevic, R., & Pal, C. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2830596 · Cited by 386 · see citing works
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346 · see citing works
Video-based emotion recognition in the wild using deep transfer learning and score fusion
Kaya, H., Gürpınar, F., & Salah, A. A. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.012 · Cited by 235 · see citing works
From individual to group-level emotion recognition: EmotiW 5.0
Dhall, A., Goecke, R., Ghosh, S., Joshi, J., Hoey, J., & Gedeon, T. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3143004 · Cited by 218 · see citing works
(2015) · doi:10.1145/2808196.2811642 · Cited by 178 · see citing works
Recurrent Dropout Without Memory LossPreprint
Semeniuta, S., Severyn, A., & Barth, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.05118 · Cited by 162 · see citing works
He, L., Jiang, D., Yang, L., Pei, E., Wu, P., & Hichem Sahli, B. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811641 · Cited by 149 · see citing works
HoloNet: towards robust emotion recognition in the wild
Yao, A., Cai, D., Hu, P., Wang, S., Sha, L., & Chen, Y. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2997639 · Cited by 115 · see citing works
Challenges in Representation Learning: A report on three machine learning contestsPreprint
Goodfellow, I., Erhan, D., Carrier, P. L., Courville, A. C., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D., et al. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.0414 · Cited by 114 · see citing works
Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks
Chen, S., & Jin, Q. (2015) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2808196.2811638 · Cited by 111 · see citing works
Multimodal Deep Convolutional Neural Network for Audio-Visual Emotion Recognition
Zhang, S., Zhang, S. S., Huang, T., & Gao, W. (2016) · International Conference on Multimedia Retrieval · doi:10.1145/2911996.2912051 · Cited by 107 · see citing works
Cognitive penetrability and emotion recognition in human facial expressions
Marchi, F., & Newen, A. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00828 · Cited by 84 · see citing works
Harnessing A.I. for Augmenting Creativity
Smith, J. R., Joshi, D., Huet, B., Hsu, W. H., & Cota, J. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3127906 · Cited by 67 · see citing works
Self-citationAudio and face video emotion recognition in the wild using deep neural networks and small datasets
Ding, W., Xu, M., Huang, D., Lin, W., Dong, M., Yu, X., & Li, H. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2997637 · Cited by 51 · see citing works
Multi-scale Temporal Modeling for Dimensional Emotion Recognition in Video
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2014) · International Workshop on Audio/Visual Emotion Challenge (AVEC) · doi:10.1145/2661806.2661811 · Cited by 39 · see citing works
Specificity of facial emotion recognition impairments in patients with multi-episode schizophrenia
Barkhof, E., de Sonneville, L. M. J., Meijer, C. J., & de Haan, L. (2015) · Schizophrenia Research Cognition · doi:10.1016/j.scog.2015.01.001 · Cited by 38 · see citing works
Chao, L., Tao, J., Yang, M., Li, Y., & Wen, Z. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472178 · Cited by 31 · see citing works
Towards multimodal emotion recognition
Paleari, M., Huet, B., & Chellali, R. (2010) · ACM International Conference on Image and Video Retrieval (CIVR) · doi:10.1145/1816041.1816069 · Cited by 27 · see citing works
Self-citationHargreaves, A., Mothersill, O., Anderson, M., Lawless, S., Corvin, A. P., & Donohoe, G. J. (2016) · Neuroscience Letters · doi:10.1016/j.neulet.2016.09.017 · Cited by 25 · see citing works
How Deep Neural Networks Can Improve Emotion Recognition on Video DataPreprint
Khorrami, P., Paine, T. L., Brady, K. J., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07377 · Cited by 10 · see citing works
EURECOM@MediaEval 2017: Media Genre Inference for Predicting Media Interestingness.
Ahmed, O. B., Wacker, J., Gaballo, A., et al. (2017) · MediaEval · Cited by 9 · see citing works
Self-citationAENet: Learning Deep Audio Features for Video AnalysisPreprint
Takahashi, N., Gygli, M., & Gool, L. V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.00599 · Cited by 5 · see citing works
DEEP FEATURES FOR MULTIMODAL EMOTION CLASSIFICATIONPreprint
Tiwari, S. N., Duong, N. Q. K., Lefebvre, F., et al. (2016) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Qirong Mao, Ming Dong, Zhengwei Huang, and Yongzhao Zhan. 2014. Learning Salient Features for Speech Emotion Recognition Using Convolutional Neural Networks. IEEE Trans. Multimedia 16 (2014), 2203-2213.
- O. Martin, I. Kotsia, B. Macq, and I. Pitas. 2006. The eNTERFACE'05 Audio-Visual Emotion Database. In International Conference on Data Engineering Workshops.
- Fabien Ringeval, Björn Schuller, Michel Valstar, Shashank Jaiswal, Erik Marchi, Denis Lalanne, Roddy Cowie, and Maja Pantic. 2015. AV+EC 2015: The First Affect Recognition Challenge Bridging Across Audio, Video, and Physiological Data. In International Workshop on Audio/Visual Emotion Challenge.
- Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke. 2016. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. In Inter- national Conference on Learning Representations Workshops.
Cited by
Sorted by citations
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142 · see citing works
- Authors
- Pini, S., Ahmed, O. B., Cornia, M., Baraldi, L., Cucchiara, R., & Huet, B.
- Affiliations
- University of Modena and Reggio Emilia (IT)
- EURECOM (FR)
- EURECOM, France
- Venue
- Proceedings of the 19th ACM International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2017-11-03
- Pages
- 536-543
- OpenAlex
- W2767488767
- Cited by
- 44
- References
- 49
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 3 · 2023: 6 · 2022: 7 · 2021: 6 · 2020: 7 · 2019: 7 · 2018: 6
- FWCI
- 6.28
- Type
- conference paper
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- HAL (submittedVersion)
- Graduate School and Research Center in Digital Science (EURECOM) (submittedVersion)
- Funders
- Bpifrance
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal fusion, facial expressions, 3D convolutional neural networks, 2D convolutional neural network, LSTM temporal modeling, fusion network
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
