Leveraging recent advances in deep learning for audio-Visual emotion recognition
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Emotional expressions are the behaviors that communicate our emotional state or attitude to others. They are expressed through verbal and non-verbal communication. Complex human behavior can be understood by studying physical features from multiple modalities; mainly facial, vocal and physical gestures. Recently, spontaneous multi-modal emotion recognition has been extensively studied for human behavior analysis. In this paper, we propose a new deep learning-based approach for audio-visual emotion recognition. Our approach leverages recent advances in deep learning like knowledge distillation and high-performing deep architectures. The deep feature representations of the audio and visual modalities are fused based on a model-level fusion strategy. A recurrent neural network is then used to capture the temporal dynamics. Our proposed approach substantially outperforms state-of-the-art approaches in predicting valence on the RECOLA dataset. Moreover, our proposed visual facial expression feature extraction network outperforms state-of-the-art results on the AffectNet and Google Facial Expression Comparison datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 3×
“We refer to this network as our ‘facial expression embedding network’, and we train it using knowledge distillation (Hinton et al. [2014]). Knowledge distillation is a two step process whereby a teacher network is trained on the task of interest, and then a (typically smaller) student network is trained on predictions…”
MethodInfluentialA concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785 · cited 12×
“Audio-only: our results (Table 3) show that our modified VG-Gish backbone feature extractor for audio segments performs well, CCCs of 0 .”
MethodInfluentialJoint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214 · cited 3×
“Visual features are extracted by taking face crops from the video sequence of interest using MTCNN and passing them to our student network (Section 3.1.2).”
MethodInfluentialMultimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · cited 1×
“Our starting point is the original VGGish model, pre-trained on the Audio Set dataset (Gemmeke et al. [2017]).”
MethodA Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Self-Training With Noisy Student Improves ImageNet Classification
Xie, Q., Luong, M.-T., Hovy, E., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01070 · Cited by 2,393
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952132 · Cited by 2,023
Universals and cultural differences in the judgments of facial expressions of emotion.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.53.4.712 · Cited by 1,763
Relational Knowledge Distillation
Park, W., Kim, D., Lu, Y., & Cho, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00409 · Cited by 1,681 · cited 2×
“A knowledge distillation loss (we specifically use ‘Relational Knowledge Distillation’ (Park et al. [2019]) for our loss function) is then calculated by comparing the output of a third output head in the student network to this 80-dimensional target vector.”
MethodA review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · cited 2×
“Several literature review papers survey existing approaches for multimodal emotion recognition (Rouast et al. [2019]; Baltrušaitis et al. [2018]; Zeng et al. [2008]; Poria et al. [2017]).”
BackgroundThe Handbook of Culture and Psychology
(2019) · doi:10.1093/oso/9780190679743.001.0001 · Cited by 952
Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771 · cited 2×
“The performances of the proposed approach have been evaluated using the REmote COLlaborative and a ff ective interactions (RECOLA) corpus (Ringeval et al. [2013]).”
MethodEnd-to-End Multimodal Emotion Recognition using Deep Neural Networks
Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2017) · arXiv · doi:10.48550/arxiv.1704.08619 · Cited by 752 · cited 2×
“In the second category, raw visual and audio signals are fed to the deep network (Tzarakis et al. [2017]; Tzirakis et al. [2018]; Basnet et al. [2019]).”
BackgroundMethodA Note On "A Concordance Correlation Coefficient to Evaluate Reproducibility"
Nickerson, C. A. E. (1997) · Biometrics · doi:10.2307/2533516 · Cited by 510
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346 · cited 1×
“Four types of strategy are reported in the literature: feature-level fusion, decision-level fusion, hybrid fusion and model-level fusion (Zhang et al. [2017]; Poria et al. [2017]).”
BackgroundLocal Learning With Deep and Handcrafted Features for Facial Expression Recognition
Georgescu, M.-I., Ionescu, R. T., & Popescu, M. (2019) · IEEE Access · doi:10.1109/access.2019.2917266 · Cited by 331
Rejaibi, E., Komaty, A., Mériaudeau, F., Agrebi, S., & Othmani, A. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103107 · Cited by 321
Self-citationEnd-to-End Speech Emotion Recognition Using Deep Neural Networks
Tzirakis, P., Zhang, J., & Schuller, B. W. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462677 · Cited by 277 · cited 3×
“Visual features are extracted by taking face crops from the video sequence of interest using MTCNN and passing them to our student network (Section 3.1.2).”
MethodInfluentialKim, D. H., Baddar, W. J., Jang, J., & Ro, Y. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2695999 · Cited by 257 · cited 1×
“It has been demonstrated that deep neural networks and especially recurrent neural networks are capable of capturing the temporal dynamics of such sequences (Kim et al. [2017]).”
BackgroundDeep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · cited 6×
“Several literature review papers survey existing approaches for multimodal emotion recognition (Rouast et al. [2019]; Baltrušaitis et al. [2018]; Zeng et al. [2008]; Poria et al. [2017]).”
BackgroundMethodInfluentialPrediction of asynchronous dimensional emotion ratings from audiovisual and physiological data
Ringeval, F., Eyben, F., Kroupi, E., Yuce, A., Thiran, J.-P., Ebrahimi, T., Lalanne, D., & Schuller, B. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.11.007 · Cited by 202 · cited 1×
“The performances of the proposed approach have been evaluated using the REmote COLlaborative and affective interactions (RECOLA) corpus (Ringeval et al. [2013]).”
MethodEfficient Facial Feature Learning with Wide Ensemble-Based Convolutional Neural Networks
Siqueira, H. L., Magg, S., & Wermter, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6037 · Cited by 175
He, L., Jiang, D., Yang, L., Pei, E., Wu, P., & Sahli, H. (2015) · Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2808196.2811641 · Cited by 147 · cited 1×
“Our deep model for audio-based emotion recognition is based on a modified version of the VGGish model (Hershey et al. [2017]).”
MethodA Compact Embedding for Facial Expression Similarity
Vemulapalli, R., & Agarwala, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00583 · Cited by 140 · cited 4×
“1) is almost identical to the model proposed in Agarwala and Vemulapalli [2019], the only difference is that we add an additional output head for the AffectNet loss.”
BackgroundMethodResultInfluentialSelf-Distillation Amplifies Regularization in Hilbert Space
Mobahi, H., Farajtabar, M., & Bartlett, P. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.05715 · Cited by 118 · cited 1×
“It has also been shown theoretically that self-distillation can improve a model’s performance via a regularization e ff ect (Mobahi et al. [2020]).”
BackgroundDimensional affect recognition using Continuous Conditional Random Fields
Baltrusaitis, T., Banda, N., & Robinson, P. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553785 · Cited by 108 · cited 1×
“In this family of approaches, multiple kernel learning (Chen et al. [2014]), and graphical models (Baltrušaitis et al.”
Emotion Recognition in the Wild with Feature Fusion and Multiple Kernel Learning
Chen, J., Chen, Z., Chi, Z., & Fu, H. (2014) · Proceedings of the 16th International Conference on Multimodal Interaction · doi:10.1145/2663204.2666277 · Cited by 105 · cited 1×
“In this family of approaches, multiple kernel learning (Chen et al. [2014]), and graphical models (Baltrušaitis et al. [2018, 2013]) have been studied, in addition to neural network-based approaches.”
AudVowelConsNet: A phoneme-level based deep CNN architecture for clinical depression diagnosis
Muzammel, M., Salam, H., Hoffmann, Y., Chétouani, M., & Othmani, A. (2020) · Machine Learning with Applications · doi:10.1016/j.mlwa.2020.100005 · Cited by 102 · cited 1×
“In the first, several handcrafted features are extracted from the video and audio signals and then fed to the deep neural network (Ringeval et al. [2015]; He et al. [2015]; Othmani et al. [2019]; Rejaibi et al. [2019]; Muzammel et al. [2020]).”
MethodSelf-citationTowards Robust Deep Neural Networks for Affect and Depression Recognition from Speech
Othmani, A., Kadoch, D., Bentounes, K. A., Rejaibi, E., Alfred, R., & Hadid, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68790-8_1 · Cited by 99 · cited 1×
“In the first, several handcrafted features are extracted from the video and audio signals and then fed to the deep neural network (Ringeval et al. [2015]; He et al. [2015]; Othmani et al. [2019]; Rejaibi et al. [2019]; Muzammel et al. [2020]).”
MethodSelf-citationStrength modelling for real-worldautomatic continuous affect recognition from audiovisual signals
Han, J., Zhang, Z., Cummins, N., Ringeval, F., & Schuller, B. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2016.11.020 · Cited by 53
Pei, E., Yang, L., Jiang, D., & Sahli, H. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344573 · Cited by 28
Estimation of affective dimensions using CNN-based features of audiovisual data
Basnet, R., Islam, M. T., Howlader, T., Rahman, S. M. M., & Hatzinakos, D. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.09.015 · Cited by 5 · cited 1×
“In the second category, raw visual and audio signals are fed to the deep network (Tzarakis et al. [2017]; Tzirakis et al. [2018]; Basnet et al. [2019]).”
Background
Cited by
Savchenko, A. V., Savchenko, L. V., & Makarov, I. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3188390 · Cited by 346
Jiang, R. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1049401 · Cited by 181
“, audio, vision and even electroencephalo-graph, using AC techniques (Schoneveld et al., 2021), which can feed back to teachers or ITSs for further monitoring and improving the EFL learning.”
MethodA Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations
Ma, H., Wang, J., Lin, H., Zhang, B., Zhang, Y., & Xu, B. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3271019 · Cited by 180
“[49] appliedKDtofurtherimproveperformanceforfacialexpres-sion recognition.”
Automated emotion recognition: Current trends and future perspectives
Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Barua, P. D., Murugappan, M., Chakole, Y., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106646 · Cited by 172
Ad-Corre: Adaptive Correlation-Based Loss for Facial Expression Recognition in the Wild
Fard, A. P., & Mahoor, M. H. (2022) · IEEE Access · doi:10.1109/access.2022.3156598 · Cited by 157
Metadata
- Authors
- Schoneveld, L., Othmani, A., & Abdelkawy, H.
- Venue
- Pattern Recognition Letters
- Publisher
- Elsevier BV
- Published
- 2021-06
- Volume
- 146
- Pages
- 1-7
- ISSN
- 0167-8655 (ISSN-L)
- OpenAlex
- W3137028092
- Cited by
- 237
- References
- 37
- FWCI
- 41.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.