The DOI matches its DataCite record; 55 of the 55 references it cites were found in a registry or an index; 1 of the 7 sources its claims cite were checked for support.
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Open accessAbstract
Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a Convolutional Neural Network (CNN) to extract features from the speech, while for the visual modality a deep residual network (ResNet) of 50 layers. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, Long Short-Term Memory (LSTM) networks are utilized. The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an end-to-end multimodal emotion recognition system that learns from raw speech and visual signals using a CNN, ResNet-50, and LSTM layers, trained to directly maximize concordance correlation coefficient (ρc). On the RECOLA database, the unimodal speech and visual models outperform several handcrafted-feature baselines, and the multimodal model improves valence prediction over prior multimodal systems.
Key findings
- For speech, the proposed raw-signal model optimized with ρc achieved test ρc of .699 for arousal and .311 for valence, compared with BLSTM eGeMAPS (.316 arousal, .195 valence) and BLSTM ComParE (.382 arousal, .187 valence); the MSE-optimized version achieved .684 arousal and .249 valence.
- For visual modality, the proposed raw-signal model achieved test ρc of .435 for arousal and .620 for valence, outperforming listed models in valence, e.g., baseline (.272 arousal, .507 valence) and Han et al. (.265 arousal, .394 valence).
- In multimodal analysis, the proposed model outperformed OA RVM-SR and Han et al. in valence “with high magnitude,” while OA RVM-SR produced the best arousal result; the provided table does not show the proposed multimodal ρc values.
- Sequence-length ablation found best speech performance at length 150 (arousal .744, valence .369) and best visual performance at length 300 (arousal .193, valence .496), though the multimodal network was trained with length 150; gate activations correlated with RMS energy range (ρ = .83), loudness (ρ = .73), and mean fundamental frequency (ρ = .72).
Methods
- Architecture: speech CNN operates on raw waveform, visual ResNet-50 operates on cropped face pixel intensities, and extracted features are concatenated into a 1920-dimensional vector (1280 speech + 640 visual) feeding a 2-layer LSTM with 256 cells each; unimodal networks are pretrained separately before end-to-end multimodal training.
- Data: RECOLA database, 46 French-speaking participants, 9.5 h of multimodal recordings, 5 minutes each in dyads; split into train (16 subjects), validation (15), and test (15), stratified by gender and age; 6 annotators. Audio is segmented into 6 s sequences at 16 kHz and further into 150 subsequences for 40 ms annotation granularity; visual images are 96×96.
- Training/evaluation: Adam optimizer with fixed learning rate 10⁻⁴, dropout p = 0.5, mini-batch 25 for audio and 2 for video, 60 epochs; metric is ρc; postprocessing includes median filtering, centering, scaling, and time-shifting selected on validation ρc.
Limitations
- The paper states that the OA RVM-SR multimodal comparison was trained using both training and validation sets, whereas this model used only the training set, and that OA RVM-SR used geometric features requiring accurate facial landmark tracking while this model operated on raw pixels; the authors expect their results could improve with similar strategies.
- The paper states hardware limitations forced a small video mini-batch of size 2, and notes memory and training instability concerns with very large recurrent sequences.
- Evident from the text: only two other multimodal models were found for comparison, RECOLA includes ECG and EDA modalities but this study uses only audio and video, and the provided multimodal results table lacks the proposed model’s exact ρc values.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
19 Own result · 5 No full text of the sources · 5 No basis given · 3 Own method · 3 Own interpretation · 1 Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the arousal dimension the OA RVM-SR produces the best results.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The only exception is the Video CNN-L4 model which performs better in the arousal dimension when appearance features are used.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This plot shows that certain cells of the model are very sensitive to different features conveyed in the original speech wave form.”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results show that our model outperforms the other models in the test set when predicting arousal dimension.”
plainn = 27- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As expected visual modality benefits the models in the valence dimension.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In all of the experiments, our model outperforms the designed features in terms of ρ c .”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we study the gate activations of the recurrent layers in the speech modality and find cells that are highly correlated with prosodic features that were always assumed to cause arousal.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our model outperforms the other two models in the valence dimension with high magnitude.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 3
Every source the paper cites for this claim was checked and supports it.
In the paper
“It is well accepted amongst the research community that certain acoustic and prosodic features play an important role in recognising the affective state [45].”
What the sources say
- Supportedcites [45] [45] K. Scherer, 'Vocal communication of emotion: A review of research paradigms,' Speech Communication , vol. 40, no. 1-2, pp. 227-256, April 2003. - source
Section 2.3, “Inference studies” (page not provided), reports that acoustic measures including pitch, energy, and duration explained a large proportion of variation in listeners’ emotion judgments. It also reports that manipulating vocal features affected judgments of a speaker’s emotional state, supporting the claim that such features matter for recognition. (LLM, conf=0.94)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
hedgedn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Some of these features, such as the mean of the fundamental frequency (F0), mean speech intensity, loudness, as well as pitch range [46], should thus be captured by our model.”
What the sources say
- Not checked — no full text of the sourcecites [46] [46] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan et al. , 'The geneva minimalistic acoustic parameter set (gemaps) for voice research and - affective computing,' IEEE Transactions on Affective Computing , vol. 7, no. 2, pp. 190-202, 2016. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
In the paper
“The task of recognising emotions is challenging because human emotions lack of temporal boundaries and different individuals express emotions in different ways [3].”
What the sources say
- Not checked — no full text of the sourcecites [3] [3] C.-N. Anagnostopoulos, T. Iliou, and I. Giannoukos, 'Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,' Artificial Intelligence Review , vol. 43, no. 2, pp. 155-177, 2015. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to perform a fair comparison we apply the proposed method on the test set of the AVEC challenge.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset is split in three partitions -train (16 subjects), validation (15 subjects) and test (15 subjects) -by stratifying (i.e., balancing) the gender and the age of the speakers.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Any of these post-processing steps is kept when an improvement is observed on the ρ c of the validation set, and applied then with the same configuration on the test partition.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
The proposed work is the first in literature to apply an end-to-end model for audio-visual emotion recognition.
“To our knowledge this is the first work in literature that applies such an end-to-end model for audio visual emotion recognition task.”
The paper relies on this without support.
Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
“The speech signals convey information about the affective state either explicitly, i.e., by linguistic means, or implicitly, i.e., by acoustic or prosodic cues.”
The paper relies on this without support.
Visual modality more easily predicts the valence dimension rather than the arousal.
“Visual modality has been shown to more easily predict the valence dimension rather than the arousal.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Dropout is applied in fullyconnected (reported in 43% of studies, e.g., [135]) and convolutional layers (14% of studies, e.g., [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We see both CNN-RNN (e.g., [18], [90]) and CNN-LSTM (e.g., [108], [105], [64]) architectures, with CNN-LSTM being the more frequent choice among the reviewed studies.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Global temporal modeling is found to lead to improved accuracies when compared with simpler methods such as pooling of spatial features (e.g., [152], [90], [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Applications of similar architectures based on the raw waveform have also been attempted [54], [108], showing that end-to-end learning can outperform shallow models.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“End-to-end training of the entire CNN-LSTM system addresses this problem, and can lead to accuracy improvements [108].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We are starting to see studies implementing end-toend training for such models [54], [108]”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Learning from raw audiovisual data with two CNNs, Tzirakis et al. [108] used a two-layer LSTM network for feature fusion, which was found to outperform the state of the art.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Similarly, for SER, Tzirakis et al. [108] found that LSTM cells learn representations similar to well-known prosodic features.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ I. INTRODUCTION
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
§ B. Speech Modality
“| Baseline [29] | eGeMAPS | .648 (.796) | .375 (.455) | | RVM [30] | eGeMAPS | - (.750) | - (.396) | | Audio BN-Multi [33] | Mixed | - (.833) | - (.503) | | Brady et al [32] | MFCC | - (.846) | - (.450) | | Weber et al. [31] | eGeMAPS | - (.793) | - (.456) | | Somandepalli et al. [34] | Mixed | - (.800) | - (.448) |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
(2010) · Cited by 14,134
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Research works printed but not linked: 1
- [37][37] F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data,' Pattern Recognition Letters , vol. 66, pp. 22-30, November 2015. -
Cited by
Sorted by citations
Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection
Schmidt, P., Reiss, A., Dürichen, R., Marberger, C., & Laerhoven, K. V. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3242985 · Cited by 1,508
EmotionMeter: A Multimodal Framework for Recognizing Human Emotions
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., & Cichocki, A. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2797176 · Cited by 1,276
Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806
Akçay, M. B., & Oğuz, K. (2019) · Speech Communication · doi:10.1016/j.specom.2019.12.001 · Cited by 743
Speech Emotion Recognition Using Deep Learning Techniques: A Review
Khalil, R. A., Jones, E., Babar, M. I., Jan, T., Zafar, M. H., & Alhussain, T. (2019) · IEEE Access · doi:10.1109/access.2019.2936124 · Cited by 702
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 713
Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos
Hazarika, D., Poria, S., Zadeh, A., Cambria, E., Morency, L., & Zimmermann, R. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1193 · Cited by 506
ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection
Hazarika, D., Poria, S., Mihalcea, R. F., Cambria, E., & Zimmermann, R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1280 · Cited by 471
Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM
Mustaqeem, Sajjad, M., & Kwon, S. (2020) · IEEE Access · doi:10.1109/access.2020.2990405 · Cited by 402
Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
Kollias, D., Tzirakis, P., Nicolaou, M. A., Papaioannou, A., Zhao, G., Schuller, B. W., Kotsia, I., & Zafeiriou, S. P. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01158-4 · Cited by 406
Self-citationEnd-to-End Audiovisual Speech Recognition
Petridis, S., Stafylakis, T., Ma, P., Cai, F., Tzimiropoulos, G., & Pantić, M. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461326 · Cited by 301
EEG-Based BCI Emotion Recognition: A Survey
Torres, E. P., Torres, E. A., Hernández-Álvarez, M., & Yoo, S. G. (2020) · Sensors · doi:10.3390/s20185083 · Cited by 292
End-to-End Speech Emotion Recognition Using Deep Neural Networks
Tzirakis, P., Zhang, J., & Schuller, B. W. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462677 · Cited by 289
Self-citationDeep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243
Investigating EEG-based functional connectivity patterns for multimodal emotion recognition
Wu, X., Zheng, W., Li, Z., & Lu, B. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac49a7 · Cited by 249
Leveraging recent advances in deep learning for audio-Visual emotion recognition
Schoneveld, L., Othmani, A., & Abdelkawy, H. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.03.007 · Cited by 243
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
Jiang, Y., Li, W., Hossain, M. S., Chen, M., Alelaiwi, A. A., & Al-Hammadi, M. H. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.06.019 · Cited by 224
Wearable-Based Affect Recognition—A Review
Schmidt, P., Reiss, A., Dürichen, R., & Laerhoven, K. V. (2019) · Sensors · doi:10.3390/s19194079 · Cited by 291
Deep learning-based late fusion of multimodal information for emotion classification of music video
Pandeya, Y. R., & Lee, J. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-08836-3 · Cited by 236
MLT-DNet: Speech emotion recognition using 1D dilated CNN based on multi-learning trick approach
Mustaqeem, M., & Kwon, S. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114177 · Cited by 205
Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion
Çimtay, Y., Ekmekcioǧlu, E., & Özhan, Ş. Ç. (2020) · IEEE Access · doi:10.1109/access.2020.3023871 · Cited by 207
Stress detection using deep neural networks
Li, R., & Liu, Z. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01299-4 · Cited by 213
Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network
Minaee, S., Minaei, M., & Abdolrashidi, A. (2021) · Sensors · doi:10.3390/s21093046 · Cited by 251
Music emotion recognition using convolutional long short term memory deep neural networks
Hızlısoy, S., Yıldırım, S., & TÜFEKCİ, Z. (2020) · Engineering Science and Technology an International Journal · doi:10.1016/j.jestch.2020.10.009 · Cited by 178
Automated emotion recognition: Current trends and future perspectives
Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Barua, P. D., Murugappan, M., Chakole, Y., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106646 · Cited by 172
Improved Deep Hybrid Networks for Urban Traffic Flow Prediction Using Trajectory Data
Duan, Z., Yang, Y., Zhang, K., Ni, Y., & Bajgain, S. (2018) · IEEE Access · doi:10.1109/access.2018.2845863 · Cited by 163
A survey of speech emotion recognition in natural environment
Fahad, M. S., Ranjan, A., Yadav, J., & Deepak, A. (2020) · Digital Signal Processing · doi:10.1016/j.dsp.2020.102951 · Cited by 167
Audiovisual emotion recognition in wild
Avots, E., Sapiński, T., Bachmann, M., & Kamińska, D. (2018) · Machine Vision and Applications · doi:10.1007/s00138-018-0960-9 · Cited by 166
Multimodal Data Fusion in Learning Analytics: A Systematic Review
Mu, S., Cui, M., & Huang, X. (2020) · Sensors · doi:10.3390/s20236856 · Cited by 180
Spontaneous Speech Emotion Recognition Using Multiscale Deep Convolutional LSTM
Zhang, S., Zhao, X., & Tian, Q. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2947464 · Cited by 156
A systematic literature review of speech emotion recognition approaches
Singh, Y. B., & Goel, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.028 · Cited by 156
Nguyen, A., Kremantzis, M., Essien, A., Petrounias, I., & Hosseini, S. (2024) · Journal of University Teaching and Learning Practice · doi:10.53761/caraaq92 · Cited by 146
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142
Transformer Encoder With Multi-Modal Multi-Head Attention for Continuous Affect Recognition
Chen, H., Jiang, D., & Hichem Sahli, B. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3037496 · Cited by 146
Zhao, Z., Bao, Z., Zhao, Y., Zhang, Z., Cummins, N., Ren, Z. X., & Schuller, B. W. (2019) · IEEE Access · doi:10.1109/access.2019.2928625 · Cited by 141
Self-citationKalateh, S., Estrada-Jimenez, L. A., Nikghadam-Hojjati, S., & Barata, J. (2024) · IEEE Access · doi:10.1109/access.2024.3430850 · Cited by 170
CNN and LSTM-Based Emotion Charting Using Physiological Signals
Dar, M. N., Akram, M. U., Khawaja, S. G., & Pujari, A. N. (2020) · Sensors · doi:10.3390/s20164551 · Cited by 137
An Active Learning Paradigm for Online Audio-Visual Emotion Recognition
Kansizoglou, I., Bampis, L., & Gasteratos, A. C. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2961089 · Cited by 126
Label-less Learning for Emotion Cognition
Chen, M., & Hao, Y. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2929071 · Cited by 124
Using transformers for multimodal emotion recognition: Taxonomies and state of the art review
Hazmoune, S., & Bougamouza, F. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108339 · Cited by 136
Integrating Deep and Shallow Models for Multi-Modal Depression Analysis—Hybrid Architectures
Yang, L., Jiang, D., & Hichem Sahli, B. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2870398 · Cited by 123
A multimodal hierarchical approach to speech emotion recognition from audio and text
Singh, P., Srivastava, R., Rana, K. P. S., & Kumar, V. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107316 · Cited by 116
A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition
Praveen, R. G., de Melo, W. C., Ullah, N., Aslam, M. H., Zeeshan, O., Denorme, T., Pedersoli, M., Koerich, A. L., Bacon, S., Cardinal, P., et al. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00278 · Cited by 115
Wu, Y., Mi, Q., & Gao, T. (2025) · Biomimetics · doi:10.3390/biomimetics10070418 · Cited by 115
X-Vectors Meet Emotions: A Study On Dependencies Between Emotion and Speaker Recognition
Pappagari, R. R., Wang, T., Villalba, J., Chen, N., & Dehak, N. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054317 · Cited by 111
Multi-Modal Domain Adaptation Variational Autoencoder for EEG-Based Emotion Recognition
Wang, Y., Qiu, S., Li, D., Du, C.-D., Lu, B.-L., & He, H.-G. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105515 · Cited by 92
Rizos, G., Hemker, K., & Schuller, B. W. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3358040 · Cited by 110
Self-citationEnd-to-end multimodal affect recognition in real-world environments
Tzirakis, P., Chen, J., Zafeiriou, S., & Schuller, B. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.10.011 · Cited by 105
Self-citationAsghar, M. A., Khan, M. J., Fawad, F., Amin, Y., Rizwan, M., Rahman, M., Badnava, S., & Mirjavadi, S. S. (2019) · Sensors · doi:10.3390/s19235218 · Cited by 105
Zhao, Z., Li, Q., Zhang, Z., Cummins, N., Wang, H., Tao, J., & Schuller, B. W. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.03.013 · Cited by 105
Self-citationZhao, Z., Bao, Z., Zhang, Z., Deng, J., Cummins, N., Wang, H., Tao, J., & Schuller, B. W. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2955012 · Cited by 110
Self-citationMulti-modal emotion recognition using EEG and speech signals
Wang, Q. H., Wang, M., Yang, Y., & Zhang, X. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105907 · Cited by 103
Context-aware Interactive Attention for Multi-modal Sentiment and Emotion Analysis
Chauhan, D. S., Akhtar, M. S., Ekbal, A., & Bhattacharyya, P. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1566 · Cited by 97
Fast Facial emotion recognition Using Convolutional Neural Networks and Gabor Filters
Zadeh, M. M. T., Imani, M., & Majidi, B. (2019) · International Conference on Knowledge-Based Engineering and Innovation · doi:10.1109/kbei.2019.8734943 · Cited by 97
Li, B., & Sano, A. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3397318 · Cited by 100
Li, J., Ji, D., Li, F., Zhang, M., & Liu, Y. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.370 · Cited by 97
Multiple classifiers in biometrics. Part 2: Trends and challenges
Fiérrez, J., Morales, A., Vera-Rodríguez, R., & Camacho, D. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.12.005 · Cited by 85
Attention-augmented End-to-end Multi-task Learning for Emotion Prediction from Speech
Zhang, Z., Wu, B., & Schuller, B. W. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682896 · Cited by 91
Self-citationWang, J. Z., Zhao, S., Wu, C., Adams, R. B., Newman, M. G., Shafir, T., & Tsachor, R. P. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3273517 · Cited by 109
Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data
Choi, W. Y., Song, K. Y., & Lee, C. W. (2018) · doi:10.18653/v1/w18-3304 · Cited by 88
Muzammel, M., Salam, H., & Othmani, A. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106433 · Cited by 91
Li, Z., Zhang, G., Wang, L., Wei, J., & Dang, J. (2023) · Journal of Neural Engineering · doi:10.1088/1741-2552/acb79e · Cited by 88
End-to-End Speech Emotion Recognition With Gender Information
Sun, T.-W. (2020) · IEEE Access · doi:10.1109/access.2020.3017462 · Cited by 88
Visual-audio emotion recognition based on multi-task and ensemble learning with multiple features
Hao, M., Cao, W., Liu, Z., Wu, M., & Xiao, P. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.01.048 · Cited by 83
Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data
Du, C., Du, C., Wang, H., Li, J., Zheng, W., Lu, B., & He, H. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240528 · Cited by 77
Attention-Enhanced Connectionist Temporal Classification for Discrete Speech Emotion Recognition
Zhao, Z., Bao, Z., Zhang, Z., Cummins, N., Wang, H., & Schuller, B. W. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1649 · Cited by 76
Self-citationMultimodal emotion recognition based on audio and text by using hybrid attention networks
Zhang, S., Yang, Y., Chen, C., Liu, R., Tao, X., Guo, W., Xu, Y., & Zhao, X. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105052 · Cited by 81
Sun, L., Lian, Z., Liu, B., & Tao, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102382 · Cited by 92
Shao, Z., Dou, W., & Pan, Y. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102113 · Cited by 79
An End-to-End Multimodal Voice Activity Detection Using WaveNet Encoder and Residual Networks
Ariav, I., & Cohen, I. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2901195 · Cited by 74
Han, J., Zhang, Z., Cummins, N., & Schuller, B. W. (2019) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2019.2901088 · Cited by 86
Self-citationTowards Robust Deep Neural Networks for Affect and Depression Recognition from Speech
Othmani, A., Kadoch, D., Bentounes, K. A., Rejaibi, E., Alfred, R., & Hadid, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68790-8_1 · Cited by 79
A blockchain-based secure Internet of medical things framework for stress detection
Qi, P., Chiaro, D., Giampaolo, F., & Piccialli, F. (2023) · Information Sciences · doi:10.1016/j.ins.2023.01.123 · Cited by 71
Tellamekala, M. K., Amiriparian, S., Schuller, B. W., André, E., Giesbrecht, T., & Valstar, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3325770 · Cited by 75
Self-citationEmotion Recognition by Fusing Time Synchronous and Time Asynchronous Representations
Wu, W., Zhang, C., & Woodland, P. C. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414880 · Cited by 70
Real-time emotional health detection using fine-tuned transfer networks with multimodal fusion
Sharma, A., Sharma, K., & Kumar, A. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-06913-2 · Cited by 69
Padi, S., Sadjadi, S. O., Manocha, D., & Sriram, R. D. (2022) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2022-57 · Cited by 82
Jaafar, N., & Lachiri, Z. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118523 · Cited by 67
Huang, K.-Y., Wu, C., Su, M.-H., & Kuo, Y.-T. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2803178 · Cited by 68
Lessons from infant learning for unsupervised machine learning
Zaadnoordijk, L., Besold, T. R., & Cusack, R. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00488-2 · Cited by 65
Multi-head attention fusion networks for multi-modal speech emotion recognition
Zhang, J., Xing, L., Tan, Z., Wang, H., & Wang, K. (2022) · Computers & Industrial Engineering · doi:10.1016/j.cie.2022.108078 · Cited by 66
EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddings
Han, J., Zhang, Z., Ren, Z. X., & Schuller, B. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2928297 · Cited by 64
Self-citationFusion Techniques for Utterance-Level Emotion Recognition Combining Speech and Transcripts
Sebastian, J., & Pierucci, P. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3201 · Cited by 63
Multimodal MR Image Synthesis Using Gradient Prior and Adversarial Learning
Liu, X., Yu, A., Wei, X., Pan, Z., & Tang, J. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3013418 · Cited by 62
Audio‐Textual Emotion Recognition Based on Improved Neural Networks
Cai, L., Hu, Y., Dong, J., & Sitong, Z. (2019) · Mathematical Problems in Engineering · doi:10.1155/2019/2593036 · Cited by 63
Audio–Visual Fusion for Emotion Recognition in the Valence–Arousal Space Using Joint Cross-Attention
Praveen, R. G., Cardinal, P., & Granger, É. (2023) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2022.3233083 · Cited by 62
Survey on multimodal approaches to emotion recognition
Aruna Gladys, A., & Vetriselvi, V. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126693 · Cited by 62
Pham, N. T., Dang, D. N. M., Nguyen, N. D., Nguyen, T. T., Nguyen, V., Manavalan, B., Lim, C. P., & Nguyen, S. D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120608 · Cited by 61
Guo, L., Wang, L., Dang, J., Liu, Z., & Guan, H. (2019) · IEEE Access · doi:10.1109/access.2019.2921390 · Cited by 62
Siddiqui, M. F. H., & Javaid, A. Y. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4030046 · Cited by 63
Deep Auto-Encoders With Sequential Learning for Multimodal Dimensional Emotion Recognition
Nguyen, D., Nguyen, T. T., Zeng, R., Nguyen, T. T., Tran, S. N., Nguyen, T., Sridharan, S., & Fookes, C. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3063612 · Cited by 67
Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks
Aslam, A., Sargano, A. B., & Habib, Z. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110494 · Cited by 60
Liu, Y., Jia, Z., & Wang, H. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612277 · Cited by 60
Deep Feature Space: A Geometrical Perspective
Kansizoglou, I., Bampis, L., & Gasteratos, A. C. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3094625 · Cited by 62
Kumar, A., Sharma, K., & Sharma, A. (2022) · Image and Vision Computing · doi:10.1016/j.imavis.2022.104483 · Cited by 59
Review and Perspectives on Human Emotion for Connected Automated Vehicles
Li, W., Li, G., Tan, R., Wang, C., Sun, Z., Li, Y., Guo, G., Cao, D., & Li, K. (2024) · Automotive Innovation · doi:10.1007/s42154-023-00270-z · Cited by 64
The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition
Kollias, D., Tzirakis, P., Cowen, A., Zafeiriou, S. P., Kotsia, I., Baird, A., Gagne, C., Shao, C., & Hu, G. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00461 · Cited by 59
Self-citationEmoCo: Visual Analysis of Emotion Coherence in Presentation Videos
Zeng, H., Wang, X., Wu, A., Wang, Y., Li, Q., Endert, A., & Qu, H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934656 · Cited by 55
Deep Learning based Emotion Recognition System Using Speech Features and TranscriptionsPreprint
Tripathi, S., Kumar, A., Ramesh, A., Singh, C., & Yenigalla, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05681 · Cited by 76
Metadata
- Authors
- Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S.
- Affiliations
- Imperial College London (GB)
- Goldsmiths University of London (GB)
- University of Augsburg (DE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 11
- Issue
- 8
- Pages
- 1301-1309
- ISSN
- 1932-4553 (ISSN-L)
- arXiv
- 1704.08619
- OpenAlex
- W2610961739
- Semantic Scholar
- 8cf0c072eeaec310aff806bdb7a7931b81e9ed99
- Cited by
- 752
- References
- 53
- Citations by year
- 2026: 67 · 2025: 90 · 2024: 81 · 2023: 99 · 2022: 108 · 2021: 102 · 2020: 76 · 2019: 82 · 2018: 42 · 2017: 5
- FWCI
- 72.53
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- Goldsmiths (University of London) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Tekes
- Engineering and Physical Sciences Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal emotion recognition, audio-visual emotion recognition, end-to-end learning, convolutional neural networks, ResNet, long short-term memory networks, feature extraction, spontaneous emotions, outlier robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
