The DOI matches its DataCite record; 55 of the 55 references it cites were found in a registry or an index; 1 of the 7 sources its claims cite were checked for support.
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Open accessAbstract
Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a Convolutional Neural Network (CNN) to extract features from the speech, while for the visual modality a deep residual network (ResNet) of 50 layers. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, Long Short-Term Memory (LSTM) networks are utilized. The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an end-to-end multimodal emotion recognition system that learns from raw speech and visual signals using a CNN, ResNet-50, and LSTM layers, trained to directly maximize concordance correlation coefficient (ρc). On the RECOLA database, the unimodal speech and visual models outperform several handcrafted-feature baselines, and the multimodal model improves valence prediction over prior multimodal systems.
Key findings
- For speech, the proposed raw-signal model optimized with ρc achieved test ρc of .699 for arousal and .311 for valence, compared with BLSTM eGeMAPS (.316 arousal, .195 valence) and BLSTM ComParE (.382 arousal, .187 valence); the MSE-optimized version achieved .684 arousal and .249 valence.
- For visual modality, the proposed raw-signal model achieved test ρc of .435 for arousal and .620 for valence, outperforming listed models in valence, e.g., baseline (.272 arousal, .507 valence) and Han et al. (.265 arousal, .394 valence).
- In multimodal analysis, the proposed model outperformed OA RVM-SR and Han et al. in valence “with high magnitude,” while OA RVM-SR produced the best arousal result; the provided table does not show the proposed multimodal ρc values.
- Sequence-length ablation found best speech performance at length 150 (arousal .744, valence .369) and best visual performance at length 300 (arousal .193, valence .496), though the multimodal network was trained with length 150; gate activations correlated with RMS energy range (ρ = .83), loudness (ρ = .73), and mean fundamental frequency (ρ = .72).
Methods
- Architecture: speech CNN operates on raw waveform, visual ResNet-50 operates on cropped face pixel intensities, and extracted features are concatenated into a 1920-dimensional vector (1280 speech + 640 visual) feeding a 2-layer LSTM with 256 cells each; unimodal networks are pretrained separately before end-to-end multimodal training.
- Data: RECOLA database, 46 French-speaking participants, 9.5 h of multimodal recordings, 5 minutes each in dyads; split into train (16 subjects), validation (15), and test (15), stratified by gender and age; 6 annotators. Audio is segmented into 6 s sequences at 16 kHz and further into 150 subsequences for 40 ms annotation granularity; visual images are 96×96.
- Training/evaluation: Adam optimizer with fixed learning rate 10⁻⁴, dropout p = 0.5, mini-batch 25 for audio and 2 for video, 60 epochs; metric is ρc; postprocessing includes median filtering, centering, scaling, and time-shifting selected on validation ρc.
Limitations
- The paper states that the OA RVM-SR multimodal comparison was trained using both training and validation sets, whereas this model used only the training set, and that OA RVM-SR used geometric features requiring accurate facial landmark tracking while this model operated on raw pixels; the authors expect their results could improve with similar strategies.
- The paper states hardware limitations forced a small video mini-batch of size 2, and notes memory and training instability concerns with very large recurrent sequences.
- Evident from the text: only two other multimodal models were found for comparison, RECOLA includes ECG and EDA modalities but this study uses only audio and video, and the provided multimodal results table lacks the proposed model’s exact ρc values.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
19 Own result · 5 No full text of the sources · 5 No basis given · 3 Own method · 3 Own interpretation · 1 Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the arousal dimension the OA RVM-SR produces the best results.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The only exception is the Video CNN-L4 model which performs better in the arousal dimension when appearance features are used.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This plot shows that certain cells of the model are very sensitive to different features conveyed in the original speech wave form.”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results show that our model outperforms the other models in the test set when predicting arousal dimension.”
plainn = 27- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As expected visual modality benefits the models in the valence dimension.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In all of the experiments, our model outperforms the designed features in terms of ρ c .”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we study the gate activations of the recurrent layers in the speech modality and find cells that are highly correlated with prosodic features that were always assumed to cause arousal.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our model outperforms the other two models in the valence dimension with high magnitude.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 3
Every source the paper cites for this claim was checked and supports it.
In the paper
“It is well accepted amongst the research community that certain acoustic and prosodic features play an important role in recognising the affective state [45].”
What the sources say
- Supportedcites [45] [45] K. Scherer, 'Vocal communication of emotion: A review of research paradigms,' Speech Communication , vol. 40, no. 1-2, pp. 227-256, April 2003. - source
Section 2.3, “Inference studies” (page not provided), reports that acoustic measures including pitch, energy, and duration explained a large proportion of variation in listeners’ emotion judgments. It also reports that manipulating vocal features affected judgments of a speaker’s emotional state, supporting the claim that such features matter for recognition. (LLM, conf=0.94)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
hedgedn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Some of these features, such as the mean of the fundamental frequency (F0), mean speech intensity, loudness, as well as pitch range [46], should thus be captured by our model.”
What the sources say
- Not checked — no full text of the sourcecites [46] [46] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan et al. , 'The geneva minimalistic acoustic parameter set (gemaps) for voice research and - affective computing,' IEEE Transactions on Affective Computing , vol. 7, no. 2, pp. 190-202, 2016. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
In the paper
“The task of recognising emotions is challenging because human emotions lack of temporal boundaries and different individuals express emotions in different ways [3].”
What the sources say
- Not checked — no full text of the sourcecites [3] [3] C.-N. Anagnostopoulos, T. Iliou, and I. Giannoukos, 'Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,' Artificial Intelligence Review , vol. 43, no. 2, pp. 155-177, 2015. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to perform a fair comparison we apply the proposed method on the test set of the AVEC challenge.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset is split in three partitions -train (16 subjects), validation (15 subjects) and test (15 subjects) -by stratifying (i.e., balancing) the gender and the age of the speakers.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Any of these post-processing steps is kept when an improvement is observed on the ρ c of the validation set, and applied then with the same configuration on the test partition.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
The proposed work is the first in literature to apply an end-to-end model for audio-visual emotion recognition.
“To our knowledge this is the first work in literature that applies such an end-to-end model for audio visual emotion recognition task.”
The paper relies on this without support.
Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
“The speech signals convey information about the affective state either explicitly, i.e., by linguistic means, or implicitly, i.e., by acoustic or prosodic cues.”
The paper relies on this without support.
Visual modality more easily predicts the valence dimension rather than the arousal.
“Visual modality has been shown to more easily predict the valence dimension rather than the arousal.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Dropout is applied in fullyconnected (reported in 43% of studies, e.g., [135]) and convolutional layers (14% of studies, e.g., [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We see both CNN-RNN (e.g., [18], [90]) and CNN-LSTM (e.g., [108], [105], [64]) architectures, with CNN-LSTM being the more frequent choice among the reviewed studies.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Global temporal modeling is found to lead to improved accuracies when compared with simpler methods such as pooling of spatial features (e.g., [152], [90], [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Applications of similar architectures based on the raw waveform have also been attempted [54], [108], showing that end-to-end learning can outperform shallow models.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“End-to-end training of the entire CNN-LSTM system addresses this problem, and can lead to accuracy improvements [108].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We are starting to see studies implementing end-toend training for such models [54], [108]”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Learning from raw audiovisual data with two CNNs, Tzirakis et al. [108] used a two-layer LSTM network for feature fusion, which was found to outperform the state of the art.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Similarly, for SER, Tzirakis et al. [108] found that LSTM cells learn representations similar to well-known prosodic features.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ I. INTRODUCTION
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
§ B. Speech Modality
“| Baseline [29] | eGeMAPS | .648 (.796) | .375 (.455) | | RVM [30] | eGeMAPS | - (.750) | - (.396) | | Audio BN-Multi [33] | Mixed | - (.833) | - (.503) | | Brady et al [32] | MFCC | - (.846) | - (.450) | | Weber et al. [31] | eGeMAPS | - (.793) | - (.456) | | Somandepalli et al. [34] | Mixed | - (.800) | - (.448) |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
(2010) · Cited by 14,134
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Research works printed but not linked: 1
- [37][37] F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data,' Pattern Recognition Letters , vol. 66, pp. 22-30, November 2015. -
Cited by
Sorted by citations
Items 101–200 of 752
A Model for EEG-Based Emotion Recognition: CNN-Bi-LSTM with Attention Mechanism
Huang, Z., Ma, Y., Wang, R., Li, W., & Dai, Y. (2023) · Electronics · doi:10.3390/electronics12143188 · Cited by 65
Jain, S., Narayan, S. P., Dewang, R. K., Bhartiya, U., Meena, N., & Kumar, V. (2019) · Students Conference on Engineering and Systems · doi:10.1109/sces46477.2019.8977211 · Cited by 58
Zheng, J., Zhang, S., Wang, Z., Wang, X., & Zeng, Z. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3144885 · Cited by 63
Speech Databases, Speech Features, and Classifiers in Speech Emotion Recognition: A Review
Dar, G. H. M., & Delhibabu, R. (2024) · IEEE Access · doi:10.1109/access.2024.3476960 · Cited by 68
Snore-GANs: Improving Automatic Snore Sound Classification With Synthesized Data
Zhang, Z., Han, J., Qian, K., Janott, C., Guo, Y., & Schuller, B. W. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2907286 · Cited by 56
Self-citationCross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition
Praveen, R. G., Granger, É., & Cardinal, P. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9667055 · Cited by 55
Multi-Channel Speech Enhancement Using Graph Neural Networks
Tzirakis, P., Kumar, A., & Donley, J. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413955 · Cited by 55
Self-citationHierarchical Attention Transfer Networks for Depression Assessment from Speech
Zhao, Z., Bao, Z., Zhang, Z., Cummins, N., Wang, H., & Schuller, B. W. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053207 · Cited by 56
Self-citationGraph Fusion Network-Based Multimodal Learning for Freezing of Gait Detection
Hu, K., Wang, Z., Martens, K. A. E., Hagenbuchner, M., Bennamoun, M., Tsoi, A. C., & Lewis, S. J. G. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105602 · Cited by 56
Is Cross-Attention Preferable to Self-Attention for Multi-Modal Emotion Recognition?
Rajan, V., Brutti, A., & Cavallaro, A. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746924 · Cited by 53
Digital Phenotyping and Feature Extraction on Smartphone Data for Depression Detection
Yang, M., Ngai, E. C. ‐., Hu, X., Hu, B., Liu, J., Erol Gelenbe, A., & Leung, V. C. M. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3542324 · Cited by 56
Harnessing multimodal data fusion to advance accurate identification of fish feeding intensity
Du, Z., Cui, M., Xu, X., Bai, Z., Han, J., Li, W., Yang, J., Liu, X., Wang, C., & Li, D. (2024) · Biosystems Engineering · doi:10.1016/j.biosystemseng.2024.08.001 · Cited by 52
Dynamic Difficulty Awareness Training for Continuous Emotion Prediction
Zhang, Z., Han, J., Coutinho, E., & Schuller, B. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2871949 · Cited by 53
Self-citation(2019) · doi:10.1109/smc.2019.8914655 · Cited by 52
DialogueTRM: Exploring Multi-Modal Emotional Dynamics in a Conversation
Mao, Y., Liu, G., Wang, X., Gao, W., & Li, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.229 · Cited by 50
Facial expression recognition in virtual reality environments: challenges and opportunities
Zhang, Z., Mir, J. M. F., & Mateu, L. G. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1280136 · Cited by 50
Self-supervised audiovisual representation learning for remote sensing data
Heidler, K., Mou, L., Hu, D., Jin, P., Li, G., Gan, C., Wen, J.-R., & Zhu, X. X. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.103130 · Cited by 57
JIANG, P., Xu, X., Tao, H., Zhao, L., & Zou, C. (2021) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3123979 · Cited by 51
Maji, B., Swain, M., & Mustaqeem, M. (2022) · Electronics · doi:10.3390/electronics11091328 · Cited by 49
Yang, D., Liu, Y., Huang, C., Li, M., Zhao, X., Wang, Y., Yang, K., Wang, Y., Zhai, P., & Zhang, L. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110370 · Cited by 47
Hybrid-Deep Learning Model for Emotion Recognition Using Facial Expressions
Verma, G., & Verma, H. (2020) · The Review of Socionetwork Strategies · doi:10.1007/s12626-020-00061-6 · Cited by 48
MIA-Net: Multi-Modal Interactive Attention Network for Multi-Modal Affective Analysis
Li, S., Zhang, T., Chen, B., & Chen, C. L. P. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3259010 · Cited by 48
Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition
Praveen, R. G., & Alam, M. J. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00483 · Cited by 54
Coronado, E., Venture, G., & Yamanobe, N. (2020) · International Journal of Social Robotics · doi:10.1007/s12369-020-00709-x · Cited by 47
MuSe 2020 Challenge and Workshop
Stappen, L., Baird, A., Rizos, G., Tzirakis, P., Du, X., Hafner, F., Schumann, L., Mallol-Ragolta, A., Schuller, B. W., Lefter, I., et al. (2020) · doi:10.1145/3423327.3423673 · Cited by 52
Self-citationA novel driver emotion recognition system based on deep ensemble classification
Zaman, K., Zhaoyun, S., Shah, B., Hussain, T., Shah, S. M., Ali, F., & Khan, U. S. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01100-9 · Cited by 49
Modeling Feedback in Interaction With Conversational Agents—A Review
Axelsson, A., Buschmeier, H., & Skantze, G. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.744574 · Cited by 45
Feature Augmented Hybrid CNN for Stress Recognition Using Wrist-based Photoplethysmography Sensor
Rashid, N., Chen, L., Dautta, M., Jimenez, A., Tseng, P., & Faruque, M. A. A. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc46164.2021.9630576 · Cited by 49
Uncertain Facial Expression Recognition via Multi-Task Assisted Correction
Liu, Y. M., Zhang, X., Kauttonen, J., & Zhao, G. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3301209 · Cited by 45
Shi, J., Liu, C., Ishi, C. T., & Ishiguro, H. (2020) · Sensors · doi:10.3390/s21010205 · Cited by 49
AIA-Net: Adaptive Interactive Attention Network for Text–Audio Emotion Recognition
Zhang, T., Li, S., Chen, B., Yuan, H., & Chen, C. L. P. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3195739 · Cited by 47
Asghar, M. A., Khan, M. J., Rizwan, M., Mehmood, R. M., & Kim, S. (2020) · Sensors · doi:10.3390/s20133765 · Cited by 44
CopyPaste: An Augmentation Method for Speech Emotion Recognition
Pappagari, R. R., Villalba, J., Żelasko, P., Moro-Velázquez, L., & Dehak, N. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9415077 · Cited by 44
Continuous Emotion Recognition for Long-Term Behavior Modeling through Recurrent Neural Networks
Kansizoglou, I., Misirlis, E., Tsintotas, K. A., & Gasteratos, A. C. (2022) · Technologies · doi:10.3390/technologies10030059 · Cited by 44
Kang, J.-S., Kavuri, S., & Lee, M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2942587 · Cited by 44
A Deep Multi-task Contextual Attention Framework for Multi-modal Affect Analysis
Akhtar, M. S., Chauhan, D. S., & Ekbal, A. (2020) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3380744 · Cited by 44
Affective computing scholarship and the rise of China: a view from 25 years of bibliometric data
Ho, T. M., Mantello, P. A., Nguyen, H. T., & Vuong, Q. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-021-00959-8 · Cited by 48
Implicit Fusion by Joint Audiovisual Training for Emotion Recognition in Mono Modality
Han, J., Zhang, Z., Ren, Z. X., & Schuller, B. W. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682773 · Cited by 42
Self-citationAnalysis and Use of the Emotional Context with Wearable Devices for Games and Intelligent Assistants
Nalepa, G. J., Kutt, K., Giżycka, B., Jemioło, P., & Bobek, S. (2019) · Sensors · doi:10.3390/s19112509 · Cited by 41
Multimodal Emotion Recognition with Capsule Graph Convolutional Based Representation Fusion
Liu, J., Chen, S., Wang, L., Liu, Z., Fu, Y., Guo, L., & Dang, J. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413608 · Cited by 42
Srinivasan, S., Huang, Z., & Kirchhoff, K. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747754 · Cited by 42
Facial Emotion Recognition with Inter-Modality-Attention-Transformer-Based Self-Supervised Learning
Chaudhari, A., Bhatt, C., Krishna, A., & Travieso, C. M. (2023) · Electronics · doi:10.3390/electronics12020288 · Cited by 44
Foundations of Multimodal Co-learning
Zadeh, A., Liang, P. P., & Morency, L.-P. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.001 · Cited by 40
Lightweight Deep Learning Framework for Speech Emotion Recognition
Akinpelu, S. A., Viriri, S., & Adegun, A. A. (2023) · IEEE Access · doi:10.1109/access.2023.3297269 · Cited by 40
EmoTour: Estimating Emotion and Satisfaction of Users Based on Behavioral Cues and Audiovisual Data
Matsuda, Y., Fedotov, D., Takahashi, Y., Arakawa, Y., Yasumoto, K., & Minker, W. (2018) · Sensors · doi:10.3390/s18113978 · Cited by 40
Learning Fine-Grained Cross Modality Excitement for Speech Emotion Recognition
Li, H., Ding, W., Wu, Z., & Liu, Z. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-158 · Cited by 40
Yadav, S., Chauhan, J., Sain, J. P., Thirunarayan, K., Sheth, A., & Schumm, J. A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.61 · Cited by 48
An End-To-End Emotion Recognition Framework Based on Temporal Aggregation of Multimodal Information
Rădoi, A., Birhala, A., Ristea, N.-C., & Duţu, L.-C. (2021) · IEEE Access · doi:10.1109/access.2021.3116530 · Cited by 39
Affect-Driven Learning of Robot Behaviour for Collaborative Human-Robot Interactions
Churamani, N., Barros, P. V. A., Güneş, H., & Wermter, S. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.717193 · Cited by 43
Torres, E. P., Torres, E. A., Hernández-Álvarez, M., & Yoo, S. G. (2020) · IEEE Access · doi:10.1109/access.2020.3035539 · Cited by 38
Pre-training Strategies and Datasets for Facial Representation Learning
Bulat, A., Cheng, S., Yang, J., Garbett, A., Sánchez, E., & Tzimiropoulos, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19778-9_7 · Cited by 38
Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition
Zhou, W., Lu, J., Xiong, Z., & Wang, W. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00610 · Cited by 37
Shan, Y., Li, S., & Chen, T. (2020) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-020-01074-x · Cited by 36
When AI meets store layout design: a review
Nguyen, K., Le, M. T. H., Martin, B. A. S., Çil, İ., & Fookes, C. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10142-3 · Cited by 45
Learning Better Representations for Audio-Visual Emotion Recognition with Common Information
Ma, F., Zhang, W., Li, Y., Huang, S., & Zhang, L. (2020) · Applied Sciences · doi:10.3390/app10207239 · Cited by 35
Hussain, W. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab487 · Cited by 36
Survey of deep emotion recognition in dynamic data using facial, speech and textual cues
Zhang, T., & Tan, Z. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17944-9 · Cited by 36
Emotion Recognition System from Speech and Visual Information based on Convolutional Neural Networks
Ristea, N.-C., Dutu, L. C., & Radoi, A. (2019) · International Conference on Speech Technology and Human-Computer Dialogue (SpeD) · doi:10.1109/sped.2019.8906538 · Cited by 35
Self-attention transfer networks for speech emotion recognition
Zhao, Z., Wang, K., Bao, Z., Zhang, Z., Cummins, N., Sun, S., Wang, H., Tao, J., & Schuller, B. W. (2021) · Virtual Reality & Intelligent Hardware · doi:10.1016/j.vrih.2020.12.002 · Cited by 35
Self-citationEnd-to-End Modeling and Transfer Learning for Audiovisual Emotion Recognition in-the-Wild
Dresvyanskiy, D., Рюмина, Е. В., Kaya, H., Markitantov, M., Karpov, A. A., & Minker, W. (2022) · Multimodal Technologies and Interaction · doi:10.3390/mti6020011 · Cited by 35
The Use of Artificial Intelligence in Assessing Affective States in Livestock
Neethirajan, S. R. (2021) · Frontiers in Veterinary Science · doi:10.3389/fvets.2021.715261 · Cited by 36
Joint Multimodal Transformer for Emotion Recognition in the Wild
Waligora, P., Aslam, M. H., Zeeshan, M., Belharbi, S., Koerich, A. L., Pedersoli, M., Bacon, S., & Granger, É. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00465 · Cited by 34
A Comprehensive Survey on Multimodal Data Representation and Information Fusion Algorithms
Gaonkar, A., Chukkapalli, Y., Raman, P. J., Srikanth, S., & Gurugopinath, S. (2021) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit51480.2021.9498415 · Cited by 34
Adversarial Training in Affective Computing and Sentiment Analysis: Recent Advances and PerspectivesPreprint
Han, J., Zhang, Z.-X., Cummins, N., & Schuller, B. (2018) · arXiv.org · doi:10.48550/arxiv.1809.08927 · Cited by 66
Self-citationJoint Multi-Scale Multimodal Transformer for Emotion Using Consumer Devices
Khan, M., Ahmad, J., Gueaieb, W., Masi, G., Karray, F. O., & Saddik, A. E. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3532322 · Cited by 37
Emotionally adaptive support: a narrative review of affective computing for mental health
Schlicher, M., Li, Y., Murthy, S. M. K., Sun, Q., & Schuller, B. W. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1657031 · Cited by 46
Self-citationSpeaker-Invariant Adversarial Domain Adaptation for Emotion Recognition
Yin, Y., Huang, B., Wu, Y., & Soleymani, M. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3418813 · Cited by 34
Harnessing emotions for depression detection
Prabhu, S. M., Mittal, H., Varagani, R., Jha, S., & Singh, S. (2021) · Pattern Analysis and Applications · doi:10.1007/s10044-021-01020-9 · Cited by 39
Zhou, Z., Asghar, M. A., Nazir, D., Siddique, K., Shorfuzzaman, M., & Mehmood, R. M. (2022) · Cluster Computing · doi:10.1007/s10586-022-03705-0 · Cited by 35
Tian, F., Zhang, L., Zhu, L., Zhao, M., Liu, J., Dong, Q., & Zhao, Q. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3420445 · Cited by 33
A Multimodal Emotion Recognition System Using Facial Landmark Analysis
Rahdari, F., Rashedi, E., & Eftekhari, M. (2018) · Iranian Journal of Science and Technology Transactions of Electrical Engineering · doi:10.1007/s40998-018-0142-9 · Cited by 32
Attention gated tensor neural network architectures for speech emotion recognition
Pandey, S., Shekhawat, H. S., & Prasanna, S. R. M. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103173 · Cited by 32
Enhancing Detection of Arabic Social Spam Using Data Augmentation and Machine Learning
Alkadri, A. M., ElKorany, A., & Ahmed, C. (2022) · Applied Sciences · doi:10.3390/app122211388 · Cited by 32
Depression Scale Recognition from Audio, Visual and Text AnalysisPreprint
Dham, S., Sharma, A., & Dhall, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.05865 · Cited by 43
Moorthy, S., & Moon, Y.-K. (2025) · Mathematics · doi:10.3390/math13071100 · Cited by 31
Jayanthi, K., Mohan, S., & Lakshmipriya, B. (2022) · International Journal of Information Technology · doi:10.1007/s41870-022-00900-5 · Cited by 31
EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts
Lee, M.-H., Shomanov, A., Begim, B., Kabidenova, Z., Nyssanbay, A., Yazıcı, A., & Lee, S. (2024) · Scientific Data · doi:10.1038/s41597-024-03838-4 · Cited by 38
Rathi, T., & Tripathy, M. (2024) · Speech Communication · doi:10.1016/j.specom.2024.103102 · Cited by 30
NLOS Identification in WLANs Using Deep LSTM with CNN Features
Nguyen, V.-H., Nguyen, M.-T., Choi, J., & Kim, Y. (2018) · Sensors · doi:10.3390/s18114057 · Cited by 30
Emotion Recognition System Based on Hybrid Techniques
Abdulsalam, W. H., Alhamdani, R. S., & Abdullah, M. N. (2019) · International Journal of Machine Learning and Computing · doi:10.18178/ijmlc.2019.9.4.831 · Cited by 30
Efficient Spatial Temporal Convolutional Features for Audiovisual Continuous Affect Recognition
Chen, H., Deng, Y., Cheng, S., Wang, Y., Jiang, D., & Hichem Sahli, B. (2019) · doi:10.1145/3347320.3357690 · Cited by 30
Context- and Knowledge-Aware Graph Convolutional Network for Multimodal Emotion Recognition
Fu, Y., Okada, S., Wang, L., Guo, L., Song, Y., Liu, J., & Dang, J. (2022) · IEEE Multimedia · doi:10.1109/mmul.2022.3173430 · Cited by 30
Emotion Recognition Based on Physiological Signals Using Convolution Neural Networks
Song, T., Lu, G., & Yan, J. (2020) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3383972.3384003 · Cited by 30
End2You -- The Imperial Toolkit for Multimodal Profiling by End-to-End LearningPreprint
Tzirakis, P., Zafeiriou, S. P., & Schuller, B. W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.01115 · Cited by 30
Self-citationDetecting Expressions with Multimodal Transformers
Parthasarathy, S., & Sundaram, S. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383573 · Cited by 29
Tang, D., Kuppens, P., Geurts, L., & van Waterschoot, T. (2021) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-021-00208-5 · Cited by 30
BLSTM and CNN Stacking Architecture for Speech Emotion Recognition
Li, D., Sun, L., Xu, X., Wang, Z., Zhang, J., & Du, W. (2021) · Neural Processing Letters · doi:10.1007/s11063-021-10581-z · Cited by 30
Let IoT Know You Better: User Identification and Emotion Recognition Through Millimeter-Wave Sensing
Yin, H., Yu, S., Zhang, Y., Zhou, A., Wang, X., Liu, L., Ma, H., Liu, J., & Yang, N. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3204779 · Cited by 29
Multimodal Utterance-level Affect Analysis using Visual, Audio and Text FeaturesPreprint
Deng, D., Zhou, Y., Pi, J., & Shi, B. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.00625 · Cited by 29
Avila, A. R., Kshirsagar, S., Tiwari, A., Lafond, D., O’Shaughnessy, D. D., & Falk, T. H. (2019) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2019.8903014 · Cited by 28
Deep Convolutional Neural Network Compression via Coupled Tensor Decomposition
Sun, W., Chen, S., Huang, L., So, H. C., & Xie, M. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3038227 · Cited by 29
Multistage linguistic conditioning of convolutional layers for speech emotion recognition
Triantafyllopoulos, A., Reichel, U. D., Liu, S., Huber, S., Eyben, F., & Schuller, B. W. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1072479 · Cited by 29
Self-citationMultimodal Emotion Recognition using Deep Learning Architectures
Hina, I., Shaukat, A., & Akram, M. U. (2022) · International Conference on Digital Futures and Transformative Technologies (ICoDT2) · doi:10.1109/icodt255437.2022.9787437 · Cited by 28
Audio and Video-based Emotion Recognition using Multimodal Transformers
John, V., & Kawanishi, Y. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956730 · Cited by 34
Zaman, K., Zengkang, G., Sun, Z., Shah, S. M., Riaz, W., Ji, J., Hussain, T., & Attar, R. W. (2025) · International Journal of Intelligent Systems · doi:10.1155/int/6611276 · Cited by 31
Real-Time Emotion Recognition on Mobile Devices
Sokolov, D., & Patkin, M. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00124 · Cited by 27
Baghizadeh, M., Maghooli, K., Farokhi, F., & Dabanloo, N. J. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2020.101902 · Cited by 28
Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text DataPreprint
Lee, C. W., Song, K. Y., Jeong, J., & Choi, W. Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.06606 · Cited by 27
Ma, F., Huang, S., & Zhang, L. (2021) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428219 · Cited by 25
A Survey on Emotion Recognition for Human Robot Interaction
Mohammed, S. N., & Hassan, A. K. A. (2021) · Journal of Computing and Information Technology · doi:10.20532/cit.2020.1004841 · Cited by 25
Metadata
- Authors
- Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S.
- Affiliations
- Imperial College London (GB)
- Goldsmiths University of London (GB)
- University of Augsburg (DE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 11
- Issue
- 8
- Pages
- 1301-1309
- ISSN
- 1932-4553 (ISSN-L)
- arXiv
- 1704.08619
- OpenAlex
- W2610961739
- Semantic Scholar
- 8cf0c072eeaec310aff806bdb7a7931b81e9ed99
- Cited by
- 752
- References
- 53
- Citations by year
- 2026: 67 · 2025: 90 · 2024: 81 · 2023: 99 · 2022: 108 · 2021: 102 · 2020: 76 · 2019: 82 · 2018: 42 · 2017: 5
- FWCI
- 72.53
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- Goldsmiths (University of London) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Tekes
- Engineering and Physical Sciences Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal emotion recognition, audio-visual emotion recognition, end-to-end learning, convolutional neural networks, ResNet, long short-term memory networks, feature extraction, spontaneous emotions, outlier robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
