The DOI matches its DataCite record; 55 of the 55 references it cites were found in a registry or an index; 1 of the 7 sources its claims cite were checked for support.
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Open accessAbstract
Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a Convolutional Neural Network (CNN) to extract features from the speech, while for the visual modality a deep residual network (ResNet) of 50 layers. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, Long Short-Term Memory (LSTM) networks are utilized. The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an end-to-end multimodal emotion recognition system that learns from raw speech and visual signals using a CNN, ResNet-50, and LSTM layers, trained to directly maximize concordance correlation coefficient (ρc). On the RECOLA database, the unimodal speech and visual models outperform several handcrafted-feature baselines, and the multimodal model improves valence prediction over prior multimodal systems.
Key findings
- For speech, the proposed raw-signal model optimized with ρc achieved test ρc of .699 for arousal and .311 for valence, compared with BLSTM eGeMAPS (.316 arousal, .195 valence) and BLSTM ComParE (.382 arousal, .187 valence); the MSE-optimized version achieved .684 arousal and .249 valence.
- For visual modality, the proposed raw-signal model achieved test ρc of .435 for arousal and .620 for valence, outperforming listed models in valence, e.g., baseline (.272 arousal, .507 valence) and Han et al. (.265 arousal, .394 valence).
- In multimodal analysis, the proposed model outperformed OA RVM-SR and Han et al. in valence “with high magnitude,” while OA RVM-SR produced the best arousal result; the provided table does not show the proposed multimodal ρc values.
- Sequence-length ablation found best speech performance at length 150 (arousal .744, valence .369) and best visual performance at length 300 (arousal .193, valence .496), though the multimodal network was trained with length 150; gate activations correlated with RMS energy range (ρ = .83), loudness (ρ = .73), and mean fundamental frequency (ρ = .72).
Methods
- Architecture: speech CNN operates on raw waveform, visual ResNet-50 operates on cropped face pixel intensities, and extracted features are concatenated into a 1920-dimensional vector (1280 speech + 640 visual) feeding a 2-layer LSTM with 256 cells each; unimodal networks are pretrained separately before end-to-end multimodal training.
- Data: RECOLA database, 46 French-speaking participants, 9.5 h of multimodal recordings, 5 minutes each in dyads; split into train (16 subjects), validation (15), and test (15), stratified by gender and age; 6 annotators. Audio is segmented into 6 s sequences at 16 kHz and further into 150 subsequences for 40 ms annotation granularity; visual images are 96×96.
- Training/evaluation: Adam optimizer with fixed learning rate 10⁻⁴, dropout p = 0.5, mini-batch 25 for audio and 2 for video, 60 epochs; metric is ρc; postprocessing includes median filtering, centering, scaling, and time-shifting selected on validation ρc.
Limitations
- The paper states that the OA RVM-SR multimodal comparison was trained using both training and validation sets, whereas this model used only the training set, and that OA RVM-SR used geometric features requiring accurate facial landmark tracking while this model operated on raw pixels; the authors expect their results could improve with similar strategies.
- The paper states hardware limitations forced a small video mini-batch of size 2, and notes memory and training instability concerns with very large recurrent sequences.
- Evident from the text: only two other multimodal models were found for comparison, RECOLA includes ECG and EDA modalities but this study uses only audio and video, and the provided multimodal results table lacks the proposed model’s exact ρc values.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
19 Own result · 5 No full text of the sources · 5 No basis given · 3 Own method · 3 Own interpretation · 1 Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the arousal dimension the OA RVM-SR produces the best results.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The only exception is the Video CNN-L4 model which performs better in the arousal dimension when appearance features are used.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This plot shows that certain cells of the model are very sensitive to different features conveyed in the original speech wave form.”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results show that our model outperforms the other models in the test set when predicting arousal dimension.”
plainn = 27- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As expected visual modality benefits the models in the valence dimension.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In all of the experiments, our model outperforms the designed features in terms of ρ c .”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we study the gate activations of the recurrent layers in the speech modality and find cells that are highly correlated with prosodic features that were always assumed to cause arousal.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our model outperforms the other two models in the valence dimension with high magnitude.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 3
Every source the paper cites for this claim was checked and supports it.
In the paper
“It is well accepted amongst the research community that certain acoustic and prosodic features play an important role in recognising the affective state [45].”
What the sources say
- Supportedcites [45] [45] K. Scherer, 'Vocal communication of emotion: A review of research paradigms,' Speech Communication , vol. 40, no. 1-2, pp. 227-256, April 2003. - source
Section 2.3, “Inference studies” (page not provided), reports that acoustic measures including pitch, energy, and duration explained a large proportion of variation in listeners’ emotion judgments. It also reports that manipulating vocal features affected judgments of a speaker’s emotional state, supporting the claim that such features matter for recognition. (LLM, conf=0.94)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
hedgedn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Some of these features, such as the mean of the fundamental frequency (F0), mean speech intensity, loudness, as well as pitch range [46], should thus be captured by our model.”
What the sources say
- Not checked — no full text of the sourcecites [46] [46] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan et al. , 'The geneva minimalistic acoustic parameter set (gemaps) for voice research and - affective computing,' IEEE Transactions on Affective Computing , vol. 7, no. 2, pp. 190-202, 2016. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
In the paper
“The task of recognising emotions is challenging because human emotions lack of temporal boundaries and different individuals express emotions in different ways [3].”
What the sources say
- Not checked — no full text of the sourcecites [3] [3] C.-N. Anagnostopoulos, T. Iliou, and I. Giannoukos, 'Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,' Artificial Intelligence Review , vol. 43, no. 2, pp. 155-177, 2015. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to perform a fair comparison we apply the proposed method on the test set of the AVEC challenge.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset is split in three partitions -train (16 subjects), validation (15 subjects) and test (15 subjects) -by stratifying (i.e., balancing) the gender and the age of the speakers.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Any of these post-processing steps is kept when an improvement is observed on the ρ c of the validation set, and applied then with the same configuration on the test partition.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
The proposed work is the first in literature to apply an end-to-end model for audio-visual emotion recognition.
“To our knowledge this is the first work in literature that applies such an end-to-end model for audio visual emotion recognition task.”
The paper relies on this without support.
Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
“The speech signals convey information about the affective state either explicitly, i.e., by linguistic means, or implicitly, i.e., by acoustic or prosodic cues.”
The paper relies on this without support.
Visual modality more easily predicts the valence dimension rather than the arousal.
“Visual modality has been shown to more easily predict the valence dimension rather than the arousal.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Dropout is applied in fullyconnected (reported in 43% of studies, e.g., [135]) and convolutional layers (14% of studies, e.g., [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We see both CNN-RNN (e.g., [18], [90]) and CNN-LSTM (e.g., [108], [105], [64]) architectures, with CNN-LSTM being the more frequent choice among the reviewed studies.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Global temporal modeling is found to lead to improved accuracies when compared with simpler methods such as pooling of spatial features (e.g., [152], [90], [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Applications of similar architectures based on the raw waveform have also been attempted [54], [108], showing that end-to-end learning can outperform shallow models.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“End-to-end training of the entire CNN-LSTM system addresses this problem, and can lead to accuracy improvements [108].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We are starting to see studies implementing end-toend training for such models [54], [108]”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Learning from raw audiovisual data with two CNNs, Tzirakis et al. [108] used a two-layer LSTM network for feature fusion, which was found to outperform the state of the art.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Similarly, for SER, Tzirakis et al. [108] found that LSTM cells learn representations similar to well-known prosodic features.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ I. INTRODUCTION
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
§ B. Speech Modality
“| Baseline [29] | eGeMAPS | .648 (.796) | .375 (.455) | | RVM [30] | eGeMAPS | - (.750) | - (.396) | | Audio BN-Multi [33] | Mixed | - (.833) | - (.503) | | Brady et al [32] | MFCC | - (.846) | - (.450) | | Weber et al. [31] | eGeMAPS | - (.793) | - (.456) | | Somandepalli et al. [34] | Mixed | - (.800) | - (.448) |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
(2010) · Cited by 14,134
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Research works printed but not linked: 1
- [37][37] F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data,' Pattern Recognition Letters , vol. 66, pp. 22-30, November 2015. -
Cited by
Sorted by citations
Items 201–300 of 752
Multimodal Knowledge Expansion
Xue, Z., Ren, S., Gao, Z., & Zhao, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00089 · Cited by 25
Trends in speech emotion recognition: a comprehensive survey
Kaur, K., & Singh, P. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14656-y · Cited by 26
Privileged Knowledge Distillation for Dimensional Emotion Recognition in the Wild
Aslam, M. H., Zeeshan, M., Pedersoli, M., Koerich, A. L., Bacon, S., & Granger, É. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00336 · Cited by 25
Jointly Aligning and Predicting Continuous Emotion Annotations
Khorram, S., McInnis, M. G., & Provost, E. M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2917047 · Cited by 25
Speech Emotion Recognition Using Semantic Information
Tzirakis, P., Nguyen, A., Zafeiriou, S. P., & Schuller, B. W. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414866 · Cited by 25
Self-citationAn Assessment of In-the-Wild Datasets for Multimodal Emotion Recognition
Aguilera, A., Mellado, D., & Rojas, F. (2023) · Sensors · doi:10.3390/s23115184 · Cited by 24
Generating Adversarial Examples in One Shot With Image-to-Image Translation GAN
Zhang, W. (2019) · IEEE Access · doi:10.1109/access.2019.2946461 · Cited by 24
Huang, C.-W., & Narayanan, S. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02901 · Cited by 24
Deep Covariance Descriptors for Facial Expression RecognitionPreprint
Otberdout, N., Kacem, A., Daoudi, M., Ballihi, L., & Berretti, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.03869 · Cited by 30
Pain-attentive network: a deep spatio-temporal attention model for pain estimation
Huang, D. J., Xia, Z., Mwesigye, J., & Feng, X. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09397-1 · Cited by 25
Cross-modal dynamic convolution for multi-modal emotion recognition
Wen, H., You, S., & Fu, Y. (2021) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2021.103178 · Cited by 23
Tian, L., Oviatt, S. L., Muszyński, M., Chamberlain, B. C., Healey, J., & Sano, A. (2022) · ACM eBooks · doi:10.1145/3502398 · Cited by 25
An Urdu speech corpus for emotion recognition
Asghar, A., Sohaib, S., Iftikhar, S., Shafi, M., & Fatima, K. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.954 · Cited by 23
WiFE: WiFi and Vision Based Unobtrusive Emotion Recognition via Gesture and Facial Expression
Gu, Y., Zhang, X., Yan, H., Huang, J., Liu, Z., Dong, M., & Ren, F. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3285777 · Cited by 29
Generative technology for human emotion recognition: A scoping review
Ma, F., Yuan, Y., Xie, Y., Ren, H., Liu, I., He, Y., Ren, F., Yu, F. R., & Ni, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102753 · Cited by 47
Dynamic Facial Models for Video-Based Dimensional Affect Estimation
Song, S., Sánchez, E., Tellamekala, M. K., Shen, L., Johnston, A., & Valstar, M. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00200 · Cited by 23
A Computational Model for Managing Impressions of an Embodied Conversational Agent in Real-Time
Biancardi, B., Wang, C., Mancini, M., Cafaro, A., Chanel, G., & Pélachaud, C. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925495 · Cited by 23
Indoor Scene Change Captioning Based on Multimodality Data
Qiu, Y., Satoh, Y., Suzuki, R., Iwata, K., & Kataoka, H. (2020) · Sensors · doi:10.3390/s20174761 · Cited by 23
Multi-Task Learning with Auxiliary Speaker Identification for Conversational Emotion RecognitionPreprint
Li, J., Zhang, M., Ji, D., & Liu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.01478 · Cited by 32
Monocular 3D Facial Expression Features for Continuous Affect Recognition
Pei, E., Oveneke, M. C., Zhao, Y. H., Jiang, D., & Hichem Sahli, B. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3026894 · Cited by 22
Sertolli, B., Ren, Z. X., Schuller, B. W., & Cummins, N. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101204 · Cited by 22
Self-citationFeng, Y., & Devillers, L. Y. (2023) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw59127.2023.10388120 · Cited by 22
A multimodal emotion recognition system using deep convolution neural networks
Almulla, M. (2024) · Journal of Engineering Research · doi:10.1016/j.jer.2024.03.021 · Cited by 27
A system for gender recognition on mobile robots
Foggia, P., Greco, A., Percannella, G., Vento, M., & Vigilante, V. (2019) · Applications Intelligent Systems · doi:10.1145/3309772.3309781 · Cited by 22
Cross-Domain Deep Feature Combination for Bird Species Classification with Audio-Visual Data
Bold, N., Zhang, C., & Akashi, T. (2019) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2018edp7383 · Cited by 22
Han, J., Zhang, Z., Pantić, M., & Schuller, B. W. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.08.002 · Cited by 22
Self-citationMulti-Modal Embeddings Using Multi-Task Learning for Emotion Recognition
Khare, A., Parthasarathy, S., & Sundaram, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1827 · Cited by 24
Yu, L., Ge, Y., Ansari, S., Imran, M. A., & Ahmad, W. (2025) · Sensors · doi:10.3390/s25154763 · Cited by 26
Emotion Recognition under Sleep Deprivation Using a Multimodal Residual LSTM Network
Tao, L.-Y., & Lu, B. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206957 · Cited by 25
End-to-End Deep Fault-Tolerant Control
Baimukashev, D., Rakhim, B., Rubagotti, M., & Varol, H. A. (2021) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2021.3100150 · Cited by 21
Spatio-Temporal Pain Estimation Network With Measuring Pseudo Heart Rate Gain
Huang, D. J., Feng, X., Zhang, H., Yu, Z., Peng, J., Zhao, G., & Xia, Z. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3096080 · Cited by 21
Yang, L., Wang, Y., Ouyang, R., Niu, X., Yang, X., & Chen, Z. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108011 · Cited by 21
Incongruity-Aware Cross-Modal Attention for Audio-Visual Fusion in Dimensional Emotion Recognition
Praveen, R. G., & Alam, J. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3422823 · Cited by 21
A Stereo Perception Framework for Autonomous Vehicles
Kemsaram, N., Das, A., & Dubbelman, G. (2020) · IEEE Vehicular Technology Conference · doi:10.1109/vtc2020-spring48590.2020.9128899 · Cited by 21
Multimodal Embeddings From Language Models for Emotion Recognition in the Wild
Tseng, S.-Y., Narayanan, S. S., & Georgiou, P. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3065598 · Cited by 23
Image–Music Synesthesia-Aware Learning Based on Emotional Similarity Recognition
Xing, B., Zhang, K., Zhang, L., Wu, X., Dou, J., & Sun, S. (2019) · IEEE Access · doi:10.1109/access.2019.2942073 · Cited by 20
Elfaik, H., & Nfaoui, E. H. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.12.015 · Cited by 22
Recursive Joint Attention for Audio-Visual Fusion in Regression Based Emotion Recognition
Praveen, R. G., Granger, É., & Cardinal, P. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095234 · Cited by 20
Yang, Z., Zhou, H., Srivastav, S., Shaffer, J. G., Abraham, K. T., Naandam, S. M., & Kakraba, S. (2025) · Data · doi:10.3390/data10010004 · Cited by 22
Applying TDNN Architectures for Analyzing Duration Dependencies on Speech Emotion Recognition
Kumawat, P., & Routray, A. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-2168 · Cited by 20
An efficient model-level fusion approach for continuous affect recognition from audiovisual signals
Pei, E., Jiang, D., & Hichem Sahli, B. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.09.037 · Cited by 21
A Deep Learning-Based Crop Disease Diagnosis Method Using Multimodal Mixup Augmentation
Lee, H., Park, Y.-S., Yang, S., Lee, H., Park, T., & Yeo, D. (2024) · Applied Sciences · doi:10.3390/app14104322 · Cited by 23
Audio-Visual Kinship Verification in the Wild
Wu, X., Granger, É., Kinnunen, T., Feng, X., & Hadid, A. (2019) · International Conference on Biometrics · doi:10.1109/icb45273.2019.8987241 · Cited by 21
Facial Image Pre-Processing and Emotion Classification: A Deep Learning Approach
Navaz, A. N., Serhani, M. A., & Mathew, S. S. (2019) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa47632.2019.9035268 · Cited by 19
A Survey on Automatic Emotion Recognition Using Audio Big Data and Deep Learning Architectures
Zhao, H., Ye, N., & Wang, R. (2018) · IEEE International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, and IEEE International Conference on Intelligent Data and Security (BigDataSecurity/HPSC/IDS) · doi:10.1109/bds/hpsc/ids18.2018.00039 · Cited by 19
Kumar, B., V, Rao, A. N., & Yedukondalu, N. (2025) · International Conference on Sustainable Computing and Data Communication Systems (ICSCDS) · doi:10.1109/icscds65426.2025.11166844 · Cited by 18
Lin, J., Cui, Y., Fang, C., Pan, B., Pan, C., Jiang, G., Zhang, S., Ma, S., & Tian, Q. C. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3652641 · Cited by 18
Comparing supervised and unsupervised approaches to multimodal emotion recognition
Carbonell, M. F., Boman, M., & Laukka, P. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.804 · Cited by 18
A Review of the Advancement in Speech Emotion Recognition for Indo-Aryan and Dravidian Languages
Monisha, S. T. A., & Sultana, S. (2022) · Advances in Human-Computer Interaction · doi:10.1155/2022/9602429 · Cited by 18
Combined CNN LSTM with attention for speech emotion recognition based on feature-level fusion
Liu, Y., Chen, A., Zhou, G., Yi, J., Jin, X., & Wang, Y. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17829-x · Cited by 18
Exploration of an Independent Training Framework for Speech Emotion Recognition
Zhong, S., Yu, B., & Zhang, H. (2020) · IEEE Access · doi:10.1109/access.2020.3043894 · Cited by 18
The Application of Capsule Neural Network Based CNN for Speech Emotion Recognition
Wen, X.-C., Liu, K., Zhang, W., & Jiang, K. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412360 · Cited by 18
Emotion Recognition from Occluded Facial Images Using Deep Ensemble燤odel
Ullah, Z., Mohmand, M. I., Rehman, S. U., Zubair, M., Driss, M., Boulila, W., Sheikh, R. R., & Alwawi, I. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2022.029101 · Cited by 17
Supervised Contrastive Learning for Affect Modelling
Pinitas, K., Makantasis, K., Liapis, A., & Yannakakis, G. N. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556584 · Cited by 17
Vaiani, L., Quatra, M. L., Cagliero, L., & Garza, P. (2022) · doi:10.1145/3551876.3554806 · Cited by 17
Estimating the Uncertainty in Emotion Class Labels With Utterance-Specific Dirichlet Priors
Wu, W., Zhang, C., Wu, X., & Woodland, P. C. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3221801 · Cited by 20
Stochastic weight averaging enhanced temporal convolution network for EEG-based emotion recognition
Yang, L., Wang, Y., Yang, X., & Chen, Z. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.104661 · Cited by 17
A Feature Fusion Model with Data Augmentation for Speech Emotion Recognition
Tu, Z., Liu, B., Zhao, W., Yan, R., & Zou, Y. (2023) · Applied Sciences · doi:10.3390/app13074124 · Cited by 20
Khan, M. T., & Sheikh, U. U. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120153 · Cited by 17
Santhosh, J., Pai, A. P., & Ishimaru, S. (2024) · IEEE Access · doi:10.1109/access.2024.3350745 · Cited by 22
Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition
Du, Y., Li, P., Cheng, L., Zhang, X., Li, M., & Li, F. (2024) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1330077 · Cited by 18
Emotional Stability Detection Using Convolutional Neural Networks
Hussein, E. S. A., Qidwai, U. A., & Al-Meer, M. H. (2020) · IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT) · doi:10.1109/iciot48696.2020.9089440 · Cited by 17
Speech Emotion Recognition Using Deep Learning LSTM for Tamil Language
Fernandes, B., & Mannepalli, K. (2021) · Pertanika journal of science & technology · doi:10.47836/pjst.29.3.33 · Cited by 17
Khan, N., Singh, A. V., & Agrawal, R. K. (2022) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001422520280 · Cited by 20
The Next Big Thing(s) in Unsupervised Machine Learning: Five Lessons from Infant LearningPreprint
Zaadnoordijk, L., Besold, T. R., & Cusack, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.08497 · Cited by 17
Speech emotion recognition using feature fusion: a hybrid approach to deep learning
Khan, W. A., Qudous, H. U., & Farhan, A. A. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18316-7 · Cited by 19
Real-world automatic continuous affect recognition from audiovisual signals
Tzirakis, P., Zafeiriou, S. P., & Schuller, B. (2019) · Multimodal Behavior Analysis in the Wild · doi:10.1016/b978-0-12-814601-9.00028-6 · Cited by 20
Self-citationSyntactic- and morphology-based text augmentation framework for Arabic sentiment analysis
Duwairi, R., & Shaqra, F. A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.469 · Cited by 16
A web crowdsourcing framework for transfer learning and personalized Speech Emotion Recognition
Vryzas, N., Vrysis, L., Kotsakis, R., & Dimoulas, C. A. (2021) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100132 · Cited by 16
Gutiérrez-Martín, L., Romero, E., Andújar, C. S. D. B., Canabal, M. F., Rodríguez-Ramos, G. E., Toro-Flores, R., López‐Ongil, S., & López-Ongil, C. (2022) · Sensors · doi:10.3390/s22114023 · Cited by 19
A Transfer Learning Method for Speech Emotion Recognition from Automatic Speech RecognitionPreprint
Sitong, Z., & Beigi, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.02863 · Cited by 22
Zhou, N., Jika, L., Wang, W., Xie, Y., Du, Y., & Soh, Y. C. (2025) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2025.3590031 · Cited by 15
Xefteris, V.-R., Tsanousa, A., Georgakopoulou, N., Diplaris, S., Vrochidis, S., & Kompatsiaris, I. Y. (2022) · Sensors · doi:10.3390/s22218198 · Cited by 15
Multimodal Arabic emotion recognition using deep learning
Roken, N. A., & Barlas, G. (2023) · Speech Communication · doi:10.1016/j.specom.2023.103005 · Cited by 15
Enhancement of Virtual Assistants Through Multimodal AI for Emotion Recognition
Rajesh, S. G., Madangarli, S. V., Pisharady, G. S., & Subrahmanyam, R. (2025) · IEEE Access · doi:10.1109/access.2025.3577664 · Cited by 17
Temporal aggregation of audio-visual modalities for emotion recognition
Birhala, A., Ristea, N.-C., Rădoi, A., & Dutu, L. C. (2020) · International Conference on Telecommunications and Signal Processing (TSP) · doi:10.1109/tsp49548.2020.9163474 · Cited by 15
Affective state recognition from hand gestures and facial expressions using Grassmann manifolds
Verma, B., & Choudhary, A. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10341-6 · Cited by 15
Multi-modal Fusion for Continuous Emotion Recognition by Using Auto-Encoders
Hamieh, S., Heiries, V., Osman, H. A., & Godin, C. (2021) · doi:10.1145/3475957.3484455 · Cited by 18
EEG-based Emotion Recognition for Multi Channel Fast Empirical Mode Decomposition using VGG-16
Asghar, M. A., Fawad, F., Khan, M. J., Amin, Y., & Akram, A. (2020) · International Conference on Engineering and Emerging Technologies (ICEET) · doi:10.1109/iceet48479.2020.9048217 · Cited by 15
From the Lab to the Wild: Affect Modeling Via Privileged Information
Makantasis, K., Pinitas, K., Liapis, A., & Yannakakis, G. N. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3265072 · Cited by 14
Automatic Fluency Assessment Method for Spontaneous Speech without Reference Text
Liu, J., Wumaier, A., Fan, C., & Guo, S. (2023) · Electronics · doi:10.3390/electronics12081775 · Cited by 17
Study on emotion recognition bias in different regional groups
Lukáč, M., Zhambulova, G., Abdiyeva, K., & Lewis, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-34932-z · Cited by 17
Uncertainty-Based Learning of a Lightweight Model for Multimodal Emotion Recognition
Rădoi, A., & Cioroiu, G. (2024) · IEEE Access · doi:10.1109/access.2024.3450674 · Cited by 18
Multimodal Joint Emotion and Game Context Recognition in League of Legends Livestreams
Ringer, C., Walker, J. A., & Nicolaou, M. A. (2019) · IEEE Conference on Games (CoG) · doi:10.1109/cig.2019.8848060 · Cited by 13
Self-citationMultimodal Emotional Classification Based on Meaningful Learning
Filali, H., Riffi, J., Boulealam, C., Mahraz, M. A., & Tairi, H. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6030095 · Cited by 15
Yu, Y., Wu, X., & Qian, Q. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105539 · Cited by 14
Transfer Learning From Sound Representations For Anger Detection in SpeechPreprint
Elshaer, M. E. A., Wisdom, S., & Mishra, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02120 · Cited by 14
A Machine-Learning Model for Detecting Depression, Anxiety, and Stress from Speech
Tasnim, M., Diaz-Ramos, R. E., Stroulia, E., & Trejo, L. A. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446567 · Cited by 15
Effective MLP and CNN based ensemble learning for speech emotion recognition
Middya, A. I., Nag, B., & Roy, S. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19017-x · Cited by 13
Attention-Based Multi-Modal Multi-View Fusion Approach for Driver Facial Expression Recognition
Jianrong, C., Dey, S., Wang, L., Bi, N., & Liu, P. (2024) · IEEE Access · doi:10.1109/access.2024.3462352 · Cited by 13
Ray, C., Tripathy, H. K., & Mishra, S. (2019) · Communications in computer and information science · doi:10.1007/978-981-13-9942-8_43 · Cited by 13
Continuous Emotion Recognition via Deep Convolutional Autoencoder and Support Vector Regressor
Allognon, S. O. C., Britto, A. D. S., & Koerich, A. L. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207250 · Cited by 13
Lahoz-Beltrá, R., & López, C. C. (2021) · Computers · doi:10.3390/computers10120170 · Cited by 15
Dementia Detection by Fusing Speech and Eye-Tracking Representation
Sheng, Z.-Y., Guo, Z., Li, X., Li, Y., & Ling, Z.-H. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747054 · Cited by 13
End-to-end speech emotion recognition using multi-scale convolution networks
Tatinati, S., Ho, M. K., Khong, A. W. H., & Wang, Y. (2017) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2017.8282026 · Cited by 13
Ali, S., & Chong, I. (2019) · Applied Sciences · doi:10.3390/app9194175 · Cited by 13
Emotion Charting Using Real-time Monitoring of Physiological Signals
Rahim, A., Sagheer, A., Nadeem, K., Dar, M. N., Rahim, A., & Akram, U. (2019) · International Conference on Robotics and Automation in Industry (ICRAI) · doi:10.1109/icrai47710.2019.8967398 · Cited by 12
Focal Loss based Residual Convolutional Neural Network for Speech Emotion RecognitionPreprint
Tripathi, S., Kumar, A., Ramesh, A., Singh, C., & Yenigalla, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05682 · Cited by 16
N24News: A New Dataset for Multimodal News Classification
Wang, Z., Xu, S., Zhang, X., & Yang, J. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5c2xbka7swxq · Cited by 29
Metadata
- Authors
- Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S.
- Affiliations
- Imperial College London (GB)
- Goldsmiths University of London (GB)
- University of Augsburg (DE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 11
- Issue
- 8
- Pages
- 1301-1309
- ISSN
- 1932-4553 (ISSN-L)
- arXiv
- 1704.08619
- OpenAlex
- W2610961739
- Semantic Scholar
- 8cf0c072eeaec310aff806bdb7a7931b81e9ed99
- Cited by
- 752
- References
- 53
- Citations by year
- 2026: 67 · 2025: 90 · 2024: 81 · 2023: 99 · 2022: 108 · 2021: 102 · 2020: 76 · 2019: 82 · 2018: 42 · 2017: 5
- FWCI
- 72.53
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- Goldsmiths (University of London) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Tekes
- Engineering and Physical Sciences Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal emotion recognition, audio-visual emotion recognition, end-to-end learning, convolutional neural networks, ResNet, long short-term memory networks, feature extraction, spontaneous emotions, outlier robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
