The DOI matches its DataCite record; 55 of the 55 references it cites were found in a registry or an index; 1 of the 7 sources its claims cite were checked for support.
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Open accessAbstract
Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a Convolutional Neural Network (CNN) to extract features from the speech, while for the visual modality a deep residual network (ResNet) of 50 layers. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, Long Short-Term Memory (LSTM) networks are utilized. The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an end-to-end multimodal emotion recognition system that learns from raw speech and visual signals using a CNN, ResNet-50, and LSTM layers, trained to directly maximize concordance correlation coefficient (ρc). On the RECOLA database, the unimodal speech and visual models outperform several handcrafted-feature baselines, and the multimodal model improves valence prediction over prior multimodal systems.
Key findings
- For speech, the proposed raw-signal model optimized with ρc achieved test ρc of .699 for arousal and .311 for valence, compared with BLSTM eGeMAPS (.316 arousal, .195 valence) and BLSTM ComParE (.382 arousal, .187 valence); the MSE-optimized version achieved .684 arousal and .249 valence.
- For visual modality, the proposed raw-signal model achieved test ρc of .435 for arousal and .620 for valence, outperforming listed models in valence, e.g., baseline (.272 arousal, .507 valence) and Han et al. (.265 arousal, .394 valence).
- In multimodal analysis, the proposed model outperformed OA RVM-SR and Han et al. in valence “with high magnitude,” while OA RVM-SR produced the best arousal result; the provided table does not show the proposed multimodal ρc values.
- Sequence-length ablation found best speech performance at length 150 (arousal .744, valence .369) and best visual performance at length 300 (arousal .193, valence .496), though the multimodal network was trained with length 150; gate activations correlated with RMS energy range (ρ = .83), loudness (ρ = .73), and mean fundamental frequency (ρ = .72).
Methods
- Architecture: speech CNN operates on raw waveform, visual ResNet-50 operates on cropped face pixel intensities, and extracted features are concatenated into a 1920-dimensional vector (1280 speech + 640 visual) feeding a 2-layer LSTM with 256 cells each; unimodal networks are pretrained separately before end-to-end multimodal training.
- Data: RECOLA database, 46 French-speaking participants, 9.5 h of multimodal recordings, 5 minutes each in dyads; split into train (16 subjects), validation (15), and test (15), stratified by gender and age; 6 annotators. Audio is segmented into 6 s sequences at 16 kHz and further into 150 subsequences for 40 ms annotation granularity; visual images are 96×96.
- Training/evaluation: Adam optimizer with fixed learning rate 10⁻⁴, dropout p = 0.5, mini-batch 25 for audio and 2 for video, 60 epochs; metric is ρc; postprocessing includes median filtering, centering, scaling, and time-shifting selected on validation ρc.
Limitations
- The paper states that the OA RVM-SR multimodal comparison was trained using both training and validation sets, whereas this model used only the training set, and that OA RVM-SR used geometric features requiring accurate facial landmark tracking while this model operated on raw pixels; the authors expect their results could improve with similar strategies.
- The paper states hardware limitations forced a small video mini-batch of size 2, and notes memory and training instability concerns with very large recurrent sequences.
- Evident from the text: only two other multimodal models were found for comparison, RECOLA includes ECG and EDA modalities but this study uses only audio and video, and the provided multimodal results table lacks the proposed model’s exact ρc values.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
19 Own result · 5 No full text of the sources · 5 No basis given · 3 Own method · 3 Own interpretation · 1 Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the arousal dimension the OA RVM-SR produces the best results.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The only exception is the Video CNN-L4 model which performs better in the arousal dimension when appearance features are used.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This plot shows that certain cells of the model are very sensitive to different features conveyed in the original speech wave form.”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results show that our model outperforms the other models in the test set when predicting arousal dimension.”
plainn = 27- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As expected visual modality benefits the models in the valence dimension.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In all of the experiments, our model outperforms the designed features in terms of ρ c .”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we study the gate activations of the recurrent layers in the speech modality and find cells that are highly correlated with prosodic features that were always assumed to cause arousal.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our model outperforms the other two models in the valence dimension with high magnitude.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 3
Every source the paper cites for this claim was checked and supports it.
In the paper
“It is well accepted amongst the research community that certain acoustic and prosodic features play an important role in recognising the affective state [45].”
What the sources say
- Supportedcites [45] [45] K. Scherer, 'Vocal communication of emotion: A review of research paradigms,' Speech Communication , vol. 40, no. 1-2, pp. 227-256, April 2003. - source
Section 2.3, “Inference studies” (page not provided), reports that acoustic measures including pitch, energy, and duration explained a large proportion of variation in listeners’ emotion judgments. It also reports that manipulating vocal features affected judgments of a speaker’s emotional state, supporting the claim that such features matter for recognition. (LLM, conf=0.94)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
hedgedn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Some of these features, such as the mean of the fundamental frequency (F0), mean speech intensity, loudness, as well as pitch range [46], should thus be captured by our model.”
What the sources say
- Not checked — no full text of the sourcecites [46] [46] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan et al. , 'The geneva minimalistic acoustic parameter set (gemaps) for voice research and - affective computing,' IEEE Transactions on Affective Computing , vol. 7, no. 2, pp. 190-202, 2016. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
In the paper
“The task of recognising emotions is challenging because human emotions lack of temporal boundaries and different individuals express emotions in different ways [3].”
What the sources say
- Not checked — no full text of the sourcecites [3] [3] C.-N. Anagnostopoulos, T. Iliou, and I. Giannoukos, 'Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,' Artificial Intelligence Review , vol. 43, no. 2, pp. 155-177, 2015. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to perform a fair comparison we apply the proposed method on the test set of the AVEC challenge.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset is split in three partitions -train (16 subjects), validation (15 subjects) and test (15 subjects) -by stratifying (i.e., balancing) the gender and the age of the speakers.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Any of these post-processing steps is kept when an improvement is observed on the ρ c of the validation set, and applied then with the same configuration on the test partition.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
The proposed work is the first in literature to apply an end-to-end model for audio-visual emotion recognition.
“To our knowledge this is the first work in literature that applies such an end-to-end model for audio visual emotion recognition task.”
The paper relies on this without support.
Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
“The speech signals convey information about the affective state either explicitly, i.e., by linguistic means, or implicitly, i.e., by acoustic or prosodic cues.”
The paper relies on this without support.
Visual modality more easily predicts the valence dimension rather than the arousal.
“Visual modality has been shown to more easily predict the valence dimension rather than the arousal.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Dropout is applied in fullyconnected (reported in 43% of studies, e.g., [135]) and convolutional layers (14% of studies, e.g., [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We see both CNN-RNN (e.g., [18], [90]) and CNN-LSTM (e.g., [108], [105], [64]) architectures, with CNN-LSTM being the more frequent choice among the reviewed studies.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Global temporal modeling is found to lead to improved accuracies when compared with simpler methods such as pooling of spatial features (e.g., [152], [90], [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Applications of similar architectures based on the raw waveform have also been attempted [54], [108], showing that end-to-end learning can outperform shallow models.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“End-to-end training of the entire CNN-LSTM system addresses this problem, and can lead to accuracy improvements [108].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We are starting to see studies implementing end-toend training for such models [54], [108]”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Learning from raw audiovisual data with two CNNs, Tzirakis et al. [108] used a two-layer LSTM network for feature fusion, which was found to outperform the state of the art.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Similarly, for SER, Tzirakis et al. [108] found that LSTM cells learn representations similar to well-known prosodic features.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ I. INTRODUCTION
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
§ B. Speech Modality
“| Baseline [29] | eGeMAPS | .648 (.796) | .375 (.455) | | RVM [30] | eGeMAPS | - (.750) | - (.396) | | Audio BN-Multi [33] | Mixed | - (.833) | - (.503) | | Brady et al [32] | MFCC | - (.846) | - (.450) | | Weber et al. [31] | eGeMAPS | - (.793) | - (.456) | | Somandepalli et al. [34] | Mixed | - (.800) | - (.448) |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
(2010) · Cited by 14,134
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Research works printed but not linked: 1
- [37][37] F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data,' Pattern Recognition Letters , vol. 66, pp. 22-30, November 2015. -
Cited by
Sorted by citations
Items 401–500 of 752
Using Voice and Biofeedback to Predict User Engagement during Requirements InterviewsPreprint
Ferrari, A., Huichapa, T., Spoletini, P., Novielli, N., Fucci, D., & Girardi, D. (2021) · Research Square · doi:10.21203/rs.3.rs-416018/v1 · Cited by 6
Real-Time Emotion Detection in Live Video Streams using Multimodal Deep Learning Architectures
Rani, K. S., Muralidhar, A., Rao, G. S., Shareef, S. K., Sudheer, P., & Bhavsingh, M. (2025) · International Conference on Pervasive Computing and Social Networking (ICPCSN) · doi:10.1109/icpcsn65854.2025.11034807 · Cited by 5
Yu, J., Zheng, Y., Wang, L., Wang, Y., & Xu, S. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00569 · Cited by 5
Jagadeesan, S., Sujigarasharma, K., Manjula, R., & Vetriselvi, T. (2023) · International Conference on Research Methodologies in Knowledge Management, Artificial Intelligence and Telecommunication Engineering (RMKMATE) · doi:10.1109/rmkmate59243.2023.10369356 · Cited by 5
Rawat, U., & Rai, C. S. (2023) · International Conference on Technological Advancements in Computational Sciences (ICTACS) · doi:10.1109/ictacs59847.2023.10390186 · Cited by 5
Are You Sure? - Multi-Modal Human Decision Uncertainty Detection in Human-Robot Interaction
Scherf, L., Gasche, L. A., Chemangui, E., & Koert, D. (2024) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3610977.3634926 · Cited by 7
Use of affect context in dyadic interactions for continuous emotion recognition
Fatima, S. N., & Erzin, E. (2021) · Speech Communication · doi:10.1016/j.specom.2021.05.010 · Cited by 5
WeaveNet: End-to-End Audiovisual Sentiment Analysis
Yu, Y., Jia, Z., Shi, F., Zhu, M., Wang, W., & Li, X. (2022) · Communications in computer and information science · doi:10.1007/978-981-16-9247-5_1 · Cited by 5
Kothuri, S. R., & R, D. R. N. (2022) · Indian Journal of Computer Science and Engineering · doi:10.21817/indjcse/2022/v13i2/221302040 · Cited by 5
Hierarchical Weighted Framework for Emotional Distress Detection using Personalized Affective Cues
Jadhav, N. (2022) · Journal of Information Systems and Telecommunication (JIST) · doi:10.52547/jist.16499.10.38.89 · Cited by 5
Deep Learning for Audio Visual Emotion Recognition
Hussain, T., Wenwu Wang, B., Bouaynaya, N. C., Fathallah‐Shaykh, H. M., & Mihaylova, L. S. (2022) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion49751.2022.9841342 · Cited by 5
Selvi, R., & Vijayakumaran, C. (2023) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs57279.2023.10113124 · Cited by 5
Huang, T., Li, J.-L., Chang, C.-M., & Lee, C.-C. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925496 · Cited by 5
Hekiert, D., & Igraś-Cybulska, M. (2019) · Roczniki Psychologiczne · doi:10.18290/rpsych.2019.22.1-2 · Cited by 5
Chen, K., Niu, M., & Chen, Q. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01540-8 · Cited by 5
Suicide Ideation Detection of Covid Patients Using Machine Learning Algorithm
Punithavathi, R., Thenmozhi, S., Jothilakshmi, R., Venugopal, E., & Ul, I. M. T. (2022) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.025972 · Cited by 5
Facial Recognition System using Decision Tree Algorithm
Sivasangari, A., Gomathi, R. M., Anandhi, T. S., Roobini, R., & Ajitha, P. (2022) · International Conference on Electronics and Sustainable Communication Systems (ICESC) · doi:10.1109/icesc54411.2022.9885489 · Cited by 5
Yalamanchili, B., Kumar, S. S., & Rao, A. K. (2023) · Journal of Information and Communication Technology · doi:10.32890/jict2023.22.1.3 · Cited by 5
D-ViSA: A Dataset for Detecting Visual Sentiment from Art Images
Kim, S., An, C., Cha, J., Kim, D., & Park, E. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00328 · Cited by 5
Multimodal Emotion Recognition Using the Symmetric S-ELM-LUPI Paradigm
Yang, L., Ban, X., Mukeshimana, M., & Chen, Z. (2019) · Symmetry · doi:10.3390/sym11040487 · Cited by 5
Stappen, L., Baird, A., Rizos, G., Tzirakis, P., Du, X., Hafner, F., Schumann, L., Mallol-Ragolta, A., Schuller, B. W., Lefter, I., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14858 · Cited by 5
Self-citationMultimodal Continuous Emotion Recognition using Deep Multi-Task Learning with Correlation LossPreprint
Köprü, B., & Erzin, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.00876 · Cited by 5
Multimodal End-to-End Group Emotion Recognition using Cross-Modal AttentionPreprint
Evtodienko, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05890 · Cited by 12
State-of-the-art Multimodal Emotion Recognition: A comprehensive survey and taxonomy
Yazici, A., Kucukyilmaz, T., Dokeroglu, T., Sharipbay, A., Lee, M.-H., & Tyler, B. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200642 · Cited by 11
Multimodal Strategy To Defend Mobile Devices Against Vishing Attacks
Mishra, M., Jain, S. K., Suraj Patel, M., Mani, S., Boragule, A., Sahni, N., & Nair, R. C. (2024) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3636534.3690683 · Cited by 4
Matsouliadis, L., Siamtanidou, E., Vryzas, N., & Dimoulas, C. A. (2025) · Information · doi:10.3390/info16030238 · Cited by 4
Implementing Fusion Techniques for the Classification of Paralinguistic Information
Vlasenko, B., Sebastian, J., Kumar, D. S. P., & Magimai.-Doss, M. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2360 · Cited by 4
Wang, X., Xie, L., Li, C., Wang, M., Liu, Z., Peng, X., & Wang, Z. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01925-z · Cited by 4
Progress in Multimodal Affective Computing: From Machine Learning to Deep Learning
Chanchal, M., & Kumar, B. V. (2023) · EAI/Springer Innovations in Communication and Computing · doi:10.1007/978-3-031-20541-5_6 · Cited by 4
Conscious observational behavior in recognizing landmarks in facial expressions
Xu, K., & Matsuka, T. (2023) · PLoS ONE · doi:10.1371/journal.pone.0291735 · Cited by 4
DriveSense: A Multi-modal Emotion Recognition and Regulation System for a Car Driver
Zhu, L., Zhong, Z., Dai, W., Chen, Y., Zhang, Y., & Chen, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60477-5_7 · Cited by 4
Multitask EfficientNet affective computing for student engagement detection
Thiruthuvanathan, M. M., & Krishnan, B. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19815-3 · Cited by 4
TCA-NET: Triplet Concatenated-Attentional Network for Multimodal Engagement Estimation
He, H., Wang, D., Hasan, M. R., Gedeon, T., & Hossain, M. Z. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10647692 · Cited by 4
Multimodal emotion recognition system for e-learning platform
Vani, R. K., & Jayashree, P. R. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-13279-6 · Cited by 4
Evaluating Citizens’ Sentiments in Smart Cities: A Deep Learning Approach
Elabora, A., Alkhatib, M., Mathew, S. S., & Barachi, M. E. (2020) · International Conference on Smart and Sustainable Technologies (SpliTech) · doi:10.23919/splitech49282.2020.9243768 · Cited by 4
Towards Learning a Joint Representation from Transformer in Multimodal Emotion Recognition
Deng, J. J., & Leung, C. H. C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86993-9_17 · Cited by 4
Fraile, M., Lindblad, J., Fawcett, C. A., Sladoje, N., & Castellano, G. (2021) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3461615.3485443 · Cited by 4
Emotion-aware Human–Robot Interaction and Social Robots
Tian, L., Oviatt, S. L., Muszyński, M., Chamberlain, B. C., Healey, J., & Sano, A. (2022) · ACM eBooks · doi:10.1145/3502398.3502407 · Cited by 4
Affect-DML: Context-Aware One-Shot Recognition of Human Affect using Deep Metric Learning
Peng, K., Roitberg, A., Schneider, D., Koulakis, M., Yang, K., & Stiefelhagen, R. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9666940 · Cited by 4
Zhang, W. (2022) · Wireless Communications and Mobile Computing · doi:10.1155/2022/7691586 · Cited by 4
X-Norm: Exchanging Normalization Parameters for Bimodal Fusion
Yin, Y., Xu, J., Zu, T., & Soleymani, M. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556581 · Cited by 4
Enhanced Human-Machine Interactive Learning for Multimodal Emotion Recognition in Dialogue System
Leung, C. H. C., Deng, J. J., & Li, Y. (2022) · International Conference on Algorithms, Computing and Artificial Intelligence (ACAI) · doi:10.1145/3579654.3579764 · Cited by 4
Multimodal Emotion Recognition Using Deep Learning Techniques
James, S. J., & Jacob, L. (2022) · International Conference on Advances in Computing, Communication Control and Networking (ICAC3N) · doi:10.1109/icac3n56670.2022.10074512 · Cited by 4
Yang, J., Li, Z., & Du, X. (2023) · Data Technologies and Applications · doi:10.1108/dta-08-2023-0414 · Cited by 4
Active Speaker Detection Using Audio, Visual, and Depth Modalities: A Survey
Robi, S. N. A. B. M., Ariffin, M. A. Z. B. M., Izhar, M. A. M., Ahmad, N. B., & Kaidi, H. M. (2024) · IEEE Access · doi:10.1109/access.2024.3426670 · Cited by 4
Granell, E., Romero, V., & Martínez-Hinarejos, C.-D. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.01.081 · Cited by 4
Emotion detection on phone calls during emergency using ensemble model with hyper parameter tuning
selvan, A. K., Nimmi, K., Janet, B., & Sivakumaran, N. (2022) · International Journal of Information Technology · doi:10.1007/s41870-022-01091-9 · Cited by 4
Ayoub, I. (2019) · uO Research (University of Ottawa) · doi:10.20381/ruor-23584 · Cited by 4
Single-Channel Speech Separation with Auxiliary Speaker EmbeddingsPreprint
Liu, S., Keren, G., & Schuller, B. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.09997 · Cited by 4
Self-citationEnhancing Driver Emotion Recognition Through Deep Ensemble Classification
Zaman, F., Xu, Z., Hussain, A., Ullah, A., & Zaman, K. (2025) · Journal of Intelligent and Connected Vehicles · doi:10.26599/jicv.2025.9210055 · Cited by 3
Adaptive multimodal emotion detection for mental health monitoring using deep learning
Arzu, G. E., Umar, M., Khan, A., Ali, U., Dang, L. M., & Moon, H. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123385 · Cited by 5
Analytical Review and Study on Emotion Recognition Strategies Using Multimodal Signals
Vala, J., & Jaliya, U. K. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-23092-9_21 · Cited by 3
Li, D. (2024) · Systems and Soft Computing · doi:10.1016/j.sasc.2024.200133 · Cited by 3
Multimodal Emotion Recognition System based on X-Vector Embeddings and Convolutional Neural Networks
Gărăiman, E., & Rădoi, A. (2024) · International Conference on Communications (COMM) · doi:10.1109/comm62355.2024.10741406 · Cited by 3
Datasets for Valence and Arousal Inference: A Survey
Schneider, H. M., Pavlitska, S., Gremmelmaier, H., & Zöllner, J. M. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00563 · Cited by 3
Praveen, R. G., Alam, J., & Charton, É. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00573 · Cited by 3
Hybrid Multidimensional Deep Convolutional Neural Network for Multimodal Fusion
Vynokurova, O. А., & Peleshko, D. (2020) · IEEE International Conference on Data Stream Mining & Processing (DSMP) · doi:10.1109/dsmp47368.2020.9204215 · Cited by 3
Hybrid Deep Convolutional Neural Network with Multimodal Fusion
Vynokurova, O. А., Peleshko, D., & Peleshko, M. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-61656-4_4 · Cited by 3
3D skeletal movement-enhanced emotion recognition networks
Shi, J., Liu, C., Ishi, C. T., & Ishiguro, H. (2021) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2021.11 · Cited by 3
Sparse Spatial-Temporal Emotion Graph Convolutional Network for Video Emotion Recognition
Liu, X., Xu, H., & Wang, M. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/3518879 · Cited by 4
A Multimodal Emotion Perception Model based on Context-Aware Decision-Level Fusion
Chen, Y., Jia, Z., Hirota, K., & Dai, Y. (2022) · Chinese Control Conference (CCC) · doi:10.23919/ccc55666.2022.9902799 · Cited by 3
Deepika, C., & Kuchibhotla, S. (2023) · Soft Computing · doi:10.1007/s00500-023-08458-5 · Cited by 3
RetractedDynamic facial expression recognition with pseudo‐label guided multi‐modal pre‐training
Yin, B., Yin, S. N., Liu, C., Zhang, Y., Xi, C., Yin, B., & Ling, Z.-H. (2023) · IET Computer Vision · doi:10.1049/cvi2.12217 · Cited by 3
Multimodal Prediction of User's Performance in High-Stress Dialogue Interactions
Gilani, S. N., Pollard, K. A., & Traum, D. R. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3610661.3617166 · Cited by 3
Emotion Recognition through Text, Speech and Image
Reddy, K. S. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.57652 · Cited by 3
Multimodal Emotion Recognition in Video, Audio, and Text using Deep and Transfer Learning
Panchal, V., Deshpande, T., Kandoi, T., Deulkar, K., & Narvekar, M. (2023) · International Conference on Advanced Computing Technologies and Applications (ICACTA) · doi:10.1109/icacta58201.2023.10392410 · Cited by 3
Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data
Song, Y., Feng, L., Zhang, W., Song, X., & Cheng, M. (2024) · IEEE China Conference on System Simulation Technology and its Applications (CCSSTA) · doi:10.1109/ccssta62096.2024.10691734 · Cited by 3
Mobile Game Evaluation Method Based on Data Mining of Affective Time Series
Ochab, J. K., Wȩgrzyn, P., Witaszczyk, P., Drążyk, D., & Nalepa, G. J. (2025) · Sensors · doi:10.3390/s25092756 · Cited by 6
Improving Emotion Recognition Performance by Random-Forest-Based Feature Selection
Egorow, O., Siegert, I., & Wendemuth, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99579-3_15 · Cited by 3
A Neural Network Based Approach for Recognition of Basic Emotions from Speech
Sham-E-Ansari, M., Disha, S. T., Chowdhury, A. I., & Hasan, M. K. (2020) · IEEE Region 10 Symposium (TENSYMP) · doi:10.1109/tensymp50017.2020.9230641 · Cited by 3
Tzirakis, P. (2020) · doi:10.1145/3423327.3423513 · Cited by 3
Self-citationTang, D., Kuppens, P., Geurts, L., & van Waterschoot, T. (2020) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco47968.2020.9287667 · Cited by 2
Choudhury, N., & Sharma, U. (2021) · Indian Journal of Computer Science and Engineering · doi:10.21817/indjcse/2021/v12i1/211201187 · Cited by 2
Deep Neural Networks for Automatic Facial Expression Recognition
Veesam, V. S., Ravichandran, S., & Babu, G. R. M. (2022) · Revue d intelligence artificielle · doi:10.18280/ria.360520 · Cited by 3
Emotion Recognition by Deep Learning and Cloud Access
Satyanarayana, P., Vardhan, D. P., Tejaswi, R., & Kumar, S. V. P. (2021) · International Conference on Advances in Computing, Communication Control and Networking (ICAC3N) · doi:10.1109/icac3n53548.2021.9725583 · Cited by 3
Multi-model Emotion Recognition Using Hybrid Framework of Deep and Machine Learning
Fahad, M. S., Juhi, A., Shambhavi, Ranjan, A., & Deepak, A. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-16-9089-1_13 · Cited by 3
Sinko, M., Medvedev, A., Smirnov, I., Laushkina, A., Kadnova, A., & Basov, O. O. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.11.001 · Cited by 4
FI-Net: A Speech Emotion Recognition Framework with Feature Integration and Data Augmentation
Xia, G., Li, F., Zhao, D., Zhang, Q., & Yang, S. (2019) · International Conference on Big Data Computing and Communications (BigCom) · doi:10.1109/bigcom.2019.00035 · Cited by 3
Multimodal Embeddings from Language ModelsPreprint
Tseng, S.-Y., Georgiou, P., & Narayanan, S. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.04302 · Cited by 3
Cross-Modal Knowledge Transfer via Inter-Modal Translation and Alignment for Affect RecognitionPreprint
Rajan, V., Brutti, A., & Cavallaro, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.00809 · Cited by 5
Maji, B., Swain, M., & Panda, R. (2022) · Research Square · doi:10.21203/rs.3.rs-1529387/v1 · Cited by 3
Automatic recognition of engagement and emotion in a group of children
Kim, J. (2018) · doi:10.3990/1.9789036545839 · Cited by 3
CNN-BLSTM with Attention Model for Speech Emotion RecognitionPreprint
Xia, Y., & Zhao, L. (2023) · Research Square · doi:10.21203/rs.3.rs-3392008/v1 · Cited by 3
Emotion Recognition Using Text and Speech Through Machine Learning
Singla, C., & Singh, S. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-3656-4_33 · Cited by 2
Current State of Deepfake Detection and Generation: A Review
Chauhan, R., Kansal, I., Popli, R., Kumar, R., & Sharma, A. (2024) · Recent Advances in Electrical & Electronic Engineering (Formerly Recent Patents on Electrical & Electronic Engineering) · doi:10.2174/0123520965269334231117112747 · Cited by 4
Recent Trends in Machine and Deep Learning for Verbal and Non-verbal Emotion Detection
Chawla, M., Panda, S. N., Khullar, V., Kansal, I., & Kumar, R. (2024) · Recent Advances in Electrical & Electronic Engineering (Formerly Recent Patents on Electrical & Electronic Engineering) · doi:10.2174/0123520965303142240430101645 · Cited by 3
Modality-specific and -shared Contrastive Learning for Sentiment Analysis
Liu, D., You, J., Xie, G., Lee, L., Wang, F. L., & Yang, Z. (2024) · International Conference on Multimedia Retrieval · doi:10.1145/3652583.3658004 · Cited by 2
Mamede, L., Sabà, R. C., Coillie, S. V., Prevot, J., Sánchez‐Ramón, S., Poli, M. C., Barasa, A. K., Schuller, B. W., Hendel, A., Garcelon, N., et al. (2025) · Frontiers in Immunology · doi:10.3389/fimmu.2025.1596971 · Cited by 2
Self-citationMultimodal contextual transformer augmented fusion for emotion recognition
Chemma, W. D., Mamuye, A. L., & Piangerelli, M. (2025) · Applied Intelligence · doi:10.1007/s10489-025-07027-7 · Cited by 2
Buechel, S., & Hahn, U. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3635686 · Cited by 4
A DLF multi-scale quantitative research method for the big five personality traits
Ning, T., Guo, Z., & Hou, Q. (2026) · Scientific Reports · doi:10.1038/s41598-025-27837-6 · Cited by 2
Joloudari, J. H., Maftoun, M., Nakisa, B., Alizadehsani, R., Yadollahzadeh‐Tabari, M., & Gaftandzhieva, S. N. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1682883 · Cited by 12
A multimodal meta-learning-augmented EmoTriSense for emotion recognition
Sowmya, V., & Rajeswari, A. (2026) · Journal of Combinatorial Optimization · doi:10.1007/s10878-026-01402-x · Cited by 2
Shyness Trait Recognition for Schoolchildren via Multi-View Features of Online Writing
Tian, X., Jing, L., Luo, F., & Zhang, S. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3077410 · Cited by 2
Speech Emotion Recognition Using Energies in six bands and Multilayer Perceptron on RAVDESS Dataset
Agrima, A., Barakat, A., Mounir, I., Farchi, A., Elmaazouzi, L., & Mounir, B. (2022) · International Conference on Advanced Communication Technologies and Networking (CommNet) · doi:10.1109/commnet56067.2022.9993940 · Cited by 2
Hierarchical emotion recognition from speech using source, power spectral and prosodic features
Haque, A., & Rao, K. S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16264-2 · Cited by 2
A Digital Twin-Based Approach for Emotion Recognition in Human-Robot Collaboration
Baratta, A., Longo, F., Mirabelli, G., Padovano, A., & Solina, V. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-38274-1_14 · Cited by 2
An emotion-driven and topic-aware dialogue framework for human–robot interaction
Lu, I.-C., Huang, J.-Y., & Lee, W. (2023) · Advanced Robotics · doi:10.1080/01691864.2023.2297902 · Cited by 4
Zhong, Q., & Shao, X.-H. (2024) · Intelligent Data Analysis · doi:10.3233/ida-230305 · Cited by 2
A Cross-Attention Emotion Recognition Algorithm Based on Audio and Video Modalities
Wu, X., Mu, X., Qi, W., & Liu, X. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10626511 · Cited by 2
Metadata
- Authors
- Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S.
- Affiliations
- Imperial College London (GB)
- Goldsmiths University of London (GB)
- University of Augsburg (DE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 11
- Issue
- 8
- Pages
- 1301-1309
- ISSN
- 1932-4553 (ISSN-L)
- arXiv
- 1704.08619
- OpenAlex
- W2610961739
- Semantic Scholar
- 8cf0c072eeaec310aff806bdb7a7931b81e9ed99
- Cited by
- 752
- References
- 53
- Citations by year
- 2026: 67 · 2025: 90 · 2024: 81 · 2023: 99 · 2022: 108 · 2021: 102 · 2020: 76 · 2019: 82 · 2018: 42 · 2017: 5
- FWCI
- 72.53
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- Goldsmiths (University of London) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Tekes
- Engineering and Physical Sciences Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal emotion recognition, audio-visual emotion recognition, end-to-end learning, convolutional neural networks, ResNet, long short-term memory networks, feature extraction, spontaneous emotions, outlier robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
