The DOI matches its DataCite record; 55 of the 55 references it cites were found in a registry or an index; 1 of the 7 sources its claims cite were checked for support.
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Open accessAbstract
Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a Convolutional Neural Network (CNN) to extract features from the speech, while for the visual modality a deep residual network (ResNet) of 50 layers. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, Long Short-Term Memory (LSTM) networks are utilized. The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an end-to-end multimodal emotion recognition system that learns from raw speech and visual signals using a CNN, ResNet-50, and LSTM layers, trained to directly maximize concordance correlation coefficient (ρc). On the RECOLA database, the unimodal speech and visual models outperform several handcrafted-feature baselines, and the multimodal model improves valence prediction over prior multimodal systems.
Key findings
- For speech, the proposed raw-signal model optimized with ρc achieved test ρc of .699 for arousal and .311 for valence, compared with BLSTM eGeMAPS (.316 arousal, .195 valence) and BLSTM ComParE (.382 arousal, .187 valence); the MSE-optimized version achieved .684 arousal and .249 valence.
- For visual modality, the proposed raw-signal model achieved test ρc of .435 for arousal and .620 for valence, outperforming listed models in valence, e.g., baseline (.272 arousal, .507 valence) and Han et al. (.265 arousal, .394 valence).
- In multimodal analysis, the proposed model outperformed OA RVM-SR and Han et al. in valence “with high magnitude,” while OA RVM-SR produced the best arousal result; the provided table does not show the proposed multimodal ρc values.
- Sequence-length ablation found best speech performance at length 150 (arousal .744, valence .369) and best visual performance at length 300 (arousal .193, valence .496), though the multimodal network was trained with length 150; gate activations correlated with RMS energy range (ρ = .83), loudness (ρ = .73), and mean fundamental frequency (ρ = .72).
Methods
- Architecture: speech CNN operates on raw waveform, visual ResNet-50 operates on cropped face pixel intensities, and extracted features are concatenated into a 1920-dimensional vector (1280 speech + 640 visual) feeding a 2-layer LSTM with 256 cells each; unimodal networks are pretrained separately before end-to-end multimodal training.
- Data: RECOLA database, 46 French-speaking participants, 9.5 h of multimodal recordings, 5 minutes each in dyads; split into train (16 subjects), validation (15), and test (15), stratified by gender and age; 6 annotators. Audio is segmented into 6 s sequences at 16 kHz and further into 150 subsequences for 40 ms annotation granularity; visual images are 96×96.
- Training/evaluation: Adam optimizer with fixed learning rate 10⁻⁴, dropout p = 0.5, mini-batch 25 for audio and 2 for video, 60 epochs; metric is ρc; postprocessing includes median filtering, centering, scaling, and time-shifting selected on validation ρc.
Limitations
- The paper states that the OA RVM-SR multimodal comparison was trained using both training and validation sets, whereas this model used only the training set, and that OA RVM-SR used geometric features requiring accurate facial landmark tracking while this model operated on raw pixels; the authors expect their results could improve with similar strategies.
- The paper states hardware limitations forced a small video mini-batch of size 2, and notes memory and training instability concerns with very large recurrent sequences.
- Evident from the text: only two other multimodal models were found for comparison, RECOLA includes ECG and EDA modalities but this study uses only audio and video, and the provided multimodal results table lacks the proposed model’s exact ρc values.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
19 Own result · 5 No full text of the sources · 5 No basis given · 3 Own method · 3 Own interpretation · 1 Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the arousal dimension the OA RVM-SR produces the best results.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The only exception is the Video CNN-L4 model which performs better in the arousal dimension when appearance features are used.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This plot shows that certain cells of the model are very sensitive to different features conveyed in the original speech wave form.”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results show that our model outperforms the other models in the test set when predicting arousal dimension.”
plainn = 27- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As expected visual modality benefits the models in the valence dimension.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In all of the experiments, our model outperforms the designed features in terms of ρ c .”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we study the gate activations of the recurrent layers in the speech modality and find cells that are highly correlated with prosodic features that were always assumed to cause arousal.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our model outperforms the other two models in the valence dimension with high magnitude.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 3
Every source the paper cites for this claim was checked and supports it.
In the paper
“It is well accepted amongst the research community that certain acoustic and prosodic features play an important role in recognising the affective state [45].”
What the sources say
- Supportedcites [45] [45] K. Scherer, 'Vocal communication of emotion: A review of research paradigms,' Speech Communication , vol. 40, no. 1-2, pp. 227-256, April 2003. - source
Section 2.3, “Inference studies” (page not provided), reports that acoustic measures including pitch, energy, and duration explained a large proportion of variation in listeners’ emotion judgments. It also reports that manipulating vocal features affected judgments of a speaker’s emotional state, supporting the claim that such features matter for recognition. (LLM, conf=0.94)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
hedgedn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Some of these features, such as the mean of the fundamental frequency (F0), mean speech intensity, loudness, as well as pitch range [46], should thus be captured by our model.”
What the sources say
- Not checked — no full text of the sourcecites [46] [46] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan et al. , 'The geneva minimalistic acoustic parameter set (gemaps) for voice research and - affective computing,' IEEE Transactions on Affective Computing , vol. 7, no. 2, pp. 190-202, 2016. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
In the paper
“The task of recognising emotions is challenging because human emotions lack of temporal boundaries and different individuals express emotions in different ways [3].”
What the sources say
- Not checked — no full text of the sourcecites [3] [3] C.-N. Anagnostopoulos, T. Iliou, and I. Giannoukos, 'Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,' Artificial Intelligence Review , vol. 43, no. 2, pp. 155-177, 2015. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to perform a fair comparison we apply the proposed method on the test set of the AVEC challenge.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset is split in three partitions -train (16 subjects), validation (15 subjects) and test (15 subjects) -by stratifying (i.e., balancing) the gender and the age of the speakers.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Any of these post-processing steps is kept when an improvement is observed on the ρ c of the validation set, and applied then with the same configuration on the test partition.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
The proposed work is the first in literature to apply an end-to-end model for audio-visual emotion recognition.
“To our knowledge this is the first work in literature that applies such an end-to-end model for audio visual emotion recognition task.”
The paper relies on this without support.
Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
“The speech signals convey information about the affective state either explicitly, i.e., by linguistic means, or implicitly, i.e., by acoustic or prosodic cues.”
The paper relies on this without support.
Visual modality more easily predicts the valence dimension rather than the arousal.
“Visual modality has been shown to more easily predict the valence dimension rather than the arousal.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Dropout is applied in fullyconnected (reported in 43% of studies, e.g., [135]) and convolutional layers (14% of studies, e.g., [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We see both CNN-RNN (e.g., [18], [90]) and CNN-LSTM (e.g., [108], [105], [64]) architectures, with CNN-LSTM being the more frequent choice among the reviewed studies.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Global temporal modeling is found to lead to improved accuracies when compared with simpler methods such as pooling of spatial features (e.g., [152], [90], [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Applications of similar architectures based on the raw waveform have also been attempted [54], [108], showing that end-to-end learning can outperform shallow models.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“End-to-end training of the entire CNN-LSTM system addresses this problem, and can lead to accuracy improvements [108].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We are starting to see studies implementing end-toend training for such models [54], [108]”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Learning from raw audiovisual data with two CNNs, Tzirakis et al. [108] used a two-layer LSTM network for feature fusion, which was found to outperform the state of the art.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Similarly, for SER, Tzirakis et al. [108] found that LSTM cells learn representations similar to well-known prosodic features.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ I. INTRODUCTION
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
§ B. Speech Modality
“| Baseline [29] | eGeMAPS | .648 (.796) | .375 (.455) | | RVM [30] | eGeMAPS | - (.750) | - (.396) | | Audio BN-Multi [33] | Mixed | - (.833) | - (.503) | | Brady et al [32] | MFCC | - (.846) | - (.450) | | Weber et al. [31] | eGeMAPS | - (.793) | - (.456) | | Somandepalli et al. [34] | Mixed | - (.800) | - (.448) |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
(2010) · Cited by 14,134
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Research works printed but not linked: 1
- [37][37] F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data,' Pattern Recognition Letters , vol. 66, pp. 22-30, November 2015. -
Cited by
Sorted by citations
Items 501–600 of 752
Improving Valence-Arousal Estimation with Spatiotemporal Relationship Learning and Multimodal Fusion
Yu, J., Zhao, G., Wang, Y., Wei, Z., Zhang, Z., Cai, Z., Xie, G., Zhu, J., Zhu, W., Yang, S., et al. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00785 · Cited by 2
Integrating gating and learned queries in audiovisual emotion recognition
Zhang, Z., Guo, Q., Lu, S., Su, J., & Tang, T. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01551-1 · Cited by 2
A Study on Multimodal Fusion and Layer Adapter in Emotion Recognition
Shi, X., Gao, Y., He, J., Mi, J., Li, X., & Toda, T. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10848773 · Cited by 2
Sonar, Y., Wajage, P., Sunke, K., & Bidkar, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-7502-9_2 · Cited by 2
A Brief Review of Facial Expressions Recognition System
Joshi, D. (2018) · Cited by 2
Convolutional Neural Networks Model for Emotion Recognition Using EEG Signal
Ahmad, I. S., Zhang, S., Wang, L., Saminu, S., Isselmou, A. E. K., Cai, Z., Javaid, I., Kamhi, S., & Kulsum, U. (2021) · International Journal of Circuits Systems and Signal Processing · doi:10.46300/9106.2021.15.46 · Cited by 2
Singular Learning of Deep Multilayer Perceptrons for EEG-Based Emotion Recognition
Guo, W., Li, G., Lu, J., & Yang, J. J. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.786964 · Cited by 2
Nonverbal conversation expressions processing for human-agent interactions
Haddad, K. E. (2017) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2017.8273663 · Cited by 2
Adversarial and Cooperative Correlated Domain Adaptation based Multimodal Emotion Recognition.
Chen, J. Q. X., & Hu, K. (2019) · National Conference on Artificial Intelligence · Cited by 2
Causality Analysis of Emotional States from EEG Response
Raha, R., Sengupta, A., & Saha, A. (2020) · IEEE International Conference for Convergence in Engineering (ICCE) · doi:10.1109/icce50343.2020.9290546 · Cited by 2
Analysis of Tourists' Nationality Effects on Behavior-based Emotion and Satisfaction Estimation
Matsuda, Y., Fedotov, D., Arakawa, Y., Suwa, H., Minker, W., & Yasumoto, K. (2020) · International Conference on Informatics, Electronics & Vision (ICIEV) · doi:10.1109/icievicivpr48672.2020.9306525 · Cited by 2
Identifying Criminal Suspects by Human Gestures Using Deep Learning
Prabha, P. A., Vighneshbalaji, S. A., Surya, C. S., & Kumar, K. V. A. (2021) · Lecture notes in networks and systems · doi:10.1007/978-981-33-4501-0_67 · Cited by 2
Artificial Intelligence and Machine Learning-Based Advanced Depression Detection System
Patil, S. M., More, N., Bhawtankar, A., Jagtap, V. P., & Jadhav, A. T. (2021) · Advances in healthcare information systems and administration book series · doi:10.4018/978-1-7998-7709-7.ch006 · Cited by 2
Contrastive Learning for Domain Transfer in Cross-Corpus Emotion Recognition
Yin, Y., Lu, L., Xiao, Y., Xu, Z. J., Cai, K., Jiang, H., Gratch, J. S., & Soleymani, M. (2021) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii52823.2021.9597453 · Cited by 2
Temporal Attentive Adversarial Domain Adaption for Cross Cultural Affect Recognition
Chen, H., Deng, Y., & Jiang, D. (2021) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3461615.3491110 · Cited by 2
Chen, M. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/5156532 · Cited by 2
Analysis on Speech-Emotion Recognition with Effective Feature Combination
Patra, S., Datta, S., & Roy, M. (2022) · OITS International Conference on Information Technology (OCIT) · doi:10.1109/ocit56763.2022.00018 · Cited by 2
Jagadeesh, M., Sajeev, N., & Kumar, N. (2023) · International Conference on Smart Systems and Inventive Technology (ICSSIT) · doi:10.1109/icssit55814.2023.10060940 · Cited by 2
Design an Optimum Feature Selection Method to Improve the Accuracy of the Speech Recognition System
Deepika, C., & Kuchibhotla, S. (2023) · SN Computer Science · doi:10.1007/s42979-023-02032-4 · Cited by 2
A Deep Learning Technique to Recommend Music Based on Facial and Speech Emotions
Reddy, R. P., Abhinaya, B., & Sahithi, A. (2023) · Smart innovation, systems and technologies · doi:10.1007/978-981-99-3982-4_3 · Cited by 2
Automated Classification of Children’s Linguistic versus Non-Linguistic Vocalisations
Zhang, Z., Cristià, A., Warlaumont, A. S., & Schuller, B. W. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2523 · Cited by 2
Self-citationA Survey on Image Acquisition Protocols for Non-posed Facial Expression Recognition Systems
Saha, P., Bhattacharjee, D., De, B. K., & Nasipuri, M. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-7596-2 · Cited by 2
Modelling Naturalistic Work Stress Using Spectral HRV Representations and Deep Learning
Hagad, J. L., Fukui, K., & Numao, M. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-39878-1_24 · Cited by 2
Towards Real-Time Contextual Touristic Emotion and Satisfaction Estimation with Wearable Devices
Fedotov, D., Matsuda, Y., Takahashi, Y., Arakawa, Y., Yasumoto, K., & Minker, W. (2019) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomw.2019.8730864 · Cited by 2
A REVIEW ON FACE RECOGNITION TECHNIQUE USING THE ARTIFICIAL NEURAL NETWORK
Technical Officer (C) in ECIL (NPCIL), T., & Panwar, A. (2020) · International Journal of Advanced Research in Computer Science · doi:10.26483/ijarcs.v11i3.6629 · Cited by 2
Prediction of instantaneous likeability of advertisements using deep learning
Saha, D., Rahman, S. M. M., Islam, M. T., Ahmad, M. O., & Swamy, M. N. S. (2021) · Cognitive Computation and Systems · doi:10.1049/ccs2.12022 · Cited by 2
Contextually Aware Multimodal Emotion Recognition
Shah, P., Raj, P. P., Suresh, P., & Das, B. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-981-15-7234-0_71 · Cited by 2
End-to-end Multimodal Emotion and Gender Recognition with Dynamic Joint Loss WeightsPreprint
Chae, M., Kim, T., Shin, Y. H., Kim, J.-W., & Lee, S.-Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00758 · Cited by 2
Audio-Visual Kinship VerificationPreprint
Wu, X., Granger, É., & Feng, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10096 · Cited by 2
Weakly Supervised Learning for Facial Behavior Analysis: A ReviewPreprint
Praveen, R. G., Cardinal, P., & Granger, É. (2024) · doi:10.36227/techrxiv.173091765.50249538/v1 · Cited by 2
Application of the General Data Protection Regulation on Household Social Robots
Gültekin-Várkonyi, G. (2020) · doi:10.14232/phd.10627 · Cited by 2
Aslam, M. H., Pedersoli, M., Koerich, A. L., & Granger, É. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5270694 · Cited by 2
Introduction to Applied Affective Computing
Tian, L., Oviatt, S. L., Muszyński, M., Chamberlain, B. C., Healey, J., & Sano, A. (2022) · ACM eBooks · doi:10.1145/3502398.3502400 · Cited by 1
Emotion Recognition in the Wild
Tian, L., Oviatt, S. L., Muszyński, M., Chamberlain, B. C., Healey, J., & Sano, A. (2022) · ACM eBooks · doi:10.1145/3502398.3502404 · Cited by 2
WiFi and Vision enabled Multimodal Emotion Recognition
Hou, Y., Zhang, X., Gu, Y., & Li, W. (2022) · International Conference on Communications · doi:10.1109/icc45855.2022.9838315 · Cited by 1
Towards Cheaper Tourists' Emotion and Satisfaction Estimation with PCA and Subgroup Analysis
Maris, L., Matsuda, Y., Sadre, R., & Yasumoto, K. (2023) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops56833.2023.10150377 · Cited by 1
Ganesh, A., & Ramachandiran, R. (2023) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-235503 · Cited by 1
RetractedFacial Emotion Recognition usign CNN
Meghana, P., Khan, M. Y., Bharti, M., Mohanta, K., & Kanaiya, V. K. (2023) · Journal Of Advanced Zoology · doi:10.17762/jaz.v44is6.2328 · Cited by 1
Kaur, D., Misra, A., & Vyas, O. P. (2024) · International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT) · doi:10.1109/icaect60202.2024.10469176 · Cited by 1
Emotion Recognition by Facial Expressions and Speech Using Deep Learning
Vardhan, J. V., Yelavarti, K. C., & Chand, A. J. (2024) · International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT) · doi:10.1109/icaect60202.2024.10469626 · Cited by 1
Facial Expression Recognition Using Convolutional Neural Network
Govindwar, V., Akbani, A., Nandeshwar, A., Wanjari, A., Agashe, P., & Pote, C. R. (2024) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit24102122 · Cited by 1
Supporting Interpersonal Emotion Regulation of Call Center Workers via Customer Voice Modulation
Lee, D., Nam, K., & Lee, U. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650968 · Cited by 1
CNN Approach for Live Speech Emotion Detection
Subramani, N., Sivasankar, S., Saravanan, K., & Sakthivel, K. (2025) · International Conference on Electronics and Renewable Systems (ICEARS) · doi:10.1109/icears64219.2025.10940376 · Cited by 1
Sun, Y., Ke, M., & Wang, L. (2025) · International Conference on Bioinformatics and Intelligent Computing (BIC) · doi:10.1145/3724979.3725068 · Cited by 1
Predicting Emotions in Interpersonal Interaction Videos: I Know What You Feel
Guerdelli, H., Ferrari, C., Berretti, S., Barhoumi, W., & Bimbo, A. D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91581-9_16 · Cited by 1
Fang, H., Ji, Y., Yuan, S., Qiu, G., Li, H., Zhang, Z., & Zhou, L. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108165 · Cited by 1
Attentional Residual Network Based Spatial Transformer Mechanism for Facial Expression Recognition
Shao, D., Gao, G., & Ma, L. (2025) · Journal of Intelligent & Fuzzy Systems · doi:10.1177/18758967251355732 · Cited by 1
Zhu, Y., Tang, J., Wei, H., Gan, K., Zhang, J., & Yin, Z. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113659 · Cited by 1
MusiQAl: A Dataset for Music Question–Answering through Audio–Video Fusion
Christodoulou, A.-M., Glette, K., Lartillot, O., & Jensenius, A. R. (2025) · Transactions of the International Society for Music Information Retrieval · doi:10.5334/tismir.222 · Cited by 5
Enhanced Speech Emotion Recognition Using Augmented Spectrograms and Transfer Learning with VGG16
Kurapati, N. R., Sinde, N., Chinta, A., Salemula, S., & Gajula, H. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-92614-3_4 · Cited by 1
Multimodal data analysis in emotion recognition: a review
Berdyshev, D. A., & Shishkin, A. G. (2025) · Modeling and Analysis of Information Systems · doi:10.18255/1818-1015-2025-3-252-281 · Cited by 1
MKDF: Knowledge Distillation Based Lightweight Multimodal Framework for Emotion Recognition
Wang, Y., Sun, Y., Qiao, B., Wang, C., Li, Z., & Yang, S. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4091-4_6 · Cited by 1
Speech emotion recognition: A systematic mega-review of techniques and pipelines
Chakhtouna, A., Sekkate, S., & Adib, A. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104161 · Cited by 2
Singh, A. K., & Shukla, A. K. (2025) · IEEE International Conference on Intelligent Signal Processing and Effective Communication Technologies (INSPECT) · doi:10.1109/inspect67393.2025.11350529 · Cited by 1
Deep learning-based stress detection using encoded physiological signal representations
Nandan, A. D. M., Praveen, K., Thejdeep, R. C., Bhat, P., & Patil, N. (2026) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-026-00730-6 · Cited by 1
Multimodal Depression Detection Using Audio, Video and Text Fusion with PCA-Based Feature Selection
Ahmed, T., & Sahoo, R. C. (2025) · International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit68829.2025.11434217 · Cited by 1
Patel, J., & Patel, P. (2026) · Intelligent Hospital · doi:10.1016/j.inhs.2026.100066 · Cited by 1
Zhang, T., Dahlan, H. A., Rosdi, F., & Wang, X. (2026) · Journal of the Chinese Institute of Engineers · doi:10.1080/02533839.2026.2683464 · Cited by 1
Hierarchical Quantized Cross-modal Masked Autoencoders for Audio-Visual Emotion Recognition
Zhang, Z., & Busso, C. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831121 · Cited by 1
Cross-culture Continuous Emotion Recognition with Multimodal Features
He, J. Y. D., Rao, P., Cao, S., Sun, B., Yu, L., Yan, H., & Xiao, Y. (2019) · International Conference on Computing and Pattern Recognition (ICCPR) · doi:10.1145/3373509.3373515 · Cited by 2
Reinforcement Learning and Affective Computing
Tian, L., Oviatt, S. L., Muszyński, M., Chamberlain, B. C., Healey, J., & Sano, A. (2022) · ACM eBooks · doi:10.1145/3502398.3502405 · Cited by 1
Multimodal Affect Recognition Using Temporal Convolutional Neural Networks
Ayoub, I., Heiries, V., & Osman, H. A. (2022) · IEEE International Joint Conference on Neural Network · doi:10.1109/ijcnn55064.2022.9892145 · Cited by 1
Cross-Modality Learning by Exploring Modality Interactions for Emotion Reasoning
Tran, T.-D., Ho, N.-H., Pant, S., Yang, H.-J., Kim, S.-H., & Lee, G.-S. (2023) · IEEE Access · doi:10.1109/access.2023.3283597 · Cited by 1
Deep Learning Approach For Emotions Detection
Anupama, Save, V., Biswas, A., Kumar, V., Lourens, M., Madhura, K., & Sharma, A. (2023) · E3S Web of Conferences · doi:10.1051/e3sconf/202339907007 · Cited by 1
Zhilenkov, A. (2023) · Artificial Societies · doi:10.18254/s207751800027272-8 · Cited by 1
Work-in-Progress: Deep Learning Classification Models for Infant Cry Diagnostic
Martinez-Cañete, Y., Cano-Ortiz, S. D., & Langmann, R. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-42467-0_62 · Cited by 1
Depression Detection and Accuracy Analysis Using Various Machine Learning Algorithms on Twitter
Johri, P., Tyagi, A., Sharma, M., & Bhardwaj, D. (2023) · International Conference on Communication, Security and Artificial Intelligence (ICCSAI) · doi:10.1109/iccsai59793.2023.10421313 · Cited by 1
Gradient-Based Dimensionality Reduction for Speech Emotion Recognition Using Deep Networks
Wang, H., & Vadakkepat, P. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447616 · Cited by 1
Emotional Expression as a Means of Communicating Virtual Human Personalities
Sonlu, S., Shabanovi, K., Güdükbay, U., & Durupınar, F. (2023) · Springer eBooks · doi:10.1007/978-3-031-50269-9_17 · Cited by 1
An integrated CycleGAN-Diffusion approach for realistic image generation
Liu, B., Ji, J., Gu, L., & Jiang, Z. (2024) · doi:10.1117/12.3027121 · Cited by 1
Emotion Detection through Human Verbal Expression Using Deep Learning Techniques
Tiwari, R., Prajapati, A., Chandran, S., Agrawal, D., Rasool, A., & Jadhav, A. (2024) · IEEE International Students' Conference on Electrical, Electronics and Computer Science (SCEECS) · doi:10.1109/sceecs61402.2024.10481854 · Cited by 1
Progressive Fusion Network with Mixture of Experts for Multimodal Sentiment Analysis
Liu, D., Yang, Z., & Guo, Z. (2024) · International Conference on Advanced Computational Intelligence (ICACI) · doi:10.1109/icaci60820.2024.10537022 · Cited by 1
Speech emotion recognition using deep learning algorithms
Kumar, T. S., Anugam, S., Irukulla, A., Grace, G. H., Daddanala, M., & Anugam, U. (2024) · AIP conference proceedings · doi:10.1063/5.0217507 · Cited by 1
Enhancing Speech Emotion Recognition Using Transfer Learning from Speaker Embeddings
Jakubec, M., Jarina, R., Lieskovská, E., Kasák, P., & Spišiak, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70566-3_17 · Cited by 1
Cai, Y., Ye, R., Xie, J., Zhou, Y., Xu, Y., & Wu, Z. (2024) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3689092.3689418 · Cited by 2
ZHANG, D., Huang, X., Li, G., Kong, S., & Liang, D. (2025) · Applied Sciences · doi:10.3390/app15094670 · Cited by 1
UMEDNet: a multimodal approach for emotion detection in the Urdu language
Majeed, A. A., & Mujtaba, H. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2861 · Cited by 2
AI based Psychometric Assessment using Multimodal Signal Data
Amaresh, R., & Aote, S. S. (2025) · International Conference on Augmented Intelligence and Sustainable Systems (ICAISS) · doi:10.1109/icaiss61471.2025.11041804 · Cited by 1
Maximum agreement linear predictors
Kim, T., Chaussé, P., Bottai, M., Doros, G. D., Giurcanu, M. C., Luta, G., & Peña, E. A. (2026) · Electronic Journal of Statistics · doi:10.1214/26-ejs2550 · Cited by 1
Ensemble System for Multimodal Emotion Recognition Challenge (MEC 2017)
Zhang, X., Xu, M., & Zheng, T. F. (2018) · Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia) · doi:10.1109/aciiasia.2018.8470352 · Cited by 1
Multi-Layer Progressive Face Alignment by Integrating Global Match and Local Refinement
Gao, N., Wang, X., & Wang, X. (2019) · Applied Sciences · doi:10.3390/app9050977 · Cited by 1
Lykartsis, A., & Kotti, M. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5939 · Cited by 2
Modelling Mental States via Computational Psychophysiology: Benefits and Challenges
Zheng, W., Cai, H., Yao, Z., Zhang, X., Li, X., & Hu, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-37429-7_67 · Cited by 1
Hayashi, R., Matsuda, Y., Fujimoto, M., Suwa, H., & Yasumoto, K. (2021) · International Conference on Mobile Computing and Ubiquitous Network (ICMU) · doi:10.23919/icmu50196.2021.9638938 · Cited by 1
[Retracted] Emotion Recognition of Violin Playing Based on Big Data Analysis Technologies
Zou, L. (2022) · Journal of Environmental and Public Health · doi:10.1155/2022/8583924 · Cited by 1
RetractedVala, J., & Jaliya, U. K. (2023) · International Journal of Engineering Applied Sciences and Technology · doi:10.33564/ijeast.2023.v08i04.021 · Cited by 1
Zhou, Z., Xu, Y., & Li, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06004 · Cited by 1
Towards Privacy-Preserving Affect Recognition: A Two-Level Deep Learning ArchitecturePreprint
Mase, J. M., Leesakul, N., Yang, F., Figueredo, G. P., & Torres, M. T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07344 · Cited by 1
A Survey on Multi-modal Emotion Detection TechniquesPreprint
Chatterjee, C., Shah, N., Bhatt, S., & Chandi, S. (2024) · Research Square · doi:10.21203/rs.3.rs-3942784/v1 · Cited by 1
Linking architecture and emotions: sensory dynamics and methodological innovations
Zhang, Z. (2024) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.5821/dissertation-2117-410676 · Cited by 1
Synergistic Perception: Adaptive Multimodal Sentiment Analysis via Multi-Task LearningPreprint
Cao, T., Ren, B., Cao, H., Chen, A., Bai, S., & Liu, N. (2025) · doi:10.36227/techrxiv.175036750.08379260/v1 · Cited by 1
Abdelkawy, H. (2021) · theses.fr (ABES) · doi:10.70675/3ae0f056z1a4az4407z8309zd6153c0e83d7 · Cited by 1
A Long Short-term Memory Neural Network for Improved Twins' Voice Differentiation
Sabatier, S. B.-R. (2018) · doi:10.33915/etd.6551
Image classification method based on improved deep residual networks
Li, W., & Hua, R. (2019) · International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS) · doi:10.1109/iciibms46890.2019.8991495
3D Skeletal Movement enhanced Emotion Recognition Network
Shi, J., Liu, C., Ishi, C. T., & Ishiguro, H. (2020) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)
Li, H., Kim, Y., Kuo, C.-H., & Narayanan, S. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01978
Priya, A. P., & Malar, J. P. L. (2021) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-1096
Research of Attention-LSTM Model for Baby Cry Detection Robot
Jian, T., Peng, Y., Peng, W., & Zhou, Y. (2021) · Proceedings of International Conference on Artificial Life and Robotics · doi:10.5954/icarob.2021.os3-2
Scaling Depression Level Through Facial Image Processing and Social Media Analysis
Bhayani, A., Meshram, P., Desai, B., Garg, A., & Jha, S. (2021) · Lecture notes in networks and systems · doi:10.1007/978-981-16-1089-9_71
Metadata
- Authors
- Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S.
- Affiliations
- Imperial College London (GB)
- Goldsmiths University of London (GB)
- University of Augsburg (DE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 11
- Issue
- 8
- Pages
- 1301-1309
- ISSN
- 1932-4553 (ISSN-L)
- arXiv
- 1704.08619
- OpenAlex
- W2610961739
- Semantic Scholar
- 8cf0c072eeaec310aff806bdb7a7931b81e9ed99
- Cited by
- 752
- References
- 53
- Citations by year
- 2026: 67 · 2025: 90 · 2024: 81 · 2023: 99 · 2022: 108 · 2021: 102 · 2020: 76 · 2019: 82 · 2018: 42 · 2017: 5
- FWCI
- 72.53
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- Goldsmiths (University of London) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Tekes
- Engineering and Physical Sciences Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal emotion recognition, audio-visual emotion recognition, end-to-end learning, convolutional neural networks, ResNet, long short-term memory networks, feature extraction, spontaneous emotions, outlier robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
