The DOI matches its DataCite record; 55 of the 55 references it cites were found in a registry or an index; 1 of the 7 sources its claims cite were checked for support.
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Open accessAbstract
Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a Convolutional Neural Network (CNN) to extract features from the speech, while for the visual modality a deep residual network (ResNet) of 50 layers. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, Long Short-Term Memory (LSTM) networks are utilized. The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an end-to-end multimodal emotion recognition system that learns from raw speech and visual signals using a CNN, ResNet-50, and LSTM layers, trained to directly maximize concordance correlation coefficient (ρc). On the RECOLA database, the unimodal speech and visual models outperform several handcrafted-feature baselines, and the multimodal model improves valence prediction over prior multimodal systems.
Key findings
- For speech, the proposed raw-signal model optimized with ρc achieved test ρc of .699 for arousal and .311 for valence, compared with BLSTM eGeMAPS (.316 arousal, .195 valence) and BLSTM ComParE (.382 arousal, .187 valence); the MSE-optimized version achieved .684 arousal and .249 valence.
- For visual modality, the proposed raw-signal model achieved test ρc of .435 for arousal and .620 for valence, outperforming listed models in valence, e.g., baseline (.272 arousal, .507 valence) and Han et al. (.265 arousal, .394 valence).
- In multimodal analysis, the proposed model outperformed OA RVM-SR and Han et al. in valence “with high magnitude,” while OA RVM-SR produced the best arousal result; the provided table does not show the proposed multimodal ρc values.
- Sequence-length ablation found best speech performance at length 150 (arousal .744, valence .369) and best visual performance at length 300 (arousal .193, valence .496), though the multimodal network was trained with length 150; gate activations correlated with RMS energy range (ρ = .83), loudness (ρ = .73), and mean fundamental frequency (ρ = .72).
Methods
- Architecture: speech CNN operates on raw waveform, visual ResNet-50 operates on cropped face pixel intensities, and extracted features are concatenated into a 1920-dimensional vector (1280 speech + 640 visual) feeding a 2-layer LSTM with 256 cells each; unimodal networks are pretrained separately before end-to-end multimodal training.
- Data: RECOLA database, 46 French-speaking participants, 9.5 h of multimodal recordings, 5 minutes each in dyads; split into train (16 subjects), validation (15), and test (15), stratified by gender and age; 6 annotators. Audio is segmented into 6 s sequences at 16 kHz and further into 150 subsequences for 40 ms annotation granularity; visual images are 96×96.
- Training/evaluation: Adam optimizer with fixed learning rate 10⁻⁴, dropout p = 0.5, mini-batch 25 for audio and 2 for video, 60 epochs; metric is ρc; postprocessing includes median filtering, centering, scaling, and time-shifting selected on validation ρc.
Limitations
- The paper states that the OA RVM-SR multimodal comparison was trained using both training and validation sets, whereas this model used only the training set, and that OA RVM-SR used geometric features requiring accurate facial landmark tracking while this model operated on raw pixels; the authors expect their results could improve with similar strategies.
- The paper states hardware limitations forced a small video mini-batch of size 2, and notes memory and training instability concerns with very large recurrent sequences.
- Evident from the text: only two other multimodal models were found for comparison, RECOLA includes ECG and EDA modalities but this study uses only audio and video, and the provided multimodal results table lacks the proposed model’s exact ρc values.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
19 Own result · 5 No full text of the sources · 5 No basis given · 3 Own method · 3 Own interpretation · 1 Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the arousal dimension the OA RVM-SR produces the best results.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The only exception is the Video CNN-L4 model which performs better in the arousal dimension when appearance features are used.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This plot shows that certain cells of the model are very sensitive to different features conveyed in the original speech wave form.”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results show that our model outperforms the other models in the test set when predicting arousal dimension.”
plainn = 27- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As expected visual modality benefits the models in the valence dimension.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In all of the experiments, our model outperforms the designed features in terms of ρ c .”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we study the gate activations of the recurrent layers in the speech modality and find cells that are highly correlated with prosodic features that were always assumed to cause arousal.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our model outperforms the other two models in the valence dimension with high magnitude.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 3
Every source the paper cites for this claim was checked and supports it.
In the paper
“It is well accepted amongst the research community that certain acoustic and prosodic features play an important role in recognising the affective state [45].”
What the sources say
- Supportedcites [45] [45] K. Scherer, 'Vocal communication of emotion: A review of research paradigms,' Speech Communication , vol. 40, no. 1-2, pp. 227-256, April 2003. - source
Section 2.3, “Inference studies” (page not provided), reports that acoustic measures including pitch, energy, and duration explained a large proportion of variation in listeners’ emotion judgments. It also reports that manipulating vocal features affected judgments of a speaker’s emotional state, supporting the claim that such features matter for recognition. (LLM, conf=0.94)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
hedgedn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Some of these features, such as the mean of the fundamental frequency (F0), mean speech intensity, loudness, as well as pitch range [46], should thus be captured by our model.”
What the sources say
- Not checked — no full text of the sourcecites [46] [46] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan et al. , 'The geneva minimalistic acoustic parameter set (gemaps) for voice research and - affective computing,' IEEE Transactions on Affective Computing , vol. 7, no. 2, pp. 190-202, 2016. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
In the paper
“The task of recognising emotions is challenging because human emotions lack of temporal boundaries and different individuals express emotions in different ways [3].”
What the sources say
- Not checked — no full text of the sourcecites [3] [3] C.-N. Anagnostopoulos, T. Iliou, and I. Giannoukos, 'Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,' Artificial Intelligence Review , vol. 43, no. 2, pp. 155-177, 2015. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to perform a fair comparison we apply the proposed method on the test set of the AVEC challenge.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset is split in three partitions -train (16 subjects), validation (15 subjects) and test (15 subjects) -by stratifying (i.e., balancing) the gender and the age of the speakers.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Any of these post-processing steps is kept when an improvement is observed on the ρ c of the validation set, and applied then with the same configuration on the test partition.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
The proposed work is the first in literature to apply an end-to-end model for audio-visual emotion recognition.
“To our knowledge this is the first work in literature that applies such an end-to-end model for audio visual emotion recognition task.”
The paper relies on this without support.
Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
“The speech signals convey information about the affective state either explicitly, i.e., by linguistic means, or implicitly, i.e., by acoustic or prosodic cues.”
The paper relies on this without support.
Visual modality more easily predicts the valence dimension rather than the arousal.
“Visual modality has been shown to more easily predict the valence dimension rather than the arousal.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Dropout is applied in fullyconnected (reported in 43% of studies, e.g., [135]) and convolutional layers (14% of studies, e.g., [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We see both CNN-RNN (e.g., [18], [90]) and CNN-LSTM (e.g., [108], [105], [64]) architectures, with CNN-LSTM being the more frequent choice among the reviewed studies.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Global temporal modeling is found to lead to improved accuracies when compared with simpler methods such as pooling of spatial features (e.g., [152], [90], [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Applications of similar architectures based on the raw waveform have also been attempted [54], [108], showing that end-to-end learning can outperform shallow models.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“End-to-end training of the entire CNN-LSTM system addresses this problem, and can lead to accuracy improvements [108].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We are starting to see studies implementing end-toend training for such models [54], [108]”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Learning from raw audiovisual data with two CNNs, Tzirakis et al. [108] used a two-layer LSTM network for feature fusion, which was found to outperform the state of the art.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Similarly, for SER, Tzirakis et al. [108] found that LSTM cells learn representations similar to well-known prosodic features.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ I. INTRODUCTION
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
§ B. Speech Modality
“| Baseline [29] | eGeMAPS | .648 (.796) | .375 (.455) | | RVM [30] | eGeMAPS | - (.750) | - (.396) | | Audio BN-Multi [33] | Mixed | - (.833) | - (.503) | | Brady et al [32] | MFCC | - (.846) | - (.450) | | Weber et al. [31] | eGeMAPS | - (.793) | - (.456) | | Somandepalli et al. [34] | Mixed | - (.800) | - (.448) |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
(2010) · Cited by 14,134
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Research works printed but not linked: 1
- [37][37] F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data,' Pattern Recognition Letters , vol. 66, pp. 22-30, November 2015. -
Cited by
Sorted by citations
Items 701–752 of 752
Cross-Modal Interactive Multimodal Emotion Recognition Based on State Space ModelPreprint
Sun, L., Wei, P., Yang, X., Chen, S., Ye, L., Li, P., & Shao, X. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6336567
An audio-video based multi-modal fusion approach for emotion recognition
Islam, S. M. J., Mustakim, S. H., Hossain, M., Mashira, M., Shourav, N. I., Ahmed, M. R., Islam, S., Islam, A. K. M. M., & Shatabda, S. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115762
Weakly Supervised Learning for Facial Affective Behavior Analysis: A Review
Praveen, R. G., Cardinal, P., & Granger, É. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3677128 · Cited by 4
Emotion Detection Based on Audio-Visual Using Deep Learning
Nicholson, D., Lie, Z. S., & Andangsari, E. W. (2026) · International Conference on Cognitive Systems and Computer Interaction (ICoSCI) · doi:10.1109/icosci66700.2026.11447989
Synthetic Resurrection – A Multimodal Pipeline for Expressive Voice and Avatar Generation
Thakur, A., Rajpoot, M., & Kaur, H. (2025) · International Conference on Emerging Technologies and Innovation for Sustainability (EmergIN) · doi:10.1109/emergin67762.2025.11450716
BiGRU-Powered Speech Emotion Recognition: A Step Toward Higher Accuracy
Nani Babu, M., S, J. P., Prabhu Shankar, B., Alex David, S., Jain, G., & Rithik, A. (2025) · International Conference on Emerging Technologies and Innovation for Sustainability (EmergIN) · doi:10.1109/emergin67762.2025.11450708
MindWell AI Assistant: A Privacy-First Intelligent Mental Health Support System
Das, S., Goud, P., Ranjini, K., Joshi, P. P., & Modi, M. (2025) · International Conference on Data, Energy and Communication Networks (DECoN) · doi:10.1109/decon67170.2025.11448081
Deep Learning-Based Multimodal System for Behavioral Analysis in Criminal Investigations
Baidya, S., Layek, A. S., Misra, B., Paul, S., Desai, L. R., & Dey, N. (2026) · International Conference on Emerging Trends and Innovations in ICT (ICEI) · doi:10.1109/icei65890.2026.11447701
Analyse d'opinion dans les interactions orales
Barriere, V. (2019) · theses.fr (ABES) · doi:10.70675/d65df106z9fe1z44f0zb9bbzb1caa1590c7a
First seconds matter : Mangaing first impressions for a more engaging virtual agent
Biancardi, B. (2019) · theses.fr (ABES) · doi:10.70675/5629459eze93dz4edazbc42z40b1b2f07116
Smart Music Recommendation System Using Both Facial Expressions and Voice Pitch Analysis
Natarajan, S., Muddammagari, D. V., & Barre, M. L. S. (2026) · IEEE International Conference on Interdisciplinary Approaches in Technology and Management for Social Innovation (IATMSI) · doi:10.1109/iatmsi68868.2026.11466041 · Cited by 1
Multimodal transformers for emotion recognition
Vazquez-Rodriguez, J. (2023) · theses.fr (ABES) · doi:10.70675/a1927be2zdba6z405azbe6fz6856529b46bb · Cited by 1
CEAT: Context-Emotion Adversarial Training Framework for Robust Emotion-Driven Fraud Detection
Li, C., Wu, S., Lu, Y., Ma, Y., Liu, J., Jia, D., Han, M., Li, F., & Hu, P. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792981
Designing Emotion Feedback for Embodied Virtual Agents: A Continuous Emotional Intensity Model
Jiang, Z., Kuai, S., & Deng, C.-L. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790353
Deep Learning Model for Predicting Foreign Language Enjoyment Using Eye-Tracking Data
Riadh, H., Bouzayenne, A., & Soui, M. (2025) · IEEE Afro-Mediterranean Conference on Artificial Intelligence (AMCAI) · doi:10.1109/amcai66110.2025.11474319
Brindha, D., Nimith, B., Rampure, S., & Srividhya, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-19681-1_21
Jayasinghe, P. T., Illesinghe, A. T., Mallawaarachchi, D. E. H., Vithanage, C. S., Silva, D. D., Chathuranga, H. M. S., Fernando, W. K. T. R., & Fernando, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-21013-5_34
Cross-Attention Audio–Visual Fusion Based on Multi-Scale Vision Transformers for Emotion Recognition
Bao, C., Chen, L., Li, M., Wu, M., Witold Pedrycz, C., & Hirota, K. (2026) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2026.p0749
Affective Computing in Education: A Bibliometric and Visualisation Analysis
Zhu, W., & Liu, X. D. (2026) · International Conference on Educational and Information Technology · doi:10.1109/iceit68991.2026.11521445
Bridging the silos in affective AI: a critical perspective from data to society
Inoshita, K. (2026) · AI & Society · doi:10.1007/s00146-026-03324-y · Cited by 1
Systematic Evaluation of Unimodal and Multimodal Fusion for Emotion Recognition
Mukherjee, A., Kumari, A., Ch, N., & Singh, P. (2026) · International Conference on Intelligent Processing, Hardware, Electronics, and Radio Systems (CIPHER) · doi:10.1109/cipher70417.2026.11523756
Live Detection of Mental Disorder Using Artifical Intelligence
R, S. G., Akshith Gowda, R., Bharath Gowda, A., & Chandana, G. (2026) · International Conference on Knowledge Engineering and Communication Systems (ICKECS) · doi:10.1109/ickecs70176.2026.11528155
SAFE-GUARD: Audio Emotion Recognition System for Real-Time Public Place Abuse Detection
Kanan, Chawla, R., & Fortino, G. (2026) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai68641.2026.11536405
ALTRUS - Adaptive Life-Support and Therapeutic Robotic Unit System
S, P. S. K., Semasinghe, S. N. S. B. K. K., S, M. L. A., Silva, H. D., Iddamalgoda, A., & Edirisinghe, B. (2026) · International Congress on Human-Computer Interaction, Optimization and Robotic Applications (ICHORA) · doi:10.1109/ichora69329.2026.11537016
Multimodal Emotion Detection Using Voice and Facial Expressions With Deep Learning
Bose, S., Shetty, A., G, C. K., Naik, P. S., & Naik, P. P. (2026) · International Conference on Artificial Intelligence and Data Engineering (AIDE) · doi:10.1109/aide69088.2026.11544807
Human and AI Perspectives: Comparing Emotion Recognition from Facial Expressions and Hand Gestures
Bougriche, Z., Osuchowski, J., Dąbroś, M., & Tomaszewski, M. (2026) · Cognitive Computation · doi:10.1007/s12559-026-10618-2
Transformer-Guided Audio Visual Emotion Recognition using Adversarial Feature Fusion
Uchhula, V., & Rana, K. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20609-1_27
Multimodal Voice Distress Detection for Real-Time Safety Monitoring in Schools
Chawla, R., Kumar, C., & Malik, A. (2026) · IEEE International Conference on Human-Machine Systems (ICHMS) · doi:10.1109/ichms69701.2026.11602270
Sai, A. M., Suryanarayanan, H., Nadar, S., Vithalani, D., & Karulkar, Y. (2026) · International Conference on Secure Cyber Computing and Communications (ICSCCC) · doi:10.1109/icsccc69031.2026.11600344
Gupta, A., Tejania, A. K., Tyagi, A., & MIshra, N. (2026) · Springer proceedings in complexity · doi:10.1007/978-3-032-21524-6_9
Bekmurat, O., Serbin, V. V., Aizhanova, A., Niyaz, N., Yerezhep, D., Sansyzbay, K., Oralbekova, A., & Sagitzhanova, M. (2026) · Multimodal Technologies and Interaction · doi:10.3390/mti10070076
DFM-NC: Disentangling fine-grained multiplex non-literal cues for multimodal sentiment analysis
Wang, T., Chen, W., Xue, F., Li, J., Liu, H., Ba, X., Xiao, Y., & Sha, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133778
Deep learning implementation of facial emotion recognition using ECG and GSR
Sudha, R., Arul, P., Abirami, N., Manishankar, R., & Padmapriya, S. (2026) · AIP conference proceedings · doi:10.1063/5.0328914
Roy, A. K., Barbhuyan, M. F. K., Nath, S., Aanjankumar, S., Dhanaraj, R. K., & Poonkuntran, S. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-44744-0.00007-2
A survey of representation learning for multi-modal emotion recognition
Wang, E., Sun, S., Xue, K., Ji, Y., Ma, T., Jiang, K., & Liu, J. (2026) · Multimedia Tools and Applications · doi:10.1007/s11042-026-21849-8 · Cited by 1
Sudhandradevi, P., & Bhuvaneswari, V. (2026) · International Conference on Sustainable Engineering and Technology Innovations (ICSETI) · doi:10.1109/icseti67678.2026.11637024
AI Elder Care: Detecting and Predicting Loneliness Using Multimodal Emotion Analysis
Ruchilasya, B., Manish, K., Vivekananda, B., & Vardhan, M. V. (2026) · Smart innovation, systems and technologies · doi:10.1007/978-3-032-25490-0_35
-, B. P., Deshpande, U. U., & Michael, G. K. O. (2026) · Discover Computing · doi:10.1007/s10791-026-10471-9
An Approach for Joint Recognition of Intent and Emotion Based on Multimodal Data
Liu, M., Liu, Z., Song, P., Yu, Z., Zhang, K., & Du, Z. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00029
SPEECH-AWARE EMOTION RECOGNITION FOR INDIVIDUALS WITH AUTISM SPECTRUM DISORDER
Swamy, B. N. (2024) · ShodhKosh Journal of Visual and Performing Arts · doi:10.29121/shodhkosh.v5.i4.2024.9152
Kalam, S., Krishna, G. V., Voddi, S., Kottu, S., Chaitanya, B., & Sharif, S. (2026) · Smart innovation, systems and technologies · doi:10.1007/978-981-95-9097-1_42
Torigaki, Y., & Matsuda, Y. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-27235-5_57
Unkelos‐Shpigel, N., Feldman, L., & Maman, M. H. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-5190-2_9
Vigil NLP: Intelligent Threat Detection and Alerting System
Sri, N. T., Kumar, A. R., Kumar, G. S. U., Pavana, V., Dharmesh, R., & Sandeep, D. S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-30957-0_37
A hierarchical multistage semantic fusion framework for multimodal sentiment analysis
Mu, G., Dai, J., Sun, H.-B., & Li, J. (2026) · Aslib Journal of Information Management · doi:10.1108/ajim-01-2026-0042
Balanced Speech Emotion Recognition for Edge and Cloud: A Practical Accuracy-Efficiency Tradeoff
Ahmed, Y., Elsherif, A., Darwish, F., & Mohammed, A. (2026) · Intelligent Methods, Systems, and Applications (IMSA) · doi:10.1109/imsa70415.2026.11700355
CDESD: The Chinese Disyllabic Emotional Speech Database
Mo, L., Sijin, L. I., Tang, J., Dong, Y., & Zhang, D. (2026) · Brain Science Advances · doi:10.26599/bsa.2026.905026
Tarade, R. V., & Satkar, A. A. (2026) · International Conference on Innovation & Growth in Next-Gen Intelligent Technology & Engineering (IGNITE) · doi:10.1109/ignite69592.2026.11705955
Bansal, M., Vyas, V., Upadhyay, G. M., Akashe, S., & Malche, T. (2026) · Discover Internet of Things · doi:10.1007/s43926-026-00496-3
Multimodal Complaint Detection Using Deep Learning with Sentiment and Emotion Analysis
Saoji, S., Mane, A., Kamunwar, A., Hadawale, B., Jadhav, T., & Rathod, S. (2026) · International Conference on Information, Implementation, and Innovation in Technology (I2ITCON) · doi:10.1109/i2itcon69735.2026.11747888
Animal Emotional State Recognition from Vocal Signals Using Deep Learning Based Acoustic Modeling
Jain, A., & Mishra, R. K. (2026) · IEEE International Conference on Advances in Computing Research on Science Engineering and Technology (ACROSET) · doi:10.1109/acroset70960.2026.11742389
Rizvi, F. R., & Gupta, S. K. (2026) · IEEE International Conference on Advances in Computing Research on Science Engineering and Technology (ACROSET) · doi:10.1109/acroset70960.2026.11742569
Metadata
- Authors
- Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S.
- Affiliations
- Imperial College London (GB)
- Goldsmiths University of London (GB)
- University of Augsburg (DE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 11
- Issue
- 8
- Pages
- 1301-1309
- ISSN
- 1932-4553 (ISSN-L)
- arXiv
- 1704.08619
- OpenAlex
- W2610961739
- Semantic Scholar
- 8cf0c072eeaec310aff806bdb7a7931b81e9ed99
- Cited by
- 752
- References
- 53
- Citations by year
- 2026: 67 · 2025: 90 · 2024: 81 · 2023: 99 · 2022: 108 · 2021: 102 · 2020: 76 · 2019: 82 · 2018: 42 · 2017: 5
- FWCI
- 72.53
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- Goldsmiths (University of London) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Tekes
- Engineering and Physical Sciences Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal emotion recognition, audio-visual emotion recognition, end-to-end learning, convolutional neural networks, ResNet, long short-term memory networks, feature extraction, spontaneous emotions, outlier robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
