The DOI matches its DataCite record; 55 of the 55 references it cites were found in a registry or an index; 1 of the 7 sources its claims cite were checked for support.
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Open accessAbstract
Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a Convolutional Neural Network (CNN) to extract features from the speech, while for the visual modality a deep residual network (ResNet) of 50 layers. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, Long Short-Term Memory (LSTM) networks are utilized. The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an end-to-end multimodal emotion recognition system that learns from raw speech and visual signals using a CNN, ResNet-50, and LSTM layers, trained to directly maximize concordance correlation coefficient (ρc). On the RECOLA database, the unimodal speech and visual models outperform several handcrafted-feature baselines, and the multimodal model improves valence prediction over prior multimodal systems.
Key findings
- For speech, the proposed raw-signal model optimized with ρc achieved test ρc of .699 for arousal and .311 for valence, compared with BLSTM eGeMAPS (.316 arousal, .195 valence) and BLSTM ComParE (.382 arousal, .187 valence); the MSE-optimized version achieved .684 arousal and .249 valence.
- For visual modality, the proposed raw-signal model achieved test ρc of .435 for arousal and .620 for valence, outperforming listed models in valence, e.g., baseline (.272 arousal, .507 valence) and Han et al. (.265 arousal, .394 valence).
- In multimodal analysis, the proposed model outperformed OA RVM-SR and Han et al. in valence “with high magnitude,” while OA RVM-SR produced the best arousal result; the provided table does not show the proposed multimodal ρc values.
- Sequence-length ablation found best speech performance at length 150 (arousal .744, valence .369) and best visual performance at length 300 (arousal .193, valence .496), though the multimodal network was trained with length 150; gate activations correlated with RMS energy range (ρ = .83), loudness (ρ = .73), and mean fundamental frequency (ρ = .72).
Methods
- Architecture: speech CNN operates on raw waveform, visual ResNet-50 operates on cropped face pixel intensities, and extracted features are concatenated into a 1920-dimensional vector (1280 speech + 640 visual) feeding a 2-layer LSTM with 256 cells each; unimodal networks are pretrained separately before end-to-end multimodal training.
- Data: RECOLA database, 46 French-speaking participants, 9.5 h of multimodal recordings, 5 minutes each in dyads; split into train (16 subjects), validation (15), and test (15), stratified by gender and age; 6 annotators. Audio is segmented into 6 s sequences at 16 kHz and further into 150 subsequences for 40 ms annotation granularity; visual images are 96×96.
- Training/evaluation: Adam optimizer with fixed learning rate 10⁻⁴, dropout p = 0.5, mini-batch 25 for audio and 2 for video, 60 epochs; metric is ρc; postprocessing includes median filtering, centering, scaling, and time-shifting selected on validation ρc.
Limitations
- The paper states that the OA RVM-SR multimodal comparison was trained using both training and validation sets, whereas this model used only the training set, and that OA RVM-SR used geometric features requiring accurate facial landmark tracking while this model operated on raw pixels; the authors expect their results could improve with similar strategies.
- The paper states hardware limitations forced a small video mini-batch of size 2, and notes memory and training instability concerns with very large recurrent sequences.
- Evident from the text: only two other multimodal models were found for comparison, RECOLA includes ECG and EDA modalities but this study uses only audio and video, and the provided multimodal results table lacks the proposed model’s exact ρc values.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
19 Own result · 5 No full text of the sources · 5 No basis given · 3 Own method · 3 Own interpretation · 1 Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the arousal dimension the OA RVM-SR produces the best results.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The only exception is the Video CNN-L4 model which performs better in the arousal dimension when appearance features are used.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This plot shows that certain cells of the model are very sensitive to different features conveyed in the original speech wave form.”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results show that our model outperforms the other models in the test set when predicting arousal dimension.”
plainn = 27- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As expected visual modality benefits the models in the valence dimension.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In all of the experiments, our model outperforms the designed features in terms of ρ c .”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From top to bottom: range of RMS energy ( ρ = 0 . 83 ), loudness ( ρ = 0 . 73 ), mean of fundamental frequency ( ρ = 0 . 72 )”
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we study the gate activations of the recurrent layers in the speech modality and find cells that are highly correlated with prosodic features that were always assumed to cause arousal.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our model outperforms the other two models in the valence dimension with high magnitude.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 3
Every source the paper cites for this claim was checked and supports it.
In the paper
“It is well accepted amongst the research community that certain acoustic and prosodic features play an important role in recognising the affective state [45].”
What the sources say
- Supportedcites [45] [45] K. Scherer, 'Vocal communication of emotion: A review of research paradigms,' Speech Communication , vol. 40, no. 1-2, pp. 227-256, April 2003. - source
Section 2.3, “Inference studies” (page not provided), reports that acoustic measures including pitch, energy, and duration explained a large proportion of variation in listeners’ emotion judgments. It also reports that manipulating vocal features affected judgments of a speaker’s emotional state, supporting the claim that such features matter for recognition. (LLM, conf=0.94)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
hedgedn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Some of these features, such as the mean of the fundamental frequency (F0), mean speech intensity, loudness, as well as pitch range [46], should thus be captured by our model.”
What the sources say
- Not checked — no full text of the sourcecites [46] [46] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan et al. , 'The geneva minimalistic acoustic parameter set (gemaps) for voice research and - affective computing,' IEEE Transactions on Affective Computing , vol. 7, no. 2, pp. 190-202, 2016. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
In the paper
“The task of recognising emotions is challenging because human emotions lack of temporal boundaries and different individuals express emotions in different ways [3].”
What the sources say
- Not checked — no full text of the sourcecites [3] [3] C.-N. Anagnostopoulos, T. Iliou, and I. Giannoukos, 'Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,' Artificial Intelligence Review , vol. 43, no. 2, pp. 155-177, 2015. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to perform a fair comparison we apply the proposed method on the test set of the AVEC challenge.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset is split in three partitions -train (16 subjects), validation (15 subjects) and test (15 subjects) -by stratifying (i.e., balancing) the gender and the age of the speakers.”
plainn = 46- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Any of these post-processing steps is kept when an improvement is observed on the ρ c of the validation set, and applied then with the same configuration on the test partition.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
The proposed work is the first in literature to apply an end-to-end model for audio-visual emotion recognition.
“To our knowledge this is the first work in literature that applies such an end-to-end model for audio visual emotion recognition task.”
The paper relies on this without support.
Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).
“The speech signals convey information about the affective state either explicitly, i.e., by linguistic means, or implicitly, i.e., by acoustic or prosodic cues.”
The paper relies on this without support.
Visual modality more easily predicts the valence dimension rather than the arousal.
“Visual modality has been shown to more easily predict the valence dimension rather than the arousal.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Dropout is applied in fullyconnected (reported in 43% of studies, e.g., [135]) and convolutional layers (14% of studies, e.g., [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We see both CNN-RNN (e.g., [18], [90]) and CNN-LSTM (e.g., [108], [105], [64]) architectures, with CNN-LSTM being the more frequent choice among the reviewed studies.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Global temporal modeling is found to lead to improved accuracies when compared with simpler methods such as pooling of spatial features (e.g., [152], [90], [108]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Applications of similar architectures based on the raw waveform have also been attempted [54], [108], showing that end-to-end learning can outperform shallow models.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“End-to-end training of the entire CNN-LSTM system addresses this problem, and can lead to accuracy improvements [108].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“We are starting to see studies implementing end-toend training for such models [54], [108]”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Learning from raw audiovisual data with two CNNs, Tzirakis et al. [108] used a two-layer LSTM network for feature fusion, which was found to outperform the state of the art.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
In the citing paper
“Similarly, for SER, Tzirakis et al. [108] found that LSTM cells learn representations similar to well-known prosodic features.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ I. INTRODUCTION
“As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
§ B. Speech Modality
“| Baseline [29] | eGeMAPS | .648 (.796) | .375 (.455) | | RVM [30] | eGeMAPS | - (.750) | - (.396) | | Audio BN-Multi [33] | Mixed | - (.833) | - (.503) | | Brady et al [32] | MFCC | - (.846) | - (.450) | | Weber et al. [31] | eGeMAPS | - (.793) | - (.456) | | Somandepalli et al. [34] | Mixed | - (.800) | - (.448) |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
(2010) · Cited by 14,134
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785
Research works printed but not linked: 1
- [37][37] F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data,' Pattern Recognition Letters , vol. 66, pp. 22-30, November 2015. -
Cited by
Sorted by citations
Items 301–400 of 752
Crude oil price prediction using temporal fusion transformer model
He, K., Zheng, L., Yang, Q., Wu, D. C. W., Yu, Y., & Zou, Y. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.08.070 · Cited by 12
Advancements in Affective and Behavior Analysis: The 8th ABAW Workshop and Competition
Kollias, D., Tzirakis, P., Cowen, A., Zafeiriou, S. P., Kotsia, I., Granger, É., Pedersoli, M., Bacon, S., Baird, A., Gagne, C., et al. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00554 · Cited by 12
Self-citationJoint Discrete and Continuous Emotion Prediction Using Ensemble and End-to-End Approaches
AlBadawy, E. A., & Kim, Y. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3242972 · Cited by 12
Emotion Dynamics Modeling via BERT
Yang, H., & Shen, J. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533860 · Cited by 12
Multimodal Emotion Recognition Using Transfer Learning on Audio and Text Data
Deng, J. J., Leung, C. H. C., & Li, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86970-0_39 · Cited by 12
Towards a Better Gold Standard
Wang, C., Lopes, P., Pun, T., & Chanel, G. (2018) · doi:10.1145/3266302.3266307 · Cited by 12
From Smart to Personal Environment: Integrating Emotion Recognition into Smart Houses
Fedotov, D., Matsuda, Y., & Minker, W. (2019) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomw.2019.8730876 · Cited by 12
Dyadic Speech-based Affect Recognition using DAMI-P2C Parent-child Multimodal Interaction Dataset
Chen, H., Zhang, Y., Weninger, F., Picard, R. W., Breazeal, C., & Park, H. W. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3418842 · Cited by 12
Robust Latent Representations Via Cross-Modal Translation and Alignment
Rajan, V., Brutti, A., & Cavallaro, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413456 · Cited by 12
Non-Uniform Attention Network for Multi-modal Sentiment Analysis
Wang, B., Dong, G., Zhao, Y., Li, R., Cao, Q., & Chao, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98358-1_48 · Cited by 12
Causal Inference for Modality Debiasing in Multimodal Emotion Recognition
Kim, J., Hong, J., & Choi, Y. (2024) · Applied Sciences · doi:10.3390/app142311397 · Cited by 12
Hantke, S., Schmitt, M., Tzirakis, P., & Schuller, B. W. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3243681 · Cited by 11
Self-citationSoft Voting Strategy for Multi-Modal Emotion Recognition Using Deep-learning- Facial Images and EEG
Chinta, U., Kalita, J. K., & Atyabi, A. (2023) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc57344.2023.10099070 · Cited by 12
Modelling Sample Informativeness for Deep Affective Computing
Rizos, G., & Schuller, B. W. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683729 · Cited by 11
Self-citationVlasenko, B., Prasad, R., & Magimai.-Doss, M. (2021) · doi:10.1145/3475957.3484448 · Cited by 11
Matsuda, Y., Fedotov, D., Takahashi, Y., Arakawa, Y., Yasumoto, K., & Minker, W. (2018) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3267305.3267687 · Cited by 11
Predicting Biological Signals from Speech: Introducing a Novel Multimodal Dataset and Results
Baird, A., Amiriparian, S., Berschneider, M., Schmitt, M., & Schuller, B. W. (2019) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2019.8901758 · Cited by 11
Self-citationAnalysis of Various Facial Expressions of Horses as a Welfare Indicator Using Deep Learning
Kim, S., & Cho, G. (2023) · Veterinary Sciences · doi:10.3390/vetsci10040283 · Cited by 10
A Review of Intelligent Systems for Driving Risk Assessment
Mase, J. M., Chapman, P., & Figueredo, G. P. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3318113 · Cited by 10
Tripartite interaction representation learning for multi-modal sentiment analysis
Wang, B., Dong, G., Zhao, Y., Li, R., Yin, W., & Lu, L. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126279 · Cited by 10
Exploring Emotion Expression Recognition in Older Adults Interacting With a Virtual Coach
Palmero, C., deVelasco, M., Hmani, M. A., Mtibaa, A., Letaifa, L. B., Buch-Cardona, P., Justo, R., Amorese, T., González–Fraile, E., Fernández-Ruanova, B., et al. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3558141 · Cited by 15
Bags in Bag: Generating Context-Aware Bags for Tracking Emotions from Speech
Han, J., Zhang, Z., Schmitt, M., Ren, Z. X., Ringeval, F., & Schuller, B. W. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-996 · Cited by 10
Self-citationAsghar, M. A., Fawad, F., Khan, M. J., Amin, Y., & Akram, A. (2019) · International Conference on Open Source Systems and Technologies (ICOSST) · doi:10.1109/icosst48232.2019.9043994 · Cited by 10
Automatic Interferogram Selection for SBAS-InSAR Based on Deep Convolutional Neural Networks
He, Y., Zhang, G., Kaufmann, H., & Xu, G. (2021) · Remote Sensing · doi:10.3390/rs13214468 · Cited by 10
Chen, Z., Xu, J., Peng, T., Yang, C., Fan, X., & Gui, W. (2021) · CAA Symposium on Fault Detection, Supervision and Safety for Technical Processes (SAFEPROCESS) · doi:10.1109/safeprocess52771.2021.9693630 · Cited by 10
Tzirakis, P., Shiarella, A., Ewers, R. M., & Schuller, B. W. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2655 · Cited by 10
Self-citationDyadic Speech-based Affect Recognition using DAMI-P2C Parent-child Multimodal Interaction Dataset
Chen, H., Zhang, Y., Weninger, F., Picard, R. W., Breazeal, C., & Park, H. (2020) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 10
Multimodal Emotion Recognition Based on Speech and Physiological Signals Using Deep Neural Networks
Bakhshi, A., & Chalup, S. K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68780-9_25 · Cited by 11
Investigations on audiovisual emotion recognition in noisy conditions
Neumann, M., & Vu, N. T. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383588 · Cited by 10
A Hybrid Technique using CNN LSTM for Speech Emotion Recognition
Qazi, H., & Kaushik, B. N. (2020) · International Journal of Engineering and Advanced Technology · doi:10.35940/ijeat.e1027.069520 · Cited by 10
Tourist Arrival Forecasting Using Multiscale Mode Learning Model
He, K., Wu, D. C. W., & Zou, Y. (2022) · Mathematics · doi:10.3390/math10162999 · Cited by 10
Music emotion classification using a hybrid CNN-LSTM model
Yakovyna, V. S., & Korniienko, V. (2023) · Applied Aspects of Information Technology · doi:10.15276/aait.06.2023.28 · Cited by 9
GHPSO-ATLSTM: a novel attention-based genetic LSTM to predict water quality indicators
Singh, R. B., Patra, K. C., & Samantra, A. (2024) · Stochastic Environmental Research and Risk Assessment · doi:10.1007/s00477-024-02694-3 · Cited by 9
Real Time Bimodal Emotion Recognition using Hybridized Deep Learning Techniques
Bhat, A. A., Kavitha, S., Satapathy, S. M., & Kavipriya, J. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.04.168 · Cited by 10
Meaningful Multimodal Emotion Recognition Based on Capsule Graph Transformer Architecture
Filali, H., Boulealam, C., Fazazy, K. E., Mahraz, M. A., Tairi, H., & Riffi, J. (2025) · Information · doi:10.3390/info16010040 · Cited by 14
VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition
Kumar, P., Malik, S., Raman, B., & Li, X. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3573180 · Cited by 14
Facial identity protection using deep learning technologies: an application in affective computing
Mase, J. M., Leesakul, N., Figueredo, G. P., & Torres, M. T. (2022) · AI and Ethics · doi:10.1007/s43681-022-00215-y · Cited by 10
Xu, C., Du, Y., Wang, J., Zheng, W., Li, T., & Yuan, Z.-S. (2023) · Computational Intelligence · doi:10.1111/coin.12607 · Cited by 9
Govoreanu, V.-C., & Neghină, M. (2020) · IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) · doi:10.1109/iccp51029.2020.9266265 · Cited by 9
ROSbag-based Multimodal Affective Dataset for Emotional and Cognitive States
Jo, W., Kannan, S. S., Cha, G.-E., Lee, A., & Min, B. (2020) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc42975.2020.9283320 · Cited by 9
An interactive VR platform with emotion recognition for self-attachment intervention
Polydorou, N., & Edalat, A. (2021) · EAI Endorsed Transactions on Pervasive Health and Technology · doi:10.4108/eai.14-9-2021.170951 · Cited by 9
A Bayesian Filtering Framework for Continuous Affect Recognition From Facial Images
Pei, E., Zhao, Y. H., Oveneke, M. C., Jiang, D., & Hichem Sahli, B. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3164248 · Cited by 11
Prediction of Continuous Emotional Measures through Physiological and Visual Data
Joudeh, I. O., Creţu, A.-M., Bouchard, S., & Guimond, S. (2023) · Sensors · doi:10.3390/s23125613 · Cited by 9
Maximum Likelihood Estimation for Multimodal Learning with Missing ModalityPreprint
Ma, F., Xu, X., Huang, S., & Zhang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10513 · Cited by 18
Multimodal Emotion Captioning Using Large Language Model with Prompt Engineering
Xu, Y., Zhou, Y., Cai, Y., Xie, J., Ye, R., & Wu, Z. (2024) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3689092.3689403 · Cited by 9
Sun, L., Lei, Y., Zhang, Z., Tang, Y., Wang, J., Ye, L., & Li, P. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.107811 · Cited by 8
Multimodal emotion recognition method in complex dynamic scenes
Liu, L., Luo, Q., Zhang, W., Zhang, M., & Zhai, B. (2025) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2025.02.004 · Cited by 12
Riaz, W., Ji, J., Zaman, K., & Zengkang, G. (2025) · Electronics · doi:10.3390/electronics14071320 · Cited by 10
Ferrero, M., Chelotti, J. O., Martinez-Rau, L. S., Vignolo, L. D., Pires, M., Galli, J. R., Giovanini, L. L., & Rufiner, H. L. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111372 · Cited by 8
Multimodal Knowledge Distillation for Emotion Recognition
Zhang, Z., & Lu, G. (2025) · Brain Sciences · doi:10.3390/brainsci15070707 · Cited by 8
Use of Affective Visual Information for Summarization of Human-Centric Videos
Köprü, B., & Erzin, E. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3222882 · Cited by 8
Crude Oil Price Prediction using Embedding Convolutional Neural Network Model
He, K., Yang, Q., & Zou, Y. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.11.265 · Cited by 7
Kshirsagar, S., Pendyala, A., & Falk, T. H. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1039261 · Cited by 14
Emo-BERT: A Multi-Modal Teacher Speech Emotion Recognition Method
Zhao, G. L., Zhang, Y., & Chu, J. (2023) · International Workshop on Artificial Intelligence and Education (WAIE) · doi:10.1109/waie60568.2023.00022 · Cited by 9
MN-UIV: Multimodal Neural Network Enabling User Identity Verification Based on Millimeter Wave Radar
Zhao, H., Ma, Y., & Wang, X. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3395287 · Cited by 8
Feature Importance and Model Performance in Deep Learning for Speech Emotion Recognition
Ramkrishnan, P., Dhaman, R. K., & Poulose, A. (2024) · International Conference on Advances in Computing and Communications (ICACC) · doi:10.1109/icacc63692.2024.10845528 · Cited by 8
Evolving Learning for Analysing Mood-Related Infant Vocalisation
Zhang, Z., Han, J., Qian, K., & Schuller, B. W. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1914 · Cited by 8
Self-citationMining Multimodal Repositories for Speech Affecting Diseases
Correia, J., Raj, B., Trancoso, I. M., & Teixeira, F. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1806 · Cited by 8
Priyasad, D., Fernando, T., Denman, S., Sridharan, S., & Fookes, C. (2019) · International Conference on Auditory-Visual Speech Processing (AVSP) · doi:10.21437/avsp.2019-5 · Cited by 10
Toward Multimodal Modeling of Emotional Expressiveness
Lin, V., Girard, J. M., Sayette, M. A., & Morency, L. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3418887 · Cited by 8
Your Smartphone Knows you Better than you May Think: Emotional Assessment ‘on the Go’ Via TapSense
Ghosh, S., Löchner, J., Mitra, B., & De, P. (2022) · Health informatics · doi:10.1007/978-3-030-94212-0_10 · Cited by 8
Forecasting Crude Oil Risk Using a Multivariate Multiscale Convolutional Neural Network Model
Zou, Y., & He, K. (2022) · Mathematics · doi:10.3390/math10142413 · Cited by 10
Analysis of infoware and software for human affective states recognition
Dvoynikova, A., Markitantov, M., Рюмина, Е. В., Uzdiaev, M., Величко, А. Н., Ryumin, D., Lyakso, E., & Karpov, A. A. (2022) · Informatics and Automation · doi:10.15622/ia.21.6.2 · Cited by 10
Pei, E., Hu, Z., He, L., Ning, H., & Berenguer, A. D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121290 · Cited by 8
A Deep Learning Approach for Human Facial Expression Recognition using Residual Network – 101
Kumari, R., & Wasim, J. (2023) · Journal of Current Science and Technology · doi:10.59796/jcst.v13n3.2023.2152 · Cited by 8
Emotion Recognition Based on Gramian Encoding Visualization
Qiu, J., Qiu, X., & Hu, K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-05587-5_1 · Cited by 10
Efremova, N., Patkin, M., & Sokolov, D. (2019) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2019.8756562 · Cited by 8
Leveraging the Deep Learning Paradigm for Continuous Affect Estimation from Facial Expressions
Oveneke, M. C., Zhao, Y. H., Pei, E., Berenguer, A. D., Jiang, D., & Hichem Sahli, B. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2944603 · Cited by 8
Shah, A. K., Kadyan, V., & Kane, L. (2025) · SN Computer Science · doi:10.1007/s42979-025-03861-1 · Cited by 7
EEG-based fusion approaches in multimodal emotion recognition: An in-depth review
Yılmaz, B. H., Yılmaz, Ç. M., & Köse, C. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132235 · Cited by 7
Ramírez-Quintana, J. A., Muñoz-Pacheco, J. J., Ramírez-Alonso, G. M. D. J., Medrano-Hermosillo, J. A., & Corral-Saenz, A. D. (2025) · Sensors · doi:10.3390/s25237264 · Cited by 7
Feedback through emotion extraction using logistic regression and CNN
Panda, M. R., Kar, S. S., Nanda, A. K., Priyadarshini, R., Panda, S., & Bisoy, S. K. (2021) · The Visual Computer · doi:10.1007/s00371-021-02260-w · Cited by 7
Chouhayebi, H., Mahraz, M. A., Riffi, J., & Tairi, H. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16779-8 · Cited by 9
Bimodal Emotion Recognition Based on Convolutional Neural Network
Chen, M., Jiang, L., Ma, C., & Sun, H. (2019) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3318299.3318347 · Cited by 7
Multimodal Emotion Recognition using Deep Convolution and Recurrent Network
Sajid, M., Afzal, M., & Shoaib, M. (2021) · doi:10.1109/icai52203.2021.9445262 · Cited by 7
Comparing Biosignal and Acoustic feature Representation for Continuous Emotion Recognition
Yadav, S., Purohit, T., Mostaani, Z., Vlasenko, B., & Magimai.-Doss, M. (2022) · doi:10.1145/3551876.3554812 · Cited by 7
TAPLoss: A Temporal Acoustic Parameter Loss for Speech Enhancement
Zeng, Y., Konan, J., Han, S., Bick, D., Yang, M., Kumar, A., Watanabe, S., & Raj, B. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094773 · Cited by 7
Bangla Speech Emotion Recognition Using 3D CNN Bi-LSTM Model
Islam, M. R., Akhand, M. A. H., & Kamal, M. A. S. (2023) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-34619-4_42 · Cited by 7
Li, J., Zhong, J., & Wang, M. (2020) · Sensors and Materials · doi:10.18494/sam.2020.2552 · Cited by 7
Dual-Waveform Emotion Recognition Model for Conversations
Zhang, J., Liu, Z., Liu, P., & Wu, B. (2021) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428327 · Cited by 7
Machine Learning Based Emotion Recognition using Speech Signal
Kumar, A., & Iqbal, D. J. L. M. (2019) · International Journal of Engineering and Advanced Technology · doi:10.35940/ijeat.a1068.1291s52019 · Cited by 10
A multi-modal deep learning system for Arabic emotion recognition
Shaqra, F. A., Duwairi, R., & Al‐Ayyoub, M. (2022) · International Journal of Speech Technology · doi:10.1007/s10772-022-09981-w · Cited by 7
Data Encoding Visualization Based Cognitive Emotion Recognition with AC-GAN Applied for Denoising
Qiu, J., & Zhao, W. (2018) · IEEE International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC) · doi:10.1109/icci-cc.2018.8482097 · Cited by 7
Shi, Q., Xu, W., & Miao, Z. (2024) · Journal of Electronic Imaging · doi:10.1117/1.jei.33.4.043042 · Cited by 6
A review on an AI-driven face robot for human-robot expression interaction
Sheng, Q., Tang, W., Qin, H., Kong, Y., Dai, H., Zhong, Y., Wang, Y., Zou, J. T., & Yang, H. (2025) · Science China Technological Sciences · doi:10.1007/s11431-025-3027-x · Cited by 7
Towards human digital twin: Reviewing human modelling and simulation
Zhu, E., & Yang, S. L. (2025) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2025.100975 · Cited by 7
Fraile, M., Fawcett, C. A., Lindblad, J., Sladoje, N., & Castellano, G. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556629 · Cited by 6
SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Video
Jegorova, M., Petridis, S., & Pantic, M. (2022) · IEEE International Conference on Automatic Face & Gesture Recognition · doi:10.1109/fg57933.2023.10042638 · Cited by 5
Panda, M., Sharma, A., & Jaiswal, G. (2024) · International Conference on Cloud Computing, Data Science & Engineering (Confluence) · doi:10.1109/confluence60223.2024.10463296 · Cited by 6
Wang, J. (2024) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1449527 · Cited by 6
Learning Long-Term Behavior through Continuous Emotion Estimation
Kansizoglou, I., Misirlis, E., & Gasteratos, A. C. (2021) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3453892.3461626 · Cited by 6
An Audiovisual Child Emotion Recognition System for Child-Robot Interaction Applications
Filntisis, P. P., Efthymiou, N., Potamianos, G., & Maragos, P. A. (2021) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco54536.2021.9616106 · Cited by 6
Subjective Evaluation of Basic Emotions from Audio–Visual Data
Kadiri, S. R., & Alku, P. (2022) · Italian National Conference on Sensors · doi:10.3390/s22134931 · Cited by 5
Distilling Privileged Multimodal Information for Expression Recognition using Optimal Transport
Aslam, M. H., Zeeshan, M., Belharbi, S., Pedersoli, M., Koerich, A. L., Bacon, S., & Granger, É. (2024) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg59268.2024.10582047 · Cited by 6
Praveen, R. G., & Alam, J. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10687371 · Cited by 6
Adaptive Humanoid Robots for Pain Management in Children
Shenoy, S., Hou, Y., Wang, X., Nikseresht, F., & Doryab, A. (2021) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3434074.3447224 · Cited by 6
Emotion Analysis of Japanese Sentences Using an Emotion-word Dictionary
Adachi, Y., Kondo, T., Kobayashi, T., ETANI, N., & ISHII, K. (2021) · Journal of the Visualization Society of Japan · doi:10.3154/jvs.41.161_21 · Cited by 8
Fusing Deep Quick Response Code Representations Improves Malware Text Classification
Ravikiran, M., & Madgula, K. (2019) · doi:10.1145/3326459.3329166 · Cited by 6
Speech Emotion Recognition Based on Multi-Task Learning
Zhao, H., Han, Z., & Wang, R. (2019) · IEEE International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, and IEEE International Conference on Intelligent Data and Security (BigDataSecurity/HPSC/IDS) · doi:10.1109/bigdatasecurity-hpsc-ids.2019.00043 · Cited by 6
Multimodal emotion recognition using deep learning techniques
Nguyen, T. D. (2020) · Queensland University of Technology · doi:10.5204/thesis.eprints.180753 · Cited by 6
Metadata
- Authors
- Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S.
- Affiliations
- Imperial College London (GB)
- Goldsmiths University of London (GB)
- University of Augsburg (DE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Volume
- 11
- Issue
- 8
- Pages
- 1301-1309
- ISSN
- 1932-4553 (ISSN-L)
- arXiv
- 1704.08619
- OpenAlex
- W2610961739
- Semantic Scholar
- 8cf0c072eeaec310aff806bdb7a7931b81e9ed99
- Cited by
- 752
- References
- 53
- Citations by year
- 2026: 67 · 2025: 90 · 2024: 81 · 2023: 99 · 2022: 108 · 2021: 102 · 2020: 76 · 2019: 82 · 2018: 42 · 2017: 5
- FWCI
- 72.53
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
- Goldsmiths (University of London) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Tekes
- Engineering and Physical Sciences Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- multimodal emotion recognition, audio-visual emotion recognition, end-to-end learning, convolutional neural networks, ResNet, long short-term memory networks, feature extraction, spontaneous emotions, outlier robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
