CoChat
Sign Up
PreprintCited by 752Emotion and Mood RecognitionSpeech and Audio ProcessingSentiment Analysis and Opinion Mining

End-to-End Multimodal Emotion Recognition using Deep Neural Networks

Open access
Tzirakis, Panagiotis, Trigeorgis, George, Nicolaou, Mihalis A., Schuller, Björn, Zafeiriou, Stefanos·2017·arXiv
DOI 10.48550/arxiv.1704.08619arXiv 1704.08619
Open this paper in CoChat — free accountOpen PDF

Abstract

Automatic affect recognition is a challenging task due to the various modalities emotions can be expressed with. Applications can be found in many domains including multimedia retrieval and human computer interaction. In recent years, deep neural networks have been used with great success in determining emotional states. Inspired by this success, we propose an emotion recognition system using auditory and visual modalities. To capture the emotional content for various styles of speaking, robust features need to be extracted. To this purpose, we utilize a Convolutional Neural Network (CNN) to extract features from the speech, while for the visual modality a deep residual network (ResNet) of 50 layers. In addition to the importance of feature extraction, a machine learning algorithm needs also to be insensitive to outliers while being able to model the context. To tackle this problem, Long Short-Term Memory (LSTM) networks are utilized. The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory and visual handcrafted features for the prediction of spontaneous and natural emotions on the RECOLA database of the AVEC 2016 research challenge on emotion recognition.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes an end-to-end multimodal emotion recognition system that learns from raw speech and visual signals using a CNN, ResNet-50, and LSTM layers, trained to directly maximize concordance correlation coefficient (ρc). On the RECOLA database, the unimodal speech and visual models outperform several handcrafted-feature baselines, and the multimodal model improves valence prediction over prior multimodal systems.

Key findings

  • For speech, the proposed raw-signal model optimized with ρc achieved test ρc of .699 for arousal and .311 for valence, compared with BLSTM eGeMAPS (.316 arousal, .195 valence) and BLSTM ComParE (.382 arousal, .187 valence); the MSE-optimized version achieved .684 arousal and .249 valence.
  • For visual modality, the proposed raw-signal model achieved test ρc of .435 for arousal and .620 for valence, outperforming listed models in valence, e.g., baseline (.272 arousal, .507 valence) and Han et al. (.265 arousal, .394 valence).
  • In multimodal analysis, the proposed model outperformed OA RVM-SR and Han et al. in valence “with high magnitude,” while OA RVM-SR produced the best arousal result; the provided table does not show the proposed multimodal ρc values.
  • Sequence-length ablation found best speech performance at length 150 (arousal .744, valence .369) and best visual performance at length 300 (arousal .193, valence .496), though the multimodal network was trained with length 150; gate activations correlated with RMS energy range (ρ = .83), loudness (ρ = .73), and mean fundamental frequency (ρ = .72).

Methods

  • Architecture: speech CNN operates on raw waveform, visual ResNet-50 operates on cropped face pixel intensities, and extracted features are concatenated into a 1920-dimensional vector (1280 speech + 640 visual) feeding a 2-layer LSTM with 256 cells each; unimodal networks are pretrained separately before end-to-end multimodal training.
  • Data: RECOLA database, 46 French-speaking participants, 9.5 h of multimodal recordings, 5 minutes each in dyads; split into train (16 subjects), validation (15), and test (15), stratified by gender and age; 6 annotators. Audio is segmented into 6 s sequences at 16 kHz and further into 150 subsequences for 40 ms annotation granularity; visual images are 96×96.
  • Training/evaluation: Adam optimizer with fixed learning rate 10⁻⁴, dropout p = 0.5, mini-batch 25 for audio and 2 for video, 60 epochs; metric is ρc; postprocessing includes median filtering, centering, scaling, and time-shifting selected on validation ρc.

Limitations

  • The paper states that the OA RVM-SR multimodal comparison was trained using both training and validation sets, whereas this model used only the training set, and that OA RVM-SR used geometric features requiring accurate facial landmark tracking while this model operated on raw pixels; the authors expect their results could improve with similar strategies.
  • The paper states hardware limitations forced a small video mini-batch of size 2, and notes memory and training instability concerns with very large recurrent sequences.
  • Evident from the text: only two other multimodal models were found for comparison, RECOLA includes ECG and EDA modalities but this study uses only audio and video, and the provided multimodal results table lacks the proposed model’s exact ρc values.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.1704.08619

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

1 of 7 cited sources checked · 1 supported · 0 flagged

  • Findingown data
    § D. Multimodal Analysis

    For the arousal dimension, the OA RVM-SR produces the best results.

    “For the arousal dimension the OA RVM-SR produces the best results.”
    Quote in the paperNot a substantial claim
  • Findingnone given
    § C. Visual Modality

    The Video CNN-L4 model performs better in the arousal dimension when appearance features are used.

    “The only exception is the Video CNN-L4 model which performs better in the arousal dimension when appearance features are used.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § B. Speech Modality

    Certain cells of the model are very sensitive to different features conveyed in the original speech wave form.

    “This plot shows that certain cells of the model are very sensitive to different features conveyed in the original speech wave form.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § B. Speech Modality

    Our model outperforms the other models in the test set when predicting the arousal dimension.

    “Results show that our model outperforms the other models in the test set when predicting arousal dimension.”
    Quote in the paperSubstantial claim
  • Findingnone given
    § C. Visual Modality

    Visual modality benefits the models in the valence dimension.

    “As expected visual modality benefits the models in the valence dimension.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 7
  • May combine two claims. 3

  • Classified as probably not a substantial claim.

    Premisecitation
    § B. Speech Modality

    Certain acoustic and prosodic features play an important role in recognising the affective state.

    “It is well accepted amongst the research community that certain acoustic and prosodic features play an important role in recognising the affective state [45].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [45] [45] K. Scherer, 'Vocal communication of emotion: A review of research paradigms,' Speech Communication , vol. 40, no. 1-2, pp. 227-256, April 2003. - source

  • May combine two claims.

    Findingown data
    § I. INTRODUCTION

    The proposed unimodal models produce the best results for both the speech and visual modalities.

    “As shown by our experiments our unimodal models produce the best results for both the speech and visual modalities.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § B. Speech Modality

    The eGEMAPS feature set provides close performance on valence.

    “One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationcitation
    § B. Speech Modality

    Our model should capture certain acoustic and prosodic features such as mean fundamental frequency (F0), mean speech intensity, loudness, and pitch range.

    “Some of these features, such as the mean of the fundamental frequency (F0), mean speech intensity, loudness, as well as pitch range [46], should thus be captured by our model.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [46] [46] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan et al. , 'The geneva minimalistic acoustic parameter set (gemaps) for voice research and - affective computing,' IEEE Transactions on Affective Computing , vol. 7, no. 2, pp. 190-202, 2016. - source
    • Rests on an assumption: Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).

  • May combine two claims.

    Premisecitation
    § I. INTRODUCTION

    Human emotions lack temporal boundaries and different individuals express emotions in different ways.

    “The task of recognising emotions is challenging because human emotions lack of temporal boundaries and different individuals express emotions in different ways [3].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [3] [3] C.-N. Anagnostopoulos, T. Iliou, and I. Giannoukos, 'Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,' Artificial Intelligence Review , vol. 43, no. 2, pp. 155-177, 2015. - source

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § I. INTRODUCTION

    The proposed method was applied on the test set of the AVEC challenge to perform a fair comparison.

    “In order to perform a fair comparison we apply the proposed method on the test set of the AVEC challenge.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § B. Speech Modality

    Valence is much more difficult to predict from speech compared to arousal.

    “One may note, however, that the eGEMAPS feature set provides close performance on valence, which is much more difficult to predict from speech compared to arousal.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § IV. DATASET

    The dataset was split into train, validation and test partitions by stratifying gender and age of the speakers.

    “The dataset is split in three partitions -train (16 subjects), validation (15 subjects) and test (15 subjects) -by stratifying (i.e., balancing) the gender and the age of the speakers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § V. EXPERIMENTS & RESULTS

    Post-processing steps were kept only when an improvement was observed on the rho_c of the validation set and applied with the same configuration on the test partition.

    “Any of these post-processing steps is kept when an improvement is observed on the ρ c of the validation set, and applied then with the same configuration on the test partition.”
    Quote in the paperNot a substantial claim
9 of 9

  • The paper relies on this without support.

    The proposed work is the first in literature to apply an end-to-end model for audio-visual emotion recognition.

    “To our knowledge this is the first work in literature that applies such an end-to-end model for audio visual emotion recognition task.”

  • The paper relies on this without support.

    Speech signals convey information about the affective state either explicitly (by linguistic means) or implicitly (by acoustic or prosodic cues).

    “The speech signals convey information about the affective state either explicitly, i.e., by linguistic means, or implicitly, i.e., by acoustic or prosodic cues.”

  • The paper relies on this without support.

    Visual modality more easily predicts the valence dimension rather than the arousal.

    “Visual modality has been shown to more easily predict the valence dimension rather than the arousal.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Dropout is applied in fully-connected layers in 43% of studies and in convolutional layers in 14% of studies.

    “Dropout is applied in fullyconnected (reported in 43% of studies, e.g., [135]) and convolutional layers (14% of studies, e.g., [108]).”

    From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]

  • CNN-LSTM is the more frequent choice among the reviewed studies compared to CNN-RNN.

    “We see both CNN-RNN (e.g., [18], [90]) and CNN-LSTM (e.g., [108], [105], [64]) architectures, with CNN-LSTM being the more frequent choice among the reviewed studies.”

    From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]

  • Global temporal modeling is found to lead to improved accuracies when compared with simpler methods such as pooling of spatial features.

    “Global temporal modeling is found to lead to improved accuracies when compared with simpler methods such as pooling of spatial features (e.g., [152], [90], [108]).”

    From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]

  • Applications of similar architectures based on the raw waveform have also been attempted, showing that end-to-end learning can outperform shallow models.

    “Applications of similar architectures based on the raw waveform have also been attempted [54], [108], showing that end-to-end learning can outperform shallow models.”

    From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]

  • End-to-end training of the entire CNN-LSTM system addresses the disconnected training problem and can lead to accuracy improvements.

    “End-to-end training of the entire CNN-LSTM system addresses this problem, and can lead to accuracy improvements [108].”

    From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [108]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Claim of best speech results conflicts with valence scores

    Minor

    The paper says its unimodal models produce the best results for both modalities, but Table IV reports a lower speech-valence test score for the proposed model (.369) than for the baseline (.375).

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. [37][37] F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, 'Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data,' Pattern Recognition Letters , vol. 66, pp. 22-30, November 2015. -

Cited by

Sorted by citations

Related

Authors
Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S.
Affiliations
  • Imperial College London (GB)
  • Goldsmiths University of London (GB)
  • University of Augsburg (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Volume
11
Issue
8
Pages
1301-1309
ISSN
1932-4553 (ISSN-L)
DOI
10.48550/arxiv.1704.08619
arXiv
1704.08619
OpenAlex
W2610961739
Semantic Scholar
8cf0c072eeaec310aff806bdb7a7931b81e9ed99
Cited by
752
References
53
Citations by year
2026: 67 · 2025: 90 · 2024: 81 · 2023: 99 · 2022: 108 · 2021: 102 · 2020: 76 · 2019: 82 · 2018: 42 · 2017: 5
FWCI
72.53
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Journal of Selected Topics in Signal Processing (publishedVersion)
  • Goldsmiths (University of London) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Spiral (Imperial College London) (submittedVersion)
  • OPUS (Augsburg University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Google
  • Tekes
  • Engineering and Physical Sciences Research Council
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
multimodal emotion recognition, audio-visual emotion recognition, end-to-end learning, convolutional neural networks, ResNet, long short-term memory networks, feature extraction, spontaneous emotions, outlier robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review