Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network
Abstract
The automatic recognition of spontaneous emotions from speech is a challenging task. On the one hand, acoustic features need to be robust enough to capture the emotional content for various styles of speaking, and while on the other, machine learning algorithms need to be insensitive to outliers while being able to model the context. Whereas the latter has been tackled by the use of Long Short-Term Memory (LSTM) networks, the former is still under very active investigations, even though more than a decade of research has provided a large set of acoustic descriptors. In this paper, we propose a solution to the problem of ‘context-aware’ emotional relevant feature extraction, by combining Convolutional Neural Networks (CNNs) with LSTM networks, in order to automatically learn the best representation of the speech signal directly from the raw time representation. In this novel work on the so-called end-to-end speech emotion recognition, we show that the use of the proposed topology significantly outperforms the traditional approaches based on signal processing techniques for the prediction of spontaneous and natural emotions on the RECOLA database.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deep learning is used for learning spatial feature representations in human affect recognition.
“Learning spatial feature representations. For images [61], short-term 6 image sequences [62] and audio segments [54], DNNs and especially CNNs are used to learn spatial feature representations.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [54]
Learned features from raw spectral representations can be shown to have similarities to handcrafted LLDs.
“Such learned features can be shown to have similarities to handcrafted LLDs [54].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [54]
Trigeorgis et al. [54] were the first to use CNNs for raw waveform in SER; they used two convolutional layers for spatial modeling, almost doubling ground truth correlation over LLDs for arousal, and slightly improving for valence.
“Trigeorgis et al. [54] were the first to do so: They used two convolutional layers for spatial modeling, almost doubling ground truth correlation over LLDs for arousal, and slightly improving for valence.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [54]
Applications of similar architectures based on the raw waveform have also been attempted, showing that end-to-end learning can outperform shallow models.
“Applications of similar architectures based on the raw waveform have also been attempted [54], [108], showing that end-to-end learning can outperform shallow models.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [54]
Studies are starting to implement end-to-end training for models that learn spatial and temporal features.
“We are starting to see studies implementing end-toend training for such models [54], [108]”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 31 · Cited by 904
Research works printed but not linked: 2
- [5][5] F. Ringeval et al., 'AV+EC 2015 - The First Affect Recognition Challenge Bridging Across Audio, Video, and Physiological Data,' in Proc. AVEC , Fabien Ringeval, Björn Schuller, Michel Valstar, Roddy Cowie, and Maja Pantic, Eds., Brisbane, Australia, October 2015, pp. 3-8, ACM. -
- [15][15] Q. Mao, M. Dong, Z. Huang, and Y. Zhan, 'Learning salient features for speech emotion recognition using convolutional neural networks,' IEEE Transactions on Multimedia , vol. 16, no. 8, pp. 2203-2213, Dec 2014. -
- Authors
- Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S.
- Affiliations
- Imperial College London (GB)
- University of Passau (DE)
- Technical University of Munich (DE)
- Goldsmiths University of London (GB)
- ORCID
- Venue
- 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2016-03
- Pages
- 5200-5204
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2399733683
- Cited by
- 904
- References
- 31
- Citations by year
- 2026: 51 · 2025: 44 · 2024: 51 · 2023: 86 · 2022: 93 · 2021: 117 · 2020: 128 · 2019: 130 · 2018: 114 · 2017: 70 · 2016: 20
- FWCI
- 128.85
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Goldsmiths (University of London) (acceptedVersion)
- Spiral (Imperial College London) (submittedVersion)
- OPUS (Augsburg University) (submittedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech emotion recognition, end-to-end learning, convolutional recurrent neural network, CNN-LSTM, audio feature extraction, spontaneous speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
