3-D Convolutional Recurrent Neural Networks With Attention Model for Speech Emotion Recognition
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Speech emotion recognition (SER) is a difficult task due to the complexity of emotions. The SER performances are heavily dependent on the effectiveness of emotional features extracted from the speech. However, most emotional features are sensitive to emotionally irrelevant factors, such as the speaker, speaking styles, and environment. In this letter, we assume that calculating the deltas and delta-deltas for personalized features not only preserves the effective emotional information but also reduces the influence of emotionally irrelevant factors, leading to reduce misclassification. In addition, SER often suffers from the silent frames and emotionally irrelevant frames. Meanwhile, attention mechanism has exhibited outstanding performances in learning relevant feature representations for specific tasks. Inspired by this, we propose a three-dimensional attention-based convolutional recurrent neural networks to learn discriminative features for SER, where the Mel-spectrogram with deltas and delta-deltas are used as input. Experiments on IEMOCAP and Emo-DB corpus demonstrate the effectiveness of the proposed method and achieve the state-of-the-art performance in terms of unweighted average recall.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Another issue faced by the time-frequency representation of speech emotion is that the distribution of emotion information in an utterance is sparse in both frequency domain and time domain [19], [23].”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [19]
In the citing paper
“However, these studies mainly embed attention into highlevel time-frequency features, leading to introduce the redundant information in the feature extraction and roughly locate the emotion-related regions in the time-frequency domain.”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [19]
In the citing paper
“In addition, they mainly focus on the selection on the time frames while ignoring the selection on the frequency bands.”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [19]
In the citing paper
“ignoring to capture the specific frequency bands associated with emotions”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [19]
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“toolkit [24], and the parameters of the model were optimized by minimizing the cross-entropy objective function, with a minibatch of 40 samples, using the Adam optimizer [25] with Nestorov momentum.”
MethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“proposed in [9] for machine translation and later applied for”
MethodTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Hierarchical Attention Networks for Document Classification
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., & Hovy, E. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1174 · Cited by 4,712 · cited 1×
“document classification [10] and speech recognition [11].”
Self-citationIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 12×
“D ACRNN obtains an absolute improvement of 2.34% and 3.44% for IEMOCAP and Emo-DB, respectively.”
MethodInfluentialA database of German emotional speech
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-446 · Cited by 2,365 · cited 1×
“form speaker-independent SER experiments on the Interactive Emotional Dyadic Motion Capture database (IEMOCAP) [21] and the Berlin Emotional Database (Emo-DB) [22].”
MethodAttention-Based Models for Speech Recognition
Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1506.07503 · Cited by 2,021
Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition
Graves, A., Fernández, S., & Schmidhuber, J. (2005) · Lecture Notes in Computer Science · doi:10.1007/11550907_126 · Cited by 925 · cited 1×
“bidirectional RNN with long short-term memory cells [19] for temporal summarization, and each direction contains 128 cells; then, we can obtain a sequence of 256-dimensional high-level”
MethodAdieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network
Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472669 · Cited by 904 · cited 1×
“In recent years, CNN has exhibited excellent performances in the field of SER [1], [4], [6].”
BackgroundSpeech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-57 · Cited by 794 · cited 4×
“1) DNN-extreme learning machine (DNN-ELM) ( see [3]).”
MethodInfluentialAutomatic speech emotion recognition using recurrent neural networks with local attention
Mirsamadi, S., Barsoum, E., & Zhang, C. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952552 · Cited by 770
Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568 · cited 2×
“In recent years, CNN has exhibited excellent performances in the field of SER [1], [4], [6].”
BackgroundMethodZhang, S., Zhang, S., Huang, T., & Gao, W. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2766843 · Cited by 441 · cited 2×
“Speech as the most convenient and natural medium for human communication, not only carries the implicit semantic information, but also contains rich affective information [1].”
BackgroundOpenEAR — Introducing the munich open-source emotion and affect recognition toolkit
Eyben, F., Wollmer, M., & Schuller, B. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349350 · Cited by 418 · cited 1×
“The log-Mels are extracted by the openEAR toolkit [23] with the window size of 25 ms and a 10-ms shift, and both the training”
MethodHigh-level feature representation using recurrent neural network for speech emotion recognition
Lee, J., & Tashev, I. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-336 · Cited by 388 · cited 1×
“[5] applied a recurrent neural network (RNN) to learn long-range temporal relationships for SER.”
BackgroundAcoustic emotion recognition: A benchmark comparison of performances
Schuller, B., Vlasenko, B., Eyben, F., Rigoll, G., & Wendemuth, A. (2009) · 2009 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2009.5372886 · Cited by 267 · cited 1×
“Inspired by the positive result of Mel-frequency cepstral coefficents (MFCC) with deltas and delta-deltas in the field of SER [7], we assume that calculated deltas and delta-deltas for personalized features are capable of reflecting the changing process of emotion and retaining effective emotional information while re…”
BackgroundNeumann, M., & Vu, N. T. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-917 · Cited by 243 · cited 1×
“In the field of SER, several works [12]–[15] have studied the effectiveness of attention mechanism in discovering emotion relevant regions for utterance-level emotion prediction and obtained promising performances in several datasets.”
BackgroundConvolutional RNN: An enhanced model for extracting features from sequential data
Keren, G., & Schuller, B. W. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727636 · Cited by 200 · cited 2×
“In this task, we set f as 40, the same as speech recognition [17], and set the value of c as 3, representing”
MethodHuang, C.-W., & Narayanan, S. S. (2017) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019296 · Cited by 145 · cited 2×
“First, we perform 3-D CNN on the whole log-Mels, different from [12], [17] which convolution”
BackgroundMethodLearning emotion-based acoustic features with deep belief networks
Schmidt, E. M., & Kim, Y. E. (2011) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/aspaa.2011.6082328 · Cited by 89 · cited 1×
“[2] employed a deep belief network to extract high-level emotional feature representations from the”
MethodAttention Assisted Discovery of Sub-Utterance Structure in Speech Emotion Recognition
Huang, C.-W., & Narayanan, S. S. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-448 · Cited by 88
Deep convolutional neural networks for acoustic modeling in low resource languages
Chan, W., & Lane, I. R. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178332 · Cited by 57 · cited 1×
“[16] found that 2-D convolution outperforms 1-D convolution with limited data, and the time-domain convolution is as important as the frequency-”
BackgroundDeep scattering spectra with deep neural networks for LVCSR tasks
Sainath, T. N., Peddinti, V., Kingsbury, B. E. D., Fousek, P., Ramabhadran, B., & Nahamoo, D. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-225 · Cited by 21 · cited 1×
“According to [18], the model parameters can be reduced effectively with no loss in accuracy by adding a linear layer before passing 3-D CNN features into the LSTM layer.”
Background
Research works printed but not linked: 2
- wei, 2016, Attention assisted discovery of sub-utterance structure in speech emotion recognition, Proc INTERSPEECH, 1387
- frano, 2016, Normalization and batch correction methods for high-throughput glycomics, Proc Joint Meet Soc Glycobiol, 1160
Cited by
Akçay, M. B., & Oğuz, K. (2020) · Speech Communication · doi:10.1016/j.specom.2019.12.001 · Cited by 721
Speech Emotion Recognition Using Deep Learning Techniques: A Review
Khalil, R. A., Jones, E., Babar, M. I., Jan, T., Zafar, M. H., & Alhussain, T. (2019) · IEEE Access · doi:10.1109/access.2019.2936124 · Cited by 702
Speech emotion recognition with deep convolutional neural networks
Issa, D., Fatih Demirci, M., & Yazici, A. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2020.101894 · Cited by 619
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 615
Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM
Mustaqeem, Sajjad, M., & Kwon, S. (2020) · IEEE Access · doi:10.1109/access.2020.2990405 · Cited by 402
Metadata
- Authors
- Chen, M., He, X., Yang, J., & Zhang, H.
- Affiliations
- Central South University (CN)
- Venue
- IEEE Signal Processing Letters
- Published
- 2018-10
- Volume
- 25
- Issue
- 10
- Pages
- 1440-1444
- ISSN
- 1070-9908 (ISSN-L)
- 1558-2361
- OpenAlex
- W2885005742
- Cited by
- 503
- References
- 23
- Citations by year
- 2026: 21 · 2025: 50 · 2024: 70 · 2023: 85 · 2022: 91 · 2021: 94 · 2020: 60 · 2019: 27 · 2018: 5
- FWCI
- 60.91
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Letters (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech emotion recognition, attention mechanism, Mel spectrogram, IEMOCAP, EMO-DB
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.