3-D Convolutional Recurrent Neural Networks With Attention Model for Speech Emotion Recognition
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Speech emotion recognition (SER) is a difficult task due to the complexity of emotions. The SER performances are heavily dependent on the effectiveness of emotional features extracted from the speech. However, most emotional features are sensitive to emotionally irrelevant factors, such as the speaker, speaking styles, and environment. In this letter, we assume that calculating the deltas and delta-deltas for personalized features not only preserves the effective emotional information but also reduces the influence of emotionally irrelevant factors, leading to reduce misclassification. In addition, SER often suffers from the silent frames and emotionally irrelevant frames. Meanwhile, attention mechanism has exhibited outstanding performances in learning relevant feature representations for specific tasks. Inspired by this, we propose a three-dimensional attention-based convolutional recurrent neural networks to learn discriminative features for SER, where the Mel-spectrogram with deltas and delta-deltas are used as input. Experiments on IEMOCAP and Emo-DB corpus demonstrate the effectiveness of the proposed method and achieve the state-of-the-art performance in terms of unweighted average recall.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Another issue faced by the time-frequency representation of speech emotion is that the distribution of emotion information in an utterance is sparse in both frequency domain and time domain [19], [23].”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [19]
In the citing paper
“However, these studies mainly embed attention into highlevel time-frequency features, leading to introduce the redundant information in the feature extraction and roughly locate the emotion-related regions in the time-frequency domain.”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [19]
In the citing paper
“In addition, they mainly focus on the selection on the time frames while ignoring the selection on the frequency bands.”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [19]
In the citing paper
“ignoring to capture the specific frequency bands associated with emotions”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [19]
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“toolkit [24], and the parameters of the model were optimized by minimizing the cross-entropy objective function, with a minibatch of 40 samples, using the Adam optimizer [25] with Nestorov momentum.”
MethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“proposed in [9] for machine translation and later applied for”
MethodTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Hierarchical Attention Networks for Document Classification
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., & Hovy, E. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1174 · Cited by 4,712 · cited 1×
“document classification [10] and speech recognition [11].”
Self-citationIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 12×
“D ACRNN obtains an absolute improvement of 2.34% and 3.44% for IEMOCAP and Emo-DB, respectively.”
MethodInfluentialA database of German emotional speech
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-446 · Cited by 2,365 · cited 1×
“form speaker-independent SER experiments on the Interactive Emotional Dyadic Motion Capture database (IEMOCAP) [21] and the Berlin Emotional Database (Emo-DB) [22].”
MethodAttention-Based Models for Speech Recognition
Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1506.07503 · Cited by 2,021
Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition
Graves, A., Fernández, S., & Schmidhuber, J. (2005) · Lecture Notes in Computer Science · doi:10.1007/11550907_126 · Cited by 925 · cited 1×
“bidirectional RNN with long short-term memory cells [19] for temporal summarization, and each direction contains 128 cells; then, we can obtain a sequence of 256-dimensional high-level”
MethodAdieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network
Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472669 · Cited by 904 · cited 1×
“In recent years, CNN has exhibited excellent performances in the field of SER [1], [4], [6].”
BackgroundSpeech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-57 · Cited by 794 · cited 4×
“1) DNN-extreme learning machine (DNN-ELM) ( see [3]).”
MethodInfluential
Research works printed but not linked: 2
- wei, 2016, Attention assisted discovery of sub-utterance structure in speech emotion recognition, Proc INTERSPEECH, 1387
- frano, 2016, Normalization and batch correction methods for high-throughput glycomics, Proc Joint Meet Soc Glycobiol, 1160
Cited by
Items 101–200 of 549
Amjad, A., Khan, L., & Chang, H.-T. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.766 · Cited by 50
“In Chen et al. (2018) , a three-dimensional convolutional recurrent neural network including an attention mechanism (ACRNN) was introduced.”
MethodResultInfluentialBAT: Block and token self-attention for speech emotion recognition
Lei, J., Zhu, X., & Wang, Y. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.09.022 · Cited by 48
Speech emotion recognition based on emotion perception
Liu, G., Cai, S., & Wang, C. (2023) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-023-00289-4 · Cited by 47
Lightweight Deep Learning Framework for Speech Emotion Recognition
Akinpelu, S. A., Viriri, S., & Adegun, A. A. (2023) · IEEE Access · doi:10.1109/access.2023.3297269 · Cited by 47
“outperforms what was obtained in [57] when comparing the model size(323.”
ResultYokotani, K., & Takano, M. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107099 · Cited by 46
Multi-Level Dual-Attention Based CNN for Macular Optical Coherence Tomography Classification
Mishra, S. S., Mandal, B., & Puhan, N. B. (2019) · IEEE Signal Processing Letters · doi:10.1109/lsp.2019.2949388 · Cited by 46
“Attention mechanism has been explored for image captioning [17], voice activity detection [18], speech emotion recognition [19] and question answering [20].”
BackgroundMurugaiyan, S., & Reddy, U. S. (2023) · Cognitive Computation · doi:10.1007/s12559-023-10127-6 · Cited by 45
“(2018) [19] Log-Mels (static deltas and deltadeltas) 3D - ACRNN IEMOCAP 64.”
MethodSpeech Emotion Recognition Using Convolutional Neural Networks with Attention Mechanism
Mountzouris, K., Perikos, I., & Hatzilygeroudis, I. (2023) · Electronics · doi:10.3390/electronics12204376 · Cited by 45
Dysarthria severity assessment using squeeze-and-excitation networks
Joshy, A. A., & Rajan, R. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.104606 · Cited by 45
Speech emotion recognition via learning analogies
Ntalampiras, S. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.01.018 · Cited by 44
An efficient speech emotion recognition based on a dual-stream CNN-transformer fusion network
Tellai, M., Gao, L., & Mao, Q. (2023) · International Journal of Speech Technology · doi:10.1007/s10772-023-10035-y · Cited by 43
An intelligent agriculture management system for rainfall prediction and fruit health monitoring
Kaplun, D., Deka, S., Bora, A., Choudhury, N., Basistha, J., Purkayastha, B., Mazumder, I. Z., Gulvanskii, V. V., Sarma, K. K., & Misra, D. D. (2024) · Scientific Reports · doi:10.1038/s41598-023-49186-y · Cited by 43
“Such a system designed for agricultural applications must successfully combine sensors in IoT packs and AI aids so as to increase efficiency 18–24 .”
BackgroundPrabhakararao, E., & Dandapat, S. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.3036314 · Cited by 43
“Recently, the attention models have shown promising results in recognition tasks by emphasizing the salient discriminative features in the classification process [30]–[36].”
BackgroundMulti-View Speech Emotion Recognition Via Collective Relation Construction
Hou, M.-X., Zhang, Z., Cao, Q. Z., Zhang, D. D., & Lu, G. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3133196 · Cited by 43
Speech emotion recognition using the novel PEmoNet (Parallel Emotion Network)
Bhangale, K. B., & Kothandaraman, M. (2023) · Applied Acoustics · doi:10.1016/j.apacoust.2023.109613 · Cited by 43
Fusion-ConvBERT: Parallel Convolution and BERT Fusion for Speech Emotion Recognition
Lee, S., Han, D. K., & Ko, H. (2020) · Sensors · doi:10.3390/s20226688 · Cited by 42
“Previous models of serial fusion that employed RNN structures following the local feature extraction process by CNNs may not be able to capture global scale contextual features [9,15].”
MethodDetecting Expressions with Multimodal Transformers
Parthasarathy, S., & Sundaram, S. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383573 · Cited by 42
“[34] proposed a 3-D convolutional recurrent neural network with an attention mechanism to focus on detected speech frames.”
BackgroundLiu, Y., Sun, H., Guan, W., Xia, Y., Li, Y., Unoki, M., & Zhao, Z. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3245401 · Cited by 42
“…bidirectional LSTM network in order to extract sequence information from MFCCs and spatiotemporal features from log-Mels for SER. Chen et al. [15] proposed 3D convolutional recurrent neural network with attention mechanism in order to automatically learn the spatiotemporal representations of…”
BackgroundMethodSpeech emotion classification using attention based network and regularized feature selection
Akinpelu, S. A., & Viriri, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-38868-2 · Cited by 41
“Three-dimensional attention-based CRNN was used by Chen et al. 27 to choose discriminative features for speech emotion classification.”
MethodHybrid deep models for parallel feature extraction and enhanced emotion state classification
Pichandi, S., Gomathy, B., & Venkatesh, C. (2024) · Scientific Reports · doi:10.1038/s41598-024-75850-y · Cited by 41
Fusing Visual Attention CNN and Bag of Visual Words for Cross-Corpus Speech Emotion Recognition
Seo, M., & Kim, M. H. (2020) · Sensors · doi:10.3390/s20195559 · Cited by 41
“…utterances as high-level features, such as gammatone, short-time Fourier transform (STFT), and mel-spectrogram, which are obtained as 2D visual data by converting the time-based signal into the frequency domain utilizing the Fourier transform and deep-learning models such as CNN and LSTM [12–15].”
MethodSelf-attention transfer networks for speech emotion recognition
Zhao, Z., Wang, K., Bao, Z., Zhang, Z., Cummins, N., Sun, S., Wang, H., Tao, J., & Schuller, B. W. (2021) · Virtual Reality & Intelligent Hardware · doi:10.1016/j.vrih.2020.12.002 · Cited by 41
A Novel Heterogeneous Parallel Convolution Bi-LSTM for Speech Emotion Recognition
Zhang, H., Huang, H., & Han, H. (2021) · Applied Sciences · doi:10.3390/app11219897 · Cited by 41
Self-citationMultiple Acoustic Features Speech Emotion Recognition Using Cross-Attention Transformer
He, Y., Minematsu, N., & Saito, D. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095777 · Cited by 40
“In [7], attention layers were used to focus on the emotionally relevant parts.”
MethodPitch-synchronous single frequency filtering spectrogram for speech emotion recognition
Gupta, S., Fahad, M. S., & Deepak, A. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09068-1 · Cited by 39
B-Scan Attentive CNN for the Classification of Retinal Optical Coherence Tomography Volumes
Das, V., Prabhakararao, E., Dandapat, S., & Bora, P. K. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.3000933 · Cited by 39
“The self attention weights ( α i ) , i = { 1 , 2 , ..., N } for the encoded CNN vectors are then obtained as [25], [26]: where w att ∈ R 1 × d l and b att ∈ R 1 are the learnable parameters of the network.”
BackgroundThirumuru, R., Gurugubelli, K., & Vuppala, A. K. (2021) · Digital Signal Processing · doi:10.1016/j.dsp.2021.103293 · Cited by 39
A statistical feature extraction for deep speech emotion recognition in a bilingual scenario
Sekkate, S., Khalil, M., & Adib, A. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-14051-z · Cited by 39
“Chen et al. [9] integrated an attention mechanism into a baseline deep recurrent neural network (RNN) to recognize emotions from samples of IEMOCAP and EMODB corpora with Mel-spectrogram augmented with deltas and delta-deltas.”
BackgroundImproved Cross-Corpus Speech Emotion Recognition Using Deep Local Domain Adaptation
Huijuan, Z., Ye, N., & Ruchuan, W. (2023) · Chinese Journal of Electronics · doi:10.23919/cje.2021.00.196 · Cited by 39
A Combined CNN Architecture for Speech Emotion Recognition
Begazo, R., Aguilera, A., Dongo, I., & Cardinale, Y. (2024) · Sensors · doi:10.3390/s24175797 · Cited by 38
“Given the satisfactory results obtained and the well-established robustness of CNNs in handling SER tasks with personalized inputs (numerical values) [66], we focused on optimizing these techniques.”
MethodCyTex: Transforming speech to textured images for speech emotion recognition
Bakhshi, A., Harimi, A., & Chalup, S. K. (2022) · Speech Communication · doi:10.1016/j.specom.2022.02.007 · Cited by 38
Consumer Personalized Gesture Recognition in UAV-Based Industry 5.0 Applications
Paikrao, P. D., Routray, S., Mukherjee, A., Khan, A. R., & Vohnout, R. (2023) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2023.3308209 · Cited by 38
DSTCNet: Deep Spectro-Temporal-Channel Attention Network for Speech Emotion Recognition
Guo, L., Ding, S., Wang, L., & Dang, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3304516 · Cited by 37
Wang, M., Ma, H., Wang, Y., & Sun, X. (2024) · Applied Acoustics · doi:10.1016/j.apacoust.2024.109886 · Cited by 37
Interactive Multimodal Attention Network for Emotion Recognition in Conversation
Ren, M., Huang, X., Shi, X., & Nie, W. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3078698 · Cited by 37
The Impact of Attention Mechanisms on Speech Emotion Recognition
Chen, S., Zhang, M., Yang, X., Zhao, Z., Zou, T., & Sun, X. (2021) · Sensors · doi:10.3390/s21227530 · Cited by 37
“Mingyi Chen [25] 3-D ACRNN MFCC EMO-DB 82.”
MethodInfluentialSpeech Emotion Recognition Using 1D CNN with No Attention
Li, Y., Baidoo, C., Cai, T., & Kusi, G. A. (2019) · International Computer Science and Engineering Conference · doi:10.1109/icsec47112.2019.8974716 · Cited by 37
“[8] proposed an RNN model extracting log-Mels feature and combining with attention to build a layer for SER.”
MethodInfluentialTang, D., Kuppens, P., Geurts, L., & van Waterschoot, T. (2021) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-021-00208-5 · Cited by 36
“This model can learn time-frequency relations from a time stack of log mel-spectrograms [12].”
MethodDeng, L., Xing, B., & Liu, X. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3396656 · Cited by 35
“Prior research has employed different forms of data for emotion understanding such as speech [3], facial expression [4], [5], [6] and electrocardiogram (ECG) signals [7], [8].”
MethodSpeech emotion recognition using data augmentation method by cycle-generative adversarial networks
Shilandari, A., Marvi, H., Khosravi, H., & Wang, W. (2022) · Signal Image and Video Processing · doi:10.1007/s11760-022-02156-9 · Cited by 35
“[37], who used 3D-ACRNNs to extract features.”
ResultTang, H., Zhang, X., Cheng, N., Xiao, J., & Wang, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446467 · Cited by 34
“We follow the SER [10] to extract a fixed size embedding from reference speech’s mel-spectrogram and its delta, delta-delta co-efficients.”
MethodZhang, H., Huang, H., & Han, H. (2020) · IEEE Access · doi:10.1109/access.2020.3047395 · Cited by 34
Self-citationLearning Mutual Correlation in Multimodal Transformer for Speech Emotion Recognition
Wang, Y., Shen, G., Xu, Y., Li, J., & Zhao, Z. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-2004 · Cited by 34
“Deep learning algorithm has improved the performance of unimodal emotion recognition [2, 3, 4, 5, 6].”
BackgroundAttention gated tensor neural network architectures for speech emotion recognition
Pandey, S., Shekhawat, H. S., & Prasanna, S. R. M. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103173 · Cited by 34
Adversarial Data Augmentation Network for Speech Emotion Recognition
Lu, Y., & Mak, M. (2019) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc47483.2019.9023347 · Cited by 34
“Table III and Table IV compare the performance of our proposed network with some end-to-end systems [28], [30] that do not use OpenSMILE features as input.”
BackgroundYan, Y., Qiu, D., & Yan, R. (2024) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-023-00380-w · Cited by 33
“Many of the existing works of the text embedding focused simply on the word representation as the input system [8, 14].”
Tao, H., Geng, L., Shan, S., Mai, J., & Fu, H. (2022) · Entropy · doi:10.3390/e24081025 · Cited by 33
“In recent years, Mel spectrum has been widely used in emotion recognition because it contains a large amount of emotional information [8,9,17,18].”
BackgroundMethodInfluentialA novel feature-fusion-based end-to-end approach for remaining useful life prediction
Zhu, Q., Xiong, Q., Yang, Z., & Yang, Y. (2022) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-022-02015-x · Cited by 33
“In many fields (Li et al., 2018b; Chen et al., 2018), the attention mechanism has proven to be effective.”
BackgroundTC-Net: A Modest & Lightweight Emotion Recognition System Using Temporal Convolution Network
Ishaq, M., Khan, M., & Kwon, S. (2023) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.037373 · Cited by 33
WISE: Word-Level Interaction-Based Multimodal Fusion for Speech Emotion Recognition
Shen, G., Lai, R., Chen, R., Zhang, Y., Zhang, K., Han, Q., & Song, H. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-3131 · Cited by 32
“There has been a good amount of research [11, 12, 13, 14, 15, 16, 17, 18, 19, 20] using unimodal features to classify speech emotions.”
BackgroundOn the Usage of Pre-Trained Speech Recognition Deep Layers to Detect Emotions
Oliveira, J., & Praça, I. (2021) · IEEE Access · doi:10.1109/access.2021.3051083 · Cited by 32
“In [21] an Attention Convolutional-Recurrent Neural Network (ACRNN) is proposed to recognize the Ekman’s six universal emotions (happy, surprise, anger, sad, disgust, fear) on a speech sample.”
BackgroundSpeech emotion recognition using deep learning
Roy, T., Marwala, T., & Chakraverty, S. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-822133-4.00009-8 · Cited by 32
A DCRNN-based ensemble classifier for speech emotion recognition in Odia language
Swain, M., Maji, B., Kabisatpathy, P., & Routray, A. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-022-00713-w · Cited by 32
Trends in speech emotion recognition: a comprehensive survey
Kaur, K., & Singh, P. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14656-y · Cited by 32
“The complications behind the emotions, make it a difficult task as well [31] and being a multifarious area, becomes more complex and difficult, as it includes variety of disciplines such as computer science, psychology, cognitive science and physiology [127, 168, 196].”
BackgroundLi, J., Zhang, X., Li, F., & Huang, L. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119649 · Cited by 32
Ibrahim, H., Loo, C. K., & Alnajjar, F. (2021) · IEEE Access · doi:10.1109/access.2021.3107858 · Cited by 31
“In the Emo-DB dataset and speaker-independent approach, authors in [62] and [65] have achieved a classification accuracy of 84.99% and 82.82% respectively using 3-”
MethodInfluentialExpressive Voice Conversion: A Joint Framework for Speaker Identity and Emotional Style Transfer
Du, Z., Şişman, B., Zhou, K., & Li, H. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9687906 · Cited by 31
Congestive Heart Failure Detection From ECG Signals Using Deep Residual Neural Network
Prabhakararao, E., & Dandapat, S. (2022) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2022.3221843 · Cited by 31
“Recently, the attention networks have shown promising results in the recognition tasks across various fields by emphasizing the salient discriminative features in the classification process [38], [39], [40].”
BackgroundGraph Isomorphism Network for Speech Emotion Recognition
Liu, J., & Wang, H. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1154 · Cited by 30
“• 3D-ACRNN[6] .”
MethodEmoFake: An Initial Dataset for Emotion Fake Audio Detection
Zhao, Y. C., Yi, J., Tao, J., Wang, C., & Dong, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8367-0_25 · Cited by 30
Compact Convolutional Recurrent Neural Networks via Binarization for Speech Emotion Recognition
Zhao, H., Xiao, Y., Han, J., & Zhang, Z. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683389 · Cited by 30
“Besides, for our experiments, only the improvised utterances were considered, resulting in 2 837 sample in total in order to keep in line with the setting in [10].”
MethodResultInfluentialTian, Y., Sui, J., Li, Y., Wu, N., & Shao, D. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3170635 · Cited by 29
“The advantage of being able to focus on useful information while suppressing useless information makes it widely used in various tasks [40]–[42].”
BackgroundLi, J., Zhang, X., Huang, L., Li, F., Duan, S., & Sun, Y. (2022) · Applied Sciences · doi:10.3390/app12199518 · Cited by 29
“In references [39,40], log-Mels, deltas, and delta-deltas were used as the inputs of the structure in the hybrid system.”
MethodSpeech emotion recognition based on improved masking EMD and convolutional recurrent neural network
Sun, C., Li, H., & Ma, L. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1075624 · Cited by 29
“Utterances in the two datasets are split into equal-length segments of 3 s, and zero padding is used for utterances with a duration of less than 3 s (Chen et al., 2018).”
MethodA Two-stage Multi-modal Affect Analysis Framework for Children with Autism Spectrum DisorderPreprint
Li, J., Bhat, A. N., & Barmaki, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09199 · Cited by 29
“Since distinct speech emotions exhibit different patterns in the energy spectrum, to capture emotion features from speech, we selected log-Mel spectrograms which have been effective in speech emotion recognition tasks in the past (Zhao, Mao, and Chen 2019; Zhang et al. 2018; Chen et al. 2018).”
Speech emotion recognition based on spiking neural network and convolutional neural network
Du, C., Liu, F., Kang, B., & Hou, T. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110314 · Cited by 28
ATDA: Attentional temporal dynamic activation for speech emotion recognition
Liu, L., Liu, W., Zhou, J., Deng, H., & Feng, L. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108472 · Cited by 28
Saleem, N., Elmannai, H., Bourouis, S., & Trigui, A. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111735 · Cited by 28
Reproducible and generalizable speech emotion recognition via an Intelligent Fusion Network
Zhang, H., Zhao, P., Tang, G., Li, Z., & Zhu, Y. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.107996 · Cited by 27
Self-citationA Contextual Attention Network for Multimodal Emotion Recognition in Conversation
Wang, T., Hou, Y., Zhou, D., & Zhang, Q. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533718 · Cited by 27
Li, X., Zhang, Z., Gan, C., & Xiang, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3157485 · Cited by 27
“Following the work [17], the utterance is split into equal-length segments of 3 s, and the zero padding is applied for the utterances with a duration less than 3 s.”
MethodInfluentialMukhamediya, A., Fazli, S., & Zollanvari, A. (2023) · IEEE Access · doi:10.1109/access.2023.3287093 · Cited by 27
“Empowered by algorithmic and computational advances in recent years, deep neural networks (DNNs) have achieved breakthrough results in various areas [5], [6], [7], [8], [9], [10], and found eminent applications in SER [11], [12], [13].”
BackgroundMethodInfluentialA supervised non-negative matrix factorization model for speech emotion recognition
Hou, M.-X., Li, J., & Lu, G. (2020) · Speech Communication · doi:10.1016/j.specom.2020.08.002 · Cited by 27
Chen, F., & Luo, Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.08065 · Cited by 27
“[23] propose a 3-”
MethodCCTG-NET: Contextualized Convolutional Transformer-GRU Network for speech emotion recognition
Tellai, M., & Mao, Q. (2023) · International Journal of Speech Technology · doi:10.1007/s10772-023-10080-7 · Cited by 26
Speaker Attentive Speech Emotion Recognition
Veillon, C. L. M., Obin, N., & Röebel, A. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-573 · Cited by 26
Pham, N. T., Nguyen, S. D., Nguyen, V. S. T., Pham, B. N. H., & Dang, D. N. M. (2023) · Journal of Information and Telecommunication · doi:10.1080/24751839.2023.2187278 · Cited by 26
“…(Koduru et al., 2020; Kuchibhotla et al., 2016), linear predictive coding, zero-crossing rate (Lingampeta & Yalamanchili, 2020), Mel spectrograms (Chen et al., 2018; Meng et al., 2019; Pham et al., 2020), and wavelet features (Abdel-Hamid, 2020; Koduru et al., 2020) are just a few of the…”
BackgroundInfluentialTemporal Attention Convolutional Network for Speech Emotion Recognition with Latent Representation
Liu, J., Liu, Z., Wang, L., Gao, Y., Guo, L., & Dang, J. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1520 · Cited by 26
“It was recognized that the classification of Happy emotion in IEMOCAP was difficult [31, 32], due to not only the dataset annotation problems but also other relevant factors that need more research in the future.”
BackgroundBidirectional parallel echo state network for speech emotion recognition
Ibrahim, H., Loo, C. K., & Alnajjar, F. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07410-2 · Cited by 25
Falahzadeh, M. R., Farsa, E. Z., Harimi, A., Ahmadi, A., & Abraham, A. (2022) · IEEE Access · doi:10.1109/access.2022.3217226 · Cited by 25
“3D CNNs have been successfully used for speaker verification [29], video scene understanding [30], action recognition [31], and also introduced as promising models to recognize the emotions on the base of feature extraction technique in the speech signals [32], [33].”
BackgroundMethodInfluentialA New Network Structure for Speech Emotion Recognition Research
Xu, C., Liu, Y., Song, W., Liang, Z., & Chen, X. (2024) · Sensors · doi:10.3390/s24051429 · Cited by 25
“Over the past few years, the attention mechanism of deep learning has been successful in the context of speech emotion recognition [32–34], which prefers to focus on emotionally charged words in people’s communication.”
BackgroundCoarse-to-Fine Speech Emotion Recognition Based on Multi-Task Learning
Huijuan, Z., Ye, N., & Ruchuan, W. (2020) · Journal of Signal Processing Systems · doi:10.1007/s11265-020-01538-x · Cited by 24
“1, after obtaining the log-Mel spectrum, we use the method of [2] to get the threechannel input, the dimension is 300*40*3.”
BackgroundMethodInfluentialArezzo, A., & Berretti, S. (2022) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3551626.3564937 · Cited by 24
“To this end, a particular architecture known as ACRNN, was presented for the first time in SER in [6].”
MethodExplore the World of Audio Deepfakes: A Guide to Detection Techniques for Non-Experts
Schäfer, K., Choi, J., & Zmudzinski, S. (2024) · ACM International Workshop on Multimedia AI against Disinformation (MAD) · doi:10.1145/3643491.3660289 · Cited by 24
Audio Multi-View Spoofing Detection Framework Based on Audio-Text-Emotion Correlations
Wu, J., Yin, Q., Sheng, Z., Lu, W., Huang, J., & Li, B. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3431888 · Cited by 24
“Similar to E-DNN [45], the emotional-view features are extracted precedes the last fully connected layer of the SER model [58], and it is followed Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
MethodSpeech Emotion Recognition Using Scalogram Based Deep Structure
Aghajani, K., & Esmaili Paeen Afrakoti, I. (2020) · International Journal of Engineering · doi:10.5829/ije.2020.33.02b.13 · Cited by 24
Neural network-based blended ensemble learning for speech emotion recognition
Yalamanchili, B., Kumar, S. S., & Anne, K. R. (2022) · Multidimensional Systems and Signal Processing · doi:10.1007/s11045-022-00845-9 · Cited by 24
“D convolutional recurrent neural networks with attention-based models were used to improve the localization and quick information analysis (Chen et al., 2018).”
MethodMultimodal Emotion Recognition: A Comprehensive Survey of Datasets, Methods, and Applications
Kumar, M. J. D., Sukesh Rao, M., & Narendra, K. (2025) · IEEE Access · doi:10.1109/access.2025.3636186 · Cited by 23
“…stage uses early fusion to discern complex relationships between synchronised modalities (e.g., speech and facial expressions), while the second stage employs late fusion to integrate these outputs with asynchronous or less reliable modalities (e.g., physiological signals, textual data) [102].”
Research on Robustness of Emotion Recognition Under Environmental Noise Conditions
Huang, Y., Xiao, J., Tian, K., Wu, A., & Zhang, G. (2019) · IEEE Access · doi:10.1109/access.2019.2944386 · Cited by 23
Deep Multimodal Sequence Fusion by Regularized Expressive Representation Distillation
Guo, X., Kong, A. W., & Kot, A. C. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3142448 · Cited by 23
A Feature Fusion Model with Data Augmentation for Speech Emotion Recognition
Tu, Z., Liu, B., Zhao, W., Yan, R., & Zou, Y. (2023) · Applied Sciences · doi:10.3390/app13074124 · Cited by 23
Speech emotion recognition using feature fusion: a hybrid approach to deep learning
Khan, W. A., Qudous, H. U., & Farhan, A. A. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18316-7 · Cited by 23
Emiru, E. D., Xiong, S., Li, Y., Fesseha, A., & Diallo, M. (2021) · Information · doi:10.3390/info12020062 · Cited by 23
Exploration of an Independent Training Framework for Speech Emotion Recognition
Zhong, S., Yu, B., & Zhang, H. (2020) · IEEE Access · doi:10.1109/access.2020.3043894 · Cited by 22
“In order to characterize the dynamic characteristic of speech signal, 3D-based model is considered, where Chen et al. [27] introduced 3D-based attention convolutional recurrent neural networks (CRNN).”
MethodInfluentialSelf-citationA Speech Emotion Recognition Framework for Better Discrimination of Confusions
Liu, J., & Wang, H. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-718 · Cited by 22
“Moreover, a combination of CNN and RNN also becomes popular for SER, where the CNN is designed for frequency information extraction and RNN is used for learning long-term dependencies [7].”
MethodInfluentialTowards adversarial learning of speaker-invariant representation for speech emotion recognitionPreprint
Tu, M., Tang, Y., Huang, J., He, X., & Zhou, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09606 · Cited by 22
“Some studies resort to extracting richer features representing emotion variation or applying more powerful deep neural networks (DNN) with careful architecture designs, expecting better generalization ability to unseen scenarios [1, 2] .”
BackgroundSelf-citationBhangale, K. B., & Kothandaraman, M. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19674-y · Cited by 21
Multi-Path and Group-Loss-Based Network for Speech Emotion Recognition in Multi-Domain Datasets
Noh, K. J., Jeong, C. Y., Lim, J., Chung, S., Kim, G., Lim, J., & Jeong, H. (2021) · Sensors · doi:10.3390/s21051579 · Cited by 21
“UA is calculated as the average of the recall values of four classes and is an important performance indicator in the evaluation of the SER model based on imbalanced datasets [19,20,26].”
BackgroundMethodSpeech Emotion Recognition System by Quaternion Nonlinear Echo State NetworkPreprint
Daneshfar, F., & Kabudian, S. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07234 · Cited by 21
Zheng, T., Gao, M., Wang, Y., Luo, Y., Zhang, H., Xiongsong, T., Sun, J., Wang, P., Dai, G., & Yang, J. (2025) · Nano Energy · doi:10.1016/j.nanoen.2024.110640 · Cited by 20
Self-citation
Metadata
- Authors
- Chen, M., He, X., Yang, J., & Zhang, H.
- Affiliations
- Central South University (CN)
- Venue
- IEEE Signal Processing Letters
- Published
- 2018-10
- Volume
- 25
- Issue
- 10
- Pages
- 1440-1444
- ISSN
- 1070-9908 (ISSN-L)
- 1558-2361
- OpenAlex
- W2885005742
- Cited by
- 503
- References
- 23
- Citations by year
- 2026: 21 · 2025: 50 · 2024: 70 · 2023: 85 · 2022: 91 · 2021: 94 · 2020: 60 · 2019: 27 · 2018: 5
- FWCI
- 60.91
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Signal Processing Letters (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech emotion recognition, attention mechanism, Mel spectrogram, IEMOCAP, EMO-DB
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.