Learning Salient Features for Speech Emotion <newline/>Recognition Using Convolutional <newline/>Neural Networks
The DOI matches its Crossref record; 46 of the 47 references it cites were found in a registry or an index.
Abstract
As an essential way of human emotional behavior understanding, speech emotion recognition (SER) has attracted a great deal of attention in human-centered signal processing. Accuracy in SER heavily depends on finding good affect- related , discriminative features. In this paper, we propose to learn affect-salient features for SER using convolutional neural networks (CNN). The training of CNN involves two stages. In the first stage, unlabeled samples are used to learn local invariant features (LIF) using a variant of sparse auto-encoder (SAE) with reconstruction penalization. In the second step, LIF is used as the input to a feature extractor, salient discriminative feature analysis (SDFA), to learn affect-salient, discriminative features using a novel objective function that encourages feature saliency, orthogonality, and discrimination for SER. Our experimental results on benchmark datasets show that our approach leads to stable and robust recognition performance in complex scenes (e.g., with speaker and language variation, and environment distortion) and outperforms several well-established SER features.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“neglecting to model the emotion-related correlations within frequency domain during the time-frequency joint learning”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [8]
Experimental — accuracy not yet measuredIn the citing paper
“recent spectrogram-based works mainly focus on modeling the long-term dependency in time domain”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [8]
Citations
Cites
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · cited 1×
“We use the spectrogram of the speech signal as the input of CNN.”
MethodRepresentation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 1×
“Recently, it has been shown that unsupervised feature leaning is very helpful for ASR [43] and image understanding [2], [10].”
BackgroundA Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
A database of German emotional speech
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-446 · Cited by 2,365 · cited 1×
“The second is Berlin Emotional Database (Emo-DB) [3], which also includes seven emotions (i.”
BackgroundSurvey on speech emotion recognition: Features, classification schemes, and databases
El Ayadi, M., Kamel, M. S., & Karray, F. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,073
Toward detecting emotions in spoken dialogs
Lee, C. M., & Narayanan, S. S. (2005) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2004.838534 · Cited by 988 · cited 1×
“An alternative procedure for detecting emotions using lexical information is found in [18].”
Method(2009) · neural information processing systems · Cited by 962
Emotional speech recognition: Resources, features, and methods
Ververidis, D., & Kotropoulos, C. (2006) · Speech Communication · doi:10.1016/j.specom.2006.04.003 · Cited by 925 · cited 1×
“In the following, we first briefly review the classification strategies, and then focus on feature extraction methods, as they are more related to this work.”
MethodSpeech emotion recognition using hidden Markov models
Nwe, T. L., Foo, S. W., & De Silva, L. C. (2003) · Speech Communication · doi:10.1016/s0167-6393(03)00099-2 · Cited by 920
Schuller, B., Rigoll, G., & Lang, M. (2004) · 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.2004.1326051 · Cited by 426 · cited 1×
“In [34], a spotting algorithm that searches for emotional keywords or phrases in the utterances was employed.”
MethodOpenEAR — Introducing the munich open-source emotion and affect recognition toolkit
Eyben, F., Wollmer, M., & Schuller, B. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349350 · Cited by 418 · cited 4×
“Table II compares the accuracy and the standard deviation for RAW, TEO, A1 [23], A2 [8], LIF, SDFA(no_or), SDFA(no_s),”
MethodInfluentialSpeech Emotion Recognition Using CNN
Huang, Z., Dong, M., Mao, Q., & Zhan, Y. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654984 · Cited by 398 · cited 1×
“The preliminary version of this work was first presented in a shortened form as a conference abstract [14].”
BackgroundSelf-citationSparse Autoencoder-Based Feature Transfer Learning for Speech Emotion Recognition
Deng, J., Zhang, Z., Marchi, E., & Schuller, B. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.90 · Cited by 366 · cited 2×
“In [6], a common emotion-specific mapping rule is learnt from a long feature vector.”
BackgroundMethodEnsemble methods for spoken emotion recognition in call-centres
Morrison, D., Wang, R., & De Silva, L. C. (2007) · Speech Communication · doi:10.1016/j.specom.2006.11.004 · Cited by 320
Deep neural networks for acoustic emotion recognition: Raising the benchmarks
Stuhlsatz, A., Meyer, C., Eyben, F., Zielke, T., Meier, G., & Schuller, B. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947651 · Cited by 288 · cited 1×
“For SER, Stuhlsatz et al. [36] used generatively pre-trained artificial neural networks (ANNs) to learn discriminative features of low dimension and found improvement in both weighted and unweighted recall on multiple emotion corpora.”
BackgroundWu, C.-H., & Liang, W.-B. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.16 · Cited by 266 · cited 1×
“More recently, there has been a focus on hybrid features, i.e., the integration of acoustic, linguistic, and context features [29], [42] Although linguistic content of the spoken utterance can help the acoustic emotion features improve the accuracy of SER, current ASR systems still cannot reliably…”
BackgroundSpeaker-dependent audio-visual emotion recognition
Haq, S., & Jackson, P. J. B. (2009) · Open Research (University of Surrey) · Cited by 228
Disentangling Factors of Variation for Facial Expression Recognition
Rifai, S., Bengio, Y., Courville, A., Vincent, P., & Mirza, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_58 · Cited by 215 · cited 1×
“In many situations where labeled data is limited or not available, deep learning is shown to have the capability to generate good features, for example, for facial expression recognition [31] and ASR [43].”
BackgroundSpeech emotion recognition approaches in human computer interaction
Ramakrishnan, S., & El Emary, I. M. M. (2013) · Telecommunication Systems · doi:10.1007/s11235-011-9624-z · Cited by 187 · cited 2×
“More recently, there has been a focus on hybrid features, i.e., the integration of acoustic, linguistic, and context features [29], [42] Although linguistic content of the spoken utterance can help the acoustic emotion features improve the accuracy of SER, current ASR systems still cannot reliably…”
BackgroundMethodEmotion recognition from speech using global and local prosodic features
Rao, K. S., Koolagudi, S. G., & Vempada, R. R. (2013) · International Journal of Speech Technology · doi:10.1007/s10772-012-9172-2 · Cited by 184 · cited 1×
“For example, prosodic speech features such as pitch and energy, can be extracted from each interval and are considered as local features [26], [30].”
BackgroundFeature Learning in Deep Neural Networks - Studies on Speech Recognition Tasks
Yu, D., Seltzer, M. L., Li, J., Huang, J.-T., & Seide, F. T. B. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3605 · Cited by 174 · cited 2×
“Recently, it has been shown that unsupervised feature leaning is very helpful for ASR [43] and image understanding [2], [10].”
BackgroundMethodFeature Analysis and Evaluation for Automatic Emotion Identification in Speech
Luengo, I., Navas, E., & Hernaez, I. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2051872 · Cited by 161 · cited 1×
“In this direction, a number of speech emotion features have been proposed in the literature, and they can be roughly classified into four categories [20]: 1) acoustic features, 2) linguistic features (words and discourse), 3) context information (e.”
BackgroundAudio-Visual Affect Recognition
Zeng, Z., Tu, J., Liu, M., Huang, T. S., Pianfetti, B., Roth, D., & Levinson, S. (2007) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.886310 · Cited by 136 · cited 1×
“The increasing applications of SER, especially those in human computer interaction and affective computing [27], [45], make it a core component in the”
BackgroundDesign, recording and verification of a danish emotional speech database
Engberg, I. S., Hansen, A. V., Andersen, O., & Dalsgaard, P. (1997) · 5th European Conference on Speech Communication and Technology (Eurospeech 1997) · doi:10.21437/eurospeech.1997-482 · Cited by 128 · cited 5×
“We divide the four public emotional speech databases into two groups: one group is SAVEE and Emo-DB, and the other one is DES and MES.”
BackgroundInfluentialModular neural-SVM scheme for speech emotion recognition using ANOVA feature selection method
Sheikhan, M., Bejani, M., & Gharavian, D. (2012) · Neural Computing and Applications · doi:10.1007/s00521-012-0814-8 · Cited by 118
Emotion recognition from spontaneous speech using Hidden Markov models with deep belief networks
Le, D., & Provost, E. M. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707732 · Cited by 117 · cited 1×
“More recently, Le et al. [17] investigated dynamic frame-level modeling with hybrid DBN-HMM classifiers on the FAU Aibo spontaneous emotion corpus [35] and achieved state-of-the-art results on the 5-class problem.”
MethodSpeech Emotion Analysis: Exploring the Role of Context
Tawari, A., & Trivedi, M. M. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2058095 · Cited by 117 · cited 1×
“They have been investigated in [39] and [11] on audio affective recognition.”
BackgroundHuman-Centred Intelligent Human Computer Interaction (HCI²): how far are we from attaining it?
Pantic, M., Nijholt, A., Pentland, A., & Huanag, T. S. (2008) · International Journal of Autonomous and Adaptive Communications Systems · doi:10.1504/ijaacs.2008.019799 · Cited by 109
Gharavian, D., Sheikhan, M., Nazerieh, A., & Garoucy, S. (2011) · Neural Computing and Applications · doi:10.1007/s00521-011-0643-1 · Cited by 104
Emotion recognition from speech using source, system, and prosodic features
Koolagudi, S. G., & Rao, K. S. (2012) · International Journal of Speech Technology · doi:10.1007/s10772-012-9139-3 · Cited by 90 · cited 1×
“On the other hand, global features such as statistics, can be obtained from thewhole speech utterance [22], [23], [16], which typically have a lower dimension than the local ones, leading to less computing time [13], [30].”
BackgroundASR for emotional speech: Clarifying the issues and enhancing performance
Athanaselis, T., Bakamidis, S., Dologlou, I., Cowie, R., Douglas‐Cowie, E., & Cox, C. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.008 · Cited by 88 · cited 1×
“In this paper, we introduce a feature learning framework to SER using CNN, in which we directly deal with the issue of entangle factors of variation through a novel object function that integrates feature saliency, discrimination, orthogonality, and reconstruction.”
MethodLearning emotion-based acoustic features with deep belief networks
Schmidt, E. M., & Kim, Y. E. (2011) · 2011 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/aspaa.2011.6082328 · Cited by 87 · cited 1×
“Schmidt and Kim [33] used deep belief networks (DBNs) to learn high-level features directly frommagnitude spectra and achieved good performances on emotional music recognition compared to other feature extraction schemes.”
BackgroundDeng, J., Xia, R., Zhang, Z., Liu, Y., & Schuller, B. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854517 · Cited by 71
Emotion Recognition in Chinese Natural Speech by Combining Prosody and Voice Quality Features
Zhang, S. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-87734-9_52 · Cited by 54 · cited 1×
“Four emotions from Chinese natural emotional speech corpus including anger, joy, sadness, and neutral are discriminated by combining prosody and voice quality features in [46].”
BackgroundEmotion recognition from speech using sub-syllabic and pitch synchronous spectral features
Koolagudi, S. G., & Krothapalli, S. R. (2012) · International Journal of Speech Technology · doi:10.1007/s10772-012-9150-8 · Cited by 53 · cited 1×
“…be further classified into four groups [24]: continuous features, qualitative features, spectral features, and Teager Energy Operator (TEO)-based features, and all of them have been intensively studied in SER. Koolagudi et al. [15] examined pitch-related features in Berlin emotion speech corpus.”
BackgroundLoss-Scaled Large-Margin Gaussian Mixture Models for Speech Emotion Classification
Yun, S., & Yoo, C. D. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2162405 · Cited by 48 · cited 1×
“Various types of classifiers have been used for the task of SER, including hidden Markov model [26], Gaussian mixture model [38], support vector machine (SVM) [23], artificial neural networks [4], k-nearest neighbor [28] and many others [29].”
MethodPao, T.-L., Chen, Y.-T., Yeh, J.-H., Cheng, Y.-M., & Lin, Y.-Y. (2007) · Springer eBooks · doi:10.1007/978-3-540-74171-8_101 · Cited by 36 · cited 1×
“In the following, we first briefly review the classification strategies, and then focus on feature extraction methods, as they are more related to this work.”
MethodInvestigating Glottal Parameters and Teager Energy Operators in Emotion Recognition
Sun, R., & Moore, E. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_54 · Cited by 35 · cited 1×
“The discrimination ability of TEO for SER is studied in [37].”
BackgroundAn Unsupervised Hierarchical Feature Learning Framework for One-Shot Image Recognition
Guo, Z., & Wang, Z. J. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2234729 · Cited by 32 · cited 1×
“Recently, it has been shown that unsupervised feature leaning is very helpful for ASR [43] and image understanding [2], [10].”
BackgroundWöllmer, M., Metallinou, A., Katsamanis, N., Schuller, B. W., & Narayanan, S. S. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288834 · Cited by 31 · cited 1×
“…al. [41] presents a method to systematically investigate the number of past and future utterance-level observations that are considered to generate an emotion prediction for a given utterance, and to examine to what extent this temporal bidirectional context contributes to the overall performance.”
BackgroundCombining Classifiers With Diverse Feature Sets For Robust Speaker Independent Emotion Recognition
Lugger, M. (2009) · doi:10.5281/zenodo.41415 · Cited by 31 · cited 1×
“A. Classifiers Various types of classifiers have been used for the task of SER, including hidden Markov model [26], Gaussian mixture model [38], support vector machine (SVM) [23], artificial neural networks [4], k-nearest neighbor [28] and many others [29].”
MethodFusion of global statistical and segmental spectral features for speech emotion recognition
Hu, H., Xu, M., & Wu, W. (2007) · Interspeech (USB) · doi:10.21437/interspeech.2007-616 · Cited by 30 · cited 1×
“On the other hand, global features such as statistics, can be obtained from thewhole speech utterance [22], [23], [16], which typically have a lower dimension than the local ones, leading to less computing time [13], [30].”
BackgroundSPEECH EMOTION RECOGNITION METHOD BASED ON IMPROVED DECISION TREE AND LAYERED FEATURE SELECTION
Mao, Q., WANG, X., & Zhan, Y. (2010) · International Journal of Humanoid Robotics · doi:10.1142/s0219843610002088 · Cited by 23 · cited 17×
“The affectsalient feature block of the final feature vector is passed to a SVM classifier to determine the emotion class of the speech utterance.”
BackgroundMethodInfluentialSelf-citationSpeech Emotion Recognition Based on Parametric Filter and Fractal Dimension
Mao, X., & Chen, L. (2010) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.e93.d.2324 · Cited by 21 · cited 5×
“We divide the four public emotional speech databases into two groups: one group is SAVEE and Emo-DB, and the other one is DES and MES.”
BackgroundMethodInfluentialSpeech Emotion Recognition using a backward context
Guven, E., & Bock, P. (2010) · IEEE Applied Imagery Pattern Recognition Workshop · doi:10.1109/aipr.2010.5759701 · Cited by 7 · cited 1×
“They have been investigated in [39] and [11] on audio affective recognition.”
BackgroundBased on EEMD-HHT Marginal Spectrum of Speech Emotion Recognition
Zhang, W., Zhang, X., & Sun, Y. (2012) · International Conference on Computing, Measurement, Control and Sensor Network · doi:10.1109/cmcsn.2012.24 · Cited by 6 · cited 1×
“In [47], the amplitude of emotional speech marginal spectrum is extracted.”
Background
Research works printed but not linked: 1
- steidl, 2009, Automatic Classification of Emotion-Related User States in Spontaneous Children's Speech
Cited by
Speech emotion recognition using deep 1D & 2D CNN LSTM networks
Zhao, J., Mao, X., & Chen, L. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2018.08.035 · Cited by 1,044
A review of emotion recognition using physiological signals
Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., & Yang, X. (2018) · Sensors · doi:10.3390/s18072074 · Cited by 923
“One is using human physical signals such as facial expression [4], speech [5], gesture, posture, etc.”
BackgroundAdieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network
Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472669 · Cited by 904
“In the field of paralinguistics, several studies have been carried out using CNNs for feature learning, e.g., recently by Milde and Biemann [14], and Mao et al. [15].”
MethodAkçay, M. B., & Oğuz, K. (2020) · Speech Communication · doi:10.1016/j.specom.2019.12.001 · Cited by 721
Speech Emotion Recognition Using Deep Learning Techniques: A Review
Khalil, R. A., Jones, E., Babar, M. I., Jan, T., Zafar, M. H., & Alhussain, T. (2019) · IEEE Access · doi:10.1109/access.2019.2936124 · Cited by 702
“Preprocessing is an aditionaltask usually required to be carried out before recognition of emotion from the speech signal [120].”
BackgroundInfluential
Metadata
- Authors
- Mao, Q., Dong, M., Huang, Z., & Zhan, Y.
- Affiliations
- Jiangsu University (CN)
- Wayne State University (US)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2014-12
- Volume
- 16
- Issue
- 8
- Pages
- 2203-2213
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2087618018
- Cited by
- 568
- References
- 47
- Citations by year
- 2026: 19 · 2025: 24 · 2024: 46 · 2023: 72 · 2022: 73 · 2021: 83 · 2020: 67 · 2019: 59 · 2018: 53 · 2017: 46 · 2016: 15 · 2015: 11
- FWCI
- 33.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Funders
- the National Nature Science Foundation of China (61272211, 61170126)
- the Six Talent Peaks Foundation of Jiangsu Province
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech emotion recognition, discriminative feature learning, local invariant features, sparse autoencoders, feature saliency, unsupervised feature learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.