The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching
Abstract
Speech emotion recognition is challenging because of the affective gap between the subjective emotions and low-level features. Integrating multilevel feature learning and model training, deep convolutional neural networks (DCNN) has exhibited remarkable success in bridging the semantic gap in visual tasks like image classification, object detection. This paper explores how to utilize a DCNN to bridge the affective gap in speech signals. To this end, we first extract three channels of log Mel-spectrograms (static, delta, and delta delta) similar to the red, green, blue (RGB) image representation as the DCNN input. Then, the AlexNet DCNN model pretrained on the large ImageNet dataset is employed to learn high-level feature representations on each segment divided from an utterance. The learned segment-level features are aggregated by a discriminant temporal pyramid matching (DTPM) strategy. DTPM combines temporal pyramid matching and optimal Lp-norm pooling to form a global utterance-level feature representation, followed by the linear support vector machines for emotion classification. Experimental results on four public datasets, that is, EMO-DB, RML, eNTERFACE05, and BAUM-1s, show the promising performance of our DCNN model and the DTPM strategy. Another interesting finding is that the DCNN model pretrained for image applications performs reasonably good in affective speech feature extraction. Further fine tuning on the target emotional speech datasets substantially promotes recognition performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS
FISHER, R. A. (1936) · Annals of Eugenics · doi:10.1111/j.1469-1809.1936.tb02137.x · Cited by 15,227
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2389824 · Cited by 11,971
Introduction to Statistical Pattern Recognition
(1990) · Elsevier eBooks · doi:10.1016/c2009-0-27872-x · Cited by 11,031
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10578-9_23 · Cited by 3,034
Graph Embedding and Extensions: A General Framework for Dimensionality Reduction
Yan, S., Xu, D., Zhang, B., Zhang, H.-J., Yang, Q., & Lin, S. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.250598 · Cited by 3,031
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
MatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444
A database of German emotional speech
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-446 · Cited by 2,365 · cited 3×
“We test the proposed method on four public datasets, including the Berlin dataset of German emotional speech (EMO-DB) [21], the RML audio-visual dataset [22], the eNTERFACE05 audio-visual dataset [23], and the BAUM-1s audio-visual dataset [24].”
BackgroundMethodConvolutional Neural Networks for Speech Recognition
Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., Deng, L., Penn, G., & Yu, D. (2014) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2014.2339736 · Cited by 2,262
Survey on speech emotion recognition: Features, classification schemes, and databases
El Ayadi, M., Kamel, M. S., & Karray, F. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,073
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · cited 5×
“Note that some pervious works [5], [62] also employ Unweighted Averaged recall (UAR), which is used Fig.”
BackgroundMethodInfluentialA convolutional neural network cascade for face detection
Li, H., Lin, Z., Shen, X., Brandt, J., & Hua, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299170 · Cited by 1,390
(2011) · doi:10.5591/978-1-57735-516-8/ijcai11-210 · Cited by 1,229
Speech emotion recognition using hidden Markov models
Nwe, T. L., Foo, S. W., & De Silva, L. C. (2003) · Speech Communication · doi:10.1016/s0167-6393(03)00099-2 · Cited by 920
Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network
Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472669 · Cited by 904
Speech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-57 · Cited by 794
Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771
Schuller, B., Steidl, S., Batliner, A., Vinciarelli, A., Scherer, K., Ringeval, F., Chetouani, M., Weninger, F., Eyben, F., Marchi, E., et al. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-56 · Cited by 731
McKeown, G., Valstar, M., Cowie, R., Pantic, M., & Schroder, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.20 · Cited by 665
The eNTERFACE’05 Audio-Visual Emotion Database
Martin, O., Kotsia, I., Macq, B., & Pitas, I. (2006) · 22nd International Conference on Data Engineering Workshops (ICDEW'06) · doi:10.1109/icdew.2006.145 · Cited by 615
AVEC 2013: the continuous audio/visual emotion and depression recognition challenge
Valstar, M., Schuller, B., Smith, K., Eyben, F., Jiang, B., Bilakhia, S., Schnieder, S., Cowie, R., & Pantic, M. (2013) · Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge · doi:10.1145/2512530.2512533 · Cited by 581
Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568 · cited 3×
“ploy 1-D convolution, such as frequency convolution [14]–[16]”
BackgroundMethodFeatures and classifiers for emotion recognition from speech: a survey from 2000 to 2011
Anagnostopoulos, C.-N., Iliou, T., & Giannoukos, I. (2015) · Artificial Intelligence Review · doi:10.1007/s10462-012-9368-5 · Cited by 428
Schuller, B., Rigoll, G., & Lang, M. (2004) · 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.2004.1326051 · Cited by 426
Speech Emotion Recognition Using Fourier Parameters
Wang, K., An, N., Li, B. N., Zhang, Y., & Li, L. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2392101 · Cited by 408
Speech Emotion Recognition Using CNN
Huang, Z., Dong, M., Mao, Q., & Zhan, Y. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654984 · Cited by 398
Automatic speech emotion recognition using modulation spectral features
Wu, S., Falk, T. H., & Chan, W.-Y. (2011) · Speech Communication · doi:10.1016/j.specom.2010.08.013 · Cited by 389
Emotion Recognition in Speech Using Neural Networks
Nicholson, J., Takahashi, K., & Nakatsu, R. (2000) · Neural Computing & Applications · doi:10.1007/s005210070006 · Cited by 344
Ensemble methods for spoken emotion recognition in call-centres
Morrison, D., Wang, R., & De Silva, L. C. (2007) · Speech Communication · doi:10.1016/j.specom.2006.11.004 · Cited by 320
LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework
Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.03.001 · Cited by 288
Deep neural networks for acoustic emotion recognition: Raising the benchmarks
Stuhlsatz, A., Meyer, C., Eyben, F., Zielke, T., Meier, G., & Schuller, B. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947651 · Cited by 288 · cited 5×
“For instance, in [12], [13] a DNN is used”
BackgroundMethodInfluentialAcoustic emotion recognition: A benchmark comparison of performances
Schuller, B., Vlasenko, B., Eyben, F., Rigoll, G., & Wendemuth, A. (2009) · 2009 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2009.5372886 · Cited by 267
Emotion recognition in speech signal: experimental study, development, and application
Petrushin, V. A. (2000) · 6th International Conference on Spoken Language Processing (ICSLP 2000) · doi:10.21437/icslp.2000-791 · Cited by 246
BAUM-1: A Spontaneous Audio-Visual Face Database of Affective and Mental States
Zhalehpour, S., Onder, O., Akhtar, Z., & Erdem, C. E. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2553038 · Cited by 216
Recognizing Human Emotional State From Audiovisual Signals
Wang, Y., & Guan, L. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.921734 · Cited by 214
Recognizing Human Emotional State From Audiovisual Signals*
Wang, Y., & Guan, L. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.927665 · Cited by 202
Speech emotion recognition approaches in human computer interaction
Ramakrishnan, S., & El Emary, I. M. M. (2013) · Telecommunication Systems · doi:10.1007/s11235-011-9624-z · Cited by 187
Speech emotion recognition with acoustic and lexical features
Jin, Q., Li, C., Chen, S., & Wu, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178872 · Cited by 168
Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks
Gulcehre, C., Cho, K., Pascanu, R., & Bengio, Y. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-662-44848-9_34 · Cited by 167
Real-time Speech and Music Classification by Large Audio Feature Space Extraction
Eyben, F. (2015) · Springer theses · doi:10.1007/978-3-319-27299-3 · Cited by 164
Feature Analysis and Evaluation for Automatic Emotion Identification in Speech
Luengo, I., Navas, E., & Hernaez, I. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2051872 · Cited by 161
Emotional space improves emotion recognition
Tato, R., Santos, R., Kompe, R., & Pardo, J. M. (2002) · 7th International Conference on Spoken Language Processing (ICSLP 2002) · doi:10.21437/icslp.2002-557 · Cited by 158
Dellaert, F., Polzin, T., & Waibel, A. (1996) · 4th International Conference on Spoken Language Processing (ICSLP 1996) · doi:10.21437/icslp.1996-462 · Cited by 158
The Relevance of Voice Quality Features in Speaker Independent Emotion Recognition
Lugger, M., & Yang, B. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.367152 · Cited by 144
Spoken emotion recognition using hierarchical classifiers
Albornoz, E. M., Milone, D. H., & Rufiner, H. L. (2010) · Computer Speech & Language · doi:10.1016/j.csl.2010.10.001 · Cited by 140
Geometric ℓp-norm feature pooling for image classification
Feng, J., Ni, B., Tian, Q., & Yan, S. (2011) · doi:10.1109/cvpr.2011.5995370 · Cited by 125
Weighted spectral features based on local Hu moments for speech emotion recognition
Sun, Y., Wen, G., & Wang, J. (2015) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2014.10.008 · Cited by 118
Speech Emotion Analysis: Exploring the Role of Context
Tawari, A., & Trivedi, M. M. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2058095 · Cited by 117
Interdependencies among Voice Source Parameters in Emotional Speech
Sundberg, J., Patel, S. A., Björkner, E., & Scherer, K. R. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.14 · Cited by 106
Affective Visualization and Retrieval for Music Video
Zhang, S., Huang, Q., Jiang, S., Gao, W., & Tian, Q. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2059634 · Cited by 98
Self-citationTiming levels in segment-based speech emotion recognition
Schuller, B. W., & Rigoll, G. (2006) · Interspeech (USB) · doi:10.21437/interspeech.2006-502 · Cited by 90
Spoken emotion recognition through optimum-path forest classification using glottal features
Iliev, A. I., Scordilis, M. S., Papa, J. P., & Falcão, A. X. (2009) · Computer Speech & Language · doi:10.1016/j.csl.2009.02.005 · Cited by 87
Recognizing Affect from Linguistic Information in 3D Continuous Space
Schuller, B. W. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.17 · Cited by 69
Segment-Based Approach to the Recognition of Emotions in Speech
Shami, M., & Kamel, M. S. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521436 · Cited by 63
Emotion Recognition in Chinese Natural Speech by Combining Prosody and Voice Quality Features
Zhang, S. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-87734-9_52 · Cited by 51
Self-citationRobust emotion recognition in noisy speech via sparse representation
Zhao, X., Zhang, S., & Lei, B. (2013) · Neural Computing and Applications · doi:10.1007/s00521-013-1377-z · Cited by 50
Self-citationProvost, E. M. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638345 · Cited by 45
Multiview emotion recognition via multi-set locality preserving canonical correlation analysis
Elmadany, N., He, Y., & Guan, L. (2016) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2016.7527309 · Cited by 27
Considering cross-cultural context in the automatic recognition of emotions
Quirós-Ramírez, M. A., & Onisawa, T. (2013) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-013-0192-2 · Cited by 15
Spoken emotion recognition via locality-constrained kernel sparse representation
Zhao, X., & Zhang, S. (2014) · Neural Computing and Applications · doi:10.1007/s00521-014-1755-1 · Cited by 14
Self-citationAcoustic and lexical representations for affect prediction in spontaneous conversations
Cao, H., Savran, A., Verma, R., & Nenkova, A. (2014) · Computer Speech & Language · doi:10.1016/j.csl.2014.04.002 · Cited by 9
Gao, L., Qi, L., & Guan, L. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472191 · Cited by 9
Cited by
Speech Emotion Recognition Using Deep Learning Techniques: A Review
Khalil, R. A., Jones, E., Babar, M. I., Jan, T., Zafar, M. H., & Alhussain, T. (2019) · IEEE Access · doi:10.1109/access.2019.2936124 · Cited by 702
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 615
3-D Convolutional Recurrent Neural Networks With Attention Model for Speech Emotion Recognition
Chen, M., He, X., Yang, J., & Zhang, H. (2018) · IEEE Signal Processing Letters · doi:10.1109/lsp.2018.2860246 · Cited by 503
“Speech as the most convenient and natural medium for human communication, not only carries the implicit semantic information, but also contains rich affective information [1].”
BackgroundKhare, S. K., Blanes-Vidal, V., Nadimi, E. S., & Acharya, U. R. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102019 · Cited by 469
Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM
Mustaqeem, Sajjad, M., & Kwon, S. (2020) · IEEE Access · doi:10.1109/access.2020.2990405 · Cited by 402
Metadata
- Authors
- Zhang, S., Zhang, S., Huang, T., & Gao, W.
- Venue
- IEEE Transactions on Multimedia
- Published
- 2018-06
- Volume
- 20
- Issue
- 6
- Pages
- 1576-1590
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2766272105
- Cited by
- 441
- References
- 70
- FWCI
- 43.14
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.