Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM
Open accessThe DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Abstract
Emotional state recognition of a speaker is a difficult task for machine learning algorithms which plays an important role in the field of speech emotion recognition (SER). SER plays a significant role in many real-time applications such as human behavior assessment, human-robot interaction, virtual reality, and emergency centers to analyze the emotional state of speakers. Previous research in this field is mostly focused on handcrafted features and traditional convolutional neural network (CNN) models used to extract high-level features from speech spectrograms to increase the recognition accuracy and overall model cost complexity. In contrast, we introduce a novel framework for SER using a key sequence segment selection based on redial based function network (RBFN) similarity measurement in clusters. The selected sequence is converted into a spectrogram by applying the STFT algorithm and passed into the CNN model to extract the discriminative and salient features from the speech spectrogram. Furthermore, we normalize the CNN features to ensure precise recognition performance and feed them to the deep bi-directional long short-term memory (BiLSTM) to learn the temporal information for recognizing the final state of emotion. In the proposed technique, we process the key segments instead of the whole utterance to reduce the computational complexity of the overall model and normalize the CNN features before their actual processing, so that it can easily recognize the Spatio-temporal information. The proposed system is evaluated over different standard dataset including IEMOCAP, EMO-DB, and RAVDESS to improve the recognition accuracy and reduce the processing time of the model, respectively. The robustness and effectiveness of the suggested SER model is proved from the experimentations when compared to state-of-the-art SER methods with an achieve up to 72.25%, 85.57%, and 77.02% accuracy over IEMOCAP, EMO-DB, and RAVDESS dataset, respectively.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484
A database of German emotional speech
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-446 · Cited by 2,365 · cited 2×
“In this paper, we utilize the three public speech emotions datasets, the IEMOCAP [49] interactive emotional dyadic motion capture dataset, Emo-DB [50] berlin emotional dataset, and RAVDESS [51] Ryerson audio-visual dataset of emotional speech and songs.”
MethodLivingstone, S. R., & Russo, F. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0196391 · Cited by 1,883
Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks
Sainath, T. N., Vinyals, O., Senior, A., & Sak, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178838 · Cited by 1,678
Approximation of dynamical systems by continuous time recurrent neural networks
Funahashi, K.-I., & Nakamura, Y. (1993) · Neural Networks · doi:10.1016/s0893-6080(05)80125-x · Cited by 980
Speech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-57 · Cited by 794
Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771
End-to-End Multimodal Emotion Recognition using Deep Neural Networks
Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2017) · arXiv · doi:10.48550/arxiv.1704.08619 · Cited by 752
Speech Emotion Recognition Using Deep Learning Techniques: A Review
Khalil, R. A., Jones, E., Babar, M. I., Jan, T., Zafar, M. H., & Alhussain, T. (2019) · IEEE Access · doi:10.1109/access.2019.2936124 · Cited by 702
ECG Arrhythmia Classification Using STFT-Based Spectrogram and Convolutional Neural Network
Huang, J., Chen, B., Yao, B., & He, W. (2019) · IEEE Access · doi:10.1109/access.2019.2928017 · Cited by 591
Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568 · cited 1×
“In the field of SER, many methods utilize CNN models with different types of input to extract salient features from speech signals to boost the recognition accuracy [30].”
Method3-D Convolutional Recurrent Neural Networks With Attention Model for Speech Emotion Recognition
Chen, M., He, X., Yang, J., & Zhang, H. (2018) · IEEE Signal Processing Letters · doi:10.1109/lsp.2018.2860246 · Cited by 503
Zhang, S., Zhang, S., Huang, T., & Gao, W. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2766843 · Cited by 441
A CNN-Assisted Enhanced Audio Signal Processing for Speech Emotion Recognition
Mustaqeem, & Kwon, S. (2019) · Sensors · doi:10.3390/s20010183 · Cited by 392
Self-citationSpeech Emotion Recognition From 3D Log-Mel Spectrograms With Deep Learning Network
Meng, H., Yan, T., Yuan, F., & Wei, H. (2019) · IEEE Access · doi:10.1109/access.2019.2938007 · Cited by 356
MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition
Song, T., Zheng, W., Lu, C., Zong, Y., Zhang, X., & Cui, Z. (2019) · IEEE Access · doi:10.1109/access.2019.2891579 · Cited by 336
DepAudioNet: An Efficient Deep Model for Audio based Depression Classification
Ma, X., Yang, H., Chen, Q., Huang, D., & Wang, Y. (2016) · Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2988257.2988267 · Cited by 298
Chouhan, S. S., Kaul, A., Singh, U. P., & Jain, S. (2018) · IEEE Access · doi:10.1109/access.2018.2800685 · Cited by 287
End-to-End Speech Emotion Recognition Using Deep Neural Networks
Tzirakis, P., Zhang, J., & Schuller, B. W. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462677 · Cited by 277
Clustering Approach Based on Mini Batch Kmeans for Intrusion Detection System Over Big Data
Peng, K., Leung, V. C. M., & Huang, Q. (2018) · IEEE Access · doi:10.1109/access.2018.2810267 · Cited by 268
Speech emotion recognition based on feature selection and extreme learning machine decision tree
Liu, Z.-T., Wu, M., Cao, W.-H., Mao, J.-W., Xu, J.-P., & Tan, G.-Z. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.07.050 · Cited by 267
SVM or deep learning? A comparative study on remote sensing image classification
Liu, P., Choo, K.-K. R., Wang, L., & Huang, F. (2017) · Soft Computing · doi:10.1007/s00500-016-2247-2 · Cited by 267
Bagged support vector machines for emotion recognition from speech
Bhavan, A., Chauhan, P., Hitkul, & Shah, R. R. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.104886 · Cited by 248
Sound Classification Using Convolutional Neural Network and Tensor Deep Stacking Network
Khamparia, A., Gupta, D., Nguyen, N. G., Khanna, A., Pandey, B., & Tiwari, P. (2019) · IEEE Access · doi:10.1109/access.2018.2888882 · Cited by 237
An efficient approximation to the K-means clustering for massive data
Capó, M., Pérez, A., & Lozano, J. A. (2017) · Knowledge-Based Systems · doi:10.1016/j.knosys.2016.06.031 · Cited by 218
Insights Into LSTM Fully Convolutional Networks for Time Series Classification
Karim, F., Majumdar, S., & Darabi, H. (2019) · IEEE Access · doi:10.1109/access.2019.2916828 · Cited by 208
Fulop, S. A., & Fitz, K. (2006) · The Journal of the Acoustical Society of America · doi:10.1121/1.2133000 · Cited by 203
An experimental study of speech emotion recognition based on deep convolutional neural networks
Zheng, W. Q., Yu, J. S., & Zou, Y. X. (2015) · 2015 International Conference on Affective Computing and Intelligent Interaction (ACII) · doi:10.1109/acii.2015.7344669 · Cited by 187
Deep features-based speech emotion recognition for smart affective services
Badshah, A. M., Rahim, N., Ullah, N., Ahmad, J., Muhammad, K., Lee, M. Y., Kwon, S., & Baik, S. W. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-017-5292-7 · Cited by 182
Self-citationSpectrogram based multi-task audio classification
Zeng, Y., Mao, H., Peng, D., & Yi, Z. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-017-5539-3 · Cited by 178
Cloud-Assisted Multiview Video Summarization Using CNN and Bidirectional LSTM
Hussain, T., Muhammad, K., Ullah, A., Cao, Z., Baik, S. W., & de Albuquerque, V. H. C. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2929228 · Cited by 170
An Image-based Deep Spectrum Feature Representation for the Recognition of Emotional Speech
Cummins, N., Amiriparian, S., Hagerer, G., Batliner, A. M., Steidl, S., & Schuller, B. W. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123371 · Cited by 167
Zhao, Z., Bao, Z., Zhao, Y., Zhang, Z., Cummins, N., Ren, Z. X., & Schuller, B. W. (2019) · IEEE Access · doi:10.1109/access.2019.2928625 · Cited by 154
Time series feature learning with labeled and unlabeled data
Wang, H., Zhang, Q., Wu, J., Pan, S., & Chen, Y. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2018.12.026 · Cited by 128
Cover the Violence: A Novel Deep-Learning-Based Approach Towards Violence-Detection in Movies
Khan, S. U., Haq, I. U., Rho, S., et al. (2019) · Applied Sciences · doi:10.3390/app9224963 · Cited by 123
Jiang, P., Fu, H., Tao, H., Lei, P., & Zhao, L. (2019) · IEEE Access · doi:10.1109/access.2019.2927384 · Cited by 118
Wang, Z.-Q., & Tashev, I. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953138 · Cited by 101
Emotion Recognition from Variable-Length Speech Segments Using Deep Learning on Spectrograms
Ma, X., Wu, Z., Jia, J., Xu, M., Meng, H., & Cai, L. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-2228 · Cited by 100
Deep Learning based Emotion Recognition System Using Speech Features and TranscriptionsPreprint
Tripathi, S., Kumar, A., Ramesh, A., Singh, C., & Yenigalla, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05681 · Cited by 97
Emotion Detection from Speech Signals using Voting Mechanism on Classified Frames
Zamil, A. A. A., Hasan, S., Baki, S. M. J., Adam, J. M., & Zaman, I. (2019) · International Conference on Robotics, Electrical and Signal Processing Techniques (ICREST) · doi:10.1109/icrest.2019.8644168 · Cited by 89
Effective Combination of DenseNet and BiLSTM for Keyword Spotting
Mengjun, Z., & Xiao, N. (2019) · IEEE Access · doi:10.1109/access.2019.2891838 · Cited by 85
Learning Temporal Clusters Using Capsule Routing for Speech Emotion Recognition
Jalal, M. A., Loweimi, E., Moore, R. K., & Hain, T. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3068 · Cited by 80
Luo, D., Zou, Y., & Huang, D. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1832 · Cited by 77
Ogawa, A., & Hori, T. (2017) · Speech Communication · doi:10.1016/j.specom.2017.02.009 · Cited by 77
Attention-based Convolutional Neural Networks for Acoustic Scene Classification
Ren, Z. X., Kong, Q., Qian, K., Plumbley, M. D., & Schuller, B. W. (2018) · Surrey Research Insight Open Access (The University of Surrey) · doi:10.5281/zenodo.3661161 · Cited by 67
Guo, L., Wang, L., Dang, J., Liu, Z., & Guan, H. (2019) · IEEE Access · doi:10.1109/access.2019.2921390 · Cited by 64
Hybrid Gaussian-cubic radial basis functions for scattered data interpolation
Mishra, P. K., Nath, S. K., Sen, M. K., & Fasshauer, G. E. (2018) · Computational Geosciences · doi:10.1007/s10596-018-9747-3 · Cited by 56
Memento: An Emotion-driven Lifelogging System with Wearables
Jiang, S., Li, Z., Zhou, P., & Li, M. (2019) · ACM Transactions on Sensor Networks · doi:10.1145/3281630 · Cited by 53
Liu, B., Qin, H., Gong, Y., Ge, W., Xia, M., & Shi, L. (2018) · IEEE Access · doi:10.1109/access.2018.2870273 · Cited by 45
Wu, L., Zhang, S., Jian, M., et al. (2019) · IEEE Access · doi:10.1109/access.2019.2922038 · Cited by 45
Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638345 · Cited by 40
Spiking Echo State Convolutional Neural Network for Robust Time Series Classification
Zhang, A., Zhu, W., & Li, J. (2018) · IEEE Access · doi:10.1109/access.2018.2887354 · Cited by 35
Robust Features for Emotion Recognition from Speech by Using Gaussian Mixture Model Classification
Navyasri, M., RajeswarRao, R., DaveeduRaju, A., & Ramakrishnamurthy, M. (2017) · Smart innovation, systems and technologies · doi:10.1007/978-3-319-63645-0_50 · Cited by 32
Convolutional Bidirectional Long Short-Term Memory for Deception Detection With Acoustic Features
Xie, Y., Liang, R., Tao, H., Zhu, Y., & Zhao, L. (2018) · IEEE Access · doi:10.1109/access.2018.2882917 · Cited by 32
Ocquaye, E. N. N., Mao, Q., Song, H., Xu, G., & Xue, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2924597 · Cited by 22
Boosting Discrimination Information Based Document Clustering Using Consensus and Classification
Sheri, A. M., Rafique, M. A., Hassan, M. T., Junejo, K. N., & Jeon, M. (2019) · IEEE Access · doi:10.1109/access.2019.2923462 · Cited by 8
Yu, Z., Chen, W., Guo, X. C., et al. (2017) · IEEE Access · doi:10.1109/access.2017.2747901 · Cited by 7
Fresnedo, Ó., Suárez-Casal, P., & Castedo, L. (2019) · IEEE Access · doi:10.1109/access.2019.2909318 · Cited by 5
Research works printed but not linked: 1
- 10.1109/JSTSP.2017.2764438 · doi:10.1109/jstsp.2017.2764438 · Paper page
Cited by
Khare, S. K., Blanes-Vidal, V., Nadimi, E. S., & Acharya, U. R. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102019 · Cited by 469
A Comprehensive Review of Speech Emotion Recognition Systems
Wani, T. M., Gunawan, T. S., Qadri, S. A. A., Kartiwi, M., & Ambikairajah, E. (2021) · IEEE Access · doi:10.1109/access.2021.3068045 · Cited by 376
Automated emotion recognition: Current trends and future perspectives
Maithri, M., Raghavendra, U., Gudigar, A., Samanth, J., Barua, P. D., Murugappan, M., Chakole, Y., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106646 · Cited by 172
Multimodal Emotion Recognition Based on Facial Expressions, Speech, and EEG
Pan, J., Fang, W., Zhang, Z., Chen, B., Zhang, Z., & Wang, S. (2024) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2023.3240280 · Cited by 121
An Overview of Emotion in Artificial Intelligence
Assunção, G., Patrão, B., Castelo-Branco, M., & Menezes, P. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3159614 · Cited by 103
Metadata
- Authors
- Mustaqeem, Sajjad, M., & Kwon, S.
- Affiliations
- Islamia College University (PK)
- Sejong University (KR)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 79861-79875
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3022013598
- Cited by
- 402
- References
- 64
- Citations by year
- 2026: 30 · 2025: 58 · 2024: 85 · 2023: 91 · 2022: 68 · 2021: 54 · 2020: 16
- FWCI
- 55.30
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Institute for Information and Communications Technology Planning and Evaluation (IITP) funded by the Korea Government through the Ministry of Science and ICT (MSIT) (2017-0-00189)
- 2020 Faculty Research Fund of Sejong University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech emotion recognition, clustering, bidirectional LSTM, temporal information, computational complexity reduction, IEMOCAP, EMO-DB, RAVDESS
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.