DNN-based Emotion Recognition Based on Bottleneck Acoustic Features and Lexical Features
Abstract
In this paper, we propose a novel emotion recognition method to reflect affect salient information using acoustic and lexical features. The acoustic features are extracted from the speech signal by applying statistical functionals of emotionally high-level features derived from Deep Neural Network (DNN). These acoustic features are early fused with two types of lexical features extracted from the text transcription of the speech signal, which are the distributed representation and affective lexicon-based dimensions. The fused features are fed to another DNN for utterance-level emotion classification. Experimental results on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) multimodal dataset showed 75.5% in unweighted accuracy recall, which outperformed the best results reported previously in the multimodal emotion recognition using acoustic and lexical features.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Term-Weighting Approaches in Automatic Text Retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 11,421 · see citing works
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · see citing works
Survey on speech emotion recognition: Features, classification schemes, and databases
Ayadi, M. M. H. E., Kamel, M. S., & Karray, F. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,089 · see citing works
Norms of valence, arousal, and dominance for 13,915 English lemmas
Warriner, A. B., Kuperman, V., & Brysbaert, M. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0314-x · Cited by 1,959 · see citing works
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · see citing works
Schuller, B. W., Batliner, A. M., Steidl, S., & Seppi, D. (2011) · Speech Communication · doi:10.1016/j.specom.2011.01.011 · Cited by 765 · see citing works
Schuller, B. W., Steidl, S., Batliner, A. M., Vinciarelli, A., Scherer, K. R., Ringeval, F., Chétouani, M., Weninger, F. J., Eyben, F., Marchi, E., et al. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-56 · Cited by 732 · see citing works
Deep neural networks for acoustic emotion recognition: Raising the benchmarks
Stuhlsatz, A., Meyer, C., Eyben, F., Zielke, T., Meier, G., & Schuller, B. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947651 · Cited by 288 · see citing works
Improved bottleneck features using pretrained deep neural networks
Yu, D., & Seltzer, M. L. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-91 · Cited by 276 · see citing works
Speech emotion recognition with acoustic and lexical features
Jin, Q., Li, C., Chen, S., & Wu, H. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178872 · Cited by 175 · see citing works
Speech Emotion Recognition Using Spectrogram & Phoneme Embedding
Yenigalla, P., Kumar, A., Tripathi, S., Singh, C., Kar, S., & Vepa, J. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1811 · Cited by 165 · see citing works
Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment
Gu, Y., Yang, K., Fu, S., Chen, S., Li, X., & Marsic, I. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1207 · Cited by 160 · see citing works
Lyric-Based Song Emotion Detection With Affective Lexicon And Fuzzy Clustering Method.
Hu, Y., Chen, X., & Yang, D. (2009) · doi:10.5281/zenodo.1417982 · Cited by 104 · see citing works
Deep Neural Networks for Emotion Recognition Combining Audio and Transcripts
Cho, J., Pappagari, R. R., Kulkarni, P., Villalba, J., Carmiel, Y., & Dehak, N. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2466 · Cited by 100 · see citing works
i‐vector representation based on bottleneck features for language identification
Song, Y., Jiang, B., Bao, Y., et al. (2013) · Electronics Letters · doi:10.1049/el.2013.1721 · Cited by 88 · see citing works
Deep Mul Timodal Learning for Emotion Recognition in Spoken Language
Gu, Y., Chen, S., & Marsic, I. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462440 · Cited by 54 · see citing works
Emotion recognition using acoustic and lexical features
Rozgić, V., Ananthakrishnan, S., Saleem, S., Kumar, R., Vembu, A. N., & Prasad, R. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-118 · Cited by 50 · see citing works
Salience based lexical features for emotion recognition
Gamage, K. W., Sethu, V., & Ambikairajah, E. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953274 · Cited by 20 · see citing works
Deep Neural Network Bottleneck Features for Acoustic Event Recognition
Mun, S., Shon, S., Kim, W., & Ko, H. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-1112 · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Speech Emotion Recognition with Dual-Sequence LSTM Architecture
Wang, J., Xue, M., Culhane, R., Diao, E., Ding, J., & Tarokh, V. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054629 · Cited by 162 · see citing works
Multimodal Emotion Recognition With Temporal and Semantic Consistency
Chen, B., Cao, Q., Hou, M., Zhang, Z., Lu, G., & Zhang, D. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3129331 · Cited by 71 · see citing works
- Authors
- Kim, E., & Shin, J. W.
- Affiliations
- Gwangju Institute of Science and Technology (KR)
- ORCID
- Shin, Jong Won
- Venue
- ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 6720-6724
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2937154351
- Cited by
- 54
- References
- 21
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 6 · 2023: 9 · 2022: 10 · 2021: 15 · 2020: 9 · 2019: 1
- FWCI
- 11.02
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- emotion recognition, multimodal emotion recognition, acoustic features, lexical features, emotion lexicon, feature fusion, IEMOCAP, speech emotion recognition, affective features
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
