The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Multimodal Speech Emotion Recognition Using Audio and Text
Open accessAbstract
Speech emotion recognition is a challenging task, and extensive reliance has been placed on models that use audio features in building well-performing classifiers. In this paper, we propose a novel deep dual recurrent encoder model that utilizes text data and audio signals simultaneously to obtain a better understanding of speech data. As emotional dialogue is composed of sound and spoken content, our model encodes the information from audio and text sequences using dual recurrent neural networks (RNNs) and then combines the information from these sources to predict the emotion class. This architecture analyzes speech data from the signal level to the language level, and it thus utilizes the information within the data more comprehensively than models that focus on audio features. Extensive experiments are conducted to investigate the efficacy and properties of the proposed model. Our proposed model outperforms previous state-of-the-art methods in assigning data to one of four emotion categories (i.e., angry, happy, sad and neutral) when the model is applied to the IEMOCAP dataset, as reflected by accuracies ranging from 68.8% to 71.8%.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
NLTK: the Natural Language Toolkit
Loper, E., & Bird, S. (2002) · Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics - · doi:10.3115/1118108.1118117 · Cited by 3,218
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110
Recent developments in openSMILE, the munich open-source multimedia feature extractor
Eyben, F., Weninger, F., Gross, F., & Schuller, B. (2013) · Proceedings of the 21st ACM international conference on Multimedia · doi:10.1145/2502081.2502224 · Cited by 1,241
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Saxe, A., McClelland, J. L., & Ganguli, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6120 · Cited by 931
Speech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-57 · Cited by 794
Automatic speech emotion recognition using recurrent neural networks with local attention
Mirsamadi, S., Barsoum, E., & Zhang, C. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952552 · Cited by 770
Effective Approaches to Attention-based Neural Machine Translation
Luong, M.-T., Pham, H., & Manning, C. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.04025 · Cited by 655
Hidden Markov model-based speech emotion recognition
Schuller, B., Rigoll, G., & Lang, M. (2003) · 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698) · doi:10.1109/icme.2003.1220939 · Cited by 570
Schuller, B., Rigoll, G., & Lang, M. (2004) · 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.2004.1326051 · Cited by 426
Efficient emotion recognition from speech using deep learning on spectrograms
Satt, A., Rozenberg, S., & Hoory, R. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-200 · Cited by 422
Speech Emotion Recognition from Spectrograms with Deep Convolutional Neural Network
Badshah, A. M., Ahmad, J., Rahim, N., & Baik, S. W. (2017) · 2017 International Conference on Platform Technology and Service (PlatCon) · doi:10.1109/platcon.2017.7883728 · Cited by 413
NLTK: the natural language toolkit
Bird, S., & Loper, E. (2004) · Proceedings of the ACL 2004 on Interactive poster and demonstration sessions - · doi:10.3115/1219044.1219075 · Cited by 403
High-level feature representation using recurrent neural network for speech emotion recognition
Lee, J., & Tashev, I. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-336 · Cited by 388
Emotion recognition using a hierarchical binary decision tree approach
Lee, C.-C., Mower, E., Busso, C., Lee, S., & Narayanan, S. (2011) · Speech Communication · doi:10.1016/j.specom.2011.06.004 · Cited by 355
Neumann, M., & Vu, N. T. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-917 · Cited by 243
Automatic Dialogue Generation with Expressed Emotions
Huang, C., Zai͏̈ane, O. R., Trabelsi, A., & Dziri, N. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2008 · Cited by 193
Speech emotion recognition using Support Vector Machines
Seehapoch, T., & Wongthanavasu, S. (2013) · International Conference on Knowledge and Smart Technology · doi:10.1109/kst.2013.6512793 · Cited by 130
Progressive Neural Networks for Transfer Learning in Emotion Recognition
Gideon, J., Khorram, S., Aldeneh, Z., Dimitriadis, D., & Provost, E. M. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1637 · Cited by 129
Using regional saliency for speech emotion recognition
Aldeneh, Z., & Provost, E. M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952655 · Cited by 126
A first look into a Convolutional Neural Network for speech emotion detection
Bertero, D., & Fung, P. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953131 · Cited by 117
Toward Effective Automatic Recognition Systems of Emotion in Speech
Busso, C., Bulut, M., & Narayanan, S. S. (2013) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195387643.003.0008 · Cited by 90
Audio-based multimedia event detection using deep recurrent neural networks
Wang, Y., Neves, L., & Metze, F. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472176 · Cited by 73
Salience based lexical features for emotion recognition
Gamage, K. W., Sethu, V., & Eliathamby Ambikairajah, A. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953274 · Cited by 23
Kim, J., Englebienne, G., Truong, K. P., et al. (2017) · University of Twente Research Information · doi:10.48550/arxiv.1708.03920 · Cited by 15
Cited by
Enhancements in Immediate Speech Emotion Detection: Harnessing Prosodic and Spectral Characteristics
Shah, Z., Zhiyong, S., & ., A. (2024) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/ijisrt24apr872 · Cited by 1,663
Speech emotion recognition with deep convolutional neural networks
Issa, D., Fatih Demirci, M., & Yazici, A. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2020.101894 · Cited by 619
A Comprehensive Review of Speech Emotion Recognition Systems
Wani, T. M., Gunawan, T. S., Qadri, S. A. A., Kartiwi, M., & Ambikairajah, E. (2021) · IEEE Access · doi:10.1109/access.2021.3068045 · Cited by 376
Speech Emotion Recognition with Dual-Sequence LSTM Architecture
Wang, J., Xue, M., Culhane, R., Diao, E., Ding, J., & Tarokh, V. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054629 · Cited by 162
ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics
Taleb, A., Kirchler, M., Monti, R., & Lippert, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02024 · Cited by 83
Metadata
- Authors
- Yoon, S., Byun, S., & Jung, K.
- Venue
- 2018 IEEE Spoken Language Technology Workshop (SLT)
- Publisher
- IEEE
- Published
- 2018-12
- Pages
- 112-118
- OpenAlex
- W2897191477
- Cited by
- 351
- References
- 30
- FWCI
- 31.57
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.