Speech Emotion Recognition with Dual-Sequence LSTM Architecture
Open accessThe DOI matches its Crossref record; 19 of the 20 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
Speech Emotion Recognition (SER) has emerged as a critical component of the next generation of human-machine interfacing technologies. In this work, we propose a new dual-level model that predicts emotions based on both MFCC features and mel-spectrograms produced from raw audio signals. Each utterance is preprocessed into MFCC features and two mel-spectrograms at different time-frequency resolutions. A standard LSTM processes the MFCC features, while a novel LSTM architecture, denoted as Dual-Sequence LSTM (DS-LSTM), processes the two mel-spectrograms simultaneously. The outputs are later averaged to produce a final classification of the utterance. Our proposed model achieves, on average, a weighted accuracy of 72.7% and an unweighted accuracy of 73.3% - a 6% improvement over current state-of-the-art unimodal models - and is comparable with multimodal models that leverage textual information as well as audio signals.
Overview
CoChat Summary
TL;DR
The paper proposes a dual-level speech emotion recognition model that combines MFCC features with two mel-spectrograms of different time-frequency resolutions, using a standard LSTM and a novel Dual-Sequence LSTM (DS-LSTM). On IEMOCAP, it reports 72.7% mean weighted accuracy and 73.3% mean unweighted accuracy, outperforming unimodal baselines and matching a multimodal model that uses text.
Key findings
- The proposed dual-level model \(M_{LSTM}+M_{DS-LSTM}\) achieves 72.7% ± 0.7 mean weighted accuracy and 73.3% ± 0.8 mean unweighted accuracy under 5-fold cross-validation.
- It outperforms all baseline models by at least 4.2% in mean weighted accuracy and at least 4.0% in mean unweighted accuracy, and outperforms state-of-the-art unimodal SER models by at least 6.8% in mean weighted accuracy and 5.9% in mean unweighted accuracy.
- Replacing two separate LSTMs with the proposed DS-LSTM increases weighted accuracy by 5% while reducing parameters by 25% compared with Base 4.
- The model is comparable in mean weighted accuracy to a multimodal model that uses textual information, despite using only audio signals.
Methods
- Dataset: IEMOCAP, using audio-only data from scripted and improvised conversations, restricted to four emotions—happy, neutral, angry, and sad—for 5531 utterances (happy 29.5%, neutral 30.8%, angry 19.9%, sad 19.5%).
- Preprocessing: openSMILE extracted 39-dimensional MFCC features (13 MFCCs plus delta and acceleration coefficients) from 25 ms frames; two mel-spectrograms per utterance were computed with 256 and 512 FFT points, then nearest-neighbor interpolation aligned variable-length spectrograms to the median number of time steps.
- Model and evaluation: a dual-level architecture averages outputs from a two-layer single-directional LSTM over MFCC features and a DS-LSTM over the two aligned mel-spectrogram sequences; models were evaluated with 5-fold cross-validation, using an Adam optimizer with learning rate 0.0001 and 200 hidden nodes.
Limitations
- The paper does not state an explicit limitations section.
- Evident from the text: evaluation is limited to the IEMOCAP dataset and only four emotion classes (happy, neutral, angry, sad), using audio-only data.
- Evident from the text: the comparison with multimodal models is limited to mean weighted accuracy against one cited model that uses textual information; the paper does not report a full multimodal comparison.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 1 Not valid · 6 Not checked · 18 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our proposed model achieves, on average, a weighted accuracy of 72.7% and an unweighted accuracy of 73.3%-a 6% improvement over current state-of-the-art unimodal models-and is comparable with multimodal models that leverage textual information as well as audio signals.”
- Also stated in: § 3.3. Results and Analysis · plain
“| Proposed M LSTM + M DS-LSTM | 72.7 ± 0.7 | 73.3 ± 0.8 |”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.3. Results and Analysis · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirical experiments suggest that the forget gate, two input gates, and output gate should incorporate the maximum amount of information, which is the concatenation of x t , y t , and h t -1 .”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More importantly, we observe that the proposed M DS-LSTM significantly outperforms Base 1 ∼ 4.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“and is comparable with multimodal models that leverage textual information as well as audio signals.”
- Also stated in: § 3.3. Results and Analysis · plain
“we see that our proposed model achieves comparable performance with [20] in mean weighted accuracy.”
What the sources say
- Not checked yetcites [20] [20] S. Yoon, S. Byun, and K. Jung, 'Multimodal speech emotion recognition using audio and text,' CoRR , vol. abs/1810.04635, 2018.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.3. Results and Analysis · plain
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“therefore, Base 4 is also slightly worse than Base 1.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
7 more claims
Create a free account to see all 25 claims and to verify the 12 references still open.
- It may state two separate claims as one. 10
- Probably not a substantial claim. 3
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“neglecting to model the emotion-related correlations within frequency domain during the time-frequency joint learning”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [13]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Furthermore, the system struggles to generalize in the face of hybrid or evolving attack vectors—such as spoofing-DoS combinations—resulting in higher false positive rates and reduced confidence in classification 19,61.”
From A Hybrid Deep Learning Model for Detection and Mitigation of DDoS Attacks in VANETs (2025) · cites this paper as [61]
Experimental — accuracy not yet measuredIn the citing paper
“recent spectrogram-based works mainly focus on modeling the long-term dependency in time domain”
From Speech Emotion Recognition via an Attentive Time–Frequency Neural Network (2023) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 2.2. Preprocessing
“For extracting MFCC features, we used the openSMILE toolkit [12], a software that automatically extracts features from an audio signal.”
§ 6. REFERENCES
“[12] Alex Graves and J¨ urgen Schmidhuber, 'Offline handwriting recognition with multidimensional recurrent neural networks,' pp. 545-552, 2009.”
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“pectrograms contain complementary information—namely, one has a better resolution along the time axis and the other has a better resolution along the frequency axis—we propose a novel variant of LSTM [9], denoted as Dual-Sequence LSTM (DS-LSTM), that can process two sequences of data simultaneously. Third, we propose…”
MethodIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 2×
“We used the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset [1] in this work, a benchmark dataset containing about 12 hours of audio and video data, as well as text transcriptions.”
MethodUncertainty Principles and Signal Recovery
Donoho, D. L., & Stark, P. B. (1989) · SIAM Journal on Applied Mathematics · doi:10.1137/0149053 · Cited by 1,054 · cited 1×
“Second, inspired by the time-frequency trade-off [9], from each utterance we calculate two mel-spectrograms of different timefrequency resolutions instead of just one.”
MethodSpeech emotion recognition using deep 1D & 2D CNN LSTM networks
Zhao, J., Mao, X., & Chen, L. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2018.08.035 · Cited by 1,044 · cited 1×
“Recently, researchers have begun transforming raw audio signals into spectrograms or mel-spectrograms [3, 4], which contain low level information and can be converted back to raw audio.”
BackgroundAutomatic speech emotion recognition using recurrent neural networks with local attention
Mirsamadi, S., Barsoum, E., & Zhang, C. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952552 · Cited by 770 · cited 2×
“Traditionally, unimodal models extract high level information from raw audio signals, such as MFCC features, and then pass the output through a recurrent neural network [2].”
BackgroundMethodOffline Arabic Handwriting Recognition with Multidimensional Recurrent Neural Networks
Graves, A. (2012) · Guide to OCR for Arabic Scripts · doi:10.1007/978-1-4471-4072-6_12 · Cited by 684
Efficient emotion recognition from speech using deep learning on spectrograms
Satt, A., Rozenberg, S., & Hoory, R. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-200 · Cited by 422 · cited 1×
“Recently, researchers have begun transforming raw audio signals into spectrograms or mel-spectrograms [3, 4], which contain low level information and can be converted back to raw audio.”
BackgroundRecurrent Batch NormalizationPreprint
Cooijmans, T., Ballas, N., Laurent, C., Gülçehre, Ç., & Courville, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.09025 · Cited by 422 · cited 2×
“1 and 0, respectively, as suggested by the original paper [16].”
BackgroundMultimodal Speech Emotion Recognition Using Audio and Text
Yoon, S., Byun, S., & Jung, K. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639583 · Cited by 351
Multi-dimensional Recurrent Neural Networks
Graves, A., Fernández, S., & Schmidhuber, J. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74690-4_56 · Cited by 303 · cited 2×
“It should be noted that previous research in multi-dimensional LSTM (MD-LSTM) [11, 12, 13], especially in ASR [14, 15], focused on adapting the LSTM to 6474978-1-5090-6631-5/20/$31.00 ©2020 IEEE ICASSP 2020 a multi-dimensional data format.”
BackgroundInfluential
Research works printed but not linked: 1
- choi s h bae, 2016, acoustic scene classification using parallel combination of lstm and cnn, Proceedings of the Detection and Classification of Acoustic Scenes and Events 2016 Workshop (DCASE2016), 11
Cited by
A systematic literature review of speech emotion recognition approaches
Singh, Y. B., & Goel, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.04.028 · Cited by 174
Speech Emotion Recognition Using Self-Supervised Features
Morais, E. D. S., Hoory, R., Zhu, W., Gat, I., Damasceno, M., & Aronowitz, H. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747870 · Cited by 203
Ai, W., Shou, Y., Meng, T., & Li, K. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3367940 · Cited by 174
“In the RNN-based multimodal emotion recognition research, Wang et al. [14] conducted dual-sequence LSTM (DS-LSTM), which uses a dual-stream LSTM to extract contextual features in the Mei-Frequency map simultaneously.”
Ye, J., Wen, X.-C., Wei, Y., Xu, Y. N., Liu, K., & Shan, H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096370 · Cited by 190
“[7] proposed a dual-level LSTM to harness temporal information from different time-frequency resolutions.”
BackgroundMethodSpeech Emotion Recognition with Multi-Task Learning
Cai, X., Yuan, J., Zheng, R., Huang, L., & Church, K. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1852 · Cited by 157
“…acc Wu et al. [37] 10-fold 72.73% Sajjad et al. [13] 5-fold 72.25% Lu et al. [35] 10-fold 72.6% Liu et al. [38] 5-fold 70.78% Wang et al. [39] 5-fold 73.3% Pappagari et al. [40] 5-fold 70.30% Peng et al. [14] 5-fold 62.6% ours 10-fold 78.15% Clearly, our proposed model significantly…”
BackgroundInfluential
Metadata
- Authors
- Wang, J., Xue, M., Culhane, R., Diao, E., Ding, J., & Tarokh, V.
- Affiliations
- Duke University (US)
- University of Minnesota (US)
- Venue
- ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2020-05
- Pages
- 6474-6478
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W3015267357
- Cited by
- 162
- References
- 20
- Citations by year
- 2026: 7 · 2025: 30 · 2024: 35 · 2023: 36 · 2022: 26 · 2021: 20 · 2020: 8
- FWCI
- 21.90
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- speech emotion recognition, Mel-frequency cepstral coefficients, Mel spectrogram, time-frequency resolution, human-machine interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Speech Emotion Recognition with Dual-Sequence LSTM Architecture. https://app.cochat.ai/papers/10.1109/icassp40776.2020.9054629
- The paper
- Wang, J., Xue, M., Culhane, R., Diao, E., Ding, J., & Tarokh, V. (2020). Speech Emotion Recognition with Dual-Sequence LSTM Architecture. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10.1109/icassp40776.2020.9054629
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.