CoChat

Speech Emotion Recognition with Dual-Sequence LSTM Architecture

Open access
Wang, Jianyou, Xue, Michael, Culhane, Ryan, Diao, Enmao, Ding, Jie, Tarokh, Vahid · 2020 · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

The DOI matches its Crossref record; 19 of the 20 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.

Abstract

Speech Emotion Recognition (SER) has emerged as a critical component of the next generation of human-machine interfacing technologies. In this work, we propose a new dual-level model that predicts emotions based on both MFCC features and mel-spectrograms produced from raw audio signals. Each utterance is preprocessed into MFCC features and two mel-spectrograms at different time-frequency resolutions. A standard LSTM processes the MFCC features, while a novel LSTM architecture, denoted as Dual-Sequence LSTM (DS-LSTM), processes the two mel-spectrograms simultaneously. The outputs are later averaged to produce a final classification of the utterance. Our proposed model achieves, on average, a weighted accuracy of 72.7% and an unweighted accuracy of 73.3% - a 6% improvement over current state-of-the-art unimodal models - and is comparable with multimodal models that leverage textual information as well as audio signals.

Overview

CoChat Summary

TL;DR

The paper proposes a dual-level speech emotion recognition model that combines MFCC features with two mel-spectrograms of different time-frequency resolutions, using a standard LSTM and a novel Dual-Sequence LSTM (DS-LSTM). On IEMOCAP, it reports 72.7% mean weighted accuracy and 73.3% mean unweighted accuracy, outperforming unimodal baselines and matching a multimodal model that uses text.

Key findings

  • The proposed dual-level model \(M_{LSTM}+M_{DS-LSTM}\) achieves 72.7% ± 0.7 mean weighted accuracy and 73.3% ± 0.8 mean unweighted accuracy under 5-fold cross-validation.
  • It outperforms all baseline models by at least 4.2% in mean weighted accuracy and at least 4.0% in mean unweighted accuracy, and outperforms state-of-the-art unimodal SER models by at least 6.8% in mean weighted accuracy and 5.9% in mean unweighted accuracy.
  • Replacing two separate LSTMs with the proposed DS-LSTM increases weighted accuracy by 5% while reducing parameters by 25% compared with Base 4.
  • The model is comparable in mean weighted accuracy to a multimodal model that uses textual information, despite using only audio signals.

Methods

  • Dataset: IEMOCAP, using audio-only data from scripted and improvised conversations, restricted to four emotions—happy, neutral, angry, and sad—for 5531 utterances (happy 29.5%, neutral 30.8%, angry 19.9%, sad 19.5%).
  • Preprocessing: openSMILE extracted 39-dimensional MFCC features (13 MFCCs plus delta and acceleration coefficients) from 25 ms frames; two mel-spectrograms per utterance were computed with 256 and 512 FFT points, then nearest-neighbor interpolation aligned variable-length spectrograms to the median number of time steps.
  • Model and evaluation: a dual-level architecture averages outputs from a two-layer single-directional LSTM over MFCC features and a DS-LSTM over the two aligned mel-spectrogram sequences; models were evaluated with 5-fold cross-validation, using an Adam optimizer with learning rate 0.0001 and 200 hidden nodes.

Limitations

  • The paper does not state an explicit limitations section.
  • Evident from the text: evaluation is limited to the IEMOCAP dataset and only four emotion classes (happy, neutral, angry, sad), using audio-only data.
  • Evident from the text: the comparison with multimodal models is limited to mean weighted accuracy against one cited model that uses textual information; the paper does not report a full multimodal comparison.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/icassp40776.2020.9054629

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

0 Valid · 1 Not valid · 6 Not checked · 18 Own results and interpretations

7 more claims

Create a free account to see all 25 claims and to verify the 12 references still open.

See all claims — free account
  • It may state two separate claims as one. 10
  • Probably not a substantial claim. 3
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 19 references

Research works printed but not linked: 1

  1. choi s h bae, 2016, acoustic scene classification using parallel combination of lstm and cnn, Proceedings of the Detection and Classification of Acoustic Scenes and Events 2016 Workshop (DCASE2016), 11

Cited by

See all 169

Metadata

Authors
Wang, J., Xue, M., Culhane, R., Diao, E., Ding, J., & Tarokh, V.
Affiliations
  • Duke University (US)
  • University of Minnesota (US)
Venue
ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Publisher
IEEE
Published
2020-05
Pages
6474-6478
ISSN
1520-6149 (ISSN-L)
OpenAlex
W3015267357
Cited by
162
References
20
Citations by year
2026: 7 · 2025: 30 · 2024: 35 · 2023: 36 · 2022: 26 · 2021: 20 · 2020: 8
FWCI
21.90
Type
conference paper
Language
en
Open access
Locations
  • IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
Keywords
speech emotion recognition, Mel-frequency cepstral coefficients, Mel spectrogram, time-frequency resolution, human-machine interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Speech Emotion Recognition with Dual-Sequence LSTM Architecture. https://app.cochat.ai/papers/10.1109/icassp40776.2020.9054629
The paper
Wang, J., Xue, M., Culhane, R., Diao, E., Ding, J., & Tarokh, V. (2020). Speech Emotion Recognition with Dual-Sequence LSTM Architecture. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10.1109/icassp40776.2020.9054629

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.