Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index; none of the 3 sources its claims cite is checked for support yet.

Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling

Open access

Abstract

In this paper, we apply a context-sensitive technique for multimodal emotion recognition based on feature-level fusion of acoustic and visual cues. We use bidirectional Long Short-Term Memory (BLSTM) networks which, unlike most other emotion recognition approaches, exploit long-range contextual information for modeling the evolution of emotion within a conversation. We focus on recognizing dimensional emotional labels, which enables us to classify both prototypical and nonprototypical emotional expressions contained in a large audiovisual database. Subject-independent experiments on various classification tasks reveal that the BLSTM network approach generally prevails over standard classification techniques such as Hidden Markov Models or Support Vector Machines, and achieves F1-measures of the order of 72 %, 65 %, and 55 % for the discrimination of three clusters in emotional space and the distinction between three levels of valence and activation, respectively. Index Terms: emotion recognition, multimodality, long shortterm memory, hidden markov models, context modeling

Overview

CoChat Summary

TL;DR

The paper presents a context-sensitive multimodal emotion recognition framework that fuses acoustic and facial features and uses bidirectional LSTM networks to model long-range context for dimensional valence/activation and emotional-space clusters. In subject-independent experiments on IEMOCAP, BLSTM generally outperformed HMM and SVM, achieving F1-measures of 72.34% for three clusters, 65.18% for three valence levels, and 55.18% for three activation levels.

Key findings

  • BLSTM achieved the best F1-measure for three-cluster discrimination (72.34%), four clusters (62.78%), and five clusters (54.84%); for three valence levels it reached 65.18% F1, versus 63.66% for unidirectional LSTM.
  • For three activation levels, HMM+LM achieved the highest F1 (56.89%), while BLSTM reached 55.18%; the paper attributes this to class imbalance, since most utterances were labeled medium activation and training data for the remaining two activation classes was insufficient.
  • Feature selection showed modality-specific patterns: for valence, on average 84% of selected features were facial, whereas for activation only 44% were facial; unimodal HMM results confirmed facial features were better for valence and acoustic features better for activation.
  • Context modeling helped: HMM+LM improved over HMM for valence and activation, and BLSTM generally prevailed over HMM and SVM, except for the five-cluster task where LSTM and BLSTM showed no significant difference.

Methods

  • Design: feature-level fusion of acoustic and visual cues; bidirectional Long Short-Term Memory networks for long-range contextual modeling, compared against fully-connected 3-state HMMs with a first-order language model and polynomial-kernel SVMs trained with SMO.
  • Data: IEMOCAP, approximately 12 hours of audio-visual data from five mixed-gender pairs of actors (ten subjects total), with utterances annotated for valence, activation, and dominance on integer scales from one to five; experiments used cyclic leave-one-speaker-out cross validation, with mean test and training utterances of 498 ± 60 and 4475 ± 61 across folds.
  • Features and analysis: 46 facial markers reduced via Principal Feature Analysis to 30 features plus first derivatives (60-dim), normalized per speaker; audio features included 12 MFCCs, 27 Mel Frequency Band coefficients, pitch, and energy plus first derivatives, extracted at 25 ms frame rate with 50 ms window; utterance-level statistical functionals were used for LSTM and SVM, with correlation-based feature subset selection yielding 66–224 features; BLSTM used 128 memory blocks per direction, learning rate 10−5, and zero-mean Gaussian input noise with standard deviation 0.6.

Limitations

  • Stated: activation classification suffers from class imbalance, as the majority of utterances are labeled medium activation and the remaining two activation classes have insufficient training data.
  • Stated future work: dynamic modeling of low-level features using a multimodal LSTM framework and inclusion of linguistic information were not yet examined.
  • Evident from the text: the experiments use an acted IEMOCAP corpus with scripted and improvised scenarios, so real-life generalization remains untested despite the subject-independent setup.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.21437/interspeech.2010-646

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 0 Not valid · 3 Not checked · 17 Own results and interpretations

7 more claims

Create a free account to see all 20 claims and every source verdict.

See all claims — free account
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 5
  • It may state two separate claims as one. 2

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 14 references

Research works printed but not linked: 1

  1. [14][14] S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland, The HTK Book. Entropic Cambridge Research Laboratory, Cambridge, England, 2006. 2365

Cited by

See all 208

Metadata

Authors
Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S.
Affiliations
  • Technical University of Munich (DE)
  • University of Southern California (US)
Venue
Interspeech 2010
Publisher
ISCA
Published
2010-09-26
Pages
2362-2365
ISSN
2308-457X (ISSN-L)
OpenAlex
W2149940198
Cited by
201
References
14
Citations by year
2026: 3 · 2025: 7 · 2024: 12 · 2023: 13 · 2022: 20 · 2021: 17 · 2020: 18 · 2019: 20 · 2018: 19 · 2017: 10 · 2016: 8 · 2015: 14 · 2014: 4 · 2013: 10 · 2012: 17
FWCI
21.93
Type
conference paper
Language
en
Open access
Licence
other-oa
Locations
  • Interspeech (USB) (publishedVersion)
  • mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
  • OPUS (Augsburg University) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
multimodal emotion recognition, bidirectional LSTM, feature-level fusion

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling. https://app.cochat.ai/papers/10.21437/interspeech.2010-646
The paper
Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S. (2010). Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling. Interspeech 2010. https://doi.org/10.21437/interspeech.2010-646

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.