The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index; none of the 3 sources its claims cite is checked for support yet.
Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling
Open accessAbstract
In this paper, we apply a context-sensitive technique for multimodal emotion recognition based on feature-level fusion of acoustic and visual cues. We use bidirectional Long Short-Term Memory (BLSTM) networks which, unlike most other emotion recognition approaches, exploit long-range contextual information for modeling the evolution of emotion within a conversation. We focus on recognizing dimensional emotional labels, which enables us to classify both prototypical and nonprototypical emotional expressions contained in a large audiovisual database. Subject-independent experiments on various classification tasks reveal that the BLSTM network approach generally prevails over standard classification techniques such as Hidden Markov Models or Support Vector Machines, and achieves F1-measures of the order of 72 %, 65 %, and 55 % for the discrimination of three clusters in emotional space and the distinction between three levels of valence and activation, respectively. Index Terms: emotion recognition, multimodality, long shortterm memory, hidden markov models, context modeling
Overview
CoChat Summary
TL;DR
The paper presents a context-sensitive multimodal emotion recognition framework that fuses acoustic and facial features and uses bidirectional LSTM networks to model long-range context for dimensional valence/activation and emotional-space clusters. In subject-independent experiments on IEMOCAP, BLSTM generally outperformed HMM and SVM, achieving F1-measures of 72.34% for three clusters, 65.18% for three valence levels, and 55.18% for three activation levels.
Key findings
- BLSTM achieved the best F1-measure for three-cluster discrimination (72.34%), four clusters (62.78%), and five clusters (54.84%); for three valence levels it reached 65.18% F1, versus 63.66% for unidirectional LSTM.
- For three activation levels, HMM+LM achieved the highest F1 (56.89%), while BLSTM reached 55.18%; the paper attributes this to class imbalance, since most utterances were labeled medium activation and training data for the remaining two activation classes was insufficient.
- Feature selection showed modality-specific patterns: for valence, on average 84% of selected features were facial, whereas for activation only 44% were facial; unimodal HMM results confirmed facial features were better for valence and acoustic features better for activation.
- Context modeling helped: HMM+LM improved over HMM for valence and activation, and BLSTM generally prevailed over HMM and SVM, except for the five-cluster task where LSTM and BLSTM showed no significant difference.
Methods
- Design: feature-level fusion of acoustic and visual cues; bidirectional Long Short-Term Memory networks for long-range contextual modeling, compared against fully-connected 3-state HMMs with a first-order language model and polynomial-kernel SVMs trained with SMO.
- Data: IEMOCAP, approximately 12 hours of audio-visual data from five mixed-gender pairs of actors (ten subjects total), with utterances annotated for valence, activation, and dominance on integer scales from one to five; experiments used cyclic leave-one-speaker-out cross validation, with mean test and training utterances of 498 ± 60 and 4475 ± 61 across folds.
- Features and analysis: 46 facial markers reduced via Principal Feature Analysis to 30 features plus first derivatives (60-dim), normalized per speaker; audio features included 12 MFCCs, 27 Mel Frequency Band coefficients, pitch, and energy plus first derivatives, extracted at 25 ms frame rate with 50 ms window; utterance-level statistical functionals were used for LSTM and SVM, with correlation-based feature subset selection yielding 66–224 features; BLSTM used 128 memory blocks per direction, learning rate 10−5, and zero-mean Gaussian input noise with standard deviation 0.6.
Limitations
- Stated: activation classification suffers from class imbalance, as the majority of utterances are labeled medium activation and the remaining two activation classes have insufficient training data.
- Stated future work: dynamic modeling of low-level features using a multimodal LSTM framework and inclusion of linguistic information were not yet examined.
- Evident from the text: the experiments use an acted IEMOCAP corpus with scripted and improvised scenarios, so real-life generalization remains untested despite the subject-independent setup.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 3 Not checked · 17 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such data are a relatively large portion of the database (approximately 17 % of the total utterances).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BLSTM v+f 67.31 52.53 58.46 55.18”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generally, multimodal classification outperforms unimodal AER.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For all cluster prediction tasks, we observe similar trends: LSTM modeling prevails over HMM and SVM classification”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BLSTM network approach generally prevails over standard classification techniques”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
7 more claims
Create a free account to see all 20 claims and every source verdict.
- The quote only partly states this claim. 1
- Probably not a substantial claim. 5
- It may state two separate claims as one. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Experimental Results and Discussion
“For four and five clusters they achieve F1-measures of 62.78 % and 55.00 % respectively.”
§ 4. Experimental Results and Discussion
“LSTM v+f 56.19 53.89 56.25 55.00 BLSTM v+f 56.31 53.76 56.13 54.84”
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Self-citationSupervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Mehrabian, A. (2017) · communication theory · doi:10.4324/9781315080918-15 · Cited by 879
Active and dynamic information fusion for facial expression understanding from image sequences
Zhang, Y., & Ji, Q. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.93 · Cited by 461
Mechrabian, A. (1968) · Medical Entomology and Zoology · Cited by 459
OpenEAR — Introducing the munich open-source emotion and affect recognition toolkit
Eyben, F., Wollmer, M., & Schuller, B. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349350 · Cited by 418 · cited 1×
“All functionals were calculated using our openEAR toolkit [12].”
MethodSelf-citationFeature selection using principal feature analysis
Lu, Y., Cohen, I., Zhou, X. S., & Tian, Q. (2007) · Proceedings of the 15th ACM international conference on Multimedia · doi:10.1145/1291233.1291297 · Cited by 341
Wollmer, M., Schuller, B., Eyben, F., & Rigoll, G. (2010) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2010.2057200 · Cited by 165
Self-citationMetallinou, A., Lee, S., & Narayanan, S. (2010) · 2010 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2010.5494890 · Cited by 89
Self-citationModeling mutual influence of interlocutor emotion states in dyadic spoken interactions
Lee, C.-C., Busso, C., Lee, S., & Narayanan, S. S. (2009) · Interspeech (USB) · doi:10.21437/interspeech.2009-480 · Cited by 81
Self-citationVisual emotion recognition using compact facial representations and viseme information
Metallinou, A., Busso, C., Lee, S., & Narayanan, S. S. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5494893 · Cited by 54
Self-citationWöllmer, M., Eyben, F., Schuller, B. W., Douglas‐Cowie, E., & Cowie, R. (2009) · Interspeech (USB) · doi:10.21437/interspeech.2009-474 · Cited by 53
Self-citation
Research works printed but not linked: 1
- [14][14] S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland, The HTK Book. Entropic Cambridge Research Laboratory, Cambridge, England, 2006. 2365
Cited by
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
DialogueRNN: An Attentive RNN for Emotion Detection in Conversations
Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., & Cambria, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016818 · Cited by 860
Speech Emotion Recognition Using Deep Learning Techniques: A Review
Khalil, R. A., Jones, E., Babar, M. I., Jan, T., Zafar, M. H., & Alhussain, T. (2019) · IEEE Access · doi:10.1109/access.2019.2936124 · Cited by 702
“Deep learning is derived from the family of machine learning, which is a broader learning technique for data representation such as emotions [57], [58].”
BackgroundA Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
“…was considerable variation in model-level fusion methods, but bidirectional long short-term memories [Eyben et al. 2010; Metallinou et al. 2012; W¨ollmer et al. 2010, 2013a], various HMM-based approaches (error-weighted semicoupled HMMs [Lin et al. (e.g., Jiang et al. [2011], Paleari et al.…”
Background
Metadata
- Authors
- Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S.
- Affiliations
- Technical University of Munich (DE)
- University of Southern California (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 2362-2365
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2149940198
- Semantic Scholar
- 923c20f9eacd1d76d95f5528654c9578906aa09b
- Cited by
- 201
- References
- 14
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 12 · 2023: 13 · 2022: 20 · 2021: 17 · 2020: 18 · 2019: 20 · 2018: 19 · 2017: 10 · 2016: 8 · 2015: 14 · 2014: 4 · 2013: 10 · 2012: 17
- FWCI
- 21.93
- Type
- conference paper
- Language
- en
- Open access
- green
- mediatum.ub.tum.de
- Licence
- other-oa
- Locations
- Interspeech (USB) (publishedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
- OPUS (Augsburg University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- multimodal emotion recognition, bidirectional LSTM, feature-level fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.