CoChat

Multimodal Emotion Recognition With Temporal and Semantic Consistency

Chen, Bingzhi, Cao, Qi, Hou, Mixiao, Zhang, Zheng, Lu, Guangming, Zhang, David · 2021 · IEEE/ACM Transactions on Audio, Speech, and Language Processing

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.

Abstract

Automated multimodal emotion recognition has become an emerging but challenging research topic in the fields of affective learning and sentiment analysis. The existing works mainly focus on developing multimodal fusion strategies to incorporate different emotion-related features. However, they fail to explore the inherent contextual consistency to reconcile the emotional information across modalities. In this paper, we propose a novel Time and Semantic Interaction Network (TSIN), which concurrently incorporates the advantages of temporal and semantic consistency into the multimodal emotion recognition task. Specifically, a well-designed Speech and Text Embedding (STE) module is devoted to formulating the initial embedding spaces by respectively building the modality-specific representations of speech and text. Instead of separately learning or directly fusing the acoustic and textual features, we propose a well-defined Time and Semantic Interaction (TSI) module to conduct the emotional parsing and sentiment refining by performing the fine-grained temporal alignment and cross-modal semantic interaction. Benefitting from temporal and semantic consistency constraints, both speech-text embeddings can be interactively optimized and fine-tuned in the learning process. In this way, the learnt acoustics and textual features can jointly and efficiently predict the final emotional state. Extensive experiments on the IEMOCAP dataset demonstrate the superiorities of our TSIN framework in comparison with state-of-the-art baselines.

Overview

Integrity signals

Registry

Crossref · 10.1109/taslp.2021.3129331

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 50 references

Cited by

See all 84

Metadata

Authors
Chen, B., Cao, Q., Hou, M., Zhang, Z., Lu, G., & Zhang, D.
Affiliations
  • South China Normal University (CN)
  • Harbin Institute of Technology (CN)
  • Chinese University of Hong Kong, Shenzhen (CN)
Venue
IEEE/ACM Transactions on Audio, Speech, and Language Processing
Published
2021
Volume
29
Pages
3592-3603
ISSN
  • 2329-9290 (ISSN-L)
  • 2329-9304
OpenAlex
W3214882941
Cited by
71
References
50
Citations by year
2026: 18 · 2025: 16 · 2024: 16 · 2023: 15 · 2022: 6
FWCI
8.93
Type
journal article
Language
en
Open access
closed
Locations
IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
Funders
  • National Key Research and Development Program of China (2018AAA0100100)
  • National Natural Science Foundation of China (62176077, 62002085)
  • Basic and Applied Basic Research Foundation of Guangdong Province (2019Bl515120055)
  • Shenzhen Key Technical Project (2020N046)
  • Shenzhen Fundamental Research (JCYJ20210324132210025, GXWD20201230155427003-20200824103320001)
  • Open Project Fund (AC01202005018)
  • Shenzhen Institute of Artificial Intelligence, and Robotics for Society
  • Medical Biometrics Perception and Analysis Engineering Laboratory, Shenzhen, China
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
multimodal emotion recognition, temporal coherence, semantic consistency, cross-modal interaction, temporal alignment, IEMOCAP, affective computing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multimodal Emotion Recognition With Temporal and Semantic Consistency. https://app.cochat.ai/papers/10.1109/taslp.2021.3129331
The paper
Chen, B., Cao, Q., Hou, M., Zhang, Z., Lu, G., & Zhang, D. (2021). Multimodal Emotion Recognition With Temporal and Semantic Consistency. IEEE/ACM Transactions on Audio, Speech, and Language Processing. https://doi.org/10.1109/taslp.2021.3129331

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.