Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 87 of the 98 references it cites were found in a registry or an index.

Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing Modalities

Liu, Rui, Zuo, Haolin, Lian, Zheng, Schuller, Björn W., Li, Haizhou · 2024 · IEEE Transactions on Affective Computing

Abstract

Multimodal emotion recognition (MER) aims to understand the way that humans express their emotions by exploring complementary information across modalities. However, it is hard to guarantee that full-modality data is always available in real-world scenarios. To deal with missing modalities, researchers focused on meaningful joint multimodal representation learning during cross-modal missing modality imagination. However, the cross-modal imagination mechanism is highly susceptible to errors due to the “modality gap” issue, which affects the imagination accuracy, thus, the final recognition performance. To this end, we introduce the concept of a modality-invariant feature into the missing modality imagination network, which contains two key modules: 1) a novel contrastive learning-based module to extract modality-invariant features under full modalities; 2) a robust imagination module based on imagined invariant features to reconstruct missing information under missing conditions. Finally, we incorporate imagined and available modalities for emotion recognition. Experimental results on benchmark datasets demonstrate that our proposed method outperforms existing state-of-the-art strategies. Compared with our previous work, our extended version is more effective on multimodal emotion recognition with missing modalities. The code is released athttps://github.com/ZhuoYulang/CIF-MMIN.

Overview

Integrity signals

Registry

Crossref · 10.1109/taffc.2024.3378570

Cites

87 of 87 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 98 references

Research works printed but not linked: 2

  1. 10.5555/3524938.3525087 · doi:10.5555/3524938.3525087
  2. Maas, Rectifier nonlinearities improve neural network acoustic models

Cited by

See all 96

Metadata

Authors
Liu, R., Zuo, H., Lian, Z., Schuller, B. W., & Li, H.
Affiliations
  • Inner Mongolia University (CN)
  • University of Bremen (DE)
  • Chinese Academy of Sciences (CN)
  • Imperial College London (GB)
  • Shenzhen Research Institute of Big Data (CN)
Venue
IEEE Transactions on Affective Computing
Published
2024-10
Volume
15
Issue
4
Pages
1856-1873
ISSN
  • 1949-3045 (ISSN-L)
  • 2371-9850
OpenAlex
W4392903647
Cited by
96
References
98
Citations by year
2026: 40 · 2025: 48 · 2024: 8
FWCI
39.06
Type
journal article
Language
en
Open access
closed
Locations
  • IEEE Transactions on Affective Computing (publishedVersion)
  • mediaTUM (Technical University of Munich) (submittedVersion)
Funders
  • National Natural Science Foundation of China (62206136, 62201572)
  • Guangdong Provincial Key Laboratory of Human Digital Twin (2022B121201 0004)
  • Inner Mongolia Science and Technology Achievement Transfer and Transformation Demonstration Zone
  • University Collaborative Innovation Base
  • University Entrepreneurship Training Base (21300-231510)
  • Internal Project Fund from Shenzhen Research Institute of Big Data (T00120220002)
  • Shenzhen Science and Technology Research Fund Fundamental Research Key Project (JCYJ20220818103001002)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
multimodal emotion recognition, missing modalities, modality gap, modality-invariant features, contrastive learning, cross-modal representation learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing Modalities. https://app.cochat.ai/papers/10.1109/taffc.2024.3378570
The paper
Liu, R., Zuo, H., Lian, Z., Schuller, B. W., & Li, H. (2024). Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing Modalities. IEEE Transactions on Affective Computing. https://doi.org/10.1109/taffc.2024.3378570

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.