CoChat

The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index; 1 of the 37 sources its claims cite were checked for support.

Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations

Open access
Mai, Sijie, Zeng, Ying, Hu, Haifeng · 2023 · IEEE Transactions on Multimedia

Abstract

Learning effective joint embedding for cross-modal data has always been a focus in the field of multimodal machine learning. We argue that during multimodal fusion, the generated multimodal embedding may be redundant, and the discriminative unimodal information may be ignored, which often interferes with accurate prediction and leads to a higher risk of overfitting. Moreover, unimodal representations also contain noisy information that negatively influences the learning of cross-modal dynamics. To this end, we introduce the multimodal information bottleneck (MIB), aiming to learn a powerful and sufficient multimodal representation that is free of redundancy and to filter out noisy information in unimodal representations. Specifically, inheriting from the general information bottleneck (IB), MIB aims to learn the minimal sufficient representation for a given task by maximizing the mutual information between the representation and the target and simultaneously constraining the mutual information between the representation and the input data. Different from general IB, our MIB regularizes both the multimodal and unimodal representations, which is a comprehensive and flexible framework that is compatible with any fusion methods. We develop three MIB variants, namely, early-fusion MIB, late-fusion MIB, and complete MIB, to focus on different perspectives of information constraints. Experimental results suggest that the proposed method reaches state-of-the-art performance on the tasks of multimodal sentiment analysis and multimodal emotion recognition across three widely used datasets. The codes are available at https://github.com/TmacMai/Multimodal-Information-Bottleneck.

Overview

CoChat Summary

TL;DR

The paper introduces a multimodal information bottleneck (MIB) framework with early-fusion, late-fusion, and complete variants to learn minimal sufficient unimodal and multimodal representations for multimodal sentiment analysis and emotion recognition. Across CMU-MOSI, CMU-MOSEI, and IEMOCAP, the variants report state-of-the-art or competitive results, with C-MIB/E-MIB often best.

Key findings

  • On CMU-MOSI, E-MIB achieves the best results: Acc7 48.6, Acc2 85.3, F1 85.3, MAE 0.711, Corr 0.798; it outperforms TFR-Net by 6.0% on Acc7, 1.3% on Acc2, and 1.4% on F1.
  • On CMU-MOSEI, C-MIB yields Acc2 86.2 and F1 86.2, improving over MISA by 0.7% on Acc2 and 1.2% on F1; Acc7 and Corr also improve over state-of-the-art methods.
  • On IEMOCAP, C-MIB is best for Angry, Sad, and Neutral emotions; L-MIB is best for Happy; E-MIB achieves Sad F1 86.9, a 2.0% improvement over the best existing result.
  • Ablation shows removing IB on acoustic or visual modalities causes significant performance drops, while removing it on language causes only a slight drop; β analysis finds best performance at β=1e-5, and β=0 drops binary accuracy by 1.5% and 7-class accuracy by 2% compared with β=1e-3 as stated. C-MIB has 109,835,748 parameters; IB components add 619,049 parameters (0.564% of total).

Methods

  • Design: Three MIB variants—E-MIB, L-MIB, and C-MIB—based on variational information bottleneck; they maximize mutual information between representation and target while minimizing mutual information between representation and input, regularizing multimodal and/or unimodal representations and remaining compatible with any fusion mechanism.
  • Data: CMU-MOSI (93 videos, utterance-level sentiment in [-3,3]), CMU-MOSEI (2928 videos, sentiment and six emotions), and IEMOCAP (151 videos, about 10K utterances, first four emotions).
  • Analysis: Comparisons against baselines on sentiment analysis and emotion recognition, ablation of IB constraints per modality, comparison across fusion strategies (Concatenation, Addition, Multiplication, PoE, Tensor Fusion, Graph Fusion), β sensitivity, t-SNE visualization, and parameter-count analysis.

Limitations

  • The paper states that applying IB on unimodal branches may ignore non-discriminative concurrence or complementary information between modalities, though β can alleviate this; L-MIB is suboptimal because it does not constrain the multimodal representation; and randomness in latent representation generation may affect performance, especially on the smaller CMU-MOSI dataset.
  • The text makes evident that there is no dedicated limitations section, evaluation is limited to three datasets and two tasks, β is sensitive (best at 1e-5 but defaulted to 1e-3), and nonverbal visual/acoustic features are described as low quality and not sentiment-sensitive.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tmm.2022.3171679

Cites

61 of 61 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

1 Valid · 7 Not valid · 25 Not checked · 86 Own results and interpretations

43 more claims

Create a free account to see all 119 claims and to verify the 23 references still open.

See all claims — free account
  • It may state two separate claims as one. 55
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • Probably not a substantial claim. 24

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 61 references

Research works printed but not linked: 1

  1. 10.1109/mis.2016.94 · DOI · Paper page

Cited by

See all 194

Metadata

Authors
Mai, S., Zeng, Y., & Hu, H.
Affiliations
Sun Yat-sen University (CN)
Venue
IEEE Transactions on Multimedia
Published
2023
Volume
25
Pages
4121-4134
ISSN
  • 1520-9210 (ISSN-L)
  • 1941-0077
OpenAlex
W4285149123
Cited by
182
References
57
Citations by year
2026: 56 · 2025: 73 · 2024: 35 · 2023: 17
FWCI
10.30
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Multimedia (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Natural Science Foundation of China (62076262)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
Keywords
information bottleneck, cross-modal representation learning, multimodal fusion, redundancy reduction, noise reduction, mutual information regularization, feature-level fusion, decision fusion, multimodal sentiment analysis, multimodal emotion recognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations. https://app.cochat.ai/papers/10.1109/tmm.2022.3171679
The paper
Mai, S., Zeng, Y., & Hu, H. (2023). Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations. IEEE Transactions on Multimedia. https://doi.org/10.1109/tmm.2022.3171679

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.