The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index; 1 of the 37 sources its claims cite were checked for support.
Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations
Open accessAbstract
Learning effective joint embedding for cross-modal data has always been a focus in the field of multimodal machine learning. We argue that during multimodal fusion, the generated multimodal embedding may be redundant, and the discriminative unimodal information may be ignored, which often interferes with accurate prediction and leads to a higher risk of overfitting. Moreover, unimodal representations also contain noisy information that negatively influences the learning of cross-modal dynamics. To this end, we introduce the multimodal information bottleneck (MIB), aiming to learn a powerful and sufficient multimodal representation that is free of redundancy and to filter out noisy information in unimodal representations. Specifically, inheriting from the general information bottleneck (IB), MIB aims to learn the minimal sufficient representation for a given task by maximizing the mutual information between the representation and the target and simultaneously constraining the mutual information between the representation and the input data. Different from general IB, our MIB regularizes both the multimodal and unimodal representations, which is a comprehensive and flexible framework that is compatible with any fusion methods. We develop three MIB variants, namely, early-fusion MIB, late-fusion MIB, and complete MIB, to focus on different perspectives of information constraints. Experimental results suggest that the proposed method reaches state-of-the-art performance on the tasks of multimodal sentiment analysis and multimodal emotion recognition across three widely used datasets. The codes are available at https://github.com/TmacMai/Multimodal-Information-Bottleneck.
Overview
CoChat Summary
TL;DR
The paper introduces a multimodal information bottleneck (MIB) framework with early-fusion, late-fusion, and complete variants to learn minimal sufficient unimodal and multimodal representations for multimodal sentiment analysis and emotion recognition. Across CMU-MOSI, CMU-MOSEI, and IEMOCAP, the variants report state-of-the-art or competitive results, with C-MIB/E-MIB often best.
Key findings
- On CMU-MOSI, E-MIB achieves the best results: Acc7 48.6, Acc2 85.3, F1 85.3, MAE 0.711, Corr 0.798; it outperforms TFR-Net by 6.0% on Acc7, 1.3% on Acc2, and 1.4% on F1.
- On CMU-MOSEI, C-MIB yields Acc2 86.2 and F1 86.2, improving over MISA by 0.7% on Acc2 and 1.2% on F1; Acc7 and Corr also improve over state-of-the-art methods.
- On IEMOCAP, C-MIB is best for Angry, Sad, and Neutral emotions; L-MIB is best for Happy; E-MIB achieves Sad F1 86.9, a 2.0% improvement over the best existing result.
- Ablation shows removing IB on acoustic or visual modalities causes significant performance drops, while removing it on language causes only a slight drop; β analysis finds best performance at β=1e-5, and β=0 drops binary accuracy by 1.5% and 7-class accuracy by 2% compared with β=1e-3 as stated. C-MIB has 109,835,748 parameters; IB components add 619,049 parameters (0.564% of total).
Methods
- Design: Three MIB variants—E-MIB, L-MIB, and C-MIB—based on variational information bottleneck; they maximize mutual information between representation and target while minimizing mutual information between representation and input, regularizing multimodal and/or unimodal representations and remaining compatible with any fusion mechanism.
- Data: CMU-MOSI (93 videos, utterance-level sentiment in [-3,3]), CMU-MOSEI (2928 videos, sentiment and six emotions), and IEMOCAP (151 videos, about 10K utterances, first four emotions).
- Analysis: Comparisons against baselines on sentiment analysis and emotion recognition, ablation of IB constraints per modality, comparison across fusion strategies (Concatenation, Addition, Multiplication, PoE, Tensor Fusion, Graph Fusion), β sensitivity, t-SNE visualization, and parameter-count analysis.
Limitations
- The paper states that applying IB on unimodal branches may ignore non-discriminative concurrence or complementary information between modalities, though β can alleviate this; L-MIB is suboptimal because it does not constrain the multimodal representation; and randomness in latent representation generation may affect performance, especially on the smaller CMU-MOSI dataset.
- The text makes evident that there is no dedicated limitations section, evaluation is limited to three datasets and two tasks, β is sensitive (best at 1e-5 but defaulted to 1e-3), and nonverbal visual/acoustic features are described as low quality and not sentiment-sensitive.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 7 Not valid · 25 Not checked · 86 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, our method can reach state-of-the-art performance even with very simple fusion methods.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results suggest that the proposed method reaches state-of-theart performance on the tasks of multimodal sentiment analysis and multimodal emotion recognition across three widely used datasets.”
- Also stated in: § A. · p. 2 · plain
“The MIB variants outperform other methods on multiple datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § A. · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, the visualization experiments demonstrate that our MIB enables the multimodal representations to be more discriminative.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“For CMU-MOSI dataset, although TFR-Net [54] performs better than the existing methods and sets up a high baseline, our MIB variants still obtain the best performance on the majority of the evaluation metrics.”
What the sources say
- Not checked yetcites [54] [54] Z. Yuan, W. Li, H. Xu, and W. Yu, 'Transformer-based feature reconstruction network for robust multimodal sentiment analysis,' in Proceedings of the 29th ACM International Conference on Multimedia , 2021, pp. 4400-4407. - source
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also refer the reader to Section IV-F for the comprehensive comparison between the MIB variants, where the average performances of the MIB variants become consistent when various fusion methods are considered.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
43 more claims
Create a free account to see all 119 claims and to verify the 23 references still open.
- It may state two separate claims as one. 55
- Borderline: one of the tests for a substantial claim is unclear. 1
- Probably not a substantial claim. 24
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Subsequently, Mai et al. [98] aimed to minimize redundancy in the fusion process by introducing the Multimodal Information Bottleneck (MIB) for MER. This method provides a non-redundant, robust multimodal representation for MER.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [98]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Previous works have tended to design complex fusion strategies to learn multimodal embedding with unimodal features, such as tensor fusion [4] and transformer-based fusion [5], [6].”
BackgroundExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 2×
“By observing the results in Table VIII, we have the following conclusions: 1) The importance of each modality: The language modality, with the powerful BERT [34] framework as its feature extractor, significantly performs better than the visual and acoustic modalities.”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“By observing the results in Table VIII, we have the following conclusions: 1) The importance of each modality: The language modality, with the powerful BERT [34] framework as its feature extractor, significantly performs better than the visual and acoustic modalities.”
BackgroundMethod(2008) · Journal of Machine Learning Research · Cited by 33,051
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 1×
“MAG-BERT [35] proposes an attachment, namely Multi-modal Adaptative Gate (MAG), to allow BERT and XL-Net [36] to accept multimodal nonverbal data during fine-tuning.”
MethodMultimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 7×
“Actually, another way to obtain z for L-MIB is referred to as the Bayesian product-of-experts (PoE) system [42], [43], [46]: where · denotes element-wise multiplication.”
BackgroundMethodInfluentialIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 3×
“3) IEMOCAP: IEMOCAP [51] is a multimodal emotion recognition dataset that contains a total number of 151 videos from 10 speakers.”
BackgroundInfluentialDeep CORAL: Correlation Alignment for Deep Domain Adaptation
Sun, B., & Saenko, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_35 · Cited by 3,757 · cited 2×
“The compared objectives include maximum mean discrepancy (MMD), Deep CORAL [57], orthogonal constraint, European distance and cosine distance.”
Background
Research works printed but not linked: 1
- 10.1109/mis.2016.94 · DOI · Paper page
Cited by
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“Subsequently, Mai et al. [98] aimed to minimize redundancy in the fusion process by introducing the Multimodal Information Bottleneck (MIB) for MER.”
MethodA Survey on Information Bottleneck
Hu, S., Lou, Z., Yan, X., & Ye, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3366349 · Cited by 173
RAFT: Robust Adversarial Fusion Transformer for multimodal sentiment analysis
Wang, R., Xu, D., Cascone, L., Wang, Y., Chen, H., Zheng, J., & Zhu, X. (2025) · Array · doi:10.1016/j.array.2025.100445 · Cited by 140
Multimodal Interaction, Interfaces, and Communication: A Survey
Dritsas, E., Τρίγκα, Μ., Troussas, C., & Phivos Mylonas, B. (2025) · Multimodal Technologies and Interaction · doi:10.3390/mti9010006 · Cited by 126
Noise Imitation Based Adversarial Training for Robust Multimodal Sentiment Analysis
Yuan, Z., Liu, Y., Xu, H., & Gao, K. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3267882 · Cited by 117
Metadata
- Authors
- Mai, S., Zeng, Y., & Hu, H.
- Affiliations
- Sun Yat-sen University (CN)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2023
- Volume
- 25
- Pages
- 4121-4134
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W4285149123
- Semantic Scholar
- 3f41112388d133cd11f7fafbbc3ab96b9d8b0e21
- Cited by
- 182
- References
- 57
- Citations by year
- 2026: 56 · 2025: 73 · 2024: 35 · 2023: 17
- FWCI
- 10.30
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62076262)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Keywords
- information bottleneck, cross-modal representation learning, multimodal fusion, redundancy reduction, noise reduction, mutual information regularization, feature-level fusion, decision fusion, multimodal sentiment analysis, multimodal emotion recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations. https://app.cochat.ai/papers/10.1109/tmm.2022.3171679
- The paper
- Mai, S., Zeng, Y., & Hu, H. (2023). Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations. IEEE Transactions on Multimedia. https://doi.org/10.1109/tmm.2022.3171679
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.