The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index; 24 of the 43 sources its claims cite were checked for support.
Robust Multimodal Learning With Missing Modalities via Parameter-Efficient Adaptation
Open accessAbstract
Multimodal learning seeks to utilize data from multiple sources to improve the overall performance of downstream tasks. It is desirable for redundancies in the data to make multimodal systems robust to missing or corrupted observations in some correlated modalities. However, we observe that the performance of several existing multimodal networks significantly deteriorates if one or multiple modalities are absent at test time. To enable robustness to missing modalities, we propose a simple and parameter-efficient adaptation procedure for pretrained multimodal networks. In particular, we exploit modulation of intermediate features to compensate for the missing modalities. We demonstrate that such adaptation can partially bridge performance drop due to missing modalities and outperform independent, dedicated networks trained for the available modality combinations in some cases. The proposed adaptation requires extremely small number of parameters (e.g., fewer than 1% of the total parameters) and applicable to a wide range of modality combinations and tasks. We conduct a series of experiments to highlight the missing modality robustness of our proposed method on five different multimodal tasks across seven datasets. Our proposed method demonstrates versatility across various tasks and datasets, and outperforms existing methods for robust multimodal learning with missing modalities.
Overview
CoChat Summary
## TL;DR
The paper proposes a parameter-efficient way to adapt pretrained multimodal networks so they remain robust when one or more modalities are missing at test time. Across five tasks and seven datasets, the adapted models generally outperform pretrained baselines and often match or beat dedicated models trained for specific modality combinations.
## Key findings
- The proposed adaptation uses fewer than 1% of total parameters in the authors’ experiments, e.g. 0.789M additional parameters for segmentation (less than 0.7%), 0.24M for NTU RGB+D (0.32%), 0.02M for sentiment analysis (0.775%), and 0.207M for UPMC Food-101 (0.18%).
- On multimodal segmentation, adapted models improve average performance over the base CMNeXt model by 8.46% mIoU on MFNet and 16.54% mIoU on NYUDv2, with 0.79M extra parameters.
- On multimodal sentiment analysis, the adapted model improves over base MulT by 3.57% accuracy and 6.79% F1 on CMU-MOSI, and by 23.78% accuracy and 33.22% F1 on CMU-MOSEI.
- On NTU RGB+D action recognition, the adapted model reaches 92.91% average accuracy versus 92.23% for UMDR, and adds 0.24M parameters (0.32% of the total model parameters).
- On UPMC Food-101 classification, the adapted method averages 79.83% accuracy, improving 9.45% over the base ViLT and 1.29% over the best prompting method.
## Methods
- The paper trains a base multimodal model on all modalities, freezes it, and learns a small set of modality-combination-specific adaptation parameters for missing-modality scenarios.
- The main adaptation mechanism is intermediate feature modulation using simple scale-and-shift layers (SSF); low-rank/additive adaptation and BitFit are also discussed and compared.
- Experiments use five tasks across seven datasets: semantic segmentation (MFNet, NYUDv2), material segmentation (MCubeS), action recognition (NTU RGB+D), sentiment analysis (CMU-MOSI, CMU-MOSEI), and classification (UPMC Food-101), with base models including CMNeXt, MulT, UMDR, and ViLT.
## Limitations
- Stated in the paper: the method only considers modalities missing at test time, not modalities missing during both training and test time.
- Stated in the paper: it learns one set of adaptation parameters for each modality combination, so parameter complexity scales with the number of combinations; for M modalities, up to 2^M - 2 adaptation sets may be needed.
- Stated in the paper: the authors did not optimize where to insert the learnable layers or how to minimize the number of parameters further, and leave this for future work.
- Evident from the text: performance is not uniformly better than dedicated models on every dataset; for example, the paper notes dedicated models perform better on MFNet in some cases.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 2 Not valid · 7 Not checked · 143 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, we observe that the performance of several existing multimodal networks significantly deteriorates if one or multiple modalities are absent at test time.”
- Also stated in: § 1 Introduction · plain
“Recent studies [14, 15, 16] have shown that MML is not inherently robust to missing modalities and performance can drop significantly when modalities are missing at test time.”
What the sources say
- Partly supportedcites [14] sourceExperimental — accuracy not yet measured
Section 3.2 (p. 3) reports dramatic performance drops for multimodal Transformers on test data with missing text. Section 5.4 (p. 6) specifies that the tests kept the image modality present while reducing or removing text; it does not establish the claim for multiple absent modalities or multimodal networks beyond Transformers. (LLM, conf=0.85)
- Partly supportedcites [15] sourceExperimental — accuracy not yet measured
Section 3.2 (p. 3) reports that the tested models were highly sensitive when the language modality was missing at test time. But the diagnostics removed only one modality at a time; they do not establish deterioration when multiple modalities are absent together. The Introduction (p. 1) also says removing audio or visual modalities did not produce statistically significant drops in the illustrated test. (LLM, conf=0.87)
- Partly supportedcites [16] McKinzie, 2023, On robustness in multimodal learning sourceExperimental — accuracy not yet measured
Section 6.1 reports that when modalities are missing at evaluation, both methods show a large drop in robustness. It describes the drop in average performance as small, however, so the claim overstates how much performance deteriorates. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“The resulting adaptation offers significant savings compared to training a separate model for each input combination or retraining the model using some specialized training strategy like modality dropout [18, 17] or knowledge distillation [20, 22].”
What the sources say
- Supportedcites [18] source
The citation identifies modality dropout as an example of a specialized training strategy, not as evidence for the citing authors’ savings claim. Section 3 describes using modality dropout during training, and Section 5.2 reports networks trained with different dropout probabilities. (LLM, conf=0.93)
- Supportedcites [17] source
The citation identifies modality dropout as an example of a specialized training strategy. The paper’s “ModDrop: multimodal dropout” section (p. 7) describes training a shared model to predict from an arbitrary number of available modalities. The savings comparison is the citing authors’ framing, not the point this citation is offered to establish. (LLM, conf=0.92)
- Not checked yetcites [20] source
Checked before — its earlier verdict is held back until a new check runs.
- Supportedcites [22] source
The citation identifies an example of a specialized training strategy, not evidence for the citing authors’ claimed savings. The paper’s Related Work section says M3L uses a knowledge-distillation framework for missing-modality robustness, and its Unsupervised training section describes the teacher–student training procedure. Page numbers are unavailable in our copy. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“prevents loss of information in the representation learning process [50]”
What the sources say
- Could not verifycites [50] sourceExperimental — accuracy not yet measured
Section 3.1 (pp. 3–4) describes group normalization and its representational properties, but does not discuss SSF or claim that it prevents information loss. Searches for “SSF,” “loss of information,” and information preservation found no passage addressing the attributed claim. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, existing works on missing modality robustness vary widely in terms of model architectures [32, 4], fusion methods [35, 63], training procedures [20, 32], and missing feature generation methods [23]. Due to this heterogeneity, a fair comparison based solely on model size/number of parameters is infeasible.”
What the sources say
- Partly supportedcites [32] Fan, 2023, SpiderMesh: Spatial-aware demand-guided recursive meshing for RGB-T semantic segmentation sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Supportedcites [4] source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Supportedcites [35] Lin, 2023, Variational probabilistic fusion network for RGB-T semantic segmentation source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Partly supportedcites [63] sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
- Not checked yetcites [20] source
Checked before — its earlier verdict is held back until a new check runs.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown in Table 5, our adaptation performs better compared to recent modality masking and generation based approach ActionMAE [23] and modality de- and re-coupling based approaches Motion-RGBD [91] and UMDR [63].”
What the sources say
- Supportedcites [23] source
The citation identifies ActionMAE as the modality-masking and reconstruction approach used for comparison. The Introduction (p. 2) describes ActionMAE as a masked autoencoder that reconstructs missing modality features; the Method section (p. 4) explains that it reconstructs missing modalities from those remaining. The claim that the citing authors’ adaptation performs better is their own Table 5 result, not a finding attributed to this source. (LLM, conf=0.94)
- Not checked — no full text of the sourcecites [91] source
- Supportedcites [63] source
The paper’s title (p. 1) identifies UMDR as a multimodal de- and re-coupling framework, supporting its identification in the citing sentence. The claim that the citing authors’ adaptation performs better is their own comparison in Table 5, not a result attributed to this source. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
102 more claims · 2 with their cited sources checked
Create a free account to see all 163 claims and to verify the 8 references still open.
- Probably not a substantial claim. 9
- It may state two separate claims as one. 46
- The quote only partly states this claim. 2
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Research works printed but not linked: 2
- Hu, LoRA: Low-rank adaptation of large language models
- 10.5555/3045118.3045167 · doi:10.5555/3045118.3045167
Cited by
MARIA: A multimodal transformer model for incomplete healthcare data
Caruso, C. M., Soda, P., & Guarrasi, V. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110843 · Cited by 32
Yang, Z., He, Q., Yu, M., Du, N., & Lu, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103571 · Cited by 37
Multimodal large model pretraining, adaptation and efficiency optimization
Ji, L., Xiao, S., Feng, J., Gao, W., & Zhang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129138 · Cited by 19
Quantum Federated Learning for Multimodal Data: A Modality-Agnostic Approach
Pokharel, A., Rahman, R., Morris, T., & Nguyen, D. C. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00059 · Cited by 15
“If these missing modalities are not properly addressed, they can significantly degrade the overall performance of the system [41].”
Uncertain Multimodal Intention and Emotion Understanding in the Wild
Yang, Q., Shi, Q., Wang, T., & Ye, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02300 · Cited by 29
Metadata
- Authors
- Reza, M. K., Prater-Bennette, A., & Asif, M. S.
- Affiliations
- University of California, Riverside (US)
- United States Air Force Research Laboratory (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2025-02
- Volume
- 47
- Issue
- 2
- Pages
- 742-754
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4403296843
- Semantic Scholar
- 36d1466dc91b3935391aeea68e5db457fb2c62fb
- Cited by
- 78
- References
- 97
- Citations by year
- 2026: 55 · 2025: 17
- FWCI
- 8.46
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Air Force Office of Scientific Research (FA9550-21-1-0330)
- NSF (CCF-2046293)
- Indiana Jetstream2 (CIS220128)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal learning, missing modalities, missing data robustness, parameter-efficient fine-tuning, feature modulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Robust Multimodal Learning With Missing Modalities via Parameter-Efficient Adaptation. https://app.cochat.ai/papers/10.1109/tpami.2024.3476487
- The paper
- Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025). Robust Multimodal Learning With Missing Modalities via Parameter-Efficient Adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2024.3476487
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.