Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index; 24 of the 43 sources its claims cite were checked for support.

Robust Multimodal Learning With Missing Modalities via Parameter-Efficient Adaptation

Open access
Reza, Md Kaykobad, Prater-Bennette, Ashley, Asif, M. Salman · 2025 · IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract

Multimodal learning seeks to utilize data from multiple sources to improve the overall performance of downstream tasks. It is desirable for redundancies in the data to make multimodal systems robust to missing or corrupted observations in some correlated modalities. However, we observe that the performance of several existing multimodal networks significantly deteriorates if one or multiple modalities are absent at test time. To enable robustness to missing modalities, we propose a simple and parameter-efficient adaptation procedure for pretrained multimodal networks. In particular, we exploit modulation of intermediate features to compensate for the missing modalities. We demonstrate that such adaptation can partially bridge performance drop due to missing modalities and outperform independent, dedicated networks trained for the available modality combinations in some cases. The proposed adaptation requires extremely small number of parameters (e.g., fewer than 1% of the total parameters) and applicable to a wide range of modality combinations and tasks. We conduct a series of experiments to highlight the missing modality robustness of our proposed method on five different multimodal tasks across seven datasets. Our proposed method demonstrates versatility across various tasks and datasets, and outperforms existing methods for robust multimodal learning with missing modalities.

Overview

CoChat Summary

## TL;DR

The paper proposes a parameter-efficient way to adapt pretrained multimodal networks so they remain robust when one or more modalities are missing at test time. Across five tasks and seven datasets, the adapted models generally outperform pretrained baselines and often match or beat dedicated models trained for specific modality combinations.

## Key findings

  • The proposed adaptation uses fewer than 1% of total parameters in the authors’ experiments, e.g. 0.789M additional parameters for segmentation (less than 0.7%), 0.24M for NTU RGB+D (0.32%), 0.02M for sentiment analysis (0.775%), and 0.207M for UPMC Food-101 (0.18%).
  • On multimodal segmentation, adapted models improve average performance over the base CMNeXt model by 8.46% mIoU on MFNet and 16.54% mIoU on NYUDv2, with 0.79M extra parameters.
  • On multimodal sentiment analysis, the adapted model improves over base MulT by 3.57% accuracy and 6.79% F1 on CMU-MOSI, and by 23.78% accuracy and 33.22% F1 on CMU-MOSEI.
  • On NTU RGB+D action recognition, the adapted model reaches 92.91% average accuracy versus 92.23% for UMDR, and adds 0.24M parameters (0.32% of the total model parameters).
  • On UPMC Food-101 classification, the adapted method averages 79.83% accuracy, improving 9.45% over the base ViLT and 1.29% over the best prompting method.

## Methods

  • The paper trains a base multimodal model on all modalities, freezes it, and learns a small set of modality-combination-specific adaptation parameters for missing-modality scenarios.
  • The main adaptation mechanism is intermediate feature modulation using simple scale-and-shift layers (SSF); low-rank/additive adaptation and BitFit are also discussed and compared.
  • Experiments use five tasks across seven datasets: semantic segmentation (MFNet, NYUDv2), material segmentation (MCubeS), action recognition (NTU RGB+D), sentiment analysis (CMU-MOSI, CMU-MOSEI), and classification (UPMC Food-101), with base models including CMNeXt, MulT, UMDR, and ViLT.

## Limitations

  • Stated in the paper: the method only considers modalities missing at test time, not modalities missing during both training and test time.
  • Stated in the paper: it learns one set of adaptation parameters for each modality combination, so parameter complexity scales with the number of combinations; for M modalities, up to 2^M - 2 adaptation sets may be needed.
  • Stated in the paper: the authors did not optimize where to insert the learnable layers or how to minimize the number of parameters further, and leave this for future work.
  • Evident from the text: performance is not uniformly better than dedicated models on every dataset; for example, the paper notes dedicated models perform better on MFNet in some cases.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tpami.2024.3476487

Cites

100 of 100 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

11 Valid · 2 Not valid · 7 Not checked · 143 Own results and interpretations

102 more claims · 2 with their cited sources checked

Create a free account to see all 163 claims and to verify the 8 references still open.

See all claims — free account
  • Probably not a substantial claim. 9
  • It may state two separate claims as one. 46
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Citations

Cites

Show all 97 references

Research works printed but not linked: 2

  1. Hu, LoRA: Low-rank adaptation of large language models
  2. 10.5555/3045118.3045167 · doi:10.5555/3045118.3045167

Cited by

See all 78

Metadata

Authors
Reza, M. K., Prater-Bennette, A., & Asif, M. S.
Affiliations
  • University of California, Riverside (US)
  • United States Air Force Research Laboratory (US)
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Published
2025-02
Volume
47
Issue
2
Pages
742-754
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
OpenAlex
W4403296843
Cited by
78
References
97
Citations by year
2026: 55 · 2025: 17
FWCI
8.46
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Air Force Office of Scientific Research (FA9550-21-1-0330)
  • NSF (CCF-2046293)
  • Indiana Jetstream2 (CIS220128)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
multimodal learning, missing modalities, missing data robustness, parameter-efficient fine-tuning, feature modulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Robust Multimodal Learning With Missing Modalities via Parameter-Efficient Adaptation. https://app.cochat.ai/papers/10.1109/tpami.2024.3476487
The paper
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025). Robust Multimodal Learning With Missing Modalities via Parameter-Efficient Adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2024.3476487

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.