Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index; 4 of the 9 sources its claims cite were checked for support.

Enhancing Multimodal Model Robustness Under Missing Modalities via Memory-Driven Prompt Learning

Open access
Zhao, Yihan, Xi, Wei, Fu, Xiao, Zhao, Jizhong · 2025 · Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence

Abstract

Existing multimodal models typically assume the availability of all modalities, leading to significant performance degradation when certain modalities are missing. Recent methods have introduced prompt learning to adapt pretrained models to incomplete data, achieving remarkable performance when the missing cases are consistent during training and inference. However, these methods rely heavily on distribution consistency and fail to compensate for missing modalities, limiting their ability to generalize to unseen missing cases. To address this issue, we propose Memory-Driven Prompt Learning, a framework that adaptively compensates for missing modalities through prompt learning. The compensation strategies are achieved by two types of prompts: generative prompts and shared prompts. Generative prompts retrieve semantically similar samples from a predefined prompt memory that stores modality-specific semantic information, while shared prompts leverage available modalities to provide cross-modal compensation. Extensive experiments demonstrate the effectiveness of the proposed model, achieving significant improvements across diverse missing-modality scenarios, with average performance increasing from 34.76% to 40.40% on MM-IMDb, 62.71% to 77.06% on Food101, and 60.40% to 62.77% on Hateful Memes. The code is available at https://github.com/zhao-yh20/MemPrompt.

Overview

CoChat Summary

TL;DR

The paper proposes Memory-Driven Prompt Learning (MemPrompt), which compensates for missing modalities using generative prompts retrieved from a prompt memory and shared prompts that transfer cross-modal information. It reports improved robustness across MM-IMDb, Food101, and Hateful Memes, including under inconsistent missing-modality scenarios between training and inference.

Key findings

  • Average performance improved from 34.76% to 40.40% on MM-IMDb, 62.71% to 77.06% on Food101, and 60.40% to 62.77% on Hateful Memes.
  • On MM-IMDb, the method improved over prior best in text-missing (36.03 to 38.47), image-missing (44.64 to 48.59), and both-missing (41.48 to 42.47) cases.
  • Under inconsistent missing scenarios on Food101, training with missing text but testing with missing image improved from 27.82 to 80.81; training with missing image but testing with missing text improved from 39.98 to 60.63.
  • Ablations show both prompt types contribute, with the full model best; memory size peaks at K=16, and distributed prompting outperforms pre-appended prompting.

Methods

  • MemPrompt uses a frozen ViLT backbone and finetunes only learnable prompts and task-specific classification layers.
  • Generative prompts are adaptively retrieved from a modality-specific prompt memory using cosine similarity and weighted summation; shared prompts are mapped to each modality via a bottleneck MLP to provide cross-modal compensation.
  • Experiments cover MM-IMDb (25,959 movies), UPMC Food101 (101 food categories), and Hateful Memes, comparing against ViLT and MAP under missing text, missing image, and missing both scenarios, including inconsistent train/inference missing cases.

Limitations

  • The paper does not state an explicit limitations section.
  • Evident from the text: the problem definition and experiments are limited to two modalities and three benchmarks with ViLT as the backbone; missing modalities are simulated with dummy inputs (empty characters/blank pixels) rather than naturally incomplete real-world data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.24963/ijcai.2025/274

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 1 Not valid · 1 Not checked · 39 Own results and interpretations

16 more claims

Create a free account to see all 45 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 4
  • It may state two separate claims as one. 20
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 5

  1. [Bao et al. , 2022] Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-modalityexperts. Advances in Neural Information Processing Systems , 35:32897-32912, 2022.
  2. [He et al. , 2023b] Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameterefficient model adaptation for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 817-825, 2023.
  3. [Li and Liang, 2021] Xiang Lisa Li and Percy Liang. Prefixtuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4582-4597, 2021.
  4. [Roy and Etemad, 2024] Shuvendu Roy and Ali Etemad. Consistency-guided prompt learning for vision-language models. In The Twelfth International Conference on Learning Representations , 2024.
  5. [Wang et al. , 2023] Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro. Multimodal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15878-15887, 2023.

Cited by

See all 6

Metadata

Authors
Zhao, Y., Xi, W., Fu, X., & Zhao, J.
Affiliations
Xi'an Jiaotong University (CN)
Venue
Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence
Published
2025-09
Pages
2458-2466
OpenAlex
W4414360615
Cited by
6
References
0
Citations by year
2026: 6
FWCI
4.53
Type
conference paper
Language
en
Open access
Locations
International Joint Conference on Artificial Intelligence (publishedVersion)
Funders
National Natural Science Foundation of China (62176205, 62372365, 62302383)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
multimodal models, missing modalities, prompt learning, robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Enhancing Multimodal Model Robustness Under Missing Modalities via Memory-Driven Prompt Learning. https://app.cochat.ai/papers/10.24963/ijcai.2025/274
The paper
Zhao, Y., Xi, W., Fu, X., & Zhao, J. (2025). Enhancing Multimodal Model Robustness Under Missing Modalities via Memory-Driven Prompt Learning. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. https://doi.org/10.24963/ijcai.2025/274

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.