The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index; 4 of the 9 sources its claims cite were checked for support.
Enhancing Multimodal Model Robustness Under Missing Modalities via Memory-Driven Prompt Learning
Open accessAbstract
Existing multimodal models typically assume the availability of all modalities, leading to significant performance degradation when certain modalities are missing. Recent methods have introduced prompt learning to adapt pretrained models to incomplete data, achieving remarkable performance when the missing cases are consistent during training and inference. However, these methods rely heavily on distribution consistency and fail to compensate for missing modalities, limiting their ability to generalize to unseen missing cases. To address this issue, we propose Memory-Driven Prompt Learning, a framework that adaptively compensates for missing modalities through prompt learning. The compensation strategies are achieved by two types of prompts: generative prompts and shared prompts. Generative prompts retrieve semantically similar samples from a predefined prompt memory that stores modality-specific semantic information, while shared prompts leverage available modalities to provide cross-modal compensation. Extensive experiments demonstrate the effectiveness of the proposed model, achieving significant improvements across diverse missing-modality scenarios, with average performance increasing from 34.76% to 40.40% on MM-IMDb, 62.71% to 77.06% on Food101, and 60.40% to 62.77% on Hateful Memes. The code is available at https://github.com/zhao-yh20/MemPrompt.
Overview
CoChat Summary
TL;DR
The paper proposes Memory-Driven Prompt Learning (MemPrompt), which compensates for missing modalities using generative prompts retrieved from a prompt memory and shared prompts that transfer cross-modal information. It reports improved robustness across MM-IMDb, Food101, and Hateful Memes, including under inconsistent missing-modality scenarios between training and inference.
Key findings
- Average performance improved from 34.76% to 40.40% on MM-IMDb, 62.71% to 77.06% on Food101, and 60.40% to 62.77% on Hateful Memes.
- On MM-IMDb, the method improved over prior best in text-missing (36.03 to 38.47), image-missing (44.64 to 48.59), and both-missing (41.48 to 42.47) cases.
- Under inconsistent missing scenarios on Food101, training with missing text but testing with missing image improved from 27.82 to 80.81; training with missing image but testing with missing text improved from 39.98 to 60.63.
- Ablations show both prompt types contribute, with the full model best; memory size peaks at K=16, and distributed prompting outperforms pre-appended prompting.
Methods
- MemPrompt uses a frozen ViLT backbone and finetunes only learnable prompts and task-specific classification layers.
- Generative prompts are adaptively retrieved from a modality-specific prompt memory using cosine similarity and weighted summation; shared prompts are mapped to each modality via a bottleneck MLP to provide cross-modal compensation.
- Experiments cover MM-IMDb (25,959 movies), UPMC Food101 (101 food categories), and Hateful Memes, comparing against ViLT and MAP under missing text, missing image, and missing both scenarios, including inconsistent train/inference missing cases.
Limitations
- The paper does not state an explicit limitations section.
- Evident from the text: the problem definition and experiments are limited to two modalities and three benchmarks with ViLT as the backbone; missing modalities are simulated with dummy inputs (empty characters/blank pixels) rather than naturally incomplete real-world data.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 1 Not valid · 1 Not checked · 39 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Existing multimodal models typically assume that all modalities are available during both training and inference [Kim et al. , 2021].”
- Also stated in: § Abstract · p. 1 · plain
“Existing multimodal models typically assume the availability of all modalities”
What the sources say
- Partly supportedcites Kim et al., 2021 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says vision-and-language pre-training models use images with aligned descriptions and are fine-tuned on tasks whose inputs involve both modalities. This supports the claim for the vision-and-language models discussed, but the paper does not establish that multimodal models generally assume every modality is available during both training and inference. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · p. 1 · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous researches on handling missing modalities primarily focus on multi-stream multimodal architectures, which often reconstruct missing modality features based on available modalities [Ma et al. , 2021; Wang et al. , 2023; Woo et al. , 2023].”
- Also stated in: § 2.1 Multimodal Learning with Missing Modalities · p. 2 · plain
“Existing methods attempt to simulate the missing modality data during training so that the model can generalize to the missing modalities scenarios. They primarily focus on reconstructing missing modality features from available ones.”
What the sources say
- Not matchedcites Ma et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Wang et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Woo et al., 2023 sourceExperimental — accuracy not yet measured
The Abstract (p. 1) and Missing Modality Predictive Coding section (p. 4) show that Woo et al.’s multimodal ActionMAE reconstructs missing modality features from the remaining features. The Related Work section (p. 7) notes a large body of missing-modality studies, but does not establish that previous methods primarily use multi-stream reconstruction architectures. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.1 Multimodal Learning with Missing Modalities · p. 2 · plain
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Extensive experiments demonstrate that our method achieves state-of-the-art performance, regardless of whether the missing modalities during training and inference are consistent or not.”
- Also stated in: § Abstract · p. 1 · boosted
“Extensive experiments demonstrate the effectiveness of the proposed model, achieving significant improvements across diverse missingmodality scenarios”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · p. 1 · boosted
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, further increasing the memory size slightly reduces performance, likely due to increased complexity.”
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, under high missing rates during training, the model achieves better performance when tested under similarly high missing rates.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
16 more claims
Create a free account to see all 45 claims and every source verdict.
- Probably not a substantial claim. 4
- It may state two separate claims as one. 20
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 Ablation Studies · p. 6
“We evaluated the model by training with different missing rates (10%, 30%, 70%, 90%) and testing under varying missing rates during inference, with results shown in Fig. 3(b).”
§ 4.4 Ablation Studies · p. 7
“(b) Model trained with different missing rates (10%, 30%,50%,70%) under missing both case, with results averaged over all missing scenarios.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 5
- [Bao et al. , 2022] Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-modalityexperts. Advances in Neural Information Processing Systems , 35:32897-32912, 2022.
- [He et al. , 2023b] Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameterefficient model adaptation for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 817-825, 2023.
- [Li and Liang, 2021] Xiang Lisa Li and Percy Liang. Prefixtuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4582-4597, 2021.
- [Roy and Etemad, 2024] Shuvendu Roy and Ali Etemad. Consistency-guided prompt learning for vision-language models. In The Twelfth International Conference on Learning Representations , 2024.
- [Wang et al. , 2023] Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro. Multimodal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15878-15887, 2023.
Cited by
Wang, Y., Wei, B., Zhang, Y., Jing, H., & Zhang, M. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00946-6
HiPMD: Hierarchical Prompting meets Multi-Layer Distillation for Incomplete Multimodal Learning
Mo, L., Jian, A., Zhou, R., Xu, L., & Dai, R. (2026) · International Conference on Communications · doi:10.1109/icc59461.2026.11586938
“Zhao et al. [15] address this limitation by employing structured prompts and modality-aware mechanisms to efficiently manage prompt combinations under missing-modality scenarios.”
Quality-Calibrated Energy Completion for Incomplete Multimodal Action Quality Assessment
Zhu, Y., He, H., & Zhang, Y. (2026) · International Conference on Electronic Technology and Information Science (ICETIS) · doi:10.1109/icetis70504.2026.11633460
“], diffusion-based modality synthesis [9], retrieval-augmented prompt tuning [10], and memory-driven prompt learning [11].”
V-Ideal: Generative Prompting for Fault-Tolerant Multimodal Web Services Under Microservice Failures
Fan, B., Seah, W. K. G., Xu, G., Xu, X., & Zhang, F. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00069
“…feature reconstruction; 3) Joint Diffusion [24]: A heavy generative model attempting to recover fine-grained modality distributions; and 4) Memory-Driven Prompt Learning (MDP) [25]: A recent state-of-the-art framework that retrieves missing modality representations from an external memory bank.”
Metadata
- Authors
- Zhao, Y., Xi, W., Fu, X., & Zhao, J.
- Affiliations
- Xi'an Jiaotong University (CN)
- Venue
- Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence
- Published
- 2025-09
- Pages
- 2458-2466
- OpenAlex
- W4414360615
- Cited by
- 6
- References
- 0
- Citations by year
- 2026: 6
- FWCI
- 4.53
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- Funders
- National Natural Science Foundation of China (62176205, 62372365, 62302383)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal models, missing modalities, prompt learning, robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Enhancing Multimodal Model Robustness Under Missing Modalities via Memory-Driven Prompt Learning. https://app.cochat.ai/papers/10.24963/ijcai.2025/274
- The paper
- Zhao, Y., Xi, W., Fu, X., & Zhao, J. (2025). Enhancing Multimodal Model Robustness Under Missing Modalities via Memory-Driven Prompt Learning. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. https://doi.org/10.24963/ijcai.2025/274
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.