Robust Multimodal Learning With Missing Modalities via Parameter-Efficient Adaptation
Open accessThe DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index; 24 of the 43 sources its claims cite were checked for support.
Abstract
Multimodal learning seeks to utilize data from multiple sources to improve the overall performance of downstream tasks. It is desirable for redundancies in the data to make multimodal systems robust to missing or corrupted observations in some correlated modalities. However, we observe that the performance of several existing multimodal networks significantly deteriorates if one or multiple modalities are absent at test time. To enable robustness to missing modalities, we propose a simple and parameter-efficient adaptation procedure for pretrained multimodal networks. In particular, we exploit modulation of intermediate features to compensate for the missing modalities. We demonstrate that such adaptation can partially bridge performance drop due to missing modalities and outperform independent, dedicated networks trained for the available modality combinations in some cases. The proposed adaptation requires extremely small number of parameters (e.g., fewer than 1% of the total parameters) and applicable to a wide range of modality combinations and tasks. We conduct a series of experiments to highlight the missing modality robustness of our proposed method on five different multimodal tasks across seven datasets. Our proposed method demonstrates versatility across various tasks and datasets, and outperforms existing methods for robust multimodal learning with missing modalities.
Overview
CoChat Summary
## TL;DR
The paper proposes a parameter-efficient way to adapt pretrained multimodal networks so they remain robust when one or more modalities are missing at test time. Across five tasks and seven datasets, the adapted models generally outperform pretrained baselines and often match or beat dedicated models trained for specific modality combinations.
## Key findings
- The proposed adaptation uses fewer than 1% of total parameters in the authors’ experiments, e.g. 0.789M additional parameters for segmentation (less than 0.7%), 0.24M for NTU RGB+D (0.32%), 0.02M for sentiment analysis (0.775%), and 0.207M for UPMC Food-101 (0.18%).
- On multimodal segmentation, adapted models improve average performance over the base CMNeXt model by 8.46% mIoU on MFNet and 16.54% mIoU on NYUDv2, with 0.79M extra parameters.
- On multimodal sentiment analysis, the adapted model improves over base MulT by 3.57% accuracy and 6.79% F1 on CMU-MOSI, and by 23.78% accuracy and 33.22% F1 on CMU-MOSEI.
- On NTU RGB+D action recognition, the adapted model reaches 92.91% average accuracy versus 92.23% for UMDR, and adds 0.24M parameters (0.32% of the total model parameters).
- On UPMC Food-101 classification, the adapted method averages 79.83% accuracy, improving 9.45% over the base ViLT and 1.29% over the best prompting method.
## Methods
- The paper trains a base multimodal model on all modalities, freezes it, and learns a small set of modality-combination-specific adaptation parameters for missing-modality scenarios.
- The main adaptation mechanism is intermediate feature modulation using simple scale-and-shift layers (SSF); low-rank/additive adaptation and BitFit are also discussed and compared.
- Experiments use five tasks across seven datasets: semantic segmentation (MFNet, NYUDv2), material segmentation (MCubeS), action recognition (NTU RGB+D), sentiment analysis (CMU-MOSI, CMU-MOSEI), and classification (UPMC Food-101), with base models including CMNeXt, MulT, UMDR, and ViLT.
## Limitations
- Stated in the paper: the method only considers modalities missing at test time, not modalities missing during both training and test time.
- Stated in the paper: it learns one set of adaptation parameters for each modality combination, so parameter complexity scales with the number of combinations; for M modalities, up to 2^M - 2 adaptation sets may be needed.
- Stated in the paper: the authors did not optimize where to insert the learnable layers or how to minimize the number of parameters further, and leave this for future work.
- Evident from the text: performance is not uniformly better than dedicated models on every dataset; for example, the paper notes dedicated models perform better on MFNet in some cases.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
11 Valid · 2 Not valid · 7 Not checked · 143 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, we observe that the performance of several existing multimodal networks significantly deteriorates if one or multiple modalities are absent at test time.”
- Also stated in: § 1 Introduction · plain
“Recent studies [14, 15, 16] have shown that MML is not inherently robust to missing modalities and performance can drop significantly when modalities are missing at test time.”
What the sources say
- Partly supportedcites [14] sourceExperimental — accuracy not yet measured
Section 3.2 (p. 3) reports dramatic performance drops for multimodal Transformers on test data with missing text. Section 5.4 (p. 6) specifies that the tests kept the image modality present while reducing or removing text; it does not establish the claim for multiple absent modalities or multimodal networks beyond Transformers. (LLM, conf=0.85)
- Partly supportedcites [15] sourceExperimental — accuracy not yet measured
Section 3.2 (p. 3) reports that the tested models were highly sensitive when the language modality was missing at test time. But the diagnostics removed only one modality at a time; they do not establish deterioration when multiple modalities are absent together. The Introduction (p. 1) also says removing audio or visual modalities did not produce statistically significant drops in the illustrated test. (LLM, conf=0.87)
- Partly supportedcites [16] McKinzie, 2023, On robustness in multimodal learning sourceExperimental — accuracy not yet measured
Section 6.1 reports that when modalities are missing at evaluation, both methods show a large drop in robustness. It describes the drop in average performance as small, however, so the claim overstates how much performance deteriorates. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“The resulting adaptation offers significant savings compared to training a separate model for each input combination or retraining the model using some specialized training strategy like modality dropout [18, 17] or knowledge distillation [20, 22].”
What the sources say
- Supportedcites [18] source
The citation identifies modality dropout as an example of a specialized training strategy, not as evidence for the citing authors’ savings claim. Section 3 describes using modality dropout during training, and Section 5.2 reports networks trained with different dropout probabilities. (LLM, conf=0.93)
- Supportedcites [17] source
The citation identifies modality dropout as an example of a specialized training strategy. The paper’s “ModDrop: multimodal dropout” section (p. 7) describes training a shared model to predict from an arbitrary number of available modalities. The savings comparison is the citing authors’ framing, not the point this citation is offered to establish. (LLM, conf=0.92)
- Not checked yetcites [20] source
Checked before — its earlier verdict is held back until a new check runs.
- Supportedcites [22] source
The citation identifies an example of a specialized training strategy, not evidence for the citing authors’ claimed savings. The paper’s Related Work section says M3L uses a knowledge-distillation framework for missing-modality robustness, and its Unsupervised training section describes the teacher–student training procedure. Page numbers are unavailable in our copy. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“prevents loss of information in the representation learning process [50]”
What the sources say
- Could not verifycites [50] sourceExperimental — accuracy not yet measured
Section 3.1 (pp. 3–4) describes group normalization and its representational properties, but does not discuss SSF or claim that it prevents information loss. Searches for “SSF,” “loss of information,” and information preservation found no passage addressing the attributed claim. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, existing works on missing modality robustness vary widely in terms of model architectures [32, 4], fusion methods [35, 63], training procedures [20, 32], and missing feature generation methods [23]. Due to this heterogeneity, a fair comparison based solely on model size/number of parameters is infeasible.”
What the sources say
- Partly supportedcites [32] Fan, 2023, SpiderMesh: Spatial-aware demand-guided recursive meshing for RGB-T semantic segmentation sourceExperimental — accuracy not yet measured
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Supportedcites [4] source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Supportedcites [35] Lin, 2023, Variational probabilistic fusion network for RGB-T semantic segmentation source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Partly supportedcites [63] sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
- Not checked yetcites [20] source
Checked before — its earlier verdict is held back until a new check runs.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown in Table 5, our adaptation performs better compared to recent modality masking and generation based approach ActionMAE [23] and modality de- and re-coupling based approaches Motion-RGBD [91] and UMDR [63].”
What the sources say
- Supportedcites [23] source
The citation identifies ActionMAE as the modality-masking and reconstruction approach used for comparison. The Introduction (p. 2) describes ActionMAE as a masked autoencoder that reconstructs missing modality features; the Method section (p. 4) explains that it reconstructs missing modalities from those remaining. The claim that the citing authors’ adaptation performs better is their own Table 5 result, not a finding attributed to this source. (LLM, conf=0.94)
- Not checked — no full text of the sourcecites [91] source
- Supportedcites [63] source
The paper’s title (p. 1) identifies UMDR as a multimodal de- and re-coupling framework, supporting its identification in the citing sentence. The claim that the citing authors’ adaptation performs better is their own comparison in Table 5, not a result attributed to this source. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
102 more claims · 2 with their cited sources checked
Create a free account to see all 163 claims and to verify the 8 references still open.
- Probably not a substantial claim. 9
- It may state two separate claims as one. 46
- The quote only partly states this claim. 2
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 1×
“[48] utilized complementary random masking, [21] used masked auto encoder, and [38] applied masked cross-modal attention for enhancing robustness of the underlying model.”
MethodMulti-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760 · cited 1×
“We experimented on two datasets for multimodal semantic segmentation tasks: MFNet dataset [17] with RGB-Thermal input modalities and NYUDv2 dataset [49] with RGB-Depth input modalities.”
MethodSegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · cited 1×
“Multimodal learning (MML) [4, 63] is a general framework for processing, combining, and understanding information from multiple, diverse data sources.”
BackgroundTarvainen, A., & Valpola, H. (2017) · arXiv · doi:10.48550/arxiv.1703.01780 · Cited by 4,275
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
Wu, Y., & He, K. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01198-w · Cited by 3,601
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317
NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
Shahroudy, A., Liu, J., Ng, T.-T., & Wang, G. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.115 · Cited by 3,148 · cited 1×
“Missing modalities at test time can cause significant degradation in the performance of multimodal systems.”
BackgroundParameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · cited 3×
“We use multimodal transformer [53] as the base model for multimodal sentiment analysis.”
MethodTensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102
Food-101 – Mining Discriminative Components with Random Forests
Bossard, L., Guillaumin, M., & Van Gool, L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10599-4_29 · Cited by 2,072
Learning Rich Features from RGB-D Images for Object Detection and Segmentation
Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_23 · Cited by 1,528 · cited 1×
“For our experiments, we use HHA-encoded images proposed by [16] instead of raw depth maps.”
MethodMultimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · cited 3×
“We tested our adaptation method for multimodal sentiment analysis on CMU-MOSI [69] and CMU-MOSEI [3] datasets, and report the results in Table 5.”
BackgroundMethodParameter-efficient fine-tuning of large-scale pre-trained language models
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00626-4 · Cited by 1,191 · cited 1×
“A number of parameter-efficient methods have been proposed for transfer learning [20, 10] and domain/task adaptation [24, 31].”
MethodEfficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · cited 1×
“Multimodal learning (MML) [4, 63] is a general framework for processing, combining, and understanding information from multiple, diverse data sources.”
BackgroundViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
Kim, W., Son, B., & Kim, I. (2021) · arXiv · doi:10.48550/arxiv.2102.03334 · Cited by 951 · cited 1×
“We use multimodal transformer [53] as the base model for multimodal sentiment analysis.”
MethodFuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902
Memory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852 · cited 2×
“We tested our adaptation method for multimodal sentiment analysis on CMU-MOSI [69] and CMU-MOSEI [3] datasets, and report the results in Table 5.”
BackgroundMethodYu, W., Xu, H., Yuan, Z., & Wu, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17289 · Cited by 816 · cited 1×
“For multimodal sentiment analysis tasks, we used the default settings for the datasets as configured in the codebase [66].”
MethodBitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.1 · Cited by 765 · cited 4×
“We summarize the results for scale only, shift only, BitFit [5], norm layer fine-tuning and LoRA [24] in Table 4.”
BackgroundMethodInfluentialMFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes
Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., & Harada, T. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206396 · Cited by 693 · cited 1×
“We experimented on two datasets for multimodal semantic segmentation tasks: MFNet dataset [17] with RGB-Thermal input modalities and NYUDv2 dataset [49] with RGB-Depth input modalities.”
MethodIntegrating Multimodal Information in Large Pretrained Transformers
Rahman, W., Hasan, M. K., Lee, S., Bagher Zadeh, A., Mao, C., Morency, L.-P., & Hoque, E. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.214 · Cited by 673
A survey of multimodal sentiment analysis
Soleymani, M., Garcia, D., Jou, B., Schuller, B., Chang, S.-F., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.08.003 · Cited by 601 · cited 1×
“In recent years, MML has achieved remarkable success in a wide range of applications, including image segmentation [6, 55, 70], captioning [71, 65], classification [15, 46], sentiment analysis [50, 28], and autonomous driving [61, 45].”
BackgroundRtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes
Sun, Y., Zuo, W., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2904733 · Cited by 546
Multimodal Fusion Transformer for Remote Sensing Image Classification
Roy, S. K., Deria, A., Hong, D., Rasti, B., Plaza, A., & Chanussot, J. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3286826 · Cited by 519 · cited 1×
“In recent years, MML has achieved remarkable success in a wide range of applications, including image segmentation [6, 55, 70], captioning [71, 65], classification [15, 46], sentiment analysis [50, 28], and autonomous driving [61, 45].”
BackgroundChen, X., Lin, K.-Y., Wang, J., Wu, W., Qian, C., Li, H., & Zeng, G. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_33 · Cited by 488 · cited 2×
“In recent years, MML has achieved remarkable success in a wide range of applications, including image segmentation [6, 55, 70], captioning [71, 65], classification [15, 46], sentiment analysis [50, 28], and autonomous driving [61, 45].”
BackgroundMethodMultimodal Transformer With Multi-View Visual Representation for Image Captioning
Yu, J., Li, J., Yu, Z., & Huang, Q. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2947482 · Cited by 468 · cited 1×
“In recent years, MML has achieved remarkable success in a wide range of applications, including image segmentation [6, 55, 70], captioning [71, 65], classification [15, 46], sentiment analysis [50, 28], and autonomous driving [61, 45].”
BackgroundMultiMAE: Multi-modal Multi-task Masked Autoencoders
Bachmann, R., Mizrahi, D., Atanov, A., et al. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2204.01678 · Cited by 448
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning
Lian, D., Zhou, D., Feng, J., & Wang, X. (2022) · neural information processing systems · doi:10.52202/068431-0009 · Cited by 446
Ba, J. L., Kiros, J., & Hinton, G. E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.06450 · Cited by 440
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428 · cited 1×
“Apart from these approaches, weight space ensembling [59], policy learning [38] and optimal fusion strategy designing [40] were also studied for robust MML for various tasks.”
MethodModDrop: Adaptive Multi-Modal Gesture Recognition
Neverova, N., Wolf, C., Taylor, G., & Nebout, F. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2461544 · Cited by 420
Multimodal semi-supervised learning for image classification
Guillaumin, M., Verbeek, J., & Schmid, C. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540120 · Cited by 418 · cited 1×
“In recent years, MML has achieved remarkable success in a wide range of applications, including image segmentation [6, 55, 70], captioning [71, 65], classification [15, 46], sentiment analysis [50, 28], and autonomous driving [61, 45].”
BackgroundFew-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., & Raffel, C. (2022) · neural information processing systems · doi:10.52202/068431-0142 · Cited by 377
SMIL: Multimodal Learning with Severely Missing Modality
Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., & Peng, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16330 · Cited by 332 · cited 1×
“Unlike existing methods, we do not try to generate [39, 2], approximate [55, 54] or distill knowledge [48, 40] from any other modality/sub-network.”
MethodMultimodal end-to-end autonomous driving
Xiao, Y., Codevilla, F., Gurram, A., Urfalioglu, O., & Lopez, A. M. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3013234 · Cited by 319 · cited 1×
“In recent years, MML has achieved remarkable success in a wide range of applications, including image segmentation [6, 55, 70], captioning [71, 65], classification [15, 46], sentiment analysis [50, 28], and autonomous driving [61, 45].”
BackgroundMultimodal Token Fusion for Vision Transformers
Wang, Y., Chen, X., Cao, L., Huang, W., Sun, F., & Wang, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01187 · Cited by 272 · cited 7×
“Unlike existing methods, we do not try to generate [39, 2], approximate [55, 54] or distill knowledge [48, 40] from any other modality/sub-network.”
BackgroundMethodInfluentialMultiMAE: Multi-modal Multi-task Masked Autoencoders
Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_20 · Cited by 268
Towards a Unified View of Parameter-Efficient Transfer Learning
He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04366 · Cited by 268
Missing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network
Sharma, A., & Hamarneh, G. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2945521 · Cited by 254 · cited 2×
“Knowledge distillation and generation methods have also become popular for robust MML. Studies by [47] and [64] used GAN based generative models while [11] used VAE based generative models for imputing missing modalities from available input modalities for unlerlying multimodal tasks.”
MethodMulti-Modal Learning with Missing Modality via Shared-Specific Feature Modelling
Wang, H., Chen, Y., Ma, C., Avery, J., Hull, L., & Carneiro, G. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01524 · Cited by 216 · cited 2×
“Unlike existing methods, we do not try to generate [39, 2], approximate [55, 54] or distill knowledge [48, 40] from any other modality/sub-network.”
BackgroundMethodAre Multimodal Transformers Robust to Missing Modality?
Ma, M., Ren, J., Zhao, L., Testuggine, D., & Peng, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01764 · Cited by 198 · cited 4×
“Apart from these approaches, weight space ensembling [59], policy learning [38] and optimal fusion strategy designing [40] were also studied for robust MML for various tasks.”
BackgroundMethodInfluentialParameter-Efficient Model Adaptation for Vision Transformers
He, X., Li, C., Zhang, P., Yang, J., & Wang, X. E. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25160 · Cited by 195 · cited 1×
“For instance, LoRA [24], QLoRA [9], KronA [12] and KAdaptataion [22] learn low-rank factors for task/domain adaptation.”
BackgroundModality Distillation with Multiple Stream Networks for Action Recognition
Garcia, N. C., Morerio, P., & Murino, V. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01237-3_7 · Cited by 194
Delivering Arbitrary-Modal Semantic Segmentation
Zhang, J., Liu, R., Shi, H., Yang, K., Reiß, S., Peng, K., Fu, H., Wang, K., & Stiefelhagen, R. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00116 · Cited by 188 · cited 6×
“NYUDv2 Dataset: For processing depth maps, we follow SA-Gate by [6] and CMNeXt by [70] and use HHA-encoded images instead of raw depth maps.”
BackgroundMethodInfluentialRecipe recognition with large multimodal food dataset
Wang, X., Kumar, D., Thome, N., Cord, M., & Precioso, F. (2015) · 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) · doi:10.1109/icmew.2015.7169757 · Cited by 179 · cited 1×
“We are committed to ensuring the reproducibility of our research and to facilitate the broader scientific community in replicating and building upon our work.”
BackgroundFEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation
Deng, F., Feng, H., Liang, M., Wang, H., Yang, Y., Gao, Y., Chen, J., Hu, J., Guo, X., & Lam, T. L. (2021) · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636084 · Cited by 170
Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation
Dorent, R., Joutard, S., Modat, M., Ourselin, S., & Vercauteren, T. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_9 · Cited by 160 · cited 2×
“Knowledge distillation and generation methods have also become popular for robust MML. Studies by [47] and [64] used GAN based generative models while [11] used VAE based generative models for imputing missing modalities from available input modalities for unlerlying multimodal tasks.”
MethodEmbraceNet: A robust deep learning architecture for multimodal classification
Choi, J.-H., & Lee, J.-S. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.02.010 · Cited by 156 · cited 1×
“[7] proposed a robust fusion strategy for multimodal classification.”
MethodMultimodal Prompting with Missing Modalities for Visual Recognition
Lee, Y.-L., Tsai, Y.-H., Chiu, W.-C., & Lee, C.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01435 · Cited by 156 · cited 7×
“We used a batch size of 4 and report single scale performance for all the datasets.”
BackgroundMethodInfluentialMitigating Modality Discrepancies for RGB-T Semantic Segmentation
Zhao, S., Liu, Y., Jiao, Q., Zhang, Q., & Han, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3233089 · Cited by 129 · cited 2×
“Our approach performs significantly better compared to masking and recursive meshing based SpiderMesh [13], variational pobabilistic fusion based VPFNet [33] and modality discrepancy reduction based MDRNet [72] models.”
MethodMultimodal Material Segmentation
Liang, Y., Wakaki, R., Nobuhara, S., & Nishino, K. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01918 · Cited by 128 · cited 1×
“We used MCubeS dataset [32] for multimodal material segmentation having RGB, Angle of Linear Polarization (AoLP), Degree of Linear Polarization (DoLP) and Near-Infrared (NIR) images as input modalities.”
MethodGraph Distillation for Action Detection with Privileged Modalities
Luo, Z., Hsieh, J.-T., Jiang, L., Niebles, J. C., & Fei-Fei, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_11 · Cited by 111 · cited 1×
“We have shown mean accuracy, F1 score and % mIoU for each experiment.”
MethodDeep Multimodal Fusion by Channel Exchanging
Wang, Y., Huang, W., Sun, F., Xu, T., Rong, Y., & Huang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.05005 · Cited by 110
3D cGAN based cross-modality MR image synthesis for brain tumor segmentation
Yu, B., Zhou, L., Wang, L., Fripp, J., & Bourgeat, P. (2018) · 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) · doi:10.1109/isbi.2018.8363653 · Cited by 109 · cited 2×
“Knowledge distillation and generation methods have also become popular for robust MML. Studies by [47] and [64] used GAN based generative models while [11] used VAE based generative models for imputing missing modalities from available input modalities for unlerlying multimodal tasks.”
MethodTowards Good Practices for Missing Modality Robust Action Recognition
Woo, S., Lee, S., Park, Y., Nugroho, M. A., & Kim, C. (2022) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2211.13916 · Cited by 101
Multimodal Sentiment Analysis: A Survey and Comparison
Kaur, R., & Kautish, S. (2019) · International Journal of Service Science, Management, Engineering, and Technology · doi:10.4018/ijssmet.2019040103 · Cited by 72 · cited 1×
“In recent years, MML has achieved remarkable success in a wide range of applications, including image segmentation [6, 55, 70], captioning [71, 65], classification [15, 46], sentiment analysis [50, 28], and autonomous driving [61, 45].”
BackgroundLearning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion
Wang, Y., Sun, F., Lu, M., & Yao, A. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413621 · Cited by 68 · cited 2×
“Our method also performs significantly better compared to dynamic channel exchange based CEN [56] and asymmetric fusion based AsymFusion [57] models.”
MethodTowards Good Practices for Missing Modality Robust Action Recognition
Woo, S., Lee, S., Park, Y., Nugroho, M. A., & Kim, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25378 · Cited by 67
MMSFormer: Multimodal Transformer for Material and Semantic Segmentation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2024) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2024.3389812 · Cited by 64 · cited 1×
“In recent years, MML has achieved remarkable success in a wide range of applications, including image segmentation [6, 55, 70], captioning [71, 65], classification [15, 46], sentiment analysis [50, 28], and autonomous driving [61, 45].”
BackgroundSelf-citationA multimodal fusion approach for image captioning
Zhao, D., Chang, Z., & Guo, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.11.004 · Cited by 61 · cited 1×
“In recent years, MML has achieved remarkable success in a wide range of applications, including image segmentation [6, 55, 70], captioning [71, 65], classification [15, 46], sentiment analysis [50, 28], and autonomous driving [61, 45].”
BackgroundRizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58 · cited 1×
“In recent years, MML has achieved remarkable success in a wide range of applications, including image segmentation [6, 55, 70], captioning [71, 65], classification [15, 46], sentiment analysis [50, 28], and autonomous driving [61, 45].”
BackgroundDecoupling and Recoupling Spatiotemporal Representation for RGB-D-based Motion Recognition
Zhou, B., Wang, P., Wan, J., Liang, Y., Wang, F., Zhang, D., Lei, Z., Li, H., & Jin, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01952 · Cited by 53 · cited 2×
“We have shown mean accuracy, F1 score and % mIoU for each experiment.”
BackgroundMethodDistillation Multiple Choice Learning for Multimodal Action Recognition
Garcia, N. C., Bargal, S. A., Ablavsky, V., Morerio, P., Murino, V., & Sclaroff, S. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00280 · Cited by 53
Analyzing Modality Robustness in Multimodal Sentiment Analysis
Hazarika, D., Li, Y., Cheng, B., Zhao, S., Zimmermann, R., & Poria, S. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.50 · Cited by 51
Complementary Random Masking for RGB-Thermal Semantic Segmentation
Shin, U., Lee, K., Kweon, I. S., & Oh, J. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611200 · Cited by 49 · cited 10×
“Unlike existing methods, we do not try to generate [39, 2], approximate [55, 54] or distill knowledge [48, 40] from any other modality/sub-network.”
BackgroundMethodInfluentialModality Dropout for Improved Performance-driven Talking Faces
Hussen Abdelaziz, A., Theobald, B.-J., Dixon, P., Knothe, R., Apostoloff, N., & Kajareker, S. (2020) · Proceedings of the 2020 International Conference on Multimodal Interaction · doi:10.1145/3382507.3418840 · Cited by 42 · cited 5×
“For instance, one approach is to adopt robust training strategies such as modality dropout during training [42, 25], partial or full modality masking [2, 48], and knowledge distillation [52, 40].”
BackgroundMethodInfluentialA Unified Multimodal De- and Re-Coupling Framework for RGB-D Motion Recognition
Zhou, B., Wang, P., Wan, J., Liang, Y., & Wang, F. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3274783 · Cited by 38
Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation
Maheshwari, H., Liu, Y.-C., & Kira, Z. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00106 · Cited by 37 · cited 8×
“Unlike existing methods, we do not try to generate [39, 2], approximate [55, 54] or distill knowledge [48, 40] from any other modality/sub-network.”
BackgroundMethodInfluentialA unified representation network for segmentation with missing modalitiesPreprint
Lau, K., Adler, J., & Sjölund, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06683 · Cited by 33
Han, W., Chen, H., Kan, M., & Poria, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.717 · Cited by 27
Efficient Yet Deep Convolutional Neural Networks for Semantic Segmentation
Kamran, S. A., & Sabbir, A. S. (2018) · International Symposium on Advanced Intelligent Informatics (SAIN) · doi:10.1109/sain.2018.8673354 · Cited by 24 · cited 1×
“Since we only insert a very small number of additional parameters (less than 0.7% of the total model parameters), it adds very limited computational overhead.”
BackgroundInfluentialExploring Missing Modality in Multimodal Egocentric Datasets
Ramazanova, M., Pardo, A., Alwassel, H., & Ghanem, B. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00013 · Cited by 18
What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?Preprint
Li, S., Du, C., Zhao, Y., Huang, Y., & Zhao, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.06383 · Cited by 16
KronA: Parameter Efficient Tuning with Kronecker AdapterPreprint
Edalati, A., Tahaei, M. S., Kobyzev, I., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.10650 · Cited by 10
SpiderMesh: Spatial-aware demand-guided recursive meshing for RGB-T semantic segmentation
Fan, S., Wang, Z., Wang, Y., & Liu, J. (2023) · arXiv · doi:10.48550/arxiv.2303.08692 · Cited by 8 · cited 4×
“Our approach performs significantly better compared to masking and recursive meshing based SpiderMesh [13], variational pobabilistic fusion based VPFNet [33] and modality discrepancy reduction based MDRNet [72] models.”
BackgroundMethodInfluentialLearning Rich Features from RGB-D Images for Object Detection and SegmentationPreprint
Gupta, S., Girshick, R., Arbeláez, P., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.5736 · Cited by 4
Variational probabilistic fusion network for RGB-T semantic segmentation
Lin, B., Lin, Z., Guo, Y., Zhang, Y., Zou, J., & Fan, S. (2023) · arXiv · doi:10.48550/arxiv.2307.08536 · Cited by 3
Analyzing Modality Robustness in Multimodal Sentiment Analysis
Hazarika, D., Li, Y., Cheng, B. H., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.15465 · Cited by 3
Supervised Contrastive Learning for Robust and Efficient Multi-modal Emotion and Sentiment Analysis
Gomaa, A., Maier, A., & Kosti, R. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956637 · Cited by 2 · cited 1×
“Such approaches include modality dropout during training [42, 25], unified representation learning [29], and supervised contrastive learning [14].”
BackgroundExploring Missing Modality in Multimodal Egocentric DatasetsPreprint
Ramazanova, M., Pardo, A., Alwassel, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.11470 · Cited by 2
(2023) · Oxford University Press eBooks · doi:10.1093/oed/7145107284 · Cited by 1
On robustness in multimodal learning
McKinzie, B., Cheng, J., Shankar, V., Yang, Y., Shlens, J., & Toshev, A. (2023) · arXiv · doi:10.48550/arxiv.2304.04385 · Cited by 1
Han, W., Chen, H., Kan, M., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.12798 · Cited by 1
Research works printed but not linked: 2
- Hu, LoRA: Low-rank adaptation of large language models
- 10.5555/3045118.3045167 · doi:10.5555/3045118.3045167
Cited by
MARIA: A multimodal transformer model for incomplete healthcare data
Caruso, C. M., Soda, P., & Guarrasi, V. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110843 · Cited by 32
Yang, Z., He, Q., Yu, M., Du, N., & Lu, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103571 · Cited by 37
Multimodal large model pretraining, adaptation and efficiency optimization
Ji, L., Xiao, S., Feng, J., Gao, W., & Zhang, H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129138 · Cited by 19
Quantum Federated Learning for Multimodal Data: A Modality-Agnostic Approach
Pokharel, A., Rahman, R., Morris, T., & Nguyen, D. C. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00059 · Cited by 15
“If these missing modalities are not properly addressed, they can significantly degrade the overall performance of the system [41].”
Uncertain Multimodal Intention and Emotion Understanding in the Wild
Yang, Q., Shi, Q., Wang, T., & Ye, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02300 · Cited by 29
Metadata
- Authors
- Reza, M. K., Prater-Bennette, A., & Asif, M. S.
- Affiliations
- University of California, Riverside (US)
- United States Air Force Research Laboratory (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2025-02
- Volume
- 47
- Issue
- 2
- Pages
- 742-754
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4403296843
- Semantic Scholar
- 36d1466dc91b3935391aeea68e5db457fb2c62fb
- Cited by
- 78
- References
- 97
- Citations by year
- 2026: 55 · 2025: 17
- FWCI
- 8.46
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Air Force Office of Scientific Research (FA9550-21-1-0330)
- NSF (CCF-2046293)
- Indiana Jetstream2 (CIS220128)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal learning, missing modalities, missing data robustness, parameter-efficient fine-tuning, feature modulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.