Efficient Multimodal Fusion via Interactive Prompting
Open accessAbstract
Large-scale pre-training has brought unimodal fields such as computer vision and natural language processing to a new era. Following this trend, the size of multimodal learning models constantly increases, leading to an urgent need to reduce the massive computational cost of finetuning these models for downstream tasks. In this paper, we propose an efficient and flexible multimodal fusion method, namely PMF, tailored for fusing unimodally pretrained transformers. Specifically, we first present a modular multimodal fusion framework that exhibits high flexibility and facilitates mutual interactions among different modalities. In addition, we disentangle vanilla prompts into three types in order to learn different optimizing objectives for multimodal learning. It is also worth noting that we propose to add prompt vectors only on the deep layers of the unimodal transformers, thus significantly reducing the training memory usage. Experiment results show that our proposed method achieves comparable performance to several other multimodal finetuning methods with less than 3% trainable parameters and up to 66% saving of training memory usage.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prompt-based multimodal fusion enables bidirectional interaction among different modalities through a 2-stream structure.
“Prompt-based multimodal fusion [143] is one such framework that enables bidirectional interaction among different modalities through a 2-stream structure, typically involving parallel construction of the multimodal model through pretrained language and image models.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [143]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3× · see citing works
“Vision Transformer (ViT) [9] adapts the Transformer [35] architecture with minimum modifications.”
MethodJakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1× · see citing works
“Unless otherwise noted, we use an ImageNet-21k [7] pre-trained vit-base model for the vision encoder and a bert-base-uncased model for the language encoder in all experiments.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 1× · see citing works
“Recent years have witnessed the great success of largescale pretrained language models [8,31,32] and visual models [6,10,23,39], leading to a surge of pretrained multimodal models [13, 14, 43, 47, 48] trying to align different modalities.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3× · see citing works
“For methods that prompt multimodally pretrained models for unimodal tasks, many recent works apply prompt tuning to pretrained vision-language models ( i.e. CLIP [30]) for unimodal vision tasks [3, 14, 25, 46, 47].”
BackgroundMethodKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 1× · see citing works
“Recent years have witnessed the great success of largescale pretrained language models [8,31,32] and visual models [6,10,23,39], leading to a surge of pretrained multimodal models [13, 14, 43, 47, 48] trying to align different modalities.”
BackgroundSelf-citationExploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01549 · Cited by 3,725 · cited 1× · see citing works
“Recent years have witnessed the great success of largescale pretrained language models [8,31,32] and visual models [6,10,23,39], leading to a surge of pretrained multimodal models [13, 14, 43, 47, 48] trying to align different modalities.”
BackgroundLearning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304 · cited 3× · see citing works
“Recent years have witnessed the great success of largescale pretrained language models [8,31,32] and visual models [6,10,23,39], leading to a surge of pretrained multimodal models [13, 14, 43, 47, 48] trying to align different modalities.”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
(2011) · Cited by 2,752 · see citing works
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 3× · see citing works
“More recently, prompting tuning [17, 19, 21, 22, 29] is proposed to address this problem by freezing all parameters of a pretrained model while tuning only the continuous prompts.”
BackgroundViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · cited 1× · see citing works
“Another important line of multimodal fusion work is through pretraining with the large-scale datasets, mostly achieved by self-supervised learning algorithms [1, 11, 24, 30, 36, 37, 45].”
BackgroundScaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · cited 3× · see citing works
“Another important line of multimodal fusion work is through pretraining with the large-scale datasets, mostly achieved by self-supervised learning algorithms [1, 11, 24, 30, 36, 37, 45].”
BackgroundMethodSelf-citationJia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., & Lim, S.-N. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19827-4_41 · Cited by 1,965 · see citing works
Conditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,918 · see citing works
CoCa: Contrastive Captioners are Image-Text Foundation ModelsPreprint
Yu, J.-H., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., & Wu, Y.-H. (2022) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2205.01917 · Cited by 1,899 · cited 1× · see citing works
“Another important line of multimodal fusion work is through pretraining with the large-scale datasets, mostly achieved by self-supervised learning algorithms [1, 11, 24, 30, 36, 37, 45].”
BackgroundFlamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555 · cited 1× · see citing works
“Another important line of multimodal fusion work is through pretraining with the large-scale datasets, mostly achieved by self-supervised learning algorithms [1, 11, 24, 30, 36, 37, 45].”
BackgroundLiu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2023) · AI Open · doi:10.1016/j.aiopen.2023.08.012 · Cited by 1,124 · see citing works
Multimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · doi:10.5555/2627435.2697059 · Cited by 1,070 · see citing works
MaPLe: Multi-modal Prompt Learning
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., & Khan, F. S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01832 · Cited by 1,011 · cited 3× · see citing works
“Recent years have witnessed the great success of largescale pretrained language models [8,31,32] and visual models [6,10,23,39], leading to a surge of pretrained multimodal models [13, 14, 43, 47, 48] trying to align different modalities.”
BackgroundMultimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. (2014) · Cited by 593 · see citing works
Prompting Visual-Language Models for Efficient Video Understanding
Ju, C., Han, T., Zheng, K., Zhang, Y., & Xie, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_7 · Cited by 423 · see citing works
ActBERT: Learning Global-Local Video-Text Representations
Zhu, L., & Yang, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00877 · Cited by 414 · cited 1× · see citing works
“Recent years have witnessed the great success of large-scale pretrained language models [8,31,32] and visual models [6,10,23,39], leading to a surge of pretrained multimodal models [13, 14, 43, 47, 48] trying to align different modalities.”
BackgroundSelf-citationAttention Bottlenecks for Multimodal Fusion
Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., & Sun, C. (2021) · arXiv · doi:10.48550/arxiv.2107.00135 · Cited by 400 · see citing works
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and TasksPreprint
Liu, X., Ji, K., Fu, Y., Tam, W. L., Du, Z., Yang, Z., & Tang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07602 · Cited by 386 · cited 2× · see citing works
“More recently, prompting tuning [17, 19, 21, 22, 29] is proposed to address this problem by freezing all parameters of a pretrained model while tuning only the continuous prompts.”
BackgroundMultimodal Few-Shot Learning with Frozen Language Models
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., & Hill, F. (2021) · arXiv · doi:10.48550/arxiv.2106.13884 · Cited by 355 · cited 3× · see citing works
“Our work is in line with deep learning-based multimodal fusion strategies [16, 18, 26, 28, 33, 34, 44].”
BackgroundResultLearning How to Ask: Querying LMs with Mixtures of Soft Prompts
Qin, G., & Eisner, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.410 · Cited by 342 · see citing works
AutoFormer: Searching Transformers for Visual Recognition
Chen, M., Peng, H., Fu, J., & Ling, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01205 · Cited by 323 · cited 1× · see citing works
“In this section, we experiment with automatic fusion structure search via AutoFormer [5].”
MethodYang, Y. P., Zhuang, Y., & Pan, Y. (2021) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2100463 · Cited by 310 · cited 1× · see citing works
“Our work is in line with deep learning-based multimodal fusion strategies [16, 18, 26, 28, 33, 34, 44].”
ResultSelf-citationGated Multimodal Units for Information FusionPreprint
Arévalo, J., Solorio, T., Montes-y-Gómez, M., & Gonzalez, F. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01992 · Cited by 302 · cited 8× · see citing works
“All experiments in this section are conducted on the MM-IMDB dataset.”
BackgroundMethodInfluentialMulti-sensor management for information fusion: issues and approaches
Xiong, N. N., & Svensson, P. (2002) · Information Fusion · doi:10.1016/s1566-2535(02)00055-6 · Cited by 297 · cited 1× · see citing works
“Multimodal fusion methods aim to simultaneously process the input of different modalities, such as audio-video [27], vision-language [2, 15], and inputs from different types of sensors [42], etc.”
BackgroundSupervised Multimodal Bitransformers for Classifying Images and TextPreprint
Kiela, D., Bhooshan, S., Firooz, H., Perez, E., & Testuggine, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02950 · Cited by 271 · cited 5× · see citing works
“Sharing the same architecture design with [15, 34], [20] uses prompt to align the feature extracted from raw images to the token space of pretrained language model.”
BackgroundMethodInfluentialLu, Y., Liu, J., Zhang, Y., Liu, Y., & Tian, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00514 · Cited by 262 · cited 1× · see citing works
“CLIP [30]) for unimodal vision tasks [3, 14, 25, 46, 47].”
BEVT: BERT Pretraining of Video Transformers
Wang, R., Chen, D., Wu, Z., Chen, Y., Dai, X., Liu, M., Jiang, Y.-G., Zhou, L., & Yuan, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01432 · Cited by 234 · cited 1× · see citing works
“Another important line of multimodal fusion work is through pretraining with the large-scale datasets, mostly achieved by self-supervised learning algorithms [1, 11, 24, 30, 36, 37, 45].”
BackgroundShaban, A., & Yousefi, S. (2024) · Springer optimization and its applications · doi:10.1007/978-3-031-53092-0_10 · Cited by 206 · see citing works
Recipe recognition with large multimodal food dataset
Wang, X., Kumar, D., Thome, N., Cord, M., & Precioso, F. (2015) · IEEE International Conference on Multimedia & Expo Workshops (ICMEW) · doi:10.1109/icmew.2015.7169757 · Cited by 199 · cited 2× · see citing works
“We conduct extensive experiments on three vision-language datasets: UPMC-Food101 [38], MM-IMDB [2], and SNLI-VE [41].”
BackgroundMethodVisual Entailment: A Novel Task for Fine-Grained Image UnderstandingPreprint
Xie, N., Lai, F., Doran, D., & Kadav, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06706 · Cited by 156 · see citing works
Domain Consensus Clustering for Universal Domain Adaptation
Li, G., Kang, G., Zhu, Y., Wei, Y., & Yang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00963 · Cited by 138 · cited 1× · see citing works
“Our work is in line with deep learning-based multimodal fusion strategies [16, 18, 26, 28, 33, 34, 44].”
ResultSelf-citationSemi-supervised Vision Transformers
Weng, Z., Yang, X., Li, A., Wu, Z., & Jiang, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20056-4_35 · Cited by 39 · see citing works
Modular and Parameter-Efficient Multimodal Fusion with Prompting
Liang, S., Zhao, M., & Schuetze, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.234 · Cited by 33 · see citing works
Mañas, O., Rodríguez, P., Ahmadi, S., Nematzadeh, A., Goyal, Y., & Agrawal, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.185 · Cited by 29 · see citing works
Prompt Tuning for Generative Multimodal Pretrained ModelsPreprint
Yang, H., Lin, J., An, Y., Wang, P., Zhou, C., & Yang, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.02532 · Cited by 18 · cited 5× · see citing works
“Recent years have witnessed the great success of largescale pretrained language models [8,31,32] and visual models [6,10,23,39], leading to a surge of pretrained multimodal models [13, 14, 43, 47, 48] trying to align different modalities.”
BackgroundMethodInfluentialMañas, O., Rodríguez, P., Ahmadi, S., Nematzadeh, A., Goyal, Y., & Agrawal, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.07179 · Cited by 7 · cited 2× · see citing works
“Our work is in line with deep learning-based multimodal fusion strategies [16, 18, 26, 28, 33, 34, 44].”
BackgroundResultBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
3 of the 53 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Prompt-based multimodal fusion [143] is one such framework that enables bidirectional interaction among different modalities through a 2-stream structure, typically involving parallel construction of the multimodal model through pretrained language and image models.”
Li, Y., Wang, H., & Sun, M. (2023) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2023.109850 · Cited by 142 · see citing works
Self-citationMultimodal Alignment and Fusion: A Survey
Li, S., & Tang, H. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02667-1 · Cited by 191 · see citing works
“Prompt-based multimodal fusion (PMF) introduces deep-layer prompts that interact across pretrained unimodal Transformers, achieving comparable performance to full fine-tuning [101].”
Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion
Gao, Z., Jiang, X., Xu, X., Shen, F., Li, Y., & Shen, H. T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02538 · Cited by 63 · see citing works
“Moreover, we also make fair comparisons with the widely adopted simple multimodal fusion strategies i.e. , Concat and Late Fusion, as well as recent well-designed multimodal fusion strategies on the bi-modal datasets, including TMC [3], ITIN [47], MMBT [2], PMF [4], MVCN [48], and QMF [5].”
MethodSelf-citationIcoCap: Improving Video Captioning by Compounding Images
Liang, Y., Zhu, L., Wang, X., & Yang, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3322329 · Cited by 35 · see citing works
“Video Representation: Representation of video [5], [10], [11], [12], [13], [14] is a long-standing problem in the representation learning [15], [16], [17], [18].”
BackgroundSelf-citationLearning Visual Prompt for Gait Recognition
Ma, K., Fu, Y., Cao, C., Hou, S., Huang, Y., & Zheng, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00063 · Cited by 34 · see citing works
“In recent years, there has been a notable surge in prompt-based research [31, 38, 57, 62] within the field of computer vision.”
Memory-Based Cross-Modal Semantic Alignment Network for Radiology Report Generation
Tao, Y., Ma, L., Yu, J., & Zhang, H. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3393018 · Cited by 31 · see citing works
“Some previous works [36], [37] also explored the combination of additional learnable parameters and transformer models.”
BackgroundMethodLi, G., Kang, G., Wang, X., Wei, Y., & Yang, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01960 · Cited by 24 · see citing works
Self-citationPsychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity
Quan, R., Wang, W., Tian, Z., Ma, F., & Yang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00030 · Cited by 24 · see citing works
“With the release of multimodal vision-language models [31, 34, 37, 38, 47, 61, 68], diffusion models [24, 50, 56–59, 71], and large-scale large-scale fMRI datasets [ 2 , 7, 25, 66] , This work represents the initial exploration of the application of MoE in the field of reconstructing images from…”
MethodSelf-citationDango: A Mixed-Initiative Data Wrangling System using Large Language Model
Chen, W., Tong, W., Case, A., & Zhang, T. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714135 · Cited by 20 · see citing works
Wang, K., Tian, Y., & Hatzinakos, D. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00190 · Cited by 17 · see citing works
“Inspired by the remarkable generalization ability of such transformer-based architectures [39], some works [4, 9, 22, 32, 36, 43, 48] explore the knowledge transfer from pre-trained foundation models into different downstream tasks by parameter-efficient transfer learning (PETL) [12, 13, 15], where…”
BackgroundUnlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning
Deng, W., Thrampoulidis, C., & Li, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00582 · Cited by 15 · see citing works
On Disentanglement of Asymmetrical Knowledge Transfer for Modality-Task Agnostic Federated Learning
Chen, J., & Zhang, A. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.29010 · Cited by 14 · see citing works
“In AGI, there has been a growing concern regarding data privacy during the pre-training and fine-tuning phases of Multimodal Foundation Models.”
BackgroundFashionERN: Enhance-and-Refine Network for Composed Fashion Image Retrieval
Chen, Y., Zhong, H., He, X., Peng, Y., Zhou, J., & Cheng, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i2.27885 · Cited by 13 · see citing works
Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation
Cai, J., Su, J. K., Li, Q., Yang, W., Wang, S., Zhao, T., He, S., & Liu, W. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00990 · Cited by 12 · see citing works
“Popular PEFT methods include adapter-based tuning [8, 20, 22] and prompt tuning [29, 32, 56].”
Hierarchical Multi-Modal Prompting Transformer for Multi-Modal Long Document Classification
Liu, T., Hu, Y., Gao, J., Sun, Y., & Yin, B. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3366935 · Cited by 11 · see citing works
“For the multi-modal baselines, we consider three categories of methods: multi-modal classification [53], [54], [55], [58], multi-modal emotion analysis [56], [57], and multi-modal transformer [9], [15].”
BackgroundMethodInfluentialMulti-modal long document classification based on Hierarchical Prompt and Multi-modal Transformer
Liu, T., Hu, Y., Gao, J., Wang, J., Sun, Y., & Yin, B. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106322 · Cited by 10 · see citing works
SAGE-Net: Single-layer augmented gated encoder network for efficient multimodal sentiment analysis
Zhou, J., Kang, X., Ding, W., Wang, L., Ding, F., Matsumoto, K., Zhang, C., & Chi, H. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113665 · Cited by 10 · see citing works
Towards Asynchronous Multimodal Signal Interaction and Fusion via Tailored Transformers
Yang, D., Kuang, H., Yang, K., Li, M., & Zhang, L. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3409211 · Cited by 9 · see citing works
“The rich information provides valuable clues to understanding human intentions [2], [3], [4], [5], [6], [7], [8], [9], [10], [11].”
BackgroundLong, K., Xie, G., Ma, L., Li, Q., Huang, M., Lv, J., & Lu, Z. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3599673 · Cited by 8 · see citing works
“• PMF-large [63] is an e ffi cient multimodal fusion method devoted to fusing unimodal pre-trained transformers.”
Enhancing the Accuracy of an Image Classification Model Using Cross-Modality Transfer Learning
Liu, J., Chui, K. T., & Lee, L. (2023) · Electronics · doi:10.3390/electronics12153316 · Cited by 7 · see citing works
MoS2: Mixture of Scale and Shift Experts for Text-Only Video Captioning
Jia, H., Xu, Y., Zhu, L., Chen, G., Wang, Y., & Yang, Y. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680686 · Cited by 7 · see citing works
“Toaddressthischallenge,researchersexploresparameter-efficient learningmethodsthatadjustonlyasubsetofparameters[15,16, 23,27,28,47,64,67].”
Self-citationPan, L., Han, X., Liu, X., & Liu, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3599563 · Cited by 6 · see citing works
“For multimodal classification tasks, the compared methods include Late Fusion [44], ConcatBERT and several robust dynamic fusion approaches: MMBT [45], TMC [35], PMF [46], MVCN [47], QMF [34], HyperDUM [29] and EAU [36].”
Visual-guided hierarchical iterative fusion for multi-modal video action recognition
Zhang, B., Zhang, Y., Zhang, J., Sun, Q., Wang, R., & Zhang, Q. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.10.003 · Cited by 6 · see citing works
SDPT: Synchronous Dual Prompt Tuning for Fusion-Based Visual-Language Pre-trained Models
Zhou, Y., Wu, Y., Saiyin, J., Wei, B., Lai, M., Chang, E., & Xu, Y. L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72967-6_19 · Cited by 6 · see citing works
End-to-End Open-Vocabulary Video Visual Relationship Detection Using Multi-Modal Prompting
Wang, Y., Wu, X., Yang, S., & Luo, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3561366 · Cited by 5 · see citing works
Multimodal Isotropic Neural Architecture with Patch Embedding
Truchan, H., Naumov, E., Abedin, R., Palmer, G., & Ahmadi, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8079-6_14 · Cited by 5 · see citing works
MTFR: An universal multimodal fusion method through Modality Transfer and Fusion Refinement
Guo, X., Tian, S., Yu, L., He, X., & Wang, Z. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108844 · Cited by 5 · see citing works
Xie, X.-D., Wu, J., Xiang, M., Tang, J.-T., & Sheng, Y.-X. (2025) · Journal of King Saud University: Computer and Information Sciences · doi:10.1007/s44443-025-00185-1 · Cited by 9 · see citing works
“PMF utilizes frozen ViT-Base and BERT-Base encoders, with three types of deep-layer prompts—query, query-con-text, and fusion-context—inserted to facilitate cross-modal alignment.”
Shi, H., Zhang, X., Cheng, N., Zhang, Y., Yu, J., Xiao, J., & Wang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5669-8_32 · Cited by 4 · see citing works
FusionSAM: Visual Multi-Modal Learning with Segment Anything Model
Li, D., Xie, W., Cao, M., Wang, Y., Zhang, Y., Fang, L., Li, Y., & Xu, C. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736973 · Cited by 4 · see citing works
“As an alternative, Transformer architectures, with their attention mechanisms and ability to model long-range dependencies, facilitate better global fusion and utilization of complementary information[24, 36, 37, 61, 65].”
Self-citationCascaded cross-modal transformer for audio–textual classification
Ristea, N.-C., Anghel, A., & Ionescu, R. T. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10869-1 · Cited by 4 · see citing works
“A wide variety of frameworks have been proposed in the literature, including approaches based on early, intermediate, and late fusion (Boulahia et al. 2021; Huang et al. 2020a, 2020b; Wang et al. 2020; Li et al. 2023; Pawłowski et al. 2023; Xu et al. 2023; Singh et al. 2021).”
Multimodal dynamic fusion framework: Multilevel feature fusion guided by prompts
Pan, L., & Wu, H. (2024) · Expert Systems · doi:10.1111/exsy.13668 · Cited by 3 · see citing works
Multimodal Rumor Detection via Multimodal Prompt Learning
Chen, F., Li, X., Li, Z., Zhou, C., & Sheng, J. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650974 · Cited by 3 · see citing works
“Prompts, through iterative learning, adeptly capture nuanced contextual signals crucial for rumor detection tasks, thereby enhancing model fusion [16].”
BackgroundDeep metric loss for multimodal learning
Moon, S., & Lee, H. (2025) · Machine Learning · doi:10.1007/s10994-024-06709-6 · Cited by 3 · see citing works
“…et al., 2019; Tsai et al., 2019; Su et al., 2020; Joze et al., 2020; Siddiqui & Javaid, 2020; Del-brouck et al., 2020; Sood et al., 2021; Saad et al., 2021; Chen et al., 2021; Moon & Lee, 2022; Chumachenko et al., 2022; Wang et al., 2022a, b; Yin et al., 2022; Li et al., 2023; Duan et al., 2023).”
MAAL: Multimodality-Aware Autoencoder-based Affordance Learning for 3D Articulated Objects
Liang, Y., Wang, X., Zhu, L., & Yang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00027 · Cited by 3 · see citing works
“Multi-modal Learning: Many tasks (e.g., VQA [3, 24], gesture generation [50, 25], video representation [23]) involve multi-modal inputs and require the network to handle the multi-modal problems [22, 44].”
BackgroundSelf-citationCoPL:Parameter-Efficient Collaborative Prompt Learning for Audio-Visual Tasks
Zhao, Y., Xi, W., Cui, Y., Bai, G., Liu, X., & Zhao, J. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681492 · Cited by 3 · see citing works
“PFM [26] proposed an efficient and flexible multi-modal fusion approach, which employ cross-attention operation in pretrained encoder layers to learn mutual interactions.”
BackgroundMulti-spectral fusion power equipment fault recognition based on prompt learning
Yiyang, Y. A. O., Zexing, D. U., Zhou, G., & Wang, Q. (2025) · Xibei Gongye Daxue Xuebao/Journal of Northwestern Polytechnical University · doi:10.1051/jnwpu/20254320410 · Cited by 3 · see citing works
Li, W., Liang, M., Fan, R., & Li, Y. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733386 · Cited by 2 · see citing works
Self-citationHybrid fusion framework for multimodal data integration
Wang, D., Liu, W., Nie, X., Liu, Y., Zhang, X., & Zhou, W. (2026) · Array · doi:10.1016/j.array.2026.100736 · Cited by 2 · see citing works
Progressive Multimodal Pivot Learning: Towards Semantic Discordance Understanding as Humans
Fang, J.-F., Wang, W., Luo, T., Huang, Y., & Lv, F. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679524 · Cited by 2 · see citing works
“(b) Models introducing interactive units (e.g., attention bottlenecks in MBT [38] or interactive prompt tokens in PMF [26]) to implement efficient information exchange across modalities. pivot memory.”
InfluentialCoupleFER: Dynamic Cross-Modal Fusion via Prompt Learning for Improved 2D+3D FER
Li, H., Yang, H., & Huang, D. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3584264 · Cited by 2 · see citing works
“Notable techniques for multi-modal representation include self-supervised learning [18], cross-attention fusion layers [21], and prompt learning [22], [23].”
Text-Dominant Interactive Attention for Cross-Modal Sentiment Analysis
Zhang, Z., Yang, S., & Pan, H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8620-6_14 · Cited by 1 · see citing works
Liu, T., Hu, Y., Li, M., Yi, J., Chang, X., Gao, J., & Yin, B. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3537759 · Cited by 1 · see citing works
“PMF [46]: An efficient and flexible multimodal fusion method that decomposes vanilla prompts into three types to learn different optimization objectives for multimodal learning.”
Guo, Z., Cheng, X., Wu, Y., & Jin, T. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i16.33868 · Cited by 2 · see citing works
Clip4Vis: Parameter-free fusion for multimodal video recognition
Zheng, Q., He, M., Duan, J., Luo, G., Wu, P., Han, Y., Min, Q., Peng, C., & Zhang, P. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131046 · Cited by 1 · see citing works
Deng, Z., Xue, M., Xiong, S., Luo, C., & Luo, Y. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3641184 · Cited by 1 · see citing works
“…categorized into data-level (direct concatenation of raw inputs), feature-level (integration of intermediate embed-dings via attention or gating), decision-level (aggregation of modality-specific predictions), and hybrid (jointly exploiting feature and decision-level fusion) [24], [25], [26].”
Adaptive graph signal processing for robust multimodal fusion with dynamic semantic alignment
Karthikeya, K., Rajasekaran, A. S., Das, A. K., Vivekananda Bhat, K., & Pal, S. (2026) · Scientific Reports · doi:10.1038/s41598-026-44641-y · Cited by 1 · see citing works
Clara: A Cross-Modal Learning Framework for Enhanced Vulnerability Detection
Peng, X., Wang, S., Lin, B., Qin, Y., Chen, L., & Mao, X. (2026) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2026.3691741 · Cited by 1 · see citing works
“…that, for specialized tasks like vulnerability detection, modality-specific prompts can extract task-relevant information during optimization [41], while modal-interaction prompts, acting as queries, facilitate bidirectional knowledge exchange via self-attention mechanisms [42], promoting a…”
SDPT: Synchronous Dual Prompt Tuning for Visual-Language Pre-Trained Models
Zhou, Y., Wu, Y., Saiyin, J., Wei, B., & Xu, Y. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3692711 · Cited by 1 · see citing works
“Among these, some have proposed asynchronous approaches, adding learnable structures to both modal branches separately for dual-modal knowledge learning, such as DPT [17], APOLLO [18], and PMF [19].”
Li, Q., Chen, C., Cai, J., Huang, J. Y., Li, Y., & Liu, W. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3710197 · Cited by 1 · see citing works
“Representative PEFT techniques include adapter-based methods [12], [49] and prompt-based approaches [13], [54].”
Self-citationWan, L. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4884439
Bilingual Multimodal Graph Modeling for Text-Image Relation Inference
Zhang, D., Lu, W., Li, S., & Zhou, G. (2025) · Lecture notes in computer science · doi:10.1007/978-981-97-5555-4_28
Research on Human-Robot Collaboration Safety Model and Key Algorithms in Assembly Systems
Li, W., Feng, Z., Kong, D., & Xia, Z. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-2684-7_30
Towards Efficient and Secure Multimodal Misinformation Detection
Wei, J., Li, Y., Li, X., Zhao, M., Zhang, J., Liu, L., & Yang, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-00828-2_14
Self-citationEPIC: Efficient Prompt Interaction for Text-Image Classification
Yu, X., Sun, H., Ling, Z., Niu, Z., Bai, Z., Qin, R., Chen, Y.-W., & Lin, L. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210028
“We also propose a prompt-tuning CLIP without interactions between modalities, denoted as P-CLIP. c) Existing prompt-based methods for multimodal interaction, PromptFuse [8], BlindPrompt [8], and PMF [9] . d) Late fusion methods with a fusion module, [15]–[18], [24].”
InfluentialCoMPLe: Cross-Modal Hybrid Prompt Learning for End-to-End Multimodal Emotion Recognition
Feng, J., Zhao, Z., Zhou, L., Pu, Y., Jieping, Y. E., Xu, D., & Gu, J. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3638415
“As a result, the prompts fail to effectively facilitate multimodal interaction [33], [34].”
Research on Efficient Feature Extraction and Fusion Algorithms for Multimodal Tasks
Jiang, H., Yan, Z., & Ren, Y. (2025) · International Conference on Image Processing, Computer Vision and Machine Learning (ICICML) · doi:10.1109/icicml67980.2025.11333422
“The efficacy of these distinct extraction methods was validated experimentally using 10,000 images and their corresponding text [5].”
Task-Driven Generative Prompt Learning for Vision-Language Pretrained Model
Wu, Q., Zhang, X., Cheng, D., Xing, Y., Liang, G., Dang, K., & Zhang, Y. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3703499
“Beyond specializing in few-shot classification challenges, learnable prompts exhibit their adaptability across a broad spectrum of down-stream endeavors [36], [37], [38], [39], [40], [41], [42], [43], [44].”
Decoupled Sub-Feature Uncertainty Modeling for Robust Multimodal Representation Learning
Zhu, A., Hu, M., Xing, Y. T., & Tang, Y. (2026) · doi:10.1109/cvprf71909.2026.00665
“…and recent state-of-the-art methods, including HyCon [28], ConKI [50], ALMT [54], CENET [42], TETFN [43], MMBT [19], TMC [13], ITIN [59], PMF [22], MVCN [46], QMF [57], EAU [9], CubeMLP [36], EMT [37], GLoMo [60], DNMCN [3], MFMB [38], DEVA [47], MISA [14], Self-MM [49], MMIM [11], DMD [23],…”
- Authors
- Li, Y., Quan, R., Zhu, L., & Yang, Y.
- Affiliations
- University of Technology Sydney (AU)
- Zhejiang University (CN)
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 2604-2613
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386047824
- Cited by
- 60
- References
- 53
- Citations by year
- 2026: 8 · 2025: 22 · 2024: 24 · 2023: 6
- FWCI
- 6.42
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Australian Research Council (ARC) (DP200100938)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal fusion, parameter-efficient fine-tuning, PMF
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
