Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
Open accessAbstract
Although Multimodal Large Language Models (MLLMs) have demonstrated promising versatile capabilities, their performance is still inferior to specialized models on down-stream tasks, which makes adaptation necessary to enhance their utility. However, fine-tuning methods require indepen-dent training for every model, leading to huge computation and memory overheads. In this paper, we propose a novel setting where we aim to improve the performance of diverse MLLMs with a group of shared parameters optimized for a downstream task. To achieve this, we propose Transferable Visual Prompting (TVP), a simple and effective approach to generate visual prompts that can transfer to different models and improve their performance on downstream tasks after trained on only one model. We introduce two strategies to address the issue of cross-model feature corruption of existing visual prompting methods and enhance the transferabil-ity of the learned prompts, including 1) Feature Consistency Alignment: which imposes constraints to the prompted feature changes to maintain task-agnostic knowledge; 2) Task Semantics Enrichment: which encourages the prompted images to contain richer task-specific semantics with language guidance. We validate the effectiveness of TVP through ex-tensive experiments with 6 modern MLLMs on a wide vari-ety of tasks ranging from object recognition and counting to multimodal reasoning and hallucination correction.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The visual perception limitations of MLLMs affect their performance on visual understanding tasks, especially when facing small low-resolution objects.
“The visual perception limitations of MLLMs affect their performance on visual understanding tasks, especially when facing small low-resolution objects [54].”
From Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping (2024) · cites this paper as Zhang et al., 2024
The Introduction says MLLMs can perform unsatisfactorily on downstream visual tasks, and Section 2.1 says they fall short of specialized models on specific tasks. Neither passage establishes that their limitations are especially pronounced for small, low-resolution objects; searches for those terms found no such finding in our copy. Page numbers were not available. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningPreprint
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J.-Q., Wang, W., Li, B. A., Fung, P., & Hoi, S. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.06500 · Cited by 4,191 · see citing works
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304 · cited 1× · see citing works
“While popular works of prompt tuning like CoOp [65, 66], VPT [24] and MaPLe [27] operate soft prompts for both modalities at the early layers of the model, even at the embedding space, they are invalid under complete black-box conditions where only discrete texts and images are accessible for input.”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Self-citationLoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428 · see citing works
Parameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · see citing works
Food-101 – Mining Discriminative Components with Random Forests
Bossard, L., Guillaumin, M., & Gool, L. V. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_29 · Cited by 2,105 · see citing works
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., & Lim, S.-N. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19827-4_41 · Cited by 1,965 · see citing works
Conditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,918 · see citing works
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781 · see citing works
Parkhi, O., Vedaldi, A., Zisserman, A., & Jawahar, C. V. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248092 · Cited by 1,563 · see citing works
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555 · see citing works
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
Hendrycks, D., & Dietterich, T. (2019) · arXiv · doi:10.48550/arxiv.1903.12261 · Cited by 1,335 · see citing works
Domain Generalization: A Survey
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195549 · Cited by 1,315 · see citing works
Fine-Grained Visual Classification of AircraftPreprint
Maji, S., Rahtu, E., Kannala, J., Blaschko, M. B., & Vedaldi, A. (2013) · HAL · doi:10.48550/arxiv.1306.5151 · Cited by 1,250 · see citing works
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks
Dong, Y., Pang, T., Su, H., & Zhu, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00444 · Cited by 1,034 · see citing works
Self-citationGLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,028 · see citing works
MaPLe: Multi-modal Prompt Learning
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., & Khan, F. S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01832 · Cited by 1,011 · see citing works
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · see citing works
A survey on multimodal large language models
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024) · National Science Review · doi:10.1093/nsr/nwae403 · Cited by 893 · see citing works
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z., Yang, Z., & Tang, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.8 · Cited by 810 · see citing works
ChatGPT and Open-AI Models: A Preliminary Review
Roumeliotis, K. I., & Tselikas, N. D. (2023) · Future Internet · doi:10.3390/fi15060192 · Cited by 767 · see citing works
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682 · see citing works
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · cited 1× · see citing works
“Though effective in various fields, such as science [37] and biomedicine [33], full-parameter fine-tuning (FFT) is computationally demanding and storage-intensive, particularly for models with billions of parameters.”
BackgroundSelf-citationLLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · neural information processing systems · doi:10.52202/075280-1240 · Cited by 590 · see citing works
Self-citationEvaluating Object Hallucination in Large Vision-Language Models
Li, Y., Du, Y., Zhou, K., Wang, J., Xin, Z., & Wen, J.-R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.20 · Cited by 580 · see citing works
Miscible Displacement of Heavy West Sak Crude by Solvents in Slim Tube
Sharma, A. K., Patil, S., KAMATH, V. A., & Sharma, G. D. (1989) · SPE California Regional Meeting · doi:10.2118/18761-ms · Cited by 559 · see citing works
PandaGPT: One Model To Instruction-Follow Them AllPreprint
Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., & Cai, D. (2023) · Tsinghua Interdisciplinary Workshop on Logic, Language and Meaning · doi:10.48550/arxiv.2305.16355 · Cited by 474 · see citing works
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469 · see citing works
The hateful memes challenge: Detecting hate speech in multimodal memes
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D. (2020) · arXiv · doi:10.48550/arxiv.2005.04790 · Cited by 352 · see citing works
GLM-130B: An Open Bilingual Pre-trained Model
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02414 · Cited by 281 · see citing works
Self-citationSelfReg: Self-supervised Contrastive Regularization for Domain Generalization
Kim, D., Yoo, Y. F., Park, S., Kim, J., & Lee, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00948 · Cited by 277 · see citing works
Self-regulating Prompts: Foundational Model Adaptation without Forgetting
Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S. A., Yang, M., & Khan, F. S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01394 · Cited by 274 · see citing works
How Much Can CLIP Benefit Vision-and-Language Tasks?Preprint
Shen, S., Li, L. H., Tan, H. X., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., & Keutzer, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06383 · Cited by 272 · see citing works
Transferable Adversarial Perturbations
Zhou, W., Hou, X., Chen, Y., Tang, M., Huang, X., Gan, X., & Yang, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_28 · Cited by 256 · see citing works
Generalization error of ensemble estimators
Ueda, N., & Nakano, R. (2002) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1996.548872 · Cited by 235 · see citing works
L2 Regularization versus Batch and Weight NormalizationPreprint
van Laarhoven, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05350 · Cited by 195 · see citing works
Simple but Effective: CLIP Embeddings for Embodied AI
Khandelwal, A., Weihs, L., Mottaghi, R., & Kembhavi, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01441 · Cited by 172 · see citing works
Detecting and Preventing Hallucinations in Large Vision Language Models
Gunjal, A., Yin, J., & Bas, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29771 · Cited by 161 · see citing works
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., & Luo, P. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3507000 · Cited by 150 · see citing works
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Xu, L., Yang, J., Zheng, X., Li, K., Sun, X. W., et al. (2025) · neural information processing systems · doi:10.52202/085713-4899 · Cited by 148 · see citing works
Exploring Visual Prompts for Adapting Large-Scale Models
Bahng, H., Jahanian, A., Sankaranarayanan, S., & Isola, P. (2022) · arXiv · doi:10.48550/arxiv.2203.17274 · Cited by 129 · see citing works
Adversarial Reprogramming of Neural NetworksPreprint
Elsayed, G. F., Goodfellow, I., & Sohl‐Dickstein, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.11146 · Cited by 120 · see citing works
When Does Diversity Help Generalization in Classification Ensembles?
Bian, Y., & Chen, H. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3053165 · Cited by 92 · see citing works
Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
Liu, Y., Yao, Y., Ton, J.-F., Zhang, X., Guo, R., Cheng, H., Klochkov, Y., Taufiq, M. F., & Li, H. (2023) · arXiv · doi:10.48550/arxiv.2308.05374 · Cited by 79 · see citing works
Diversity-Aware Meta Visual Prompting
Huang, Q., Dong, X., Chen, D., Zhang, W., Wang, F., Hua, G., & Yu, N. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01047 · Cited by 67 · see citing works
Tsai, Y.-Y., Chen, P., & Ho, T.-Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.08714 · Cited by 58 · see citing works
Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
Luo, G., Zhou, Y., Ren, T., Chen, S. L., Sun, X., & Ji, R. (2023) · neural information processing systems · doi:10.52202/075280-1288 · Cited by 50 · see citing works
BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning
Oh, C., Hwang, H., Lee, H., Lim, Y., Jung, G., Jung, J., Choi, H., & Song, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02320 · Cited by 47 · see citing works
Potter, D. M. (2015) · AccessScience · doi:10.1036/1097-8542.731800 · Cited by 19 · see citing works
How Robust is Google's Bard to Adversarial Image Attacks?Preprint
Dong, Y., Chen, H., Chen, J., Fang, Z., Yang, X. H., Zhang, Y., Tian, Y., Su, H., & Zhu, J. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.11751 · Cited by 12 · see citing works
Self-citationUnleashing the Power of Visual Prompting At the Pixel LevelPreprint
Wu, J., Li, X., Chen, W. R., Wang, H., Yuille, A., Zhou, Y., & Xie, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.10556 · Cited by 11 · see citing works
Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models
Wu, Q., Yu, W., Zhou, Y., Huang, S., Sun, X., & Ji, R. (2023) · neural information processing systems · doi:10.52202/075280-1786 · Cited by 6 · see citing works
Quantifying Privacy Risks of Prompts in Visual Prompt LearningPreprint
Wu, Y., Wen, R., Michael Backes, A., Berrang, P., Humbert, M., Shen, Y., & Zhang, Y. (2023) · arXiv (Cornell University) · doi:10.22028/d291-47514 · Cited by 3 · see citing works
Self-citationSEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., & Shan, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.16125
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning
Zhao, H., Cai, Z., Si, S., Ma, X., An, K., Chen, L., Liu, Z., Wang, S., Han, W., & Chang, B. (2023) · arXiv · doi:10.48550/arxiv.2309.07915
Flagged first, then by citations
4 of the 69 references couldn’t be matched to a known paper.
Research works printed but not linked: 9
- Huanran Chen, Yichi Zhang, Yinpeng Dong, and Jun Zhu. Rethinking model ensemble in transfer-based adversarial at- tacks. arXiv preprint arXiv:2303.09105, 2023. 7 · arXiv
- Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial at- tacks with momentum. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9185-9193, 2018. 3, 7
- Dan Hendrycks and Thomas Dietterich. Benchmarking neu- ral network robustness to common corruptions and perturba- tions. In International Conference on Learning Representa- tions, 2019. 8, 2, 4
- Wenbo Hu, Yifan Xu, Y Li, W Li, Z Chen, and Z Tu. Bliva: A simple multimodal llm for better handling of text-rich vi- sual questions. arXiv preprint arXiv:2308.09936, 2023. 1, 3, 5 · arXiv
- Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. In Computer Vision -ECCV 2022, pages 709-727, 2022. 2, 3, 6, 8
- Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bis- sacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Work- shop on Deep Learning and Unsupervised Feature Learning 2011, 2011. 1, 5
- Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic for- getting in multimodal large language models. arXiv preprint arXiv:2309.10313, 2023. 1 · arXiv
- Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. Transfer visual prompt generator across llms. In Advances in Neural Information Processing Sys- tems, 2023. 2, 5, 7, 1
- Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xi- angqi Huang, Xiang Gan, and Yong Yang. Transferable ad- versarial perturbations. In Computer Vision -ECCV 2018, pages 452-467, 2018. 2, 3, 4
Cited by
Sorted by citations
A Survey of Attacks on Large Vision–Language Models: Resources, Advances, and Future Trends
Liu, D., Yang, M., Qu, X., Zhou, P., Cheng, Y., & Hu, W. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3592935 · Cited by 79 · see citing works
Video-Bench: Human-Aligned Video Generation Benchmark
Han, H., Li, S., Chen, J., Yuan, Y., Wu, Y., Deng, Y., Leong, C. T., Du, H., Fu, J., Li, Y., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01757 · Cited by 10 · see citing works
Multi-scale interaction network for multimodal entity and relation extraction
Zhang, Y., Li, G., Gao, H., & Dang, D. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121787 · Cited by 9 · see citing works
Self-citationVisual Position Prompt for MLLM Based Visual Grounding
Tang, W., Sun, Y., Gu, Q., & Li, Z. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3654372 · Cited by 9 · see citing works
Xiao, J., Wan, Y., Huo, Y., Wang, Z., Xu, X., Wang, W., Xu, Z., Wang, Y., & Lyu, M. R. (2024) · arXiv · doi:10.48550/arxiv.2411.03292 · Cited by 7 · see citing works
Liao, Y., Peng, C., Li, X., Wang, X., & Deng, Y. (2025) · IEEE Transactions on Power Delivery · doi:10.1109/tpwrd.2025.3586984 · Cited by 7 · see citing works
Evaluating the Adversarial Robustness of Vision-Language Models via Internal Feature Perturbations
Liu, C., Wang, Y., Cao, H., Liu, B., & Jiang, D. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3625396 · Cited by 3 · see citing works
Can large language models understand depth in monocular images without prior vision knowledge?
Xia, Z., Wu, T., & Wu, B. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111680 · Cited by 4 · see citing works
Multimodal Affect Perception With Large Language Model Enhancement Network
Yang, K., Luo, Y., Zhang, Z., Chen, C. L. P., & Zhang, T. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3607727 · Cited by 2 · see citing works
Route Optimization Reimagined: Multi-Modal Large Language Models for Next-Generation Vehicle Routing
Albalkhi, S. Y., Alotaibi, D. F., Dimitriou, T. D., & Ahmad, I. (2026) · IEEE Access · doi:10.1109/access.2026.3663141 · Cited by 2 · see citing works
CoolPrompt: Automatic Prompt Optimization Framework for Large Language Models
Kulin, N., Zhuravlev, V. N., Khairullin, A., Sitkina, A. N., & Muravyov, S. (2025) · Conference of Open Innovations Association (FRUCT) · doi:10.23919/fruct67853.2025.11239071 · Cited by 1 · see citing works
A multi-dimensional transfer learning framework for studying reward-guided behaviors across species
Liu, Y. M., Turnbull, A. G., Adeli, E., Zhao, G., Wang, K. H., & Lin, F. V. (2025) · Nature Mental Health · doi:10.1038/s44220-025-00547-8 · Cited by 1 · see citing works
Bridging radiology and pathology: domain-generalized cross-modal learning for clinical
Zhong, X., Gu, Z., Manimurugan, S., Li, M., Sun, H., Du, M., Chen, Q., & Jiang, G. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02423-w · Cited by 2 · see citing works
VisionCoach : Reinforcing Grounded Video Reasoning via Visual-Perception Prompting
Lee, D., Yu, S., Zhang, Y., & Bansal, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37432-5_35
Self-citationModel-free Domain Adaptation for Concealed Multimodal Large-Language Models
Mitsuzumi, Y., Kimura, A., & Kashima, H. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00126
Aspect-Oriented Prompt with Adaptive Cross-Modal Fusion for Multimodal Sentiment Analysis
Zhang, J., Li, Y., Hu, Y., Mao, X., Yu, F., Lee, L., Wang, F. L., & Yang, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0369-7_31
From Adversarial ML to Agentic Offense in AI-Enabled Threat Evolution
Mohapatra, H. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0413-5.ch001
- Authors
- Zhang, Y., Dong, Y., Zhang, S., Min, T., Su, H., & Zhu, J.
- Affiliations
- Tsinghua University (CN)
- Venue
- 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2024-06-16
- Pages
- 26552-26562
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4402727038
- Cited by
- 17
- References
- 69
- Citations by year
- 2026: 7 · 2025: 9 · 2024: 1
- FWCI
- 3.52
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Key Research and Development Program of China (2020AAA0I06302)
- NSFC (92370124,62350080,92248303,62276149,U2341228,62061136001,62076147)
- Tsinghua University
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal large language models, visual prompting, cross-model transfer, multimodal reasoning, hallucination correction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
