The DOI matches its DataCite record.
Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference
Open accessAbstract
Multimodal large language models (MLLMs) demand considerable computations for inference due to the extensive parameters and the additional input tokens needed for visual information representation. Herein, we introduce Visual Tokens Withdrawal (VTW), a plug-and-play module to boost MLLMs for rapid inference. Our approach is inspired by two intriguing phenomena we have observed: (1) the attention sink phenomenon that is prevalent in LLMs also persists in MLLMs, suggesting that initial tokens and nearest tokens receive the majority of attention, while middle vision tokens garner minimal attention in deep layers; (2) the presence of information migration, which implies that visual information is transferred to subsequent text tokens within the first few layers of MLLMs. As per our findings, we conclude that vision tokens are unnecessary in the deep layers of MLLMs. Thus, we strategically withdraw them at a certain layer, enabling only text tokens to engage in subsequent layers. To pinpoint the ideal layer for VTW, we initially analyze a limited set of tiny datasets and choose the first layer that meets the Kullback-Leibler divergence criterion. Our VTW approach can cut computational overhead by over 40\% across diverse multimodal tasks while maintaining performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In language models, token reduction has commonly been implemented through early-exit mechanisms and token-skipping strategies, which reduce the number of intermediate tokens processed and thus lower computational overhead.
“In language models, token reduction has commonly been implemented through early-exit mechanisms and token-skipping strategies [87, 149], which reduce the number of intermediate tokens processed and thus lower computational overhead.”
From Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality (2025) · cites this paper as [87]
The Vision Token Reduction section (page unavailable) says pruning and merging reduce token counts, inference time, and memory use. The Introduction (page unavailable) describes withdrawing visual tokens after a layer so only text tokens are processed in deeper layers. The paper does not establish that early-exit mechanisms and token-skipping strategies are commonly used for token reduction in language models generally; its discussion concerns visual tokens in multimodal models. (LLM, conf=0.82)
Experimental — accuracy not yet measuredEmpirical studies show that beyond the first few layers, many image tokens contribute little new information.
“② Visual Redundancy: Empirical studies [87, 149] show that beyond the first few layers, many image tokens contribute little new information,”
From Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality (2025) · cites this paper as [87]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 37
- Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 Technical Report. arXiv:2303.08774. · arXiv
- Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966. · arXiv
- Bolya, D.; Fu, C.-Y.; Dai, X.; Zhang, P.; Feichtenhofer, C.; and Hoffman, J. 2023. Token Merging: Your ViT But Faster. In ICLR, 1-20.
- Cao, J.; Ye, P.; Li, S.; Yu, C.; Tang, Y.; Lu, J.; and Chen, T. 2024. MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Trans- former. arXiv:2403.02991. · arXiv
- Chen, L.; Zhao, H.; Liu, T.; Bai, S.; Lin, J.; Zhou, C.; and Chang, B. 2024. An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision- Language Models. arXiv:2403.06764. · arXiv
- Chen, M.; Lin, M.; Li, K.; Shen, Y.; Wu, Y.; Chao, F.; and Ji, R. 2023a. CF-ViT: A General Coarse-to-Fine Method for Vision Transformer. In AAAI, 7042-7052.
- Chen, M.; Shao, W.; Xu, P.; Lin, M.; Zhang, K.; Chao, F.; Ji, R.; Qiao, Y.; and Luo, P. 2023b. DiffRate : Differen- tiable Compression Rate for Efficient Vision Transformers. In ICCV, 17164-17174.
- Dao, T.; Fu, D.; Ermon, S.; Rudra, A.; and Ré, C. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS, 16344-16359.
- Duan, H.; Yang, J.; Qiao, Y.; Fang, X.; Chen, L.; Liu, Y.; Dong, X.; Zang, Y.; Zhang, P.; Wang, J.; Lin, D.; and Chen, K. 2024. VLMEvalKit: An Open-Source Toolkit for Evalu- ating Large Multi-Modality Models. arXiv:2407.11691. · arXiv
- Fu, C.; Chen, P.; Shen, Y.; Qin, Y.; Zhang, M.; Lin, X.; Yang, J.; Zheng, X.; Li, K.; Sun, X.; Wu, Y.; and Ji, R. 2023. MME: A Comprehensive Evaluation Benchmark for Multi- modal Large Language Models. arXiv:2306.13394. · arXiv
- Fu, C.; Dai, Y.; Luo, Y.; Li, L.; Ren, S.; Zhang, R.; Wang, Z.; Zhou, C.; Shen, Y.; Zhang, M.; et al. 2024. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. arXiv:2405.21075. · arXiv
- Guo, Z.; Xu, R.; Yao, Y.; Cui, J.; Ni, Z.; Ge, C.; Chua, T.-S.; Liu, Z.; and Huang, G. 2024. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. In ECCV, 390- 406.
- Jang, Y.; Song, Y.; Yu, Y.; Kim, Y.; and Kim, G. 2017. Tgif- qa: Toward spatio-temporal reasoning in visual question an- swering. In CVPR, 2758-2766.
- Kembhavi, A.; Salvato, M.; Kolve, E.; Seo, M.; Hajishirzi, H.; and Farhadi, A. 2016. A Diagram is Worth A Dozen Images. In ECCV, 235-251.
- Lai, X.; Tian, Z.; Chen, Y.; Li, Y.; Yuan, Y.; Liu, S.; and Jia, J. 2023. LISA: Reasoning Segmentation via Large Lan- guage Model. arXiv:2308.00692. · arXiv
- Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023a. Blip-2: Boot- strapping Language-Image Pre-Training with Frozen Image Encoders and Large Language Models. In ICML, 19730- 19742.
- Li, Y.; Du, Y.; Zhou, K.; Wang, J.; Zhao, W. X.; and Wen, J.- R. 2023b. Evaluating Object Hallucination in Large Vision- Language Models. arXiv:2305.10355. · arXiv
- Li, Y.; Wang, C.; and Jia, J. 2023. LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models. arXiv:2311.17043. · arXiv
- Liang, Y.; Chongjian, G.; Tong, Z.; Song, Y.; Wang, J.; and Xie, P. 2022. EViT: Expediting Vision Transformers via To- ken Reorganizations. In ICLR, 1-21.
- Lin, B.; Zhu, B.; Ye, Y.; Ning, M.; Jin, P.; and Yuan, L. 2023a. Video-Llava: Learning United Visual Representation by Alignment Before Projection. arXiv:2311.10122. · arXiv
- Lin, B.; Zhu, B.; Ye, Y.; Ning, M.; Jin, P.; and Yuan, L. 2023b. Video-LLaVA: Learning United Visual Represen- tation by Alignment Before Projection. arXiv:2311.10122. · arXiv
- Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2023a. Improved Base- lines with Visual Instruction Tuning. arXiv:2310.03744. · arXiv
- Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023b. Visual Instruc- tion Tuning. In NeurIPS, 1-19.
- Liu, Y.; Duan, H.; Zhang, Y.; Li, B.; Zhang, S.; Zhao, W.; Yuan, Y.; Wang, J.; He, C.; Liu, Z.; et al. 2023c. MM- Bench: Is Your Multi-modal Model an All-around Player? arXiv:2307.06281. · arXiv
- Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.- C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to Explain: Multimodal Reasoning via Thought Chains for Sci- ence Question Answering. In NeurIPS, 2507-2521.
- Marin, D.; Chang, J.-H. R.; Ranjan, A.; Prabhu, A.; Raste- gari, M.; and Tuzel, O. 2021. Token Pooling in Vision Trans- formers. arXiv:2110.03860. · arXiv
- Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transfer- able Visual Models From Natural Language Supervision. In ICML, 8748-8763.
- Rao, Y.; Zhao, W.; Liu, B.; Lu, J.; Zhou, J.; and Hsieh, C.-J. 2021. Dynamicvit: Efficient Vision Transformers with Dy- namic Token Sparsification. In NeurIPS, 13937-13949.
- Shang, Y.; Cai, M.; Xu, B.; Lee, Y. J.; and Yan, Y. 2024. LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models. arXiv:2403.15388. · arXiv
- Shi, D.; Tao, C.; Rao, A.; Yang, Z.; Yuan, C.; and Wang, J. 2023. Crossget: Cross-guided Ensemble of Tokens for Accelerating Vision-Language Transformers. arXiv:2305.17455. · arXiv
- Team, G.; Anil, R.; Borgeaud, S.; Wu, Y.; Alayrac, J.-B.; Yu, J.; Soricut, R.; Schalkwyk, J.; Dai, A. M.; Hauth, A.; et al. 2023. Gemini: A Family Of Highly Capable Multimodal Models. arXiv:2312.11805. · arXiv
- Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023a. Llama: Open and Efficient Foun- dation Language Models. arXiv:2302.13971. · arXiv
- Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023b. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288. · arXiv
- Wu, X.; Zeng, F.; Wang, X.; Wang, Y.; and Chen, X. 2023. PPT: Token Pruning and Pooling for Efficient Vision Trans- formers. arXiv:2310.01812. · arXiv
- Xiao, G.; Tian, Y.; Chen, B.; Han, S.; and Lewis, M. 2023. Efficient Streaming Language Models with Attention Sinks. arXiv:2309.17453. · arXiv
- Xu, Y.; Zhang, Z.; Zhang, M.; Sheng, K.; Li, K.; Dong, W.; Zhang, L.; Xu, C.; and Sun, X. 2022. Evo-Vit: Slow-Fast Token Evolution for Dynamic Vision Transformer. In AAAI, 2964-2972.
- Yue, X.; Ni, Y.; Zhang, K.; Zheng, T.; Liu, R.; Zhang, G.; Stevens, S.; Jiang, D.; Ren, W.; Sun, Y.; et al. 2023. Mmmu: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert Agi. arXiv:2311.16502. · arXiv
Cited by
Sorted by citations
Kong, Z., Li, Y., Zeng, F., Xin, L., Messica, S., Lin, X., Zhao, P., Kellis, M., Tang, H., & Zitnik, M. (2025) · arXiv · doi:10.48550/arxiv.2505.18227 · Cited by 51
“Recent work has explored visual token pruning by addressing attention inefficiencies of deep trans-former layers in MLLMs [4, 8, 87, 114].”
Metadata
- Authors
- Lin, Z., Lin, M., Lin, L., & Ji, R.
- Affiliations
- Xiamen University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Volume
- 39
- Issue
- 5
- Pages
- 5334-5342
- ISSN
- 2159-5399 (ISSN-L)
- arXiv
- 2405.05803
- OpenAlex
- W4409367031
- Cited by
- 31
- References
- 0
- Citations by year
- 2026: 23 · 2025: 8
- FWCI
- 23.43
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62072387, 62025603, 62072389, No. 62025603, 62002305, 2022J06001, 62272401, U23A20383, U21A20472, 62176222, 62176226, U22B2051, U21B2037, 62176223, 62072386)
- Natural Science Foundation of Fujian Province (2021J06003, 2022J06001)
- National Science and Technology Major Project
- National Science Fund for Distinguished Young Scholars (No. 62025603, 62025603)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- multimodal large language models, computational efficiency, layer pruning, Kullback-Leibler divergence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
