What matters when building vision-language models?
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
The growing interest in vision-language models (VLMs) has been driven by improvements in large language models and vision transformers. Despite the abundance of literature on this subject, we observe that critical decisions regarding the design of VLMs are often not justified. We argue that these unsupported decisions impede progress in the field by making it difficult to identify which choices improve model performance. To address this issue, we conduct extensive experiments around pre-trained models, architecture choice, data, and training methods. Our consolidation of findings includes the development of Idefics2, an efficient foundational VLM of 8 billion parameters. Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
MiniCPM-V: A GPT-4V Level MLLM on Your PhonePreprint
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.01800 · Cited by 1,628
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. (2024) · neural information processing systems · doi:10.52202/079017-2771 · Cited by 1,210
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
Li, F., Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., & Li, C. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2407.07895 · Cited by 1,008
DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal UnderstandingPreprint
Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.10302 · Cited by 996
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
Wu, H., Li, D., Chen, B., & Li, J. (2024) · neural information processing systems · doi:10.52202/079017-0907 · Cited by 849
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
Shukor, M., Aubakirova, D., Capuano, F., Kooijmans, P., Palma, S., Zouitine, A., Aractingi, M., Pascal, C., Russi, M., Marafioti, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01844 · Cited by 695
Self-citationLMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
Zhang, K., Li, B., Zhang, P., Pu, F., Cahyono, J. A., Hu, K., Liu, S., Zhang, Y., Yang, J., Li, C., et al. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.51 · Cited by 495
CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
Wang, Z., Xia, M., He, L., Chen, H., Liu, Y., Zhu, R., Liang, K., Wu, X., Liu, H., Malladi, S., et al. (2024) · neural information processing systems · doi:10.52202/079017-3609 · Cited by 480
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models
Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., & Zhou, J. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2408.04840 · Cited by 404
Perception Encoder: The best visual embeddings are not at the output of the network
Bolya, D., Huang, P.-Y., Sun, P., Cho, J. H., Madotto, A., Wei, C., Ma, T., Zhi, J., Rajasegaran, J., Rasheed, H., et al. (2025) · neural information processing systems · doi:10.52202/085713-2036 · Cited by 401
MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations
Ma, Y., Zang, Y., Chen, M., Chen, M., Jiao, Y., Li, X., Li, X., Liu, Z., Ma, Y., Dong, X., et al. (2024) · neural information processing systems · doi:10.52202/079017-3041 · Cited by 355
PaliGemma 2: A Family of Versatile VLMs for TransferPreprint
Steiner, A., Pinto, A. S., Tschannen, M., Keysers, D., Wang, X. R., Bitton, Y., Gritsenko, A. A., Minderer, M., Sherbondy, A. J., Long, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.03555 · Cited by 339
NVILA: Efficient Frontier Visual Language Models
Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang, S. W., Xi, H., Cao, S., Gu, Y., Li, D., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00390 · Cited by 338
MANTIS: Interleaved Multi-Image Instruction Tuning
Jiang, D., He, X., Zeng, H., Wei, C., Ku, M., Liu, Q., & Chen, W. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2405.01483 · Cited by 313
VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
Shi, Y., Tang, C., Xu, B., Cui, J., Ran, J., Yan, Y., Liu, Z., Wang, S., Han, X., Liu, Z., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.10594 · Cited by 297
Building and better understanding vision-language models: insights and future directionsPreprint
Laurençon, H., Marafioti, A., Sanh, V., & Tronchon, L. (2024) · arXiv.org · doi:10.48550/arxiv.2408.12637 · Cited by 209
Self-citationMerlin: a computed tomography vision–language foundation model and dataset
Blankemeier, L., Kumar, A., Cohen, J., Liu, J., Liu, L., Veen, D. V., Gardezi, S. J. S., Yu, H., Paschali, M., Chen, Z., et al. (2026) · Nature · doi:10.1038/s41586-026-10181-8 · Cited by 104
MetaMorph: Multimodal Understanding and Generation via Instruction Tuning
Tong, S., Fan, D., Zhu, J., Xiong, Y., Chen, X., Sinha, K., Rabbat, M., LeCun, Y., Xie, S., & Liu, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01579 · Cited by 258
Efficient multimodal large language models: a survey
Jin, Y., Li, J., Gu, T., Liu, Y., Zhao, B., Lai, J., Gan, Z., Wang, Y., Wang, C., Tan, X., et al. (2025) · Visual Intelligence · doi:10.1007/s44267-025-00099-6 · Cited by 241
Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H. T., & Shi, G. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.02189 · Cited by 238
Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
Dong, Y., Liu, Z., Sun, H., Yang, J., Hu, W., Rao, Y., & Liu, Z. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00847 · Cited by 217
MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
Zhao, Y., Zhang, H., Xie, L., Hu, T., Gan, G., Long, Y., Hu, Z., Chen, W., Li, C., Xu, Z., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00793 · Cited by 216
MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding
Wang, F., Fu, X., Huang, J. Y., Li, Z., Liu, Q., Liu, X., Ma, M. D., Xu, N., Zhou, W., Zhang, K., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2406.09411 · Cited by 211
LLaDA-V: Large Language Diffusion Models with Visual Instruction TuningPreprint
You, Z., Nie, S., Zhang, X., Hu, J., Zhou, J., Lu, Z., Wen, J.-R., & Li, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.16933 · Cited by 205
Unifying Multimodal Retrieval via Document Screenshot Embedding
Ma, X., Lin, S.-C., Li, M., Chen, W., & Lin, J. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.373 · Cited by 200
M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document UnderstandingPreprint
Cho, J., Mahata, D., İrsoy, O., He, Y., & Bansal, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.04952 · Cited by 194
Efficient Multimodal Learning from Data-centric PerspectivePreprint
He, M., Liu, Y., Wu, B., Yuan, J., Wang, Y., Huang, T., & Zhao, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.11530 · Cited by 190
mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding
Hu, A., Xu, H., Zhang, L., Ye, J., Yan, M., Zhang, J., Jin, Q., Huang, F., & Zhou, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.291 · Cited by 188
Foundation models for materials discovery – current state and future directions
Pyzer‐Knapp, E. O., Manica, M., Staar, P., Morin, L., Ruch, P., Laino, T., Smith, J. R., & Curioni, A. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01538-0 · Cited by 185
Benchmarking Vision Language Models for Cultural Understanding
Nayak, S., Jain, K., Awal, R., Reddy, S., van Steenkiste, S., Hendricks, L. A., Stańczak, K., & Agrawal, A. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.329 · Cited by 182
Aria: An Open Multimodal Native Mixture-of-Experts ModelPreprint
Li, D., Liu, Y., Wu, H., Wang, Y., Shen, Z., Qu, B., Niu, X., Zhou, F., Huang, C., Li, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.05993 · Cited by 179
Baichuan-Omni-1.5 Technical ReportPreprint
Li, Y., Liu, J., Zhang, T., Zhang, T., Chen, S., Li, T., Li, Z., Liu, L., Ming, L., Dong, G., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.15368 · Cited by 176
OCRBench: on the hidden mystery of OCR in large multimodal models
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., & Bai, X. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4235-6 · Cited by 167
Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution
Liu, Z., Dong, Y., Liu, Z., Hu, W., Lu, J., & Rao, Y. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2409.12961 · Cited by 165
Multimodal Large Language Models in Medical Imaging: Current State and Future Directions
Nam, Y., Kim, D., Kyung, S., Seo, J., Song, J. M., Kwon, J., Kim, J., Jo, W., Park, H. J., Sung, J., et al. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.0599 · Cited by 154
A Survey on Post-training of Large Language ModelsPreprint
Tie, G., Zhao, Z., Song, D., Wei, F., Zhou, R., Dai, Y., Yin, W., Yang, Z., Yan, J., Su, Y., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.06072 · Cited by 144
M-LLM Based Video Frame Selection for Efficient Video Understanding
Hu, K., Gao, F., Nie, X., Peng, Z., Tran, S., Neiman, T., Wang, L., Shah, M., Hamid, R., Yin, B., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01279 · Cited by 138
X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great AgainPreprint
Geng, Z., Wang, Y., Ma, Y., Li, C., Rao, Y., Gu, S., Zhong, Z., Lu, Q., Hu, H., Zhang, X., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.22058 · Cited by 132
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Zohar, O., Wang, X., Dubois, Y., Mehta, N., Xiao, T., Hansen-Estruch, P., Yu, L., Wang, X., Juefei-Xu, F., Zhang, N., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01760 · Cited by 130
OmniBench: Towards The Future of Universal Omni-Language Models
Li, Y., Zhang, G., Ma, Y., Yuan, R., Zhu, K., Guo, H., Liang, Y., Liu, J., Wang, N., Yang, J., et al. (2025) · neural information processing systems · doi:10.52202/085713-1036 · Cited by 126
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark
Chai, W., Song, E., Du, Y., Meng, C., Madhavan, V., Bar-Tal, O., Hwang, J., Xie, S., & Manning, C. D. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.03051 · Cited by 125
Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H. T., & Shi, G. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00147 · Cited by 123
Liu, Z., Chu, T., Zang, Y., Wei, X., Dong, X., Zhang, P., Liang, Z., Xiong, Y., Qiao, Y., Lin, D., et al. (2024) · neural information processing systems · doi:10.52202/079017-0278 · Cited by 123
Ola: Pushing the Frontiers of Omni-Modal Language ModelPreprint
Liu, Z., Dong, Y., Wang, J., Liu, Z., Hu, W., Lu, J., & Rao, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.04328 · Cited by 123
A Survey of Robot Intelligence with Large Language Models
Jeong, H., Lee, H., Kim, C., & Shin, S. (2024) · Applied Sciences · doi:10.3390/app14198868 · Cited by 120
MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
Daxberger, E., Wenzel, N., Griffiths, D., Gang, H., Lazarow, J., Kohavi, G., Kang, K., Eichner, M. L., Yang, Y., Dehghan, A., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00694 · Cited by 118
A Survey of Small Language ModelsPreprint
Nguyen, C. V., Shen, X., Aponte, R., Xia, Y., Basu, S., Hu, Z., Chen, J., Parmar, M., Kunapuli, S., Barrow, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.20011 · Cited by 119
NoteLLM-2: Multimodal Large Representation Models for Recommendation
Zhang, C., Zhang, H., Wu, S., Wu, D., Xu, T., Zhao, X., Gao, Y. F., Hu, Y., & Chen, E. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709440 · Cited by 117
VHELM: A Holistic Evaluation of Vision Language Models
Lee, T., Tu, H., Wong, C. H., Zheng, W., Zhou, Y., Mai, Y., Roberts, J. S., Yasunaga, M., Yao, H., Xie, C., et al. (2024) · neural information processing systems · doi:10.52202/079017-4464 · Cited by 117
GenAI Arena: An Open Evaluation Platform for Generative Models
Jiang, D., Ku, M., Li, T., Ni, Y., Sun, S., Fan, R., & Chen, W. (2024) · neural information processing systems · doi:10.52202/079017-2538 · Cited by 107
WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences
Lu, Y., Jiang, D., Chen, W., Wang, W. Y., Choi, Y., & Lin, B. Y. (2024) · neural information processing systems · doi:10.52202/079017-1528 · Cited by 105
Needle In A Multimodal Haystack
Wang, W., Zhang, S., Ren, Y., Duan, Y., Li, T., Liu, S., Hu, M., Chen, Z., Zhang, K., Lu, L., et al. (2024) · neural information processing systems · doi:10.52202/079017-0649 · Cited by 103
Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench
Liu, Z., Dou, G., Jia, M., Tan, Z., Zeng, Q., Yuan, Y., & Jiang, M. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.207 · Cited by 98
VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning
Liu, Y., Lin, K. Q., Chen, C. W., & Shou, M. Z. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2503.13444 · Cited by 98
Efficient GPT-4V level multimodal large language model for deployment on edge devices
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Chen, C., Li, H., Zhao, W., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-61040-5 · Cited by 97
VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents
Wang, K., Zhang, P., Wang, Z., Gao, Y., Li, L., Wang, Q., Chen, H., Lu, Y., Yang, Z., Yang, Z., et al. (2025) · neural information processing systems · doi:10.52202/085713-5754 · Cited by 93
MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?
Chen, Z., Wen, Z., Du, Y., Zhou, Y., Cui, C., Han, S., Weng, J., Wang, C., Tong, Z., HUANG, L., et al. (2025) · neural information processing systems · doi:10.52202/085713-2081 · Cited by 90
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
Zhang, H., Gao, M., Gan, Z., Dufter, P., Wenzel, N., Huang, F., Shah, D., Du, X., Zhang, B., Li, Y., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2409.20566 · Cited by 90
MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
Meng, F., Wang, J., Li, C., Lu, Q., Tian, H., Liao, J., Zhu, X., Dai, J., Qiao, Y., Luo, P., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2408.02718 · Cited by 88
FineVision: Open Data Is All You NeedPreprint
Wiedmann, L., Zohar, O., Mahla, A., Wang, X.-H., Li, R., Frere, T., von Werra, L., Gosthipaty, A. R., & Marafioti, A. (2025) · arXiv.org · doi:10.48550/arxiv.2510.17269 · Cited by 55
CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
Li, J., Wang, X., Zhu, S., Kuo, C.-W., Xu, L. X., Chen, F. W., Jain, J., Shi, H., & Wen, L. (2024) · neural information processing systems · doi:10.52202/079017-4170 · Cited by 85
Baichuan-Omni Technical ReportPreprint
Li, Y., Sun, H., Lin, M., Li, T., Dong, G., Zhang, T., Ding, B., Song, W., Cheng, Z., Huo, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.08565 · Cited by 84
Wang, H., Shi, H., Tan, S., Qin, W., Wang, W., Zhang, T., Nambi, A. U., Ganu, T., & Wang, H. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.166 · Cited by 81
SmolDocling: An Ultra-Compact Vision-Language Model for End-To-End Multi-Modal Document Conversion
Nassar, A., Omenetti, M., Lysak, M., Livathinos, N., Auer, C., Morin, L., Lima, R. T. D., Kim, Y., Dolfi, M., & Staar, P. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02040 · Cited by 80
CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models
Cheng, Z., Chen, Q., Zhang, J., Fei, H., Feng, X., Che, W., Li, M., & Qin, L. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i22.34538 · Cited by 77
Zhao, B., Zong, Y., Zhang, L., & Hospedales, T. M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.12742 · Cited by 76
T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation
Li, L., Shi, Z., Hu, X., Dong, B., Qin, Y., Liu, X., Sheng, L., & Shao, J. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01249 · Cited by 74
MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
Xu, Y., Zhu, L., & Yang, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01642 · Cited by 73
Hidden in plain sight: VLMs overlook their visual representationsPreprint
Fu, S., Bonnen, T., Guillory, D., & Darrell, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.08008 · Cited by 73
Scalable Vision Language Model Training via High Quality Data Curation
Dong, H., Kang, Z., Yin, W., LiangXiao, L., ChaoFeng, C., & Jiao, R. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1595 · Cited by 72
CF-VLM:CounterFactual Vision-Language Fine-tuning
zhang, J., Cai, K., Fan, Y., Wang, J., & Wang, K. (2025) · neural information processing systems · doi:10.52202/085713-1946 · Cited by 72
Li, Y., Cao, Y., He, H., Cheng, Q., Fu, X., Xiao, X., Wang, T., & Tang, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.04633 · Cited by 72
OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature ExtractionPreprint
Huang, H., Liu, F., Fu, L., Wu, T., Mukadam, M., Malik, J., Goldberg, K., & Abbeel, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.03734 · Cited by 72
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
Huang, B., Mitra, C., Arbelle, A., Karlinsky, L., Darrell, T., & Herzig, R. (2024) · neural information processing systems · doi:10.52202/079017-0697 · Cited by 70
Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering
Cocchi, F., Moratelli, N., Cornia, M., Baraldi, L., & Cucchiara, R. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00859 · Cited by 69
Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language ModelsPreprint
Chen, G., Li, Z., Wang, S., Jiang, J., Liu, Y., Lü, L., Huang, D.-A., Byeon, W., Le, M., Rintamaki, T., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.15271 · Cited by 69
Aya Vision: Advancing the Frontier of Multilingual MultimodalityPreprint
Dash, S., Nan, Y., Dang, J., Ahmadian, A., Singh, S., Smith, M., Venkitesh, B., Shmyhlo, V., Aryabumi, V., Beller-Morales, W., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.08751 · Cited by 68
Zhou, B., Yang, H., Chen, D., Ye, J., Bai, T., Yu, J., Zhang, S., Lin, D., He, C., & Li, W. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i10.33163 · Cited by 67
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Chen, K., Gou, Y., Huang, R., Liu, Z., Tan, D., Xu, J., Wang, C., Zhu, Y., Zeng, Y., Yang, K., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00513 · Cited by 67
Scaling Laws for Native Multimodal Models
Shukor, M., Fini, E., da Costa, V. G. T., Cord, M., Susskind, J. M., & El-Nouby, A. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00009 · Cited by 68
Self-citationThe Evolution of Multimodal Model ArchitecturesPreprint
Wadekar, S. N., Chaurasia, A., Chadha, A. R., & Culurciello, E. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.17927 · Cited by 64
VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning
Zong, Y., Bohdal, O., & Hospedales, T. M. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2403.13164 · Cited by 60
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text
Li, Q., Chen, Z., Wang, W., Wang, W., Ye, S., Jin, Z., Chen, G., He, Y., Gao, Z., Cui, E., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2406.08418 · Cited by 62
MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs
Qian, Y., Ye, H., Fauconnier, J.-P., Grasch, P., Yang, Y., & Gan, Z. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2407.01509 · Cited by 60
FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture
Li, W., Zhang, C., Li, J., Peng, Q., Tang, R., Zhou, L., Zhang, W., Hu, G., Yuan, Y., Søgaard, A., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1063 · Cited by 59
F-LMM: Grounding Frozen Large Multimodal Models
Wu, S., Jin, S., Zhang, W., Xu, L., Liu, W., Li, W., & Loy, C. C. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02301 · Cited by 59
Teaching multimodal LLMs to comprehend 12-lead electrocardiographic images
Liu, R., Bai, Y., Yue, X., & Zhang, P. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02551-3 · Cited by 59
Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward
Xiao, T. Y., Xu, X., Huang, Z., Gao, H., Liu, Q., Liu, Q., & Chen, E. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2506.07218 · Cited by 57
LEGO-Puzzles: How Good Are MLLMs at Multi-step Spatial Reasoning?
Tang, K., Gao, J., Zeng, Y., Duan, H., Sun, Y., Xing, Z., Liu, W., Chen, K., & Lyu, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37553-7_1 · Cited by 55
Zhao, Y., Xue, F., Reed, S., Fan, L., Zhu, Y., Kautz, J., Yu, Z., Krähenbühl, P., & Huang, D.-A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.05178 · Cited by 55
He, H., Yao, W., Ma, K., Yu, W., Zhang, H., Fang, T., Lan, Z., & Yu, D. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1336 · Cited by 54
LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding
Wang, S., Liu, S., Kuang, Y., Wei, X., Liu, Y., Li, Z., Man, Y., Guo, C., Tao, A., Liu, G., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37718-0_20 · Cited by 57
How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive PromptsPreprint
Qian, Y., Zhang, H., Yang, Y., & Apple, Z. G. (2024) · arXiv (Cornell University) · doi:10.13140/rg.2.2.31472.84485 · Cited by 54
DocVLM: Make Your VLM an Efficient Reader
Nacson, M. S., Aberdam, A., Ganz, R., Avraham, E. B., Golts, A., Kittenplon, Y., Mazor, S., & Litman, R. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02701 · Cited by 53
Granite Vision: a lightweight, open-source multimodal model for enterprise IntelligencePreprint
Karlinsky, L., Arbelle, A., Daniels, A., Nassar, A. S., Alfassi, A., Wu, B., Schwartz, E., Joshi, D., Kondic, J., Shabtay, N., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2502.09927 · Cited by 35
VAMBA: Understanding Hour-Long Videos with Hybrid Mamba-Transformers
Ren, W., Ma, W., Yang, H., Wei, C., Zhang, G., & Chen, W. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01969 · Cited by 51
One-D-Piece: Image Tokenizer Meets Quality-Controllable CompressionPreprint
Miwa, K., Sasaki, K., Arai, H., Takahashi, T., & Yamaguchi, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.10064 · Cited by 51
Image2Struct: Benchmarking Structure Extraction for Vision-Language Models
Roberts, J. S., Lee, T., Wong, C. H., Yasunaga, M., Mai, Y., & Liang, P. (2024) · neural information processing systems · doi:10.52202/079017-3653 · Cited by 50
Similarity-Aware Token Pruning: Your VLM but FasterPreprint
Jeddi, A., Baghbanzadeh, N., Dolatabadi, E., & Taati, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.11549 · Cited by 50
VILA$^2$: VILA Augmented VILAPreprint
Fang, Y., Zhu, L., Lu, Y., Wang, Y., Molchanov, P., Kautz, J., Cho, J. H., Pavone, M., Han, S., & Yin, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.17453 · Cited by 50
Metadata
- Authors
- Laurençon, H., Tronchon, L., Cord, M., & Sanh, V.
- ORCID
- Cord, Matthieu
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 87874-87907
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W4415796627
- Cited by
- 66
- References
- 0
- Citations by year
- 2026: 39 · 2025: 23 · 2024: 4
- FWCI
- 13.67
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- vision-language models, model design, multimodal representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.