The DOI matches its Crossref record; 127 of the 127 references it cites were found in a registry or an index.
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites
Open accessAbstract
In this paper, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple improvements. (1) Strong vision encoder: we explored a continuous learning strategy for the large-scale vision foundation model — InternViT-6B, boosting its visual understanding capabilities, and making it can be transferred and reused in different LLMs. (2) Dynamic high-resolution: we divide images into tiles ranging from 1 to 40 of 448×448 pixels according to the aspect ratio and resolution of the input images, which supports up to 4K resolution input. (3) High-quality bilingual dataset: we carefully collected a high-quality bilingual dataset that covers common scenes, document images, and annotated them with English and Chinese question-answer pairs, significantly enhancing performance in optical character recognition (OCR) and Chinese-related tasks. We evaluate InternVL 1.5 through a series of benchmarks and comparative studies. Compared to both open-source and proprietary commercial models, InternVL 1.5 shows competitive performance, achieving state-of-the-art results in 8 of 18 multimodal benchmarks. Code and models are available at https://github.com/OpenGVLab/InternVL.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.1993.341010 · Cited by 14,265
Proceedings European Conference on Computer Vision 2002
Heyden, A., Sparr, G., Nielsen, M., & Johansen, P. (2002) · Lund University Publications (Lund University) · Cited by 5,153
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 2×
“…(en&zh) [41], ST-VQA (en) [11] Document DocVQA (en) [22], Common Crawl PDF (en&zh) Grounding RefCOCO/+/g (en) [75, 126], Visual Genome (en) [42] LLaVA-150K (en&zh) [59], LVIS-Instruct4V (en) [110], ALLaVA (en&zh) [16], Laion-GPT4V (en) [1], Conversation TextOCR-GPT4V (en) [2], SVIT (en&zh)…”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
Proceedings for the 5th International Conference on Learning Representations
Srivastava, A., & Sutton, C. A. (2017) · Cited by 1,837
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2310.03744 · Cited by 1,733
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
CogVLM: Visual Expert for Pretrained Language Models
Wang, W., Lv, Q., Yu, W., Hong, W., Ji, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al. (2024) · neural information processing systems · doi:10.52202/079017-3860 · Cited by 1,362
Self-citationMM-Vet: Evaluating Large Multimodal Models for Integrated CapabilitiesPreprint
Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., & Wang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.02490 · Cited by 1,294
Self-citationDeepSeek-VL: Towards Real-World Vision-Language UnderstandingPreprint
Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Yang, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.05525 · Cited by 1,289
Self-citationGeneration and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.9 · Cited by 1,174
Sigmoid Loss for Language Image Pre-Training
Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01100 · Cited by 1,127
Kosmos-2: Grounding Multimodal Large Language Models to the WorldPreprint
Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., & Wei, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.14824 · Cited by 1,096
Self-citationProceedings of the 24th annual meeting on Association for Computational Linguistics -
(1986) · doi:10.3115/981131 · Cited by 1,081
(2016) · doi:10.1007/978-3-319-46475-6_5 · Cited by 1,081
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078 · cited 1×
“Currently, models like CLIP-ViT [87] and SigLIP [131] are prevalently utilized; however, many studies have been conducted to find the most suitable vision encoders for MLLMs [53, 67, 72, [58, 60], combining a pre-trained InternViT-6B [20] with InternLM2-20B [13] through a MLP projector.”
Shikra: Unleashing Multimodal LLM's Referential Dialogue MagicPreprint
Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., & Zhao, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.15195 · Cited by 1,049
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsPreprint
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T.-Y., Li, Z., Weller, A., & Liu, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.12284 · Cited by 1,028
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990
Self-citationGLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.26 · Cited by 955
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908
SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., & Shan, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.16125 · Cited by 882
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction ModelPreprint
Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.15010 · Cited by 837
Self-citationNExT-GPT: Any-to-Any Multimodal LLMPreprint
Wu, S., Fei, H., Qu, L., Ji, W., & Chua, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05519 · Cited by 791
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02283 · Cited by 734
Self-citationOK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00331 · Cited by 703
Objects365: A Large-Scale, High-Quality Dataset for Object Detection
Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., & Sun, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00852 · Cited by 673 · cited 3×
“Detection and grounding tasks utilize datasets like Ob-jects365 [93], GRIT [86], and All-Seeing [114], making up 5.2%.”
InfluentialMMBench: Is Your Multi-modal Model an All-Around Player?
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72658-3_13 · Cited by 668
Self-citationOtter: A Multi-Modal Model With In-Context Instruction Tuning
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Cahyono, J. A., Yang, J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3571946 · Cited by 655
Self-citationLiu, F., Emerson, G., & Collier, N. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00566 · Cited by 563
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
Lin, B., Tang, Z., Ye, Y., Huang, J., Zhang, J., Pang, Y., Jin, P., Ning, M., Luo, J., & Yuan, L. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3654458 · Cited by 542
Self-citationMonkey: Image Resolution and Text Label are Important Things for Large Multi-Modal Models
Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., & Bai, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02527 · Cited by 535
Self-citationMiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469
LISA: Reasoning Segmentation via Large Language Model
Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., & Jia, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00915 · Cited by 466
Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Wei, X., Zhang, S., Duan, H., Cao, M., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.16420 · Cited by 455
Self-citationMini-Gemini: Mining the Potential of Multi-Modality Vision Language Models
Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., Liu, S., & Jia, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3637265 · Cited by 446
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429
Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L., Han, X., Qiu, H., Lin, C., Shao, W., Chen, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.07575 · Cited by 377
Self-citationGPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
Zhang, S., Sun, P., Chen, S., Xiao, M., Shao, W., Zhang, W., Liu, Y., Chen, K., & Luo, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91813-1_4 · Cited by 376
Self-citationChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
Masry, A., Long, D., Tan, J. Q., Joty, S., & Hoque, E. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.177 · Cited by 373
Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., & Zhu, S. C. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.528 · Cited by 371
Learn to Explain: Multimodal Reasoning Via Thought Chains for Science Question Answering
Lu, P., Mishra, S. R., Xia, T., Qiu, L., Chang, K.-W., Zhu, S. C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · neural information processing systems · doi:10.52202/068431-0182 · Cited by 362
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332
Self-citationGemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306
Self-citationLLaVA-UHD: An LMM Perceiving Any Aspect Ratio and High-Resolution Images
Guo, Z., Xu, R., Yao, Y., Cui, J., Ni, Z., Ge, C., Chua, T., Liu, Z., & Huang, G. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73010-8_23 · Cited by 294
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingPreprint
McKinzie, B., Gan, Z., Fauconnier, J., Dodge, S., Zhang, B.-W., Dufter, P., Shah, D., Du, X.-Z., Peng, F., Weers, F., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2403.09611 · Cited by 279
Self-citationA-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge
Schwenk, D., Khandelwal, A., Clark, C., Marino, K., & Mottaghi, R. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20074-8_9 · Cited by 269
DVQA: Understanding Data Visualizations via Question Answering
Kafle, K., Price, B., Cohen, S., & Kanan, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00592 · Cited by 254
Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., & Hajishirzi, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.571 · Cited by 244 · cited 2×
“…(en) [28], GQA (en) [34], OKVQA (en) [76], General QA VSR (en) [55], VisualDialog (en) [24] Science AI2D (en) [39], ScienceQA (en) [69], TQA (en) [40] ChartQA (en) [77], MMC-Inst (en) [57], DVQA (en) [38], Chart PlotQA (en) [81], LRV-Instruction (en) [56] GeoQA+ (en) [14], TabMWP (en) [70],…”
TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document
Liu, Y., Yang, B., Liu, Q., Li, Z., Ma, Z., Zhang, S., & Bai, X. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3653415 · Cited by 243
Self-citationShareGPT4V: Improving Large Multi-modal Models with Better Captions
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., & Lin, D. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72643-9_22 · Cited by 237
Self-citationmPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding
Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L., Zhang, B., Zhang, J., Jin, Q., Huang, F., & Zhou, J. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.175 · Cited by 237
Self-citationOCR-VQA: Visual Question Answering by Reading Text in Images
Mishra, A., Shekhar, S., Singh, A. K., & Chakraborty, A. (2019) · 2019 International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00156 · Cited by 227
TextCaps: A Dataset for Image Captioning with Reading Comprehension
Sidorov, O., Hu, R., Rohrbach, M., & Singh, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58536-5_44 · Cited by 225
CogVLM: Visual Expert for Pretrained Language Models
Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.03079 · Cited by 217
Self-citationDong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Zhang, S., Duan, H., Zhang, W., Li, W., et al. (2024) · neural information processing systems · doi:10.52202/079017-1348 · Cited by 216
Self-citationWukong: A 100 Million Large-Scale Chinese Cross-Modal Pre-Training Benchmark
Gu, J., Meng, X., Lu, G., Hou, L., Minzhe, N., Liang, X., Yao, L., Huang, R., Zhang, W., Jiang, X., et al. (2022) · neural information processing systems · doi:10.52202/068431-1916 · Cited by 208
Self-citationProceedings of the International Conference on Document Analysis and Recognition
Value, N. (2003) · Cited by 207
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
Wang, Y., Li, K., Li, X., Yu, J., He, Y., Chen, G., Pei, B., Zheng, R., Wang, Z., Shi, Y., et al. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73013-9_23 · Cited by 205
Self-citationCogAgent: A Visual Language Model for GUI Agents
Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01354 · Cited by 202
Self-citationICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text - RRC-ArT
Chng, C. K., Ding, E., Liu, J., et al. (2019) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2019.00252 · Cited by 202
COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images
Veit, A., Matera, T., Neumann, L., Matas, J., & Belongie, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.07140 · Cited by 191
SVIT: Scaling up Visual Instruction TuningPreprint
Zhao, B., Wu, B., He, M., & Huang, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.04087 · Cited by 191
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., & Xie, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00914 · Cited by 189
Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models
Ma, C., Jiang, Y., Wu, J., Yuan, Z., & Qi, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72658-3_24 · Cited by 187
VideoLLM: Modeling Video Sequence with Large Language ModelsPreprint
Guo, C., Zheng, Y.-D., Wang, J., Xu, J., Huang, Y., Pan, J., Wang, Y., Wang, Y., Qiao, Y., Lü, T., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13292 · Cited by 185
Self-citationVideoChat: chat-centric video understanding
Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., & Qiao, Y. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4321-9 · Cited by 185
Self-citationYing, K., Meng, F., Wang, J., Li, Z., Lin, H., Yang, Y., Zhang, H., Zhang, W., Lin, Y., Liu, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.16006 · Cited by 176
Self-citationTo See is to Believe: Prompting GPT-4V for Better Visual Instruction TuningPreprint
Wang, J., Meng, L., Weng, Z., He, B., Wu, Z., & Jiang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.07574 · Cited by 172
Self-citationSkywork: A More Open Bilingual Foundation ModelPreprint
Wei, T., Zhao, L., Zhang, L., Zhu, B., Wang, L., Yang, H., Li, B., Cheng, C., Lü, W., Hu, R., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.19341 · Cited by 169
OCRBench: on the hidden mystery of OCR in large multimodal models
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., & Bai, X. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4235-6 · Cited by 167
(2023) · doi:10.48550/arxiv.2310.14566 · Cited by 166
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
Zhang, R., Han, J., Liu, C., Gao, P., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., & Qiao, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.16199 · Cited by 166
Self-citationVisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al. (2023) · neural information processing systems · doi:10.52202/075280-2688 · Cited by 162
Self-citationCLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical ReasoningPreprint
Lindström, A. D., & Abraham, S. S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.05358 · Cited by 161
A Large Chinese Text Dataset in the Wild
Yuan, T., Zhu, Z., Xu, K., Cheng-jun, L., Mu, T., & Hu, S. (2019) · Journal of Computer Science and Technology · doi:10.1007/s11390-019-1923-y · Cited by 160
TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text
Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., & Hassner, T. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00869 · Cited by 158
PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR SystemPreprint
Li, C., Liu, W., Guo, R., Yin, X., Jiang, K., Du, Y., Du, Y., Zhu, L., Lai, B., Hu, X., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.03001 · Cited by 158
ICDAR 2019 Competition on Large-Scale Street View Text with Partial Labeling - RRC-LSVT
Sun, Y., Karatzas, D., Chan, C. S., Jin, L., Ni, Z., Chng, C.-K., Liu, Y., Luo, C., Ng, C. C., Han, J., et al. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00250 · Cited by 155
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., & Wang, L. (2023) · arXiv · doi:10.48550/arxiv.2306.14565 · Cited by 149
Self-citationSuper-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning
Li, Z., Wong, X., Stengel-Eskin, E., Kortylewski, A., Ma, W., Durme, B. V., & Yuille, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01437 · Cited by 149
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Xu, L., Yang, J., Zheng, X., Li, K., Sun, X. W., et al. (2025) · neural information processing systems · doi:10.52202/085713-4899 · Cited by 148
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond LanguagePreprint
Liu, Z., He, Y., Wang, W., Wang, W., Wang, Y., Chen, S., Zhang, Q., Lai, Z., Yang, Y., Li, Q., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.05662 · Cited by 142
Self-citationICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboard
Zhang, R., Yang, M., Bai, X., Shi, B., Karatzas, D., Lu, S., Jawahar, C. V., Zhou, Y., Jiang, Q., Song, Q., et al. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00253 · Cited by 132
Self-citationFeast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language ModelsPreprint
Luo, G., Zhou, Y., Zhang, Y., Zheng, X., Sun, X., & Ji, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.03003 · Cited by 129
KOSMOS-2.5: A Multimodal Literate ModelPreprint
Lv, T., Huang, Y., Chen, J., Zhao, Y., Jia, Y., Cui, L., Ma, S., Chang, Y., Huang, S., Wang, W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.11419 · Cited by 127
Self-citationBaichuan 2: Open Large-scale Language Models
Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.10305 · Cited by 125
Self-citationThe All-Seeing Project V2: Towards General Relation Comprehension of the Open World
Wang, W., Ren, Y., Luo, H., Li, T., Yan, C., Chen, Z., Wang, W., Li, Q., Lu, L., Zhu, X., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73414-4_27 · Cited by 120
Self-citationYe, J., Hu, A., Xu, H., Ye, Q., Yan, M., Xu, G., Li, C., Tian, J., Qian, Q., Zhang, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.05126 · Cited by 111
OtterHD: A High-Resolution Multi-modality ModelPreprint
Li, B., Zhang, P., Yang, J., Zhang, Y., Pu, F., & Liu, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.04219 · Cited by 100
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99 · cited 1×
“Large language models (LLMs) [3, 6, 9, 10, 13, 26, 99, 101, 103, 107, 108, 117, 118, 136] have greatly advanced AGI by enabling complex language tasks previously thought human-exclusive.”
Self-citationReka Core, Flash, and Edge: A Series of Powerful Multimodal Language ModelsPreprint
Reka Team, Ormazabal, A., Zheng, C., d’Autume, C. D. M., Yogatama, D., Fu, D., Ong, D., Chen, E., Lamprecht, E., Pham, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.12387 · Cited by 96
VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. (2024) · Proceedings of the 32nd ACM International Conference on Multimedia · doi:10.1145/3664647.3685520 · Cited by 95
Self-citationControlLLM: Augment Language Models with Tools by Searching on Graphs
Liu, Z., Lai, Z., Gao, Z., Cui, E., Li, Z., Zhu, X., Lu, L., Chen, Q., Qiao, Y., Dai, Y., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73254-6_6 · Cited by 95
Self-citationYe, J., Hu, A., Xu, H., Ye, Q., Yan, M., Xu, G., Li, C., Tian, J., Qian, Q., Zhang, J., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.187 · Cited by 94
Vary: Scaling up the Vision Vocabulary for Large Vision-Language ModelsPreprint
Wei, H., Kong, L., Chen, J., Zhao, L., Ge, Z., Yang, J., Sun, J., Han, C., & Zhang, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.06109 · Cited by 85
The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open WorldPreprint
Wang, W., Shi, M., Li, Q., Wang, W., Huang, Z., Xing, L., Chen, Z., Li, H., Zhu, X., Cao, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01907 · Cited by 83
Self-citationMM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature SynchronizerPreprint
Tian, C., Zhu, X., Xiong, Y., Wang, W., Chen, Z., Wang, W., Chen, Y., Lu, L., Lü, T., Zhou, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.10208 · Cited by 78
Self-citationLarge Multilingual Models Pivot Zero-Shot Multimodal Learning across LanguagesPreprint
Hu, J., Yao, Y., Wang, C., Wang, S., Pan, Y., Chen, Q., Yu, T., Wu, H., Zhao, Y., Zhang, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.12038 · Cited by 74
COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuningPreprint
Bai, Y., Du, X., Liang, Y., Jin, Y., Zhou, J., Liu, Z., Fang, F., Chang, M., Zheng, T., Zhang, X., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.18058 · Cited by 70
WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large ModelsPreprint
He, C., Jin, Z., Xu, C., Qiu, J., Wang, B., Li, W., Yan, H., Wang, J., & Lin, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.10755 · Cited by 61
Self-citationALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language ModelsPreprint
Chen, G. H., Chen, S., Zhang, R., Chen, J., Wu, X., Zhang, Z., Chen, Z. H., Li, J., Wan, X., & Wang, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.11684 · Cited by 59
Self-citationYi: Open Foundation Models by 01.AI
AI, 0., :, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Wang, G., Li, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.04652 · Cited by 50
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning
Lu, P., Qiu, L., Chang, K.-W., Wu, Y. N., Zhu, S.-C., Rajpurohit, T., Clark, P., & Kalyan, A. (2022) · arXiv · doi:10.48550/arxiv.2209.14610 · Cited by 45
ViQuAE, a Dataset for Knowledge-based Visual Question Answering about Named Entities
Lerner, P. J., Ferret, O., Guinaudeau, C., Borgne, H. L., Besançon, R., Moreno, J. G., & Lovón-Melgarejo, J. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531753 · Cited by 39
Liu, S., Ying, K., Zhang, H., Yang, Y., Lin, Y., Zhang, T., Li, C., Qiao, Y., Luo, P., Shao, W., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.20194 · Cited by 35
Self-citationCai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.17297 · Cited by 30
Self-citationmPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality CollaborationPreprint
Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., Huang, F., & Zhou, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.04257 · Cited by 28
Shimizu, N., Rong, N., & Miyazaki, T. (2018) · International Conference on Computational Linguistics · Cited by 25
Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic ReasoningPreprint
Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., & Zhu, S. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04165 · Cited by 14
MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction TuningPreprint
Liu, F., Wang, X., Yao, W., Chen, J., Song, K., Cho, S., Yacoob, Y., & Yu, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.10774 · Cited by 4
LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution ImagesPreprint
Xu, R., Yao, Y., Guo, Z., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.11703 · Cited by 3
Cited by
A survey on multimodal large language models
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024) · National Science Review · doi:10.1093/nsr/nwae403 · Cited by 893
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.02245 · Cited by 291
OCRBench: on the hidden mystery of OCR in large multimodal models
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., & Bai, X. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4235-6 · Cited by 167
“5-Sonnet [ 70] Anthropic 788 LLaVA-Next-Yi-34B [71] UW–Madison 574 GPT-4o-mini-20240718 [72] OpenAI 785 TextMonkey [30] HUST 561 InternVL2-4B [64] Shanghai AI Lab 784 Monkey-Chat [29] HUST 534 InternVL2-2B [64] Shanghai AI Lab 781 InternLM-XComposer2 [73] Shanghai AI Lab 532 GLM-4v-9B [67] Zhipu AI…”
InfluentialThinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.00994 · Cited by 120
“For open-source models, we evaluate models from InternVL2 [14], ViLA [45], LongViLA [91], LongVA [101], LLaVA-OneVision [40], and LLaVA-Video [104].”
OmniGen: Unified Image Generation
Xiao, S., Wang, Y., Zhou, J., Yuan, H., Xing, X., Yan, R., Li, C., Wang, S., Huang, T., & Liu, Z. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.01241 · Cited by 75
Metadata
- Authors
- Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., Ma, J., Wang, J., Dong, X., Yan, H., Guo, H., He, C., Shi, B., Jin, Z., Xu, C., Wang, B., Wei, X., Li, W., Zhang, W., Zhang, B., Cai, P., Wen, L., Yan, X., Dou, M., Lu, L., Zhu, X., Lu, T., Lin, D., Qiao, Y., Dai, J., & Wang, W.
- Venue
- Science China Information Sciences
- Published
- 2024-12
- Volume
- 67
- Issue
- 12
- ISSN
- 1869-1919 (ISSN-L)
- 1674-733X
- OpenAlex
- W4405595839
- Cited by
- 325
- References
- 127
- FWCI
- 45.91
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.