OCRBench: on the hidden mystery of OCR in large multimodal models
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Application-driven benchmarks targeting a single perceptual application, such as OCRBench and ScreenSpot, are built top-down with predefined categories from a specific application domain.
“benchmarks targeting a single perceptual application, such as OCRBench (Yuliang Liu et al. 2024) and ScreenSpot (K. Li et al. 2025), provide more direct measures of perception but are built top-down: their categories are predefined and populated with questions from a specific application domain.”
From PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models (2026) · cites this paper as Yuliang Liu et al. 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1× · see citing works
“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Hawkins, C., Sorgi, M., & Lock, S. (1985) · Springer London · doi:10.1007/978-1-4471-3519-7 · Cited by 9,160 · cited 1×
“5 [64] Shanghai AI Lab 720 DeepSeek-VL-7B [81] DeepSeek 435 Claude3-Opus [82] Anthropic 694 OmniLMM-12B [83] OpenBMB 420 RekaFlash [74] Reka AI 692 TransCore-M [84] PCI Research 405 InternLM-XComposer2.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1× · see citing works
“Among them, LLaMA [3] is an open-source LLM that matches the performance of GPT-3, followed by Alpaca [4], Vicuna [5], GPT-4-LLM [6] to improve the LLM’s alignment ability to follow human instruction, reporting impressive performance compared with proprietary LLMs.”
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428 · see citing works
Self-citationGLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z.-X., Qian, Y., Liu, X., Ding, M., Qiu, J.-Z., Yang, Z.-L., & Tang, J. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2022.acl-long.26 · Cited by 2,032 · cited 1× · see citing works
“5 [64] Shanghai AI Lab 639 MMAlaya [93] DataCanvas 223 Phi-3-Vision [94] Microsoft 637 VisualGLM [95] Tsinghua 170 WeMM [96] WechatCV 628 OpenFlamingo v2 [97] UW 149 IDEFICS2-8B [98] Hugging Face 626 PandaGPT-13B [99] Tencent AI Lab 36 Step-1V [100] StepFun 625 Chameleon-30B [101] Meta 27 Table 4:…”
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical ReasoningPreprint
Masry, A., Long, D. X., Tan, J., Joty, S. R., & Hoque, E. (2022) · Findings · doi:10.48550/arxiv.2203.10244 · Cited by 2,023 · cited 1× · see citing works
“We assess these LMMs on DocVQA[55], InfographicVQA [56] and ChartQA [57].”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · cited 1× · see citing works
“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”
Self-citationMME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
Fu, C., Chen, P., Shen, Y.-H., Qin, Y.-L., Zhang, M.-D., Lin, X., Yang, J., Zheng, X.-W., Li, K., Sun, X., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.13394 · Cited by 1,917
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748 · cited 1× · see citing works
“Other works, such as [20, 21], also focus on aligning the vision module and LLM for improved multimodal understanding.”
Self-citationToward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Self-citationICDAR 2015 competition on Robust Reading
Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S. K., Bagdanov, A. D., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar, V., Lu, S.-J., et al. (2015) · IEEE International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2015.7333942 · Cited by 1,622 · cited 2× · see citing works
“…OCR text recognition datasets, including (1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly…”
The IAM-database: an English sentence database for offline handwriting recognition
Marti, U.-V., & Bunke, H. (2002) · International Journal on Document Analysis and Recognition · doi:10.1007/s100320200071 · Cited by 1,589 · see citing works
Synthetic Data for Text Localisation in Natural Images
Gupta, A., Vedaldi, A., & Zisserman, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.254 · Cited by 1,563 · see citing works
ICDAR 2013 Robust Reading Competition
Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., Bigorda, L. G. I., Mestre, S., Mas, J., Mota, D. F., Almazán, J., & Heras, L.-P. D. L. (2013) · IEEE International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2013.221 · Cited by 1,506 · cited 1× · see citing works
“…Recognition : We evaluate LMM using widely-adopted OCR text recognition datasets, including (1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3)…”
mPLUG-Owl: Modularization Empowers Large Language Models with MultimodalityPreprint
Ye, Q.-H., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P.-C., Shi, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2304.14178 · Cited by 1,318 · cited 2× · see citing works
“Additionally, [22, 23] emphasize modality collaboration for image and text.”
Self-citationShareGPT4V: Improving Large Multi-Modal Models with Better CaptionsPreprint
Chen, L., Li, J., Dong, X.-W., Zhang, P., He, C.-H., Wang, J., Zhao, F., & Lin, D.-H. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2311.12793 · Cited by 1,271 · see citing works
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · cited 2× · see citing works
“Flamingo [13] and OpenFlamingo [19] enhance a frozen pretrained LLM by incorporating novel gated cross-attention-dense layers, enabling conditioning on visual inputs.”
Synthetic Data and Artificial Neural Networks for Natural Scene Text RecognitionPreprint
Jaderberg, M., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.2227 · Cited by 915 · see citing works
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · cited 1× · see citing works
“BLIP2 [18] introduces the Querying Transformer (Q-Former) as a means to bridge the gap between vision and language models.”
GIT: A Generative Image-to-text Transformer for Vision and LanguagePreprint
Wang, J., Yang, Z.-Y., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z.-C., Liu, C., & Wang, L.-J. (2022) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2205.14100 · Cited by 817 · cited 1× · see citing works
“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”
Self-citationTowards VQA Models That Can Read
Singh, A., Natarajan, V., Shah, M., Jiang, Y. L., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00851 · Cited by 777 · cited 1× · see citing works
“Scene Text-Centric VQA : We test LMMs on STVQA [51], TextVQA [52], OCRVQA [53] and ESTVQA [54].”
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningPreprint
Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z.-C., Zhang, P.-C., Krishnamoorthi, R., Chandra, V., Xiong, Y.-Y., & Elhoseiny, M. (2023) · arXiv.org · doi:10.48550/arxiv.2310.09478 · Cited by 764 · cited 1× · see citing works
“MiniGPT4V2 [26] uses unique identifiers for different tasks when training the model to better distinguish each task instruction effortlessly.”
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Ye, Q.-H., Xu, H., Ye, J., Yan, M.-S., Hu, A., Liu, H.-W., Qian, Q., Zhang, J., Huang, F., & Zhou, J.-R. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.01239 · Cited by 763 · cited 1× · see citing works
“Additionally, [22, 23] emphasize modality collaboration for image and text.”
OCR-VQA: Visual Question Answering by Reading Text in Images
Mishra, A., Shekhar, S., Singh, A., & Chakraborty, A. (2019) · IEEE International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2019.00156 · Cited by 746 · cited 1× · see citing works
“TextVQA dataset comprises 45,000+ questions on 28,000+ images sampled from specific OpenImages dataset categories expected to contain text.”
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682 · cited 4× · see citing works
“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”
InfluentialSelf-citationOpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language ModelsPreprint
Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2308.01390 · Cited by 674 · cited 1× · see citing works
“Flamingo [13] and OpenFlamingo [19] enhance a frozen pretrained LLM by incorporating novel gated cross-attention-dense layers, enabling conditioning on visual inputs.”
COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural ImagesPreprint
Veit, A., Matera, T., Neumann, L., Matas, J., & Belongie, S. J. (2016) · arXiv.org · doi:10.48550/arxiv.1601.07140 · Cited by 641 · cited 1× · see citing works
“…(1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly occluded scene text (WOST) and heavily…”
What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis
Baek, J., Kim, G., Lee, J., Park, S., Han, D., Yun, S., Oh, S. J., & Lee, H. (2019) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2019.00481 · Cited by 608 · see citing works
PaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · cited 1× · see citing works
“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · cited 1× · see citing works
“Additionally, the scarcity of training data for this specific task further complicates the recognition process for LMMs.”
Self-citationInfographicVQAPreprint
Mathew, M., Bagal, V., Tito, R. P., Karatzas, D., Valveny, E., & Jawahar, C. V. (2021) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.48550/arxiv.2104.12756 · Cited by 573 · see citing works
DocVQA: A Dataset for VQA on Document Images
Mathew, M., Karatzas, D., & Jawahar, C. V. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00225 · Cited by 570 · cited 1× · see citing works
“We have divided the ESTVQA dataset into ESTVQA(CN) and ESTVQA(EN), which specifically include questions and answers in Chinese or English respectively.”
Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition
Chng, C.-K., & Chan, C. S. (2017) · IEEE International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2017.157 · Cited by 563 · cited 1× · see citing works
“…SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly occluded scene text (WOST) and heavily occluded scene text (HOST); (4 LMMs primarily rely on…”
A robust arbitrary text detection system for natural scene images
Risnumawan, A., Shivakumara, P., Chan, C. S., & Tan, C. L. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2014.07.008 · Cited by 529 · cited 1× · see citing works
“…datasets, including (1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly occluded scene text…”
Recognizing Text with Perspective Distortion in Natural Scenes
Phan, T. Q., Shivakumara, P., Tian, S.-X., & Tan, C.-L. (2013) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.76 · Cited by 491 · cited 1× · see citing works
“…text recognition datasets, including (1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly occluded scene…”
MMBench: Is Your Multi-modal Model an All-around Player?
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.06281 · Cited by 487 · cited 1× · see citing works
“[31, 32] have developed effective systems to assess LMMs. [15, 33, 22] introduces GPT-4 or human evaluation to assess the output of LMMs. MMBench [34] evaluates LMM using multiple-choice questions across various dimensions of ability.”
Self-citationMiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469 · see citing works
ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction
Huang, Z., Chen, K., He, J.-H., Bai, X., Karatzas, D., Lu, S.-J., & Jawahar, C. (2019) · IEEE International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2019.00244 · Cited by 464 · see citing works
Self-citationHow far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4231-5 · Cited by 437 · cited 8× · see citing works
“5-Sonnet [ 70] Anthropic 788 LLaVA-Next-Yi-34B [71] UW–Madison 574 GPT-4o-mini-20240718 [72] OpenAI 785 TextMonkey [30] HUST 561 InternVL2-4B [64] Shanghai AI Lab 784 Monkey-Chat [29] HUST 534 InternVL2-2B [64] Shanghai AI Lab 781 InternLM-XComposer2 [73] Shanghai AI Lab 532 GLM-4v-9B [67] Zhipu AI…”
InfluentialWhat matters when building vision-language models?Preprint
Laurençon, H., Tronchon, L., Cord, M., & Sanh, V. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2405.02246 · Cited by 413 · see citing works
FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents
Jaume, G., Ekenel, H. K., & Thiran, J. (2019) · International Conference on Document Analysis and Recognition Workshops (ICDARW) · doi:10.1109/icdarw.2019.10029 · Cited by 412 · cited 1× · see citing works
“We conduct experiments on SROIE [58], FUNSD [59] and POIE [60].”
Dong, X.-W., Zhang, P., Zang, Y., Cao, Y.-H., Wang, B., Ouyang, L.-K., Wei, X.-L., Zhang, S.-Y., Duan, H., Cao, M., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2401.16420 · Cited by 401 · see citing works
Florence: A New Foundation Model for Computer Vision
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.11432 · Cited by 389 · cited 1× · see citing works
“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”
Self-citationLMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
Zhang, K.-C., Li, B., Zhang, P.-Y., Pu, F., Cahyono, J. A., Hu, K.-R., Dong, Y., Liu, S., Zhang, Y.-H., Yang, J., et al. (2024) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2025.findings-naacl.51 · Cited by 387 · cited 1× · see citing works
“[31, 32] have developed effective systems to assess LMMs. [15, 33, 22] introduces GPT-4 or human evaluation to assess the output of LMMs. MMBench [34] evaluates LMM using multiple-choice questions across various dimensions of ability.”
Self-citationTop-down and bottom-up cues for scene text recognition
Mishra, A., Karteek, A., & Jawahar, C. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247990 · Cited by 359 · cited 1× · see citing works
“Text Recognition : We evaluate LMM using widely-adopted OCR text recognition datasets, including (1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3)…”
Towards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338 · cited 1× · see citing works
“Additionally, the scarcity of training data for this specific task further complicates the recognition process for LMMs.”
Self-citationLLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image UnderstandingPreprint
Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D.-Y., & Sun, T.-F. (2023) · arXiv.org · doi:10.48550/arxiv.2306.17107 · Cited by 335 · cited 1× · see citing works
“Additionally, [22, 23] emphasize modality collaboration for image and text.”
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · see citing works
Scene Text Recognition with Permuted Autoregressive Sequence Models
Bautista, D., & Atienza, R. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2207.06966 · Cited by 332 · cited 3×
“We assess these LMMs on DocVQA[55], InfographicVQA [56] and ChartQA [57].”
Curved scene text detection via transverse and longitudinal sequence connection
Liu, Y.-L., Jin, L., Zhang, S., Luo, C., & Zhang, S. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.02.002 · Cited by 322 · see citing works
Self-citationScene Text Visual Question Answering
Biten, A. F., Tito, R., Mafla, A., Gómez, L., Rusiñol, M., Jawahar, C. V., Valveny, E., & Karatzas, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00439 · Cited by 311 · cited 1× · see citing works
“The responses generated by LMM often include many explanatory terms, so the exact matching approach or Average Normalized Levenshtein Similarity(ANLS) [36] used in the original dataset are not suitable for evaluating LMM in zero-shot scenarios.”
DePlot: One-shot visual language reasoning by plot-to-table translation
Liu, F., Eisenschlos, J. M., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Chen, W., Collier, N. H., & Altun, Y. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.10505 · Cited by 257
Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C.-Y., Wang, L.-J., Liu, Z.-C., & Gao, J.-F. (2022) · Foundations and Trends in Computer Graphics and Vision · doi:10.48550/arxiv.2210.09263 · Cited by 237 · cited 1×
“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”
Self-citationBLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
Hu, W., Xu, Y., Li, Y., Li, W., Chen, Z., & Tu, Z. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2308.09936 · Cited by 233 · cited 1×
“BLIVA [25] combines instruction-aware and global visual features to capture richer image information.”
Li, B., Fang, G., Yang, Y., Wang, Q.-S., Ye, W., Zhao, W., & Zhang, S.-K. (2023) · arXiv.org · doi:10.48550/arxiv.2304.11633 · Cited by 223 · cited 1× · see citing works
“Moreover, a study by Li et al. [104], evaluates ChatGPT across seven detailed information extraction tasks.”
Zhang, P., Dong, X.-W., Zang, Y., Cao, Y.-H., Qian, R., Chen, L., Guo, Q.-P., Duan, H., Wang, B., Ouyang, L.-K., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2407.03320 · Cited by 218 · cited 1× · see citing works
“5 [85] Shanghai AI Lab 686 LLaVA-InternLM2-7B [86] Shanghai AI Lab 402 Qwen-VL-Max [78] Alibaba 684 ShareGPT4V-13B [87] Shanghai AI Lab 398 Gemini-1.”
CogVLM: Visual Expert for Pretrained Language Models
Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.03079 · Cited by 217 · see citing works
Self-citationELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual ModelsPreprint
Li, C.-Y., Liu, H., Li, L., Zhang, P.-C., Aneja, J., Yang, J.-W., Jin, P., Lee, Y. J., Hu, H., Liu, Z.-C., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2204.08790 · Cited by 215 · cited 1× · see citing works
“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”
Self-citationDong, X.-W., Zhang, P., Zang, Y., Cao, Y.-H., Wang, B., Ouyang, L.-K., Zhang, S.-Y., Duan, H., Zhang, W.-W., Li, Y., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2404.06512 · Cited by 199
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2308.12966 · Cited by 193 · see citing works
From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network
Wang, Y., Xie, H., Fang, S., Wang, J., Zhu, S., & Zhang, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01393 · Cited by 188 · cited 1× · see citing works
“…Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly occluded scene text (WOST) and heavily occluded scene text (HOST); (4 LMMs primarily rely on semantic understanding to…”
Peng, B., Li, C., He, P., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2304.03277 · Cited by 183 · cited 1× · see citing works
“Among them, LLaMA [3] is an open-source LLM that matches the performance of GPT-3, followed by Alpaca [4], Vicuna [5], GPT-4-LLM [6] to improve the LLM’s alignment ability to follow human instruction, reporting impressive performance compared with proprietary LLMs.”
Self-citationMonkey: Image Resolution and Text Label are Important Things for Large Multi-Modal Models
Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., & Bai, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02527 · Cited by 174 · cited 2× · see citing works
“…[71] UW–Madison 574 GPT-4o-mini-20240718 [72] OpenAI 785 TextMonkey [30] HUST 561 InternVL2-4B [64] Shanghai AI Lab 784 Monkey-Chat [29] HUST 534 InternVL2-2B [64] Shanghai AI Lab 781 InternLM-XComposer2 [73] Shanghai AI Lab 532 GLM-4v-9B [67] Zhipu AI 776 LLaVA-Next-Vicuna-7B [71]…”
Self-citationStrucTexT: Structured Text Understanding with Multi-Modal Transformers
Li, Y., Qian, Y., Yu, Y., Qin, X., Zhang, C.-Q., Liu, Y., Yao, K., Han, J.-Y., Liu, J.-T., & Ding, E.-R. (2021) · ACM Multimedia · doi:10.1145/3474085.3475345 · Cited by 159 · see citing works
Self-citationMitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., & Wang, L. (2023) · arXiv · doi:10.48550/arxiv.2306.14565 · Cited by 149 · cited 1× · see citing works
“[31, 32] have developed effective systems to assess LMMs. [15, 33, 22] introduces GPT-4 or human evaluation to assess the output of LMMs. MMBench [34] evaluates LMM using multiple-choice questions across various dimensions of ability.”
Generative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147 · see citing works
Feng, H., Liu, Q., Liu, H., Tang, J.-Q., Zhou, W.-G., Li, H.-Q., & Huang, C. (2023) · Science China Information Sciences · doi:10.1007/s11432-024-4250-y · Cited by 147 · cited 1× · see citing works
“Docpedia [28] directly processes visual input in the frequency domain rather than the pixel space.”
On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering
Wang, X., Liu, Y.-L., Shen, C.-H., Ng, C. C., Luo, C., Jin, L., Chan, C.-S., van den Hengel, A., & Wang, L. (2020) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.2002.10215 · Cited by 135 · cited 1×
“Scene Text-Centric VQA : We test LMMs on STVQA [51], TextVQA [52], OCRVQA [53] and ESTVQA [54].”
Self-citationICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboard
Zhang, R., Yang, M., Bai, X., Shi, B., Karatzas, D., Lu, S., Jawahar, C. V., Zhou, Y., Jiang, Q., Song, Q., et al. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00253 · Cited by 132 · see citing works
Self-citationERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding
Peng, Q.-M., Pan, Y.-X., Wang, W., Luo, B., Zhang, Z., Huang, Z.-J., Hu, T., Yin, W., Chen, Y., Zhang, Y., et al. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.06155 · Cited by 114
Syntax-Aware Network for Handwritten Mathematical Expression Recognition
Yuan, Y., Liu, X., Dikubab, W., Liu, H., Ji, Z.-L., Wu, Z.-Q., & Bai, X. (2022) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00451 · Cited by 108 · cited 1× · see citing works
“Handwritten Mathematical Expression Recognition(HMER) : We evaluate on HME100K [61], which consists of 74,502 images for training and 24,607 images for testing, with 245 symbol classes.”
Self-citationTextMonkey: An OCR-Free Large Multimodal Model for Understanding Document
Liu, Y., Yang, B., Liu, Q., Li, Z., Ma, Z., Zhang, S., & Bai, X. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3653415 · Cited by 103 · cited 1× · see citing works
“Monkey [29] enhanced the LMM’s ability to perceive details at a low cost by the generated detailed caption and a high-resolution model architecture.”
Self-citationVLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. (2024) · Proceedings of the 32nd ACM International Conference on Multimedia · doi:10.1145/3664647.3685520 · Cited by 95
Self-citationFeng, H., Wang, Z., Tang, J.-Q., Lu, J., Zhou, W.-G., Li, H.-Q., & Huang, C. (2023) · arXiv.org · doi:10.48550/arxiv.2308.11592 · Cited by 92 · cited 1×
“MiniGPT4V2 [26] uses unique identifiers for different tasks when training the model to better distinguish each task instruction effortlessly.”
Toward Understanding WordArt: Corner-Guided Transformer for Scene Text Recognition
Xie, X.-D., Fu, L., Zhang, Z., Wang, Z.-W., & Bai, X. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2208.00438 · Cited by 75
Self-citationOBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A. M., Kiela, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.16527 · Cited by 72 · see citing works
When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition
Li, B., Yuan, Y., Liang, D.-K., Liu, X., Ji, Z.-L., Bai, J.-F., Liu, W.-Y., & Bai, X. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2207.11463 · Cited by 69
Self-citationCambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16860 · Cited by 68
Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solution
Kuang, J., Hua, W., Liang, D.-K., Yang, M.-K., Jiang, D., Ren, B., Zhou, Y., & Bai, X. (2023) · IEEE International Conference on Document Analysis and Recognition · doi:10.48550/arxiv.2305.07498 · Cited by 66 · cited 1×
“We conduct experiments on SROIE [58], FUNSD [59] and POIE [60].”
Self-citationICDAR 2019 Competition on Scene Text Visual Question Answering
Biten, A. F., Tito, R., Mafla, A., Gómez, L., Rusiñol, M., Mathew, M., Jawahar, C. V., Valveny, E., & Karatzas, D. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00251 · Cited by 60 · cited 1× · see citing works
“Scene Text-Centric VQA : We test LMMs on STVQA [51], TextVQA [52], OCRVQA [53] and ESTVQA [54].”
AttentionHTR: Handwritten Text Recognition Based on Attention Encoder-Decoder Networks
Kass, D., & Vats, E. (2022) · International Workshop on Document Analysis Systems · doi:10.48550/arxiv.2201.09390 · Cited by 51
He, J., Wang, L., Hu, Y., Liu, N., Liu, H., Xu, X., & Shen, H. T. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01785 · Cited by 40 · cited 1× · see citing works
“Recent work such as the ICL-D3IE [102], introduces an in-context learning framework leveraging large language models like GPT-3 and ChatGPT for document information extraction.”
ICFHR 2014 Competition on Handwritten Digit String Recognition in Challenging Datasets (HDSRC 2014)
Diem, M., Fiel, S., Kleber, F., Sablatnig, R., Saavedra, J. M., Contreras, D., Barrios, J. M., & Oliveira, L. (2014) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr.2014.136 · Cited by 40 · cited 1×
“To confirm this, we create two datasets: Semantic Text (ST) and Non-Semantic Text (NST) using the IIIT5k dictionary.”
End-to-end scene text recognition using tree-structured models
Shi, C., Wang, C., Xiao, B., Gao, S., & Hu, J. (2014) · Pattern Recognition · doi:10.1016/j.patcog.2014.03.023 · Cited by 38 · cited 1× · see citing works
“Text Recognition : We evaluate LMM using widely-adopted OCR text recognition datasets, including (1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3)…”
PaliGemma: A versatile 3B VLM for transfer
Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D., Neumann, M., Alabdulmohsin, I., Tschannen, M., Bugliarello, E., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.07726 · Cited by 17 · cited 1× · see citing works
“[64] Shanghai AI Lab 842 PaliGemma-3B-mix-448 [65] Google 614 CongRong [66] CloudWalk 827 Cambrian-13B [63] NYU 610 GLM-4v [67] Zhipu AI 814 MiniCPM-V-2 [68] OpenBMB 605 MiniMonkey-2B [69] HUST 802 Cambrian-34B [63] NYU 591 InternVL2-8B [64] Shanghai AI Lab 794 CogVLM-17B-Chat [67] Zhipu AI 590…”
Journal, I. (2022) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem14709 · Cited by 13 · cited 1×
“Additionally, LLaVA-Med [106] attempts to extend multimodal instruction-tuning to the biomedical domain and demonstrates excellent chat abilities with domain knowledge.”
Qiao, Y., Chen, H., Wang, J., Zhao, S.-S., Chen, Y., Ye, X., Li, Z., Qi, X., Gao, P., & Xie, G.-T. (2021) · arXiv.org · doi:10.48550/arxiv.2106.15332 · Cited by 9
Self-citationCross-Modal Attention Networks with Modality Disentanglement for Scene-Text VQA
Fang, Z., Li, L., Xie, Z., & Yuan, J.-L. (2022) · IEEE International Conference on Multimedia and Expo · doi:10.1109/icme52920.2022.9859666 · Cited by 8
DUBLIN: Visual Document Understanding By Language-Image Network
Aggarwal, K., Khandelwal, A., Tanmay, K., Mohammed, O., Liu, Q., Choudhury, M., Chauhan, H. H., Som, S., Chaudhary, V., & Tiwary, S. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-industry.65 · Cited by 7 · see citing works
DUBLIN - Document Understanding By Language-Image Network
Aggarwal, K., Khandelwal, A., Tanmay, K., Mohammed, O., Liu, Q., Choudhury, M., Chauhan, H., Som, S., Chaudhary, V., & Tiwary, S. (2023) · arXiv.org · doi:10.48550/arxiv.2305.14218 · Cited by 6
Ovis: Structural Embedding Alignment for Multimodal Large Language Model
Lu, S., Li, Y., Chen, Q.-G., Xu, Z., Luo, W., Zhang, K., & Ye, H.-J. (2024) · arXiv · doi:10.48550/arxiv.2405.20797 · Cited by 5 · see citing works
An Efficient Prototype-Based Model for Handwritten Text Recognition with Multi-loss Fusion
Yu, M., Zhang, H., Yin, F., & Liu, C.-L. (2022) · International Conference on Frontiers in Handwriting Recognition · doi:10.1007/978-3-031-21648-0_28 · Cited by 2
Self-citationOPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Flagged first, then by citations
16 of the 110 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Info- graphicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697-1706, 2022.
- Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. Cord: A consolidated receipt dataset for post-ocr parsing. In Document Intelligence Workshop at Neural Information Processing Systems, 2019.
Cited by
Sorted by citations
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAsPreprint
Abouelenin, A., Ashfaq, A., Atkinson, A., Awadalla, H., Nguyen, B., Bao, J., Benhaim, A., Cai, M., Chaudhary, V., Chen, C., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2503.01743 · Cited by 697 · see citing works
Self-citationMMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429 · see citing works
Self-citationLLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal TrainingPreprint
An, X., Xie, Y., Yang, K.-C., Zhang, W.-K., Zhao, X., Cheng, Z.-X., Wang, Y.-R., Xu, S.-C., Chen, C., Wu, C., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.23661 · Cited by 278 · see citing works
“5 RL (Fast) Qwen 2.5-VL”
InfluentialSelf-citationGLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement LearningPreprint
Hong, W.-Y., Yu, W., Gu, X., Wang, G., Gan, G., Tang, H.-M., Cheng, J.-L., Qi, J., Ji, J., Pan, L.-H., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2507.01006 · Cited by 238 · see citing works
“From solving scientific problems [67; 32; 34] to developing autonomous agents [21; 62; 43], the demands on VLMs have far surpassed simple visual content perception [31], with an increasing emphasis on advanced reasoning abilities.”
Self-citationTextMonkey: An OCR-Free Large Multimodal Model for Understanding Document
Liu, Y., Yang, B., Liu, Q., Li, Z., Ma, Z., Zhang, S., & Bai, X. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3653415 · Cited by 103 · see citing works
“The selection of test datasets and the formulation of evaluation criteria were carried out in accordance with the methodology described in [10].”
InfluentialSelf-citationRoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete
Ji, Y., Tan, H., Shi, J., Hao, X., Zhang, Y., Zhang, H., Wang, P., Zhao, M., Mu, Y., An, P., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00168 · Cited by 32 · see citing works
TokenPacker: Efficient Visual Projector for Multimodal LLM
Li, W., Yuan, Y., Liu, J., Tang, D., Wang, S., Qin, J., Zhu, J., & Zhang, L. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02491-7 · Cited by 63 · see citing works
MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe
Yu, T.-Y., Wang, Z., Wang, C., Huang, F.-W., Ma, W., He, Z., Cai, T., Chen, W., Huang, Y.-X., Zhao, Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.18154 · Cited by 211
“[53], ChartQA [54], TextVQA [55], DocVQA [56], and OmniDocBench [57], testing ability to extract, interpret, and reason about textual information in various visual contexts, including documents and charts.”
Qwen3.5-Omni Technical ReportPreprint
Qwen Team (2026) · arXiv.org · doi:10.48550/arxiv.2604.15804 · Cited by 210 · see citing works
“For the general visual question answering, the model is evaluated on RealWorldQA (Zhang et al., 2024), MMStar (Chen et al., 2024a), HallusionBench (Guan et al., 2024), and SimpleVQA (Cheng et al., 2025).”
Cambrian-S: Towards Spatial Supersensing in Video
Yang, S.-S., Yang, J.-H., Huang, P.-Z., Brown, E., Yang, Z., Yu, Y., Tong, S.-B., Zheng, Z.-H., Xu, Y.-F., Wang, M.-H., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2511.04670 · Cited by 167
Efficient multimodal large language models: a survey
Jin, Y., Li, J., Gu, T., Liu, Y., Zhao, B., Lai, J., Gan, Z., Wang, Y., Wang, C., Tan, X., et al. (2025) · Visual Intelligence · doi:10.1007/s44267-025-00099-6 · Cited by 108 · see citing works
“For tasks requiring fine-grained text recognition, OCR-Bench [205] provides a robust testbed for evaluating optical character recognition capabilities in diverse scenes.”
Self-citationZhang, Y.-F., Lu, X., Yin, S.-K., Fu, C.-Y., Chen, W., Hu, X., Wen, B., Jiang, K.-Y., Liu, C.-Y., Zhang, T.-K., et al. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2508.11630 · Cited by 159
“These benchmarks include Hallucination bench (Li et al., 2023), MMStar (Chen et al., 2024), MMVet Hard (Yu et al., 2024), OCR Bench (Liu et al., 2023), Chart QA (Masry et al., 2022), and BLINK (Fu et al., 2024a).”
Self-citationFrom Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
Zhang, J.-H., Chen, Y., Zhou, Y., Xu, Y., Huang, Z., Mei, J.-L., Chen, J., Yuan, Y.-J., Cai, X., Huang, G., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2503.22976 · Cited by 153
“Existing multi-modal datasets primarily provide single-view or limited multi-view images, often designed for image-text retrieval or basic visual question answering, making them insufficient for advanced 3D reasoning [33, 38, 39, 54].”
ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model
Zhou, Z.-Y., Zhu, Y.-C., Zhu, M., Wen, J.-J., Liu, N., Xu, Z.-Y., Meng, W.-B., Cheng, R., Peng, Y.-X., Shen, C.-M., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2502.14420 · Cited by 151
“We also tested against more challenging benchmarks designed for MLLMs, i.e., MMMU [62], MMStar [9], MME [19], OCR-Bench [35], HallBench [20] and MMBench [34].”
MMSearch-R1: Incentivizing LMMs to Search
Wu, J., Deng, Z.-H., Li, W., Liu, Y., You, B., Li, B., Ma, Z.-J., & Liu, Z. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2506.20670 · Cited by 137
Self-citationBLIP-3: A Family of Open Large Multimodal Models
Xue, L., Shu, M., Awadalla, A., Wang, J., Yan, A., Purushwalkam, S., Zhou, H., Prabhu, V., Dai, Y.-T., Ryoo, M. S., et al. (2024) · 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw69036.2025.00644 · Cited by 129 · see citing works
Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H. T., & Shi, G. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00147 · Cited by 72 · see citing works
“…[204], MM-Vet [234] Optical Character Recognition (OCR) Test models’ ability to extract objects from visual inputs MM-Vet [234], OCRBench [140], MME [54], MMT-Bench [232] Chart graphic understanding Evaluates models’ ability to interpret graphic-related data infographic VQA [152], AI2D…”
Self-citationDeepEyesV2: Toward Agentic Multimodal ModelPreprint
Hong, J., Zhao, C.-X., Zhu, C.-L., Lu, W., Xu, G.-H., & Yu, X. (2025) · arXiv.org · doi:10.48550/arxiv.2511.05271 · Cited by 122 · see citing works
Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
Zhang, Q.-Z., Liu, M.-Z., Li, L.-C., Lu, M., Zhang, Y., Pan, J.-W., She, Q., & Zhang, S.-H. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2506.10967 · Cited by 122
“…MMBench [Liu et al., 2025], MMBench-CN [Liu et al., 2025] and MM-Vet [Yu et al., 2023], 4 text-oriented VQA tasks such as TextVQA [Singh et al., 2019], ChartQA [Masry et al., 2022], AI2D [Kembhavi et al., 2016] and OCRBench [Liu et al., 2024e], and 1 multi-turn dialog task MMDU [Liu et al., 2024f].”
X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again
Geng, Z., Wang, Y., Ma, Y., Li, C., Rao, Y., Gu, S.-Y., Zhong, Z., Lu, Q.-L., Hu, H., Zhang, X., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2507.22058 · Cited by 120
Self-citationRADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models
Heinrich, G., Ranzinger, M., Yin, H.-X., Lu, Y., Kautz, J., Tao, A., Catanzaro, B., & Molchanov, P. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.02094 · Cited by 118 · see citing works
“We leverage insights from Cambrian[42] to select the benchmarks that are most sensitive to the quality of visual features report scores on TextVQA[40], ChartQA[27], DocVQA[29], InfoVQA[28] and OCRBench[26].”
VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language ModelsPreprint
Xu, W.-Y., Wang, J., Wang, W.-Y., Chen, Z., Zhou, W.-G., Yang, A.-J., Lu, L., Li, H.-Q., Wang, X.-H., Zhu, X.-Z., et al. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2504.15279 · Cited by 114 · see citing works
“Subsequent works expand the capability coverage of benchmarks into more specialized domains: OCRBench [49], Chartqa [56] and DocVQA [57] assess textual content extraction; AgentBench [48] and ToolEyes [86] test tool usage capabilities; and egocentric perception benchmarks [54, 17] quantify…”
Arif, K. H. I., Yoon, J., Nikolopoulos, D. S., Vandierendonck, H., John, D., & Ji, B. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2408.10945 · Cited by 114
“To effectively distribute, we first introduce a hyperparameter α ∈ [0 , 1] to determine the token budget for the full image.”
Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding
Yu, R., Ma, X., & Wang, X.-C. (2025) · arXiv.org · doi:10.48550/arxiv.2505.16990 · Cited by 110
FastVLM: Efficient Vision Encoding for Vision Language Models
Vasu, P. K. A., Faghri, F., Li, C.-L., Koc, C., True, N., Antony, A., Santhanam, G., Gabriel, J., Grasch, P., Tuzel, O., et al. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.01841 · Cited by 108
“4, we report performance of FastVLM on ChartQA [63], OCRBench [56] and InfoVQA [64] to compare FastVLM against recent methods on text-rich benchmarks.”
Self-citationLuo, G., Yang, X., Dou, W., Wang, Z., Liu, J., Dai, J., Qiao, Y., & Zhu, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02324 · Cited by 23 · see citing works
Scaling Spatial Intelligence with Multimodal Foundation Models
Cai, Z., Wang, R.-S., Gu, C.-Y., Pu, F., Xu, J., Wang, Y.-B., Yin, W.-Q., Yang, Z.-T., Wei, C., Sun, Q., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2511.13719 · Cited by 101
“…training on spatial intelligence data ( i.e. , SenseNova-SI-8M), we evaluate its performance on four additional multimodal benchmarks: MMBench-En [33] and MMStar [9] for holistic multimodal understanding, AI2D [22] for scientific diagram reasoning, and OCRBench [34] for character recognition (OCR).”
InfluentialLu, S.-Y., Li, Y., Xia, Y., Hu, Y.-W., Zhao, S.-S., Ma, Y., Wei, Z.-C., Li, Y.-L., Duan, L.-H., Zhao, J.-S., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2508.11737 · Cited by 101
“It also scores 87.9 on OCRBench, underscoring its precise text recognition, and 65.1 on HallusionBench, indicating fewer hallucinations and higher reliability.”
InfluentialMiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
Cui, J.-B., Xu, B.-K., Wang, C., Yu, T., Sun, W.-Y., Xu, Y., Wang, T.-R., He, Z., Ma, W., Cai, T., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.27393 · Cited by 100
“On OCRBench [41], TextVQA [42], and DocVQA [43], MiniCPM-o 4.5 is on par with top-tier models.”
InfluentialMonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm
Li, Z., Liu, Y.-L., Liu, Q., Ma, Z., Zhang, Z., Zhang, S., Yang, B., Guo, Z., Zhang, J.-R., Wang, X., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2506.05218 · Cited by 98
“These findings indicate that while smaller models are sufficient for simpler tasks, a larger parameter capacity is crucial for maintaining performance on complex tasks.”
Self-citationYang, R., Zhu, Z., Li, Y.-W., Huang, J.-J., Yan, S., Zhou, S., Liu, Z., Li, X.-T., Li, S.-Y., Wang, W.-Q., et al. (2025) · European Conference on Computer Vision · doi:10.48550/arxiv.2511.05491 · Cited by 95
InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation
Yang, S., Li, H., Chen, Y.-L., Wang, B., Tian, Y., Wang, T., Wang, H.-Q., Zhao, F., Liao, Y., & Pang, J.-M. (2025) · arXiv.org · doi:10.48550/arxiv.2507.17520 · Cited by 83
“We use the automatic evaluation from VLMEvalKit (Duan et al., 2024) including MMMU(Val) (Yue et al., 2024), MMStar (Chen et al., 2024a), MME (Fu et al., 2024), OCRBench (Liu et al., 2024e), HallB(Avg) (Guan et al., 2024), MMB(Dev En V1.”
Deng, H., Zou, D., Ma, R., Luo, H.-C., Cao, Y., & Kang, Y. (2025) · arXiv.org · doi:10.48550/arxiv.2503.07065 · Cited by 81
“• OCRBench [27] converts text in images into readable format, is a fundamental task in document understanding.”
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward ModelPreprint
Zang, Y., Dong, X.-W., Zhang, P., Cao, Y.-H., Liu, Z.-Y., Ding, S.-Y., Wu, S.-X., Ma, Y.-B., Duan, H., Zhang, W.-W., et al. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2501.12368 · Cited by 80 · see citing works
“We also report the performance on other categories, such as MMBench [52], MMMU [106] and MMStar [12] for general knowledge, MathVerse [113] and MathVision [89] for math reasoning, TextVQA [79], ChartQA [63] and OCRbench [49] for text-rich document understanding.”
Kwai Keye-VL 1.5 Technical Report
Yang, B., Wen, B., Ding, B., Liu, C.-Y., Chu, C., Song, C., Rao, C.-L., Yi, C., Li, D., Zang, D., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.01563 · Cited by 76
Self-citationEagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language ModelsPreprint
Chen, G., Li, Z.-Q., Wang, S.-H., Jiang, J.-D., Liu, Y., Lu, L.-D., Huang, D.-A., Byeon, W., Lê, M., Rintamaki, T., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2504.15271 · Cited by 76 · see citing works
“The model also performs well in optical character recognition with 869 on OCRBench [65], comparable to other models in its category.”
Self-citationChain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
Qin, Y.-M., Wei, B., Ge, J.-X., Kallidromitis, K., Fu, S., Darrell, T., & Wang, X.-D. (2025) · European Conference on Computer Vision · doi:10.48550/arxiv.2511.19418 · Cited by 75
“Besides, we also evaluate C O VT on some non-vision-centric visual benchmarks such as OCRBench [32], MME [17], MUIRBench [48], HallusionBench [21], A-OKVQA [40], TaskMeAny-thing [68], WeMATH [38], and WorldMedQA-V [36].”
VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception
Yan, Z., He, Y.-N., Li, X.-H., Yue, Z.-R., Zeng, X.-Y., Wang, Y.-L., Qiao, Y., Wang, L.-M., & Wang, Y. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2509.21100 · Cited by 75
“Additionally, on OCRBench [37], the score increases from 856 to 866, reflecting improved fine-grained visual perception.”
Wang, G., Zhao, S.-S., Zhang, X., Cao, L., Zhan, P., Duan, L.-H., Lu, S.-Y., Fu, M.-H., Chen, X., Zhao, J.-S., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2506.23044 · Cited by 75
“…Multi-modal Academic Benchmarks 1 , including MMBench (Liu et al., 2024a), MMStar (Chen et al., 2024b), MMMU-Val (Yue et al., 2024), MathVista-Mini (Lu et al., 2023), HallusionAvg (Guan et al., 2024), AI2D-Test (Kembhavi et al., 2016), OCRBench (Liu et al., 2024b), MMVet (Yu et al., 2024).”
Yue, Z., Lin, Z., Song, Y.-H., Wang, W.-K., Ren, S.-Q., Gu, S.-H., Li, S., Li, P.-D., Zhao, L., Li, L., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2506.03569 · Cited by 73
Self-citationEfficient GPT-4V level multimodal large language model for deployment on edge devices
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Chen, C., Li, H., Zhao, W., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-61040-5 · Cited by 50 · see citing works
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Chen, K., Gou, Y.-H., Huang, R., Liu, Z.-L., Tan, D.-X., Xu, J., Wang, C., Zhu, Y., Zeng, Y., Yang, K., et al. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.00513 · Cited by 65 · see citing works
Self-citationLarge multimodal models evaluation: a survey
Zhang, Z.-C., Wang, J.-Y., Wen, F.-R., Guo, Y.-J., Zhao, X., Fang, X., Ding, S.-Y., Jia, Z.-H., Xiao, J.-H., Shen, Y., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-025-4676-4 · Cited by 64 · see citing works
Self-citationEgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering
Zhou, S., Xiao, J., Li, Q., Li, Y.-C., Yang, X., Guo, D., Wang, M., Chua, T.-S., & Yao, A. (2025) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.00319 · Cited by 64 · see citing works
“Recent advancements in Multimodal Large Language Models (MLLMs) [38, 49, 52] have shown significant potential in addressing scene-text VQA tasks [2, 23, 35].”
EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience
Xue, T., Peng, C., Huang, M.-Q., Guo, L., Han, T., Wang, H., Wang, J., Zhang, X.-C., Yang, X., Zhao, D., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.15876 · Cited by 58
Self-citationUniToken: Harmonizing Multimodal Understanding and Generation Through Unified Visual Encoding
Jiao, Y., Qiu, H.-B., Jie, Z.-Q., Chen, S.-X., Chen, J.-J., Ma, L., & Jiang, Y.-G. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00345 · Cited by 56 · see citing works
MM-IFEngine: Towards Multimodal Instruction Following
Ding, S.-Y., Wu, S.-X., Zhao, X., Zang, Y., Duan, H., Dong, X.-W., Zhang, P., Cao, Y.-H., Lin, D.-H., & Wang, J. (2025) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.00110 · Cited by 55
“We first take an analysis of the performance of the current open-source MLLMs and proprietary MLLMs on several benchmarks [25, 34], and find that for instruction-following capability, the most powerful open-source MLLM like InternVL2.”
MBQ: Modality-Balanced Quantization for Large Vision-Language Models
Li, S.-Y., Hu, Y.-C., Ning, X.-F., Liu, X.-H., Hong, K., Jia, X., Li, X.-H., Yan, Y.-Q., Ran, P., Dai, G.-H., et al. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.00394 · Cited by 54
“Specifically, we use OCRBench [32] and TextVQA [40] for text recognition and comprehension, VizWiz [18] and SEED-Bench [19] to test visual perception, and ScienceQA [34] and MMMU [47] to evaluate visual reasoning.”
DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search
Narayan, K., Xu, Y., Cao, T., Nerella, K., Patel, V. M., Shiee, N., Grasch, P., Jia, C., Yang, Y.-F., & Gan, Z. (2025) · arXiv.org · doi:10.48550/arxiv.2510.12801 · Cited by 49
“To quantify the impact of SFT+RL training on the proposed model’s general VQA and reasoning ability, we benchmark DeepMMSearchR1-7B (RL) on a range of benchmarks, including OCRBench (Liu et al., 2024d), MMVet (Yu et al., 2023), AI2D (Kembhavi et al., 2016), MathVista (Lu et al., 2024a), MMBench (Liu et al., 2024c), Do…”
Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning
Liu, Y., Zhang, B., Zang, Y., Cao, Y.-H., Xing, L., Dong, X.-W., Duan, H., Lin, D.-H., & Wang, J. (2025) · arXiv.org · doi:10.48550/arxiv.2510.27606 · Cited by 48
“Second, we evaluate fine-grained visual perception on OCRBench [40], ChartQA test [45], and SeedBench2-plus [31], which require accurate recognition of dense text, charts, and detailed visual information.”
Self-citationKe, W., Zheng, Y., Li, Y., Xu, H., Nie, D., Wang, P., & He, Y. (2025) · ACM Transactions on Information Systems · doi:10.1145/3768156 · Cited by 30 · see citing works
Leveraging large language models for structured information extraction from pathology reports
Balasubramanian, J. B., Adams, D., Roxanis, I., de González, A. B., Coulson, P., Almeida, J. S., & García‐Closas, M. (2025) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2025.100521 · Cited by 31 · see citing works
“35 Evaluating the ability of LLMs tohandle reports with multiple specimen descriptions is another important area for future study.”
Beyond Language Modeling: An Exploration of Multimodal Pretraining
Tong, S., Fan, D., Nguyen, J., Brown, E., Zhou, G.-Y., Qian, S., Zheng, B., Vallaeys, T., Han, J., Fergus, R., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.03276 · Cited by 42
OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations
Ouyang, L., Qu, Y., Zhou, H., Zhu, J., Zhang, R., Lin, Q., Wang, B., Zhao, Z., Jiang, M., Zhao, X., et al. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.02313 · Cited by 61
“OCRBench [31] proposes five OCR-related tasks with a greater emphasis on evaluating the model’s visual understanding and reasoning capabilities.”
Self-citationStep-GUI Technical ReportPreprint
GELab-Team, & StepFun (2025) · arXiv.org · doi:10.48550/arxiv.2512.15431 · Cited by 41 · see citing works
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
You, K., Zhang, H., Schoop, E., Weers, F., Swearngin, A., Nichols, J., Yang, Y., & Gan, Z. (2024) · arXiv · doi:10.48550/arxiv.2404.05719 · Cited by 40 · see citing works
SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture
Diao, H.-W., Wu, P.-H., Deng, H.-M., Wang, J.-H., Bai, S.-H., Wu, S.-L., Fan, W.-C., Ye, W., Tong, W.-W., Fan, X.-Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.12500 · Cited by 40
“…(MMMU [159], MMMU-Pro [160], MathVista [87], MathVision [132]), general VLM understanding (MMBench-EN [82], MMStar [16]), OCR (InfoVQA [94], OCRBench [83], OCRBench-v2 [37], AI2D [51]), hallucination detection (HallusionBench [49]), and advanced visual reasoning (BabyVision [15], TiR [68]).”
Xu, M.-Z., Gao, M.-F., Li, S.-Y., Lu, J.-S., Gan, Z., Lai, Z.-F., Cao, M., Kang, K., Yang, Y.-F., & Dehghan, A. (2025) · arXiv.org · doi:10.48550/arxiv.2503.18943 · Cited by 40
“We include OCRBench (Liu et al., 2024d), TextVQA (Singh et al., 2019) and DocVQA (Mathew et al., 2021) measuring OCR, scene text and document understanding, respectively.”
InfluentialChatRex: Taming Multimodal LLM for Joint Perception and Understanding
Jiang, Q., Luo, G., Yang, Y.-Q., Xiong, Y.-D., Zeng, Z.-Y., Chen, Y.-H., Ren, T.-H., & Zhang, L. (2024) · IEEE transactions on multimedia · doi:10.48550/arxiv.2411.18363 · Cited by 40
“…on various academic multimodal benchmarks including MME [23], MMBench [53], SEED I [35], MMstar [11], MM-Vet [96], MMMU [98], AI2D [31], OCRBench [54], TextVQA [77], POPE [40], and HallusionBench [24] shown in Table 3, ChatRex demonstrates strong multimodal capabilities, though there remains a…”
LongCat-Next: Lexicalizing Modalities as Discrete Tokens
Xiao, M., Wang, C., Li, C., Zhang, C., Peng, C., Yu, H., Yang, H., Yan, H., Sun, H., Zhao, H., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.27538 · Cited by 39
“To comprehensively evaluate the model’s capability in document understanding and information extraction across diverse document types, we conduct assessments on OCRBench [72], OCRBenchV2 [33], DocVQA [79], and OmniDocBench [88].”
Self-citationAToken: A Unified Tokenizer for Vision
Lu, J.-S., Song, L., Xu, M.-Z., Ahn, B., Wang, Y., Chen, C., Dehghan, A., & Yang, Y.-F. (2025) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.2509.14476 · Cited by 39
“…on 7 standard benchmarks, including RW-QA * , AI2D (Kembhavi et al., 2016), SQA (Lu et al., 2022b), and MMMU (Yue et al., 2024), and MathVISTA (Lu et al., 2024b) for general image QA, as well as OCRBench (Liu et al., 2024a) and TextVQA (Singh et al., 2019) for text and document understanding.”
DocVLM: Make Your VLM an Efficient Reader
Nacson, M. S., Aberdam, A., Ganz, R., Ben-Avraham, E., Golts, A., Kittenplon, Y., Mazor, S., & Litman, R. (2024) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.02701 · Cited by 39
LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
Wang, X., Li, C., Yang, J., Zhang, K., Liu, B., Xiong, T., & Huang, F.-R. (2025) · arXiv.org · doi:10.48550/arxiv.2509.00676 · Cited by 38
“We conduct experiment on five multimodel reasoning and understanding benchmarks: MathVista, MathVision, MathVerse, MMMU, and MMStar.”
InfluentialSelf-citationGLM-5V-Turbo: Toward a Native Foundation Model for Multimodal AgentsPreprint
GLM-5V-Turbo Team (2026) · arXiv.org · doi:10.48550/arxiv.2604.26752 · Cited by 37 · see citing works
“…[23] and PointBench [6] respectively), video understanding (+5.6% on MVBench [24]), 3D grounding (+7.7% on SUNRGBD [33]), OCR (+4.2% on OCRBench [25]), and chart understanding (+7.7% on CharXiv [40]); on reasoning -heavy tasks such as STEM (+1.8% on MMMU_Val [46], MMMU_Pro [47], MathVista [26]…”
InfluentialTUNA: Taming Unified Visual Representations for Native Unified Multimodal Models
Liu, Z.-H., Ren, W.-M., Liu, H.-Z., Zhou, Z.-J., Chen, S., Qiu, H., Huang, X., An, Z.-C., Yang, F., Patel, A., et al. (2025) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.2512.02014 · Cited by 37
“…understanding capabilities on nine benchmarks, including general VQA benchmarks such as MME [11], GQA [18], RealWorldQA [66] and SEED-Bench [22]; knowledge-intensive benchmarks such as MMMU [78], MMStar [5], and AI2D [20]; and text-centric benchmarks including ChartQA [36] and OCRBench [32].”
Lei, W., Wang, J., Wang, H.-C., Li, X.-T., Liew, J., Feng, J.-S., & Huang, Z.-L. (2025) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.01930 · Cited by 36
“Specifically, MLLM benchmarks encompass MMBench-EN [50], SEEDBench-IMG [40], MMVet [85], MME [46], HallusionBench [30], MathVista MINI [53], and OCR-Bench [51].”
MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale
Wang, B., He, T.-Y., Ouyang, L.-K., Wu, F., Zhao, Z.-Y., Chu, T., Qu, Y., Jin, Z.-J., Zeng, W., Miao, Z.-Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.04771 · Cited by 35
“OmniDocBench [27] integrates all three metrics and provides one of the most comprehensive document parsing evaluation frameworks to date; OCRBench [20] and CC-OCR [44] focus on evaluating multimodal models’ overall OCR capabilities.”
Self-citationKwai Keye Team, Yang, B., Wen, B., Liu, C.-Y., Chu, C., Song, C.-R., Rao, C.-L., Yi, C., Li, D., Zang, D., et al. (2025) · doi:10.48550/arxiv.2507.01949 · Cited by 34
Self-citationNemotron 3 Nano Omni: Efficient and Open Multimodal IntelligencePreprint
Deshmukh, N. A. S., Chumachenko, K., Rintamaki, T., Lê, M., Poon, T., Taheri, D., Karmanov, I., Liu, G., Seppanen, J., Goel, A., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.24954 · Cited by 33 · see citing works
“In particular, it attains leading results in document understanding, audio-visual reasoning, and audio benchmarks, ranking at or near the top of leaderboards such as OCRBench-V2 (Liu et al., 2024), MMLongBench-DOC (Ma et al., 2024), VoiceBench (Chen et al., 2024a), WorldSense (Hong et al., 2026), and DailyOmni (Zhou e…”
Self-citationLongCat-Flash-Omni Technical ReportPreprint
Meituan LongCat Team (2025) · arXiv.org · doi:10.48550/arxiv.2511.00279 · Cited by 33 · see citing works
“• Text Recognition & Document Understanding : ChartQA [Masry et al., 2022], DocVQA [Mathew et al., 2021], and OCRBench [Liu et al., 2024c], OmniDocBench EN/ZH [Ouyang et al., 2024].”
InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue
Tong, W.-W., Guo, H.-W., Ran, D.-C., Chen, J.-N., Lu, J.-F., Wang, K., Li, K.-Q., Zhu, X.-X., Li, J., Li, K.-H., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.13747 · Cited by 33
“1 [99], MMStar [24], MMMU [183], MathVista [103], HallusionBench [60], AI2D [79], and OCRBench [100].”
Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer
Huang, Z.-Y., Zheng, D.-D., Zou, C., Liu, R., Wang, X.-L., Ji, K., Chai, W., Sun, J.-X., Wang, L.-B., Lv, Y.-J., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.06590 · Cited by 33
“However, we observe a performance gap on OCRBench (Liu et al., 2024b) and MMMU (Yue et al., 2024), where fine-grained recognition is critical.”
Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence
Karlinsky, L., Arbelle, A., Daniels, A., Nassar, A. S., Alfassi, A., Wu, B., Schwartz, E., Joshi, D., Kondic, J., Shabtay, N., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2502.09927 · Cited by 33
MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models
Liu, Z.-Y., Zang, Y., Dong, X.-W., Zhang, P., Cao, Y.-H., Duan, H., He, C.-H., Xiong, Y., Lin, D.-H., & Wang, J. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.17637 · Cited by 33
“Subse-quently, we also test the model on several single-image benchmarks: MMStar (Chen et al., 2024a), ScienceQA (Lu et al., 2022), MMVet (Yu et al., 2023), POPE (Li et al., 2023c), MMBench (Liu et al., 2023), MathVista (Lu et al., 2023), AI2D (Kembhavi et al., 2016), and OCRBench (Liu et al., 2024c).”
Scaling Language-Centric Omnimodal Representation Learning
Xiao, C.-H., Chan, H.-P., Zhang, H., Xu, W.-W., Aljunied, M., & Rong, Y. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2510.11693 · Cited by 32
“Generative performance is measured by averaging results from TextVQA [56], DocVQA [45], OCRBench [42], and ChartQA [44].”
Yang, C., Dong, X., Zhu, X., Su, W., Wang, J., Tian, H., Chen, Z., Wang, W., Lu, L., & Dai, J. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02322 · Cited by 10 · see citing works
Valley2: Exploring Multimodal Models with Scalable Vision-Language Design
Wu, Z.-H., Chen, Z.-H., Luo, R.-P., Zhang, C., Gao, Y., He, Z.-T., Wang, X., Lin, H., & Qiu, M.-H. (2025) · arXiv.org · doi:10.48550/arxiv.2501.05901 · Cited by 31
“Vision and OCR benchmarks (e.g., OCR-Bench [45], ChartQA [58], RealWorldQA) primarily target visual perception and textual recognition.”
Visual-Advantage On-Policy Distillation for Vision-Language Models
Liu, R., Lv, X., Li, G., Zhu, X.-M., Wang, Z., Zhang, Z.-B., Chen, J., Li, Z.-H., Li, B., Gao, J., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.21924 · Cited by 30
“We evaluate on eight benchmarks grouped into mathematical reasoning (WeMath [11], MathVista [12], MathVerse [13]) and visual understanding (HallusionBench [14], AI2D [15], MMMU [16], MMStar [17], OCRBench [18]).”
Self-citationSTEP3-VL-10B Technical ReportPreprint
Huang, A., Yao, C.-Y., Han, C.-R., Wan, F.-Q., Guo, H.-Y., Lv, H.-R., Zhou, H.-Y., Wang, J., Zhou, J., Sun, J.-J., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.09668 · Cited by 30 · see citing works
“S TEP 3-VL-10B exhibits frontier-class document intelligence, achieving 86.75% on OCRBench and 89.35% on AI2D.”
InfluentialSelf-citationVINO: A Unified Visual Generator with Interleaved OmniModal Context
Chen, J.-Y., He, T., Fu, Z.-J., Wan, P.-F., Gai, K., & Ye, W.-C. (2026) · arXiv.org · doi:10.48550/arxiv.2601.02358 · Cited by 30
Huang, M., Shi, Y.-X., Peng, D.-Z., Lai, S., Xie, Z., & Jin, L.-W. (2025) · arXiv.org · doi:10.48550/arxiv.2505.17163 · Cited by 30
“First, since most of our data collection and annotation processes are performed manually, the high costs associated with these processes have resulted in our dataset size being comparable to previous methods [31, 58].”
InfluentialSelf-citationEfficient Multi-modal Large Language Models via Progressive Consistency Distillation
Wen, Z., Wang, S.-B., Zhou, Y., Zhang, J.-Y., Zhang, Q.-T., Gao, Y., Chen, Z.-R., Wang, B., Li, W.-J., He, C.-H., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2510.00515 · Cited by 29
“• OCRBench [46] is a comprehensive benchmark for evaluating the OCR capabilities of multi-modal language models across five key tasks: text recognition, scene text-centric and document-oriented VQA, key information extraction, and handwritten mathematical expression recognition.”
MLLM-CL: Continual Learning for Multimodal Large Language Models
Zhao, H., Zhu, F., Wang, M., Wang, R., Meng, G., & Zhang, Z. (2025) · arXiv.org · doi:10.48550/arxiv.2506.05453 · Cited by 29
“…of the SFT data, we collect the relevant data from LLaVA-OneVision [34], Monkey [37], ScreenQA [22], Screen2Words [58], MultiUI [42], Math-LLaVA [53], MAVIS [71], CLVER [28] and TallyQA [1] and test them on OCRBench [44], MathVista [46], MMTBench-GUI [66] and CV-Bench-Counting [57], respectively.”
Cao, S., Li, J.-Y., Li, X.-H., Pu, Y.-D., Zhu, K., Gao, Y., Luo, S.-Q., Xin, Y., Qin, Q., Zhou, Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2512.21675 · Cited by 28
“MMBench [33] evaluates comprehensive perception and reasoning across everyday images, MathVista [35] centers on mathematical and geometric reasoning within visual scenes, and OCRBench [34] specifically tests a model’s capability to recognize and interpret text embedded in images.”
Self-citationBai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.21631 · Cited by 27 · see citing works
“…(Wang et al., 2024b), MathVision-Wild photo (hereafter MathVision WP ) , MathVista (Lu et al., 2023), We-Math (Qiao et al., 2024), MathVerse (Zhang et al., 2024), DynaMath (Zou et al., 2024), Math-VR (Duan et al., 2025), LogicVista (Xiao et al., 2024), VisualPuzzles (Song et al., 2025b),…”
InfluentialSelf-citationDuan, S., Xue, Y.-Q., Wang, W.-H., Sū, Z., Liu, H., Yang, S., Gan, G., Wang, G., Wang, Z., Yan, S., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.10910 · Cited by 27
“Besides, GLM-OCR delivers strong performance across text recognition, formula recognition, table parsing, and key information extraction, reaching 94.0 on OCRBench (Text) [17] and 96.5 on UniMERNet [32], and achieving 85.2 on PubTabNet [36] and 86.0 on TEDS 1 .”
InfluentialSelf-citationMME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios
Shi, Y., Wang, H.-Q., Xie, W.-L., Zhang, H.-Y., Zhao, L.-J., Zhang, Y.-F., Li, X.-F., Fu, C.-Y., Wen, Z.-E., Liu, W.-T., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2505.21333 · Cited by 27
“This straightforward yet effective method is widely adopted in previous work [21, 24, 55].”
A Survey on Fine-Grained Multimodal Large Language Models
Peng, Y., Wang, Z., Li, G., Zheng, X., Yin, S., & He, H. (2026) · Chinese Journal of Electronics · doi:10.23919/cje.2025.00.336 · Cited by 5 · see citing works
Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and GenerationPreprint
Ма, Й., Zou, C., Du, C.-K., Yan, C., Jin, C.-X., Shen, C.-J., Lian, C.-Y., Fan, C.-Z., Zheng, D.-D., Wang, F.-D., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2510.24821 · Cited by 26 · see citing works
“Extensive experiments demonstrate that Ming-Flash-Omni achieves comparable performance with leading MLLMs. Vision → Text (Understanding) As shown in”
Robust Multimodal Large Language Models Against Modality Conflict
Zhang, Z., Zhou, W., Zhao, J., & Li, H.-Q. (2025) · International Conference on Machine Learning · doi:10.48550/arxiv.2507.07151 · Cited by 26
“To analyze the alignment tax of our methods, we test the performance changes of the models on a wide range of vision-language tasks after the fine-tuning, including HallusionBench (Guan et al., 2024), MMBench (Liu et al., 2025), MMStar (Chen et al., 2024), MMMU (Yue et al., 2024), MathVista (Lu et al., 2024), OCRBench…”
Lee, J., Park, J., Kim, J., Kim, Y., Oh, J., Oh, J., & Choi, J. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2411.09909 · Cited by 26
Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery
Xin, M., Priyadarshi, S., Xin, J.-Y., Kartal, B., Vavre, A., Thekkumpate, A. K., Chen, Z.-J., Mahabaleshwarkar, A., Shahaf, I., Bercovich, A., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2601.20088 · Cited by 25
“We believe this is attributed to the simple training pipeline and also the small accuracy drop from PTQ. Kem-bhavi et al., 2016), ChartQA (Masry et al., 2022), DocVQA (Mathew et al., 2021b), InfoVQA (Mathew et al., 2021a), OCRBench (Liu et al., 2024a)”
SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
Deng, J., Li, W., Zhou, J. T., & He, Y. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2510.24214 · Cited by 25
“To further demonstrate the effectiveness of SCOPE, we conducted additional experiments on more OCR-specific benchmarks including DocVQA [32], ChartQA [31] and OCRBench [28].”
Benchmarking Vision-Language Models on Chinese Ancient Documents: From OCR to Knowledge Reasoning
Yu, H.-Y., Wu, Y.-C., Shi, F., Liao, L., Lu, J., Ge, X., Wang, H., Zhuo, M., Wu, X.-C., Fei, X., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.09731 · Cited by 25
“To alleviate the above problem of single tasks, OCR-Bench (Liu et al. 2024b) and its enhanced version OCR-Bench v2 (Fu et al. 2024) have introduced more diverse settings, covering multiple subtasks, including character recognition, text localization, and some document-level question answering and logical reasoning pro…”
Gong, Z.-Y., Li, W., Ma, X.-Z., Li, S.-Y., Wang, Z., Li, S.-Z., Ji, J.-Y., Yang, X., Luo, G., Yan, J.-C., et al. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2506.07966 · Cited by 25
“With the expanding capabilities of MLLMs, an increasing number of studies have focused on systematically evaluating their performance across diverse tasks, including caption generation [9, 3, 65], visual question answering (VQA) [22, 44, 17, 71, 36, 37, 48, 38], and code generation [27].”
Yu, W., Yang, Z.-B., Wan, J.-Q., Song, S.-B., Tang, J., Cheng, W., Liu, Y.-L., & Bai, X. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3677075 · Cited by 25
“3) We further investigate the incorporation of SPOT prompting into multimodal large language models (MLLMs) for VsTP, a domain where MLLMs traditionally underper-formed [15], [16], [17].”
Self-citationQianfan-OCR: A Unified End-to-End Model for Document Intelligence
Dong, D.-X., Zheng, M., Xu, D., Luo, C., Zhuang, B., Li, Y., He, R., Wang, H., Zhang, W., Wang, W.-B., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.13398 · Cited by 24
“Note that these benchmarks include not only pure OCR recognition but also understanding and key information extraction (KIE) tasks, where specialized OCR models generally underperform.”
InfluentialAdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition
Lin, Z.-C., Liu, Y.-C., Yang, Y., Tao, L., & Ye, D.-H. (2025) · arXiv.org · doi:10.48550/arxiv.2512.03794 · Cited by 23
“We conduct experiments on several general VQA benchmarks, including ChartQA [25], OCRBench [22], DocVQA [26], MME [7], MMVet [47], RealWorldQA [41], POPE [18], MathVista [24], MathVerse [48].”
Self-citationYu, W., Yang, Z.-B., Liu, Y.-L., & Bai, X. (2025) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.00086 · Cited by 23
“To further evaluate DocThinker beyond the Visual CoT Benchmark [16], we assess its performance on OCRBench [12], a widely used benchmark for text-centric multimodal understanding.”
Self-citationAurangzeb, I., & Yoon, J. H. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106286 · Cited by 23 · see citing works
- Authors
- Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., & Bai, X.
- Affiliations
- Huazhong University of Science and Technology (CN)
- South China University of Technology (CN)
- Microsoft (United States) (US)
- University of Science and Technology Beijing (CN)
- Corresponding author
- Bai, Xiang
- Venue
- Science China Information Sciences
- Published
- 2024-12
- Volume
- 67
- Issue
- 12
- ISSN
- 1869-1919 (ISSN-L)
- 1674-733X
- OpenAlex
- W4405399726
- Cited by
- 167
- References
- 110
- Citations by year
- 2026: 82 · 2025: 64 · 2024: 16 · 2023: 5
- FWCI
- 22.68
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Science China Information Sciences (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62225603)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- optical character recognition, large multimodal models, evaluation benchmark, key information extraction, handwritten mathematical expression recognition, multilingual text recognition, GPT-4V, Gemini
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
