OCRBench: on the hidden mystery of OCR in large multimodal models

“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”

“5 [64] Shanghai AI Lab 720 DeepSeek-VL-7B [81] DeepSeek 435 Claude3-Opus [82] Anthropic 694 OmniLMM-12B [83] OpenBMB 420 RekaFlash [74] Reka AI 692 TransCore-M [84] PCI Research 405 InternLM-XComposer2.”

“Among them, LLaMA [3] is an open-source LLM that matches the performance of GPT-3, followed by Alpaca [4], Vicuna [5], GPT-4-LLM [6] to improve the LLM’s alignment ability to follow human instruction, reporting impressive performance compared with proprietary LLMs.”

“We assess these LMMs on DocVQA[55], InfographicVQA [56] and ChartQA [57].”

“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”

“Other works, such as [20, 21], also focus on aligning the vision module and LLM for improved multimodal understanding.”

“…OCR text recognition datasets, including (1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly…”

“4 FTBI-13B can still output correct content when noisy text detection content is introduced, whereas TFI-13B blindly follows the erroneous content across comprehensive VQA benchmarks, including VQA v 2 , GQA ∗ , and MME, as well as perceptual and reasoning tasks evaluated by MMBench and SEED-Bench.”

“To assess whether language supervision preserves the back-bone’s capabilities, we conducted ablation studies comparing Mantis with other VLA models [53, 57] on several VQA and multimodal understanding benchmarks [30, 52].”

“On OCRBench (Liu et al., 2023b), R-4B-RL achieves a robust score of 83.6%, proving its solid text recognition capabilities in diverse scenes.”

“General Vision Understanding Method Vision Tower LLM I n f o VQA [ 46 ] D o c VQA [ 45 ] C h a r t QA [ 44 ] T e x t VQA [ 59 ] O CRB e n c h [ 41 ] O CRB e n c h V 2 [ 41 ] L i v e X i v VQA [ 58 ] O CR A vg A I 2 D [ 29 ] MM B E N Table 1.”

“5-665K (Liu et al., 2023b), LLaVA-Next-779K (Liu et al., 2024b), Bunny-2M (He et al., 2024), and LLaVA-OV-4M (Li et al., 2024) to isolate the effect of data volume.”

“…large language models [33, 3, 78, 37, 10, 43, 17, 57, 18, 67, 73, 72, 21, 52, 23, 77] have facilitated the development of various MLLM benchmarks [36, 9, 35, 68, 26, 66, 19, 28, 79, 16, 29, 76, 21, 42, 15] via scaled pre-training, supervised fine-tuning, and post-training on massive multimodal…”

“• OCRBench (Liu et al., 2024c) evaluates OCR abilities, testing models on diverse text extraction tasks from images with varying fonts, layouts, and noise levels.”

“Further experiments on ICDAR 2015 [25] and OCRBench [26] provide solid evidence that LMMs frequently hallucinate scene text answers based on semantic priors rather than actual visual grounding.”

“This leaderboard contains 8 different multimodal benchmarks, including complex VQA (MMBench [127], MMStar [25], MMMU [255], AI2D [94], and MMVet [249]), multimodal reasoning (MathVista [136]), hallucination evaluation (Hallusion-bench [72]), and OCR (OCRBench [128]).”

“5-Instruct (Yang et al., 2024) here, showing its forgetting behavior during post-training on math reasoning, along with the corresponding results.”

“Performance is reported on a broad suite, including: MME [13], GQA [15], ChartQA [35], POPE [24], TextVQA [41], OCR [31], SEED [17], MMVet [50], and SQA [33].”

“We evaluated baselines using the combat understanding benchmark (CUBench), general benchmarks (i.e., MME [9], VideoMME [10], OCRBench [25]), and practical tests.”

“As shown in Figure 6, on the three understanding benchmarks (OCRBench (Liu et al., 2024), MMVP (Tong et al., 2024) and V* (Wu and Xie, 2024)), Tuna-R actually outperforms Tuna-2 during the early stage of the training.”

“For instance, on the transformed OCRBench subset, CodeVision-7B achieves an average score of 73.4, a +17.4 improvement over its base model, demonstrating its robust ability to recognize the correct transformation and apply the corresponding tool to restore the canonical image view for successful reasoning.”

“Even more significantly, OCR systems fill a critical gap in acquiring high-quality corpora for Large Language Models, acting as an essential instrument for unlocking the content of specialized books and historical archives Zhang et al. (2024b).”

“• Text-rich Document & Chart Understanding : ChartQA [58], TextVQA [78], DocVQA [60], InfoVQA [59], and OCRBench [53].”

“For OCR & Chart understanding, we specifically use AI2D [34] for diagram understanding, SEED-Bench-2-Plus [37] for text-rich visual comprehension, DocVQA [47] for document understanding, and OCRBench [43] for comprehensive OCR evaluation.”

“DREAM is evaluated across eight diverse benchmarks: MMT-Bench [58], SEED-Bench-2 [21], ScienceQA [37], OCRBench [35] , ChartQA [39], and MathVista [36].”

“VLMs are known to possess optical character recognition abilities [36], and so this result is not surprising, but, verifies the AS maps show expected associations.”

“…standard and vision splits of MMMU-Pro (Yue et al., 2025), and CharXiv (Wang et al., 2024), together with general recognition through OCRBench (Liu et al., 2024c), RealWorldQA (xAI, 2024), SimpleVQA (Cheng et al., 2025) for multimodal factuality, and CountQA (Tamarapalli et al., 2025) for…”

“We also observe solid performance on OCRBench [18], AI2D [12], CountBenchQA [27], and RefCOCO [44], demonstrating that dots.mocr maintains broad visual grounding and reasoning abilities beyond parsing.”

“The current OCRBench [17, 8, 37] for evaluating MLLMs primarily targets line-granularity recognition.”

“OCRBench [76] is a comprehensive evaluation benchmark designed to assess the optical character recognition capabilities of large vision-language models across diverse text-related visual understanding tasks.”

“• OCR & Chart benchmark: TextVQA[58], OCRBench[44], ChartQA[51] and DocVQA[53].”

“Modern applications require a comprehensive interface capable of supporting diverse text-centric visual tasks, ranging from document parsing [3–5] and information extraction to visual question answering [6], text-image translation [7, 8], and multi-image document understanding [9, 10].”

“In addition to above, OCRBench(Liu et al., 2024d) supplied the evaluation on text recognition and document understanding.”

“1 [90] (bilingual multimodal benchmark), MMVet [82] (integrated tasks), MMStar [91] (multimodal understanding), HallusionBench [92] (multimodal hallucination), AI2D [78] (science diagrams), and OCRBench [93] (OCR capabilities), RealWorldQA [94] (real-world comprehension).”

“To comprehensively assess their multimodal capabilities, we evaluate them across four representative benchmarks: OCRBench (Liu et al., 2024d) for foundational visual-text recognition, ChartQA (Masry et al., 2022) and MMMU (Yue et al., 2024) for complex multimodal reasoning and domain-specific knowledge, and MM-SafetyB…”

“Current OCR evaluation methods predominantly rely on standardized benchmark metrics [34], creating a disconnect between reported performance and real-world reliability.”

“(2) Neglect visual text tasks : While understanding text in images is key to many MLLM use cases [17], benchmarks for OCR [36] and visual document retrieval remain sparse.”

“However, the evaluation of visual language models has traditionally focused on atomic tasks such as visual question answering [14], image understanding and captions [18], OCR recognition [31], and diagram analysis [40], where single-instance understanding is prioritized over contextual coherence.”

“Unlike OCRBench [7, 17], which evaluates OCR capabilities of LMMs on high-level tasks like VQA, OmniDocBench is more fitting for benchmarking PP-OCRv5 as a specialized text recognition engine, with its focus purely on recognition accuracy.”

“They offer compelling advantages for practical deployment: they enable inference on consumer GPUs and edge devices, support privacy-preserving local processing, and demonstrate superior cost-performance ratios for specialized tasks (Belcak et al., 2025).”

“[31] models on three OCR-related benchmarks (TextVQA [69], OCRBench [70] (from VLMEvalKit [65]), and OCR-VQA [71]) shows that increasing LLM backbone size leads to improved typographic understanding.”

“…et al., 2024) and leaf area index (Chen et al., 2022; Du et al., 2023); text analysis technology can extract valuable germplasm descriptions (L. Liu et al., 2024; Sansaloni et al., 2020), breeding optimization strategies (J. Xu et al., 2023; Yao et al., 2022) and market trends (Garg et al.,…”

“5-1.5B [28] for the language model, and SigLIP ViT-L 384 [30] for the vi-087 Bench [18], ChartQA [21], InfoVQA [22], OCRBench [19], RealWorldQA 1 , and MMStar [4] using the lmms-eval frame-101 work [31].”

“The OCR category features TextVQA [44] and OCRBench [31], both assessing Table 1.”

“Robust OCR: An integration of OCRBench [21] and Kaggle competition data.”

“5 [34], OCRBench [29], TEDS, PubTabNet [53] and FireRedBench.”

“We still employ SimplerEnv for robotic manipulation evaluation, and assess multimodal tasks on MMMU [66], MM-Vet [65], MMStar [5], OCRBench [37], MMB [36], TextVQA [50], InfoVQA [41] and DocVQA [40] for all ablation studies.”

“Multimodal chain-of-thought (MCoT) prompting [27, 66, 46] offers a step-by-step reasoning trajectory when MLLM faces hard questions including detail grounding [72, 42], agent planing [27], etc. Specifically, MCoT decomposes the question into a group of reasoning steps and builds a chain to guide…”

“This includes developing more sophisticated vision encoders (Wang et al., 2024a; Bai et al., 2025; Zhang et al., 2024b; Ge et al., 2024; Luo et al., 2024), supplementing the LLM with full high-resolution images (Liu et al., 2024a; Zhu et al., 2025; Huang et al., 2024; Tong et al., 2024a; Cha et…”

“Image tasks include ChartQA (Masry et al., 2022), DocVQA (Mathew et al., 2021), InfoVQA (Mathew et al., 2022), AI2D (Kembhavi et al., 2016), MMBench-EN (Fu et al., 2023), OCRBench (Liu et al., 2024b), OCRBench v2 (Liu et al., 2024b), MMStar (Chen et al., 2024a), and RealWorldQA (Corp., 2024).”

“…established benchmarks to cover diverse difficulty and varying reliance on modalities: (1) Document-centric OCR and understanding: using OCRBench [17] and SEED-Bench v2 Plus [12], which contain scanned pages, dense layouts, and long-form documents that stress OCR quality, layout parsing, and…”

“73.3), with only minor differences on tasks like OCR [38] (537 vs .”

“OCRBench [33] is a comprehensive benchmark for evaluating the OCR capabilities of multi-modal language models across five key tasks: text recognition, scene text-centric and document-oriented VQA, key information extraction, and handwritten mathematical expression recognition.”

“While MMStar is specifically curated to eliminate language bias, OCRB [53] evaluates fine-grained text recognition capabilities.”

“Perception : Following (Tong et al., 2024a), we conduct experiments on 4 types of perception task ( i.e. , general, knowledge, ocr, and vision-centric) across 16 benchmarks: MME (Fu et al., 2023), MMBench (Liu et al., 2025), SEED (Li et al., 2024), GQA (Hudson & Manning, 2019), ScienceQA (Lu et al., 2022), MMMU (Yue e…”

“Recent advancements in vision-language models (VLMs) have spurred the development of numerous multimodal benchmarks designed to evaluate their performance across a wide array of tasks [29, 12, 19, 54, 30, 37, 36, 33, 16, 49, 39].”

“It includes recognizing and interpreting the visual-linguistic concepts, such as text recognition (OCRBench (Liu et al., 2024d)), Visual Grounding (RefCOCO (Yu et al., 2016), FineCops-Ref (Liu et al., 2024a), HC-RefLoCo (Wei et al., 2024)) and Referring Expression Generation (Visual Genome)…”

“To evaluate text-centric perception, we use TextVQA val , OCRBench, and DocVQA [33, 36, 44].”

“ChartQA [33] OCR [30] Avg performance by 2.51 (3.7”

“Benchmarks cover diverse image datasets (GQA [14], MMBench [31], MME [9], POPE [20], SEED [17], TextVQA [42], VizWiz [11], OCRBench[32]) and video datasets (MVBench [19], EgoSchema [35], MLVU [61], VideoMME [10]).”

“…VLMEvalKit[13] with default prompts: MME[14], MMStar[7], MMBench test en [36], SeedBench image [27], ScienceQA val [53], RealworldQA[65], AI2D w/o M [24], TextVQA val [55], OCRBench[37], ChartQA test [41], DocVQA test [42], MMMU val [71], MathVista mini [38], Video-MME[15], and LongVideoBench[64].”

“Benefiting from large-scale vision-language pretraining, these visual encoders naturally exhibit strong OCR ability and robust text–vision alignment (Yuan et al., 2025; Lin et al., 2025; Liu et al., 2024), making them a promising alternative to conventional text tokenization.”

“In addition, we evaluate on more benchmarks, including OCRBench (Liu et al., 2024c), MMStar (Chen et al., 2024a), MMMU (Yue et al., 2024), MMB 1.1 (Liu et al., 2024b), and POPE (Li al., 2023b), where D E F ACTO achieves competitive performance against both leading closed-source models (e.g., GPT-4o…”

“…[213], MM-Vet [247] Optical Character Recognition (OCR) Test models’ ability to extract objects from visual inputs MM-Vet [247], OCRBench [135], MME [49], MMT-Bench [244] Chart graphic understanding Evaluates models’ ability to interpret graphic-related data infographic VQA [149], AI2D…”

“…et al., 2016), ChartQA (Masry et al., 2022), HallusionBench (Guan et al., 2024), MathVerse (Zhang et al., 2024), MathVision (Wang et al., 2024), MathVista (Lu et al., 2024), MMMU (Yue et al., 2024), MMStar (Chen et al., 2024), OCRBench (Liu et al., 2024b), and ScreenSpot v2 sc (Cheng et al., 2024).”

“On OCR-related benchmarks, Qwen3-VL-8B-Instruct scores 96.1 on DocVQA_test [24] and 896 on OCRBench [21], whereas InternVL3.”

“CQA, DVQA, TVQA, IVQA, OCRB, MMB. MVista, MMS and MMV represents ChartQA [35], DocVQA [36], TextVQA [42], InfoVQA [37], OCRBench [30], MMBench [29], MathVista [31], MMstar [10] and MMVet [64], respectivelly.”

“The evaluation includes 9 image-language benchmarks, namely GQA [15], MMB [25], MME [5], POPE [20], SQA [27], TextVQA [33], OCRBench [26], SEEDBench [18], and OKVQA [29], as well as 4 video-language benchmarks, namely EgoSchema [28], NExTQA [39], VideoMME [12], and EgoPlan [8].”

“5 [11] model, with an input resolution of 336x336 pixels, is effective for general visual reasoning, but it struggles significantly when processing small text or when performing detailed visual analysis [12].”

“We select two detail-sensitive tasks, DocVQA (Tito, Karatzas, and Valveny 2023) and OCR-Bench (Liu et al. 2024f), to evaluate the accuracy degradation introduced by token reduction under different compression ratios on LLaVA-NeXT-7B (Liu et al. 2024c).”

“These models excel in visual perception[15], visual question answering (VQA)[16], and multimodal reasoning[17, 18], and have been applied in medicine[19], autonomous driving[20], remote sensing[21, 22], and OCR[23].”

“Second, to assess the role of OCR capability , we refer to each model’s performance on standard OCR benchmarks such as OCRBench [56].”

“UniAR significantly outperforms existing unified models on OCR-oriented tasks, such as OCRBench [32], DocVQA [39], and InfoVQA [40], as well as video understanding benchmark like MVBench [25].”

“Specifically, text-only LLMs are evaluated on LongBench-E and the Needle-in-a-Haystack (NIAH) benchmark [6, 24]; multi-modal LLMs on OCRBench and DocVQA [37, 39]; and omni-modal LLMs on MMAU-Pro [26].”

“To further demonstrate that our model has the general QA capabilities, we benchmark it on MMbench [21], OCRBench [22], RealWordQA [50], ChartQA [30], and MMStar [4].”

“Similar findings have been reported in previous works on document and OCR-based tasks, where end-to-end MLLMs consistently outperform pipeline-based approaches in both latency and performance [48,49].”

“( iii ) Chart/Table Understanding: OCRBench [65], DocVQA [66], ChartQA [67] (report accuracy).”

“: We evaluate our framework across two benchmark categories demanding fine-grained perception: 1) Document & OCR (DocVQA [29], InfoVQA [30], ChartQA [31], OCRBench [71], and TextVQA [72]); and 2) High-Resolution & Vision-Centric (V* [33], MME-RealWorld [35], and HR-Bench [34]).”

“For visual understanding, we evaluate on POPE [26], GQA [19], MMB [30], SEED [24], DocVQA [33], and OCRB [31], covering object recognition, relational reasoning, and document-based QA across diverse domains.”

“OCR-based document and chart understanding benchmarks include TextVQA [42], AI2D [22], OCRBench [28], and ChartQA [31].”

“…[39] (comprehensive multimodal capability), MMMU [40] (massive multi-discipline multimodal understanding), MMB [25] (multimodal understanding capability), MathVista [28] (mathematical reasoning in visual contexts), OCR-Bench [26] (OCR robustness), and HallusionBench [16] (multimodal hallucination).”