OCRBench: on the hidden mystery of OCR in large multimodal models

“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”

“5 [64] Shanghai AI Lab 720 DeepSeek-VL-7B [81] DeepSeek 435 Claude3-Opus [82] Anthropic 694 OmniLMM-12B [83] OpenBMB 420 RekaFlash [74] Reka AI 692 TransCore-M [84] PCI Research 405 InternLM-XComposer2.”

“Among them, LLaMA [3] is an open-source LLM that matches the performance of GPT-3, followed by Alpaca [4], Vicuna [5], GPT-4-LLM [6] to improve the LLM’s alignment ability to follow human instruction, reporting impressive performance compared with proprietary LLMs.”

“We assess these LMMs on DocVQA[55], InfographicVQA [56] and ChartQA [57].”

“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”

“Other works, such as [20, 21], also focus on aligning the vision module and LLM for improved multimodal understanding.”

“…OCR text recognition datasets, including (1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly…”

“The Charts and OCR/Document group contains ChartQA [23], DocVQA [24], TextVQA [25], and OCRBench [26], and it systematically examines chart reading and integrated OCR capability.”

“…abilities were benchmarked against a comprehensive suite of benchmarks, including MMBench-EN (Liu et al., 2024a), MM-Star (Chen et al., 2024b), OCRBench (Liu et al., 2024b), SEED-Bench-2-Plus (Li et al., 2024b), ScienceQA (Lu et al., 2022), AI2D (Kembhavi et al., 2016), and POPE (Li et al., 2023b).”

“Please refer to Appendix B for more metric details.”

“Thanks to the large-scale and diverse training data, recent LMMs deliver impressive OCR-related performance, such as text recognition [Liu et al. , 2024e] and document parsing [Wei et al. , 2024].”

“All evaluations are conducted using VLMEvalKit [26], assessing performance across 16 popular benchmarks, including GQA [42], DocVQA [60], InfoVQA [61], ScienceQA [58], TextVQA [77], VizWiz [36], OCRVQA [63], OCRBench [56], MME [30], MMMU [98], SEEDBench_IMG [48], MathVista_MINI [59], AI2D [46], HallusionBench [34], PO…”

“For datasets containing far more than 5000 image-question pairs, we selected the first 5000 pairs for the test, similar to [84].”

“We assess quantized models on the following benchmarks: (1) Multimodal benchmarks, including MME [13], OCR-Bench [32], DocVQA [37], RealWorldQA [57], and VLMBlind.”

“As shown in Table 1, they achieve competitive or leading results on a broad range of understanding benchmarks, including MMStar, SEED, MME and OCRBench, Table 2: Comparison with state-of-the-art text-to-image generation methods on GenEval (Ghosh et al., 2023) and DPG-Bench (Hu et al., 2024). demonstrating that task-pr…”

“For general utility, we follow prior LVLM work and report performance on OCRBench [17, 29], MathVista [30] and MMStar [5].”

“OCRBench [29] OCRBench contains 29 datasets, including Text Recognition, Scene Text-Centric Visual Question Answering (VQA), Document-Oriented VQA, Key Information Extraction (KIE), and Handwritten Mathematical Expression Recognition (HMER), making it the most comprehensive OCR evaluation benchmark.”

“To evaluate the multimodal capabilities of our models, we utilize a diverse suite of vision-based benchmarks, including AI2D [104], MMStar [105], RealWorldQA [106], MathVista [41], and OCRBench [107].”

“To avoid overlap with the test set, for the 750 instances sampled from the evaluation dataset in [30], we remove any data that also appears in OCR-Bench [30] and exclude samples from TextVQA [40], DocVQA [35], InfoVQA [36], and ChartQA [34].”

“As noted in [59], such models tend to overfit to the context and language model.”

“OCR contains DocVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022), InfoVQA (Mathew et al., 2022), OCRBench (Liu et al., 2024b), and AI2D (Kembhavi et al., 2016).”

“• OCRBench [91] is a benchmark for OCR and text-centric visual understanding, consisting of 1,000 manually verified question-answer pairs across five tasks: text recognition, scene-text VQA, document-oriented VQA, key information extraction, and handwritten mathematical expression recognition.”

“Vision–language models (VLMs) [3, 7, 12] deliver strong results across tasks such as multimodal mathematics [26, 38], chart analysis [24, 27], and general VQA [50].”

“In contrast, combining direct answer and full format reduces tokens but lowers accuracy (1–5%), particularly on OCRBench, implying that the lack of perception-only format harms visual grounding.”

“Visual Reasoning Tasks: Evaluated on OCRBench [20], TextVQA [30], Vizwiz [12], ScienceQA [23] and MMMU [42].”

“Applied to historical documents, VLMs show robustness to noise and layout variability but also exhibit characteristic behaviors such as orthographic normalization, modernization of spelling, and hallucinated content in visually ambiguous regions [19, 7, 10].”

“For evaluation, we assess spatial reasoning with MMSI-Bench (Yang et al., 2025) and EmbSpatial (Du et al., 2024), multimodal medical QA with MMMU’s medical sub-set and PathVQA (He et al., 2020), and OCR performance with OCRBench (Liu et al., 2024b) and CharXiv (Wang et al., 2024).”

“…[76] focuses on university-level subject knowledge; MME [15] and MMBench [42] measure general-purpose visual understanding capabilities; OCRBench [43], MathVista [44], and HallusionBench [22] evaluate OCR competence, mathematical reasoning, and hallucination resistance, respectively; WISE [47]…”

“The model is examined off the reasoning framework using standard benchmarks including RefCOCO[11], OCR [14], and POPE [ ?”

“Recent vision–language models (VLMs) have a proven state-of-the-art performance across multiple Optical Character Recognition (OCR) tasks, from general-purpose benchmarks [3] to specialized consumer product applications [4,5].”

“We assess OmniBridge’s capabilities on a wide array of OCR-centric tasks: AI2D [71] (mask/no-mask settings), ChartQA [72] (test accuracy), CharXiv [73] (RQ/DQ tasks), TextVQA [74] (validation VQA accuracy), DocVQA [75], InfoVQA [76], OCRBench [77], and SEEDBench-2-Plus [78].”

“When increasing the maximum grid size to 8 × 8 and the token sequence length to (16+1) × 576 , the model exhibits improved performance on OCRBench [52] compared to the original setting.”

“The strongest recovery appears on MME-RealWorld (Chinese), where it improves Qwen2.”

“In the fi eld of image understanding, Liu et al. 17 introduced OCRBench, the fi rst benchmark dedicated to multimodal text recognition across diverse scenarios.”

“For some questions (especially religion), we provided a little textual context (e.g. the hint “the image depicts a religious scene”) because otherwise questions like “who is the person on the left of the image” are not answerable unambiguously.”

“As MLLMs continue to evolve, a series of large-scale benchmarks have been proposed to assess their capabilities across diverse scnarios and tasks [90, 56, 62, 61].”

“In contrast, most OCR-related benchmarks [16, 17, 18, 19, 20, 21, 22, 23, 24, 25] lack complexity, potentially overstating LMM progress in integrating reading, understanding, and reasoning.”

“We evaluate our models and the state-of-the-art lightweight VLMs on a myriad of multimodal benchmarks such as MMMU [84], MMBench [83], MMStar [7], MMVet [82], Math-Vista [46], AI2D [30], HallusionBench [24], OCRBench [42], MME [20], SEEDBench [34].”

“(b): Comparison with other MLLMs [34, 40, 54, 57] on 11 vision-language tasks [14, 22, 26, 29, 41, 42, 46, 48, 49, 55, 64] using Llama3-8B [44] as the LLM. LEO-M INI achieves competitive performance while using only 64 visual tokens performance in image and video understanding tasks [14, 22, 26,…”

“In particular, recent evaluations suggest that strong multimodal reasoning ability does not automatically translate into robust handwritten text recognition or accurate localization [67, 69].”

“OCRBench [37] RealWorldQA.”

“The compared systems differ in training data, post-training, layout modules, region-refinement strategies, and inference pipelines; therefore, this comparison VisionEncoder Params Overall DocVQA[74] InfoVQA[73] DF[96] KLC[94] WTQ[79] ChartQA[72] DT-VQA[134] OCRBench[66] FrozenvisionencoderspairedwiththesameQwen3-1. ca…”

“5-VL is evaluated on 8 benchmarks: TextVQA, ChartQA [45], AI2D [23], OCRBench [42], HallBench [17], MME, MMBench, and MMBench-CN.”

“[Liu et al., 2024c] ocrbench 1000 OCR, VQA ocrbench_accuracy MMMU [Yue et al., 2024] mmmu_val 900 college-level reasoning, VQA mmmu_acc RealWorldQA [xAI, 2024] realworldqa 765 real-world, VQA exact_match”

“The dominant benchmarks, e.g., OCR-Bench [35], OCRBench v2 [15], CC-OCR [58], and OmniDocBench [41], evaluate models on Latin, CJK, and a small number of other mid-resource scripts.”

“Active vision has been extensively studied in view planning (i.e., determining optimal sensor view-points) [70], object recognition [5], scene and 3D shape reconstruction [56], and robotic manipulation [8].”

“To evaluate text recognition and detail-sensitive reasoning, we include TextVQA [33] and the comprehensive OCRBench [28], which covers scene-text, document-text, and key information extraction subtasks.”

“…OCR, and accessibility: LISA (Lai et al., 2024), MMMU-Pro (Yue et al., 2024), AI2D (Kembhavi et al., 2016), MathVista (Lu et al., 2023), MathVision (Liu et al., 2024c), MathVerse (Zhang et al., 2024b), MMBench (Liu et al., 2024a), VizWiz (Gurari et al., 2018), and OCRBench v2 (Fu et al., 2024).”

“Our multi-faceted assessment is strategically divided into three critical capability domains: • General Multimodal Ability (5 benchmarks) : We assess core understanding and reasoning using established large-scale datasets, including MM-IFEval [6], MMMU [26], MMStar [2], AI2D [13], and OCRBench [17].”

“The models also show significant improvements in OCR tasks, with scores of 873 on OCRBench (Chen et al., 2024c) for the 70B variant.”

“OpenCompass has 8 evaluation datasets across diverse multimodal tasks, including AI2D [46], HallusionBench [37], MMMU [100], MMStar [12], MMVet [99], OCRBench [60], MMBench [59], and MathVista_MINI [63].”

“In the OCRBench [34] and MME [11], Mixpert achieves scores of 732 and 2143 , surpasses LLaVA-OV-7B [24] with 31 and 34 , respectively.”

“OCRBench [79] specifically targets the model’s capability to recognize and reason about text in the visual domain, covering a range of document layouts, fonts, and multilingual content with both exact-match and reasoning-based questions.”

“…limited impact on text with high contextual dependence, in cases such as random strings of letters or random sequences of numbers, where context does not contribute, the recognition capability of individual characters can be directly tested, and performance degradation may become pronounced [7, 8].”

“Although many various tasks such as VQA [2, 24, 47, 49, 51, 55, 79], Image Caption [33, 44], and OCR [18, 45, 74, 77] are care-fully designed, as shown in Fig.”

“RealWorldQA (xAI, 2024) evaluates spatial and physical understanding of everyday scenes, while SimpleVQA (Cheng et al., 2025b) et al., 2023) for hallucination, AI2D (Kembhavi et al., 2016) and CharXiv (Wang et al., 2024b) for diagram and chart understanding, OCRBench (Liu et al., 2023b) and CC-OCR (Yang et al., 2024)…”

“Five directly target the fine-grained visual grounding capabilities that CVPD is designed to improve: OCRBench [17] (localized text understanding), InfoVQA [20] (information extraction from dense infographics), and three MMStar [3] subscales: Fine-Grained Perception, Instance Reasoning, and Logical…”

“However, different MLLMs exhibit substantially different OCR capabilities [35].”

“This ability to exploit contextual information has enabled state-of-the-art performance on numerous OCR and document understanding benchmarks and has often improved cross-domain robustness [29,36].”

“Later work has continued in this direction with an emphasis on OCR and layout robustness; see OCRBench [14] and OCRBench v2 [9], OmniDocBench [23], Real5-OmniDocBench [36], RoDLA [4], and MMDocBench [39].”

“Best and second-best Rel. values within each budget are shown in bold and underlined, respectively. an identical selected subset under raw selection order and native raster-scan order on MME-OCR [4], OCRBench [15], and TextVQA [22].”

“We evaluate CLSE on nine diverse multimodal benchmarks spanning visual reasoning (GQA [31], SQA [48], VizWiz [25]), hallucination detection (POPE [38]), OCR (VQA TEXT [55], OCR Bench [47]), and holistic understanding (MME [23], MMB [46], MMB CN [46], MMVet [80]).”

“The multimodal rows draw from MMMU [40], MMMU-Pro [41], MathVista [42], DynaMath [43], BlindTest/VLMs-Are-Blind [44], MMBench [45], MMStar [46], RealWorldQA [47], SimpleVQA [48], CharXiv [49], OCRBench [50], CountBench [51], RefCOCO [52], ERQA [53], and Video-MME [54].”

“To verify general utility , ensuring our intervention preserves fundamental capabilities, we evaluate LVLM performance on OCRBench-v2 [10, 23], Real-WorldQA [40], MMStar [5] and MMMU [44].”

“Our models match their original VLM backbones and even improve on OCRBench [37] and POPE [32].”

“…(VQA) (Antol et al., 2015; Goyal et al., 2017; Hudson & Manning, 2019), to expert knowledge (Liu et al., 2024a; Yue et al., 2024; Lu et al., 2022), to document understanding (Mathew et al., 2021; Singh et al., 2019; Liu et al., 2024b), and video understanding (Fu et al., 2025a; Li et al., 2024).”

“Text recognition and understanding are tested on OCRBench [25] and TextVQA [38]; document and infographic comprehension on DocVQA [32] and In-foVQA [33]; chart reasoning on ChartQA [31]; and general visual perception on VizWiz-VQA [14].”

“The MLLM-CL Ability benchmark focuses on task-incremental learning and includes four tasks: OCR [60, 68], Math and Logic [84, 119], Visual Perception [1, 43], and GUI Agent [35, 102, 113].”

“Benchmarks such as OCRBench [10], CC-OCR [11], and Reasoning-OCR [12] systematically evaluated MLLMs across document parsing, key information extraction, multilingual recognition, and reasoning.”

“We evaluate our models on common VLM benchmarks on a variety of tasks: document (DocVQA [28]) and chart (ChartQA [27], InfoVQA [29]) understanding, text recognition (TextVQA [37], OCR-Bench [24]), and general QA (RealWorldQA [45], AI2D [16], GQA [15], MME [12]).”

“…for evaluating MLLMs on visual question answering (VQA) cover a wide range of domains, e.g ., mathematics [27, 42, 47], document under-1 standing [26, 31, 45, 50], video comprehension [11, 22, 46, 58], embodied tasks [10, 12, 18, 21, 34, 48], common sense knowledge [3, 13, 17, 30, 36, 37, 57],…”

“All evaluations use VLMEvalKit 3 (Duan et al., 2024) with English prompts matching our training format (e.g., “Return only the letter of the best answer option” for multiple-choice, “Respond very briefly” for open-ended questions).”

“Comprehensive evaluations like OCRBench [24] and Seed-Bench Plus 2 [20] assess models across key information extraction, text location, etc.”

“For text and chart understanding visual tasks, we include AI2D [10], ChartQA [21], OCRBench [17], and TextVQA [25].”

“OCRBench [14] measures text recognition ability on real-world visual-text inputs.”

“Additionally,we include OCRBench [25] for scene-centric text-VQA and for key information extraction from the receipt images.”

“To demonstrate the effectiveness of PPE, we construct our training datasets for supervised fine-tuning (SFT) referring to the public projects: LLaVA-Video-178k Zhang et al. (2024b) and LLaVA-OneVision Li et al. (2024a).”

“…and representative multimodal benchmarks, consisting of MMBench [Liu et al., 2024a], MMStar [Chen et al., 2024b], MMMU [Yue et al., 2024], MathVista [Lu et al., 2023], HallusionBench [Guan et al., 2024], AI2D [Kembhavi et al., 2016], OCRBench [Liu et al., 2024b] and MMVet [Yu et al., 2024].”

“5-7B [ 22] and LLaVA-Next-7B [23], and conduct experiments on widely used benchmarks [1, 10, 12, 18, 24, 25, 27, 32].”

“We expand the scope of evaluation with OCRBench [Liu et al., 2024b], which provides a multifaceted assessment that includes text recognition, scene text-centric VQA, document-oriented VQA, key information extraction, and handwritten mathematical expression recognition.”

“However, another cause could be poor quality of uploaded fi les for analysis such as PDFs (e.g., bad scans, formatting issues, OCR errors) (Liu et al. 2024; Piryani et al. 2025).”

“As shown in Table 5, we evaluate the overall performance of MindVL on MMBench [62], MME [10], OCRBench [63], DocVQA [72], ChartQA [70], InfoVQA [71].”

“…we tune the pretrained VLM on 3M samples from Infinity-MM [19] stage3 data with all parameters unfrozen, and use a series of publicly available VLM benchmarks for model evaluation, such as LLaVABench [30], MMVet [51], MMStar [10], DocVQA [34], OCRBench [31], TextVQA [40], DetailCaps-4870 [17], etc.”