OCRBench: on the hidden mystery of OCR in large multimodal models

“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”

“5 [64] Shanghai AI Lab 720 DeepSeek-VL-7B [81] DeepSeek 435 Claude3-Opus [82] Anthropic 694 OmniLMM-12B [83] OpenBMB 420 RekaFlash [74] Reka AI 692 TransCore-M [84] PCI Research 405 InternLM-XComposer2.”

“Among them, LLaMA [3] is an open-source LLM that matches the performance of GPT-3, followed by Alpaca [4], Vicuna [5], GPT-4-LLM [6] to improve the LLM’s alignment ability to follow human instruction, reporting impressive performance compared with proprietary LLMs.”

“We assess these LMMs on DocVQA[55], InfographicVQA [56] and ChartQA [57].”

“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”

“Other works, such as [20, 21], also focus on aligning the vision module and LLM for improved multimodal understanding.”

“…OCR text recognition datasets, including (1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly…”

“Model Merging integrates models finetuned on different tasks within the parameter space to assimilate diverse capabilities (Yang et al., 2024b).”

“Unlike traditional OCR-related benchmarks [45, 26] or DocVQA [28], MMMU-Pro requires not only text recognition but also complex reasoning that integrates both visual and textual information, thereby pushing the capabilities of LMMs beyond standard document understanding.”

“Recently, multi-modal LLMs have been applied to the task of entity extraction from document images [5-8].”

“Vision-language tasks We report results on ALM-Bench (Vayani et al., 2024), a cultural understanding multilingual 4 visual QA benchmark, OCRBench (Liu et al., 2024b) and cc-OCR (Yang et al., 2024) for English and multilingual 5 OCR-centric capabilities respectively, and TextVQA (Singh et al., 2019), assessing scientif…”

“Existing OCR benchmarks [8, 3, 14] primarily focus on measuring average accuracy across standardized test sets, providing limited insight into the reliability of individual predictions.”

“Lastly, we provide a generalization bound and experimental validation to show that in practice, a small subset of samples from the dataset is enough to determine the viability of skipping, without access to ground truth labels.”

“The evaluation results align with a persistent and well-documented performance gap between closed-source frontier models and open-source VLMs [16].”

“Similarly, prior progressive pruning methods (Ye et al., 2024; Xing et al., 2024; Zhang et al., 2024c) also leverage network structures (layer positions) for finer granularity.”

“…Answering, such as MMBench [45], SEED-Bench [38], and MMMU [83]; (2) Optical Character Recognition (OCR), such as OCR-VQA [56] and OCRBench [46]; (3) Graphic Understanding, including AI2D [33], CharXiv [80], and MultiChartQA [88]; and (4) Mathematics, such as MathVista [48], MathVision…”

“…we select a set of OCR-extensive benchmarks (DocVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022), InfoVQA (Mathew et al., 2022), OCRBench (Liu et al., 2024c), TextVQA (Singh et al., 2019)) that require fine-grained visual information, thus being highly sensitive to potential information…”

“On OCRBench, it shows strong OCR capabilities, outperforming many similar-sized competitors, and on AI2D, it scores 84.3, surpassing all rivals.”

“For text understanding, we evaluate on OCRBench [41], a specialized benchmark for scene text recognition, document text recognition, and key information extraction.”

“Recent benchmarking efforts have begun evaluating large-scale OCR and multimodal models on complex text recognition tasks [20].”

“Specifically, we report results on vision-centric benchmarks such as RealWorldQA [49] and V-Star [48], as well as OCR-and chart-focused datasets including TextVQA [36], ChartQA [29], and OCRBench [25].”

“(Baird, 2007; Guyon, Haralick, Hull, & Phillips, 2000) Leveraging LVLMs for more specialized and integrated OCR tasks and integrating further interference steps as in VQA have been explored in multiple benchmarks such as OCRBench (Liu et al., 2023) and DocVQA (Mathew, Karatzas, & Jawahar, 2020).”

“…and language modalities, and has made remarkable progress, as indicated in the recent research literature, such as document layout analysis [1], assessment of Large Multimodal Models [2], especially in English script-specific scene text recognition (STR) [3], [4], [5], [6], [7], [8], [9], [10].”

“Meanwhile, benchmarks targeting a single perceptual application, such as OCRBench (Yuliang Liu et al. 2024) and ScreenSpot (K. Li et al. 2025), provide more direct measures of perception but are built top-down: their categories are predefined and populated with questions from a specific application domain.”

“We also compare VisionReasoner’s VQA (Masry et al., 2022; Mathew et al., 2021; Liu et al., 2024b; Yue et al., 2024; xAI, 2024) ability with Qwen2VL (Wang et al., 2024) and the baseline model Qwen2.”

“Specifically, the image benchmarks include MMMU (Yue et al., 2024), MMBench (Liu et al., 2024a), MMStar (Chen et al., 2024), OCRBench (Liu et al., 2024b), AI2D (Kembhavi et al., 2016), RealWorldQA (X.AI., 2024), DocVQA (Mathew et al., 2021), TextVQA Singh et al. (2019), InfoVQA (Mathew et al., 2022), and ChartQA (Masr…”

“These capabilities highlight its potential as a foundation model for document understanding in OCRBench [18; 8].”

“The OCR ability of general-purpose VLMs has been bench-marked across scripts and tasks [9], mostly on Latin-script and high-resource text.”

“Some of the datasets we use for evaluation have overlap with the OCRBench dataset [11, 21], but we modify input and output tokens to focus more on text transcription compared to general text VQA and OCR bounding box detection.”

“[26] 859 836 OCRBench v2 (EN, M) [12] 63.92 62.91 MathVista (M) [27] 80.50 79.60 MathVision (M) [47] 64.14 67.10 MMMU-Pro [52] 70.23 69.88 CharXiv (reasoning) [50] 46.80 48.90 ChartQAPro [30] 63.21 63.68 HallusionBench [16] 64.90 62.29”

“For text recognition and text-centric understanding, we use OCRBench (Liu et al., 2024), OCRBench V2 (Fu et al., 2026), and TextVQA (Singh et al., 2019); for document and infographic understanding, we use DocVQA (Mathew et al., 2021), InfoVQA (Mathew et al., 2022), and OmniDocBench (Ouyang et al., 2025), which evaluat…”

“At the same time, OCR benchmarks have shown that multimodal language models still face difficulties with text recognition, particularly for multi-lingual and visually challenging documents [17], while recent work demonstrates that OCR errors can significantly affect downstream question answering [21].”

“OCRBench [29].”

“Liu et al. [13] evaluated multimodal models using OCRBench across text-related image tasks and reported useful capabilities alongside continuing weaknesses in multilingual and other challenging text settings.”

“Evaluation uses LMMS-Eval (Li et al., 2024a; Zhang et al., 2024).”

“This design follows the controlled evaluation approach used in recent OCR benchmarks [11, 15, 19].”

“OCRBench (Liu et al., 2024) systematically evaluates text-centric visual capabilities, while MMLongBench-Doc (Ma et al., 2024) extends document reasoning to long-context settings in which relevant evidence may be distributed across multiple pages and heterogeneous document modalities.”

“(1) General Question Answering: GQA [17], MME [12], RealWorldQA (RWQA) [37]; (2) Comprehensive Knowledge Reasoning: ScienceQA-IMG (SQA) [29], AI2D [18], MMMU [39], MMStar [5]; (3) OCR and Chart Parsing: ChartQA [30], DocVQA [31], TextVQA [33], and OCRBench [28].”

“Table 2 separately reports ChartQA (Masry et al., 2022), OCRBench (Liu et al., 2024b), DocVQA (Mathew et al., 2021), InfoVQA (Mathew et al., 2022), POPE (Li et al., 2023b), HallusionBench (Guan et al., 2024), and CountBenchQA (Beyer et al., 2024).”

“We also include 4 text-oriented benchmarks, namely TextVQA (Singh et al., 2019), ChartQA (Masry et al., 2022), AI2D (Kembhavi et al., 2016), and OCRBench (Liu et al., 2024c), to assess OCR-related and document-level reasoning abilities across various visual scenarios.”

“Prior work improves pixel-text understanding through reading-oriented data and architectures (Zhang et al., 2024), benchmarks text-rich comprehension (Li et al., 2024), and studies the symbolic-to-pixel modality gap (Sun et al., 2026; Fu et al., 2025).”

“…image understanding (Masry et al., 2022; Mathew et al., 2021, 2022; Singh et al., 2019), general perception and diagram understanding (Kembhavi et al., 2016; Liu et al., 2024a; Li et al., 2024), text recognition (Liu et al., 2024c), and visual reasoning (Chen et al., 2024; Lu et al., 2024b).”

“We evaluate on GQA, MMBench-en/cn [16], MME [7], POPE [12], MMStar [5], OCRBench [17], and ChartQA using official metrics.”

“We further evaluate DIPrune on detailed captioning (Detail-Caps (Dong et al., 2024)), open-ended hallucination (CHAIR (Rohrbach et al., 2018)), multi-turn reasoning (ConvBench (Liu et al., 2024b)), and fine-grained perception (OCRBench (Liu et al., 2024d), ChartQA (Masry et al., 2022)).”

“Additional H1 evaluations cover TextVQA, MMMU, OCRBench, and POPE (Singh et al., 2019; Yue et al., 2024; Liu et al., 2023; Li et al., 2023).”

“OCRBench, a 29-dataset evaluation of large multimodal models, similarly reports that these models perform poorly on non-semantic character strings [37].”

“5-4B, it improves ChartQA and EvoChart by 7.20 and 10.11 points, respectively, while also improving CountQA and OCRBench by 2.53 and 1.93 points.”

“Recent MLLMs have demonstrated strong capabilities in document understanding, optical character recognition, and chart reasoning Achiam et al. (2023); Wang et al. (2024); Liu et al. (2023, 2026).”

“Perception extracts structured signals from non-text media and aligns them with language Baechler et al. (2024); Liu et al. (2024e); Radford et al. (2021).”

“Vision–language performance is evaluated on MMMU, OCRBench, VizWiz, ScienceQA, and TextVQA using task accuracy (Yue et al., 2024; Liu et al., 2024; Gurari et al., 2018; Lu et al., 2022; Singh et al., 2019).”

“Comprehensive OCR evaluations, including OCRBench (Liu et al., 2024) and OCRBench v2 (Fu et al., 2025), reveal substantial performance variations among different OCR capability dimensions, indicating that a unified multimodal interface alone is insufficient to establish balanced OCR intelligence.”

“Spatial reasoning is evaluated on MMSI-Bench (Yang et al., 2025) and EmbSpatial (Du et al., 2024), optical character recognition on OCRBench (Liu et al., 2024b), and medical multimodal reasoning on MMMU-Med (Yue et al., 2024) and PathVQA (He et al., 2020).”

“We then examine visual feature responses in ConvCue and use a case study to compare how answer preferences change across language-model layers in the original model, the matched-training baseline, and ConvCue.”

“OCRBench (Liu et al., 2023b) et al., 2023).”

“OCRBench [40] broadens the assessment with both recognition and question-answering tasks.”

“…POPE, MMMU, OCRBench, and ChartQA (Liu et al., 2024b; Yu et al., 2024; Singh et al., 2019; Mathew et al., 2020; Li et al., 2023b; Yue et al., 2024; Liu et al., 2024c; Masry et al., 2022), covering multimodal understanding, text recognition, documents, object hallucination, multidisciplinary…”

“Document and chart understanding includes ChartQA (Masry et al., 2022), the descriptive (DQ) and reasoning (RQ) tracks of CharXiv (Wang et al., 2024), DocVQA (Mathew et al., 2021), OCRBench (Liu et al., 2024b), and the English-task average of OCRBench v2 (Fu et al., 2026).”

“Table 4 covers MathVision (Wang et al., 2024a), MathVista (Lu et al., 2024), MMMU (Yue et al., 2024a), MMMU-Pro (Yue et al., 2024b), RealWorldQA (xAI, 2024), MM-Star (Chen et al., 2024b), MMBench (Liu et al., 2024), HallusionBench (Guan et al., 2024), AI2D (Kembhavi et al., 2016), CC-OCR (Yang et al., 2025), OCRBench…”

“We evaluate on 17 public benchmarks spanning four capability areas: document understanding and OCR (OCRBench [28], DocVQA [33], ChartQA [32], InfoVQA [34], TextVQA [42]), visual perception (MME [11], SEEDBench [21], MMBench [27], RealWorldQA [52]), reasoning and knowledge (Sci-enceQA [29], AI2D…”

“We evaluate quantization quality on a broad suite of vision-language benchmarks through the LMMs-Eval framework [42]: OCRBench [43] and TextVQA [44] for text recognition and understanding.”

“…LLMs. End-to-end task accuracy is assessed across six multi-modal benchmarks chosen to stress different activation topologies: logical reasoning (MMMU [3]); text-rich visual comprehension (OCRBench [40], TextVQA [41], ChartQA [42]); and general perception (VizWiz [43], SEED-Bench-2+ [44]).”

“…evaluate EmbodiedMind on benchmarks in four categories: general multimodal understanding (AI2D [35], MMSTAR [36], MMMU [37], and OCR-Bench [38]), 2D spatial reasoning (CV-Bench [39], RoboSpa-tial [40], RefSpatial [28], EMbSpatial [23], ERQA [41], and CrossPoint [26]), 3D spatial…”

“5-VL-7B ( Bai et al. 2025) on GQA, MMB, MME, POPE, SQA, VQA Text , and OCRBench (Liu et al. 2024d).”

“Recent benchmarks such as OCR-Bench, OCRBench v2, and MMDocBench further probe OCR, localization, fine-grained perception, and reasoning over complex document layouts [8, 24, 55].”

“VTC-Bench [19] constructs its pruning-sensitive Group A evaluation set from eight established benchmarks: GQA [11], MMBench and MMBench-CN [22], MME [7], POPE [17], MMStar [5], OCRBench [21], and ChartQA [24].”

“• OCRBench [28]: The benchmark includes 1000 question-answer pairs evaluating the comprehensive ability of MLLMs, including Text Recognition, SceneText-Centric VQA, Document-Oriented VQA, Key Information Extraction, and Handwritten Mathematical Expression Recognition.”

“Taken together, robust gauge-reading therefore requires not only general visual understanding or OCR capabilities but also precise visual grounding, contextual reasoning, uncertainty awareness, and resistance to misleading textual and visual cues.”

“5-[27] 46.1 ‡ + S 2 [41] 52.6 ‡ + ViCrop [56] 57.2 ‡ + SD-RPN [42] 58.8 ‡ + Q-Zoom [43] 58. trained model on OCRBench [28] without OCRBench-specific training.”

“We evaluate on public vision–language benchmarks including OCRBench (Liu et al., 2024b), Math-Vista (Lu et al., 2024), MM-Vet (Yu et al., 2024), HallusionBench (Guan et al., 2024), MM-Star (Chen et al., 2024), MMMU (Yue et al., 2024), AI2D (Kembhavi et al., 2016), and MM-Bench (Liu et al., 2024a).”

“(3) General multi-modal understanding is evaluated across a suite of standard benchmarks: MMBench [24] (MMB_CN, MMB_EN), MMStar [12], and OCR-Bench [25].”

“T We evaluate performance via the LMMs-Eval [Zhang et al. , 2024] framework on diverse benchmarks, including OCRBench (text recognition) [Liu et al. , 2024c], MMMU val (multi-discipline reasoning) [Yue et al. , 2024], RealWorldQA (real-world QA) [ Fu et al. , 2024 ] , and AI2D (diagram reasoning) […”

“B ENCH ), consistency of a spatial model across limited views (M IND C UBE ), spatial reasoning over video (VSI-B ENCH ), and multi-image perceptual primitives (BLINK).”

“…capabilities of large multimodal models across text recognition, scene-text VQA, document-oriented VQA, key information extraction, and handwritten mathematical expression recognition, showing that visual text processing is an important but challenging capability of MLLMs (Liu et al. 2024).”

“21 × and 2.50, with OCRBench becoming slower than autoregressive decoding at 0 .”

“…the effectiveness of our method, we pre-train our model on the BLIP_LATION_CC_SBU_558k dataset and fine-tune it on COCO [39], GQA [40], OCR-VQA [41], TextVQA [42], Visual Genome [43] and LLaVA Visual Instruct 150K. • BLIP_LATION_CC_SBU_558k :consistingofacurated mixture of CC-3M, LAION, SBU…”

“OCRBench excludes digit-string recognition, non-semantic text recognition, handwritten mathematical expressions, and subsets already represented by TextVQA, ChartQA, DocVQA, or InfographicVQA; its visual-document contribution is key-information extraction.”

“Image/video understanding ChartQA [50]; DocVQA [51]; InfoVQA [52]; OCRBench [45]; TextVQA [58]; MMBench [44]; MMStar [9]; MME [25]; LongVideoBench [63]; LVBench [61]; MotionBench [32]; VideoMME and VideoMME-Short [24].”