OCRBench: on the hidden mystery of OCR in large multimodal models

“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”

“5 [64] Shanghai AI Lab 720 DeepSeek-VL-7B [81] DeepSeek 435 Claude3-Opus [82] Anthropic 694 OmniLMM-12B [83] OpenBMB 420 RekaFlash [74] Reka AI 692 TransCore-M [84] PCI Research 405 InternLM-XComposer2.”

“Among them, LLaMA [3] is an open-source LLM that matches the performance of GPT-3, followed by Alpaca [4], Vicuna [5], GPT-4-LLM [6] to improve the LLM’s alignment ability to follow human instruction, reporting impressive performance compared with proprietary LLMs.”

“We assess these LMMs on DocVQA[55], InfographicVQA [56] and ChartQA [57].”

“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”

“Other works, such as [20, 21], also focus on aligning the vision module and LLM for improved multimodal understanding.”

“…OCR text recognition datasets, including (1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly…”

“Recently, large-scale benchmarks such as Union14M-Benchmark [31] and OCRBench [32] have emerged to provide more standardized evaluation.”

“…GQA [10] (test-dev), MME [7], MMBench [22] (English dev, lite subset), POPE [17] (adversarial split), TextVQA [26] (validation, lite subset), ScienceQA-IMG [24], SEED-Bench [14] (lite subset), OCRBench [23], VizWiz-VQA [9] (validation, lite sub-set), and VQAv2 [8] (validation, lite subset).”

“• Document: OCRBench [52], InfoVQA [53], CharXiv descriptive and CharXiv reasoning [54], targeting OCR, text-rich images, and chart/document comprehension.”

“We evaluate DocVQA [20], Infograph-icVQA [21], TextVQA [22], OCRBench [23], ChartQA [24], and V*Bench [4].”

“Liu et al.[37] proposed OCRBench, which systematically evaluates the capabilities of large multimodal models in text recognition, document understanding, mathematical expressions, and multilingual OCR tasks.”

“It incorporates OCRBench [36], InfoVQA [37], and CharXiv [38].”

“The recent proliferation of Large Vision-Language Models (LVLMs) [24, 56, 70] has driven unprecedented advancements across a broad spectrum of multimodal tasks [23,27,28,34,37,69].”

“[39] includes text recognition and key information extraction, covering 29 sub-datasets.”

“It combines controlled MME perception questions [Fu et al., 2023] with one larger source for each capability: OCRBench for direct text recognition [Liu et al., 2023b], TallyQA for counting [Acharya et al., 2019], POPE for object existence [Li et al., 2023], VQAv2 color questions for attributes…”

“Relative to Mini-o3 [Lai et al., 2025], it performs better on OCRBench [Liu et al., 2024b], ChartQA [Masry et al., 2022], and CV-Bench [Tong et al., 2024].”

“T o k e n s p e r f r a m e / t il e TT F T ( m s ) T e x t V Q A [ 37 ] D o c V Q A [ 38 ] I n f o V Q A [ 39 ] O C R B e n c h [ 40 ] O C R B e n c h v 2 -”

“5 [17] families across four benchmarks, covering OCR [11], VQA [10], instruction following [6], and mathematical reasoning [13] tasks.”

“Notably, parameter count alone cannot distinguish between models with substantially different textual capabilities but similar parameter scales, leading to nearly identical VLM performance predictions for equally-sized backbones.”

“…and SEED-Bench (SEED) [16] for the General category; ScienceQA (SQA) [23], MMMU [48], MathVista [22], and AI2D [14] for Knowledge; ChartQA [24], OCRBench (OCRB) [21], TextVQA [33], and VizWiz [11] for OCR; and MMVP [36], RealWorldQA (RWQA) [44], Blink [6], V* [43], and CVBench [34] for Vision.”

“…(pt-PT) is an important language, with native speakers in Portugal and across the world, yet remains significantly underexplored in Optical character recognition (OCR) benchmarks [15,17,18,24,30], which are heavily skewed toward high-resource languages such as English (EN) and Chinese (ZH).”

“Benchmarks such as OCRBench and CC-OCR show that OCR remains a non-trivial capability for large multimodal models, especially for handwritten and fine-grained text-related tasks Liu et al (2024); Yang et al (2024).”

“OmniDocBench [18] provides diverse page types with fine-grained annotations and edit-distance–style metrics; OCRBench [10] aggregates OCR-centric tasks for multimodal models.”

“To provide a broader evaluation, the extended benchmark group includes MMStar[3], RealWorldQA (RWQA), AI2D[7], OCRBench[15], ChartQA[17], and MMBench-en[14].”

“The in-3051 2.37k 9.9 cluded benchmarks encompass MME [8], MMBench [36], SeedBench [14], MuirBench [32], Mirb [41], MMStar [6], OCRBench [20], SynthDog [13], ChartQA [23], AI2D [12], and ConBench [40], covering diverse domains, question types and metrics.”

“CNN/LSTM models often misinterpret cursive writing and medical abbreviations, while transformer models require domain-specific data and high computational resources [8].”

“Liu et al. [5] developed OCRBench, a benchmark revealing that large multimodal models still face challenges in text recognition, formula recognition, and key information extraction from document images.”

“These benchmarks span text recognition, visual question answering, key information extraction, and handwritten text scenarios, revealing persistent limitations of large models in handwritten, non-semantic, and fine-grained text recognition tasks when compared to specialized OCR systems [7].”

“On eight widely used image understanding benchmarks [7, 9, 14, 20, 21, 23, 24, 28], we evaluate M IP KV against four recent token-pruning baselines[3, 4, 34, 38] and Resize.”

“OCR-Bench [33] assesses the model’s ability to read and comprehend text in images under various scenarios.”

“Specifically, we adopt OCRBench [19], RealWorldQA [20], and MME-reasoning (MME-R) [21], respectively.”

“…evaluation landscape has grown in parallel, with benchmarks covering general VQA and multi-discipline reasoning [85, 86], STEM reasoning [87], OCR [88, 3], chart understanding [74], code generation from screenshots [89], video understanding [90, 91], GUI grounding and computer-use agents [92, 93,…”

“While effective on many benchmarks, this strategy incurs irreversible information loss that degrades performance on tasks where relevant content is distributed broadly across the scene [18], such as document understanding [21, 22] and visual grounding [23].”

“We conducted extensive evaluations across a comprehensive set of 14 benchmark datasets, includin MMMU [45], RealWorldQA [43], MMBench [22], MMStar [4], OK-VQA [24], GQA [13], ScienceQA [32], MMVP [39], OCRBench [21], TextVQA [34], AI2D [12], InfographicVQA [27], DocVQA [26] and POPE [19].”

“We evaluate the fine-grained perception capability on OCRBench (Liu et al., 2024c) for text recognition and AI2D (Kembhavi et al., 2016) for diagram interpretation.”

“These evaluation settings enable balanced assessment of TamOCR, consistent with recent OCR benchmarking practices [16], capturing low-level OCR accuracy, higher-level transliteration refinement, computational efficiency, and practical deployment feasibility under realistic degradation and resource…”

“Multimodal chain-of-thought (MCoT) prompting [28,31,44] offers a step-by-step reasoning trajectory when MLLM faces hard questions including detail grounding [30,55], agent planing [28], etc .”

“Fine-grained Image Benchmarks We leverage Open-compass [14] image benchmark for evaluation, including MMBench [50], MathVista [57], HallusionBench [25], OCRBench [52], AI2D [35], MMVet [108], MMStar [9], MMMU [109].”

“Application-specific benchmarks include OCRBench (Liu et al., 2024b), ScreenSpot-Pro (Li et al., 2025), MMSI-Bench (circular) (Yang et al., 2026), 3DSRBench (Ma et al., 2025), and LEGO (circular) (Tang et al., 2025).”

“For fine-grained visual understanding, we additionally evaluate on 4 text-oriented benchmarks: DocVQA [27], InfoVQA [28], ChartQA [26], and OCRBench [23].”

“While benchmarks focused on grounding exist, such as OCR tasks (Liu et al., 2024b) or hallucination detection (e.g., POPE (Li et al., 2023)), they largely test the retrieval of specific semantic entities (text or objects).”

“5-VL [27], which achieved exceptional performance across a range of multimodal tasks [28], [29], [30], [31], [32].”

“For fine-grained perception and OCR, OCRBench [13] exposes the systematic weakness of large multimodal models on text-rich images.”

“OCRBench [9,19] and CC-OCR [44] broaden the scope to multilingual OCR for VLMs.”

“Specifically, we include the following datasets: Re-alWorldQA [46], ScienceQA [31], GQA [15], MME [57], MMSTAR [6], MMBench [28], POPE [23], AI2D [18], ChartQA [32], TextVQA [41], InfoVQA [34], OCR-Bench [29], and DocVQA [33].”

“Representative examples include TextHawk (Yu et al., 2024), with token compression and multi-level cross-attention, and VLM-FO1 (Liu et al., 2025b), which strengthens regional OCR using auxiliary high-resolution encoders and explicit region tokens.”

“( 3) OCR & Chart: AI2D [57], OCRBench [58], DocVQA [59], ChartQA [60], InfoVQA [61], and TextVQA [62].”

“Text-centric benchmarks such as TextVQA [Singh et al., 2019], DocVQA [Mathew et al., 2021], and OCRBench [Liu et al., 2024d] evaluate whether models can read text correctly.”

“We evaluate DocCogito on six benchmarks, including DocVQA [24], WTQ [28], ChartQA [22], TextVQA [31], OCRBench [18], and InfoVQA [23].”

“…this progress, a substantial body of work has assessed and improved spatial reasoning in VLMs through dedicated benchmarks (Fu et al., 2024; Liu et al., 2024b; Yue et al., 2024; Lu et al., 2023; Kazemzadeh et al., 2014; Yang et al., 2025; Batra et al., 2025; Ogezi & Shi, 2025; Gan et al.,…”

“W.r.t. integrating multiple modalities more in general, this is a long-standing challenge in machine learning (Baltrušaitis et al., 2019; Huang et al., 2023; Li et al., 2024d).”

“Importantly, while some failure cases exist, successful results emerge not only for datasets whose (part of) their training sets were in the mixture, but also for different tasks such as OCR-Bench, MMStar, or MME.”

“However, this degradation is not uniform: Ostrakon-VL retains strong performance on tasks aligned with FSRS scenarios—such as Chinese-OCRBench (88.5, close to the best The results suggest that Ostrakon-VL successfully prioritizes task-relevant skills without catastrophic forgetting of foundational abilities, striking…”

“Document understanding benchmarks focus on MLLMs’ processing of structured/unstructured document images for OCR, chart reasoning, etc. Liu et al. [2] developed OCRBench, the most comprehensive OCR-focused benchmark covering 29 datasets for text recognition, document VQA, and KIE, identifying model…”

“Recent advances in multimodal foundation models (e.g., GPT, Gemini, and Qwen-VL) have shown the capacity to interpret images directly, performing zero-shot OCR and structured data extraction with minimal fine-tuning [11].”

“Language classification is frequently the first step in system building, as noted in [7], followed by method classification into character-based, word-based, and sequence-based approaches.”

“Non-subject Questions Subject-oriented Questions [13, 15, 16, 14].”

“…benchmarks: AI2D [11] (diagram understanding), MMMU [30] (massive multi-discipline multimodal understanding), MMStar [5] (complex multimodal reasoning), OCRBench [16] (OCR robustness), MM-Vet [29](comprehensive multimodal capability), and Mathvista [17] (mathematical reasoning in visual contexts).”

“…a diverse set of established benchmarks, including AI2D [12], MMBench [24], MMMU [41], MM-Vet [40], and MathVista [26] for general QA; OCR-Bench [25], ChartQA [27], and TextVQA [32] for OCR; OVOBench [28], MVBench [21], and VideoMME [11] for video understanding; Aishell-1 [5], Aishell-2…”

“For example, Deepak et emergence of benchmarks like OCRBench [11], OCRBench 2[12],andMMMU[13],whichevaluatelargelanguagemod-els (LLMs) and vision-language models (VLMs).”

“After extensive evaluation, we find that multi-modal based evaluators [20] [21] [22] [23] [24], although effective at capturing many visual aspects of images, tend to miss certain key properties such as depth [11], artifacts [14] [15] and character-level text-rendering in the image [12] [13] etc.”

“State-of-the-art LLMs exhibit poor performance in character recognition when there is little contextual information or the text is handwritten [14].”

“Finally, OCRBench [27] focuses on the model’s text recognition and understanding performance in real-world scenarios.”

“By using its dual understanding of visual features and textual meaning, the model delivered more accurate extractions across diverse invoice structures [11] [12].”

“Previous studies have shown that multimodal models perform much worse on non-semantic text and random character combinations than on ordinary words, because models are more likely to rely on semantic priors to recognize common words rather than read image evidence character by character [8], [21].”

“Known tasks for LVLMs also include traditional CV tasks such as OCR (Liu et al., 2023; Yang et al., 2024), which is also a topic of this paper, as well as more integrated tasks with reasoning steps such as VQA (Mishra, Shekhar, Singh, & Chakraborty, 2019).”

“…[66], assessing performance across 16 popular benchmarks, including GQA [67], DocVQA [68], InfoVQA [69], ScienceQA [70], TextVQA [71], VizWiz [72], OCRVQA [73], OCR-Bench [74], MME [75], MMMU [76], SEEDBench_IMG [77], MathVista_MINI [78], AI2D [79], HallusionBench [80], POPE [81], HRBench4K [82].”

“For OCR capabilities, the model is assessed using the OCRbench [27] and OCR-Reasoning [21] benchmarks.”