OCRBench: on the hidden mystery of OCR in large multimodal models

“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”

“5 [64] Shanghai AI Lab 720 DeepSeek-VL-7B [81] DeepSeek 435 Claude3-Opus [82] Anthropic 694 OmniLMM-12B [83] OpenBMB 420 RekaFlash [74] Reka AI 692 TransCore-M [84] PCI Research 405 InternLM-XComposer2.”

“Among them, LLaMA [3] is an open-source LLM that matches the performance of GPT-3, followed by Alpaca [4], Vicuna [5], GPT-4-LLM [6] to improve the LLM’s alignment ability to follow human instruction, reporting impressive performance compared with proprietary LLMs.”

“We assess these LMMs on DocVQA[55], InfographicVQA [56] and ChartQA [57].”

“The success of large models has also been extended to the multimodal vision-language space [7], leading to a line of research on large multimodal models (LMM), including contrastive learning [8, 9, 10, 11] and generative modeling [2, 12, 13, 14, 15].”

“Other works, such as [20, 21], also focus on aligning the vision module and LLM for improved multimodal understanding.”

“…OCR text recognition datasets, including (1) Regular Text Recognition: IIIT5K [37], SVT [38], IC13 [39]; (2)Irregular Text Recognition: IC15 [40], SVTP [41], CT80 [42], COCOText(COCO) [43], SCUT-CTW1500 (CTW) [44], Total-Text (TT) [45]; (3) Occlusion Scene Text [46], encompassing weakly…”

“While this assumption has been adopted by many subsequent studies Zhang et al. (2024b); Zhao et al. (2024); Xing et al. (2024); Tan et al. (2025); Huang et al. (2024); Wen et al. (2025), we believe it remains open to question: Are all visual tokens in the shallow layers of LLM truly indispensable…”

“We formalize this approach as Layer-wise Adaptive Freeze-Tuning (LW-AFT), which identifies and updates only critical parameters through layer-wise sensitivity analysis.”

“For LLaVA-Video, we use a 10k subset of LLaVA-Video-178k (Zhang et al., 2024c) to maintain its capability in the video domain.”

“Recent studies have highlighted the superior performance of Large Multimodal Models (LMM) in OCR tasks, with benchmarks emphasizing their OCR capabilities [96], [118], [119].”

“Recent benchmark suites such as OCRBench-v1 [16], OCRBench-v2 [6], and OmniDocBench [25] broaden coverage by including tasks like text localization, recognition, layout analysis, and table extraction, evaluated with character accuracy, edit distance, and mean average precision.”

“…MMMU (val) [76], CharXiv (RQ) [67], Math-Vision [61] and MathVista (mini) [36]; (vision knowledge) SimpleVQA [13] and WorldVQA 2 ; (perception) ZeroBench (w/ and w/o tools) [48], BabyVision [12], BLINK [18] and MMVP [57]; (OCR & document) OCR-Bench [35], OmniDocBench 1.5 [42] and InfoVQA [38].”

“…demonstrate impressive capabilities, showing strong performance on diverse multimodal tasks [14, 17] including (i) perception & grounding [12, 19, 27], (ii) knowledge-intensive multimodal reasoning [13, 15, 26], and (iii) text-centric understanding that requires reading in-image text [25, 28, 33].”

“Recent studies have evaluated AI models from psychological perspectives, including behavioral analysis (Coda-Forno et al., 2024), personality (Huang et al., 2024b;a), emotion (Huang et al., 2024b), and mental disorder (Coda-Forno et al., 2023).”

“…evaluation scores for all 23 models on OpenVLM, we end up with AI2D [Kembhavi et al., 2016], HallusionBench [Guan et al., 2024], MMBench [Liu et al., 2025], MMStar [Chen et al., 2024b], MMMU [Yue et al., 2024], MathVista [Lu et al., 2023], MM-Vet [Yu et al., 2023], OCRBench [Liu et al., 2024b].”

“…et al., 2023), MathVista (Lu et al., 2024b), and HallusionBench (Guan et al., 2024), as well as five document understanding-related benchmarks: TextVQA (Singh et al., 2019), DocVQA (Tito et al., 2021), AI2D (Kembhavi et al., 2016), InfoVQA (Mathew et al., 2021a), and OCRBench (Liu et al., 2024c).”

“At Qwen-256, it leads five, while MMTok-style and Top-K lead OCRBench and MMBench-EN, respectively.”

“Capability benchmarks complement identity tests by evaluating task behavior, including OCR [28], multimodal reasoning [53], software engineering [26], mathematics [31], and broad knowledge [49, 50].”

“MathVista [21], ChartQA [22], MMMU [23], OCRBench [24], DocVQA [25], AI2D [26], and Real-WorldQA [27] are reformulated under an agentic setting requiring multi-step tool calls.”

“Table 10 reports five full-dataset VL point estimates in no-thinking mode: the routed Tall service is higher than the native base on OCRBench (Liu et al., 2023b), MMBench-EN (Liu et al., 2023a), MMMU (Yue et al., 2023), and MME (Fu et al., 2023) cognition, and lower by 52.99 points on MME…”

“• OCR text recognition : OCRBench v1 [77] and OCRBench v2 [78] (reported separately for Chinese and English), evaluating general text recognition and document parsing; • Referring grounding : RefCOCO [79], measuring pixel-level element referring and grounding accuracy in general scenarios; • GUI…”

“For instance, document images can be converted into markdown representations through OCR and layout analysis [7,10,19,32,39,63], which is a common practice in retrieval-augmented generation (RAG) pipelines.”

“In addition, several comprehensive OCR benchmarks have been introduced, such as OCRBench [30], OCRBench v2 [11], and SEED-Bench-2-Plus [23], which integrate multiple OCR-related tasks to provide a holistic evaluation of MLLMs’ text understanding capabilities [1,19].”

“Most MLLM benchmarks frame evaluation as per-question correctness: MMMU (Yue et al., 2024) and MMMU-Pro (Yue et al., 2025) measure college-level multimodal reasoning, MMBench (Liu et al., 2024a) spans 20 ability dimensions, and OCRBench (Liu et al., 2024b) targets text recognition accuracy.”

“…MMBench [24]), expert-level knowledge and reasoning (MMMU [25], GQA [26], ScienceQA [27]), object hallucination sensitivity (POPE [28]), and specialized tasks such as structured diagram understanding (AI2D [29]), in-the-wild VQA (VizWiz [30]), and fine-grained text recognition (OCRBench [31]).”

“• Benchmarks : We evaluate on a diverse set of vision-language benchmarks encompassing general visual question answering (VQAv2 [13], GQA [14]), OCR and document understanding (TextVQA [28], OCRBench [22]).”

“We report nine metrics across benchmark suites: MMBench-En/Cn [24], MME-P/C [9], AI2D [14], SQA-IMG [26], POPE-A/P [20], and OCRBench [25].”

“MMBench = MM-Bench_DEV_EN_V11 [30], MMStar [5], OCRBench [31] (scored out of 1000), TQA = TruthfulQA (MC2) [26], and MMLU-R = MMLU-Redux [17].”

“We further evaluate our method on VQA datasets [23, 37, 43, 48, 54, 94] to investigate how object-level Reinforcement learning influences image-level understanding.”

“These models now support a broad range of vision-language tasks, including visual question answering [14, 19, 30], optical character recognition [24, 13, 41], document and chart understanding, spatial reasoning, and multimodal mathematical reasoning [25, 34, 43].”

“We evaluate on eight benchmarks covering three capability dimensions: (i) general VQA : MMBench EN [26], MMBench CN [26], MMStar [7]; (ii) knowledge & reasoning : MMMU [50], MathVista [29], AI2D [19], HallusionBench [13]; (iii) fine-grained perception : OCRBench [27].”

“Following mainstream MLLMs [8, 33], we focus on a wide range of classic benchmarks, including ChartQA [50], OCRBench [46], InfoVQA [52], AI2D [30], TextVQA [60], DocVQA [51] and SEED-Bench-2-Plus [34].”

“For vision-centric perception, we assess document understanding (Mathew et al., 2021; 2022; Liu et al., 2024b) and visual grounding (Kazemzadeh et al., 2014; Yu et al., 2016).”

“Here we extend the evaluation to two additional benchmarks covering complementary capability domains: OCRBench [24] (text recognition) and MathVista [26] (mathematical reasoning).”

“Most datasets [5, 12, 30, 42] center on English or other high-resource languages, offering limited evidence of generalization to low-resource scripts.”

“We evaluate the model’s multimodal understanding capabilities on multiple benchmarks, including general VQA benchmarks such as GQA Hudson & Manning (2019), RealWorldQA xAI (2024), and SEED-Bench Li et al. (2023); knowledge-intensive benchmarks such as MMMU (Yue et al., 2024); and text-centric benchmarks including Char…”

“Notable examples of such benchmarks include image cap-tioning [21], visual question answering [16], [17], and optical character recognition (OCR) [32].”

“Each image contains Social Security numbers, addresses, and income data that VLMs can extract through OCR (Liu et al., 2024).”

“The default leaderboard reports the average score across eight diverse VLM benchmarks: MMBench (Liu et al., 2024b), MMStar (Chen et al., 2024b), MMMU (Val) (Yue et al., 2024), MathVista (Lu et al., 2024), OCRBench (Liu et al., 2024c), AI2D (Kembhavi et al., 2016), HallusionBench (Guan et al., 2024), and MMVet (Yu et a…”

“Additional benchmarks/ablations are reported in Appendix A.4.”

“To comprehensively evaluate the performance of our proposed TLQ across diverse vision–language understanding scenarios, we adopt multiple widely used multimodal benchmarks, including MMMU-Val (Yue et al., 2024), OCRBench (Liu et al., 2024), VizWiz-VQA-Val (Gurari et al., 2018), TextVQA-Val (Singh et al., 2019), ChartQ…”

“Specifically, LMMs excel in high-level perception and reasoning tasks, such as image captioning, visual question answering, and cross-modality grounding, as validated by benchmarks like OCRBench [1].”

“We also notice that DiffEIC improves on POPE, OCRBench and MMB but exhibits a very slight drop on SEEDBench, MME and GQA.”

“For document understanding and question answering, we evaluate on AI2D (Kembhavi et al., 2016), OCRBench (Liu et al., 2024f), DocVQA (Mathew et al., 2021), InfoVQA (Mathew et al., 2022), and ChartQA (Masry et al., 2022).”

“OCRBench [Liu et al., 2024c] A general OCR benchmark covering natural scenes, scanned documents, and receipts, focusing on robustness to complex fonts, multilingual text, and noisy environments.”

“…answering, MMMU [125] for massive multi-discipline understanding, MMStar (MM ∗ ) [20] for challenging multi-modal reasoning, OCR-Bench (OCRB) [71] for text recognition capabilities, TextVQA (TVQA) [98] for reading text in images, ChartQA (CQA) [78] for chart understanding, AI2D [53] for…”

“We assess OCR ability using OCRBench [39] and TextVQA [48].”

“…[51, 53]; zero-shot out-of-domain tasks and benchmarks include Lisa-grounding [54], counting (Pixmo-Count [55]), mathematical diagrams perception (MATHGLANCE [56]), and general ability benchmarks of MLLMs from VLMEvalKit [57] such as MMBench [58], MMVet [59], SEEDBench [60], OCRBench [61], etc.”

“We also include the average regression percentages of these fine-tuned models across four benchmarks: MMVP [56], OCRBench [39], MME [13] and MMMU [69] (each representing one task category [54]).”

“We adopt evaluation protocols consistent with PLM and those followed by LMMS-Eval (Zhang et al., 2024; Li* et al., 2024).”

“The advent of LLMs has opened the way to their use in OCR tasks [23].”

“6) and report the average score of eight evaluation datasets (i.e., MMBench [32], MMStar [7], MMMU [70], Math-Vista [33], AI2D [22], HallusionBench [17], MMVet [69], OCRBench [31]) based on VLMEvalKit [13].”

“Second, when evaluated on the general benchmarks including BFCL-v3 Patil et al. (2025), MME-perception Fu et al. (2023), MME-reasoning Yuan et al. (2025), and OCRBench Liu et al. (2024c), DaMo outperforms DML by 2.57% in terms of average score.”

“For OCR capabilities, the model is assessed using the OCRbench [27] and OCR-Reasoning [21] benchmarks.”

“Si-multaneously Si-multaneously, the number of benchmarks for evaluating such models has grown significantly [7], [8], [9], [10], [11], [12], [13], [14], most of which require a combination of perception, reasoning, and advanced world knowledge to be solved.”

“According to multiple visual-task specific benchmarks [84], [85], [86], the model excels in understanding visual elements at a comparable level to GPT-4o-mini [87].”

“• Text-rich: TextVQA [48], ChartQA [49], DocVQA [37], InfoVQA [50], OCRBench [38].”

“With the rapid MLLMs, multimodal benchmark has evolved from early visual perception tasks to specialized domains, including OCR [61] Answering [62], Document Visual Question Answering [63], Agent Benchmark [64], Tool Vision Benchmark [65], and first-person perspective perception tasks [66, 67].”

“OCR and Text-Rich This group evaluates abilities on OCR and understanding text-rich images, including OCRBench (Liu et al., 2024d) and SeedBench-Plus (Li et al., 2024c) for general OCR and text-rich tasks, ChartQA (Masry et al., 2022) for chart perception and reasoning, DocVQA (Mathew et al., 2021)…”

“While research has advanced each individual component independently [27–31], the integration of these capabilities must simultaneously interpret visual symbolic representations and extract spatial relationships to enable coherent path finding.”

“The benchmarks are VQA v2 [11], GQA [16], MMBench [27], OK-VQA [32], ScienceQA-IMG [30], AI2D [19], TextVQA [39], OCRBench [28], DocVQA [34], ChartQA [33], POPE [21], RefCOCO, RefCOCO+ [54] and RefCOCOg [31].”

“A page can reach the reader as a pixel array, as in OCR-free models [8, 9], or as a symbolic transcript, as in OCR-augmented ones [10], and a VLM may use evidence in the second form that it fails to extract from the first [11, 12].”

“We assess capability preservation with general multimodal benchmarks (Yue et al., 2024; Liu et al., 2024a; Mathew et al., 2021; 2022; Fu et al., 4).”

“…(2) general VQA benchmarks, GQA (Hudson & Manning, 2019), MMMU (Yue et al., 2024), MMB (en) (Liu et al., 2024a) and MME (Fu et al., 2025) (3) visual hallucination benchmarks, POPE (Li et al., 2023); (4) OCR benchmarks, TextVQA (TVQA) (Singh et al., 2019) and OCRBench (OCRB) (Liu et al., 2024b).”

“5, we use AI2D [32], ChartQA [33], TextVQA, OCRBench [34], MME, and MMBench-EN/CN. Video evaluation uses MVBench [35], LongVideoBench [36], and Video-MME [37].”

“We follow the vision-language setup of Yang, Shi, and Liu (2026) and evaluate on MMSI-Bench (Yang et al. 2025), EmbSpatial (Du et al. 2024), MMMU-Med (Yue et al. 2024), PathVQA (He et al. 2020), OCRBench (Liu et al. 2024b), and CharXiv (Wang et al. 2024).”

“We evaluate unrelated utility on the ten benchmarks listed in Table 3 using their official protocols (Fu et al. 2023; Liu et al. 2024a; Li et al. 2024a; Liu et al. 2024b; Lu et al. 2022; Qian et al. 2025; Lu et al. 2024; Wang et al. 2024; Li et al. 2023; Guan et al. 2024).”

“…(AI, 2024); (3) visual perception on MMStar (Chen et al., 2024b), MMVP (Tong et al., 2024b), CV-Bench (2D and 3D) (Tong et al., 2024a), and OCRBench (Liu et al., 2024c); and (4) hallucination robustness on POPE (Li et al., 2023c), HallusionBench (Guan et al., 2023), and AMBER (Wang et al., 2023).”

“A wide range of benchmarks has been proposed to evaluate the growing capabilities of MLLMs, spanning vision question answering (VQA) [30, 31], image captioning [32, 33], and OCR/text understanding [34].”

“Comprehensive evaluations in benchmarks like OCRBench [45] -v2[14] and TextSquare [62] reveal that standard MLLMs struggle significantly with dense text.”

“These include MMMU [37], RealWorldQA [35], MMStar [4], GQA [13], SEED-Bench [16], POPE [18], ScienceQA [22], MME [9], AI2D [14], OCRBench [21], CountBench [2], DocVQA [24], ChartQA [23], TextVQA [28], InfoVQA [3], BLINK [11], MUIRBENCH [32], ERQA [8], EmbSpatialBench [7], MVBench [17], VideoMME…”

“We evaluate on a wide set of benchmarks across several categories including General VQA [5,11,19,21,55,59,60], OCR and Document Understanding [26,32, 33,43]; Chart and Diagram Understanding [17,31]; Spatial Understanding [9,16]; Captioning [16, 24] and Mathematics [53].”

“Moreover, MMVet [59] and OCRBench [33] exhibit a surprisingly low correlation ( r ≈ 0 .”

“We follow MBQ [26] and report performance on six widely used multimodal benchmarks via the LMMs-Eval suite: OCRBench [31], TextVQA [38], VizWiz [18], SEED-Bench [22], ScienceQA-IMG [32], and MMMU [50].”

“Archives, libraries and cultural heritage institutions hold vast quantities of documents that look nothing like those from IAM [15] or OCRBench [14].”

“• OCRBench [25]: An OOD comprehensive OCR evaluation suite covering text recognition, key information extraction, scene text QA, document and chart QA, and formula recognition.”

“Model performance is evaluated with the LMMs-Eval benchmark suite (Zhang et al., 2024b) across a wide range of vision–language tasks, including: (1) Text recognition and understanding: OCRBench (Liu et al., 2024a) (scene text recognition), TextVQA(Singh et al., 2019) (text-centric visual question answering).”

“Specifically, we augment the original categories of general VQA [5, 84], vision-centric perception [58, 60], chart/OCR [36, 44], and mathematical reasoning [38, 39] with spatial reasoning [32, 83] and hallucination resistance [21, 28] to enable a more holistic assessment.”

“We evaluate on a suite of multimodal benchmarks covering both perception-intensive and reasoning-oriented capabilities us-ing the VLMEvalKit evaluation framework (Duan et al., 2024).”

“In document processing, recent benchmarks for OCR and document understanding [21] have revealed performance gaps in handling multilingual and handwritten text, while systematic evaluation on production-scale datasets and environmental impact assessment remains limited.”

“1 [65] and RealWorldQA [66]), multimodal reasoning (MMMU [67], MMMU-Pro [68], and MathVista [69]), and text recognition (OCRBench [70]).”

“…the ability of ERASE to preserve fine-grained visual information, we conduct experiments on six challenging benchmarks covering text recognition (TextVQA [27], OCRBench [28]), document and chart understanding (ChartQA [29], DocVQA [30], InfoVQA [31]), and complex visual reasoning (MathVista [32]).”

“…of VLMs many benchmarks have also been released which test various capabilities such as VQA [111, 112], STEM understanding and reasoning [113], OCR [114, 35], chart and plot understanding [100], coding [115], video understanding [116, 117], GUI understanding and computer use [118, 119, 120, 121],…”

“LVLM-based approaches [32, 33] directly transcribe textual regions using the model’s joint vision-language understanding.”

“As shown in Table 3, we evaluate them on both retrieval and VQA tasks, including MMMU [42], RealWorldQA [46], and OCRBench [27], to assess the preservation of pre-trained capabilities.”

“For general benchmarks, MathVision [55] is reported as accuracy over 3,040 questions spanning 16 mathematical sub-fields, OCRBench [52] is reported using the normalized final score, defined as the raw number of correct OCR sub-task answers divided by 10 to obtain a 0–100 scale, and RealWorldQA [76]…”

“Due to space constraints, the main paper focuses on two representative benchmarks for understanding: MMLU [17] (general knowledge reasoning) and OCRBench [34] (fine-grained visual perception).”

“We further analyze OCRBench degradation and find that these tasks rely on visual grounding signals unavailable to the text-only teacher (Appendix F).”

“OCR-focused suites [9, 24] target scene text recognition and structured extraction.”

“To assess whether the merged model preserves the capabilities of the original specialist models, we evaluate it on three domain-specific benchmarks: MMLU [13] for language understanding, OCR-Bench [22] for vision–language reasoning, and MMAU [31] for audio–language tasks.”

“Performance evaluation necessitates balancing parameter efficiency, inference latency, and accuracy[18].”

“The document understanding benchmarks consist of ChartQA [38], DocVQA [40], TextVQA [46], AI2D [23], InfoVQA [39] and OCRBench [35].”

“OCRBench OCRBench [25] is a comprehensive benchmark for evaluating the OCR capabilities of large-scale multimodal models, covering five task categories: text recognition, scene text VQA, document VQA, key information extraction, and handwritten formula recognition.”

“However, when confronted with few-shot text and real-world complexities, VLMs still face challenges in handling intricate problems and achieving practical deployment [12] [13].”

“OCRVQA (Pham et al., 2025) and OCRBench (Liu et al., 2024b; Fu et al., 2024a) further stress OCR-centric reasoning under complex layouts and challenging visual conditions, focusing on accuracy and robustness of text grounding.”