Vision language models are blind: Failing to translate detailed visual features into words

“Recent VLM diagnostics further isolate primitives such as intersections, overlapping shapes, circled letters, grids, and counting [24,26,29].”

“Rahmanzadehgervi et al. (2024) show that state-of-the-art models perform poorly on elementary perceptual judgments such as whether two lines intersect or how many shapes overlap, and Fu et al. (2024) find that perception-demanding tasks which “resist mediation through natural language” remain near…”

“We further evaluate extensibility on four specialized OOD benchmarks: ERQA [18], OCRBench [25], VlmsAreBlind [42], and Humaneval_V [73], which use the augmented registry described above.”

“Recent works [5–7] show that vision–language models underperform their underlying vision encoders on pure vision tasks, suggesting that visual information may not be fully utilised in multimodal reasoning.”

“…[26] ✓ Low Massive High ✓ SuperSplat [66], Pointcosm [85] ✓ Ultra Large Medium ✗ Thus, we cast high-level planning as a selection problem over discrete human-centric action primitives [61], avoiding the inherent limitations of VLMs [76] in continuous control regression and spatial reasoning.”

“For example, they fail to understand viewpoints and perspectives [29], hallucinate object existence [17], and struggle with low-level visual concepts [26].”

“Rich line of benchmarks and evaluation protocols pointed out that MLLMs often struggle at even basic spatial understanding [20, 30, 60, 79, 80, 88, 96, 116], and showed that their spatial cognition can be improved by well-curated data [12, 22, 40, 49, 57, 87], introducing novel architecture designs…”

“Prior work have shown MLLMs to fail rudimentary vision tests (Rahmanzadehgervi et al., 2024), perceive graph structures poorly (Li et al., 2024d), and lack compositional understanding of visual attributes, relations, and ordering (Yuksekgonul et al., 2022).”

“34,35 In Experiment 1, to account for the subjective nature of human similarity perception, we employ a relative-comparison task in which participants judge which of two target graphs is more similar to a reference graph.”

“Although recent VLMs promise powerful capabilities, they still face critical limitations: they struggle with negation [2], basic geometric reasoning [3], data leakage [4], and reliability in sensitive tasks such as medical diagnosis [5].”

“However,VLMs encounter limitations in tasks requiring precise localization, such as pedestrian detection, where they struggle to recognize small or distant pedestrians [14], [15].”

“For example, Rahmanzadehgervi et al. (2025) show multimodal models lag considerably behind humans at identifying simple tasks such as whether two circles overlap, with Huang et al. (2025) showing similar conclusions on visual arithmetic.”

“However, despite their promising performance, VLMs continue to struggle with some simple artifacts in image understand-This ICCV Workshop paper is the Open Access version, provided by the Computer Vision Foundation.”

“For vision language models, as pointed by [16], current VLMs perform poorly with tasks that require precise spatial information awareness, which is exactly the ability that sign language tasks requires.”

“Although it is known that VLMs’ performance in interpreting visualizations is currently limited [5], rapid advancements have spurred various attempts to leverage VLMs for visual interpretation [3, 4].”

“One benchmark, where solving tasks often requires visualising scenes, finds that LLMs still perform well below humans (Philip & Hemang, 2024), suggesting that LLMs struggle with visualisation (see also Rahmanzadehgervi et al., 2024).”

“Recent work, Vision language models are blind [31], explored the seven BlindTest, a basic visual tasks, and identified that VLM has frequently failed to understand the visual structure with textual prompts.”

“Accurate visual perception serves as a foundational prerequisite for Multimodal Large Language Models (MLLMs) to perform downstream reasoning and environmental interaction (Xingyu Fu et al. 2024; Rahmanzadehgervi et al. 2024; J.-T. Huang et al. 2025).”

“Counting offers a sharper diagnostic because answers are objective and depend purely on visual enumeration (Paiss et al., 2023; Qharabagh et al., 2025; Rahmanzadehgervi et al., 2024; Sengupta et al., 2025; Campbell et al., 2024).”

“Under a fixed visual input and encoding budget, these spatial structures remain difficult to recover accurately from visual observations alone (Liu et al., 2018; Rahmanzadehgervi et al., 2024; Chen et al., 2024).”

“General-purpose VLM evaluations examine fine-grained perception through spatial judgments and counting (Fu et al., 2024; Rahmanzadehgervi et al., 2024; Zhou et al., 2025).”

“Despite this progress, MLLMs can still struggle with precise spatial perception: they perform well on high-level image-text benchmarks yet fail on simple tasks requiring exact spatial information, especially when objects are close, over-lapping, or small [26].”

“Rahmanzadehgervi et al. (2024) study a complementary failure in which a visual fact is represented but not used, while other studies infer limited visual access from model behavior (Liu et al., 2026; Shahgir et al., 2026; Osaka et al., 2026).”

“Visual judgments can fail before a model chooses an experiment [84, 85, 86].”

“Fail on overlapping circles, line intersections, and ring counts [199]; CLIP-blind pairs [240]; answers survive total blurring, late layers suppress vision [289]; success tracks whether the target has a name [214]; text wins conflicts [54]; RL shortcuts [272] The language prior predicts the answer…”

“That multimodal benchmarks can be answered without looking is well documented [20, 23–26], and the failure generalizes past multimodal work [27–29] (Sec.”

“Subsequent studies on instruction-tuned multimodal LLMs have further revealed that even state-of-the-art architectures struggle with fine-grained visual grounding [22] and elementary spatial or geometric reasoning that humans perform effortlessly [18].”

“…we probe the baseline capabilities of MLLMs without augmentation to assess whether they guide search via human-like feature relationships (cf. Becker et al., 2014; D’Zmura, 1991 ; Sobel & Cave, 2002 ) or rely on superficial patterns that break under complexity (Rahmanzadehgervi et al., 2024).”

“Intersection, collinearity, and parallel-line counts resemble low-level visual tasks on which four tested models averaged 58.07% across a seven-task synthetic benchmark [24].”

“Recent studies suggest a gap between high-level reasoning and low-level visual perception in VLMs [14], raising concerns that strong benchmark performance may not reflect genuine numerical understanding.”

“MMMU, MMBench and BLINK assess broad knowledge and perception (Yue et al., 2024; Liu et al., 2024; Fu et al., 2024b); MathVista and elementary visual-relation tests probe more specific skills (Lu et al., 2024; Rahmanzadehgervi et al., 2024).”

“Although some studies on VLMs report strong performance on multimodal benchmarks [40], others document failures even on basic perceptual tasks [29]; still, whether VLMs can distinguish merely unusual situations from genuine physical danger, or understand the context when making safety judgments,…”

“BLINK [14] collects tasks a person answers in the blink of an eye, BlindTest [36] poses questions so simple that a sighted human never misses them, ArtQA [41] asks painting questions that require looking, and BabyVision [9], ChildBench [11], and KidGym [49] draw on tasks an infant or child can…”

“These inconsistencies point to a fundamental question regarding the minimum reliable resolving power of VLMs, and FineSightBench is designed to systematically investigate this limitation [34, 52].”

“VLM failures on geometric primitives have been shown to depend upon spurious correlations rather than image evidence [9,12,29,32,36,39,48].”

“…and language, including image captioning, visual question answering (VQA), document understanding, and chart interpretation [Liu et al., 2023, Chen et al., 2024a, Wang et al., 2024, Rahmanzadehgervi et al., 2024, Grattafiori et al., 2024, Team et al., 2024, Dai et al., 2024, Deitke et al., 2024].”

“Prior studies have documented broad shortcomings in the visual abilities of multimodal models [6, 11, 14, 34], as well as failures on basic geometric judgments, visual-only perception, and core visual tasks designed to reduce linguistic shortcuts [8, 19, 30].”

“…show that CLIP often relies on coarse or spurious cues, failing to capture detailed object attributes and subtle visual distinctions (Yuksekgonul et al., 2023; Rahmanzadehgervi et al., 2024; Adila et al., 2024; Wang et al., 2024; Zhang et al., 2024; Tong et al., 2024; Koishigarina et al., 2026).”

“In these settings, models consistently predict the illusion even when the image has been counterfactually altered so that the stimulus is objectively unambiguous [6].”

“…et al., 2024) it comes with two major drawbacks: For every subject model a new fiber model has to be trained entirely from scratch, and there is no control over the achieved level of invariance, i.e. how close samples from the fiber model are to their target representation under the subject model.”

“Studies [32,40] have shown that VLMs assign little attention to visual tokens, generating responses driven primarily by textual context rather than visual evidence.”

“However, LVLMs often struggle with fundamental visual perception such as identifying fine-grained attributes or understanding spatial relationships, leading to hallucinations or incorrect reasoning based on misperceived visual input [4, 13, 14].”

“A recent benchmark study found that vision-language models tend to hallucinate or produce vague spatial cues when asked to support blind navigation [13], limiting their standalone utility in fine-grained object localization.”

“Prior benchmarks have explored VLMs’ response sensitivity across object plurality [24], preposition [23], and plausibility [22].”

“Interface designs that mediate user interaction with these models also require careful attention to privacy, explainability, and interpretability [12][13].”

“Furthermore, models with vision capabilities have been shown to be brittle, highlighting their weaknesses [47, 65, 77, 93, 104, 117, 119].”

“In a recent paper [21], explore the limitations of VLMs when it comes to low-level vision tasks that require precise spatial understanding.”

“The limited capability in figure recognition is further illustrated by Rahman-zadehgervi et al. [20], who reveal that even state-of-the-art models like GPT-4V [16] fail at simple visual tasks such as counting overlapping shapes or determining line segment intersections.”

“Although DC can eliminate the need for training additional task-specific modules, recent work shows that LVLMs can struggle even with small counts [3,33] (e.g., up to ten), suggesting limited generalization beyond the count range seen during (pre-)training.”

“Vision-language-limit studies, including saliency benchmarks, 6 demonstrations that capable models miss obvious features, 30 and findings that models fail on representationally confusable inputs, 31 The deployment-safety literature our thesis ultimately addresses concerns benchmark scores,…”

“…range of challenging tasks, such as solving advanced college exam questions (Lu et al., 2023; Yue et al., 2024a;b) and spatial understanding tasks (Yang et al., 2024a;b), they also exhibit limitations in visual perception (Zhai et al., 2024a;b; Tong et al., 2024c;d; Rahmanzadehgervi et al., 2024).”

“…and mathematical figure interpretation (Roberts et al., 2024b; Lu et al., 2023; Wang et al., 2024b), visual coding (Zhang et al., 2024), geospatial and remote sensing (Roberts et al., 2024c; Muhtar et al., 2024), video game glitch detection (Taesiri et al., 2024) and medicine (Chen et al., 2024).”

“VLMs are shown to be imprecise at spatial information understanding and geometry (Rahmanzadehgervi et al., 2024; Kamath et al., 2023).”