Vision language models are blind: Failing to translate detailed visual features into words

“Failure scenario”

“…success of VLMs across many tasks, these models still perform poorly on several surprisingly simple multi-object reasoning tasks – such as counting [23, 25, 40], relational image generation [7], relational scene understanding [15, 31], and simple visual analogy tasks [20, 38] – on which humans…”

“Note that this model does not currently feature an option for manually setting the temperature, and the provided documentation does not specify what the default temperature is.”

“1, an +8.3% gain on Visual Puzzles [43], and a dramatic leap on VLMsAreBlind-mini [41], where PyVision -Claude-4.”

“To date, work on identifying such biases has concentrated on general visual understanding benchmarks, 22,24,29,30 while medical benchmarks, which we show here to be the most susceptible, remain largely unexamined.”

“8 secures the second place, highlighting its robust perceptual capabilities [25, 56, 68], strong general visual proficiency [12, 37, 89], and consistent multi-image understanding performance [66, 69].”

“Numerous evaluations revealed that VLMs struggle on even elementary spatial-perception tasks [18, 50, 51, 62, 65] and higher-level spatial reasoning based on images or videos [26, 32, 39, 58, 60, 72].”

“DeafTest , a suite of four low-level auditory tasks (examples are illustrated in Figure 1), is introduced to examine the fundamental ability of MLLMs to listen , inspired by the Schwabach test in audiology [26] and BlindTest [61] in vision domain.”

“Please refer to Appendix A.3 high-level reasoning tasks on similar domains (Paiss et al., 2023; Rahmanzadehgervi et al., 2024), demonstrating the long-standing Moravec’s Paradox: tasks that are easy to humans could be extremely difficult to machines and vice versa (Moravec, 1988).”

“These results indicate that poor performance is not solely attributable to domain shifts, aligning with prior studies [38, 53], which emphasize the inherent complexity of perception tasks that may require multi-task integration.”

“Other examples include visual reasoning and designs tasks defined in [118].”

“…challenging benchmarks: MathVista (Lu et al., 2024), MathVision (Wang et al., 2024a), MathVerse (Zhang et al., 2024b), MMMU (Yue et al., 2024), EMMA (Hao et al., 2025), Blind (Rahmanzadehgervi et al., 2024), V ∗ (Wu and Xie, 2023), VisuLogic (Xu et al., 2025), and ZeroBench (Roberts et al., 2025).”

“However, a purely textual CoT may be sub-optimal for vision-centric (Fu et al., 2024; Rahmanzadehgervi et al., 2024; Hua et al., 2024; Tong et al., 2024) and vision-language tasks (Yu et al., 2023, 2024b; Hao et al., 2025).”

“As recent LVLMs evolve to be more powerful and intricate, 2D vision-language (VL) benchmarks have advanced towards meticulously designed evaluation or detailed analysis [6, 19, 41, 43, 44, 68, 74, 87, 89, 90].”

“We may attribute part of the models’ failure in analogical reasoning to an inability to correctly recognize the given transformation T given .”

“Most existing multimodal benchmarks remain limited to think about images : they focus on passive visual Q&A without active interactions (Rahmanzadehgervi et al., 2024; Yue et al., 2024; Wang et al., 2024b; Lu et al., 2023; Zou et al., 2024; Wu & Xie, 2024; Scale AI, 2025), restrict tool use to…”

“5-VL-72B-Instruct improves from 35.2% to 40.1% on MathVision [57], from 61.3% to 65.8% on VLMsAreBlind [46], and from 45.5% to 49.4% on Charxiv [63].”

“However, some works like BlindTest [23] show that LVLMs don’t seem to understand images truly; in other words, the models look at images too superficially.”

“Many works investigate the vision capabilities of MFMs via VQA-style benchmarks that combine visual and textual inputs to generate textual outputs [4, 21, 31, 39, 41, 48, 57, 58, 71].”

“In addition, we evaluate driving-oriented understanding on DriveBench [102], and assess broader multimodal capability on several general VQA benchmarks, including MMStar [15], MMMU [109], RealWorldQA, AI2D, MME [21], VLMsAreBlind [75], and ChartQA [71].”

“Vision-centric tasks have proven challenging for vision-language models (VLMs), especially when the tasks require object counting and localization, scene understanding, and 2D/3D spatial reasoning (Liao et al., 2024; Rahmanzadehgervi et al., 2024; Campbell et al., 2024).”

“However, despite these advancements, studies (Rahmanzadehgervi et al. 2024; Fu et al. 2024; Tong et al. 2024; Li et al. 2024c) show that SOTA VLMs still struggle with fine-grained visual perception tasks, such as detecting line intersections or object boundaries—tasks that are trivial for humans.”

“Adjacent to recent work that suggest MLMs overlook the rich information already present in the visual encoders [18, 34, 35, 45], offers a simple approach to help models to use and manipulate visual signals for improved multimodal reasoning performance.”

“…on which they are based have dramatic shortcomings in the case of reasoning (Nezhurina et al., 2024) and visual perception (Kamath et al., 2023; Rahmanzadehgervi et al., 2024; Gei-gle et al., 2024; Gou et al., 2024) or their combination (Zhou et al., 2023; Zhang et al., 2024; Wang et al.,…”

“Given recent findings that VLMs struggle with basic visual recognition tasks involving simple shapes and patterns, it is possible that they fail the Three Mountain Task adaptation not due to cognitive limitations but because of visual recognition difficulties (Rahmanzadehgervi et al., 2024).”

“Note: The results show the best-performing marker type per model. as identifying relative positions of objects on images [33,39,15].”

“However, as indicated in [50], VLMs struggle to provide precise point-level correspondences, particularly for tasks requiring high precision (e.g., hammer - pound - nail ).”

“Multiple recent works document such failures (Tong et al., 2024b; Rahmanzadehgervi et al., 2024), and weaknesses can be seen through benchmarks focused on object localization (Kazemzadeh et al., 2014; Wang et al., 2021), counting (Acharya et al., 2019) and relational question-answering (Liu et al.,…”

“VLMs perform very poorly on many tasks that are easy for humans, such as counting or visual search [Rahmanzadehgervi et al., 2024], and many of these tasks seem to involve the need to accurately bind features in-context.”

“This is because individual models can exhibit biased outcomes influenced by their training data and architectural constraints, and can be susceptible to hallucinations — generating plausible but incorrect information (Tong et al., 2024; Gervi et al., 2024).”

“When the support data involves composite tasks and becomes more complex and noisy (e.g., BlindTest, RealWorld, CharXiv, and MathVision), non-agentic methods are more likely to yield unstable or even degraded performance, whereas ContextNav consistently improves MLLMs’ performance.”

“In contrast to previous approaches that focus on LVLM output (Rahmanzadehgervi et al., 2024; Vo et al., 2025; Lee et al., 2025; Luo et al., 2025), we leverage the chain-of-embedding for fine-grained analysis of LVLM, which is defined as a sequence of hidden states across layers, i.e. , ( Z denotes…”

“Recent investigations have revealed that many VLMs and vision backbones [10,20,55] lack basic visual reasoning abilities such as determining whether lines intersect, how shapes are oriented, and counting items in a visual array .”

“…by visual encoders differ from the discrete token representations expected by language models; second, the information bottle-neck problem, wherein images compressed through visual encoders may lose substantial detailed information, particularly affecting low-frequency visual patterns [43].”

“However, existing MLLMs often struggle with mathematical images [8, 40, 52] and extract unreliable information from visual contents, which inevitably hampers In Rt△ABC, ∠C=90°, AC=4, O is a point on BC, and OC=3, E is the midpoint of AO, if a circle is drawn with O as the center and OC as the…”

“However, current VLMs consistently fail at physical reasoning tasks, struggling with basic spatial relationships (e.g., object positioning, counting) [33], object attributes [40], and physical interactions (e.g., stability assessment, dynamics prediction) [10, 14].”

“They frequently miscount objects in cluttered or overlapping scenes [6, 7], perform poorly in spatial reasoning (e.g., identifying left of or above) [8], and often fail to accurately locate or recognize salient targets during visual search [9].”

“We assess seven state-of-the-art VLMs on the counting task by asking closed-ended questions about an image [10], [17].”

“…spatial understanding of VLMs (What’sUp (Kamath et al., 2023a) and Embspatial-bench (Du et al., 2024)), and core visual perception abilities (BLINK (Fu et al., 2024) for various aspects like visual correspondence and BlindTestbasic (Rahmanzadehgervi et al., 2024) for recognizing basic patterns).”

“But more alarming is the fact that VLMs struggle with simple, low-level vision tasks like whether two lines intersect or two geometric primitives overlap or are close together [41].”

“Multi-turn conversation for self-correction Most existing studies rely on single -turn conversations, where the model is queried once per task (Rahmanzadehgervi et al., 2024).”

“This suggests that while LLMs can engage in high-level strategic planning, they need significant support systems to translate these plans into tactical actions.”

“Additionally, [Rahmanzadehgervi et al. , 2024] characterized MLLMs as effectively “blind” after they failed simple low-level visual tasks such as counting intersecting lines or circles—tasks trivial for humans with minimal world knowledge.”

“…reading report persistent weaknesses: SalBench (Dahou et al., 2025) high-11 lights difficulties with low-level perceptual cues, while BlindTest (Rahmanzadehgervi et al., 2024), SRBench (Stogiannidis et al., 2025), and VisOnlyQA (Kamoi et al., 2025) expose brittle shape, geometry, and spatial…”

“…visual properties that are generally salient to humans, such as intersections between lines, overlap between shapes, and the number of simple visual elements in a scene [69] — all foundational abilities needed to succeed on the visualization understanding tasks investigated in the present work.”

“We speculate that Vittle ’s information control prevents the reliance on a partial feature of a single modality [72], e.g., textual feature, which is a common source of hallucination.”

“For instance, many models struggle with the perception of fine-grained visual details (Rahmanzadehgervi et al., 2024).”

“Recent studies [33, 35, 40] have also identified various visual shortcomings in these generalist VLMs, such as weak spatial reasoning and localization awareness, highlighting the need for enhanced visual processing capabilities.”

“The underlying causes of these limitations are multifaceted, encompassing factors such as inadequate spatial and temporal understanding [16, 17, 18], vision-language representational misalignments [26, 22], challenges in processing long visual sequences due to context length constraints [21, 22,…”

“Occlusions impact VLMs’ understanding of objects and text in images 13,16,17 , and VLMs’ interpretations of spatial context—such as relative position, orientation, direction and perspective—can be unreliable 18 .”

“A growing line of benchmarks focuses specifically on quantity understanding (Rane et al., 2024; Paiss et al., 2023; Rahmanzadehgervi et al., 2024; Yuksekgonul et al., 2022).”

“…yet they exhibit puzzling failures on seemingly simple multi-object tasks: miscounting objects, confusing which color belongs to which shape, and producing “illusory conjunctions” that mirror errors observed in human rapid visual processing (Campbell et al., 2024; Rahmanzadehgervi et al., 2024).”

“Studies reveal frequent failures in both perception and reasoning, even on relatively simple tasks [11, 16, 24, 25, 41, 46, 49, 50].”

“Prior work has mainly assessed counting through behavioral accuracy or prompt-based evaluations [3, 5, 19, 20, 30].”

“[16] shows that LVLMs perform poorly with stacked or stair plots; [18] describes how reducing space between points or words can lead to sharp drop-offs in LVLM visual understanding.”

“Despite these striking capabilities, recent work on probing the workings of MLLMs have begun to uncover significant limitations in how current MLLMs integrate visual information [3, 12, 20].”

“The backbone of the agent is based on LLMs rather than vision-language models (VLMs), as VLMs consistently struggle with tasks requiring precise spatial information [RBTN24].”

“Interestingly, similar results have also been observed for tasks that do not involve relations, such as counting (Rane et al., 2024; Rahmanzadehgervi et al., 2024), or multi-object scene description (Campbell et al., 2024).”

“However, recent studies reveal that VLMs struggle with simple, low-level visual tasks that humans easily solve, highlighting a need to enhance their visual reasoning and understanding [5].”

“Since then, many recent studies [41,10,26,52,11,55,46,40,43] have shown that modern VLMs still exhibit clear weaknesses on basic vision-centric tasks, such as spatial reasoning, object counting, geometric perception, visual analogy, and fine-grained recognition.”

“Prior studies show that vision does not reliably benefit from increased inference length alone (Tian et al., 2025; Rahmanzadehgervi et al., 2024; Fu et al., 2025; Qin et al., 2025), and verbose reasoning can even exacerbate visually grounded errors by amplifying reliance on language priors (Chu et…”

“This failure mainly stems from the fact that the CLIP’s vision encoder [50] is designed for standard classification tasks and is not equipped to handle the more fine-grained visual understanding tasks required by the language modality [59, 51, 27].”

“However, despite the superb capabilities of state-of-the-art models, VLMs suffer from notable limitations in processing fine-grained visual details [18].”

“Analyzing the textual chain-of-thought responses of VLM judges, we find that they are often blind to critical image details [43] and miss differences between image pairs (Appendix G.4.2).”

“Frontier models still fail simple geometric probes such as whether two lines intersect or how many circles overlap [65].”

“When a VLM emits a wrong answer, is it Perceptual Blindness (the encoder cannot see the image), a Language Shortcut (the model has the visual signal but ignores it), or * Corresponding author.”

“Stage 1 (Perception): The What and Where Circuits perform Implicit Relevance Detection (IRD), taking the image and question as input to output relevant crop coordinates (e.g., localizing the woman’s ear).”

“Failures have been observed in geometric reasoning [17], compositional generalization [2, 9, 15], and bias driven by strong visual or linguistic priors [23].”

“As recent studies [23, 25, 29, 33] highlight, VLMs frequently suffer from hallucination and lack temporal or causal grounding, especially in long or complex videos.”

“Despite strong performance on high-level vision tasks, state-of-the-art VLLMs struggled with basic geometric tasks that are straightforward for humans [28].”

“BlindTest [31] evaluates the visual perception of four MLLMs, GPT-4o, Gemini-1.”

“…what requires a rudimentary conceptual understanding of quantity, at least in these dimensions, which converges with recent studies reporting that major language models and as well as their corresponding vision models fail at simple counting tasks [Rane et al., 2024, Rahmanzadehgervi et al., 2024].”

“…from the model’s internal decision process [60, 27, 6, 91]; in VLMs specifically, image-based biases influence answers yet rarely surface in CoT [46, 53, 57, 2], RL training improves accuracy while degrading CoT faithfulness [100], and thinking-mode reasoning amplifies text bias rather than…”

“VLMs exhibit systematic failures under occlusion and suboptimal viewpoints [21], yet provide no calibrated uncertainty signal [22].”

“For this reason, understanding the different failure mechanisms is important for proposing meaningful improvements to VLMs.”

“…architectures and training meth-ods, VLMs continue to struggle on simple visual tasks that are trivial for any human observer, such as counting objects in a scene or making judgements about their interactions (Rahmanzadehgervi et al., 2024; Schulze Buschoff et al., 2025a; Balazadeh et al., 2024).”

“Though there have been a few works discussing the shortfalls of visual capabilities of MLLMs, such as [20] and [23], it’s the first time that MLLMs have been reported to perform poorly on such simple structured perception tasks, to the best of our knowledge.”

“…) on benchmark datasets, including MAD-Bench (Li et al., 2023), MM-SafetyBench (Liu et al., 2023b), MathVista (Lu et al., 2023), and a variety of arithmetic-focused questions (Rahmanzadehgervi et al., 2024), demonstrate that leveraging multiple tokens leads to more reliable hallucination detection.”

“…domains of intuitive physics and causal reasoning, VLMs have also been shown to have a number of visual deficiencies, and they often struggle with simple visual tasks that would be trivial for a human observer (Rahmanzadehgervi et al., 2024; Schulze Buschoff et al., 2025; Balazadeh et al., 2024).”

“Unfortunately, despite this tremendous progress, existing MLLMs still struggle with tasks that require precise points or pixel-level reasoning, such as localization, counting, or salient keypoint detection [8, 37, 56].”

“For instance, some studies have rigorously evaluated LLVMs on basic perception tasks that are trivially easy for humans by introducing “blind” pairs of image datasets (Tong et al., 2024; Fu et al., 2024; Rahmanzadehgervi et al., 2024).”

“These limitations are further exemplified in our benchmark, where grid-based image observations differ significantly from the natural image-text pairs on which many VLMs are trained (Yu et al., 2023; Rahmanzadehgervi et al., 2024).”

“Rahmanzadehgervi et al. (2024) introduce BlindTest to expose a bottleneck in decoding basic geometric primitives, finding that vision encoders preserve sufficient spatial details but language de-coders fail to translate them accurately.”

“…object recognition, they struggle with precise enumeration, often exhibiting “numerosity blindness” or hallucination when counts exceed small subitizing ranges ( N < 5 ) (Paiss et al., 2023; Hsieh et al., 2023; Rahmanzadehgervi et al., 2024; Guan et al., 2024; Beyer et al., 2024; Guo et al., 2025).”

“"cl_value": “ … ", "rl_value": “ … ", "vdd": “ … ", "cc": “ … ", "l_m1": “ … ", "l_m10": “ … ", "l_m11": “ … ", "l_m12": “ … ",”

“Some recent studies focus exclusively on the perception capabilities of VLMs and show that these remain limited, particularly when it comes to fine-grained image details [15], [16], [17], [18], [19].”

“Other benchmarks like Pope, TextVQA and VizWiz performed significantly higher than random chance.”

“Prior studies indicate that VLMs often rely heavily on language priors from the LLM component [10] and sometimes make blind guesses rather than deriving conclusions based on visual data [15].”

“Recent work has also probed VLMs and found them incapable of solving straightforward tasks that specifically require visual information processing, where answers cannot be guessed from text alone [20].”

“Indeed, in visual domains, it has been repetitively discovered that common language models do not have sufficient visual commonsense knowledge, such as the color of common objects, leading to a poor performance on visual tasks [5]–[8].”