The DOI matches its DataCite record, with a small difference in title or year; 9 of the 9 references it cites were found in a registry or an index.
Vision language models are blind: Failing to translate detailed visual features into words
Open accessAbstract
While large language models with vision capabilities (VLMs), e.g., GPT-4o and Gemini 1.5 Pro, score high on many vision-understanding benchmarks, they are still struggling with low-level vision tasks that are easy to humans. Specifically, on BlindTest, our suite of 7 very simple tasks, including identifying (a) whether two circles overlap; (b) how many times two lines intersect; (c) which letter is being circled in a word; and (d) the number of circles in an Olympic-like logo, four state-of-the-art VLMs are only 58.07% accurate on average. Claude 3.5 Sonnet performs the best at 77.84% accuracy, far from the human expected accuracy of 100%. Across different image resolutions and line widths, VLMs including slow-thinking models consistently struggle with those tasks that require precise spatial information when geometric primitives overlap or are close. Yet, VLMs perform at near-100% accuracy when much more space is added to separate shapes and letters. Linear probing experiments show that vision encoders contain sufficient visual information to solve BlindTest and that language models fail to decode this information into correct answers. Code and data are at: https://vlmsareblind.github.io
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have identified key limitations of VLMs that align with our findings, including biases toward natural image-text pairs, optimization for image description rather than action-oriented reasoning, and challenges with out-of-distribution inputs (Tan et al., 2024; Tong et al., 2024; Rahmanzadehgervi et al., 2024; Zang et al., 2024; Guan et al., 2023).”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Rahmanzadehgervi et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These limitations are further exemplified in our benchmark, where grid-based image observations differ significantly from the natural image-text pairs on which many VLMs are trained (Yu et al., 2023; Rahmanzadehgervi et al., 2024).”
From BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (2024) · cites this paper as Rahmanzadehgervi et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Accurate visual perception serves as a foundational prerequisite for Multimodal Large Language Models (MLLMs) to perform downstream reasoning and environmental interaction (Xingyu Fu et al. 2024; Rahmanzadehgervi et al. 2024; J.-T. Huang et al. 2025).”
From PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models (2026) · cites this paper as Rahmanzadehgervi et al. 2024
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 2
- [14]Hughes, J.F.: Computer graphics: principles and practice. Pearson Education (2014)
- [18]Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., Farhadi, A.: A dia- gram is worth a dozen images. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14. pp. 235–251. Springer (2016) https://www.anthropic.com/news/claude-3-5-sonnet https://www.anthropic.com/news/claude-3-5-sonnet https://www.anthropic.com/news/claude-3-family https://www.anthropic.com/news/claude-3-family https://gemini-spatial-example.grantcuster.com/ https://gemini-spatial-example.grantcuster.com/ https://www.zeiss.com/vision-care/us/eye-health-and-care/vision-screening.html https://www.zeiss.com/vision-care/us/eye-health-and-care/vision-screening.html
Cited by
Sorted by citations
Do generative video models understand physical principles?
Motamed, S., Culp, L., Swersky, K., Jaini, P., & Geirhos, R. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00099 · Cited by 147
“Failure scenario”
Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem
Campbell, D., Cohen, J. D., Sabbata, N. D., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2411.00238 · Cited by 104
“…success of VLMs across many tasks, these models still perform poorly on several surprisingly simple multi-object reasoning tasks – such as counting [23, 25, 40], relational image generation [7], relational scene understanding [15, 31], and simple visual analogy tasks [20, 38] – on which humans…”
Visual cognition in multimodal large language models
Schulze Buschoff, L. M., Akata, E., Bethge, M., & Schulz, E. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00963-y · Cited by 83
“Note that this model does not currently feature an option for manually setting the temperature, and the provided documentation does not specify what the default temperature is.”
PyVision: Agentic Vision with Dynamic ToolingPreprint
Zhao, S., Zhang, H., Lin, S., Li, M., Wu, Q., Zhang, K., & Chen, W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.07998 · Cited by 77
“1, an +8.3% gain on Visual Puzzles [43], and a dramatic leap on VLMsAreBlind-mini [41], where PyVision -Claude-4.”
InfluentialMIRAGE: The Illusion of Visual UnderstandingPreprint
Asadi, M., O’Sullivan, J. W., Cao, F., Nedaee, T., Rajabalifardi, K., Fei-Fei, L., Adeli, E., & Ashley, E. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.21687 · Cited by 78
“To date, work on identifying such biases has concentrated on general visual understanding benchmarks, 22,24,29,30 while medical benchmarks, which we show here to be the most susceptible, remain largely unexamined.”
Metadata
- Authors
- Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., & Nguyen, A. T.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2407.06581
- OpenAlex
- W4400518804
- Semantic Scholar
- ad12e9233dad5408da24b5581b05558f27b183f1
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision-language models, geometric shape recognition, spatial reasoning, image resolution effects, letter recognition, GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
