Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
Open accessAbstract
Large Vision Language Models (LVLMs) excel in various vision-language tasks.Yet, their robustness to visual variations in position, scale, orientation, and context that objects in natural scenes inevitably exhibit due to changes in viewpoint and environment remains largely underexplored.To bridge this gap, we introduce V 2 R-Bench, a comprehensive benchmark framework for evaluating Visual Variation Robustness of LVLMs, which encompasses automated evaluation dataset generation and principled metrics for thorough robustness assessment.Through extensive evaluation on 21 LVLMs, we reveal a surprising vulnerability to visual variations, in which even advanced models that excel at complex visionlanguage tasks significantly underperform on simple tasks such as object recognition.Interestingly, these models exhibit a distinct visual position bias that contradicts theories of effective receptive fields, and demonstrate a humanlike visual acuity threshold.To identify the source of these vulnerabilities, we present a systematic framework for component-level analysis, featuring a novel visualization approach for aligned visual features.Results show that these vulnerabilities stem from error accumulation in the pipeline architecture and inadequate multimodal alignment.Complementary experiments with synthetic data further demonstrate that these limitations are fundamentally architectural deficiencies, scoring the need for architectural innovations in future LVLM designs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
VLMs can fail when the input image is distorted.
“Besides lacking knowledge of image content, VLMs can also fail when the input image is distorted.”
From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [43]
Initial studies have examined the negative impact of visual variations and the effect of synthetic image degradation on captioning output.
“Initial studies have examined the negative impact of visual variations [43] and the effect of synthetic image degradation [57, 107] on captioning output, but the literature on systematically understanding the impact of real-world image distortions on captioning accuracy is limited.”
From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 3
- Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and Volker Tresp. 2023b. Benchmarking ro- bustness of adaptation methods on pre-trained vision- language models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
- Shujin Wu, Yi Fung, Sha Li, Yixin Wan, Kai-Wei Chang, and Heng Ji. 2024a. Macaroon: Training vision- language models to be your engaged partners. In Findings of the Association for Computational Lin- guistics: EMNLP 2024, pages 7715-7731.
- Xiyang Wu, Ruiqi Xian, Tianrui Guan, Jing Liang, Souradip Chakraborty, Fuxiao Liu, Brian M. Sadler, Dinesh Manocha, and Amrit Bedi. 2024b. On the safety concerns of deploying LLMs/VLMs in robotics: Highlighting the risks and vulnerabilities. In First Vision and Language for Autonomous Driv- ing and Robotics Workshop.
Cited by
Sorted by citations
Liu, Z., Kou, J., Zhang, W., Gu, C., Fang, X., Huang, Z., Hao, Y., Li, H., Lu, X., Yin, A., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189137 · Cited by 9 · see citing works
Garg, K., Tang, X., Heo, J., Morgan, D. R., Gergle, D., Sudderth, E. B., & Piper, A. M. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791309 · Cited by 3 · see citing works
“Initial studies have examined the negative impact of visual variations [43] and the effect of synthetic image degradation [57, 107] on captioning output, but the literature on systematically understanding the impact of real-world image distortions on captioning accuracy is limited.”
TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models
Lee, J. W., Ke, X., Ekbote, C., Polisetty, S., Fung, Y. R., & Liang, P. P. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.359 · Cited by 3 · see citing works
Self-citationFang, X., Yuan, H. R., Li, H., Kou, J., Gu, C., Zhang, W., Duan, X., & Fang, Y. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189217 · Cited by 3 · see citing works
Kou, J., Lu, X., Yin, A., Zhang, W., Fang, X., Li, Y., Pan, T., Zhang, L., Cheng, G., Yuan, H., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189154 · Cited by 2 · see citing works
Bao, H., Huang, Y., Wang, Y., Ye, J., Wang, X., Chen, X., Zhao, Y., Zhou, T., Elhoseiny, M., & Zhang, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817506
Self-citationWhen Harmful Content Goes Invisible: Unveiling Perception Failure of LVLMs with CAMHARMTI
Li, Y., Zhou, Q., Xu, Z., & Guo, H. (2026) · doi:10.1109/cvprf71909.2026.00200
- Authors
- Fan, Z., Wang, Y., Polisetty, S., & Fung, Y. R.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Venue
- Findings of the Association for Computational Linguistics: ACL 2025
- Published
- 2025
- Pages
- 20222-20242
- OpenAlex
- W4412887737
- Cited by
- 4
- References
- 0
- Citations by year
- 2026: 1 · 2024: 3
- FWCI
- 0.76
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- large vision-language models, benchmarking framework, object recognition, position bias, receptive fields, visual acuity thresholds, multimodal alignment, error accumulation, feature visualization, synthetic data, perspective shift
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
