ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
Open accessAbstract
Large Multimodal Models (LMMs) exhibit shortfalls when interpreting images and, by some measures, have poorer spatial cognition than young children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by model progress. This creates a need for difficult benchmarks that remain relevant for longer. We introduce ZeroBench - a lightweight visual reasoning benchmark curated using adversarial filtering to be "impossible" for frontier LMMs at its original release, with initial SotA scores of 0% pass@1 and pass^5. We track progress on ZeroBench over the subsequent year, observing SotA reaching 6% pass^5 and 19% pass@5, indicating the potential longevity of the benchmark. We evaluate 46 LMMs on ZeroBench, compare performance to a human baseline, analyse strengths and weaknesses, chart a year of progress in visual capabilities, and publicly release ZeroBench at https://zerobench.github.io.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Does Spatial Cognition Emerge in Frontier Models?
Ramakrishnan, S. K., Wijmans, E., Kraehenbuehl, P., & Koltun, V. (2024) · arXiv · doi:10.48550/arxiv.2410.06468 · cited 3×
“SPACE (Ramakrishnan et al., 2024) systematically evaluates large-and small-scale spatial cognition, finding the spatial intelligence of frontier LMMs to to be inferior to animals.”
Vision language models are blind: Failing to translate detailed visual features into words
Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., & Nguyen, A. T. (2024) · arXiv · doi:10.48550/arxiv.2407.06581 · cited 2×
“…and mathematical figure interpretation (Roberts et al., 2024b; Lu et al., 2023; Wang et al., 2024b), visual coding (Zhang et al., 2024), geospatial and remote sensing (Roberts et al., 2024c; Muhtar et al., 2024), video game glitch detection (Taesiri et al., 2024) and medicine (Chen et al., 2024).”
Self-citation
Flagged first, then by citations
Research works printed but not linked: 46
- Agrawal, P., Antoniak, S., Hanna, E. B., Bout, B., Chap- lot, D., Chudnovsky, J., Costa, D., De Monicault, B., Garg, S., Gervet, T., et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024. · arXiv
- Bitton-Guetta, N., Slobodkin, A., Maimon, A., Habba, E., Rassin, R., Bitton, Y., Szpektor, I., Globerson, A., and Elovici, Y. Visual riddles: a commonsense and world knowledge challenge for large vision and language mod- els. arXiv preprint arXiv:2407.19474, 2024. · arXiv
- Chen, P., Ye, J., Wang, G., Li, Y., Deng, Z., Li, W., Li, T., Duan, H., Huang, Z., Su, Y., et al. Gmai- mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. arXiv preprint arXiv:2408.03361, 2024. · arXiv
- Chollet, F. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. · arXiv
- Chollet, F., Knoop, M., Kamradt, G., and Landers, B. Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604, 2024. · arXiv
- Dai, W., Lee, N., Wang, B., Yang, Z., Liu, Z., Barker, J., Rintamaki, T., Shoeybi, M., Catanzaro, B., and Ping, W. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402, 2024. · arXiv
- Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. · arXiv
- Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N. A., Ma, W.-C., and Krishna, R. Blink: Multi- modal large language models can see but not perceive. In European Conference on Computer Vision, pp. 148-166. Springer, 2025.
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. · arXiv
- Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and ele- mentary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
- Jones, A. L. Scaling scaling laws with board games. arXiv preprint arXiv:2104.03113, 2021. · arXiv
- Kazemi, M., Dikkala, N., Anand, A., Devic, P., Dasgupta, I., Liu, F., Fatemi, B., Awasthi, P., Guo, D., Gollapudi, S., et al. Remi: A dataset for reasoning with multiple images. arXiv preprint arXiv:2406.09175, 2024. · arXiv
- Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
- Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
- Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal llms with gener- ative comprehension. arXiv preprint arXiv:2307.16125, 2023. · arXiv
- Li, J., Lu, W., Fei, H., Luo, M., Dai, M., Xia, M., Jin, Y., Gan, Z., Qi, D., Fu, C., et al. A survey on benchmarks of multimodal large language models. arXiv preprint arXiv:2408.08632, 2024. · arXiv
- Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2025.
- Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. · arXiv
- Mensink, T., Uijlings, J., Castrejon, L., Goel, A., Cadar, F., Zhou, H., Sha, F., Araujo, A., and Ferrari, V. Encyclo- pedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3113- 3124, 2023.
- Miller, E. Adding error bars to evals: A statistical ap- proach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024. · arXiv
- Muhtar, D., Li, Z., Gu, F., Zhang, X., and Xiao, P. Lhrs- bot: Empowering remote sensing with vgi-enhanced large multimodal language model. arXiv preprint arXiv:2402.02544, 2024. · arXiv
- Northcutt, C. G., Athalye, A., and Mueller, J. Pervasive la- bel errors in test sets destabilize machine learning bench- marks. arXiv preprint arXiv:2103.14749, 2021. · arXiv
- Padlewski, P., Bain, M., Henderson, M., Zhu, Z., Relan, N., Pham, H., Ong, D., Aleksiev, K., Ormazabal, A., Phua, S., et al. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models. arXiv preprint arXiv:2405.02287, 2024. · arXiv
- Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Shaaban, M., Ling, J., Shi, S., Choi, M., et al. Humanity's last exam, 2025. URL https://arxiv.org/abs/ 2501.14249.
- Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., and Nguyen, A. T. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vi- sion, pp. 18-34, 2024.
- Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. · arXiv
- Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. · arXiv
- Reka-Team, R., Ormazabal, A., Zheng, C., d'Autume, C. d. M., Yogatama, D., Fu, D., Ong, D., Chen, E., Lam- precht, E., Pham, H., et al. Reka core, flash, and edge: A series of powerful multimodal language models. arXiv preprint arXiv:2404.12387, 2024. · arXiv
- Roberts, J., Han, K., and Albanie, S. GRAB: A Challeng- ing GRaph Analysis Benchmark for Large Multimodal Models. arXiv preprint arXiv:2408.11817, 2024a. · arXiv
- Roberts, J., Han, K., Houlsby, N., and Albanie, S. Sci- FIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation. arXiv preprint arXiv:2405.08807, 2024b. · arXiv
- Roberts, J., Lüddecke, T., Sheikh, R., Han, K., and Albanie, S. Charting new territories: Exploring the geographic and geospatial capabilities of multimodal LLMs. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 554-563, 2024c.
- Schmidgall, S., Su, Y., Wang, Z., Sun, X., Wu, J., Yu, X., Liu, J., Liu, Z., and Barsoum, E. Agent laboratory: Using llm agents as research assistants. arXiv preprint arXiv:2501.04227, 2025. · arXiv
- Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.03314, 2024. · arXiv
- Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. · arXiv
- Taesiri, M. R., Feng, T., Bezemer, C.-P., and Nguyen, A. Glitchbench: Can large multimodal models detect video game glitches? In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 22444-22455, 2024.
- Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. · arXiv
- Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024a. · arXiv
- Wang, Z., Xia, M., He, L., Chen, H., Liu, Y., Zhu, R., Liang, K., Wu, X., Liu, H., Malladi, S., Chevalier, A., Arora, S., and Chen, D. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. arXiv preprint arXiv:2406.18521, 2024b. · arXiv
- Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
- Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. "Transformers: State-of- the-Art Natural Language Processing". In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Compu- tational Linguistics. URL https://www.aclweb. org/anthology/2020.emnlp-demos.
- Xiao, Y., Sun, E., Liu, T., and Wang, W. Logicvista: Multi- modal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973, 2024. · arXiv
- Xu, F. F., Song, Y., Li, B., Tang, Y., Jain, K., Bao, M., Wang, Z. Z., Zhou, X., Guo, Z., Cao, M., Yang, M., Lu, H. Y., Martin, A., Su, Z., Maben, L., Mehta, R., Chi, W., Jang, L., Xie, Y., Zhou, S., and Neubig, G. Theagentcompany: Benchmarking llm agents on consequential real world tasks, 2024. URL https: //arxiv.org/abs/2412.14161.
- Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., and Wang, L. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9 (1):1, 2023. · arXiv
- Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multi- modal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. · arXiv
- Zerroug, A., Vaishnav, M., Colin, J., Musslick, S., and Serre, T. A benchmark for compositional visual reasoning. arXiv preprint arXiv:2206.05379, 2022. · arXiv
- Zhang, F., Wu, L., Bai, H., Lin, G., Li, X., Yu, X., Wang, Y., Chen, B., and Keung, J. Humaneval-v: Evaluating visual understanding and reasoning abilities of large mul- timodal models through coding tasks. arXiv preprint arXiv:2410.12381, 2024. · arXiv
Cited by
Sorted by citations
Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.21631 · Cited by 27 · see citing works
“…al., 2024), MathVerse (Zhang et al., 2024), DynaMath (Zou et al., 2024), Math-VR (Duan et al., 2025), LogicVista (Xiao et al., 2024), VisualPuzzles (Song et al., 2025b), VLM are Blind (Rahmanzadehgervi et al., 2025), ZeroBench (Main/Subtasks) (Roberts et al., 2025), and VisuLogic (Xu et al., 2025).”
Self-citationMeasuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14499 · Cited by 14 · see citing works
Kimi K2.5: Visual Agentic Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., Cai, S. H., Cao, Y., Chai, Z., Charles, Y., Che, H. S., Chen, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.02276 · Cited by 2 · see citing works
Measuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-3086 · Cited by 1 · see citing works
“The recent ZeroBench [34] involve difficult reasoning, but in the context of visual puzzles rather than economically valuable tasks.”
Kimi K3: Open Frontier Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., C., M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24653
PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
Lin, Z., Xie, Y., Qu, B., Wang, H., Li, J., Wu, H., Dong, Y., Yang, Z., Zhu, J., Lu, H., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24957
HCAST: Human-Calibrated Autonomy Software Tasks
Rein, D., Becker, J., Deng, A., Nix, S., Canal, C., O'Connel, D., Arnott, P., Bloom, R., Broadley, T., Garcia, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.17354
“For example, ZeroBench (Roberts et al., 2025) consists of questions that require reasoning about parts of a provided image, while ARC-AGI (Chollet, 2019) questions involve generating 2D pixel grids using a rule that must be deduced from examples.”
- Authors
- Roberts, J., Taesiri, M. R., Sharma, A., Gupta, A., Roberts, S., Croitoru, I., Bogolin, S.-V., Tang, J., Langer, F., Raina, V., Raina, V., Xiong, H., Udandarao, V., Lu, J., Chen, S., Purkis, S., Yan, T., Lin, W., Shin, G., Yang, Q., Nguyen, A. T., Atkinson, D. I., Baranwal, A., Coca, A., Dang, M., Dziadzio, S., Kunz, J. D., Liang, K., Lo, A., Pulfer, B., Walton, S., Yang, C., Han, K., & Albanie, S.
- Affiliations
- University of Cambridge (GB)
- University of Alberta (CA)
- University of Hong Kong (HK)
- University of Oxford (GB)
- Auburn University (US)
- Northeastern University (US)
- University of Central Florida (US)
- University of Tübingen (DE)
- College of Southern Maryland (US)
- Princeton University (US)
- University of Geneva (CH)
- University of Oregon (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2502.09696
- OpenAlex
- W4407631974
- Cited by
- 0
- References
- 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- large multimodal models, visual reasoning, benchmark design, spatial cognition, image interpretation, human comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
