PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns
Open accessAbstract
Large multimodal models extend the impressive capabilities of large language models by integrating multimodal understanding abilities.However, it is not clear how they can emulate the general intelligence and reasoning ability of humans.As recognizing patterns and abstracting concepts are key to general intelligence, we introduce PUZZLEVQA, a collection of 2000 puzzle instances based on abstract patterns.With this dataset, we evaluate large multimodal models with abstract patterns based on fundamental concepts, including colors, numbers, sizes, and shapes.Through our experiments on state-of-the-art large multimodal models, we find that they are not able to generalize well to simple abstract patterns.Notably, GPT-4V achieves a score of 46.4% on single-concept puzzles, which shows that state-of-the-art models struggle on our dataset.To diagnose the reasoning challenges in large multimodal models, we progressively guide the models with our ground truth reasoning explanations for visual perception, inductive reasoning, and deductive reasoning.Our systematic analysis finds that the main bottlenecks of GPT-4V are weaker visual perception and inductive reasoning abilities.Through this work, we hope to shed light on the limitations of large multimodal models and how they can better emulate human cognitive processes in the future 1 .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent literature indicates that vision language models are primarily bottlenecked by their vision module.
“This is consistent with recent findings in the literature that vision language models are primarily bottlenecked by their vision module49,50.”
From Generating dermatopathology reports from gigapixel whole slide images with HistoGPT (2025) · cites this paper as [50]
Section 5.1 (page unavailable) identifies visual perception as one of the main bottlenecks for GPT-4V and Claude 3 Opus on abstract puzzles, alongside inductive reasoning. It identifies deductive reasoning as the main bottleneck for LLaVA-13B and Gemini Pro. The paper therefore does not support the broader claim that vision-language models are primarily bottlenecked by their vision module. (LLM, conf=0.89)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 11
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question An- swering. In International Conference on Computer Vision (ICCV).
- Tom Brown, Benjamin Mann, Nick Ryder, et al. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Susan Carey. 2000. The origin of concepts. Journal of Cognition and Development, 1:37 -41.
- Charles Cole. 1996. Fluid concepts and creative analo- gies: Computer models of the fundamental mech- anisms of thought. Journal of the Association for Information Science and Technology, 47:403-404.
- Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq R. Joty, and Boyang Albert Li. 2022. Is gpt-3 a good data annotator? In Annual Meeting of the Association for Computational Linguistics.
- Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for com- positional language and elementary visual reasoning. In CVPR.
- Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199-22213.
- Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Informa- tion Processing Systems.
- Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual ques- tion answering benchmark requiring external knowl- edge. In Conference on Computer Vision and Pattern Recognition (CVPR).
- Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xi- ang Ren. 2024. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In The Twelfth Interna- tional Conference on Learning Representations.
- Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019. Raven: A dataset for rela- tional and analogical visual reasoning. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Cited by
Sorted by citations
Generating dermatopathology reports from gigapixel whole slide images with HistoGPT
Tran, M., Schmidle, P., Guo, R. R., Wagner, S. J., Koch, V., Lupperger, V., Novotny, B., Murphree, D. H., Hardway, H. D., D’Amato, M., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-60014-x · Cited by 36 · see citing works
MAPM: multiscale attention pre-training model for TextVQA
Yang, Y., Yu, Y., & Li, Y. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05727-0 · Cited by 3 · see citing works
MLRQA: A Dataset with Multimodal Logical Reasoning Challenges
Xiao, J., Lin, G., & Li, P. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0122-6_1
VQA-Induct: Instruction Induction for Visual Question Answering
Chen, P.-C., Huang, H., & Chen, H. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760963
GenEscape: Hierarchical Multi-Agent Generation of Escape Room Puzzles
Shan, M., Curless, B., Kemelmacher-Shlizerman, I., & Seitz, S. M. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00212
AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution
Wang, Z., Zeng, Y., 東, 公., Guo, Y., Zhu, F., Zhang, H., Zhang, W., & Zuo, W. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37383-0_24
FSMVision: Semantic Extraction of Finite State Machines from Diagrams via Multimodal AI
Bakchy, S. C., Islam, M. A., Rahman, S., & Rahman, M. T. (2026) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed69900.2026.11534790
- Authors
- Chia, Y. K., Toh, V., Ghosal, D., Bing, L., & Poria, S.
- Affiliations
- Singapore University of Technology and Design (SG)
- Venue
- Findings of the Association for Computational Linguistics ACL 2024
- Published
- 2024
- Pages
- 16259-16273
- OpenAlex
- W4402684064
- Cited by
- 10
- References
- 0
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 2
- FWCI
- 2.07
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Ministry of Education, India
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- large multimodal models, pattern recognition, general intelligence, visual perception, inductive reasoning, deductive reasoning, GPT-4V, visual puzzles, human cognitive processes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
