MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
Open accessAbstract
Large Language Models (LLMs) and Large Multimodal Models (LMMs) exhibit impressive problem-solving skills in many tasks and domains, but their ability in mathematical reasoning in visual contexts has not been systematically studied. To bridge this gap, we present MathVista, a benchmark designed to combine challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets (i.e., IQTest, FunctionQA, and PaperQA). Completing these tasks requires fine-grained, deep visual understanding and compositional reasoning, which all state-of-the-art foundation models find challenging. With MathVista, we have conducted a comprehensive, quantitative evaluation of 12 prominent foundation models. The best-performing GPT-4V model achieves an overall accuracy of 49.9%, substantially outperforming Bard, the second-best performer, by 15.1%. Our in-depth analysis reveals that the superiority of GPT-4V is mainly attributed to its enhanced visual perception and mathematical reasoning. However, GPT-4V still falls short of human performance by 10.4%, as it often struggles to understand complex figures and perform rigorous reasoning. This significant gap underscores the critical role that MathVista will play in the development of general-purpose AI agents capable of tackling mathematically intensive and visually rich real-world tasks. We further explore the new ability of self-verification, the application of self-consistency, and the interactive chatbot capabilities of GPT-4V, highlighting its promising potential for future research. The project is available at https://mathvista.github.io/.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Open-source models such as Qwen2.5-VL-72B-Instruct and Llama-4-Maverick perform far worse on SPOT than proprietary models, despite matching them on existing multi-modal benchmarks like MMMU or MathVista.
“Surprisingly, open-source models such as Qwen2.5-VL-72B-Instruct and Llama-4-Maverick, which match proprietary models on existing multi-modal benchmarks like MMMU [49] or MathVista [50], perform far worse on SPOT.”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [50]
Cited by
Sorted by citations
Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01363 · Cited by 166 · see citing works
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98 · see citing works
“MathVista (Math V ) (Lu et al., 2023b) The evaluation score is computed by an LLM such as GPT-4, which is asked to compare a candidate answer with a reference one.”
A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist
Zhang, W., Zhao, L., Xia, H., Sun, S., Sun, J., Qin, M., Li, X., Zhao, Y., Zhao, Y., Cai, X., et al. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671801 · Cited by 74 · see citing works
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv · doi:10.48550/arxiv.2305.10415 · Cited by 66 · see citing works
“The implementation details is provided in the supplementary materials.”
BackgroundImproving Multi-Agent Debate with Sparse Communication Topology
Li, Y., Du, Y., Zhang, J., Hou, L., Grabowski, P., Li, Y., & Ie, E. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.427 · Cited by 20 · see citing works
Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.21631 · Cited by 27 · see citing works
“We evaluate the Qwen3-VL series on a wide range of multimodal reasoning benchmarks, primarily focusing on STEM-related tasks and visual puzzles, including MMMU (Yue et al., 2024a), MMMU-Pro (Yue et al., 2024b), MathVision (Wang et al., 2024b), MathVision-Wild photo (hereafter MathVision WP ) , MathVista (Lu et al., 20…”
InfluentialSelf-citationPathVLM-Eval: Evaluation of open vision language models in histopathology
Gilal, N. U., Zegour, R., Al-Thelaya, K., Özer, E., Agus, M., Schneider, J., & Boughorbel, S. (2025) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2025.100455 · Cited by 7 · see citing works
When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research
Son, G., Hong, J., Fan, H., Nam, H., Ko, H., Lim, S., Song, J., Choi, J., Paulo, G., Yu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.11855 · Cited by 5 · see citing works
“Figure 3 illustrates the performance of o3 on six benchmarks: MathVista [50], MMLU-Pro [51], GPQA Di-amond [4], MMMU [49], HLE [52] and S POT (re-call). o3 exceeds 80% on the first four benchmarks, demonstrating robust general reasoning and code understanding.”
Attacking Vision-Language Computer Agents via Pop-ups
Zhang, Y., Yu, T., & Yang, D. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.411 · Cited by 4 · see citing works
Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
Fan, Z., Wang, Y., Polisetty, S., & Fung, Y. R. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.1037 · Cited by 4 · see citing works
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., & Huang, G. (2025) · arXiv · doi:10.48550/arxiv.2504.13837 · Cited by 3 · see citing works
“5-VL-7B (Bai et al., 2025) on Geometry3K (Lu et al., 2021), and evaluate its visual reasoning capabilities on filtered MathVista-TestMini (Lu et al., 2024) and MathVision-TestMini (Wang et al., 2024).”
Kimi K2.5: Visual Agentic Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., Cai, S. H., Cao, Y., Chai, Z., Charles, Y., Che, H. S., Chen, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.02276 · Cited by 2 · see citing works
Self-citationPerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
Lin, Z., Xie, Y., Qu, B., Wang, H., Li, J., Wu, H., Dong, Y., Yang, Z., Zhu, J., Lu, H., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24957
Phan, L. (2025) · doi:10.21203/rs.3.rs-6615297/v1 · Cited by 65 · see citing works
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306 · see citing works
Self-citation
- Authors
- Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., & Gao, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.02255
- OpenAlex
- W4387356472
- Cited by
- 12
- References
- 0
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Mathematics, Computing, and Information Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- mathematical reasoning, visual reasoning, large language models, large multimodal models, benchmark, GPT-4V, foundation models, compositional reasoning, visual perception, self-consistency, self-verification, multimodal dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
