The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Open accessAbstract
We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and text-books, covering six core disciplines: Art & Design, Busi-ness, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly het-erogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. Unlike existing benchmarks, MMMU focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of 28 open-source LMMs as well as the propri-etary GPT-4V(ision) and Gemini highlights the substantial challenges posed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieve accuracies of 56% and 59% respectively, indicating significant room for improvement. We believe MMMU will stimulate the community to build next-generation multimodal foundation models towards expert artificial general intelligence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Surprisingly, open-source models such as Qwen2.5-VL-72B-Instruct and Llama-4-Maverick, which match proprietary models on existing multi-modal benchmarks like MMMU [49] or MathVista [50], perform far worse on SPOT.”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [49]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Self-explanations are a prevailing approach on par with traditional model explanations like LIME saliency maps [71] for understanding LLM-generated responses [25, 32, 53] and are widely used in analyzing language model behavior [66, 99].”
From Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2025) · cites this paper as [99]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“This work relies on pre-trained visual representations like Faster RCNN features [61] to minimize the training sample complexity.”
MethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3×
“Later on, CLIP [60], ALIGN [27], SimVLM [71], CoCa [78], Flamingo [2], BLIP-2 [32], and Fuyu [6] (inter alia) have been proposed to train visual representation using ViT [18] from scratch with massive amount of web data.”
BackgroundMethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 3×
“Rapid advances in large language models (LLMs) [13, 54, 67] have sparked broad discussions on the controversial concept of artificial general intelligence (AGI), often used to describe AI systems that perform on par or surpass humans at most tasks [1, 7, 21, 29, 49, 52].”
BackgroundMethodVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“Rapid advances in large language models (LLMs) [13, 54, 67] have sparked broad discussions on the controversial concept of artificial general intelligence (AGI), often used to describe AI systems that perform on par or surpass humans at most tasks [1, 7, 21, 29, 49, 52].”
BackgroundLiu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · cited 1×
“LXMERT [64], UNITER [10], VinVL [80], Oscar [34], VilBert [45], and VLP [86] are among the earliest work to train universal vision-language models to tackle many multimodal tasks.”
Background
Research works printed but not linked: 19
- Arcas, 2023, Artificial general intelligence is already here, Noema Magazine
- Bavishi, 2023, Introducing our multimodal models
- 2024, GitHub Repository
- Chiang, 2023, Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
- Dosovitskiy, 2021, An image is worth 16×16 words: Transformers for image recognition at scale
- Team, 2024, Adept fuyu-heavy: A new multimodal model
- Li, 2023, Seed-bench: Benchmarking multimodalllms with generative comprehension, arXiv preprint
- Li, 2023, Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models
- Li, 2023, M3it: A large-scale dataset towards multimodal multilingual instruction tuning, arXiv preprint
- Liu, 2023, Aligning large multimodal model with robust instruction tuning, arXiv preprint
- Liu, 2024, Llava-next: Improved reasoning, ocr, and world knowledge
- 2024, Minicpm-v
- 2024, Ominilmm-12b
- 2023, Gpt-4v(ision) system card, OpenAI
- Ormazabal, 2024, Reka flash: An efficient and capable multimodal language model
- 2023, Qwen-vl-plus
- 2024, Qwen-vl-max
- 2024, Introducing the next generation of claude
- 2024, Infimm: Advancing multimodal understanding from flamingos legacy through diverse llm integration, InfiMM Team.
Cited by
Sorted by citations
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4231-5 · Cited by 1,058
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.02245 · Cited by 291
Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., & Chang, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73004-7_2 · Cited by 359
Multimodal generative AI for medical image interpretation
Rao, V. M., Hla, M., Moor, M., Adithan, S., Kwak, S., Topol, E. J., & Rajpurkar, P. (2025) · Nature · doi:10.1038/s41586-025-08675-y · Cited by 134
Generative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Zhan, Q., Liang, Z., Ying, Z., & Kang, D. (2024) · Findings of the Association for Computational Linguistics: ACL 2024 · doi:10.18653/v1/2024.findings-acl.624 · Cited by 147
Foundation models for materials discovery – current state and future directions
Pyzer‐Knapp, E. O., Manica, M., Staar, P., Morin, L., Ruch, P., Laino, T., Smith, J. R., & Curioni, A. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01538-0 · Cited by 144
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., & Luo, P. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3507000 · Cited by 273
Self-citationThinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.00994 · Cited by 120
“For MCA tasks, we follow standard practice [24, 31, 99] by using Accuracy ( ACC ), based on exact matching (with possible fuzzy matching), as the primary metric.”
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98
“Similarly, MiniGPT-5 (Zheng et al., 2023) includes the reconstruction loss of diffusion models in addition to the alignment loss of GILL.”
MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K., Qiao, Y., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73242-3_10 · Cited by 338
Self-citationBLINK: Multimodal Large Language Models Can See but Not Perceive
Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N. A., Ma, W.-C., & Krishna, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73337-6_9 · Cited by 345
Large language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71
Honeybee: Locality-Enhanced Projector for Multimodal LLM
Cha, J., Kang, W., Mun, J., & Roh, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01311 · Cited by 141
VisionZip: Longer is Better but Not Necessary in Vision Language Models
Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., & Jia, J. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01843 · Cited by 229
Self-citationLi, Z., Wu, X., Du, H., Liu, F., Nghiem, H. T., & Shi, G. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00147 · Cited by 91
On scalable oversight with weak LLMs judging strong LLMs
Kenton, Z., Siegel, N. Y., Kramár, J., Brown-Cohen, J., Albanie, S., Bulian, J., Agarwal, R., Lindner, D., Tang, Y., Goodman, N. D., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.04622 · Cited by 66
MM1: Methods, Analysis and Insights from Multimodal LLM Pre-training
McKinzie, B., Gan, Z., Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P., Shah, D., Du, X., Peng, F., Belyi, A., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73397-0_18 · Cited by 309
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Wen, L., Xie, Z., Yu, X., Ruan, C., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01210 · Cited by 334
Vision Language Models are blind
Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., & Nguyen, A. T. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-981-96-0917-8_17 · Cited by 54
Efficient GPT-4V level multimodal large language model for deployment on edge devices
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Chen, C., Li, H., Zhao, W., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-61040-5 · Cited by 60
Doris, A. C., Grandi, D., Tomich, R., Alam, M. F., Ataei, M., Cheong, H., & Ahmed, F. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4067333 · Cited by 76
NVILA: Efficient Frontier Visual Language Models
Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang, S. W., Xi, H., Cao, S., Gu, Y., Li, D., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00390 · Cited by 190
Mini-InternVL: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance
Gao, Z., Chen, Z., Cui, E., Ren, Y., Wang, W., Zhu, J., Tian, H., He, J., He, J., Zhu, X., et al. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00067-6 · Cited by 84
TokenPacker: Efficient Visual Projector for Multimodal LLM
Li, W., Yuan, Y., Liu, J., Tang, D., Wang, S., Qin, J., Zhu, J., & Zhang, L. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02491-7 · Cited by 159
Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data
Zhang, C., & Wang, S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00780 · Cited by 60
RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics
Song, C. H., Blukis, V., Tremblay, J., Tyree, S., Su, Y., & Birchfield, S. T. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01470 · Cited by 79
Self-citationTowards Open-Ended Visual Quality Comparison
Wu, H., Zhu, H., Zhang, Z., Zhang, E., Chen, Z., Liao, L., Li, C., Wang, A., Sun, W., Yan, Q., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72646-0_21 · Cited by 48
Putting ChatGPT vision (GPT-4V) to the test: risk perception in traffic images
Driessen, T., Dodou, D., Bazilinskyy, P., & de Winter, J. (2024) · Royal Society Open Science · doi:10.1098/rsos.231676 · Cited by 38
Large language models for batteries
Zuo, W., Zheng, H., He, T., Vishwanath, V., Chan, M. K. Y., Stevens, R. L., Amine, K., & Xu, G. (2025) · Joule · doi:10.1016/j.joule.2025.102037 · Cited by 35
From System 1 to System 2: A Survey of Reasoning Large Language Models
Zhang, D., Li, Z., Zhang, M.-L., Zhang, J., Liu, Z., Yao, Y., Xu, H., Zheng, J., Chen, X., Zhang, Y., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3637037 · Cited by 522
A Survey on Efficient Vision‐Language Models
Shinde, G., Ravi, A., Dey, E., Sakib, S., Rampure, M., & Roy, N. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70036 · Cited by 46
TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation
Qu, L., Zhang, H., Liu, Y., Wang, X., Jiang, Y., Gao, Y., Ye, H., Du, D. K., Yuan, Z., & Wu, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00243 · Cited by 131
Self-citationBenchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A SurveyPreprint
Li, Z., Wu, X., Du, H., Nghiem, H. T., & Shi, G. (2025) · Qeios · doi:10.32388/gxr68q · Cited by 33
Ma, Y., Liu, X., Chen, X., Liu, W., Wu, C., Wu, Z., Pan, Z., Xie, Z., Zhang, H., Yu, X., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00725 · Cited by 118
What Makes Multimodal In-Context Learning Work?
Baldassini, F. B., Shukor, M., Cord, M., Soulier, L., & Piwowarski, B. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00161 · Cited by 35
How should we test AI for human-level intelligence? OpenAI’s o3 electrifies quest
Jones, N. (2025) · Nature · doi:10.1038/d41586-025-00110-6 · Cited by 25
When Do We Not Need Larger Vision Models?
Shi, B., Wu, Z., Mao, M., Wang, X., & Darrell, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73242-3_25 · Cited by 54
PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology
Sun, Y., Wu, H., Zhu, C., Zheng, S., Chen, Q., Zhang, K., Zhang, Y., Wan, D., Lan, X., Zheng, M., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73033-7_4 · Cited by 36
Self-citationHuang, K.-H., Chan, H. P., Fung, Y. R., Qiu, H., Zhou, M., Joty, S., Chang, S., & Ji, H. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3513320 · Cited by 56
Yang, X., Wu, W., Feng, S., Wang, M., Wang, D., Li, Y., Sun, Q., Zhang, Y., Fu, X., & Poria, S. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103204 · Cited by 29
Attention Prompting on Image for Large Vision-Language Models
Yu, R., Yu, W., & Wang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73404-5_15 · Cited by 28
Seidel, L., Gehringer, S., Raczok, T., Ivens, S.-N., Eckardt, B., & Maerz, M. (2025) · Drones · doi:10.3390/drones9050347 · Cited by 20
Zhang, Q., Gao, S., Chen, W., Zhao, Y., Nie, Y., Chen, Z., Chen, S., Su, Y., & Sun, H. (2025) · Transactions in GIS · doi:10.1111/tgis.70135 · Cited by 28
Self-citationBai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.21631 · Cited by 27
“We evaluate the Qwen3-VL series on a wide range of multimodal reasoning benchmarks, primarily focusing on STEM-related tasks and visual puzzles, including MMMU (Yue et al., 2024a), MMMU-Pro (Yue et al., 2024b), MathVision (Wang et al., 2024b), MathVision-Wild photo (hereafter MathVision WP ) , MathVista (Lu et al., 20…”
InfluentialSelf-citationWords or Vision: Do Vision-Language Models Have Blind Faith in Text?
Deng, A., Cao, T., Chen, Z., & Hooi, B. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00366 · Cited by 19
ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object
Zhang, C., Pan, F., Kim, J., Kweon, I. S., & Mao, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02055 · Cited by 21
Medical multimodal large language models: A systematic review
Hu, Y., Xu, C., Lin, B., Yang, W., & Tang, Y. Y. (2025) · Intelligent Oncology · doi:10.1016/j.intonc.2025.09.005 · Cited by 14
AgroBench: Vision-Language Model Benchmark in Agriculture
Shinoda, R., Inoue, N., Kataoka, H., Onishi, M., & Ushiku, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00716 · Cited by 14
A Causality-Aware Paradigm for Evaluating Creativity of Multimodal Large Language Models
Huang, Z., Zhong, S., Zhou, P., Gao, S., Žitnik, M., & Lin, L. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3539433 · Cited by 16
Multimodal large language model for wheat breeding: A new exploration of smart breeding
Yang, G., Li, Y., He, Y., Zhou, Z., Ye, L., Fang, H., Luo, Y., & Feng, X. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.03.027 · Cited by 17
Luo, G., Yang, X., Dou, W., Wang, Z., Liu, J., Dai, J., Qiao, Y., & Zhu, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02324 · Cited by 23
Truong, T.-D., Nguyen, H.-Q., Nguyen, X.-B., Dowling, A. P. G., Li, X., & Luu, K. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02521-4 · Cited by 17
EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark
Li, M., Zhong, J., Chen, T., Lai, Y., & Psounis, K. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01245 · Cited by 12
Miao, Z., Zhang, Y., Fabian, Z., Celis, A. H., Beery, S., Li, C., Liu, Z., Gupta, A., Nasir, M., Li, W., et al. (2025) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.70120 · Cited by 25
Project Sid: Many-agent simulations toward AI civilization
AL, A., Ahn, A., Becker, N., Carroll, S., Christie, N., Cortes, M., Demirci, A., Du, M., Li, F., Luo, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.00114 · Cited by 12
Barman, K. G., Caron, S., Sullivan, E., de Regt, H. W., de Austri, R. R., Boon, M., Färber, M., Fröse, S., Golling, T., Lopez, L. G., et al. (2025) · The European Physical Journal C · doi:10.1140/epjc/s10052-025-14707-8 · Cited by 42
Li, Y., Yuan, S., Wang, H., Li, Q., Liu, M., Chen, X., Shi, G., & Zuo, W. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02035 · Cited by 15
Nasir, M. F., Rehman, M. U., & Hussain, I. (2026) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1735096 · Cited by 11
“To connect model performance in agricultural tasks with overall scale, Figure 6 shows the relationship between parameter count, training compute, and general reasoning ability, measured by the MMMU (Massive Multi-discipline Multimodal Understanding) benchmark (Yue et al., 2024).”
Imp: Highly Capable Large Multimodal Models for Mobile Devices
Shao, Z., Zhou, Y., Yu, J., Ouyang, X., Zheng, L., Gai, Z., Wang, M., Kuang, Z., & Ding, J. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3557680 · Cited by 27
Patnaik, S., Jain, R., Krishnamurthy, B., & Sarkar, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02207 · Cited by 11
All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages
Vayani, A., Dissanayake, D., Watawana, H., Ahsan, N., Sasikumar, N., Thawakar, O., Ademtew, H. B., Hmaiti, Y., Kumar, A., Kuckreja, K., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01822 · Cited by 25
Moëll, B., Farestam, F., & Beskow, J. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1557920 · Cited by 15
PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset
Liu, J., Fu, Y., Xie, R., Xie, R., Sun, X., Lian, F., Kang, Z., & Li, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01849 · Cited by 16
MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
Zhao, Y., Zhang, H., Xie, L., Hu, T., Gan, G., Long, Y., Hu, Z., Chen, W., Li, C., Xu, Z., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00793 · Cited by 80
Self-citationProvatas, N., Konstantinou, I., & Koziris, N. (2025) · IEEE Access · doi:10.1109/access.2025.3535085 · Cited by 11
POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models
He, J., Wang, X., Liu, S., Wu, G., Silva, C. T., & Qu, H. (2025) · IEEE Pacific Visualization Conference (PacificVis) · doi:10.1109/pacificvis64226.2025.00010 · Cited by 15
Instruction-guided fusion of multi-layer visual features in Large Vision-Language Models
Xu, L., Zheng, Y., Chen, H., Chen, X., Liang, Y., Lai, C., Li, B., & Xue, X. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111932 · Cited by 10
FaceXBench: Evaluating Multimodal LLMs on Face Understanding
Narayan, K., Vibashan, V. S., & Patel, V. M. (2026) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2026.3655668 · Cited by 15
Teaching multimodal LLMs to comprehend 12-lead electrocardiographic images
Liu, R., Bai, Y., Yue, X., & Zhang, P. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02551-3 · Cited by 22
Self-citationPeng, H., Cui, S., Wei, J., Lu, T., Cai, Y., & Wang, S. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3567609 · Cited by 10
Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillation
Shi, Y., Di, S., Chen, Q., & Xie, W. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00797 · Cited by 8
How Many Are in This Image A Safety Evaluation Benchmark for Vision LLMs
Tu, H., Cui, C., Wang, Z., Zhou, Y., Zhao, B., Han, J., Zhou, W., Yao, H., & Xie, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72983-6_3 · Cited by 8
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Chen, K., Gou, Y., Huang, R., Liu, Z., Tan, D., Xu, J., Wang, C., Zhu, Y., Zeng, Y., Yang, K., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00513 · Cited by 49
Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models
Wang, J., Lv, C., Li, X., Dong, S., Li, H., Yao, K., Li, C., Shao, W., & Luo, P. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00400 · Cited by 7
Pan, X., Bai, Y., Zou, K., Zhou, Y., Zhou, J., Fu, H., Tham, Y. C., & Liu, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103631 · Cited by 8
Self-citationUniToken: Harmonizing Multimodal Understanding and Generation Through Unified Visual Encoding
Jiao, Y., Qiu, H.-B., Jie, Z.-Q., Chen, S.-X., Chen, J.-J., Ma, L., & Jiang, Y.-G. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00345 · Cited by 56
Mitigating Hallucinations in Large Vision-Language Models via Accumulative Decoding
Zhang, H., & Zhang, Z. (2025) · International Conference on Electronic Information Engineering and Computer Science (EIECS) · doi:10.1109/eiecs67708.2025.11283452 · Cited by 7
SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews
Huotala, A., Kuutila, M., & Mäntylä, M. (2025) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem64174.2025.00053 · Cited by 12
EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning
Chen, Y., Ge, Y., Ge, Y., Ding, M., Li, B., Wang, R., Xu, R., Shan, Y., & Liu, X. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02676-0 · Cited by 91
Tihanyi, N., Bisztray, T., Dubniczky, R. A., Tóth, R., Borsos, B., Cherif, B., Jain, R., Muzsai, L., Ferrag, M. A., Marinelli, R. A., et al. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825051 · Cited by 12
PathVLM-Eval: Evaluation of open vision language models in histopathology
Gilal, N. U., Zegour, R., Al-Thelaya, K., Özer, E., Agus, M., Schneider, J., & Boughorbel, S. (2025) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2025.100455 · Cited by 7
MiniCPM-V LLaMA Model for Image Recognition: A Case Study on Satellite Datasets
Kömürcü, K., & Petkevičius, L. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3547144 · Cited by 10
Space-LLaVA: A Vision-Language Model Adapted to Extraterrestrial Applications
Foutter, M., Gammelli, D., Kruger, J., Foss, E., Bhoj, P., Guffanti, T., D’Amico, S., & Pavone, M. (2025) · IEEE Aerospace Conference (AERO) · doi:10.1109/aero63441.2025.11068757 · Cited by 21
Chen, R., Lee, E., Wang, Y., Yadav, A., Zhong, M., Pragti, P., Sun, Y., & Diao, J. (2025) · Aggregate · doi:10.1002/agt2.70118 · Cited by 6
Self-citationBlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices
Lü, X., Chen, Y., Chen, C., Tan, H. S., Chen, B., Xie, Y., Hu, R., Tan, G., Wu, R., Hu, Y., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00392 · Cited by 12
Yang, C., Dong, X., Zhu, X., Su, W., Wang, J., Tian, H., Chen, Z., Wang, W., Lu, L., & Dai, J. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02322 · Cited by 16
Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation
Zhang, Y., Su, Y., Liu, Y., Wang, X., Burgess, J., Sui, E., Wang, C., Aklilu, J. G., Lozano, A., Wei, A., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02754 · Cited by 19
Self-citationVLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models
Lee, B., Hachiuma, R., Wang, Y.-C. F., Ro, Y. M., & Wu, Y.-H. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02751 · Cited by 10
MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
Chitty-Venkata, K. T., Howland, S., Azar, G. A., Soboleva, D., Vassilieva, N., Raskar, S., Emani, M. K., & Vishwanath, V. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3731599.3767706 · Cited by 8
Visual language models show widespread visual deficits on neuropsychological tests
Tangtartharakul, G., & Storrs, K. R. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01179-y · Cited by 6
Park, S., Cui, C., Ma, Y., Moradipari, A., Gupta, R., Han, K., & Wang, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00756 · Cited by 6
MMInstruct: a high-quality multi-modal instruction tuning dataset with extensive diversity
Liu, Y., Cao, Y. Q., Gao, Z., Wang, W., Chen, Z., Wang, W., Tian, H., Lu, L., Zhu, X., Lu, T., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4187-3 · Cited by 34
Self-citationWhen AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research
Son, G., Hong, J., Fan, H., Nam, H., Ko, H., Lim, S., Song, J., Choi, J., Paulo, G., Yu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.11855 · Cited by 5
“Figure 3 illustrates the performance of o3 on six benchmarks: MathVista [50], MMLU-Pro [51], GPQA Di-amond [4], MMMU [49], HLE [52] and S POT (re-call). o3 exceeds 80% on the first four benchmarks, demonstrating robust general reasoning and code understanding.”
Videogamebunny: Towards Vision Assistants for Video Games
Taesiri, M. R., & Bezemer, C. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00144 · Cited by 13
Accuracy-Aware MLLM Task Offloading and Resource Allocation in UAV-Assisted Satellite Edge Computing
Yan, H., Huang, H., Zhao, Z., Wang, Z., & Zhao, Z. (2025) · Drones · doi:10.3390/drones9070500 · Cited by 5
Jung, J., Kim, E., Kim, S., Lee, J. H., Kim, B., & Chang, B. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01330 · Cited by 10
Symmetry-Aware Advances in Multimodal Large Language Models: Architectures, Training, and Evaluation
Liu, X., & Liu, H. (2025) · Symmetry · doi:10.3390/sym17091400 · Cited by 7
Benchmarking Knowledge and Capability of Large Language Models in Building Science Domain
Jiang, G., Chen, K., Liu, J., Chen, J.-L., Zhe, S.-D., & Wang, Z. (2025) · Energy Use · doi:10.59717/ipj.energy-use.2025.100026 · Cited by 4
LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
Cocchi, F., Moratelli, N., Caffagni, D., Sarto, S., Baraldi, L., Cornia, M., & Cucchiara, R. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00450 · Cited by 25
Metadata
- Authors
- Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., & Chen, W.
- Affiliations
- IN. AI Research
- University of Waterloo (CA)
- Independent
- The Ohio State University (US)
- Carnegie Mellon University (US)
- Princeton University
- University of Victoria (CA)
- ORCID
- Venue
- 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2024-06-16
- Pages
- 9556-9567
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4402716477
- Semantic Scholar
- b50d19c5c298f6562c3b3c6c3822a351bdc89260
- Cited by
- 429
- References
- 80
- Citations by year
- 2026: 213 · 2025: 175 · 2024: 41
- FWCI
- 88.55
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal benchmark, large language models, domain knowledge, visual reasoning, charts and diagrams, GPT-4V, multimodal foundation models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
