Cited by 445Multimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot LearningAdvanced Neural Network Applications

The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.

MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Open access

Yue, Xiang, Ni, Yuansheng, Zheng, Tianyu, Zhang, Kai, Liu, Ruoqi, Zhang, Ge, Stevens, Samuel, Jiang, Dongfu, Ren, Weiming, Sun, Yuxuan, Wei, Cong, Yu, Botao, Yuan, Ruibin, Sun, Renliang, Yin, Ming, Zheng, Boyuan, Yang, Zhenzhu, Liu, Yibo, Huang, Wenhao, Sun, Huan, Su, Yu, Chen, Wenhu·2024·2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

DOI 10.1109/cvpr52733.2024.00913

Open this paper in CoChat — free account Open PDF

Abstract

We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and text-books, covering six core disciplines: Art & Design, Busi-ness, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly het-erogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. Unlike existing benchmarks, MMMU focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of 28 open-source LMMs as well as the propri-etary GPT-4V(ision) and Gemini highlights the substantial challenges posed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieve accuracies of 56% and 59% respectively, indicating significant room for improvement. We believe MMMU will stimulate the community to build next-generation multimodal foundation models towards expert artificial general intelligence.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1109/cvpr52733.2024.00913

Cites

88 of 88 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“This work relies on pre-trained visual representations like Faster RCNN features [61] to minimize the training sample complexity.”
Method
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3×
“Later on, CLIP [60], ALIGN [27], SimVLM [71], CoCa [78], Flamingo [2], BLIP-2 [32], and Fuyu [6] (inter alia) have been proposed to train visual representation using ViT [18] from scratch with massive amount of web data.”
BackgroundMethod
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 3×
“Rapid advances in large language models (LLMs) [13, 54, 67] have sparked broad discussions on the controversial concept of artificial general intelligence (AGI), often used to describe AI systems that perform on par or surpass humans at most tasks [1, 7, 21, 29, 49, 52].”
BackgroundMethod
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“Rapid advances in large language models (LLMs) [13, 54, 67] have sparked broad discussions on the controversial concept of artificial general intelligence (AGI), often used to describe AI systems that perform on par or surpass humans at most tasks [1, 7, 21, 29, 49, 52].”
Background
Visual instruction tuning
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · cited 1×
“LXMERT [64], UNITER [10], VinVL [80], Oscar [34], VilBert [45], and VLP [86] are among the earliest work to train universal vision-language models to tackle many multimodal tasks.”
Background

Show all 80 references

Research works printed but not linked: 19

Arcas, 2023, Artificial general intelligence is already here, Noema Magazine
Bavishi, 2023, Introducing our multimodal models
2024, GitHub Repository
Chiang, 2023, Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
Dosovitskiy, 2021, An image is worth 16×16 words: Transformers for image recognition at scale
Team, 2024, Adept fuyu-heavy: A new multimodal model
Li, 2023, Seed-bench: Benchmarking multimodalllms with generative comprehension, arXiv preprint
Li, 2023, Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models
Li, 2023, M3it: A large-scale dataset towards multimodal multilingual instruction tuning, arXiv preprint
Liu, 2023, Aligning large multimodal model with robust instruction tuning, arXiv preprint
Liu, 2024, Llava-next: Improved reasoning, ocr, and world knowledge
2024, Minicpm-v
2024, Ominilmm-12b
2023, Gpt-4v(ision) system card, OpenAI
Ormazabal, 2024, Reka flash: An efficient and capable multimodal language model
2023, Qwen-vl-plus
2024, Qwen-vl-max
2024, Introducing the next generation of claude
2024, Infimm: Advancing multimodal understanding from flamingos legacy through diverse llm integration, InfiMM Team.

Sorted by citations

Items 101–200 of 445

Evaluating the robustness and readiness of large frontier models in health AI applications
池谷, 裕., Fu, J., Liu, X., Valanarasu, J. M. J., Codella, N., Tan, R., Liu, Q., Jin, Y., Zhang, S., Wang, J., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04501-8 · Cited by 48
“To address this, we conducted a structured, clinician-guided audit of nine representative benchmarks 26–30 .”
Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective
Ma, X., Xie, H., & Qin, S. J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103419 · Cited by 7
Open Vocabulary Emotion Prediction Based on Large Multimodal Models
Zhang, Z., Dong, Z., Gao, Z., Gao, S., Wang, D., Chen, C., Nie, Y., & Zhao, H. (2024) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3689092.3689402 · Cited by 5
Generative Multimodal Models for Social Science: An Application with Satellite and Streetscape Imagery
Law, T. L., & Roberto, E. (2025) · Sociological Methods & Research · doi:10.1177/00491241251339673 · Cited by 8
Less is More: High-value Data Selection for Visual Instruction Tuning
Liu, Z., Zhou, K., Zhao, W. X., Gao, D., Li, Y., & Wen, J.-R. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755160 · Cited by 39
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Wang, W., Chen, Z., Liu, Y., Cao, Y., Wang, W., Zhu, X., Lu, L., Lü, T., Qiao, Y., & Dai, Y. (2025) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-031-94969-2_2 · Cited by 4
Self-citation
A Survey on Fine-Grained Multimodal Large Language Models
Peng, Y., Wang, Z., Li, G., Zheng, X., Yin, S., & He, H. (2026) · Chinese Journal of Electronics · doi:10.23919/cje.2025.00.336 · Cited by 10
Generative AI in a new era of computer model-informed tobacco research: a short report
Vassey, J., Kennedy, C. J., Chang, H.-C. H., & Unger, J. B. (2025) · Tobacco Control · doi:10.1136/tc-2024-058887 · Cited by 4
CrysMTM: a multiphase, temperature-resolved, multimodal dataset for crystalline materials
Polat, C., Serpedin, E., Kurban, M., & Kurban, H. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/adf9bc · Cited by 6
Dynamic Model Deployment, Batch Scheduling, and Resource Allocation in MLLM-Enabled Edge–Cloud Networks: A Multiagent Two-Timescale DRL Approach
Huang, H., Du, Y., Zhan, W., Duan, H., Peng, K., Cheng, Y., Ye, Y., & Zhao, Z. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3611003 · Cited by 4
A Benchmark and Chain-of-Thought Prompting Strategy for Large Multimodal Models with Multiple Image Inputs
Zhang, D., Yang, J., Lyu, H., Jin, Z., Yao, Y., Chen, M., & Luo, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78456-9_15 · Cited by 62
Application of Large Language Models in Power System Operation and Control
Zhang, Y. (2024) · Journal of Computing and Electronic Information Management · doi:10.54097/sb9qdz28 · Cited by 5
SensorBench: Benchmarking LLMs in Coding-Based Sensor Processing
Quan, P., Ouyang, X., Jeyakumar, J. V., Wang, Z., Xing, Y., & Srivastava, M. B. (2025) · IEEE Workshop on Mobile Computing Systems and Applications (WMCSA) · doi:10.1145/3708468.3711882 · Cited by 6
TinyVision: Distributed Vision-Language Model with Efficiency and Privacy for Edge Deployment
Lou, S., Ge, S., Yu, J., & Zhang, G. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9849-3_15 · Cited by 4
Self-citation
COCO is “ALL” You Need for Visual Instruction Fine-tuning
Han, X., Wang, Y., Zhai, B., You, Q., & Yang, H. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10687511 · Cited by 6
New frontiers in AI for biodiversity research and conservation with multimodal language modelsPreprint
Miao, Z., Zhang, Y., Fabian, Z., Celis, A. H., Beery, S., Li, C., Liu, Z., Gupta, A., Nasir, M., Li, W., et al. (2024) · doi:10.32942/x22s6f · Cited by 4
Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression
Sarkar, S., Che, Y., Gavin, A., Beerel, P. A., & Kundu, S. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.631 · Cited by 4
Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Xu, W., Chan, H. P., Li, L., Aljunied, M., Yuan, R., Wang, J., Xiao, C., Chen, G., Liu, C., Li, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.07044 · Cited by 4
Self-citation
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Gou, B., Huang, Z., Ning, Y., Gu, Y., Lin, M., Qi, W., Kopanev, A., Yu, B., Gutiérrez, B. J., Shu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.21506 · Cited by 3
Self-citation
GPT-4o for Visual Political Communication: Toward Automated Image Type Analysis
Achmann-Denkler, M., Haim, M., & Wolff, C. G. (2025) · ACM Web Science Conference (WebSci) · doi:10.1145/3717867.3717881 · Cited by 3
UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation
Zhang, Q., Ning, M., Liu, Z., Huang, Y., Yang, S., Wang, Y., Ye, J., Chen, X., Song, Y., & Li, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00856 · Cited by 4
VLM@school: Evaluation of AI Image Understanding on German Middle School Knowledge
Peinl, R., & Tischler, V. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-07986-2_41 · Cited by 4
MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution
Zhu, C., Wang, Y., Zeng, J., Jiang, J., & Lin, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37550-6_32 · Cited by 16
Look Further Ahead: Testing the Limits of GPT-4 in Path Planning
Aghzal, M., Plaku, E., & Yao, Z. (2024) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case59546.2024.10711396 · Cited by 6
Vision Generalist Model: A Survey
Wang, Z., Rao, Y., Sun, S., Liu, X., Wei, Y., Yu, X., Liu, Z., Wang, Y., Liu, H., Zhou, J. N., et al. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02502-7 · Cited by 3
Can Machines Understand Composition? Dataset and Benchmark for Photographic Image Composition Embedding and Understanding
Zhao, Z., Lu, P., Zhang, A., Li, P., Li, X., Liu, X., Hu, Y., Chen, S., Wang, L., & Guo, W. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01344 · Cited by 5
Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
Gafni, T., Karnieli, A., & Hanani, Y. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00308 · Cited by 3
Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis
Huang, J., Jia, B., Wang, Y., Zhu, Z., Linghu, X., Li, Q., Zhu, S. C., & Huang, S. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02288 · Cited by 8
Kimi K2.5: Visual Agentic Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., Cai, S. H., Cao, Y., Chai, Z., Charles, Y., Che, H. S., Chen, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.02276 · Cited by 770
“• Image Understanding : (math & reasoning) MMMU-Pro [75], MMMU (val) [76], CharXiv (RQ) [67], Math-Vision [61] and MathVista (mini) [36]; (vision knowledge) SimpleVQA [13] and WorldVQA 2 ; (perception) ZeroBench (w/ and w/o tools) [48], BabyVision [12], BLINK [18] and MMVP [57]; (OCR & document)…”
Self-citation
V-RoAst: Visual Road Assessment Can VLM be a Road Safety Assessor using the iRAP Standard?
Jongwiriyanurak, N., Zeng, Z., Goo, J. M., Wang, X., Ilyankou, I., Sriroongvikrai, K., Christie, N., Wang, M., Chen, H., & Haworth, J. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00176 · Cited by 7
Development of a Large-Scale Dataset of Chest Computed Tomography Reports in Japanese and a High-Performance Finding Classification Model: Dataset Development and Validation Study
Yamagishi, Y., Nakamura, Y., Kikuchi, T., Sonoda, Y., Hirakawa, H., Kano, S., Nakamura, S., Hanaoka, S., Yoshikawa, T., & Abe, O. (2025) · JMIR Medical Informatics · doi:10.2196/71137 · Cited by 4
Application of Gemini in Public Health Amid the Artificial Intelligence Era
Tatar, M., Farokhi, S., Foumani, A. A., Uzunlar, E., & Araz, Ö. M. (2025) · IEEE Engineering Management Review · doi:10.1109/emr.2025.3577913 · Cited by 4
“Gemini presents a model,which is designed to tackle complex tasks while offering efﬁciency,accuracy, and adaptability [1],[2],[3].”
PathBench: Advancing the Benchmark of Large Multimodal Models for Pathology Image Understanding at Patch and Whole Slide Level
Sun, Y., Wu, H., Zhu, C., Si, Y., Chen, Q., Zhang, Y., Zhang, K., Li, J., Cai, J., Wang, Y., et al. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3584857 · Cited by 2
Self-citation
Multi-modal multi-agent debate-based dialectical knowledge transfer learning
Wang, H., Yu, W., Du, X., Chen, Q., Chu, Z., Yan, L., Jiang, J., & Guan, Y. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113551 · Cited by 3
SVLTA: Benchmarking Vision-Language Temporal Alignment via Synthetic Video Situation
Du, H., Wu, B., Lu, Y. H., & Mao, Z. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01288 · Cited by 3
CASP: Compression of Large Multimodal Models Based on Attention Sparsity
Gholami, M., Akbari, M., Cannons, K. J., & Zhang, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00875 · Cited by 3
radio-llava: Advancing vision-language models for radio astronomical source analysis
Riggi, S., Cecconello, T., Pilzer, A., Palazzo, S., Gupta, N., Hopkins, A., Trigilio, C., & Umana, G. (2025) · Publications of the Astronomical Society of Australia · doi:10.1017/pasa.2025.10082 · Cited by 14
Evaluation of Multimodal Image and Text Processing Models from an Uncertainty Perspective
Kostyumov, V., Nutfullin, B., & Pilipenko, O. G. (2025) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661825700166 · Cited by 47
MMCircuitEval: A Comprehensive Multimodal Circuit-Focused Benchmark for Evaluating LLMs
Zhao, C., Shi, Z., Wen, X., Liu, C., Liu, Y., Zhou, Y., Zhao, Y., Feng, H., Zhu, Y., Wan, G.-W., et al. (2025) · doi:10.1109/iccad66269.2025.11240653 · Cited by 7
Self-citation
Automated UML Generation: A Framework for Class Diagram Synthesis and Multimodal Validation
Nguyen, V.-V., Khánh, N. H., Nguyen, K.-S., Thi, M.-H. L., Nguyen, V. T., & Vu, D.-Q. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4724-1_15 · Cited by 5
S 3 : A simple strong sample-effective multimodal dialog system
Rykov, E., & Panchenko, A. (2025) · Data & Knowledge Engineering · doi:10.1016/j.datak.2025.102538 · Cited by 2
DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
Zeng, L., Yao, J., Liao, B., Tao, H., Liu, W., & Wang, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37369-4_18 · Cited by 15
Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
Huang, Y., Thede, L., Mancini, M., Xu, W., & Akata, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-12840-9_21 · Cited by 3
Beyond LLaVA-HD: Diving Into High-Resolution Multimodal Large Language Models
Zhang, Y., Wen, Q., Fu, C., Wang, K., Wang, X., Zhang, Z., Wang, L., & Jin, R. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3650761 · Cited by 18
Instruction-Following Evaluation of Large Vision-Language Models
Shiono, D., Miyawaki, S., Tanaka, R., & Suzuki, J. (2026) · New Generation Computing · doi:10.1007/s00354-025-00308-2 · Cited by 2
Benchmarking Gaslighting Negation Attacks Against Reasoning Models
Zhu, B., Yin, H., Chen, J., & Jiang, Y.-G. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-6960-1_14 · Cited by 6
CAFE: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning
Yu, H., Zhao, Z., Yan, S., Korycki, Ł., Wang, J., He, B., Liu, J., Zhang, L., Fan, X., & Yu, H. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00659 · Cited by 3
WP-CLIP: Leveraging CLIP to Predict Wölfflin's Principles in Visual Art
Ghildyal, A., Wang, L., & Liu, F. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00047 · Cited by 2
EdgeSD: Efficient Speculative Decoding With Vision-Decoding Disaggregation for MLLM Inference in Edge-Cloud Networks
Huang, H., Zhan, W., Duan, H., Peng, K., Min, G., Zhao, Z., Zhao, Z., & Ye, Y. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3667591 · Cited by 2
“The EdgeSD is tested on three visual question answering tasks: DocVQA [35], ChartVQA [36] and InfoVQA [37], and two tasks for comprehensive evaluation of MLLMs’ capabilities: MMBench [38] and MMMU [39].”
TalkSketch: Multimodal Generative AI for Real-Time Sketch Ideation with Speech
Shi, W., Upadhyay, S., Quek, G., & Choo, K. T. W. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-16893-1_6 · Cited by 2
FDPT: Federated Discrete Prompt Tuning for Black-Box Visual-Language Models
Wu, J., Chen, S., Tang, J., Yang, Y., Chen, Y., Wang, L., Lin, S., Wang, Z., Chen, W., & Tian, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00237 · Cited by 2
Self-citation
Towards Long-Horizon Vision-Language-Action System: Reasoning, Acting and Memory
Li, D., Zhang, Y., Cao, M., Donglai, L., Xie, W., Hui, T., Lin, L., Xie, Z., & Li, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00643 · Cited by 2
Token-Efficient VLM: High-Resolution Image Understanding Via Dynamic Region Proposal
Jiang, Y., Gu, J., Xue, T., Cheung, K. C., Molchanov, P. A., Yin, H., & Liu, S. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02238 · Cited by 2
HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
Wei, Z., Wang, G., Ma, X., Mei, K., Chen, H., Jin, Y., & Rao, F. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02084 · Cited by 3
4D-Bench: Benchmarking Multi-Modal Large Language Models for 4D Object Understanding
Zhu, W., Li, B., Zheng, C., Mai, J., Chen, J., Jiang, L., Hamdi, A., Martinez, S. R., Lin, C., Elhoseiny, M., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01963 · Cited by 5
Empowering Source-Free Domain Adaptation via MLLM-Guided Reliability-Based Curriculum Learning
Chen, D., Patwari, K., Lai, Z., Zhu, X., Cheung, S.-C. S., & Chuah, C. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00415 · Cited by 2
Retrospective Analysis of Google Gemini
Yadav, N. S., Goar, V. K., Yadav, P. S., Rajpurohit, J., & Kotecha, K. V. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-6106-7_35 · Cited by 4
Learning Chain of Counterfactual Thought for Bias-Robust Vision-Language Reasoning
Zhang, Y., Jiang, M. X., & Zhao, Q. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73242-3_19 · Cited by 2
Evaluating fine-tuned GPT models on different datasets in the healthcare domain
Lim, Y. H., Yeoh, P. S. Q., & Lai, K. W. (2025) · Innovation and Emerging Technologies · doi:10.1142/s2737599425500124 · Cited by 3
Aligning Vision Language Models with Contrastive Learning
Ak, K. E., Mohta, J., Dimitriadis, D., Manchanda, S., Xu, Y., & Shen, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91672-4_3 · Cited by 3
Benchmarking Vision Language Models on German Factual Data
Peinl, R., & Tischler, V. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-96235-6_20 · Cited by 2
VisionArena: 230K Real World User-VLM Conversations with Preference Labels
Chou, C., Dunlap, L., Mashita, K., Mandal, K., Darrell, T., Stoica, I. G., Gonzalez, J. E., & Chiang, W.-L. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00367 · Cited by 3
Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark
Song, E., Chai, W., Xu, W., Xie, J., Liu, Y., & Wang, G. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00642 · Cited by 3
Self-citation
Deception-Based Benchmarking: Measuring LLM Susceptibility to Induced Hallucination in Reasoning Tasks Using Misleading PromptsPreprint
Dou, R. (2024) · Preprints.org · doi:10.20944/preprints202407.0120.v1 · Cited by 2
VISTA: A Visual and Textual Attention Dataset for Interpreting Multimodal Models
Harshit, & Taşdizen, T. (2025) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw65960.2025.00048 · Cited by 1
ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning
Hao, Z., Guo, J., Shen, L., Luo, Y., Hu, H., & Wen, Y. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02440-4 · Cited by 5
Seeing Beyond Hallucinations: LLM-based Compositional Information Extraction for Multimodal Reasoning
Li, X., Lin, L., Wang, S., & Wu, H. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730081 · Cited by 3
UTTBench: Benchmarking Large Multimodal Models for text recognition in underwater thermal turbulent environments
Gao, H., Chen, J., He, M., Wang, J., Wu, S., Zhong, H. F., Bo, J., Zhou, Y. J., & Fan, L. (2025) · Displays · doi:10.1016/j.displa.2025.103181 · Cited by 1
Generative Multimodal Models Are In-Context Learners
Cui, Y., Sun, Q., Zhang, X., Zhang, F., Yu, Q., Luo, Z., Wang, Y., Rao, Y., Liu, J., Huang, T., et al. (2025) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-031-94969-2_4 · Cited by 1
KOFFVQA: An Objectively Evaluated Free-Form VQA Benchmark for Large Vision-Language Models in the Korean Language
Kim, Y., & Jung, J. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00062 · Cited by 1
A Novel Pipeline for Automatic UML Sequence Diagram Synthesis and Multimodal Scoring
Nguyen, V.-V., Khánh, N. H., Nguyen, K.-S., Thi, M.-H. L., Nguyen, V. T., & Vu, D.-Q. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-3355-8_34 · Cited by 1
Omnidirectional Spatial Modeling from Correlated Panoramas
Zhang, X., Fu, T., & Xu, Z. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3743093.3770977 · Cited by 1
CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision-Language Models on Cartographic Map Understanding
Ung, H. Q., Habault, G., Nishimura, Y., Niu, H., Legaspi, R., Oya, T., Kojima, R., Taya, M., Ono, C., Minamikawa, A., et al. (2025) · ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (SIGSPATIAL GIS) · doi:10.1145/3748636.3762755 · Cited by 1
Self-citation
Mitigating Hallucinations in Large Vision-Language Models via Grounded-Contrastive Decoding
Zhang, Z., & Zhang, H. (2025) · International Conference on Electronic Information Engineering and Computer Science (EIECS) · doi:10.1109/eiecs67708.2025.11283461 · Cited by 1
VLMs-in-the-Wild: Bridging the Gap Between Academic Benchmarks and Enterprise Reality
Bandraupalli, S., Gajera, M., Saifee, A. A., & Parwar, A. (2025) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1109/aimlsystems67835.2025.11330553 · Cited by 1
Design Practices and Lessons from Deploying On-device Vision-Language Interaction in Robotic Guide Dogs
Kwon, J., Lee, J., & Kwon, Y. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00256 · Cited by 1
Domain-Aware Enhancements to Vision-Language Models for Urban Traffic Safety Question Answering
Ha, V. T. D., Tran, T. H., Dong, G. T., Chu, N. C., Vu, H., & Nguyen, T. C. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00567 · Cited by 1
Multimodal AI teacher: Integrating edge computing and reasoning models for enhanced student error analysis
Xu, T., Zhang, Y., Chu, Z., & Wen, Q. (2026) · AI Magazine · doi:10.1002/aaai.70053 · Cited by 1
Leveraging Multimodal Large Language Models for Fall Risk Reduction in Older Adults in the Home: Proposed Model Design
Do, J., Suresh, V., Zhang, L., Chavre, B. M., Cha, J., & Pugliese, R. (2026) · JMIR Aging · doi:10.2196/77591 · Cited by 1
Can Multimodal Large Language Models Understand Graphic Design? A Comparative Study
Lin, J., Huang, D., Zhao, T., Zhan, D., & Lin, C.-Y. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3684992 · Cited by 1
Structured Policy Optimization: Enhance Large Vision-Language Model via Self-Referenced Dialogue
Sun, G., Qin, C., Feng, Y., Chen, Z., Xu, R., Dianat, S. A., Rabbani, M., Rao, R., & Tao, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00077 · Cited by 1
$\mathcal{F}_{M}$ FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging
Tang, Z., Haihong, E., Liu, J., Yang, Z., Li, R., Rong, Z., He, H., Hao, Z., Hu, X., Ji, K., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00311 · Cited by 2
Self-citation
GenieBlue: Integrating Both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devices
Lu, X., Chen, Y., Wu, R., Gao, H., Chen, X., Yang, X., Zhao, X., Zhou, A., Li, F., Wen, Y., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00400 · Cited by 1
Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension
Wang, X., Yang, Z., Li, L., Lu, H., Xu, Y., Lin, C.-C., Lin, K., Huang, F., & Wang, L. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00117 · Cited by 1
Self-citation
VSP: Diagnosing the Dual Challenges of Perception and Reasoning in Spatial Planning Tasks for MLLMS
Wu, Q., Zhao, H., Saxon, M., Bui, T., Wang, W. Y., Zhang, Y., & Chang, S. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00219 · Cited by 1
ADIEE: Automatic Dataset Creation and Scorer for Instruction-Guided Image Editing Evaluation
Chen, X., Wei, Y., Zhou, L., & Kumar, S. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01705 · Cited by 7
Acknowledging Focus Ambiguity in Visual Questions
Chen, C., Tseng, Y., Li, Z., Venkatesh, A., & Gurari, D. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00122 · Cited by 7
VFLowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization
Yang, S., Xu, R., Cui, C., Wang, T., Lin, D., & Pang, J. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02218 · Cited by 1
ProJudge: A Multi-Modal Multi-Discipline Benchmark and Instruction-Tuning Dataset for Mllm-Based Process Judges
Ai, J., Zhou, P., Xu, Z., Li, M., Zhang, F., Li, Z., Sun, J., Feng, Y., Huang, B., Wang, Z., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00445 · Cited by 1
Self-citation
Understanding Artificial General Intelligence: Defining Characteristics and Benchmarks
Muhsen, D. K., & Sadiq, A. T. (2025) · Journal of Artificial Intelligence & Control Systems · doi:10.66967/jaics.2025.v1i103 · Cited by 1
Multimodal large language model (MLLM) benchmark for intelligent construction in underground engineering
Li, T., Ge, X., Wang, Z., Ren, P., & Xu, Z. (2026) · Automation in Construction · doi:10.1016/j.autcon.2026.106997 · Cited by 3
Large Language Model Benchmarks: A Taxonomy of Capabilities, Scientific Quality Assessment, and Saturation Analysis
Gómez, R., Miranda, C. E., Romero-González, J.-A., Esparza, D. M. C., Alfonso-Francia, G., Chávez‐Urbiola, E. A., Ramírez-Pedraza, A., & Terven, J. R. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8060141 · Cited by 2
First-class AI? Assessing LLMs on undergraduate physics exams
Dowsett, B., Rey, M. P., & Pizzochero, M. (2026) · Physics Education · doi:10.1088/1361-6552/ae68b3 · Cited by 1
Boosting multimodal CoT reasoning through DPO with Error-prone Sample Synthesis
Lou, Y., Ke, W., He, Y., Wang, P., Liu, Q., & Xu, H. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104952 · Cited by 1
Beyond accuracy: cognitive, pedagogical, and practical readiness of multimodal LLMs for K-12 science education
Jiang, K., & Xiao, X. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2679730 · Cited by 1
Token-Based Affordance Grounding with Large Vision-Language Models
Lee, S. I., Lei, Q., Xu, D., Yang, D., Tan, R. T., Chen, Y., & Wang, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36846-1_24 · Cited by 1
InternVL-X: Advancing and Accelerating InternVL Series with Efficient Visual Token Compression
Lu, D., Zhang, Z., Huang, L., Sun, Y., Zeng, J., Shu, M., & Cao, H. (2026) · doi:10.1109/cvprf71909.2026.00554 · Cited by 2
Self-citation
Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
Zhan, Y., Zhu, Y., Zhao, H., Yang, F., Zheng, S., Tang, M., & Wang, J. (2026) · doi:10.1109/cvprf71909.2026.00558 · Cited by 3
Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructions
Gao, J., Gan, L., Li, Y., Ye, Y., & Wang, D. Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72998-0_23 · Cited by 3
Level Up Your Tutorials: VLMs for Game Tutorials Quality Assessment
Cambrin, D. R., Militone, G. S., Colomba, L., Malnati, G., Apiletti, D., & Garza, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92387-6_26 · Cited by 2

Authors: Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., & Chen, W.
Affiliations: IN. AI Research
University of Waterloo (CA)
Independent
The Ohio State University (US)
Carnegie Mellon University (US)
Princeton University
University of Victoria (CA)
ORCID: Zhang, Kai
Zhang, Ge
Stevens, Samuel
Jiang, Dongfu
Ren, Weiming
Sun, Yuxuan
Wei, Cong
Yuan, Ruibin
Zheng, Boyuan
Liu, Yibo
Huang, Wenhao
Sun, Huan
Su, Yu
Chen, Wenhu
Venue: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher: IEEE
Published: 2024-06-16
Pages: 9556-9567
ISSN: 1063-6919 (ISSN-L)
DOI: 10.1109/cvpr52733.2024.00913
OpenAlex: W4402716477
Semantic Scholar: b50d19c5c298f6562c3b3c6c3822a351bdc89260
Cited by: 429
References: 80
Citations by year: 2026: 213 · 2025: 175 · 2024: 41
FWCI: 88.55
Type: conference paper
Language: en
Open access: green
arxiv.org
Licence: doi.org/10.15223/policy-029 (stm-asf)
doi.org/10.15223/policy-037 (stm-asf)
Locations: IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Topics: Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords: multimodal benchmark, large language models, domain knowledge, visual reasoning, charts and diagrams, GPT-4V, multimodal foundation models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks