Cited by 445Multimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot LearningAdvanced Neural Network Applications

The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.

MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Open access

Yue, Xiang, Ni, Yuansheng, Zheng, Tianyu, Zhang, Kai, Liu, Ruoqi, Zhang, Ge, Stevens, Samuel, Jiang, Dongfu, Ren, Weiming, Sun, Yuxuan, Wei, Cong, Yu, Botao, Yuan, Ruibin, Sun, Renliang, Yin, Ming, Zheng, Boyuan, Yang, Zhenzhu, Liu, Yibo, Huang, Wenhao, Sun, Huan, Su, Yu, Chen, Wenhu·2024·2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

DOI 10.1109/cvpr52733.2024.00913

Open this paper in CoChat — free account Open PDF

Abstract

We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and text-books, covering six core disciplines: Art & Design, Busi-ness, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly het-erogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. Unlike existing benchmarks, MMMU focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of 28 open-source LMMs as well as the propri-etary GPT-4V(ision) and Gemini highlights the substantial challenges posed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieve accuracies of 56% and 59% respectively, indicating significant room for improvement. We believe MMMU will stimulate the community to build next-generation multimodal foundation models towards expert artificial general intelligence.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1109/cvpr52733.2024.00913

Cites

88 of 88 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“This work relies on pre-trained visual representations like Faster RCNN features [61] to minimize the training sample complexity.”
Method
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3×
“Later on, CLIP [60], ALIGN [27], SimVLM [71], CoCa [78], Flamingo [2], BLIP-2 [32], and Fuyu [6] (inter alia) have been proposed to train visual representation using ViT [18] from scratch with massive amount of web data.”
BackgroundMethod
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 3×
“Rapid advances in large language models (LLMs) [13, 54, 67] have sparked broad discussions on the controversial concept of artificial general intelligence (AGI), often used to describe AI systems that perform on par or surpass humans at most tasks [1, 7, 21, 29, 49, 52].”
BackgroundMethod
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“Rapid advances in large language models (LLMs) [13, 54, 67] have sparked broad discussions on the controversial concept of artificial general intelligence (AGI), often used to describe AI systems that perform on par or surpass humans at most tasks [1, 7, 21, 29, 49, 52].”
Background
Visual instruction tuning
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · cited 1×
“LXMERT [64], UNITER [10], VinVL [80], Oscar [34], VilBert [45], and VLP [86] are among the earliest work to train universal vision-language models to tackle many multimodal tasks.”
Background

Flagged first, then by citations

Show all 80 references

Research works printed but not linked: 19

Arcas, 2023, Artificial general intelligence is already here, Noema Magazine
Bavishi, 2023, Introducing our multimodal models
2024, GitHub Repository
Chiang, 2023, Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
Dosovitskiy, 2021, An image is worth 16×16 words: Transformers for image recognition at scale
Team, 2024, Adept fuyu-heavy: A new multimodal model
Li, 2023, Seed-bench: Benchmarking multimodalllms with generative comprehension, arXiv preprint
Li, 2023, Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models
Li, 2023, M3it: A large-scale dataset towards multimodal multilingual instruction tuning, arXiv preprint
Liu, 2023, Aligning large multimodal model with robust instruction tuning, arXiv preprint
Liu, 2024, Llava-next: Improved reasoning, ocr, and world knowledge
2024, Minicpm-v
2024, Ominilmm-12b
2023, Gpt-4v(ision) system card, OpenAI
Ormazabal, 2024, Reka flash: An efficient and capable multimodal language model
2023, Qwen-vl-plus
2024, Qwen-vl-max
2024, Introducing the next generation of claude
2024, Infimm: Advancing multimodal understanding from flamingos legacy through diverse llm integration, InfiMM Team.

Sorted by citations

Items 301–400 of 445

EVA: Editing for Versatile Alignment Against Jailbreaks
Wang, Y., Qiu, H., Xu, Y., Yang, S., Qin, Z., Huang, M., & Wang, W. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3694189 · Cited by 1
“VLM utility is evaluated on MM-Vet-v2 [69], MMMU [70], and MMStar [71].”
Efficient Long-Horizon Mobile Manipulation with RAG-Assisted Vision Language Models
Han, Y., Qu, Y., Ren, J., Xu, M., Zhang, X., & Lu, H. (2025) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait67930.2025.11522616
Cambrian-P: Pose-Grounded Video Understanding
Yang, J., Zhao, Z., Pan, X., Yang, S., Zhang, J., Xu, H., Li, S.-W. D., & Xie, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37252-9_18 · Cited by 7
CLGA: A Linguistic Arena for Testing LLMs’ Prowess in Chinese Literary Games
Wang, Z., Ren, X., Wang, J., Wen, F., Zhang, T., LUO, Z., Sun, W., Zhou, Y., & Zhou, Y. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-9507-5_14
AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution
Wang, Z., Zeng, Y., 東, 公., Guo, Y., Zhu, F., Zhang, H., Zhang, W., & Zuo, W. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37383-0_24
Multimodal Fusion in the Era of Large Models: Methods, Evaluation, and Frontier Challenges
蒋, 松. (2026) · Artificial Intelligence and Robotics Research · doi:10.12677/airr.2026.153086
Interfaze: The Future of AI is built on Task-Specific Small Models
Khurdula, H. V., Agarwal, V., & Khemlani, Y. D. (2026) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai68641.2026.11536205
“This makes it difﬁcult to see which parts of the perception stack matter for benchmarks such as AI2D, ChartQA, MMMU, or Common Voice, where reliably reading diagrams, charts, and speech is the core challenge [14]–[16], [26].”
Anchored, Not Graded: Vision-Language Models Fail at Slant-from-Texture Perception
Zhang, Q., Golovanevsky, M., Domini, F., & Tompkin, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37393-9_18
Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency
Huang, Y., Thede, L., Mancini, M., Xu, W., & Akata, Z. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-02865-5 · Cited by 2
“4.2.3.”
Influential
Revisiting InternVL: A Systematic Technical Framework for Building Powerful Open-Source Vision-Language Models
Chen, Z., Wang, W., Zhu, J., Wang, X., Liu, Y., Liu, Z. T., Luo, G., Sun, P., Su, W., Tian, H., et al. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3702168
Self-citation
Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Models
Zhou, Z., Zhu, D., Wang, H., Zhang, H., 申, 嬌., & Zhao, Y. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810788
Optimize Multimodal Data Mixture for Pre-Training with Loss Regression
Shang, L., Cheng, H., Zhou, B., & Zhang, Y. (2026) · Algorithms · doi:10.3390/a19060482
Benchmarking Gaslighting Negation Attacks Against Multimodal Large Language Models
Zhu, B., Gui, Y., Qi, H., Chen, J., Ngo, C., & Lim, E. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810624 · Cited by 8
SKG-VLA: Scene Knowledge Graph Priors for Structured Scene Semantics and Multimodal Reasoning for Decision Making
Li, Z., & Li, L. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810830
Fine-Grained Cross-Modal Retrieval in Art via Region-Level Grounding of Symbolic Narratives
Bindila, M.-B., Wang, S., & Englebienne, G. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810645
Beyond Inconsistent Reasoning: Intermediate Process Direct Preference Optimization on Small Multi-modal Large Language Models for Visual Question Answering
Miao, Y., Su, P., Yin, H., Li, S., Tang, J., & Wang, T. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810807
Synergizing Agentic Data Generation and Efficient Representation Learning for Medical Cross-Modal Understanding
Cheng, L. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810827
Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safety
Liu, J., Xiao, J., Tang, W., Wang, Z., Xie, Z., Wang, W., Zhang, M., & Yu, S. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810787 · Cited by 7
OpenImplicit: Benchmarking Implicit Reasoning in MLLMs via Open-Ended Evaluation
Li, J., Yin, X., Guo, Y., Zhou, S., Tu, Y., Zhao, H., Duan, S., Liu, G., & Zhu, H. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810855
UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer
Wang, S., Li, L., Chen, Y., Gao, R., Teng, Y., & Wang, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37422-6_17 · Cited by 2
ElecThinker: A Three-Stage Framework to Enhance Electronic Diagram Reasoning in Multimodal LLMs
Chen, Q., Bai, Q., Jin, D., Li, J., Chen, J., Chen, S. F., Ye, S., & Li, W. (2026) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas66217.2026.11561947
“However, when applied to tasks that demand deep domain expertise, such as circuit analysis and design, they still reveal significant limitations [6].”
SPAR: Semantic-Pixel Self-alignment and Adaptive Routing for Unified Multimodal Models
Li, H., Chen, H., Zhu, C., Huang, X., Cai, J., Jiang, X., Hu, Y., & Chen, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37255-0_18 · Cited by 1
Unsteady Metrics and Benchmarking Cultures of AI Model Builders
Baack, S., Buschek, C., & Boháček, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812240
Improving Reasoning in Vision-Language Models via Perception Verified Self-training
Sharma, S., Gupta, S. K., & Thakur, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37592-6_19 · Cited by 1
Efficient and High Fidelity Multimodal Machine Unlearning for Information With Deep Memory
Zhang, J., Hou, J., Zhang, H., Xiao, T., & Huang, Y. (2026) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2026.3706840
“The evaluation covers popular multimodal evaluation benchmarks such as MME [36], MMB [37], MMMU [38], MMStar [39], AI2D [40], DocVQA [41], HallusionBench [42], and RealWorldQA.”
ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP
Zhang, S., Naseer, M., Xie, B., Suryanto, N., Qiu, S., Bentahar, J., Akhtar, N., & Shah, M. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37013-6_12
ViQ: Text-Aligned Visual Quantized Representations at Any Resolution
Yu, X., Liu, Z., Yang, Z., Dong, Y. H., Qian, S., Lu, J., Hu, H., & Rao, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37356-4_30 · Cited by 2
Self-citation
DiCoBench: Benchmarking Multi-image Fine-Grained Perception via Differential and Commonality Visual Cues
Li, G., & Peng, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37383-0_10
Paying More Attention to Visual Tokens in Self-evolving Large Multimodal Models
Venkatraman, S., Thawkar, R., Thawakar, O., Anwer, R. M., Cholakkal, H., Khan, S., & Khan, F. S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37464-6_3
Do Instructional Behaviors Generalize Across Disciplines? an Empirical Study with Fine-Tuned Multimodal LLMs
Bian, C., Ye, S., & Lu, W. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29760-0_2
Beyond Price: A Technical Quality Framework for AI Antitrust
Wu, P. Y. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812338
“Benchmarks are also commonly used to compare the performance of LLMs [see, e.g., 68, 133, 178, 183].”
On Test-Time Scaling for Vision-Language Models
Sammani, F., Chamiti, T., & Deligiannis, N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37271-0_10
CoLT: Teaching Multi-modal Models to Think with Chain of Latent Thoughts
Hu, L., Qin, S., Liao, Z., Guo, Q., Wan, L., Feng, W., & Liu, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37016-7_28
Self-citation
Information-Regularized Attention for Visual-Centric Reasoning
Sun, G. X., Wang, X., Patel, Y., Liu, M. Y., Tao, Z., & Krishnan, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37321-2_30
ReShift: Aha-Moment-Driven Reasoning-Level Backdoor Attacks on Vision–Language Models
Dou, Z., Zhao, Q., Gao, Z., & Biswas, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37627-5_9
LASER: A Corrective Lens for LVLMs via Visual Attention Preservation and Sink Suppression
Yuan, B., Wang, Z., Luo, Y., Wang, S., & Huang, Z. H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37211-6_20
The IN-OUT Framework for Studying Blackbox LLMs via Human-AI Interaction: Behaviorism in the Digital World
Grenzebach, J., & Radüntz, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29596-5_7
Token-Level Response-Visual Attention Guidance for Multimodal LLMs Knowledge Distillation
Jang, J., Yoon, E., Yoon, H. S., Eom, S., Hasegawa‐Johnson, M., & Yoo, C. D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37255-0_35
Transferability Between Understanding and Generation in Unified Multimodal Models
Kang, J., Yoon, H., Jung, J., Min, J., Jeon, M., Hwang, B., Jung, S., & Kim, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37132-4_35 · Cited by 4
AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning
Oh, K., & Kim, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37211-6_33
Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking
Chung, S., Yu, Y., Kim, S., Chee, Y., & Ro, Y. M. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3707796 · Cited by 2
A survey of transformers based on their input modalities
Sharma, P., & Jain, S. K. (2026) · Applied Intelligence · doi:10.1007/s10489-026-07378-9 · Cited by 1
“To evaluate vision-language grounding, multi-modal massive multi-task understanding (MMMU) [ 71] was introduced.”
On Locality and Length Generalization in Visual Reasoning
Madan, P., Haresh, S., Ebrahimi, R., Panchal, S., Bhattacharyya, A., & Memisevic, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37595-7_9
OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents
Chen, Y., Li, Y., Shen, Y., Liu, M., Zheng, T., Fu, B., Lin, Q., Yu, Z., & Shi, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37032-7_19
Self-citation
Preliminary Study of an Evaluation Benchmark for Vision–Language Models in Fashion E-Commerce
Shimizu, R., HtaungKham, S., Sakurai, S., & Shimizu, Y. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808442
“VLM evaluation has evolved from VQA-style benchmarks [2] to broader assessments including MMMU [44], MMBench [27], SEED-Bench [19], and MME [8], yet high scores do not reliably predict accuracy on narrower operational tasks [39], and none target fashion attributes or e-commerce extraction.”
HiRAD: Hierarchical Reasoning Adapter with Decoding Intervention for Efficient Visual Understanding in Compact Multimodal Models
Gao, Y., & Li, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3507-0_12
A Self-correcting Neuro-symbolic AI Reasoning Framework
Wooding, B., Brennan, K., Tumlin, A. M., Zhang, H., & Johnson, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-32357-6_9
A novel automated method for UML modeling integrating instruction, reasoning LLMs and vision language model evaluation
Nguyen, V.-V., Nguyen, H.-K., Nguyen, K.-S., Luong, T. M.-H., Bui, A.-T., Vu, D.-Q., & Nguyen, T.-V. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01846-4
“These weights are taken directly from the models’ scores on the MMMU benchmark [44], a standard benchmark for assessing multimodal inference performance.”
Evaluating Vision-Language Models vs. OCR Pipelines for Spatial Grounding in ISO/IEC 17025 Audits
Rasyid, S. A., Wiranata, A., Syamsul, M., Prihatini, Y. I., Usman, K., & Pratondo, A. (2026) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict71158.2026.11620745
“Benchmarks such as MMBench [12] and MMMU [13] test classification-level understanding, not metric coordinate prediction.”
SparseVLM+: Visual Token Sparsification With Improved Text-Visual Attention Pattern
Zhang, Y., Ma, J., Zhang, Q., Fan, C.-K., Zheng, W., Cheng, K., Lu, J., & Zhang, S. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3718245
“…on ﬁfteen widely adopted benchmarks, covering diverse tasks: VQAv2 (VQA V2 ) [50] and GQA [51] for general visual understanding; SQA [52] and MMMU [53] for multimodal reasoning; VQA Text (TextVQA) [54], AI2D [55], and ChartQA [56] for optimal character recognition; MME [57], SEED-Bench (SEED)…”
Exploring MLLMs for ultra-fine-grained agricultural classification
Wang, C., & Zhang, P. (2026) · The Knowledge Engineering Review · doi:10.48130/ker-0026-0010
PhotoTalk: an LLM-based photo album browsing assistance with chatbot for people with visual impairments
Lee, Y., Moon, S., Joh, H., Park, S., & Oh, U. (2026) · Universal Access in the Information Society · doi:10.1007/s10209-026-01369-w
“These models were plausible candidates because they supported image-based interaction and were representative of contemporary MLLMs with multimodal understanding and visual reasoning capabilities [36, 37].”
From detection data to structured knowledge: Augmenting multimodal large language models for substation safety inspection
Lai, Y., Wang, S., Xu, W., Fei, L., Wang, J., & Ning, D. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.115711
Systematic Survey of Large Language Model Evaluation: Methods, Benchmarks, and Comprehensive Taxonomic Framework
iBRAHIMOĞLU, N., Demirezen, M. U., & Yıldız, F. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01490-x · Cited by 1
REFLEX-Med: Reinforcement with Label-Free Explainability for Unified Medical Reasoning
Tang, L., Cai, Z., Tian, Q., Li, Z., Liu, Q., Bae, K. T., & Chen, C. (2026) · Multidimensional Signal Processing Workshop · doi:10.1109/ivmsp68335.2026.11618840
“Evaluation covers in-domain test sets plus out-of-domain suites (PMC-VQA [28], Quilt-VQA [29], MedXpertQA [30]) and the MMMU [31] Health & Medicine track to probe answer quality and explanation veriﬁability.”
REVERIE+: Generalized Reflective Instruction Tuning for Hallucination Mitigation in Advanced VLMs
Bi, M., Zhang, J., Wang, X., Jiang, X., Lu, Y., Wang, P., & Zheng, F. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-02945-6
“Evaluation is conducted on widely used LVLM benchmarks: ScienceQA (Lu et al., 2022), POPE (Li et al., 2023), MMBench (Liu et al., 2023a), MMMU (Yue et al., 2024), MME (Fu et al., 2023), GQA (Hudson & Man-ning, 2019), andMM-Vet (Yuet al., 2023).”
Standardized assessment of LLM English proficiency
Min, S., Wang, S., Gao, X., Wang, H., Jin, Z., Ling, C., & Ding, N. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100656
AILMIR: Agent and incremental learning-based multimodal information retrieval for complex multi-hop multimedia queries
Fan, Y., Yang, Z., Qin, H., & Wu, S. (2026) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-026-00413-y
Self-citation
Triplet Decomposition and Extensions: A General Framework for Parameter-Efficient Fine-Tuning
Du, Z., Min, Y., Zhang, D., Lü, K., Li, J., & Shen, H. T. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3720458
“We evaluate the ﬁne-tuned models on four multimodal benchmarks: MMMU [77], MMMU-Pro [78] and SciQA [79], which collectively cover scientiﬁc reasoning and multi-disciplinary university-level multimodal understanding.”
Zero-Shot Product Attribute Labeling with Vision-Language Models: A Three-Tier Evaluation Framework
Shukla, S., & Sonalkar, K. (2026) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw68408.2026.00169
“Comprehensive benchmarks like MMMU [20] have established rigorous evaluation standards for VLMs across 30 academic disciplines.”
EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs
Xie, Y., Han, K., Xiao, Y., Cui, H., Wu, G., Zhang, Z., Shu, K., Lu, J., Hu, X., & Yang, C. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817571 · Cited by 2
Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery
Yang, C., Lin, X., Li, S., Wang, W., Guo, R., Feng, F., & Chua, T. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817545 · Cited by 1
“Benchmarks such as Super-GLUE [29], MMLU [9], MMLU-Pro [30], MMMU [33], C-Eval [11], Xiezhi [8], GPQA [25], and HLE [24] typically construct multiple-choice or short-answer questions derived from textbooks, standardized exams, or academic papers.”
Learning Graph Rationales to Compress Long Chains of Thought in Multimodal Reasoning
Wang, Y., Yue, L., Wang, D.-B., Wei, T., & Zhang, M.-L. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817627
AutoDavis: Automatic and Dynamic Evaluation Protocol of Large Vision-Language Models on Visual Question-Answering
Bao, H., Huang, Y., Wang, Y., Ye, J., Wang, X., Chen, X., Zhao, Y., Zhou, T., Elhoseiny, M., & Zhang, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817506
“To address the major concern regarding examiner bias and circularity risk, we conducted a comprehensive ablation study us-ing three conditions: Baseline (Multi-Examiner) , LOEO (Leave-One-Examiner-Out) , and Single-Examiner .”
OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understanding
Tao, H., Huang, C., Wang, N., Lyu, H., Zhang, L., Ke, G., & Fang, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817510
“140 on MMMU [43], and +0 .”
Nonparametric domain adaptation for multimodal connoisseurship of Buddhist statuary across dynasties (ChronoStyleNet2.0)Preprint
Yu, Z., Xing, J., Zhao, L., Lei, D., Chen, L., Han, Y., & Ren, W. (2026) · Research Square · doi:10.21203/rs.3.rs-10211545/v1
Self-citation
Training-Free Video Corpus Moment Retrieval via Synergistic Collaboration and Adaptive Calibration
Zhao, J., Li, H., Zhang, Y., Wang, W., & Hua, C. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3723243
“This task plays a crucial role in multimodal understanding [6], [7], [8], [9], [10], [11], video retrieval [12], [13], [14], [15], [16], [17], and moment localization [18], [19], [20], [21], [22], [23].”
GDAS-Merge: Grouped Differentiable Adaptive Search for multimodal model merging
Gu, M., Wang, L., Li, M., Zhu, J., Yan, J., Wu, Q., & Zhang, M. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116848
Can VLMs Understand Handwritten Mathematical Documents?
Mitra, S., Mondal, A., & Jawahar, C. V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36039-7_37
Stroke-Level Connectivity Verification: Grounding Vision-Language Models Against Topology Hallucination in Diagram Understanding
Arean, A. I. H., Samin, N. H., Rahman, M. A., Suity, R. A., Noshin, J. A., & Rahman, M. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36023-6_19
Evaluating Multimodal Large Language Models for End-to-End Diagram Recognition
Burmester, M., Scharfenberger, J., & Funk, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36207-0_15
Concept Tree Learner (CTL): An Incremental and Interpretable Symbolic Framework for Binary String Rule Induction
Ertekin, M. T., & Genç, B. (2026) · Applied Sciences · doi:10.3390/app16178542
Identifying Entrance Accessibility in Subsidized Housing Using a Vision Language Model-Assisted Pipeline with a Chain of Refinement
Jang, S., Yong, G., Baek, F., Rollings, K. A., & Lee, S. H. (2026) · ASCE Construction Research Congress (CRC) · doi:10.1061/9780784487006.054
Evaluating Large Language Models’ Ability to Reason About Anatomical Spatial Relations
Tsourma, M., Blackwell, R. E., Cohn, A. G., Drosou, A., & Koubarakis, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36716-7_8
PercepTax: Benchmarking Physical Intelligence Through Cross-Property Reasoning in Vision-Language Models
Lee, J., Wang, X., Peng, J., Ye, L., Zheng, Z., Zhang, T., Wang, T., Ma, W., Chen, S., Chou, Y.-C., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37026-6_5
UniReflect: Self-reflection Tuning for Unified Multimodal Understanding and Generation
Wei, C., Huang, Z., Tan, H., Liang, D., Gao, L., Yan, P., & Wei, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37577-3_19
Self-citation
Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation
Zhang, Y., Peng, D., Guo, Z., Zhang, Z., Yang, X., Sun, T., Sun, S., Zhang, Y., Li, Y., Zhao, H., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37098-3_22 · Cited by 4
Self-citation
Preventing Expert Collapse in MoE-dVLMs via Modality-Wise Norm Alignment
Liu, Z., Cao, Z., Zhang, H., Yu, C., Niu, L., & Meng, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37359-5_20
ATOMIC: A Domain-Specific Vision-Language Model for Transmission Electron Microscopy
Tu, C.-R., Hsueh, H.-W., & Hsu, S.-H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37132-4_24
EmbedCopilot-Bench: Evaluating Vision-Language Models for Hardware-Aware Embedded System Development
Yuan, D., Deng, Y., & Chen, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37132-4_19
From Correct Answers to Usable Outputs: Layered Instruction Stability in Multimodal Large Language Models
Ruan, J. (2026) · International Conference on Computer Information Science and Application Technology (CISAT) · doi:10.1109/cisat71030.2026.11682121
“Other multimodal benchmarks, including VisIT-Bench, MMBench, MMMU, MM-Vet, and CrafText, evaluate realistic user requests, fine-grained capabilities, expert-level reasoning, and integrated multimodal skills [4,5,6,7,9].”
Circuit-MLLM: Topological Logic-Guided Latent-Space Visual Reasoning for Circuit Schematic Understanding
Deng, J., Jiang, Y., Huang, W., Li, X., Sun, Q., & Zhuo, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36846-1_22
“However, applying MLLMs to circuit schematics remains challenging due to their dense component layouts and intricate topological dependencies [20].”
Self-citation
Bidimensional Reinforcement Learning for Multimodal Diffusion Language Models
Li, B., Wang, Y., Zhang, Y., Liu, J., Guo, Y., Wu, J., Wang, M., & Yang, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37464-6_37
Learning from Primitive: Probing Visual Reasoning of LVLMs via Counting
Che, L., Xue, Z., Quan, Y., Liu, B., Shi, Z., Tang, R., Krishna, R., & Pavlović, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37023-5_37 · Cited by 4
SWIFT: Spatial-Window Integrated Frequency-Aware Token Pruning for Efficient MLLMs on Edge Devices
Liu, G., Chen, J., & Yu, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37553-7_3
CabinSI: Omni-Cabin Spatial Reasoning Through Explicit Visual Cognitive Maps
Qu, M., Hu, H., Ma, M., Lei, M., Gao, J., Ding, H., Zhao, Y., Wu, K., & Wei, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37718-0_6
Exploring Efficient Reasoning Segmentation with Small Language Models
Wen, C., Peng, Z., Huang, Y., Yang, X., & Shen, W. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37574-2_32
DIGS: Differentiable, Incremental, Global, Scalable Pruning for Language Models
Li, B., Xian, J., Jiang, T., Lin, J., Zou, C., & Zhang, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37235-2_11
Self-citation
VoxAnchor: Explicit Voxel-Semantic Grounding for Spatial Understanding in Videos
Li, X., Long, T., Zhou, J., Zeng, C., Chen, J., Yang, J., & Xie, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37255-0_19
SpecV: Specification Verification for Robust Unified Multimodal Evaluation
Yu, W., Fang, R., Cai, Y., Huang, L., Yang, Y., Zhuang, X., Lin, J., Yuan, Y., & Bai, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37029-7_5
HERO: Enhancing Multimodal Faithfulness via Dynamic Entropy-Aware Reinforcement Learning
Yang, X., Zhang, Q., Zhang, Y., Lin, K.-Y., Hu, J., Jiang, D., & Zheng, W.-S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37624-4_25
Self-citation
Evaluating BLIP-2 for Fine-Grained Fashion Attribute Annotation: Prompt-Based Label Mapping and Error Taxonomy
Song, Y., & Du, B. (2026) · International Conference on Image, Vision and Computing (ICIVC) · doi:10.1109/icivc70586.2026.11686736
“More broadly, multimodal benchmarks such as MMBench [11], SEED-Bench [12], and MMMU [13], together with recent failure-oriented evaluation such as ODE [14], show the value of structured and diagnostic assessment beyond fluent free-form outputs.”
Intuitions of compromise: Utilitarianism, egalitarianism, contractualism
Moore, J., Choi, Y., & Levine, S. (2026) · Cognition · doi:10.1016/j.cognition.2026.106694
Enem-VL: A Multimodal Benchmark of Brazilian Exams for Evaluating Vision-Language Models
Soares, A. L. B., Ferreira, L. N., Leme, G. M., Scatolin, M. F., Interian, R., & Dias, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39882-6_21
Advancing Scientific Chart Understanding: The SCI-CQA Benchmark and Beyond
Shen, L., Qigqi, Ding, K., Xu, J. H., Meng, G., & Xiang, S. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3725719
“Existing benchmarks such as MMBench [13], MME [14], MMMU [15], and POPE [16] cover diverse tasks, including video understanding [17], high-resolution images [18], visual localization and perception [19], medical image analysis [20], and related domains.”
LightFusion: A Lightweight Double-Fusion Framework for Unified Multimodal Understanding and Generation
Wang, Z., Chen, Z., Gou, C., Li, F., Deng, C., Zhu, D., Li, K., Yu, W., Tu, H., Fan, H., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37135-5_23 · Cited by 16
MANGO: Unleashing Image Generation Capability of Unified Multimodal Models
Wang, Y., Liu, M., He, W., Yuan, H., Huang, Z., Zhang, G., Fang, W., Jiang, H., Zhang, S., Dai, W., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37520-9_10
Self-citation
Vision-Language Models Evaluation for Telecom Field Safety Auditing
Grčić, N., Ribičić, I., Romić, K., Lenard, D. B., & Leventić, H. (2026) · International Symposium ELMAR (ELMAR) · doi:10.1109/elmar71231.2026.11712238
“General VLM benchmarks (MMBench [11], MMMU [12]) target broad capability assessment, but domain-specific benchmarks for telecommunications field auditing remain sparse; to our knowledge, none combine a multi-requirement checklist with real and synthetic imagery.”
Learning Consistency in Reward Modeling for Multi-modal Reasoning
Li, C., Ni, B., Zhang, B., Zhang, J., Ye, K., Peng, H., Hu, H., & Zheng, N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37258-1_16
MapQA: A Map-Question-Answering Benchmark for Visual Language Model Reasoning
Arnold, C., Alini, A., Alabdulkareem, A., Wang, J., Feenstra, P., Arnold, C., DeWitt, J. K., Ritsema, N. C., Yae, J. H., Katz, B., et al. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831218

Authors: Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., & Chen, W.
Affiliations: IN. AI Research
University of Waterloo (CA)
Independent
The Ohio State University (US)
Carnegie Mellon University (US)
Princeton University
University of Victoria (CA)
ORCID: Zhang, Kai
Zhang, Ge
Stevens, Samuel
Jiang, Dongfu
Ren, Weiming
Sun, Yuxuan
Wei, Cong
Yuan, Ruibin
Zheng, Boyuan
Liu, Yibo
Huang, Wenhao
Sun, Huan
Su, Yu
Chen, Wenhu
Venue: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher: IEEE
Published: 2024-06-16
Pages: 9556-9567
ISSN: 1063-6919 (ISSN-L)
DOI: 10.1109/cvpr52733.2024.00913
OpenAlex: W4402716477
Semantic Scholar: b50d19c5c298f6562c3b3c6c3822a351bdc89260
Cited by: 429
References: 80
Citations by year: 2026: 213 · 2025: 175 · 2024: 41
FWCI: 88.55
Type: conference paper
Language: en
Open access: green
arxiv.org
Licence: doi.org/10.15223/policy-029 (stm-asf)
doi.org/10.15223/policy-037 (stm-asf)
Locations: IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Topics: Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords: multimodal benchmark, large language models, domain knowledge, visual reasoning, charts and diagrams, GPT-4V, multimodal foundation models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks