Cited by 445Multimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot LearningAdvanced Neural Network Applications

The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.

MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Open access

Yue, Xiang, Ni, Yuansheng, Zheng, Tianyu, Zhang, Kai, Liu, Ruoqi, Zhang, Ge, Stevens, Samuel, Jiang, Dongfu, Ren, Weiming, Sun, Yuxuan, Wei, Cong, Yu, Botao, Yuan, Ruibin, Sun, Renliang, Yin, Ming, Zheng, Boyuan, Yang, Zhenzhu, Liu, Yibo, Huang, Wenhao, Sun, Huan, Su, Yu, Chen, Wenhu·2024·2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

DOI 10.1109/cvpr52733.2024.00913

Open this paper in CoChat — free account Open PDF

Abstract

We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and text-books, covering six core disciplines: Art & Design, Busi-ness, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly het-erogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. Unlike existing benchmarks, MMMU focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of 28 open-source LMMs as well as the propri-etary GPT-4V(ision) and Gemini highlights the substantial challenges posed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieve accuracies of 56% and 59% respectively, indicating significant room for improvement. We believe MMMU will stimulate the community to build next-generation multimodal foundation models towards expert artificial general intelligence.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1109/cvpr52733.2024.00913

Cites

88 of 88 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“This work relies on pre-trained visual representations like Faster RCNN features [61] to minimize the training sample complexity.”
Method
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3×
“Later on, CLIP [60], ALIGN [27], SimVLM [71], CoCa [78], Flamingo [2], BLIP-2 [32], and Fuyu [6] (inter alia) have been proposed to train visual representation using ViT [18] from scratch with massive amount of web data.”
BackgroundMethod
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 3×
“Rapid advances in large language models (LLMs) [13, 54, 67] have sparked broad discussions on the controversial concept of artificial general intelligence (AGI), often used to describe AI systems that perform on par or surpass humans at most tasks [1, 7, 21, 29, 49, 52].”
BackgroundMethod
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“Rapid advances in large language models (LLMs) [13, 54, 67] have sparked broad discussions on the controversial concept of artificial general intelligence (AGI), often used to describe AI systems that perform on par or surpass humans at most tasks [1, 7, 21, 29, 49, 52].”
Background
Visual instruction tuning
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · cited 1×
“LXMERT [64], UNITER [10], VinVL [80], Oscar [34], VilBert [45], and VLP [86] are among the earliest work to train universal vision-language models to tackle many multimodal tasks.”
Background

Flagged first, then by citations

Show all 80 references

Research works printed but not linked: 19

Arcas, 2023, Artificial general intelligence is already here, Noema Magazine
Bavishi, 2023, Introducing our multimodal models
2024, GitHub Repository
Chiang, 2023, Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
Dosovitskiy, 2021, An image is worth 16×16 words: Transformers for image recognition at scale
Team, 2024, Adept fuyu-heavy: A new multimodal model
Li, 2023, Seed-bench: Benchmarking multimodalllms with generative comprehension, arXiv preprint
Li, 2023, Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models
Li, 2023, M3it: A large-scale dataset towards multimodal multilingual instruction tuning, arXiv preprint
Liu, 2023, Aligning large multimodal model with robust instruction tuning, arXiv preprint
Liu, 2024, Llava-next: Improved reasoning, ocr, and world knowledge
2024, Minicpm-v
2024, Ominilmm-12b
2023, Gpt-4v(ision) system card, OpenAI
Ormazabal, 2024, Reka flash: An efficient and capable multimodal language model
2023, Qwen-vl-plus
2024, Qwen-vl-max
2024, Introducing the next generation of claude
2024, Infimm: Advancing multimodal understanding from flamingos legacy through diverse llm integration, InfiMM Team.

Sorted by citations

Items 201–300 of 445

Mavors: Multi-granularity Video Representation for Multimodal Large Language Model
Shi, Y., Liu, J., Guan, Y., Wu, Z., Zhang, Y., Wang, Z., Lin, W., Hua, J., Wang, Z., Chen, X., et al. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755363 · Cited by 1
Remember, Retrieve and Generate: Understanding Infinite Visual Concepts as Your Personalized AssistantPreprint
Hao, H., Han, J., Li, C., Li, Y.-F., & Yue, X. (2024) · Qeios · doi:10.32388/95w7kc · Cited by 2
Self-citation
Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ DocumentsPreprint
Chen, J., Xu, D., Fei, J., Feng, C.-M., & Elhoseiny, M. (2024) · Qeios · doi:10.32388/ajmacy · Cited by 1
iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokensin Large Multimodal ModelsPreprint
Hu, L., Shang, F., Wan, L., & Feng, W. (2024) · Qeios · doi:10.32388/69kflk · Cited by 1
PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
Lin, Z., Xie, Y., Qu, B., Wang, H., Li, J., Wu, H., Dong, Y., Yang, Z., Zhu, J., Lu, H., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24957
“…(Antol et al. 2015; Hudson et al. 2019; Y. Zhu et al. 2016) test scene understanding through question answering, while recent holistic benchmarks (Yue, Ni, et al. 2024; Yuan Liu et al. 2024; Lin Chen et al. 2024; Yue, Zheng, et al. 2025) extend this paradigm to college-level reasoning and…”
Self-citation
Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data
Arviv, O., Greenewald, K., Perlitz, Y., Mulian, H., Shmueli-Scheuer, M., & Choshen, L. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.43
Does Spatial Cognition Emerge in Frontier Models?
Ramakrishnan, S. K., Wijmans, E., Kraehenbuehl, P., & Koltun, V. (2024) · arXiv · doi:10.48550/arxiv.2410.06468 · Cited by 49
S$$^3$$: A Simple Strong Sample-Effective Multimodal Dialog System
Rykov, E., Malkershin, E., & Panchenko, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70239-6_17
What is the True Performance of Large Multimodal Models in Visual Context-Based Mathematical Reasoning? An Analysis of Multiple Datasets and Future Research Directions
Park, J., Park, S., Jeon, H., & Lim, J. (2024) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc62082.2024.10827294
Evaluating Large Language Models in Cybersecurity Knowledge with Cisco Certificates
Keppler, G., Kunz, J., Hagenmeyer, V., & Elbez, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-79007-2_12
Are Vision-Language Models Truly Understanding Multi-vision Sensor?Preprint
Chung, S., Yu, Y., Chee, Y., Kim, S. Y., Lee, B., & Ro, Y. M. (2025) · Qeios · doi:10.32388/vn42c2
Enhancing Binary Neural Network with Response Deep Supervision
Yang, Y., Liu, J., Ma, H., Zhao, Y., Ma, Z., & Ma, L. (2024) · China Automation Congress (CAC) · doi:10.1109/cac63892.2024.10864592
Detecting Cyberbullying in Visual Content: A Large Vision-Language Model Approach
Mu, J., Cong, D., Qin, H., Ajay, I., Guo, K., Vishwamitra, N., & Hu, H. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00257
Disambiguating Ambiguous Questions using Eye-Gaze in Visual Question Answering
Inadumi, S., Kawano, S., Yuguchi, A., Kawanishi, Y., & Yoshino, K. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.3
An Approach to Complex Visual Data Interpretation with Vision-Language Models
Nguyen, T., Huynh, V.-T., Nguyen, V.-L., & Tran, M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-2641-0_23
Grounding Image Understanding to Oil and Gas Product Manuals: Refining LLaVA through Contextual Instruction Tuning
Wang, H., & Benslimane, S. (2025) · doi:10.1109/aixmm62960.2025.00018
ChipVQA: Benchmarking Visual Language Models for Chip Design
Yang, H., Huang, Q., Pinckney, N., Turner, W. J., Zhou, W., Zhang, Y., Ho, C.-T., Chang, C.-C., & Ren, H. M. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992791
VisCon-100K: Leveraging Contextual Web Data for Fine-Tuning Vision Language Models
Kumar, G. K., Chaabane, I., & Wu, K. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8298-0_31
A Multilingual Multimodal Medical Examination Dataset for Visual Question Answering in Healthcare
Riccio, G., Romano, A., Barone, M., Orlando, G. M., Russo, D., Postiglione, M., Gatta, V. L., & Moscato, V. (2025) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms65348.2025.00093
HTAN: A Hierarchical Table-Aware Network for Complex Table Visual Understanding
Wu, S., Wang, R., Zhang, M., Wang, Z., Gan, X., & Wu, Y. (2025) · IEEE International Conference on Pattern Recognition, Machine Vision and Artificial Intelligence (PRMVAI) · doi:10.1109/prmvai65741.2025.11108551
EMMU: Efficient Information-Level Multimodal Machine Unlearning with High Model Fidelity
Zhang, J., Hou, J., Xiao, T., Huang, Y., & Li, X.-Y. (2025) · International Workshop on Quality of Service · doi:10.1109/iwqos65803.2025.11143267
TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos
Charoenpitaks, K., Nguyen, V.-Q., Suganuma, M., Arai, K., Totsuka, S., Ino, H., & Okatani, T. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00230 · Cited by 2
RadFig-VQA: A Multi-imaging-Modality Radiology Benchmark for Evaluating Vision-Language Models in Clinical Practice
Yamagishi, Y., Hanaoka, S., Nakamura, Y., Kikuchi, T., Shimizu, A., Yoshikawa, T., & Abe, O. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06004-4_23
HW-MLVQA: a novel handwritten multilingual dataset for visual question answering and evaluation
Pal, A., Mondal, A., & Jawahar, C. V. (2025) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-025-00560-8
reVISION: A Polish Benchmark for Evaluating Vision-Language Models on Multimodal National Exam Data
Ciesiółka, M., & Graliński, F. (2025) · Annals of Computer Science and Information Systems · doi:10.15439/2025f2608
Specify Privacy Yourself: Assessing Inference-Time Personalized Privacy Preservation Ability of Large Vision-Language Models
Wang, X., Yi, X.-Y., Xie, X., & Jia, J. (2025) · ACM Multimedia · doi:10.1145/3746027.3758156 · Cited by 2
Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security
Dai, M., Liu, S., Zhao, Z., Gao, J., Sun, H. S., & Li, X. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755236
Self-citation
LAVA Grand Challenge 2025: Benchmarking Japanese-English Document Understanding with Large Vision-Language Models
Sato, D., Vo, D. M., Salam, K. M. A., Shoji, H., Matsuoka, Y., Taniguchi, T., Baba, K., & Nakayama, H. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3761994
Gating-TinyLLaVA: A Compact Multimodal Model with Dual Visual Encoders
Zhang, X., Wu, Y., Chen, L., & Chen, X. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4384-7_1
FinMR: A Knowledge-Intensive Multimodal Benchmark for Advanced Financial Reasoning
Deng, S., Peng, H., Xu, J.-C., Mao, R., Giurcăneanu, C., & Liu, J. (2025) · International Conference on AI in Finance · doi:10.1145/3768292.3770365 · Cited by 6
DLLaVA: A Novel Multimodal Architecture with Enhanced Vision Encoder and Curriculum Learning
Mao, Y., & Zhou, J. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228908
Zero-shot Artistic Text Recognition with Multimodal Language Models
Do, T., Tran, T., Le, D.-D., & Ngo, T. D. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249117
Künstliche Intelligenz zur Verbesserung der Zugänglichkeit von Business-Graphiken für Menschen mit Sehbeeinträchtigungen
Hölzl, S., & Langenbach, C. (2025) · HMD Praxis der Wirtschaftsinformatik · doi:10.1365/s40702-025-01222-3
Boosting Multi-Modal Large Language Model With Enhanced Visual Features
Ma, Y., Lin, W., Wang, Z., Ji, J., Sun, X., Lin, C., & Ji, R. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3644851
Dual Thinking and Logical Processing in Human Vision and Multimodal Large Language Models
Dayanandan, K., Kumar, N., Sinha, A., & Lall, B. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3606452
“…in ImageNet-a confuse deep learning models but not human vision [39]; (d) example from synthetically generated IllusionBench dataset [40]; (e) example from MMMU dataset for MLLMs [41]; and (f) example from WHOOPS dataset showing Einstein using a phone that was not available during his time [42].”
Vowel Articulation and Tongue Position in Vision Language Models
Sakajo, H., Sakai, Y., Kamigaito, H., & Watanabe, T. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.859
Research Process for JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark
Miyai, A. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.983
KnowBench: Evaluating the Knowledge Alignment on Large Visual Language Models
Ma, Z., Yang, H.-T., Zhang, J.-B., & Chen, J.-J. (2025) · Journal of Computer Science and Technology · doi:10.1007/s11390-025-5512-y
EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning
Kulkarni, Y., & Fazli, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37627-5_35
Vision-Language Memory for Spatial Reasoning
Liu, Z., Du, Y., Fu, T., Su, S., Ho, C., & Wang, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37356-4_17 · Cited by 6
MME-VirtualWorld: Simplifying Multimodal Assessment via Programmable Synthetic Benchmarking
Liu, Z., Chen, L., Chen, S., He, X., Fu, A., Liu, J., & Feng, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5761-5_28
A Unified View of Multimodal Emotion Recognition Systems
Subhashini, N., Baby, V., & Akhil, K. (2025) · IEEE International Conference on ICT in Business Industry & Government (ICTBIG) · doi:10.1109/ictbig68706.2025.11323715
From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion
Chen, C., Guo, Y., Zeng, P., Song, J., Di, P., Yu, H., & Gao, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37167-6_35
Enhancing Technical Question Answering Quality Through Multimodal Document Segmentation
Lvov, D., Smirnov, I., Volokha, V., Laushkina, A., & Boukhanovsky, A. V. (2026) · IEEE Access · doi:10.1109/access.2026.3655813
Multimodal Augmented Advantage Policy Optimization Method and Its Application
Qian, T., Li, C., & Zuo, G. (2025) · International Conference on Intelligent Medicine and Health (ICIMH) · doi:10.1145/3784929.3784931
Conformal Prediction in the Age of Multimodal Foundation Models: A Survey
Stavan, C., Tilaganji, K., Krishnan, S. M., Kancheti, S. S., & Balasubramanian, V. N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15120-9_13
The Ten Minutes That Shocked the World—Teaching Generative AI to Analyze the Trump–Zelensky Multimodal Debate
Poggi, I., Scaramella, T., Violini, S., Careri, S., Epure, M. D., & Dragoni, D. (2026) · Information · doi:10.3390/info17020136
Evaluating and Mitigating Relationship Hallucinations in Large Vision-Language Models
Wu, M.-K., Li, J.-L., Ji, J.-Y., Hao, F., Sun, X.-S., & Ji, R.-R. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3656175 · Cited by 5
“There are lots of benchmarks [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44] for evaluating LVLMs.”
Enriching Process-Related UI Logs Via Screenshot-Based Activity Labeling by Using Vision-Language Models
Rodríguez-Ruíz, A., Martínez-Rojas, A., Enríquez, J. G., Ramirez, A. J., & Agostinelli, S. (2026) · Business & Information Systems Engineering · doi:10.1007/s12599-026-00990-6
“Furthermore, the best-performing model in the MMMU benchmark (Yue et al. 2024), timed out while giving responses, taking more than 1,500 s to generate a response and exceeding the maximum number of output tokens set before ﬁnishing its reasoning, preventing us from fairly evaluating its performance.”
RLVPR: Process-level Supervision for Thinking with images in Multimodal Language Models
Tian, X., Zhang, Y., Fu, T., & Xie, X. (2025) · International Conference on Artificial Intelligence and Computer Engineering (ICAICE) · doi:10.1109/icaice68195.2025.11382399
Standardized Assessment of LLM English ProficiencyPreprint
Min, S., Wang, S., Gao, X., Wang, H., Jin, Z., Ling, C., & Ding, N. (2026) · Research Square · doi:10.21203/rs.3.rs-8820245/v1
Evaluating Variance in Visual Question Answering Benchmarks
Nikitha, S. R. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00442 · Cited by 2
MoPEQ: Mixture of Mixed Precision Quantized Experts
Chitty-Venkata, K. T., Ye, J., & Emani, M. K. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00424
EVLM: Self-reflective Multimodal Reasoning and KTO Alignment for Cross-Dimensional Visual Editing
Khalid, U., Munir, K., Iqbal, H., Farooq, A., Hua, J., Rahnavard, N., Chen, C., Zhu, V., & Ji, Z. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-16893-1_8
SEC : Enabling MLLMs for Low-Latency IoT Video Analysis via Semantic-Aware Edge–Cloud Collaboration
Yang, M., Tian, Y., Cong, P., Zhang, L., Xi, G., Wang, S., & Wang, W. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3669144
“Toassess whether these e ﬃ ciency gains come at the cost of semantic ﬁdelity, we evaluate S EC on three widely adopted multimodal benchmarks: MME [83], MMVet [84], and MMMU [85].”
Establishing benchmarks for large language models in nuclear engineering: A NucBench evaluation of GPT-4.1, Claude 3.7 Sonnet, and Gemini 2.5 Pro
Khuwaileh, B. A., & Matesha, P. (2026) · Annals of Nuclear Energy · doi:10.1016/j.anucene.2026.112223 · Cited by 1
SHC: Deeply Activating Human-Like Cognitive Ability for Visual Question Answering
Wang, F., Wang, Z., An, G., Lang, C., & Wu, D. O. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3673273
“We selected several representative benchmarks: VQA-v2, GQA, ScienceQA [94], Text-VQA [95] and VizWiz [89], as well as multiple multimodal evaluation benchmarks, including MMB [96], MME [97], MM-Vet [98], MMMU [99] and POPE [100].”
EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generation
Li, Y., Liao, N., Zhao, X., Zhang, S., Wang, X., Yang, Y., Yan, J., & Yang, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37553-7_11 · Cited by 2
GRADE: Benchmarking Discipline-Informed Reasoning in Image Editing
Liu, M., Fan, Z., Wang, Z., Gu, L., Zhu, Z., He, Y., Yang, Y., Tian, C., Zhao, X., Liao, N., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37435-6_29 · Cited by 8
Performance Comparison of Multimodal Vision-Language Models in Classifying Turkish Dishes
Bıçakçı, Y. S. (2026) · Sakarya University Journal of Computer and Information Sciences · doi:10.35377/saucis...1727583
A Thai Industrial Certification E-license Verification Framework using Vision LLMs
Soongswang, K., Buranasaksee, U., & Tangsombatvichit, P. (2025) · IEEE International Conference on Information, Communication and Networks (ICICN) · doi:10.1109/icicn67355.2025.11430509
Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation
Barrios, W., & Jin, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37020-4_25
LearnPhy: Progressive Hints to Incentivize Model Learning for Olympiad Physics
Li, H., Zou, Y., Lin, H., Qian, S., & Zhang, J. (2025) · doi:10.1145/3788149.3788207
Chain-of-Cognition Through Multi-Agent Collaboration Against Cross-Modal Cognitive Interference
He, L., Li, Z.-C., Shen, M.-J., Wang, P., & Zhang, L.-F. (2026) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslpro.2026.3675798 · Cited by 1
ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models
Min, T. D., Roy, S., Lathuilière, S., Ricci, E., & Mancini, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37235-2_30
Is Monitoring Enough? Strategic Agent Selection For Stealthy Attack in Multi-agent Discussions
Xiang, Q., Qu, H., Rahmani, H., & Liu, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37432-5_24 · Cited by 2
Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs
Narnaware, V., Gupta, A., Zhai, K., Wang, Z., & Shah, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37447-9_23 · Cited by 1
Pioneering a DeepSeek R1-Generated UML Dataset and an Automated Multimodal Visual Validation Framework
Nguyen, V. T., Nguyen, H.-K., Nguyen, K.-S., Luong, T. M.-H., Nguyen, T.-V., Nguyen, H.-C., & Vu, D.-Q. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-18159-6_37
A Comparative Study of Object and Sequence Diagrams in Automated UML Understanding
Nguyen, V.-V., Bui, A.-T., Nguyen, H.-K., Nguyen, K.-S., Luong, T. M.-H., Vu, D.-Q., Nguyen, T.-V., & Bui, Q.-A. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-21631-1_28
Foundation Vision-Language models
Laurençon, H. (2025) · theses.fr (ABES) · doi:10.70675/ec0311a0zb6a4z4249zb9fbz01bf51a02c7c
A Causal Lens on Non-RGB Vision Sensor Understanding in Vision-Language Models
Yu, Y., & Ro, Y. M. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3675431
Shape vs. Context: Examining Human--AI Gaps in Ambiguous Japanese Character Recognition
Haraguchi, D. (2026) · CHI Extended Abstracts · doi:10.1145/3772363.3798606
From Image to Pixels: Towards Fine-Grained Medical Vision-Language Models
Shen, L., Huang, X., Shang, F., Zhang, X., Yang, Y., Fan, B., & Xiang, S. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3682684
PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading
Ravishankara, M. (2026) · IEEE SoutheastCon · doi:10.1109/southeastcon63549.2026.11476291
“More recent MLLM-focused suites (e.g., ChartBench) broaden chart types and question styles for chart comprehension evaluation [5], while general multimodal reasoning benchmarks (e.g., Math-Vista and MMMU) include charts among many heterogeneous modalities and tasks [6], [7].”
Generative Reasoning Meets Multimodal Evaluation: A Novel Approach to Activity and Package Diagram Comparison in UML Modeling
Nguyen, V.-V., Nguyen, H.-K., Nguyen, K.-S., Luong, T. M.-H., Bui, A.-T., Vu, D.-Q., & Nguyen, T.-V. (2026) · Springer eBooks · doi:10.1007/978-3-032-19488-6_7
iMathBench: Is Your Multi-Modal Large Language Model Ready to Solve Mathematical Problems Embedded in Images?
Guo, J., Jiang, X., Wang, G., Lu, R., & Tang, S. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464297
Towards Accurate Quantization for Large Vision-Language Models via Zeroth-Order Gradient Optimization and Sectioned Logarithmic Quantizer
Zhang, J., Chen, J. Y., Guo, X., Tong, X., & Huang, D. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460568
SVCF: Enabling Zero-Shot Correction of Reasoning Steps in Multi-Modal Large Language Models
Jiang, B.-Y., Huang, T.-X., Yang, Y., Zhang, Y., Duan, G., & He, T. (2026) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp55912.2026.11462050
ZIP Your Data: Length-Adaptive Visual Token Optimization for Efficient Multi-Modal Training
Sun, N., Wu, F., Zhang, Y., Chen, S., Tao, X., & Yang, F. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461954
ECM: Enhancing Compressibility of Quantized Vision Encoder and LLM for Large Vision-Language Models
Wang, W., Mao, Y., Tang, D., Guan, N., & Xue, C. J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460785
Enrich Visual Features by Holistic Sampling and Hierarchical Condensing in Multimodal Large Language Models
Bai, Y., Bai, S., 冉健莉, Su, T., & Bai, Z. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463161
A survey on edge multimodal large models: Compression, inference acceleration, and applications
Chen, S., Shu, Y., Wang, C., & Chen, J. (2026) · National Science Open · doi:10.1360/nso/20260016
Creation-Mmbench: Assessing Context-Aware Creative Intelligence in Mllms
Fang, X., Chen, Z., Lan, K., Ma, L., Ding, S., Liang, Y., Zhao, X., Wen, F., Zhang, Z., Zhang, G., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00049
Self-citation
Zero-Shot Composed Image Retrieval via Dual-Stream Instruction-Aware Distillation
Zhong, W., Barton, R., An, W., Jiang, F., Ma, H., Guo, Y., Dan, A., Sam, S., Bouyarmane, K., & Huang, J. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02063
UINavBench: A Framework for Comprehensive Evaluation of Interactive Digital Agents
Agrawal, H., Schoop, E., Pan, X., Mahajan, A., Seff, A., Feng, D., Cheng, R., y Terán, A. R. M., Gomez, E., Sundararajan, A., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02168
RMultiplex200K: Toward Reliable Multimodal Process Supervision for Visual Language Models on Telecommunications
Chen, S., & Song, B. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00165
VisionMath: Vision-Form Mathematical Problem-Solving
Ma, Z., Chen, Y., Zhang, Z., Oi, Z., Yuan, C., Zhu, S., Zhuo, C., Li, B., Liu, Y., Li, Z., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00116
Self-citation
Layer-Wise Vision Injection With Disentangled Attention for Efficient LVLMs
Zhang, X., Li, D., Liu, B., Bao, Z., Zhou, Y., Yang, B., Liu, Z., Zhong, Y., & Yuan, T. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00658
VideoSetDiff: Identifying and Reasoning Similarities and Differences in Similar Videos
Qiu, Y., Sun, Y., Yagi, T., Egami, S., Miyata, N., Fukuda, K., Hara, K., & Sagawa, R. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01138
Self-citation
MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning
Gao, T., Fu, Y., Wu, W., Yue, H., Liu, S., & Zhang, G. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00146
Self-citation
VLM-Guard : Defending Jailbreaks by Monitoring Only Hundreds of Safety-Critical Neurons
Hu, J., Zhou, J., Xie, M., Yang, Z., Li, J. W., Zheng, H., Shi, J., He, D., & Li, Y. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3690008
Self-citation
GHOST: Getting to the Bottom of Hallucinations with A Multi-round Consistency Benchmark
Vibashan, V. S., Chang, N., Schmalfuß, J., Patel, V. M., Yu, Z., & Álvarez, J. M. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00596
Seeing and Understanding: A Human-Centric Evaluation of Multimodal Language Models in Chinese Contexts
Li, J. J., Jiang, Z. J., Liu, Y., & Xu, H. (2025) · International Conference on Human-Engaged Computing (ICHEC) · doi:10.1145/3786995.3787004
Self-citation
Multimodal Validation in UML Synthesis: A Dual-Case Study on Sequence and Class Diagram Generation Pipelines
Nguyen, V.-V., Nguyen, H.-K., Nguyen, K.-S., Thi, M.-H. L., Bui, A.-T., Nguyen, T.-V., & Vu, D.-Q. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-15839-0_37
Context-Aware Visual Multi-turn Conversation Generation from Wikipedia and Wikidata
Shbita, B., Li, P., & Gentile, A. L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-25159-6_19
A Vision-Language Fusion Framework for Ethnic Minority Costume Image Captioning
Zhang, J., Hu, K., Huang, Y., & Zhang, D. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7084-3_21
Reformulation is All You Need: Addressing Malicious Textual Features in DNNs
Jiang, Y., Ma, O., Yang, Y., Zhang, T., & Ji, S. (2026) · Machine Intelligence Research · doi:10.1007/s11633-025-1594-9
Reinforcement learning-powered co-optimization: Bridging critic model and multimodal LLM reasoning abilities
Wang, Q., Liu, S., Zhang, Z., Du, J., Zhang, J., & Liu, Q. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113955
9. Artificial Intelligence, Accessibility and Inclusion
D’Alonzo, L., & Pozza (2026) · Open reports series · doi:10.11647/obp.0530.09
HumanEval-V: Systematic Evaluation of Visual Reasoning in Large Multimodal Models for Code Generation
Zhang, F., Wu, L., Bai, H., Lin, G., Li, X. F., Yu, X., Wang, Y., Chen, B., & Keung, J. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3813804 · Cited by 1
“In parallel, a variety of multimodal benchmarks, such as MMMU [87], MathVista [49], ChartQA [52], ARC-AGI [16], and RAVEN [88], have been proposed.”
Influential

Authors: Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., & Chen, W.
Affiliations: IN. AI Research
University of Waterloo (CA)
Independent
The Ohio State University (US)
Carnegie Mellon University (US)
Princeton University
University of Victoria (CA)
ORCID: Zhang, Kai
Zhang, Ge
Stevens, Samuel
Jiang, Dongfu
Ren, Weiming
Sun, Yuxuan
Wei, Cong
Yuan, Ruibin
Zheng, Boyuan
Liu, Yibo
Huang, Wenhao
Sun, Huan
Su, Yu
Chen, Wenhu
Venue: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher: IEEE
Published: 2024-06-16
Pages: 9556-9567
ISSN: 1063-6919 (ISSN-L)
DOI: 10.1109/cvpr52733.2024.00913
OpenAlex: W4402716477
Semantic Scholar: b50d19c5c298f6562c3b3c6c3822a351bdc89260
Cited by: 429
References: 80
Citations by year: 2026: 213 · 2025: 175 · 2024: 41
FWCI: 88.55
Type: conference paper
Language: en
Open access: green
arxiv.org
Licence: doi.org/10.15223/policy-029 (stm-asf)
doi.org/10.15223/policy-037 (stm-asf)
Locations: IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Topics: Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords: multimodal benchmark, large language models, domain knowledge, visual reasoning, charts and diagrams, GPT-4V, multimodal foundation models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks