The DOI matches its DataCite record.
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
Abstract
Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks, showing amazing emergent abilities in recent studies, such as writing poems based on an image. However, it is difficult for these case studies to fully reflect the performance of MLLM, lacking a comprehensive evaluation. In this paper, we fill in this blank, presenting the first comprehensive MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks. In order to avoid data leakage that may arise from direct use of public datasets for evaluation, the annotations of instruction-answer pairs are all manually designed. The concise instruction design allows us to fairly compare MLLMs, instead of struggling in prompt engineering. Besides, with such an instruction, we can also easily carry out quantitative statistics. A total of 30 advanced MLLMs are comprehensively evaluated on our MME, which not only suggests that existing MLLMs still have a large room for improvement, but also reveals the potential directions for the subsequent model optimization. The data are released at the project page https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MME serves as an extensive evaluative benchmark for MLLMs, aiming to assess their perceptual and cognitive aptitudes.
“MME [46] serves as an extensive evaluative benchmark, aiming to assess their perceptual and cognitive aptitudes.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Self-citationMM-LLMs: Recent Advances in MultiModal Large Language Models
Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., & Yu, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.738 · Cited by 260
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.16449 · Cited by 176
OCRBench: on the hidden mystery of OCR in large multimodal models
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., & Bai, X. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4235-6 · Cited by 167
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98
“TextVQA (VQA T ) (Singh et al., 2019) is a dataset based on pictures from Open Images (Kuznetsova et al., 2020) Comprehensively describing the visual input is another important skill desired in MLLMs.”
InfluentialMLVU: Benchmarking Multi-task Long Video Understanding
Zhou, J., Shu, Y., Zhao, B., Wu, B., Liang, Z., Xiao, S., Qin, M., Yang, X., Xiong, Y., Zhang, B., et al. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.01278 · Cited by 44
Hallucination of Multimodal Large Language Models: A Survey
Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., & Shou, M. Z. (2024) · arXiv · doi:10.48550/arxiv.2404.18930 · Cited by 38
Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression
Sarkar, S., Che, Y., Gavin, A., Beerel, P. A., & Kundu, S. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.631 · Cited by 4
Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
Fan, Z., Wang, Y., Polisetty, S., & Fung, Y. R. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.1037 · Cited by 4
RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics
Yuan, W., Duan, J., Blukis, V., Pumacay, W., Krishna, R., Murali, A., Mousavian, A., & Fox, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10721 · Cited by 7
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
Bui, N., Nguyen, H. T., Cohan, A., & Ying, R. (2026) · arXiv · doi:10.48550/arxiv.2605.09649
“We evaluate on three standard short-form VQA benchmarks: VQAText [Singh et al., 2019], MME [Fu et al., 2025a], and GQA [Hudson and Manning, 2019].”
Does Spatial Cognition Emerge in Frontier Models?
Ramakrishnan, S. K., Wijmans, E., Kraehenbuehl, P., & Koltun, V. (2024) · arXiv · doi:10.48550/arxiv.2410.06468 · Cited by 49
Enhancing Vision Foundation Models via Multimodal Continual Pre-Training
Chen, Y., Meng, L., Peng, W., Tao, J., Xu, C., Wu, Z., & Jiang, Y.-G. (2025) · arXiv · doi:10.48550/arxiv.2503.18931
Metadata
- Authors
- Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., Ji, R., Shan, C., & He, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- arXiv
- 2306.13394
- References
- 0
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
