Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation
Open accessAbstract
Assessing long-form responses generated by Vision-Language Models (VLMs) is challenging. It not only requires checking whether the VLM follows the given instruction but also verifying whether the text output is properly grounded on the given image. Inspired by the recent approach of evaluating LMs with LMs, in this work, we propose to evaluate VLMs with VLMs. For this purpose, we present a new feedback dataset called the Perception Collection, encompassing 15K customized score rubrics that users might care about during assessment. Using the Perception Collection, we train Prometheus-Vision, the first open-source VLM evaluator model that can understand the user-defined score criteria during evaluation. Prometheus-Vision shows the highest Pearson correlation with human evaluators and GPT-4V among open-source models, showing its effectiveness for transparent and accessible evaluation of VLMs. We open-source our code, dataset, and model at https://github.com/kaistAI/prometheus-vision
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing reward approaches for GUI agents struggle to achieve both scalability and performance.
“Existing reward approaches struggle to achieve both scalability and performance.”
From OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards (2026) · cites this paper as Lee et al., 2024
LLM-as-a-judge offers a flexible and scalable paradigm by harnessing generalized reasoning capabilities and extensive world knowledge of foundation models.
“LLM-as-a-judge offers a flexible and scalable paradigm by harnessing the generalized reason-ing capabilities and extensive world knowledge of foundation models”
From OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards (2026) · cites this paper as Lee et al., 2024
Cites
Still being checked.
Cited by
Sorted by citations
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40 · see citing works
Personalized Preference Fine-tuning of Diffusion Models
Dang, M., Singh, A., Zhou, L., Ermon, S., & Song, J. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00751 · Cited by 5 · see citing works
MotIF: Motion Instruction Fine-Tuning
Hwang, M., Hejna, J., Sadigh, D., & Bisk, Y. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3527290 · Cited by 4 · see citing works
VisionArena: 230K Real World User-VLM Conversations with Preference Labels
Chou, C., Dunlap, L., Mashita, K., Mandal, K., Darrell, T., Stoica, I. G., Gonzalez, J. E., & Chiang, W.-L. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00367 · Cited by 3 · see citing works
Kim, Y., & Jung, J. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00062 · Cited by 1 · see citing works
Keith, B., Villegas, C. M., Matus, M., Castro, M. C., & Urrutia, D. (2025) · Electronics · doi:10.3390/electronics14214199 · Cited by 1 · see citing works
OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards
Li, Z., Wu, Z., Zhao, Y., Yang, B., Xie, J., Liu, Z., Liu, Z., Jin, K., Liang, J., Li, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2603.19191
“ing capabilities and extensive world knowledge of foundation models (Lee et al., 2024; Li et al., 2024; Zhou et al., 2025).”
FacGPT: An Effective and Efficient Method for Evaluating Knowledge-Based Visual Question Answering
Cheng, S., Zhang, S., Wu, J., Lan, M., & Sun, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9431-7_16
Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Charts
Kim, S. G., Choi, J. Y., Rossi, R. A., Koh, E., & Lee, T. Y. (2025) · IEEE Pacific Visualization Conference (PacificVis) · doi:10.1109/pacificvis64226.2025.00040
Self-citationCan Vision-Language Models See the World Like You Do?
He, Z., He, J., & Zhang, X. (2025) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss55030.2025.11313959
MedGemma-Critic: Fine-Tuning Medical Language Models for Domain-Specialised Text Evaluation
Liu, S.-R., & Kim, J. (2025) · International Conference on Digital Image Computing: Techniques and Applications · doi:10.1109/dicta68720.2025.11302446
Grounded Multi-modal Conversation for Zero-shot Visual Question Answering
Zarei, M. R., Akkasi, A., & Komeili, M. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343454
Understanding Generative AI Capabilities in Everyday Image Editing Tasks
Collins, B., Taesiri, M. R., Bolton, L., Lai, V. D., Dernoncourt, F., Bui, T., & Nguyen, A. T. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00221
From Sensor Data to Sequence of Scenes: Diffusion Model-Based Visualization of Human Activity
Matsuishi, K., Ukita, K., & Okita, T. (2026) · International Conference on Activity and Behavior Computing (ABC) · doi:10.1109/abc68169.2026.11567210
Semantic Browsing: Controllable Diversity for Image Generation
Dorfman, S., Vishnevsky, M., Dahary, O., Patashnik, O., & Cohen‐Or, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37602-2_24
AI Pictures and the Thousand-Word Problem
Stabauer, M., & Mittermayr, J. C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30047-8_15
H-Adapter: Pose-Robust Hairstyle Transfer via Attention-Derived, Source-Aligned Hair Masks
Jeong, S., Cho, Y., & 박상훈 (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37235-2_33
Exploring LMM-as-a-Judge for Image Harmonization Evaluation
Baek, J., & Noh, E. (2026) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr70517.2026.00016
- Authors
- Lee, S., Kim, S., Park, S. H., Kim, G., & Seo, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 11286-11315
- arXiv
- 2401.06591
- OpenAlex
- W4402670365
- Cited by
- 16
- References
- 0
- Citations by year
- 2026: 6 · 2025: 9 · 2024: 1
- FWCI
- 3.31
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- vision-language models, fine-grained evaluation, automated assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
