PathVLM-Eval: Evaluation of open vision language models in histopathology
Open accessAbstract
The emerging trend of vision language models (VLMs) has introduced a new paradigm in artificial intelligence (AI). However, their evaluation has predominantly focused on general-purpose datasets, providing a limited understanding of their effectiveness in specialized domains. Medical imaging, particularly digital pathology, could significantly benefit from VLMs for histological interpretation and diagnosis, enabling pathologists to use a complementary tool for faster morecomprehensive reporting and efficient healthcare service. In this work, we are interested in benchmarking VLMs on histopathology image understanding. We present an extensive evaluation of recent VLMs on the PathMMU dataset, a domain-specific benchmark that includes subsets such as PubMed, SocialPath, and EduContent. These datasets feature diverse formats, notably multiple-choice questions (MCQs), designed to aid pathologists in diagnostic reasoning and support professional development initiatives in histopathology. Utilizing VLMEvalKit, a widely used open-source evaluation framework-we bring publicly available pathology datasets under a single evaluation umbrella, ensuring unbiased and contamination-free assessments of model performance. Our study conducts extensive zero-shot evaluations of more than 60 state-of-the-art VLMs, including LLaVA, Qwen-VL, Qwen2-VL, InternVL, Phi3, Llama3, MOLMO, and XComposer series, significantly expanding the range of evaluated models compared to prior literature. Among the tested models, Qwen2-VL-72B-Instruct achieved superior performance with an average score of 63.97% outperforming other models across all PathMMU subsets. We conclude that this extensive evaluation will serve as a valuable resource, fostering the development of next-generation VLMs for analyzing digital pathology images. Additionally, we have released the complete evaluation results on our leaderboard PathVLM-Eval: https://huggingface.co/spaces/gilalnauman/PathVLMs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In dermatopathology, VLMs can enable automatic slide annotation, region-of-interest highlighting, and natural language explanations of histopathologic findings.
“In dermatopathology, this can enable automatic slide annotation, region-of-interest highlighting, and natural language explanations of histopathologic findings [81].”
From Artificial Intelligence in Dermatopathology: An Update and Review of the Current Literature (2026) · cites this paper as [81]
The Introduction describes VLMs analyzing histological images, and the clinical-workflow section proposes draft diagnostic reports and case triage. Those passages support potential language-based reporting, but they do not address dermatopathology specifically or describe automatic slide annotation or region-of-interest highlighting. The supplied passages have no page numbers. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 39 · Cited by 7
Research works printed but not linked: 20
- [5]Bai J. Bai S. Yang S. Qwen-vl: a frontier large vision-language model with versatile abilities Retrieved from https://arxiv.org/abs/2308.12966 2023 · arXiv
- [7]Brown T. Mann B. Ryder N. Language models are few-shot learners. Advances in neural information process- ing systems 33 2020 1877 1901
- [8]Cai R. Song Z. Guan D. Benchlmm: benchmarking cross-style visual capability of large multimodal models European Conference on Computer Vision 2025 Springer 340 358
- [9]Chen L. Li J. Dong X. Sharegpt4v: improving large multi-modal models with better captions European Conference on Computer Vision 2025 Springer 370 387
- [16]Deitke M. Clark C. Lee S. Tripathi R. Yang Y. Park J.S. Salehi M. Muennighoff N. Lo K. Soldaini L. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language mod- els, in: Proceedings of the Computer Vision and Pattern Recognition Conference 2025 91 104
- [27]Huang M. Liu Y. Liang D. Jin L. Bai X. Mini-monkey: multi-scale adaptive cropping for multimodal large language models 2024 CoRR
- [29]Li B. Ge Y. Ge Y. Seed-bench: benchmarking multimodal large language models Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 13299 13308
- [31]Li J. Li D. Savarese S. Hoi S. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models International Conference on Machine Learning 2023 PMLR 19730 19742
- [32]Li Z. Yang B. Liu Q. Monkey: image resolution and text label are important things for large multi-modal models Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 26763 26773
- [38]Liu Y. Duan H. Zhang Y. Mmbench: is your multi-modal model an all-around player? European Conference on Computer Vision 2025 Springer 216 233
- [40]Liu Y. Yang B. Liu Q. Textmonkey: An OCR-Free Large Multimodal Model for Understanding Document Retrieved from https://arxiv.org/abs/2403.04473 2024 41587255 · arXiv
- [44]Mathew M. Karatzas D. Jawahar C. Docvqa: a dataset for vqa on document images Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 2021 2200 2209
- [48]OpenAI R. Gpt-4 technical report 2023 arxiv 2303.08774. View in Article 2 · arXiv
- [49]Qi L. Lv S. Li H. Dureadervis: a chinese dataset for open-domain document visual question answering Findings of the Association for Computational Linguistics 2022 2022 ACL 1338 1351
- [50]Radford A. Kim J.W. Hallacy C. Learning transferable visual models from natural language supervision International Conference on Machine Learning 2021 PMLR 8748 8763
- [52]Sun Y. Wu H. Zhu C. Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology European Conference on Computer Vision 2024 Springer 56 73
- [53]Team G Anil R. Borgeaud S. Gemini: a family of highly capable multimodal models Retrieved from https://arxiv.org/abs/2312.11805 2025 · arXiv
- [55]Vaswani A. Shazeer N. Parmar N. Attention is all you need. Advances in neural information processing systems 30 2017
- [57]Wang P. Bai S. Tan S. Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution Retrieved from https://arxiv.org/abs/2409.12191 2024 · arXiv
- [64]Zhang K. Li B. Zhang P. Lmms-eval: reality check on the evaluation of large multimodal models Rertrieved from https://arxiv.org/abs/2407.12772 2024 · arXiv
- Authors
- Gilal, N. U., Zegour, R., Al-Thelaya, K., Özer, E., Agus, M., Schneider, J., & Boughorbel, S.
- Affiliations
- Hamad bin Khalifa University (QA)
- Sidra Medical and Research Center (QA)
- Corresponding author
- Gilal, Nauman Ullah
- Venue
- Journal of Pathology Informatics
- Publisher
- Elsevier BV
- Published
- 2025-08
- Volume
- 18
- Pages
- 100455
- ISSN
- 2153-3539 (ISSN-L)
- PMC
- PMC12276438
- OpenAlex
- W4411050245
- Cited by
- 7
- References
- 39
- Citations by year
- 2026: 7
- FWCI
- 2.46
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Journal of Pathology Informatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Qatar National Library (HBKU-INT-VPR-TG-02-04)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- vision-language models, digital pathology, zero-shot evaluation, Qwen2-VL, LLaVA, model benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
