CoChat
Sign Up
PreprintCited by 22AI in cancer detectionRadiomics and Machine Learning in Medical ImagingArtificial Intelligence in Healthcare and Education

PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology

Open access
Shaikovski, George, Casson, Adam, Severson, Kristen, Zimmermann, Eric, Wang, Yi Kan, Kunz, Jeremy D., Retamero, Juan A., Oakley, Gerard +14 more·2024·arXiv
DOI 10.48550/arxiv.2405.10254arXiv 2405.10254
Open this paper in CoChat — free accountOpen PDF

Abstract

Foundation models in computational pathology promise to unlock the development of new clinical decision support systems and models for precision medicine. However, there is a mismatch between most clinical analysis, which is defined at the level of one or more whole slide images, and foundation models to date, which process the thousands of image tiles contained in a whole slide image separately. The requirement to train a network to aggregate information across a large number of tiles in multiple whole slide images limits these models' impact. In this work, we present a slide-level foundation model for H&E-stained histopathology, PRISM, that builds on Virchow tile embeddings and leverages clinical report text for pre-training. Using the tile embeddings, PRISM produces slide-level embeddings with the ability to generate clinical reports, resulting in several modes of use. Using text prompts, PRISM achieves zero-shot cancer detection and sub-typing performance approaching and surpassing that of a supervised aggregator model. Using the slide embeddings with linear classifiers, PRISM surpasses supervised aggregator models. Furthermore, we demonstrate that fine-tuning of the PRISM slide encoder yields label-efficient training for biomarker prediction, a task that typically suffers from low availability of training data; an aggregator initialized with PRISM and trained on as little as 10% of the training data can outperform a supervised baseline that uses all of the data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2405.10254

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Slide foundation models are developed with orders of magnitude fewer samples and limited optimization of SSL recipes compared to patch foundation models, leading to slide representations with restricted generalization capability.

    “Second, whereas current patch foundation models are trained with millions of histology image patches, slide foundation models are developed with orders of magnitude fewer samples and limited optimization of SSL recipes, leading to slide representations with restricted generalization capability58,62,82,83.”

    From A multimodal whole-slide foundation model for pathology (2025) · cites this paper as [62]

    The Related Work section says most pre-trained pathology models focus on tiles and identifies HIPT and LongViT as slide-level exceptions; it also notes that their self-supervised objectives lack clinical-report supervision. The introduction says slide-level aggregators trained from scratch can overfit small datasets. Neither passage establishes that slide foundation models generally use orders of magnitude fewer samples, have had limited optimization of SSL recipes, or consequently produce representations with restricted generalization; §4.1 reports that PRISM was trained on 587,196 whole-slide images. Page numbers were not available in our copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The pretrained weights of TITANV can serve as a good initialization for task-specific training.

    “This suggests that the pretrained weights of TITANV can serve as a good initialization for task-specific training, in line with previous works62,64.”

    From A multimodal whole-slide foundation model for pathology (2025) · cites this paper as [62]

    Section 5.2 reports that fine-tuning PRISM’s pretrained slide encoder outperformed training it from scratch in six of nine biomarker tasks. This supports the general point that pretrained weights can be a useful starting point for task-specific training, but the paper does not establish that result for TITANV. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • New types of foundation models have recently been proposed for encoding entire WSIs into slide-level general-purpose feature representations.

    “To overcome these limitations, new types of foundation models have recently been proposed for encoding entire WSIs into slide-level general-purpose feature representations57–72.”

    From A multimodal whole-slide foundation model for pathology (2025) · cites this paper as [62]

  • From cancer subtyping to molecular classification, TITAN consistently outperforms other state-of-the-art slide encoders such as PRISM, GigaPath and CHIEF.

    “From cancer subtyping to molecular classification, TITAN consistently outperforms other state-of-the-art slide encoders, such as PRISM62, GigaPath58 and CHIEF83.”

    From A multimodal whole-slide foundation model for pathology (2025) · cites this paper as [62]

  • PRISM uses the Perceiver architecture as the slide encoder, incorporating CoCa-based vision and language alignment on 195,344 specimen-report pairs comprising 587,196 WSIs.

    “PRISM62 uses the Perceiver architecture136 as the slide encoder, incorporating CoCa-based vision and language alignment95 on 195,344 specimen-report pairs, which comprise a total of 587,196 WSIs, each containing one or more WSIs.”

    From A multimodal whole-slide foundation model for pathology (2025) · cites this paper as [62]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 22

Research works printed but not linked: 44

  1. Josh Achiam et al. "Gpt-4 technical report". In: arXiv preprint arXiv:2303.08774 (2023). · arXiv
  2. Shekoofeh Azizi et al. "Robust and data-efficient generalization of self-supervised machine learning for diag- nostic imaging". In: Nature Biomedical Engineering 7.6 (2023), pp. 756-779.
  3. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. "Layer normalization". In: arXiv preprint arXiv:1607.06450 (2016). · arXiv
  4. Geert Berx and Frans Van Roy. "The E-cadherin/catenin complex: an important gatekeeper in breast cancer tumorigenesis and malignant progression". In: Breast Cancer Research 3 (2001), pp. 289-293. DOI: 10.1186/ bcr309. · DOI · Paper page
  5. Gustav Bredell et al. "Aggregation Model Hyperparameters Matter in Digital Pathology". In: arXiv preprint arXiv:2311.17804 (2023). · arXiv
  6. Gabriele Campanella et al. "Clinical-grade computational pathology using weakly supervised deep learning on whole slide images". In: Nature medicine 25.8 (2019), pp. 1301-1309.
  7. Gabriele Campanella et al. "Computational Pathology at Health System Scale-Self-Supervised Foundation Models from Three Billion Images". In: arXiv preprint arXiv:2310.07033 (2023). · arXiv
  8. Debyani Chakravarty et al. "OncoKB: a precision oncology knowledge base". In: JCO precision oncology 1 (2017), pp. 1-16.
  9. Richard J Chen et al. "Scaling vision transformers to gigapixel images via hierarchical self-supervised learning". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 16144- 16155.
  10. Richard J Chen et al. "Towards a general-purpose foundation model for computational pathology". In: Nature Medicine (2024), pp. 1-13.
  11. Diana T Cheng et al. "Memorial Sloan Kettering-Integrated Mutation Profiling of Actionable Cancer Targets (MSK-IMPACT): A Hybridization Capture-Based Next-Generation Sequencing Clinical Assay for Solid Tumor Molecular Oncology". In: Journal of Molecular Diagnostics 17.3 (2015), pp. 251-264. DOI: 10 . 1016 / j . jmoldx.2014.12.006. · DOI · Paper page
  12. Nicolas Coudray et al. "Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning". In: Nature medicine 24.10 (2018), pp. 1559-1567.
  13. Jiayu Ding et al. "Longnet: Scaling transformers to 1,000,000,000 tokens". In: arXiv preprint arXiv:2307.02486 (2023). · arXiv
  14. Jonas Dippel et al. "RudolfV: A Foundation Model by Pathologists for Pathologists". In: arXiv preprint arXiv:2401.04079 (2024). · arXiv
  15. Alexey Dosovitskiy et al. "An image is worth 16x16 words: Transformers for image recognition at scale". In: arXiv preprint arXiv:2010.11929 (2020). · arXiv
  16. Alexandre Filiot et al. "Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling". In: medRxiv (2023), pp. 2023-07.
  17. Jevgenij Gamper and Nasir Rajpoot. "Multiple instance captioning: Learning representations from histopathol- ogy textbooks and articles". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. 2021, pp. 16549-16559.
  18. Zhi Huang et al. "A visual-language foundation model for pathology image analysis using medical twitter". In: Nature medicine 29.9 (2023), pp. 2307-2316.
  19. Maximilian Ilse, Jakub Tomczak, and Max Welling. "Attention-based deep multiple instance learning". In: International conference on machine learning. PMLR. 2018, pp. 2127-2136.
  20. Andrew Jaegle et al. "Perceiver: General perception with iterative attention". In: International conference on machine learning. PMLR. 2021, pp. 4651-4664.
  21. Gregory P. Kalemkerian et al. "Molecular Testing Guideline for the Selection of Patients With Lung Cancer for Treatment With Targeted Tyrosine Kinase Inhibitors: American Society of Clinical Oncology Endorsement of the College of American Pathologists/International Association for the Study of Lung Cancer/Association for Molecular Pathology Clinical Practice Guideline Update". In: Journal of Clinical Oncology 36(9) (2018). DOI: 10.1200/JCO.2017.76.7293. · DOI · Paper page
  22. Mingu Kang et al. "Benchmarking self-supervised learning on diverse pathology datasets". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 3344-3354.
  23. Bin Li, Yin Li, and Kevin W Eliceiri. "Dual-stream multiple instance learning network for whole slide im- age classification with self-supervised contrastive learning". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 14318-14328.
  24. Fuxiao Liu et al. "Aligning Large Multi-Modal Model with Robust Instruction Tuning". In: arXiv preprint arXiv:2306.14565 (2023). · arXiv
  25. Ming Y Lu et al. "Data-efficient and weakly supervised computational pathology on whole-slide images". In: Nature biomedical engineering 5.6 (2021), pp. 555-570.
  26. Ming Y Lu et al. "Towards a visual-language foundation model for computational pathology". In: arXiv preprint arXiv:2307.12914 (2023). · arXiv
  27. Ming Y Lu et al. "Visual language pretrained multiple instance zero-shot transfer for histopathology images". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023, pp. 19764- 19775.
  28. Maxime Oquab et al. "DINOv2: Learning robust visual features without supervision". In: arXiv preprint arXiv:2304.07193 (2023). · arXiv
  29. Sudhir Perincheri et al. "An independent assessment of an artificial intelligence system for prostate cancer detection shows strong diagnostic accuracy". In: Modern Pathology 34.8 (2021), pp. 1588-1595.
  30. Wenkang Qin et al. "What a whole slide image can tell? subtype-guided masked transformer for pathological image captioning". In: arXiv preprint arXiv:2310.20607 (2023). · arXiv
  31. Patricia Raciti et al. "Clinical validation of artificial intelligence-augmented pathology diagnosis demonstrates significant gains in diagnostic accuracy in prostate cancer detection". In: Archives of Pathology & Laboratory Medicine 147.10 (2023), pp. 1178-1185.
  32. Patricia Raciti et al. "Novel artificial intelligence system increases the detection of prostate cancer in whole slide images of core needle biopsies". In: Modern Pathology 33.10 (2020), pp. 2058-2066.
  33. Alec Radford et al. "Learning transferable visual models from natural language supervision". In: International conference on machine learning. PMLR. 2021, pp. 8748-8763.
  34. Zhuchen Shao et al. "Transmil: Transformer based correlated multiple instance learning for whole slide image classification". In: Advances in neural information processing systems 34 (2021), pp. 2136-2147.
  35. Noam Shazeer. "Glu variants improve transformer". In: arXiv preprint arXiv:2002.05202 (2020). · arXiv
  36. Leonard M da Silva et al. "Independent real-world application of a clinical-grade automated prostate cancer detection system". In: The Journal of pathology 254.2 (2021), pp. 147-158.
  37. Masayuki Tsuneki and Fahdi Kanavati. "Inference of captions from histopathological patches". In: International Conference on Medical Imaging with Deep Learning. PMLR. 2022, pp. 1235-1250.
  38. Vishaal Udandarao et al. "No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance". In: arXiv preprint arXiv:2404.04125 (2024). · arXiv
  39. Ashish Vaswani et al. "Attention is all you need". In: Advances in neural information processing systems 30 (2017).
  40. Eugene Vorontsov et al. Virchow: A Million-Slide Digital Pathology Foundation Model. 2024. arXiv: 2309. 07778 [eess.IV].
  41. S. J. Wagner et al. "Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study". In: Cancer Cell 41.9 (Sept. 2023), pp. 1650-1661.
  42. Wenhui Wang et al. "When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology". In: arXiv preprint arXiv:2312.03558 (2023). · arXiv
  43. Jiahui Yu et al. "Coca: Contrastive captioners are image-text foundation models". In: arXiv preprint arXiv:2205.01917 (2022). · arXiv
  44. Renyu Zhang et al. "Evaluating and interpreting caption prediction for histopathology images". In: Machine Learning for Healthcare Conference. PMLR. 2020, pp. 418-435.

Authors
Shaikovski, G., Casson, A., Severson, K., Zimmermann, E., Wang, Y. K., Kunz, J. D., Retamero, J. A., Oakley, G., Klimstra, D., Kanan, C., Hanna, M., Zelechowski, M., Viret, J., Tenenholtz, N., Hall, J., Fusi, N., Yousfi, R., Hamilton, P., Moye, W. A., Vorontsov, E., Liu, S., & Fuchs, T. J.
Affiliations
  • Microsoft Research (United Kingdom) (GB)
  • University of Rochester (US)
  • Memorial Sloan Kettering Cancer Center (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2405.10254
arXiv
2405.10254
OpenAlex
W4397028860
Cited by
22
References
0
Citations by year
2026: 10 · 2025: 10 · 2024: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
whole slide images, histopathology, H&E staining, medical report generation, cancer subtyping, biomarker prediction, multimodal pre-training

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review