PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialogue
Open accessAbstract
Recent rapid progress in the field of computational pathology has been enabled by foundation models. These models are beginning to move beyond encoding image patches towards whole-slide understanding but their clinical utility remains limited. In this work, we present PRISM2, a multimodal slide-level foundation model trained on data from 700,000 diagnostic specimen-report pairs, the largest vision (2.3 million whole slide images) and language (14M question-answer pairs) histopathology dataset to date. By learning through clinical-dialogue supervision, PRISM2 aligns histomorphologic features with the language of diagnostic reasoning, producing slide-level representations that support both direct diagnostic question-answering and transferable embeddings for downstream tasks. Without additional training, PRISM2 matches or exceeds the cancer-detection performance of clinical-grade products. This is observed without loss of generality on other tasks, where PRISM2 achieves top performance. Finally, using survival prediction as the example, we show that task-specific finetuning with a large dataset can outperform task-specific models, further improving performance. These results demonstrate how language-supervised pretraining provides a scalable, clinically grounded signal for learning generalizable pathology representations, bridging human diagnostic reasoning and foundation-model performance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Mass-340K contains fewer slides compared to other pretraining datasets used for patch encoders and slide encoders.
“Finally, Mass-340K contains fewer slides compared to other pretraining datasets used for patch encoders12,13,126 and slide encoders62,72.”
From A multimodal whole-slide foundation model for pathology (2025) · cites this paper as [72]
The Introduction (p. 3) reports that PRISM2 was trained on 2.3 million slides, more than the approximately 340,000 slides attributed to TITAN in Comparison models (p. 15). But that section also reports slide-encoder datasets of approximately 3,000 slides for COBRA and 170,000 for Prov-Gigapath—both smaller than 340,000—and does not establish the claimed comparison with patch-encoder datasets. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428 · see citing works
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287 · see citing works
Memorial Sloan Kettering-Integrated Mutation Profiling of Actionable Cancer Targets (MSK-IMPACT)
Cheng, D. T., Mitchell, T. N., Zehir, A., Shah, R. H., Benayed, R., Syed, A., Chandramohan, R., Liu, Z. Y., Won, H. H., Scott, S. N., et al. (2015) · The Journal of Molecular Diagnostics · doi:10.1016/j.jmoldx.2014.12.006 · Cited by 2,119 · see citing works
CoCa: Contrastive Captioners are Image-Text Foundation ModelsPreprint
Yu, J.-H., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., & Wu, Y.-H. (2022) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2205.01917 · Cited by 1,899 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“Base Embedding”
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560 · cited 2× · see citing works
“…models into the computational pathology modeling pipeline has improved performance, robustness, and data efficiency in a variety of tasks such as cancer detection, subtyping, and biomarker quantification [ wang2022transformer , filiot2023scaling , chen˙general-purpose˙2023 ,…”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · see citing works
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · see citing works
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078 · see citing works
Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · see citing works
Transformer-based unsupervised contrastive learning for histopathological image classification
Wang, X., Yang, S., Zhang, J., Wang, M., Zhang, J., Yang, W., Huang, J., & Han, X. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102559 · Cited by 751 · see citing works
Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning
Chen, R. J., Chen, C., Li, Y., Chen, T. Y., Trister, A. D., Krishnan, R. G., & Mahmood, F. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01567 · Cited by 621 · see citing works
A pathology foundation model for cancer diagnosis and prognosis prediction
Wang, X., Zhao, J., Marostica, E., Yuan, W., Jin, J., Zhang, J., Li, R., Tang, H., Wang, K., Li, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07894-z · Cited by 580 · see citing works
A foundation model for clinical-grade computational pathology and rare cancers detection
Vorontsov, E., Bozkurt, A., Casson, A., Shaikovski, G., Zelechowski, M., Severson, K., Zimmermann, E., Hall, J., Tenenholtz, N., Fusi, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03141-0 · Cited by 572 · see citing works
Self-citationGQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023) · arXiv · doi:10.48550/arxiv.2305.13245 · Cited by 447 · see citing works
Artificial intelligence in histopathology: enhancing cancer research and clinical oncology
Shmatko, A., Ghaffari Laleh, N., Gerstung, M., & Kather, J. N. (2022) · Nature Cancer · doi:10.1038/s43018-022-00436-4 · Cited by 446 · see citing works
Artificial intelligence for digital and computational pathology
Song, A. H., Jaume, G., Williamson, D. F. K., Lu, M. Y., Vaidya, A., Miller, T. R., & Mahmood, F. (2023) · Nature Reviews Bioengineering · doi:10.1038/s44222-023-00096-8 · Cited by 356 · see citing works
OpenAI, :, Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., et al. (2024) · arXiv · Cited by 180 · see citing works
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.14219 · Cited by 166 · see citing works
Self-citationReal-world deployment of a fine-tuned pathology foundation model for lung cancer biomarker detection
Campanella, G., Kumar, N., Nanda, S., Singi, S., Fluder, E., Kwan, R., Muehlstedt, S., Pfarr, N., Schüffler, P. J., Häggström, I., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03780-x · Cited by 69 · see citing works
Self-citationA Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model
Xu, Y., Wang, Y., Zhou, F., Ma, J., Jin, C., Yang, S., Li, J., Zhang, Z., Zhao, C., Zhou, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.15362 · Cited by 60 · see citing works
Self-citationVirchow2: Scaling Self-Supervised Mixed Magnification Models in Pathology
Zimmermann, E., Vorontsov, E., Viret, J., Casson, A., Zelechowski, M., Shaikovski, G., Tenenholtz, N., Hall, J., Klimstra, D., Yousfi, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2408.00738 · Cited by 53 · see citing works
Self-citationTranscriptomics-Guided Slide Representation Learning in Computational Pathology
Jaume, G., Oldenburg, L., Vaidya, A., Chen, R. J., Williamson, D. F. K., Peeters, T., Song, A. H., & Mahmood, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00920 · Cited by 51 · see citing works
Liu, J., Lichtenberg, T. M., Hoadley, K. A., Poisson, L., Lazar, A. J., Cherniack, A. D., Kovatich, A. J., Benz, C. C., Levine, D. A., Lee, A. V., et al. (2018) · Cancer Research · doi:10.1158/1538-7445.am2018-3287 · Cited by 37 · see citing works
Multistain Pretraining for Slide Representation Learning in Pathology
Jaume, G., Vaidya, A., Zhang, A., H. Song, A., J. Chen, R., Sahai, S., Mo, D., Madrigal, E., Phi Le, L., & Mahmood, F. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73414-4_2 · Cited by 27 · see citing works
Multimodal Whole Slide Foundation Model for PathologyPreprint
Ding, T., Wagner, S. J., Song, A. H., Chen, R. J., Lu, M. Y., Zhang, A., Vaidya, A. J., Jaume, G., Shaban, M., Kim, A., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.19666 · Cited by 27 · see citing works
PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology
Shaikovski, G., Casson, A., Severson, K., Zimmermann, E., Wang, Y. K., Kunz, J. D., Retamero, J. A., Oakley, G., Klimstra, D., Kanan, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.10254 · Cited by 22 · see citing works
Self-citationMAIRA-2: Grounded Radiology Report Generation
Bannur, S., Bouzid, K., Castro, D. C., Schwaighofer, A., Thieme, A., Bond-Taylor, S., Ilse, M., Pérez-García, F., Salvatelli, V., Sharma, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.04449 · Cited by 19 · see citing works
Unsupervised Foundation Model-Agnostic Slide-Level Representation Learning
Lenz, T., Neidlinger, P., Ligero, M., Wölflein, G., van Treeck, M., & Kather, J. N. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02869 · Cited by 15 · see citing works
Molecular-driven Foundation Model for Oncologic PathologyPreprint
Vaidya, A. J., Zhang, A., Jaume, G., Song, A. H., Ding, T., Wagner, S. J., Lu, M. Y., Doucet, P., Robertson, H., Almagro-Pérez, C., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.16652 · Cited by 13 · see citing works
Wang, Y. K., Tydlitatova, L. T., Kunz, J. D., Oakley, G. J., Chow, B. K. B., Godrich, R., Lee, M. C. H., Aghdam, H., Bozkurt, A., Zelechowski, M., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.09554 · Cited by 4 · see citing works
Self-citationPathAlign: A vision-language model for whole slide images in histopathologyPreprint
Ahmed, F., Sellergren, A., Yang, L., Xu, S., Babenko, B., Ward, A., Olson, N. H., Mohtashamian, A., Matias, Y., Corrado, G. S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.19578 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- [11][11] Tri Dao. 'FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning'. In: International Conference on Learning Representations (ICLR) . 2024. -
- [14][14] Utku Evci et al. 'Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning'. In: Proceedings of the 39th International Conference on Machine Learning (ICML) . Vol. 162. Proceedings of Machine Learning Research. July 2022, pp. 1-25. url : https://proceedings.mlr.press/v162/ evci22a/evci22a.pdf . -
- [16][16] Andrew Jaegle et al. 'Perceiver: General perception with iterative attention'. In: International conference on machine learning . PMLR. 2021, pp. 4651-4664. -
- [24][24] Ming Y Lu et al. 'A visual-language foundation model for computational pathology'. In: Nature Medicine 30.3 (2024), pp. 863-874. -
- [37][37] John N Weinstein et al. 'The cancer genome atlas pan-cancer analysis project'. In: Nature genetics 45.10 (2013), pp. 1113-1120. -
- [38][38] Hanwen Xu et al. 'A whole-slide foundation model for digital pathology from real-world data'. In: Nature 630.8015 (2024), pp. 181-188. -
Cited by
Sorted by citations
A multimodal whole-slide foundation model for pathology
Ding, T., Wagner, S. J., Song, A. H., Chen, R. J., Lu, M. Y., Zhang, A., Vaidya, A. J., Jaume, G., Shaban, M., Kim, A., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03982-3 · Cited by 171 · see citing works
- Authors
- Vorontsov, E., Shaikovski, G., Casson, A., Viret, J., Zimmermann, E., Tenenholtz, N., Wang, Y. K., Bernhard, J. H., Godrich, R. A., Retamero, J. A., Shia, J., Gonen, M., Weiser, M. R., Klimstra, D. S., Yousfi, R., Fusi, N., Fuchs, T. J., Severson, K., & Liu, S.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2506.13063
- OpenAlex
- W4415107977
- Cited by
- 0
- References
- 33
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- computational pathology, multimodal foundation models, whole slide images, histopathology, diagnostic reasoning, cancer detection, survival prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
