A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model
Open accessAbstract
Remarkable strides in computational pathology have been made in the task-agnostic foundation model that advances the performance of a wide array of downstream clinical tasks. Despite the promising performance, there are still several challenges. First, prior works have resorted to either vision-only or image-caption data, disregarding pathology reports with more clinically authentic information from pathologists and gene expression profiles which respectively offer distinct knowledge for versatile clinical applications. Second, the current progress in pathology FMs predominantly concentrates on the patch level, where the restricted context of patch-level pretraining fails to capture whole-slide patterns. Even recent slide-level FMs still struggle to provide whole-slide context for patch representation. In this study, for the first time, we develop a pathology foundation model incorporating three levels of modalities: pathology slides, pathology reports, and gene expression data, which resulted in 26,169 slide-level modality pairs from 10,275 patients across 32 cancer types, amounting to over 116 million pathological patch images. To leverage these data for CPath, we propose a novel whole-slide pretraining paradigm that injects the multimodal whole-slide context into the patch representation, called Multimodal Self-TAught PRetraining (mSTAR). The proposed paradigm revolutionizes the pretraining workflow for CPath, enabling the pathology FM to acquire the whole-slide context. To the best of our knowledge, this is the first attempt to incorporate three modalities at the whole-slide context for enhancing pathology FMs. To systematically evaluate the capabilities of mSTAR, we built the largest spectrum of oncological benchmark, spanning 7 categories of oncological applications in 15 types of 97 practical oncological tasks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
New types of foundation models have recently been proposed for encoding entire WSIs into slide-level general-purpose feature representations.
“To overcome these limitations, new types of foundation models have recently been proposed for encoding entire WSIs into slide-level general-purpose feature representations57–72.”
From A multimodal whole-slide foundation model for pathology (2025) · cites this paper as [63]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Springer series in statistics · doi:10.1007/978-1-4612-4380-9_16 · Cited by 18,704 · see citing works
The Molecular Signatures Database Hallmark Gene Set Collection
Liberzon, A., Birger, C., Thorvaldsdóttir, H., Ghandi, M., Mesirov, J. P., & Tamayo, P. (2015) · Cell Systems · doi:10.1016/j.cels.2015.12.004 · Cited by 13,807 · cited 1× · see citing works
“We performed evaluation on the same test set of each dataset as that of Slide Classification.”
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741 · see citing works
Self-citationThe cancer genome atlas pan-cancer analysis project
The Cancer Genome Atlas Research Network, Weinstein, J. N., Collisson, E. A., Mills, G. B., Shaw, K. R. M., Ozenberger, B. A., Ellrott, K., Shmulevich, I., Sander, C., & Stuart, J. M. (2013) · Nature Genetics · doi:10.1038/ng.2764 · Cited by 9,092 · see citing works
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293 · see citing works
Self-citationAdjuvant Trastuzumab in HER2-Positive Breast Cancer
Slamon, D. J., EIERMANN, W., Robert, N. J., Pieńkowski, T., Martín, M., Press, M. F., Mackey, J. R., Glaspy, J. A., Chan, A., Pawlicki, M., et al. (2011) · New England Journal of Medicine · doi:10.1056/nejmoa0910383 · Cited by 2,779 · see citing works
OncoKB: A Precision Oncology Knowledge Base
Chakravarty, D., Gao, J., Phillips, S. M., Kundra, R., Zhang, H., Wang, J., Rudolph, J. E., Yaeger, R. D., Soumerai, T. E., Nissan, M. H., et al. (2017) · JCO Precision Oncology · doi:10.1200/po.17.00011 · Cited by 2,735 · see citing works
Self-citationData-efficient and weakly supervised computational pathology on whole-slide images
Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Chen, R. J., Barbieri, M., & Mahmood, F. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-020-00682-w · Cited by 2,282 · see citing works
Gene-expression profiles predict survival of patients with lung adenocarcinoma
Beer, D. G., Kardia, S. L. R., Huang, C.-C., Giordano, T. J., Levin, A. M., Misek, D. E., Lin, L., Chen, G., Gharib, T. G., Thomas, D. G., et al. (2002) · Nature Medicine · doi:10.1038/nm733 · Cited by 2,025 · see citing works
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560 · cited 4× · see citing works
“In the realm of prior investigations into pathology foundation models, two prominent categories have emerged: vision-only models 2,4,46 and vision-language models 1,3 .”
InfluentialSelf-citationscGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,383 · see citing works
A practical guide to single-cell RNA-sequencing for biomedical research and clinical applications
Haque, A., Engel, J., Teichmann, S. A., & Lönnberg, T. (2017) · Genome Medicine · doi:10.1186/s13073-017-0467-4 · Cited by 1,071 · see citing works
MaPLe: Multi-modal Prompt Learning
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., & Khan, F. S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01832 · Cited by 1,011 · see citing works
A visual-language foundation model for computational pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02856-4 · Cited by 966 · cited 4× · see citing works
“8 and Supplementary Table 3, the involvement of pathology reports and gene expression data can bring extra performance gains.”
InfluentialHuang, W., Takebayashi, N., Qi, Y., & Hickerson, M. J. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-1 · Cited by 952 · see citing works
A whole-slide foundation model for digital pathology from real-world data
Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07441-w · Cited by 921 · see citing works
Self-citationA visual–language foundation model for pathology image analysis using medical twitter
Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T. J., & Zou, J. (2023) · Nature Medicine · doi:10.1038/s41591-023-02504-3 · Cited by 693 · see citing works
Pan-cancer integrative histology-genomic analysis via multimodal deep learning
Chen, R. J., Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Noor, Z., Shaban, M., Shady, M., Williams, M., Joo, B., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.07.004 · Cited by 660 · see citing works
Keap1 loss promotes Kras-driven lung cancer and results in dependence on glutaminolysis
Romero, R., Sayin, V. I., Davidson, S. M., Bauer, M. R., Singh, S. X., LeBoeuf, S. E., Karakousi, T. R., Ellis, D. C., Bhutkar, A., Sánchez-Rivera, F. J., et al. (2017) · Nature Medicine · doi:10.1038/nm.4407 · Cited by 652 · see citing works
Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J. (2021) · doi:10.1101/2021.12.05.471261 · Cited by 617 · see citing works
RNA sequencing: new technologies and applications in cancer research
Hong, M., Tao, S., Zhang, L., Diao, L., Huang, X., Huang, S., Xie, S., Xiao, Z., & Zhang, H. (2020) · Journal of Hematology & Oncology · doi:10.1186/s13045-020-01005-x · Cited by 600 · see citing works
A pathology foundation model for cancer diagnosis and prognosis prediction
Wang, X., Zhao, J., Marostica, E., Yuan, W., Jin, J., Zhang, J., Li, R., Tang, H., Wang, K., Li, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07894-z · Cited by 580 · see citing works
Self-citationA foundation model for clinical-grade computational pathology and rare cancers detection
Vorontsov, E., Bozkurt, A., Casson, A., Shaikovski, G., Zelechowski, M., Severson, K., Zimmermann, E., Hall, J., Tenenholtz, N., Fusi, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03141-0 · Cited by 572 · see citing works
Self-citationBandi, P., Geessink, O., Manson, Q., Van Dijk, M., Balkenhol, M., Hermsen, M., Ehteshami Bejnordi, B., Lee, B., Paeng, K., Zhong, A., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2867350 · Cited by 569 · see citing works
Self-citationThe CPTAC Data Portal: A Resource for Cancer Proteomics Research
Edwards, N. J., Oberti, M., Thangudu, R. R., Cai, S., McGarvey, P. B., Jacob, S., Madhavan, S., & Ketchum, K. A. (2015) · Journal of Proteome Research · doi:10.1021/pr501254j · Cited by 565 · see citing works
Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge
Bulten, W., Kartasalo, K., Chen, P.-H. C., Ström, P., Pinckaers, H., Nagpal, K., Cai, Y., Steiner, D. F., van Boven, H., Vink, R., et al. (2022) · Nature Medicine · doi:10.1038/s41591-021-01620-2 · Cited by 559 · see citing works
Self-citationMultimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images
Chen, R. J., Lu, M. Y., Weng, W.-H., Chen, T. Y., Williamson, D. F., Manz, T., Shady, M., & Mahmood, F. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00398 · Cited by 398 · see citing works
Five-Year Survival in EGFR -Mutant Metastatic Lung Adenocarcinoma Treated with EGFR-TKIs
Lin, J. J., Cardarella, S., Lydon, C., Dahlberg, S. E., Jackman, D. M., Janne, P. A., & Johnson, B. E. (2015) · Journal of Thoracic Oncology · doi:10.1016/j.jtho.2015.12.103 · Cited by 355 · see citing works
Targeting the PI3-kinase pathway in triple-negative breast cancer
Pascual, J., & Turner, N. C. (2019) · Annals of Oncology · doi:10.1093/annonc/mdz133 · Cited by 330 · see citing works
Breast-Cancer Stromal Cells with TP53 Mutations and Nodal Metastases
Patócs, A., Zhang, L., Xu, Y., Weber, F., Caldés, T., Mutter, G. L., Platzer, P., & Eng, C. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa071825 · Cited by 297 · see citing works
Self-citationPrognostic and Therapeutic Implications of EGFR and KRAS Mutations in Resected Lung Adenocarcinoma
Marks, J. L., Broderick, S., Zhou, Q. C., Chitale, D. A., Li, A. R., Zakowski, M. F., Kris, M. G., Rusch, V. W., Azzoli, C. G., Seshan, V., et al. (2008) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e318160c607 · Cited by 268 · see citing works
Sarkar, A., & Stephens, M. (2021) · Nature Genetics · doi:10.1038/s41588-021-00873-4 · Cited by 224 · see citing works
Cancer gene mutation frequencies for the U.S. population
Mendiratta, G., Ke, E., Aziz, M., Liarakos, D., Tong, M. M., & Stites, E. C. (2021) · Nature Communications · doi:10.1038/s41467-021-26213-y · Cited by 208 · see citing works
Bias in Cross-Entropy-Based Training of Deep Survival Networks
Zadeh, S. G., & Schmid, M. C. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2979450 · Cited by 202 · see citing works
Xu, Y., & Chen, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01942 · Cited by 144 · see citing works
Self-citationCross-Modal Translation and Alignment for Survival Analysis
Zhou, F., & Chen, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01964 · Cited by 129 · see citing works
Self-citationVisual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images
Lu, M. Y., Chen, B., Zhang, A., Williamson, D. F. K., Chen, R. J., Ding, T., Le, L. P., Chuang, Y.-S., & Mahmood, F. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01893 · Cited by 122 · see citing works
MambaMIL: Enhancing Long Sequence Modeling with Sequence Reordering in Computational Pathology
Yang, S., Wang, Y., & Chen, H. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72083-3_28 · Cited by 114 · see citing works
Self-citationDeep learning-based histotype diagnosis of ovarian carcinoma whole-slide pathology images
Farahani, H., Boschman, J., Farnell, D., Darbandsari, A., Zhang, A. A., Ahmadvand, P., Jones, S. J. M., Huntsman, D. D., Köbel, M., Gilks, C. B., et al. (2022) · Modern Pathology · doi:10.1038/s41379-022-01146-z · Cited by 92 · see citing works
CIC protein instability contributes to tumorigenesis in glioblastoma
Bunda, S., Heir, P., Metcalf, J. L., Li, A. S. C., Agnihotri, S., Pusch, S., Mamatjan, Y., Li, M., Burrell, K. E., Mansouri, S., et al. (2019) · Nature Communications · doi:10.1038/s41467-018-08087-9 · Cited by 83 · see citing works
Xu, F., Zhu, C., Tang, W., Wang, Y., Zhang, Y., Li, J., Jiang, H., Shi, Z., Liu, J., & Jin, M. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.759007 · Cited by 80 · see citing works
Self-citationPTEN Expression in Endometrial Biopsies as a Marker of Progression to Endometrial Carcinoma
Lacey, J. V., Mutter, G. L., Ronnett, B. M., Ioffe, O. B., Duggan, M. A., Rush, B. B., Glass, A. G., Richesson, D. A., Chatterjee, N., Langholz, B. M., et al. (2008) · Cancer Research · doi:10.1158/0008-5472.can-08-1154 · Cited by 80 · see citing works
The Digital Brain Tumour Atlas, an open histopathology resource
Roetzer-Pejrimovsky, T., Moser, A., Atli, B., Vogel, C., Mercea, P. A., Prihoda, R., Gelpí, E., Haberler, C., Höftberger, R., Hainfellner, J. A., et al. (2022) · Scientific Data · doi:10.1038/s41597-022-01157-0 · Cited by 70 · see citing works
Zhu, Z., Shen, H., Xu, J., Fang, Z., Wo, G., Ma, Y., Yang, K., Wang, Y., Yu, Q., & Tang, J. (2023) · Drug Resistance Updates · doi:10.1016/j.drup.2023.100974 · Cited by 57 · see citing works
Self-citationTranscriptomics-Guided Slide Representation Learning in Computational Pathology
Jaume, G., Oldenburg, L., Vaidya, A., Chen, R. J., Williamson, D. F. K., Peeters, T., Song, A. H., & Mahmood, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00920 · Cited by 51 · see citing works
Li, W., Xu, R., Yan, X., Liang, D., Zhang, L., Qin, X., Caiyin, Q., Zhao, G.-R., Xiao, W., Hu, Z., et al. (2019) · BMC Genomics · doi:10.1186/s12864-018-5397-z · Cited by 39 · see citing works
Guo, Z., Ma, J., Xu, Y., Wang, Y., Wang, L., & Chen, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72083-3_18 · Cited by 28 · see citing works
Self-citationMolecular markers of metastatic disease in KRAS-mutant lung adenocarcinoma
Boiarsky, D., Lydon, C., Chambers, E. S., Sholl, L. M., Nishino, M., Skoulidis, F., Heymach, J. V., Luo, J., Awad, M. M., Janne, P. A., et al. (2023) · Annals of Oncology · doi:10.1016/j.annonc.2023.04.514 · Cited by 27 · see citing works
A multimodal dataset for precision oncology in head and neck cancer
Dörrich, M., Balk, M., Heusinger, T., Beyer, S., Mirbagheri, H., Fischer, D. J., Kanso, H., Matek, C., Hartmann, A., Iro, H., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-62386-6 · Cited by 26 · see citing works
Asadi-Aghbolaghi, M., Farahani, H., Zhang, A. W., Akbari, A., Kim, S., Chow, A., Dane, S., Huntsman, D. D., Gilks, C. B., Ramus, S. J., et al. (2024) · medRxiv · doi:10.1101/2024.04.19.24306099 · Cited by 22 · see citing works
Enolase-1 as a Prognostic Biomarker of Sepsis
Ζάρρος, Α. (2023) · Journal of Exploratory Research in Pharmacology · doi:10.14218/jerp.2022.00089 · Cited by 18 · see citing works
Yu, Y., He, Z., Wang, Z., Lin, R., li, T., Zhang, Z., Ren, W., Mao, L., & Yao, H. (2023) · Annals of Oncology · doi:10.1016/j.annonc.2023.10.134 · Cited by 3 · see citing works
Self-citationSLEEPNOW, & YX_Xu (2025) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.17273572 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 22
- [3][3] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visuallanguage foundation model for computational pathology. Nature Medicine , 30(3):863-874, 2024. -
- [5][5] Saghir Alfasly, Peyman Nejat, Sobhan Hemati, Jibran Khan, Isaiah Lahr, Areej Alsaafin, Abubakr Shafique, Nneka Comfere, Dennis Murphree, Chady Meroueh, et al. When is a foundation model a foundation model. arXiv preprint arXiv:2309.11510 , 2023. - · arXiv
- [6][6] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023. - · arXiv
- [7][7] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multimodal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19113-19122, 2023. -
- [11][11] Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning , pages 2127-2136. PMLR, 2018. -
- [13][13] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems , 34:2136-2147, 2021. -
- [21][21] Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier Gonz´ alez, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature , pages 1-8, 2024. -
- [25][25] Gaurav Mendiratta, Eugene Ke, Meraj Aziz, David Liarakos, Melinda Tong, and Edward C Stites. Cancer gene mutation frequencies for the us population. Nature communications , 12(1):5961, 2021. -
- [26][26] Mike R Wilson, Jake J Reske, Jeanne Holladay, Genna E Wilber, Mary Rhodes, Julie Koeman, Marie Adams, Ben Johnson, RenWei Su, Niraj R Joshi, et al. Arid1a and pi3-kinase pathway mutations in the endometrium drive epithelial transdifferentiation and collective invasion. Nature communications , 10(1):3554, 2019. -
- [34][34] Ian F Pollack, Sydney D Finkelstein, Judith Burnham, Emiko J Holmes, Ronald L Hamilton, Allan J Yates, Jonathan L Finlay, Richard Sposto, and Children's Cancer Group. Age and tp53 mutation frequency in childhood malignant gliomas: results in a multi-institutional cohort. Cancer research , 61(20):7404-7407, 2001. -
- [36][36] Hironori Tashiro, Marian S Blazes, Rong Wu, Kathleen R Cho, Shikha Bose, Steven I Wang, Jing Li, Ramon Parsons, and Lora Hedrick Ellenson. Mutations in pten are frequent in endometrial carcinoma but rare in other common gynecological malignancies. Cancer research , 57(18):3935-3940, 1997. -
- [40][40] Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, and Hao Chen. Prototypical information bottlenecking and disentangling for multimodal cancer survival prediction. In The Twelfth International Conference on Learning Representations . -
- [41][41] Nathan Mantel et al. Evaluation of survival data and two new rank order statistics arising in its consideration. Cancer Chemother Rep , 50(3):163-170, 1966. -
- [44][44] Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Siqi Liu, Philippe Mathieu, Alexander van Eck, Donghun Lee, Julian Viret, et al. Virchow: A million-slide digital pathology foundation model. arXiv preprint arXiv:2309.07778 , 2023. - · arXiv
- [47][47] Albert Gu and Tri Dao. Mamba: Lineartime sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023. - · arXiv
- [48][48] Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486 , 2023. - · arXiv
- [55][55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning - transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. -
- [58][58] Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737 , 2017. - · arXiv
- [59][59] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020. - · arXiv
- [61][61] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022. - · arXiv
- [69][69] Michael Denkowski and Alon Lavie. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the sixth workshop on statistical machine translation , pages 85-91, 2011. -
- [70][70] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74-81, 2004. -
Cited by
Sorted by citations
A multimodal whole-slide foundation model for pathology
Ding, T., Wagner, S. J., Song, A. H., Chen, R. J., Lu, M. Y., Zhang, A., Vaidya, A. J., Jaume, G., Shaban, M., Kim, A., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03982-3 · Cited by 171 · see citing works
Foundation Model for Advancing Healthcare: Challenges, Opportunities and Future Directions
He, Y., Huang, F., Jiang, X., Nie, Y., Wang, M., Wang, J., & Chen, H. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3496744 · Cited by 130 · see citing works
Self-citationZhang, Y., Gao, Z., He, K., Li, C., & Mao, R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103027 · Cited by 23 · see citing works
Zhang, X.-M., Gao, T.-H., Cai, Q.-Y., Xia, J.-B., Sun, Y., Yang, J. M., Li, W., Zhang, S.-X.-M., Lou, H.-R., Yu, X., et al. (2026) · Military Medical Research · doi:10.1186/s40779-025-00680-6 · Cited by 32 · see citing works
Self-citationHMIL: Hierarchical Multi-Instance Learning for Fine-Grained Whole Slide Image Classification
Jin, C., Luo, L., Lin, H., Hou, J., & Chen, H. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3520602 · Cited by 21 · see citing works
Self-citationSurGen: 1020 H&E-stained whole-slide images with survival and genetic markers
Myles, C., Um, I. H., Marshall, C., Harris‐Birtill, D., & Harrison, D. J. (2025) · GigaScience · doi:10.1093/gigascience/giaf086 · Cited by 15 · see citing works
A Systematic Review and Implementation Guidelines of Multimodal Foundation Models in Medical ImagingPreprint
Huang, S.-C., Jensen, M., Yeung, S. Y., Lungren, M. P., Poon, H., & Chaudhari, A. S. (2025) · Research Square · doi:10.21203/rs.3.rs-5537908/v1 · Cited by 17 · see citing works
Panayides, A. S., Chen, H., Filipović, N., Geroski, T., Hou, J., Lekadir, K., Kostas Marias, A., Matsopoulos, G. K., Papanastasiou, G., Sarder, P., et al. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3649496 · Cited by 13 · see citing works
Self-citationLeveraging multi-modal foundation models for analysing spatial multi-omic and histopathology data
Liu, T., Huang, T., Ding, T., Wu, H., Humphrey, P., Perincheri, S., Schalper, K., Ying, R., Xu, H., Zou, J., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01602-6 · Cited by 21 · see citing works
How artificial intelligence is transforming pathology
Kwon, D. (2025) · Nature · doi:10.1038/d41586-025-01576-0 · Cited by 11 · see citing works
Generalist biological artificial intelligence in modeling the language of life
Rao, V. M., Zhang, S., Plosky, B. S., Hsu, P. D., Wang, B., Zou, J., Žitnik, M., Topol, E. J., & Rajpurkar, P. (2026) · Nature Biotechnology · doi:10.1038/s41587-026-03064-w · Cited by 12 · see citing works
Hu, D., Jiang, Z., Shi, J., Xie, F., Wu, K., Tang, K., Cao, M. W., Huai, J., & Zheng, Y. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108677 · Cited by 9 · see citing works
Wang, Y., Gu, Y., Zhang, X., Wang, B., Wang, R., Li, X., Liu, Y., Qu, F., Ren, F., Yan, R., et al. (2025) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003790 · Cited by 9 · see citing works
Self-citationDing, W., Zhang, J.-X., Jin, Z., Hua, H. J., Zu, Q., Yang, S., Wang, W., Liu, S., Zhou, H.-F., & Shi, H.-B. (2025) · Discover Oncology · doi:10.1007/s12672-025-03254-z · Cited by 10 · see citing works
Self-citationTang, W., Huang, S., Fang, H., Zhou, F., Liu, B., & Liu, Q. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02587-0 · Cited by 6 · see citing works
Self-citationCost-effective instruction learning for pathology vision and language analysis
Chen, K., Liu, M., Yan, F., Ма, Л., Shi, X., Wang, L., Wang, X., Zhu, L., Wang, Z., Zhou, M., et al. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00818-5 · Cited by 6 · see citing works
Self-citationA cancer-type-aware framework for robust multimodal survival prediction under missing modalities
Song, Y., Zhan, Z., Xie, F., Wang, N., Peng, Y., Zhang, R., & Lin, M. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag124 · Cited by 4 · see citing works
Zhang, Y., Zhang, X., Qi, X., Wu, X., Chen, F., Yang, G., & Fu, H. (2025) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2025.3619086 · Cited by 3 · see citing works
Self-citationFoundation models: Insights and implications for gastrointestinal cancer
Shi, L., Huang, R., Zhao, L.-L., & Guo, A.-J. (2025) · World Journal of Gastroenterology · doi:10.3748/wjg.v31.i47.112921 · Cited by 3 · see citing works
Learning with less supervision: A survey of label-efficient learning for medical image analysis
Jin, C., Guo, Z., Lin, Y., Luo, L., & Chen, H. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.104062 · Cited by 31 · see citing works
Self-citationBoosting pathology foundation models via few-shot prompt-tuning for rare cancer subtyping
He, D., Zhou, X., Guan, W., Zhang, L., Zhang, X., Xu, S., Wang, G., Wang, L., Yuan, X., Ma, J., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-71715-2 · Cited by 4 · see citing works
Self-citationEnd-to-end multimodal pathology foundation model with clinical dialogue
Vorontsov, E., Shaikovski, G., Casson, A., Viret, J., Zimmermann, E., Tenenholtz, N., Wang, Y. K., Bernhard, J., Godrich, R., Retámero, J. A., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04521-4 · Cited by 10 · see citing works
Self-citationUttarwar, M., Khandare, J. J., Shivamurthy, P. M., Satpute, A., Panwar, M., Kothavade, H., Ramesh, A., Iyer, S., & Shafi, G. (2026) · Diagnostics · doi:10.3390/diagnostics16020356 · Cited by 2 · see citing works
Data- and knowledge-driven multimodal learning in computational pathology: A comprehensive survey
Liu, M., Cai, C., Chen, D., Li, J., Li, J., Ming, W., & Xu, J. (2026) · EngMedicine · doi:10.1016/j.engmed.2026.100132 · Cited by 2 · see citing works
Self-citationFoundation models in computational pathology: methods, applications and clinical implications
Sali, R., Aschenaki, Y., Leveillee, R. J., Baba, F., Tessema, L., Dixon, C. M., Bachhamba, K., & Zhang, D. Y. (2026) · BMJ Oncology · doi:10.1136/bmjonc-2026-001102 · Cited by 3 · see citing works
Artificial Intelligence in Oncology: Clinical Applications, Challenges, and Opportunities
Parkinson, G., Patel, R. D., Bergstrom, C. P., Vadasz, B., Amgad, M., Cooper, L. A. D., Sparano, J. A., & Lu, J. (2026) · American Society of Clinical Oncology Educational Book · doi:10.1200/edbk-26-520716 · Cited by 2 · see citing works
Wang, Y., Li, Y., Zhou, C., Ma, X., & Qi, J. (2026) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbbio.2026.3658320 · Cited by 1 · see citing works
Self-citationPRET is a few-shot system for pan-cancer recognition without example training
Li, Y., Ning, Z., Xiang, T., Zhang, Q., Lin, Z., Yi, M., Feng, F., Zeng, B., Qian, X., Sun, L., et al. (2026) · Nature Cancer · doi:10.1038/s43018-026-01141-2 · Cited by 3 · see citing works
Foundation Models in Cancer Pathology: Techniques, Applications, and Future Directions
Zhang, B., Cui, V. Y., Wu, T., Su, G., Huang, Q., Shang, B., Kong, M., Yu, W., Yuan, Y., & Tu, H. (2026) · Research · doi:10.34133/research.1332 · Cited by 1 · see citing works
Munawar, S., Hanif, F., Raza, A., Farman, M. T., & Hınçal, E. (2026) · Scientific Reports · doi:10.1038/s41598-026-55588-5 · Cited by 1 · see citing works
Steele, S., Mazengenya, P., & Chambuso, R. (2026) · Journal of Translational Medicine · doi:10.1186/s12967-026-08426-1 · Cited by 1 · see citing works
Unified multimodal computational pathology with missing-modality robustness via Riemannian learning
Yang, G., Qu, M., Di, D., Yi, K., Xu, H. J., Su, T., & Fan, L. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104633 · Cited by 1 · see citing works
Corso, F., Zec, A., Favali, M., Ligero, M., Hilgers, L., Invernizzi, L. M., Mazzeo, L., Morales, A. M., Müller‐Franzes, G. A., Machado, I. P., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2026.102966 · Cited by 2 · see citing works
H2O: A Foundation Model Bridging Histopathology to Spatial Multi-Omics ProfilingPreprint
Gu, Y., Wu, Z., Yan, R., Wang, Z., Li, Y., Lin, S., Cui, Y., Lai, H., Luo, X., Zhou, S. K., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.04.21.717342 · Cited by 1 · see citing works
He, Q., Ge, S., Huang, Q., Li, J., Guan, T., Wang, Z., & He, Y. (2026) · Applied Intelligence · doi:10.1007/s10489-026-07109-0
Self-citationYe, C., Lv, J., Li, G., Wu, R., Zeng, S., Jiang, N., Liu, B., Wu, J., Ji, D., & Zhang, H. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00672-z
MOOZY: A Patient-First Foundation Model for Computational Pathology
Kotp, Y., Trinh, V. Q., Pal, C., & Hosseini, M. S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37359-5_26
HiAdapter: Histopathology-Induced Adapter for Pathology Foundation Models
Liu, Q., Xie, P., Dai, Z., & Bai, X. (2026) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2026.3694387
Histology image sequence representation using a deep generative neural network
Thulasidharan, P. P., Mary, L., Nath, K., & Sahoo, J. (2026) · ETRI Journal · doi:10.4218/etrij.2025-0494
A unified vision-language model for precision oncology and biomarker prediction in neuroblastoma
Zhu, J., Hu, R., Yang, S., Sun, Q., Zhao, Z., Cao, J. Y., Pan, P., Wang, K., 崔丽燕, Tang, H., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-74865-5
Self-citationCoupland, S. E., Rao, A., Jonigk, D., Bülow, R. D., Chetty, R., Rane, S. U., Dave, S., & Naresh, K. N. (2026) · Journal of Clinical Pathology · doi:10.1136/jcp-2026-210720
Jumaah, N. S., Jabbar, M. S., Abbood, S. H., Ayesha, N., Mohamed, M. B. A., & Khan, M. I. (2026) · Computational and Systems Oncology · doi:10.1002/cso2.70022
Chen, Z., Hu, Y., Xiao, Y., Ju, Y., Zhang, T., Xu, Y., Li, W., Chen, H., Rittscher, J., & Yang, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38224-5_20
Self-citationGuo, Y. (2026) · International Conference on Computer Vision, Image and Deep Learning (CVIDL) · doi:10.1109/cvidl70130.2026.11637802
Artificial intelligence agents and agentic artificial intelligence applied to precision medicine.
Costa, F. F. (2026) · Drug Discovery Today · doi:10.1016/j.drudis.2026.104781
Suresh, S., Parvathy, A. S., Raj, M., S., K. G., Priya, M. L., Biswas, R., Ramaiah, S., & Anbarasu, A. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1872286
Lei, Q.-K., Wu, T.-X., Zhang, C.-L., & Zheng, Y. (2026) · Liver Research · doi:10.1016/j.livres.2026.08.001
Dalani, T., Jerry, W. K. H., Wu, J., Fang, X., Feng, T., Xie, F., Huang, M., Liang, Y., To, K. F., Kang, W., et al. (2026) · Cancers · doi:10.3390/cancers18172802
Self-citationNeighbor-constrained attention for multiple instance learning in whole-slide histopathology
Li, M., & Li, W. (2026) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-026-03627-8
The role of pathology foundation models in cancer diagnosis, biomarker discovery, and prognosis
Embaye, K. S., Tubelleza, R., Pavilion, G. L., Kilgallon, A., & Kulasinghe, A. (2026) · Trends Open · doi:10.1016/j.treopn.2026.08.006
Nguyen, N. H., Lý, K. N., & Le, N. Q. K. (2026) · Computers · doi:10.3390/computers15090626
Role of foundation models in data-driven tissue diagnostics
Bilal, M., Aadam, Raza, M., Alsuhaibani, A., Altherwy, Y. N., Alabduljabbar, A., Almarshad, F. A., Golding, P., & Rajpoot, N. (2026) · Molecular Aspects of Medicine · doi:10.1016/j.mam.2026.101504
One Slide, Many Views: Unifying Complementary Foundation Model Perspectives for WSI Analysis
Wang, Y., Xu, Y., Yang, S., Bie, Y., Ma, J., Zhou, F., & Chen, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37321-2_27
Self-citationSpectral-Semantic Consensus for Few-Shot Whole Slide Image Classification
Du, H., Zhou, P., Shi, W., Wei, Z., Chen, Y., Du, B., & Xu, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38224-5_48
Self-citationCao, C., Ji, Z., He, C., Wang, J., Luo, X., Liu, J., Jing, X., Yan, F., Chen, Y., Dong, Z., et al. (2026) · Advanced Science · doi:10.1002/advs.78019
Self-citationMitigating Batch Effects in Histopathology via Language-Mediated Robust Embedding Generation
Zhang, Y., Wu, S., Zhang, Z., Li, D., Yao, H., Li, Y., Carmichael, I. D., Hoadley, K. A., Zhu, H., Wu, D., et al. (2026) · doi:10.1109/cvprf71909.2026.00500
Self-citationZhang, W., Zhang, S., Wei, Y., Zhao, Y., Yuan, W., Wu, X., Lu, X., Yin, T., Yang, L., Yi, Y., et al. (2026) · Laboratory Investigation · doi:10.1016/j.labinv.2026.106169
Towards scalable cancer prognosis by aligning semi-supervised learning with survival modeling
Huang, Y., Cai, L., Feng, Z., Zhao, M., Tu, X., Yuan, Z., & Wang, Z. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.104347
Foundation Models and Multimodal Learning for Medical Data with Applications in Oncology
Waqas, A., Tripathi, A., Sivaram, S., Ali, F., Kuanar, S., Singh, Y., Ajami, H., Ünal, O., Huo, Y., Elhanashi, A., et al. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02319-x
Lee, S. Y. (2026) · Blood Research · doi:10.1007/s44313-026-00177-8
Interpreting interpretable AI in digital pathology
Wang, Z., Popel, A. S., & Sulam, J. (2026) · Patterns · doi:10.1016/j.patter.2026.101664
PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialogue
Vorontsov, E., Shaikovski, G., Casson, A., Viret, J., Zimmermann, E., Tenenholtz, N., Wang, Y. K., Bernhard, J. H., Godrich, R. A., Retamero, J. A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.13063
Self-citation
- Authors
- Xu, Y., Wang, Y., Zhou, F., Ma, J., Jin, C., Yang, S., Li, J., Zhang, Z., Zhao, C., Zhou, H., Li, Z., Lin, H., Wang, X., Wang, J., Han, A., Chan, R. C. K., Liang, L., Zhang, X., & Chen, H.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Nanfang Hospital (CN)
- Kunming Medical University (CN)
- Chinese University of Hong Kong (HK)
- Sun Yat-sen University (CN)
- First Affiliated Hospital Zhejiang University (CN)
- Corresponding author
- Chen, Hao
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Volume
- 16
- Issue
- 1
- Pages
- 11406-11406
- ISSN
- 2041-1723 (ISSN-L)
- arXiv
- 2407.15362
- PMC
- PMC12738713
- OpenAlex
- W4417263460
- Cited by
- 60
- References
- 56
- Citations by year
- 2026: 47 · 2025: 11 · 2024: 2
- FWCI
- 23.26
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- arxiv.org/licenses/nonexclusive-distrib/1.0/
- Locations
- Nature Communications (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Innovation and Technology Commission (PRP/034/22FX and ITCPD/17-9)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- computational pathology, pathology foundation models, multimodal learning, pathology reports, gene expression data, slide-level analysis, molecular prediction, cancer, MSTAR, self-supervised pre-training, multimodal dataset
- MeSH terms
- Humans, Image Processing, Computer-Assisted, Neoplasms, Pathology, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
