Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Integrating whole-slide images (WSIs) and bulk tran-scriptomics for predicting patient survival can improve our understanding of patient prognosis. However, this multi-modal task is particularly challenging due to the different nature of these data: WSIs represent a very high-dimensional spatial description of a tumor, while bulk tran-scriptomics represent a global description of gene expression levels within that tumor. In this context, our work aims to address two key challenges: (1) how can we tokenize transcriptomics in a semantically meaningful and interpretable way?, and (2) how can we capture dense multi-modal interactions between these two modalities? Here, we propose to learn biological pathway tokens from transcriptomics that can encode specific cellular functions. Together with histology patch tokens that encode the slide morphology, we argue that they form appropriate reasoning units for interpretability. We fuse both modalities using a memoryefficient multimodal Transformer that can model interactions between pathway and histology patch tokens. Our model, Survpath, achieves state-of-the-art performance when evaluated against unimodal and multimodal baselines on five datasets from The Cancer Genome Atlas. Our interpretability framework identifies key multimodal prognostic factors, and, as such, can provide valuable insights into the interaction between genotype and phenotype. Code available at https://github.com/mahmoodlab/SurvPath.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., Gillette, M. A., Paulovich, A., Pomeroy, S. L., Golub, T. R., Lander, E. S., et al. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0506580102 · Cited by 53,625
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
The Molecular Signatures Database Hallmark Gene Set Collection
Liberzon, A., Birger, C., Thorvaldsdóttir, H., Ghandi, M., Mesirov, J. P., & Tamayo, P. (2015) · Cell Systems · doi:10.1016/j.cels.2015.12.004 · Cited by 13,807
The basics of epithelial-mesenchymal transition
Kalluri, R., & Weinberg, R. A. (2009) · Journal of Clinical Investigation · doi:10.1172/jci39104 · Cited by 10,196 · cited 1×
“EMT is thought to underlie tumor cells’ ability to invade and metastasize [33], and the inverse importance of this pathway for the low-and high-risk cases is compatible with this analysis.”
ResultVisualizing and interpreting cancer genomics data via the Xena platform
Goldman, M. J., Craft, B., Hastie, M., Repečka, K., McDade, F., Kamath, A., Banerjee, A., Luo, Y., Rogers, D., Brooks, A. N., et al. (2020) · Nature Biotechnology · doi:10.1038/s41587-020-0546-8 · Cited by 4,481 · cited 1×
“Pathway collection: We used the Xena database [23] to access raw transcriptomics from TCGA ( N G = 60 , 499 in total) along with DSS labels.”
MethodAxiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
Coudray, N., Ocampo, P. S., Sakellaropoulos, T., Narula, N., Snuderl, M., Fenyö, D., Moreira, A. L., Razavian, N., & Tsirigos, A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0177-5 · Cited by 2,930 · cited 1×
“Predicting patient prognosis is a fundamental task in computational pathology (CPATH) that aims to utilize histology whole-slide images (WSIs) for automated risk assessment, patient stratification, and response-to-treatment prediction [3, 5, 6, 13, 61, 62].”
BackgroundClinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · cited 3×
“This formulation can be seen as a sparse attention pattern [4] on a multimodal sequence, where sparsity is imposed between patch tokens.”
Self-Normalizing Neural Networks
Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.02515 · Cited by 2,267
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · cited 1×
“In parallel, the use of Transformers for multimodal fusion has gained significant attention in classification and generative tasks [58, 67, 83].”
Self-citationViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 1×
“Multimodal baselines: (a) Late fusion: We combine ABMIL [30], AMISL [85], and TransMIL [59] with an S-MLP using concatenation [48], denoted as ABMIL (Cat) , AMISL (Cat) , and TransMIL (Cat) , and Kronecker product [9, 21, 78, 86], denoted as ABMIL (KP) , AMISL (KP) , and TransMIL (KP) .”
MethodMultimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · cited 1×
“Multimodal baselines: (a) Late fusion: We combine ABMIL [30], AMISL [85], and TransMIL [59] with an S-MLP using concatenation [48], denoted as ABMIL (Cat) , AMISL (Cat) , and TransMIL (Cat) , and Kronecker product [9, 21, 78, 86], denoted as ABMIL (KP) , AMISL (KP) , and TransMIL (KP) .”
MethodFlamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
Pancancer survival analysis of cancer hallmark genes
Nagy, Á., Munkácsy, G., & Győrffy, B. (2021) · Scientific Reports · doi:10.1038/s41598-021-84787-5 · Cited by 1,381 · cited 1×
“For instance, bulk transcriptomics, which represents the average gene expression in a tissue, can reveal a richer global landscape of cell types and cell states [37, 77] and has been shown to be a strong predictor of patient survival [24, 51, 56].”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · cited 2×
“…reduce memory and computational requirements [4, 12, 14, 15, 29, 31, 80, 82], for instance, by approximating self-attention with a low-rank decomposition [44, 82], using latent bottleneck distillation [31, 32, 50], by optimizing GPU reads/writes [14, 15] or using sparse attention patterns [4, 53].”
BackgroundPredicting cancer outcomes from histology and genomics using convolutional networks
Mobadersany, P., Yousefi, S., Amgad, M., Gutman, D. A., Barnholtz-Sloan, J. S., Velázquez Vega, J. E., Brat, D. J., & Cooper, L. A. D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1717139115 · Cited by 1,084 · cited 2×
“Multimodal baselines: (a) Late fusion: We combine ABMIL [14], AMISL [6], and TransMIL [15] with an S-MLP using concatenation [61], denoted as ABMIL (Cat), AMISL (Cat), and TransMIL (Cat), and Kronecker product [60], [78]–[80], denoted as ABMIL (KP), AMISL (KP), and TransMIL (KP).”
Kather, J. N., Krisam, J., Charoentong, P., Luedde, T., Herpel, E., Weis, C.-A., Gaiser, T., Marx, A., Valous, N. A., Ferber, D., et al. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002730 · Cited by 1,082
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
Transformer-based unsupervised contrastive learning for histopathological image classification
Wang, X., Yang, S., Zhang, J., Wang, M., Zhang, J., Yang, W., Huang, J., & Han, X. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102559 · Cited by 751 · cited 1×
“Formally, we employ a pre-trained feature extractor f ( · ) to map each patch h i into a patch embedding as x In this work, we use a Swin Transformer encoder that was pre-trained via contrastive learning on more than 15 million pan-cancer histopathology patches [73, 74].”
MethodChen, R. J., Lu, M. Y., Wang, J., Williamson, D. F. K., Rodig, S. J., Lindeman, N. I., & Mahmood, F. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3021387 · Cited by 729
Self-citationRethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695
Pan-cancer integrative histology-genomic analysis via multimodal deep learning
Chen, R. J., Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Noor, Z., Shaban, M., Shady, M., Williams, M., Joo, B., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.07.004 · Cited by 660
Self-citationCaptum: A unified and generic model interpretability library for PyTorch
Kokhlikyan, N., Miglani, V., Martin, M., Wang, E., Alsallakh, B., Reynolds, J., Melnikov, A., Kliushkina, N., Araya, C., Yan, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.07896 · Cited by 658
Bandi, P., Geessink, O., Manson, Q., Van Dijk, M., Balkenhol, M., Hermsen, M., Ehteshami Bejnordi, B., Lee, B., Paeng, K., Zhong, A., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2867350 · Cited by 569 · cited 1×
“Predicting patient prognosis is a fundamental task in computational pathology (CPATH) that aims to utilize histology whole-slide images (WSIs) for automated risk assessment, patient stratification, and response-to-treatment prediction [3, 5, 6, 13, 61, 62].”
BackgroundSelf-citationYao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., & Huang, J. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101789 · Cited by 522
Biologically informed deep neural network for prostate cancer discovery
Elmarakeby, H. A., Hwang, J., Arafeh, R., Crowdis, J., Gang, S., Liu, D., AlDubayan, S. H., Salari, K., Kregel, S., Richter, C., et al. (2021) · Nature · doi:10.1038/s41586-021-03922-4 · Cited by 456 · cited 3×
“However, sparse networks have shown to be particularly parameter-efficient when the number of genes considered drastically increases and are more interpretable than regular MLPs [20].”
Artificial intelligence in histopathology: enhancing cancer research and clinical oncology
Shmatko, A., Ghaffari Laleh, N., Gerstung, M., & Kather, J. N. (2022) · Nature Cancer · doi:10.1038/s43018-022-00436-4 · Cited by 446 · cited 2×
“Predicting patient prognosis is a fundamental task in computational pathology (CPATH) that aims to utilize histology whole-slide images (WSIs) for automated risk assessment, patient stratification, and response-to-treatment prediction [3, 5, 6, 13, 61, 62].”
Using deep learning to model the hierarchical structure and function of a cell
Ma, J., Yu, M. K., Fong, S., Ono, K., Sage, E., Demchak, B., Sharan, R., & Ideker, T. (2018) · Nature Methods · doi:10.1038/nmeth.4627 · Cited by 435 · cited 1×
“This can be viewed as learning a sparse multi-layer perceptron (S-MLP) [20, 25, 45] that maps transcriptomics g ∈ R N G to tokens x ( P ) ∈ N P d .”
BackgroundDeep learning with multimodal representation for pancancer prognosis prediction
Cheerla, A., & Gevaert, O. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz342 · Cited by 421 · cited 1×
“Fusion can be based on vector concatenation [48], modality-level alignment [7], bilinear pooling ( i.e., Kronecker product) [9, 77], or factor-ized bilinear pooling [37, 54].”
BackgroundPerceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414 · cited 3×
“…these three forms of interaction is achieved by the following: (1) designing the queries, keys, and values to share parameters across token types [31, 32], and (2) simplifying the attention layer to ignore patch-to-patch interactions, which we find through experimentation to be not as effective…”
BackgroundAttention Bottlenecks for Multimodal Fusion
Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., & Sun, C. (2021) · arXiv · doi:10.48550/arxiv.2107.00135 · Cited by 400
Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images
Chen, R. J., Lu, M. Y., Weng, W.-H., Chen, T. Y., Williamson, D. F., Manz, T., Shady, M., & Mahmood, F. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00398 · Cited by 398 · cited 9×
“Early vs. Late fusion: Early fusion methods (MCAT [10], MOTCat [84] and S URV P ATH ) outperform all late fusion methods.”
InfluentialSelf-citationNyströmformer: A Nyström-based Algorithm for Approximating Self-Attention
Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., & Singh, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17664 · Cited by 376
The impact of site-specific digital histology signatures on deep learning model accuracy and bias
Howard, F. M., Dolezal, J., Kochanny, S., Schulte, J., Chen, H., Heij, L., Huo, D., Nanda, R., Olopade, O. I., Kather, J. N., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-24698-1 · Cited by 330
Long-term cancer survival prediction using multimodal deep learning
Vale-Silva, L. A., & Rohr, K. (2021) · Scientific Reports · doi:10.1038/s41598-021-92799-4 · Cited by 268
Deep learning-based survival prediction for multiple cancer types using histopathology images
Wulczyn, E., Steiner, D. F., Xu, Z., Sadhwani, A., Wang, H., Flament-Auvigne, I., Mermel, C. H., Chen, P.-H. C., Liu, Y., & Stumpe, M. C. (2020) · PLOS ONE · doi:10.1371/journal.pone.0233678 · Cited by 266
TransPath: Transformer-Based Self-supervised Learning for Histopathological Image Classification
Wang, X., Yang, S., Zhang, J., Wang, M., Zhang, J., Huang, J., Yang, W., & Han, X. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87237-3_18 · Cited by 178 · cited 2×
“Formally, we employ a pre-trained feature extractor f ( · ) to map each patch h i into a patch embedding as x In this work, we use a Swin Transformer encoder that was pre-trained via contrastive learning on more than 15 million pan-cancer histopathology patches [73, 74].”
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2307.08691 · Cited by 143 · cited 2×
“…reduce memory and computational requirements [4, 12, 14, 15, 29, 31, 80, 82], for instance, by approximating self-attention with a low-rank decomposition [44, 82], using latent bottleneck distillation [31, 32, 50], by optimizing GPU reads/writes [14, 15] or using sparse attention patterns [4, 53].”
BackgroundTransformer Quality in Linear Time
Hua, W., Dai, Z., Liu, H., & Le, Q. V. (2022) · arXiv · doi:10.48550/arxiv.2202.10447 · Cited by 137 · cited 1×
“Several models have proposed re-formulations of self-attention to reduce memory and computational requirements [4, 12, 14, 15, 29, 31, 80, 82], for instance, by approximating self-attention with a low-rank decomposition [44, 82], using latent bottleneck distillation [31, 32, 50], by optimizing GPU…”
Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology
Song, A. H., Chen, R. J., Ding, T., Williamson, D. F. K., Jaume, G., & Mahmood, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01099 · Cited by 79
Self-citationFoundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L.-P. (2022) · arXiv · doi:10.48550/arxiv.2209.03430 · Cited by 38
Self-citation
Cited by
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560
Self-citationMachine learning methods for histopathological image analysis: updates in 2024
Komura, D., Ochi, M., & Ishikawa, S. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.12.033 · Cited by 76
Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology
Song, A. H., Chen, R. J., Ding, T., Williamson, D. F. K., Jaume, G., & Mahmood, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01099 · Cited by 79
Self-citationTranscriptomics-Guided Slide Representation Learning in Computational Pathology
Jaume, G., Oldenburg, L., Vaidya, A., Chen, R. J., Williamson, D. F. K., Peeters, T., Song, A. H., & Mahmood, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00920 · Cited by 51
“…While the representation learning capabilities of (S+E) pre-training remain poorly understood, the multimodal integration of histology with gene expression data has been extensively studied in cancer-specific and pan-cancer works, especially for prognostication [3, 9, 11, 34, 50, 59, 69, 86, 89].”
BackgroundMethodSelf-citationGenerative Artificial Intelligence in Anatomic Pathology
Brodsky, V., Ullah, E., Bychkov, A., Song, A. H., Walk, E. E., Louis, P., Rasool, G., Singh, R. S., Mahmood, F., Bui, M. M., et al. (2025) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0215-ra · Cited by 48
Self-citation
Metadata
- Authors
- Jaume, G., Vaidya, A., Chen, R. J., Williamson, D. F. K., Liang, P. P., & Mahmood, F.
- Venue
- 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2024-06-16
- Pages
- 11579-11590
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4402716294
- Cited by
- 150
- References
- 43
- FWCI
- 44.42
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.