Survival Prediction via Hierarchical Multimodal Co-Attention Transformer: A Computational Histology-Radiology Solution
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
The rapid advances in deep learning-based computational pathology and radiology have demonstrated the promise of using whole slide images (WSIs) and radiology images for survival prediction in cancer patients. However, most image-based survival prediction methods are limited to using either histology or radiology alone, leaving integrated approaches across histology and radiology relatively underdeveloped. There are two main challenges in integrating WSIs and radiology images: (1) the gigapixel nature of WSIs and (2) the vast difference in spatial scales between WSIs and radiology images. To address these challenges, in this work, we propose an interpretable, weakly-supervised, multimodal learning framework, called Hierarchical Multimodal Co-Attention Transformer (HMCAT), to integrate WSIs and radiology images for survival prediction. Our approach first uses hierarchical feature extractors to capture various information including cellular features, cellular organization, and tissue phenotypes in WSIs. Then the hierarchical radiology-guided co- attention (HRCA) in HMCAT characterizes the multimodal interactions between hierarchical histology-based visual concepts and radiology features and learns hierarchical co- attention mappings for two modalities. Finally, HMCAT combines their complementary information into a multimodal risk score and discovers prognostic features from two modalities by multimodal interpretability. We apply our approach to two cancer datasets (365 WSIs with matched magnetic resonance [MR] images and 213 WSIs with matched computed tomography [CT] images). Our results demonstrate that the proposed HMCAT consistently achieves superior performance over the unimodal approaches trained on either histology or radiology data alone, as well as other state-of-the-art methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Radiomics: Images Are More than Pictures, They Are Data
Gillies, R. J., Kinahan, P. E., & Hricak, H. (2016) · Radiology · doi:10.1148/radiol.2015151169 · Cited by 8,421
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293
Clinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799
Data-efficient and weakly supervised computational pathology on whole-slide images
Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Chen, R. J., Barbieri, M., & Mahmood, F. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-020-00682-w · Cited by 2,282
Self-citationSelf-Normalizing Neural Networks
Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.02515 · Cited by 2,267
Tertiary lymphoid structures in the era of cancer immunotherapy
Sautès-Fridman, C., Petitprez, F., Calderaro, J., & Fridman, W. H. (2019) · Nature reviews. Cancer · doi:10.1038/s41568-019-0144-6 · Cited by 1,884
Checking the Cox model with cumulative sums of martingale-based residuals
LIN, D. Y., WEI, L. J., & YING, Z. (1993) · Biometrika · doi:10.1093/biomet/80.3.557 · Cited by 1,432 · cited 3×
“For survival prediction using deep learning, a commonly used loss function is the Cox partial likelihood from the Cox Proportional Hazards (CoxPH) model [23].”
Predicting cancer outcomes from histology and genomics using convolutional networks
Mobadersany, P., Yousefi, S., Amgad, M., Gutman, D. A., Barnholtz-Sloan, J. S., Velázquez Vega, J. E., Brat, D. J., & Cooper, L. A. D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1717139115 · Cited by 1,084 · cited 5×
“proportional hazards model [23] for survival outcome prediction [9].”
InfluentialLi, B., Li, Y., & Eliceiri, K. W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01409 · Cited by 957 · cited 3×
“normalizing network (SNN) [41] for radiomics and deep radiology features, Deep Sets which is one of the first neural network architectures for set-based deep learning, attentionbased MIL (AMIL) [20], [24] which is a set-based neural network with GAP, dual-stream MIL (DSMIL) [42] which uses masked non-local pooling fun…”
The Cancer Genome Atlas: Creating Lasting Value beyond Its Data
Hutter, C., & Zenklusen, J. C. (2018) · Cell · doi:10.1016/j.cell.2018.03.042 · Cited by 871
Chen, R. J., Lu, M. Y., Wang, J., Williamson, D. F. K., Rodig, S. J., Lindeman, N. I., & Mahmood, F. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3021387 · Cited by 729
Self-citationHarnessing multimodal data integration to advance precision oncology
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., & Shah, S. P. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-021-00408-3 · Cited by 662 · cited 4×
“form new data-driven insights for patients with cancer [4].”
InfluentialAI-based pathology predicts origins for cancers of unknown primary
Lu, M. Y., Chen, T. Y., Williamson, D. F. K., Zhao, M., Shady, M., Lipkova, J., & Mahmood, F. (2021) · Nature · doi:10.1038/s41586-021-03512-4 · Cited by 636 · cited 1×
“THE past few decades have witnessed significant progress in cancer diagnosis and prognosis [1], [2], [3].”
Self-citationScaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning
Chen, R. J., Chen, C., Li, Y., Chen, T. Y., Trister, A. D., Krishnan, R. G., & Mahmood, F. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01567 · Cited by 621
Bandi, P., Geessink, O., Manson, Q., Van Dijk, M., Balkenhol, M., Hermsen, M., Ehteshami Bejnordi, B., Lee, B., Paeng, K., Zhong, A., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2867350 · Cited by 569 · cited 2×
“and biomarker identification [10], [11].”
Deep learning-based classification of mesothelioma improves prediction of patient outcome
Courtiol, P., Maussion, C., Moarii, M., Pronier, E., Pilcer, S., Sefta, M., Manceron, P., Toldo, S., Zaslavskiy, M., Le Stang, N., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0583-3 · Cited by 544
Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., & Huang, J. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101789 · Cited by 522
Deep learning with multimodal representation for pancancer prognosis prediction
Cheerla, A., & Gevaert, O. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz342 · Cited by 421 · cited 1×
“Multimodal learning in the biomedical domain seeks to aggregate, integrate, and analyze disparate heterogeneous data modalities to complete prediction tasks in areas such as cancer diagnosis and therapeutic response prediction [34],”
Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images
Chen, R. J., Lu, M. Y., Weng, W.-H., Chen, T. Y., Williamson, D. F., Manz, T., Shady, M., & Mahmood, F. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00398 · Cited by 398 · cited 9×
“former [20] that learns how histology instances attend to genes, we learn how hierarchical histology instances attend to radiology features when predicting patient survival.”
InfluentialSelf-citationThe present and future of deep learning in radiology
Saba, L., Biswas, M., Kuppili, V., Cuadrado Godia, E., Suri, H. S., Edla, D. R., Omerzu, T., Laird, J. R., Khanna, N. N., Mavrogeni, S., et al. (2019) · European Journal of Radiology · doi:10.1016/j.ejrad.2019.02.038 · Cited by 353
Deep learning-based survival prediction for multiple cancer types using histopathology images
Wulczyn, E., Steiner, D. F., Xu, Z., Sadhwani, A., Wang, H., Flament-Auvigne, I., Mermel, C. H., Chen, P.-H. C., Liu, Y., & Stumpe, M. C. (2020) · PLOS ONE · doi:10.1371/journal.pone.0233678 · Cited by 266
Jiang, Y., Jin, C., Yu, H., Wu, J., Chen, C., Yuan, Q., Huang, W., Hu, Y., Xu, Y., Zhou, Z., et al. (2021) · Annals of Surgery · doi:10.1097/sla.0000000000003778 · Cited by 169
Self-citationBraman, N., Gordon, J. W. H., Goossens, E. T., Willis, C., Stumpe, M. C., & Venkataraman, J. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87240-3_64 · Cited by 107
Jiang, Y., Liang, X., Han, Z., Wang, W., Xi, S., Li, T., Chen, C., Yuan, Q., Li, N., Yu, J., et al. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00065-0 · Cited by 58 · cited 1×
“radiology images [7], [8] and gene expression signatures [9].”
Self-citation
Cited by
Multimodal Deep Learning Approaches for Precision Oncology: A Comprehensive Review
Yang, H., Yang, M., Chen, J., Yao, G., Zou, Q., & Jia, L. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae699 · Cited by 115
Radiology and multi-scale data integration for precision oncology
Paverd, H., Zormpas-Petridis, K., Clayton, H., Burge, S., & Crispin-Ortuzar, M. (2024) · npj Precision Oncology · doi:10.1038/s41698-024-00656-0 · Cited by 41
Hou, J., Zhang, R., Xie, Y., Li, C., & Qin, W. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02257-y · Cited by 38
“Regarding multimodal fusion, par-ticularlyinthecontextoftumorsurvivalprediction,thesestudies 14,26 – 29 haveall employed feature-level fusion to enable cross-modal information interaction, whichrepresentsakeyadvantageofthisapproachoverearlyfusion(data-level fusion) and late fusion (decision-level…”
Metadata
- Authors
- Li, Z., Jiang, Y., Lu, M., Li, R., & Xia, Y.
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2023-09
- Volume
- 42
- Issue
- 9
- Pages
- 2678-2689
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W4361802136
- Cited by
- 83
- References
- 25
- FWCI
- 7.29
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.