The DOI matches its Crossref record; 73 of the 75 references it cites were found in a registry or an index.
Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images
Abstract
Survival outcome prediction is a challenging weakly-supervised and ordinal regression task in computational pathology that involves modeling complex interactions within the tumor microenvironment in gigapixel whole slide images (WSIs). Despite recent progress in formulating WSIs as bags for multiple instance learning (MIL), representation learning of entire WSIs remains an open and challenging problem, especially in overcoming: 1) the computational complexity of feature aggregation in large bags, and 2) the data heterogeneity gap in incorporating biological priors such as genomic measurements. In this work, we present a Multimodal Co-Attention Transformer (MCAT) framework that learns an interpretable, dense co-attention mapping between WSIs and genomic features formulated in an embedding space. Inspired by approaches in Visual Question Answering (VQA) that can attribute how word embed-dings attend to salient objects in an image when answering a question, MCAT learns how histology patches attend to genes when predicting patient survival. In addition to visualizing multimodal interactions, our co-attention trans-formation also reduces the space complexity of WSI bags, which enables the adaptation of Transformer layers as a general encoder backbone in MIL. We apply our proposed method on five different cancer datasets (4,730 WSIs, 67 million patches). Our experimental results demonstrate that the proposed method consistently achieves superior performance compared to the state-of-the-art methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In the comparison between the FSM and MCAT [6] models across the TCGA and CPTAC cohorts, the FSM model consistently demonstrated superior performance in predicting survival outcomes for both LUAD and LUSC cases.”
From Graph attention-based fusion of pathology images and gene expression for prediction of cancer survival (2024) · cites this paper as [6]
In the citing paper
“TAH is applicable to models employing attention-weighted pooling of instances, while CoAttn suits multimodal models that fuse WSI and genomic features using co-attention.”
From Graph attention-based fusion of pathology images and gene expression for prediction of cancer survival (2024) · cites this paper as [6]
In the citing paper
“While other researchers have previously explored the development of mid-level-fusion and mixer architectures [6], [7] as well as the use of graph-based representations of WSIs [8], our work is unique in mixing node and edge embeddings along with fusion of bulk gene expression embeddings to learn a multimodal topographic mapping to predict survival.”
From Graph attention-based fusion of pathology images and gene expression for prediction of cancer survival (2024) · cites this paper as [6]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., Gillette, M. A., Paulovich, A., Pomeroy, S. L., Golub, T. R., Lander, E. S., et al. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0506580102 · Cited by 53,625
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Regression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793
Regression Models and Life-Tables
Cox, D. R. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_37 · Cited by 26,297
Graph Attention NetworksPreprint
Veličković, P., Casanova, A., Liò, P., Cucurull, G., Romero, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.17863/cam.48429 · Cited by 19,745
The 2016 World Health Organization Classification of Tumors of the Central Nervous System: a summary
Louis, D. N., Perry, A., Reifenberger, G., von Deimling, A., Figarella‐Branger, D., Cavenee, W. K., Ohgaki, H., Wiestler, O. D., Kleihues, P., & Ellison, D. W. (2016) · Acta Neuropathologica · doi:10.1007/s00401-016-1545-1 · Cited by 15,703
Research works printed but not linked: 1
- zaheer, 2017, Deep sets, Advances in neural information processing systems
Cited by
Sorted by citations
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560
Self-citationTransformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,523
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
A visual-language foundation model for computational pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02856-4 · Cited by 966
Self-citationArtificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Self-citationScaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning
Chen, R. J., Chen, C., Li, Y., Chen, T. Y., Trister, A. D., Krishnan, R. G., & Mahmood, F. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01567 · Cited by 621
Self-citationVision Transformers in medical computer vision—A contemplative retrospection
Parvaiz, A., Khalid, M. A., Zafar, R., Ameer, H., Ali, M. U., & Fraz, M. M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106126 · Cited by 302
Li, Y., El Habib Daho, M., Conze, P.-H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108635 · Cited by 274
Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review
Cui, C., Yang, H., Wang, Y., Zhao, S., Asad, Z., Coburn, L. A., Wilson, K. T., Landman, B. A., & Huo, Y. (2023) · Progress in Biomedical Engineering · doi:10.1088/2516-1091/acc2fe · Cited by 256
Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions
Lotter, W., Hassett, M. J., Schultz, N., Kehl, K. L., Van Allen, E. M., & Cerami, E. (2024) · Cancer Discovery · doi:10.1158/2159-8290.cd-23-1199 · Cited by 196
Multimodal data integration for oncology in the era of deep neural networks: a review
Waqas, A., Tripathi, A., Ramachandran, R. P., Stewart, P. A., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1408843 · Cited by 165
Jaume, G., Vaidya, A. J., Chen, R. J., Williamson, D. F. K., Liang, P. P., & Mahmood, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01100 · Cited by 184
Self-citationRecent Advances of Deep Learning for Computational Histopathology: Principles and Applications
Wu, Y., Cheng, M., Huang, S., Pei, Z., Zuo, Y., Liu, J., Yang, K., Zhu, Q., Zhang, J., Hong, H., et al. (2022) · Cancers · doi:10.3390/cancers14051199 · Cited by 148
Toward Explainable Artificial Intelligence for Precision Pathology
Klauschen, F., Dippel, J., Keyl, P., Jurmeister, P. S., Bockmayr, M. L., Möck, A., Buchstab, O., Alber, M., Ruff, L., Montavon, G., et al. (2023) · Annual Review of Pathology Mechanisms of Disease · doi:10.1146/annurev-pathmechdis-051222-113147 · Cited by 164
Waqas, A., Bui, M. M., Glassy, E. F., Naqa, I. E., Borkowski, P., Borkowski, A., & Rasool, G. (2023) · Laboratory Investigation · doi:10.1016/j.labinv.2023.100255 · Cited by 158
Xu, Y., & Chen, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01942 · Cited by 160
Vision Transformers for Computational Histopathology
Xu, H., Xu, Q., Cong, F., Kang, J., Han, C., Liu, Z., Madabhushi, A., & Lu, C. Y. (2023) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2023.3297604 · Cited by 153
Fast and scalable search of whole-slide images via self-supervised deep learning
Chen, C., Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Schaumberg, A. J., & Mahmood, F. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00929-8 · Cited by 130
Self-citationIntegrating multi-omics data with EHR for precision medicine using advanced artificial intelligence
Tong, L., Shi, W., Isgut, M., Zhong, Y., Lais, P., Gloster, L., Sun, J., Swain, A., Giuste, F., & Wang, M. D. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2023.3324264 · Cited by 130
Cross-Modal Translation and Alignment for Survival Analysis
Zhou, F., & Chen, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01964 · Cited by 140
Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images
Lu, M. Y., Chen, B., Zhang, A., Williamson, D. F. K., Chen, R. J., Ding, T., Le, L. P., Chuang, Y.-S., & Mahmood, F. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01893 · Cited by 122
Self-citationAdversarial attacks and adversarial robustness in computational pathology
Laleh, N. G., Truhn, D., Veldhuizen, G. P., Han, T., van Treeck, M., Buelow, R. D., Langer, R., Dislich, B., Boor, P., Schulz, V., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33266-0 · Cited by 137
MambaMIL: Enhancing Long Sequence Modeling with Sequence Reordering in Computational Pathology
Yang, S., Wang, Y., & Chen, H. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72083-3_28 · Cited by 114
Prediction of recurrence risk in endometrial cancer with multimodal deep learning
Volinsky-Fremond, S., Horeweg, N., Andani, S., Wolf, J. B., Lafarge, M. W., de Kroon, C. D., Ørtoft, G., Høgdall, E. V. S., Dijkstra, J., Jobsen, J. J., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02993-w · Cited by 111
Atabansi, C. C., Nie, J., Liu, H., Song, Q., Yan, L., & Zhou, X. (2023) · BioMedical Engineering OnLine · doi:10.1186/s12938-023-01157-0 · Cited by 114
Teoh, J. R., Dong, J., Zuo, X., Lai, K. W., Hasikin, K. B., & Wu, X. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2298 · Cited by 117
Attention-Challenging Multiple Instance Learning for Whole Slide Image Classification
Zhang, Y., Li, H., Sun, Y., Zheng, S., Zhu, C., & Yang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73668-1_8 · Cited by 107
Li, Z., Jiang, Y., Lu, M., Li, R., & Xia, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3263010 · Cited by 93
Self-citationArtificial intelligence applications in histopathology
Bahadir, C. D., Omar, M., Rosenthal, J., Marchionni, L., Liechty, B., Pisapia, D. J., & Sabuncu, M. R. (2024) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-023-00012-7 · Cited by 93
Machine learning in computational histopathology: Challenges and opportunities
Cooper, M. C., Ji, Z., & Krishnan, R. G. (2023) · Genes Chromosomes and Cancer · doi:10.1002/gcc.23177 · Cited by 87
Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis
Liu, J., Cen, X., Yi, C., Wang, F.-A., Ding, J., Cheng, J., Wu, Q., Gai, B., Zhou, Y., He, R., et al. (2025) · Genomics, Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzaf011 · Cited by 75
Hou, W., Lin, C., Yu, L., Qin, J., Yu, R., & Wang, L. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3253760 · Cited by 74
Recent advances of Transformers in medical image analysis: A comprehensive review
Xia, K., & Wang, J. (2023) · MedComm – Future Medicine · doi:10.1002/mef2.38 · Cited by 81
Improving diagnosis and prognosis of lung cancer using vision transformers: a scoping review
Ali, H., Mohsen, F., & Shah, Z. (2023) · BMC Medical Imaging · doi:10.1186/s12880-023-01098-z · Cited by 76
Al-Thelaya, K., Gilal, N. U., Alzubaidi, M. S., Majeed, F., Agus, M., Schneider, J., & Househ, M. S. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100335 · Cited by 67
Di, D., Zou, C., Feng, Y., Zhou, H., Ji, R., Dai, Q., & Gao, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3209652 · Cited by 63
Xiong, C., Chen, H., Sung, J. J. �., & King, I. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/176 · Cited by 73
MuRCL: Multi-Instance Reinforcement Contrastive Learning for Whole Slide Image Classification
Zhu, Z., Yu, L., Wu, W., Yu, R., Zhang, D., & Wang, L. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3227066 · Cited by 61
Zheng, Y., Li, J., Shi, J., Xie, F., Huai, J., Cao, M., & Jiang, Z. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3264781 · Cited by 63
A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model
Xu, Y., Wang, Y., Zhou, F., Ma, J., Jin, C., Yang, S., Li, J., Zhang, Z., Zhao, C., Zhou, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.15362 · Cited by 60
A multimodal knowledge-enhanced whole-slide pathology foundation model
Xu, Y., Wang, Y., Zhou, F., Ma, J., Jin, C., Yang, S., Li, J., Zhang, Z., Zhao, C., Zhou, H., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-66220-x · Cited by 118
Qu, L., Ma, Y., Luo, X., Guo, Q., Wang, M., & Song, Z. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3400876 · Cited by 61
Stegmüller, T., Bozorgtabar, B., Spahr, A., & Thiran, J. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00611 · Cited by 58
Liang, M., Chen, Q., Li, B., Wang, L., Wang, Y., Zhang, Y., Wang, R., Jiang, X. Y., & Zhang, C. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107268 · Cited by 55
Efficient High-Resolution Deep Learning: A Survey
Bakhtiarnia, A., Zhang, Q., & Iosifidis, A. (2024) · ACM Computing Surveys · doi:10.1145/3645107 · Cited by 75
Harnessing multimodal data fusion to advance accurate identification of fish feeding intensity
Du, Z., Cui, M., Xu, X., Bai, Z., Han, J., Li, W., Yang, J., Liu, X., Wang, C., & Li, D. (2024) · Biosystems Engineering · doi:10.1016/j.biosystemseng.2024.08.001 · Cited by 52
Generative Artificial Intelligence in Anatomic Pathology
Brodsky, V., Ullah, E., Bychkov, A., Song, A. H., Walk, E. E., Louis, P., Rasool, G., Singh, R. S., Mahmood, F., Bui, M. M., et al. (2025) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0215-ra · Cited by 48
Self-citationPathology-and-Genomics Multimodal Transformer for Survival Outcome Prediction
Ding, K., Zhou, M., Metaxas, D., & Zhang, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43987-2_60 · Cited by 48
Chen, C., Song, M., Song, W., Guo, L., & Jian, M. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3203421 · Cited by 50
Deep learning quantifies pathologists’ visual patterns for whole slide image diagnosis
Nan, T., Zheng, S., Qiao, S., Quan, H., Gao, X., Niu, J., Zheng, B., Guo, C., Zhang, Y., Wang, X., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-60307-1 · Cited by 54
Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review
Couture, H. D. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12122022 · Cited by 45
Redlich, J.-P., Feuerhake, F., Weis, J., Schaadt, N. S., Teuber‐Hanselmann, S., Buck, C., Luttmann, S., Eberle, A., Nikolin, S., Appenzeller, A., et al. (2024) · npj Imaging · doi:10.1038/s44303-024-00020-8 · Cited by 49
Zheng, Y., Conrad, R. D., Green, E. J., Burks, E. J., Betke, M., Beane, J. E., & Kolachalama, V. B. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3386108 · Cited by 44
“MCAT [6] implements a co-attention mechanism to merge WSI and genomic features, which are then processed by a : We conducted a series of ablation experiments to assess the impact of different feature extractors, node connectivity types, and the various components of the graph mixer layer (GML),…”
BackgroundMethodInfluentialApplication of Deep Learning in Histopathology Images of Breast Cancer: A Review
Zhao, Y., Zhang, J., Hu, D., Qu, H. Y., Tian, Y., & Cui, X. (2022) · Micromachines · doi:10.3390/mi13122197 · Cited by 44
Shao, Z., Chen, Y., Bian, H., Zhang, J., Liu, G., & Zhang, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25315 · Cited by 45
MoME: Mixture of Multimodal Experts for Cancer Survival Prediction
Xiong, C., Chen, H., Zheng, H., Wei, D., Zheng, Y., Sung, J. J. �., & King, I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72083-3_30 · Cited by 45
Liu, M., Liu, Y., Cui, H., Li, C., & Ma, J. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385897 · Cited by 42
Bazargani, R., Fazli, L., Gleave, M. E., Goldenberg, L., Bashashati, A., & Salcudean, S. E. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103197 · Cited by 42
Hou, J., Zhang, R., Xie, Y., Li, C., & Qin, W. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02257-y · Cited by 38
Pancancer outcome prediction via a unified weakly supervised deep learning model
Yuan, W. Q., Chen, Y. M., Zhu, B., Yang, S., Zhang, J., Mao, N., Xiang, J., Ji, Y., Ji, Y., Luo, X., et al. (2025) · Signal Transduction and Targeted Therapy · doi:10.1038/s41392-025-02374-w · Cited by 42
Qiu, L., Zhao, L., Hou, R., Zhao, W., Zhang, S., Lin, Z., Teng, H., & Zhao, J. (2023) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2022.102176 · Cited by 35
Applications of machine and deep learning to thyroid cytology and histopathology: a review
Slabaugh, G., Beltrán, L., Rizvi, H., Deloukas, P., & Marouli, E. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.958310 · Cited by 38
Cohort-Individual Cooperative Learning for Multimodal Cancer Survival Analysis
Zhou, H., Zhou, F., & Chen, H. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3455931 · Cited by 39
Shao, Z., Wang, Y., Chen, Y., Bian, H., Liu, S., Wang, H., & Zhang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01965 · Cited by 31
Li, L. K., Pan, H., Liang, Y., Shao, M., Xie, S., Lu, S., & Liao, S. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111502 · Cited by 30
Ji, Z., Ge, Y., Chukwudi, C. C., Kaicheng, U., Zhang, S. M., Peng, Y., Zhu, J., Zaki, H. A., Zhang, X., Yang, S., et al. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3548048 · Cited by 31
Mondol, R. K., Millar, E. K. A., Sowmya, A., & Meijering, E. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3418341 · Cited by 29
Ding, S., Li, J., Wang, J., Ying, S., & Shi, J. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3405535 · Cited by 31
Holste, G., Lin, M., Zhou, R., Wang, F., Liu, L., Yan, Q., Tassel, S. H. V., Kovacs, K. D., Chew, E. Y., Lu, Z., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01207-4 · Cited by 34
Ma, Y., Jamdade, S., Konduri, L., & Sailem, H. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01524-2 · Cited by 32
Discrepancy and Gradient-Guided Multi-modal Knowledge Distillation for Pathological Glioma Grading
Xing, X., Chen, Z., Zhu, M., Hou, Y., Gao, Z., & Yuan, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_61 · Cited by 29
Multi-Scale Efficient Graph-Transformer for Whole Slide Image Classification
Ding, S., Li, J., Wang, J., Ying, S., & Shi, J. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3317067 · Cited by 32
One label is all you need: Interpretable AI-enhanced histopathology for oncology
Tavolara, T. E., Su, Z., Gurcan, M. N., & Niazi, M. K. K. (2023) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2023.09.006 · Cited by 28
Zhi-hua, W., Yu, L., Ding, X., Liao, X., & Wang, L. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3287256 · Cited by 27
Lin, H., Hua, J., Gong, Z., Chen, M., Qiu, B., Wu, Y., He, W., Wang, Y., Feng, Z., Liang, Y., et al. (2025) · Cancer Letters · doi:10.1016/j.canlet.2025.217557 · Cited by 30
Temporal Context Matters: Enhancing Single Image Prediction with Disease Progression Representations
Konwer, A., Xu, X., Bae, J., Chen, C., & Prasanna, P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01826 · Cited by 30
Multimodal analysis methods in predictive biomedicine
Qoku, A., Katsaouni, N., Flinner, N., Buettner, F., & Schulz, M. H. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.11.011 · Cited by 29
HONeYBEE: enabling scalable multimodal AI in oncology through foundation model-driven embeddings
Tripathi, A. G., Waqas, A., Schabath, M. B., Yılmaz, Y., & Rasool, G. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02003-4 · Cited by 37
Hierarchical Transformer for Survival Prediction Using Multimodality Whole Slide Images and Genomics
Li, C., Zhu, X., Yao, J., & Huang, J. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956296 · Cited by 25
Huang, Q., Wu, S., Ou, Z., & Gao, Y. (2025) · Intelligent Oncology · doi:10.1016/j.intonc.2025.03.004 · Cited by 29
Histopathology-based protein multiplex generation using deep learning
Andani, S., Chen, B., Ficek, J., Heinke, S., Casanova, R., Hild, B. F., Sobottka, B., Bodenmiller, B., Aebersold, R. H., Ak, M., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01074-y · Cited by 29
Multi stain graph fusion for multimodal integration in pathology
Dwivedi, C., Nofallah, S., Pouryahya, M., Iyer, J. S., Leidal, K., Chung, C., Watkins, T. R., Billin, A. N., Myers, R. P., Abel, J., et al. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00200 · Cited by 24
Development of Prognostic Biomarkers by TMB-Guided WSI Analysis: A Two-Step Approach
Liu, X., Liu, Z., Yan, Y., Wang, K., Wang, A., Ye, X., Wang, L., Wei, W., Li, B., Sun, C., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3249354 · Cited by 23
Digital pathology and multimodal learning on oncology data
Waqas, A., Naveed, J., Shahnawaz, W., Asghar, S., Bui, M. M., & Rasool, G. (2024) · BJR|Artificial Intelligence · doi:10.1093/bjrai/ubae014 · Cited by 21
GTP-4o: Modality-Prompted Heterogeneous Graph Learning for Omni-Modal Biomedical Representation
Li, C., Liu, X., Wang, C., Liu, Y., Yu, W., Shao, J., & Yuan, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73235-5_10 · Cited by 27
Zhou, Q., Zhong, W., Guo, Y., Xiao, M., Ma, H., & Huang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72083-3_35 · Cited by 29
Benkirane, H., Vakalopoulou, M., Planchard, D., Adam, J., Olaussen, K. A., Michiels, S., & Cournède, P.-H. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013012 · Cited by 22
Histo-Genomic Knowledge Association for Cancer Prognosis From Histopathology Whole Slide Images
Wang, Z., Zhang, Y., Xu, Y., Imoto, S., Chen, H., & Song, J. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3526816 · Cited by 37
Multimodal integration strategies for clinical application in oncology
Zhang, B., Wan, Z., Luo, Y., Zhao, X., Samayoa, J. A., Zhao, W., & Wu, S. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1609079 · Cited by 28
Wang, W., Wang, R., Mao, L., & Min, B. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341395 · Cited by 20
Shao, W., Shi, H., Liu, J., Zuo, Y., Sun, L., Xia, T., Chen, W., Wan, P., Sheng, J., Zhu, Q., et al. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3362852 · Cited by 21
Hashimoto, N., Hanada, H., Miyoshi, H., Nagaishi, M., Sato, K., Hontani, H., Ohshima, K., & Takeuchi, I. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100359 · Cited by 20
Yan, R., Zhang, X., Jiang, Z., Wang, B., Bian, X., Ren, F., & Zhou, S. K. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3611531 · Cited by 20
Qu, L., Yang, Z., Duan, M., Ma, Y., Wang, S., Wang, M., & Song, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01962 · Cited by 21
Yang, Z., Wang, X., Xiang, J., Zhang, J., Yang, S., Wang, X., Yang, W., Li, Z., Han, X., & Liu, Y. (2023) · Archiv für Pathologische Anatomie und Physiologie und für Klinische Medicin · doi:10.1007/s00428-023-03502-z · Cited by 18
Palmal, S., Arya, N., Saha, S., & Tripathy, S. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111379 · Cited by 18
Fu, M., Fang, M. X., Khan, R. A., Liao, B., Hu, Z., & Wu, F. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102972 · Cited by 19
Identifying progression-specific Alzheimer’s subtypes using multimodal transformer
Machado Reyes, D., Chao, H., Hahn, J., Shen, L., Yan, P., & Initiative, F. T. A. D. N. (2024) · Journal of Personalized Medicine · doi:10.3390/jpm14040421 · Cited by 17
Elforaici, M. E. A., Montagnon, E., Romero, F., Le, W., Azzi, F., Trudel, D., Nguyen, B., Turcotte, S., Tang, A., & Kadoury, S. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103346 · Cited by 19
A survey on cell nuclei instance segmentation and classification: Leveraging context and attention
Nunes, J. D., Montezuma, D., Oliveira, D., Pereira, T., & Cardoso, J. S. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103360 · Cited by 19
Metadata
- Authors
- Chen, R. J., Lu, M. Y., Weng, W.-H., Chen, T. Y., Williamson, D. F., Manz, T., Shady, M., & Mahmood, F.
- Affiliations
- Broad Institute (US)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Brigham and Women's Hospital (US)
- ORCID
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 3995-4005
- OpenAlex
- W3203898052
- Cited by
- 398
- References
- 75
- Citations by year
- 2026: 106 · 2025: 121 · 2024: 94 · 2023: 55 · 2022: 22
- FWCI
- 19.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- survival prediction, whole slide images, multimodal representation learning, multiple instance learning, genomic features, tumor microenvironment, visual question answering, ordinal regression, cancer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
