The DOI matches its Crossref record; 73 of the 75 references it cites were found in a registry or an index.
Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images
Abstract
Survival outcome prediction is a challenging weakly-supervised and ordinal regression task in computational pathology that involves modeling complex interactions within the tumor microenvironment in gigapixel whole slide images (WSIs). Despite recent progress in formulating WSIs as bags for multiple instance learning (MIL), representation learning of entire WSIs remains an open and challenging problem, especially in overcoming: 1) the computational complexity of feature aggregation in large bags, and 2) the data heterogeneity gap in incorporating biological priors such as genomic measurements. In this work, we present a Multimodal Co-Attention Transformer (MCAT) framework that learns an interpretable, dense co-attention mapping between WSIs and genomic features formulated in an embedding space. Inspired by approaches in Visual Question Answering (VQA) that can attribute how word embed-dings attend to salient objects in an image when answering a question, MCAT learns how histology patches attend to genes when predicting patient survival. In addition to visualizing multimodal interactions, our co-attention trans-formation also reduces the space complexity of WSI bags, which enables the adaptation of Transformer layers as a general encoder backbone in MIL. We apply our proposed method on five different cancer datasets (4,730 WSIs, 67 million patches). Our experimental results demonstrate that the proposed method consistently achieves superior performance compared to the state-of-the-art methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In the comparison between the FSM and MCAT [6] models across the TCGA and CPTAC cohorts, the FSM model consistently demonstrated superior performance in predicting survival outcomes for both LUAD and LUSC cases.”
From Graph attention-based fusion of pathology images and gene expression for prediction of cancer survival (2024) · cites this paper as [6]
In the citing paper
“TAH is applicable to models employing attention-weighted pooling of instances, while CoAttn suits multimodal models that fuse WSI and genomic features using co-attention.”
From Graph attention-based fusion of pathology images and gene expression for prediction of cancer survival (2024) · cites this paper as [6]
In the citing paper
“While other researchers have previously explored the development of mid-level-fusion and mixer architectures [6], [7] as well as the use of graph-based representations of WSIs [8], our work is unique in mixing node and edge embeddings along with fusion of bulk gene expression embeddings to learn a multimodal topographic mapping to predict survival.”
From Graph attention-based fusion of pathology images and gene expression for prediction of cancer survival (2024) · cites this paper as [6]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., Gillette, M. A., Paulovich, A., Pomeroy, S. L., Golub, T. R., Lander, E. S., et al. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0506580102 · Cited by 53,625
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Regression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793
Regression Models and Life-Tables
Cox, D. R. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_37 · Cited by 26,297
Graph Attention NetworksPreprint
Veličković, P., Casanova, A., Liò, P., Cucurull, G., Romero, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.17863/cam.48429 · Cited by 19,745
The 2016 World Health Organization Classification of Tumors of the Central Nervous System: a summary
Louis, D. N., Perry, A., Reifenberger, G., von Deimling, A., Figarella‐Branger, D., Cavenee, W. K., Ohgaki, H., Wiestler, O. D., Kleihues, P., & Ellison, D. W. (2016) · Acta Neuropathologica · doi:10.1007/s00401-016-1545-1 · Cited by 15,703
The Molecular Signatures Database Hallmark Gene Set Collection
Liberzon, A., Birger, C., Thorvaldsdóttir, H., Ghandi, M., Mesirov, J. P., & Tamayo, P. (2015) · Cell Systems · doi:10.1016/j.cels.2015.12.004 · Cited by 13,807
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1710.10903 · Cited by 12,107
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Breslow, N. E., Cox, D. R., & Oakes, D. (1985) · Biometrics · doi:10.2307/2530888 · Cited by 4,381
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293 · cited 1×
“…occupies a tiny proportion of the total image ( i.e. - a needle-in-a-haystack problem such as differentiating normal tissue vs micro-metastases) [4, 3, 5, 45], or 2) spanning the entire composition of a WSI and dependent on the interactions of its components ( i.e. - a fine-grained visual…”
BackgroundClinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799
(2011) · Cited by 2,752
Data-efficient and weakly supervised computational pathology on whole-slide images
Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Chen, R. J., Barbieri, M., & Mahmood, F. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-020-00682-w · Cited by 2,282
Self-citationSelf-Normalizing Neural Networks
Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.02515 · Cited by 2,267
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102
Katzman, J. L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., & Kluger, Y. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0482-1 · Cited by 1,995
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582
Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images
Graham, S., Vu, Q. D., Raza, S. E. A., Azam, A. S., Tsang, Y. W., Kwak, J. T., & Rajpoot, N. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101563 · Cited by 1,516
Attention-based Deep Multiple Instance LearningPreprint
Ilse, M., Tomczak, J. M., & Welling, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04712 · Cited by 1,428
Hierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,357
Predicting cancer outcomes from histology and genomics using convolutional networks
Mobadersany, P., Yousefi, S., Amgad, M., Gutman, D. A., Barnholtz-Sloan, J. S., Velázquez Vega, J. E., Brat, D. J., & Cooper, L. A. D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1717139115 · Cited by 1,084
Deep Modular Co-Attention Networks for Visual Question Answering
Zhou, Y., Yu, J., Cui, Y., Tao, D., & Tian, Q. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00644 · Cited by 1,033
Saltz, J., Gupta, R., Hou, L., Kurc, T., Singh, P., Nguyen, V., Samaras, D., Shroyer, K. R., Zhao, T., Batiste, R., et al. (2018) · Cell Reports · doi:10.1016/j.celrep.2018.03.086 · Cited by 1,027
Stand-Alone Self-Attention in Vision Models
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., & Shlens, J. (2019) · arXiv · doi:10.48550/arxiv.1906.05909 · Cited by 977
Patch-Based Convolutional Neural Network for Whole Slide Tissue Image Classification
Hou, L., Samaras, D., Kurç, T., Gao, Y., Davis, J., & Saltz, J. H. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.266 · Cited by 921
The single-cell pathology landscape of breast cancer
Jackson, H. W., Fischer, J. R., Zanotelli, V. R., Ali, H. R., Mechera, R., Soysal, S. D., Moch, H., Muenst, S. E., Varga, Z., Weber, W. P., et al. (2020) · Nature · doi:10.1038/s41586-019-1876-x · Cited by 892
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867
Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering
Zhou, Y., Yu, J., Jianping Fan, B., & Tao, D. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.202 · Cited by 810
Kim, J.-H., Jun, J., & Zhang, B.-T. (2018) · arXiv · doi:10.48550/arxiv.1805.07932 · Cited by 761
Chen, R. J., Lu, M. Y., Wang, J., Williamson, D. F. K., Rodig, S. J., Lindeman, N. I., & Mahmood, F. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3021387 · Cited by 729
Self-citationHadamard Product for Low-rank Bilinear PoolingPreprint
Kim, J.-H., On, K.-W., Lim, W., Kim, J., Ha, J.-W., & Zhang, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.04325 · Cited by 702
Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
Lee, J., Lee, Y., Kim, J., Kosiorek, A. R., Choi, S., & Teh, Y. W. (2018) · arXiv · doi:10.48550/arxiv.1810.00825 · Cited by 691
DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation
Chen, H., Qi, X., Yu, L., & Heng, P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.273 · Cited by 671
AI-based pathology predicts origins for cancers of unknown primary
Lu, M. Y., Chen, T. Y., Williamson, D. F. K., Zhao, M., Shady, M., Lipkova, J., & Mahmood, F. (2021) · Nature · doi:10.1038/s41586-021-03512-4 · Cited by 636
Self-citationBandi, P., Geessink, O., Manson, Q., Van Dijk, M., Balkenhol, M., Hermsen, M., Ehteshami Bejnordi, B., Lee, B., Paeng, K., Zhong, A., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2867350 · Cited by 569
Self-citationDeep learning-based classification of mesothelioma improves prediction of patient outcome
Courtiol, P., Maussion, C., Moarii, M., Pronier, E., Pilcer, S., Sefta, M., Manceron, P., Toldo, S., Zaslavskiy, M., Le Stang, N., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0583-3 · Cited by 544
Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., & Huang, J. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101789 · Cited by 522
Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images
Mahmood, F., Borders, D., Chen, R. J., McKay, G. N., Salimian, K. J., Baras, A. S., & Durr, N. J. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2927182 · Cited by 361
Self-citationNguyen, D.-K., & Okatani, T. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00637 · Cited by 352
SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
Fuchs, F. B., Worrall, D. E., Fischer, V., & Welling, M. (2020) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.2006.10503 · Cited by 346
Towards a Neural StatisticianPreprint
Edwards, H. S., & Storkey, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02185 · Cited by 331
WSISA: Making Survival Prediction from Whole Slide Histopathological Images
Zhu, X., Yao, J., Zhu, F., & Huang, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.725 · Cited by 320
Diao, J. A., Wang, J. K., Chui, W. F., Mountain, V., Gullapally, S. C., Srinivasan, R., Mitchell, R. N., Glass, B., Hoffman, S., Rao, S. K., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-21896-9 · Cited by 289
Geospatial immune variability illuminates differential evolution of lung adenocarcinoma
AbdulJabbar, K., Raza, S. E. A., Rosenthal, R., Jamal‐Hanjani, M., Veeriah, S., Akarca, A. U., Lund, T., Moore, D. A., Salgado, R., Bakir, M. A., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0900-x · Cited by 287
Federated learning for computational pathology on gigapixel whole slide images
Lu, M. Y., Chen, R. J., Kong, D., Lipkova, J., Singh, R., Williamson, D. F. K., Chen, T. Y., & Mahmood, F. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102298 · Cited by 267
Self-citationDeep learning-based survival prediction for multiple cancer types using histopathology images
Wulczyn, E., Steiner, D. F., Xu, Z., Sadhwani, A., Wang, H., Flament-Auvigne, I., Mermel, C. H., Chen, P.-H. C., Liu, Y., & Stumpe, M. C. (2020) · PLOS ONE · doi:10.1371/journal.pone.0233678 · Cited by 266
Neural Image Compression for Gigapixel Histopathology Image Analysis
Tellez, D., Litjens, G. J. S., van der Laak, J. A. W. M., & Ciompi, F. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2936841 · Cited by 246
Zhao, Y. S., Yang, F., Fang, Y., Liu, H., Zhou, N., Zhang, J., Sun, J., Yang, S., Menze, B. H., Fan, X., et al. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00489 · Cited by 208
Bias in Cross-Entropy-Based Training of Deep Survival Networks
Zadeh, S. G., & Schmid, M. C. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2979450 · Cited by 202
CAMEL: A Weakly Supervised Learning Framework for Histopathology Image Segmentation
Xu, G., Song, Z., Sun, Z., Ku, C., Yang, Z., Liu, C., Wang, S., Ma, J., & Xu, W. H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01078 · Cited by 196
Shaban, M., Khurram, S. A., Fraz, M. M., Alsubaie, N. M., Masood, I., Mushtaq, S., Hassan, M., Loya, A., & Rajpoot, N. (2019) · Scientific Reports · doi:10.1038/s41598-019-49710-z · Cited by 176
Wong, W. H. (1986) · The Annals of Statistics · doi:10.1214/aos/1176349844 · Cited by 172
HistoSegNet: Semantic Segmentation of Histological Tissue Type in Whole Slide Images
Chan, L., Hosseini, M. S., Rowsell, C. H., Plataniotis, K. N., & Damaskinos, S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01076 · Cited by 152
Chen, R. J., Janković, F., Marinsek, N., Foschini, L., Kourtis, L. C., Signorini, A., Pugh, M., Shen, J., Yaari, R., Maljkovic, V., et al. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330690 · Cited by 119
Self-citationFast Transformers with Clustered AttentionPreprint
Vyas, A., Katharopoulos, A., & Fleuret, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04825 · Cited by 86
Deep Multi-instance Learning for Survival Prediction from Whole Slide Images
Yao, J., Zhu, X., & Huang, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32239-7_55 · Cited by 81
Lu, M. Y., Chen, R. J., Wang, J., Dillon, D., & Mahmood, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10825 · Cited by 80
Self-citationMultiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles
Gamper, J., & Rajpoot, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01628 · Cited by 75
Capturing Cellular Topology in Multi-Gigapixel Pathology Images
Lu, W., Graham, S., Bilal, M., Rajpoot, N., & Minhas, F. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00138 · Cited by 63
Hosseini, M. S., Chan, L., Tse, G., Tang, M. C. S., Deng, J., Norouzi, S., Rowsell, C. H., Plataniotis, K. N., & Damaskinos, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01202 · Cited by 62
Fine-grained histopathological image analysis via robust segmentation and large-scale retrieval
Zhang, X., Su, H., Yang, L., & Zhang, S. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299174 · Cited by 57
Multimodal Multitask Representation Learning for Pathology Biobank Metadata PredictionPreprint
Weng, W., Cai, Y., Lin, A. S., Tan, F. E., & Chen, P.-H. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.07846 · Cited by 23
Self-citationNeedles in Haystacks: On Classifying Tiny Objects in Large ImagesPreprint
Pawlowski, N., Bhooshan, S., Ballas, N., Ciompi, F., Glocker, B., & Drozdzal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06037 · Cited by 23
No pixel-level annotations needed
van der Laak, J. A. W. M., Ciompi, F., & Litjens, G. J. S. (2019) · Nature Biomedical Engineering · doi:10.1038/s41551-019-0472-6 · Cited by 22
Learning Permutation Invariant Representations Using Memory Networks
Kalra, S., Adnan, M., Taylor, G., & Tizhoosh, H. R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58526-6_40 · Cited by 22
Polyp segmentation and classification using predicted depth from monocular endoscopy
Mahmood, F., Yang, Z., Durr, N. J., Chen, R., Xu, W., & Borders, D. (2019) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.2513117 · Cited by 20
Self-citationChen, R. J., Lu, M. Y., Shady, M., Lipková, J., Chen, T., Williamson, D. F. K., Joo, B., & Mahmood, F. (2021) · Clinical Cancer Research · doi:10.1158/1557-3265.adi21-po-002 · Cited by 7
Self-citation
Research works printed but not linked: 1
- zaheer, 2017, Deep sets, Advances in neural information processing systems
Cited by
Sorted by citations
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560
Self-citationTransformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,523
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
A visual-language foundation model for computational pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02856-4 · Cited by 966
Self-citationArtificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Self-citation
Metadata
- Authors
- Chen, R. J., Lu, M. Y., Weng, W.-H., Chen, T. Y., Williamson, D. F., Manz, T., Shady, M., & Mahmood, F.
- Affiliations
- Broad Institute (US)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Brigham and Women's Hospital (US)
- ORCID
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 3995-4005
- OpenAlex
- W3203898052
- Cited by
- 398
- References
- 75
- Citations by year
- 2026: 106 · 2025: 121 · 2024: 94 · 2023: 55 · 2022: 22
- FWCI
- 19.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- survival prediction, whole slide images, multimodal representation learning, multiple instance learning, genomic features, tumor microenvironment, visual question answering, ordinal regression, cancer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
