The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Deep learning with multimodal representation for pancancer prognosis prediction
Open accessPublished version of a preprint
Abstract
Motivation: Estimating the future course of patients with cancer lesions is invaluable to physicians; however, current clinical methods fail to effectively use the vast amount of multimodal data that is available for cancer patients. To tackle this problem, we constructed a multimodal neural network-based model to predict the survival of patients for 20 different cancer types using clinical data, mRNA expression data, microRNA expression data and histopathology whole slide images (WSIs). We developed an unsupervised encoder to compress these four data modalities into a single feature vector for each patient, handling missing data through a resilient, multimodal dropout method. Encoding methods were tailored to each data type—using deep highway networks to extract features from clinical and genomic data, and convolutional neural networks to extract features from WSIs. Results: We used pancancer data to train these feature encodings and predict single cancer and pancancer overall survival, achieving a C-index of 0.78 overall. This work shows that it is possible to build a pancancer model for prognosis that also predicts prognosis in single cancer sites. Furthermore, our model handles multiple data modalities, efficiently analyzes WSIs and represents patient multimodal data flexibly into an unsupervised, informative representation. We thus present a powerful automated tool to accurately determine prognosis, a key step towards personalized treatment for cancer patients. Availability and implementation: https://github.com/gevaertlab/MultimodalPrognosis
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Different approaches have been proposed, such as multi-task learning [70], generative models [63] and multimodal dropout [78, 101].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [78]
The cited source says what the claim attributes to it.
In the citing paper
“Cheerla and Gevaert (2019) constructed a multimodal neural network-based model to predict the survival of patients for 20 different cancer types using clinical data, mRNA expression data, microRNA expression data, and histopathology whole slide images (WSIs).”
From Navigating the landscape of multimodal AI in medicine: A scoping review on technical challenges and clinical applications (2025) · cites this paper as Cheerla and Gevaert, 2019
In the citing paper
“Indeed, it has been shown that training a model on multiple tissues or cancer variants can improve detection performance for each cancer53, as many morphological features are observed across cancers from different topographies54.”
From A foundation model for clinical-grade computational pathology and rare cancers detection (2024) · cites this paper as [53]
In the citing paper
“although we posit that increased diversity helps to learn robust and rare features. Indeed, it has been shown that training a model on multiple tissues or cancer variants can improve detection performance for each cancer53, as many morphological features are observed across cancers from different topographies54.”
From A foundation model for clinical-grade computational pathology and rare cancers detection (2024) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
(2008) · Journal of Machine Learning Research · Cited by 33,051
The cancer genome atlas pan-cancer analysis project
The Cancer Genome Atlas Research Network, Weinstein, J. N., Collisson, E. A., Mills, G. B., Shaw, K. R. M., Ozenberger, B. A., Ellrott, K., Shmulevich, I., Sander, C., & Stuart, J. M. (2013) · Nature Genetics · doi:10.1038/ng.2764 · Cited by 9,092
MicroRNA signatures in human cancers
Calin, G. A., & Croce, C. M. (2006) · Nature reviews. Cancer · doi:10.1038/nrc1997 · Cited by 7,668
Cox, D. R., & Oakes, D. (2018) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315137438 · Cited by 6,269
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201
Oncomirs—MicroRNAs with a role in cancer
Esquela-Kerscher, A., & Slack, F. J. (2006) · Nature Reviews Cancer · doi:10.1038/nrc1840 · Cited by 5,884
Learning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036
The Versatile Role of microRNA-30a in Human Cancer
Wang, C., Yang, X., Chen, Y., & Chen, L. (2017) · Cellular Physiology and Biochemistry · doi:10.1159/000471111 · Cited by 3,414
(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534
Machine Learning Identifies Stemness Features Associated with Oncogenic Dedifferentiation
Malta, T. M., Sokolov, A., Gentles, A. J., Burzykowski, T., Poisson, L., Weinstein, J. N., Kamińska, B., Huelsken, J., Omberg, L., Gevaert, O., et al. (2018) · Cell · doi:10.1016/j.cell.2018.03.034 · Cited by 2,180
Self-citationKatzman, J. L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., & Kluger, Y. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0482-1 · Cited by 1,995
Wide Residual NetworksPreprint
Zagoruyko, S., & Komodakis, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07146 · Cited by 1,850
Image analysis and machine learning in digital pathology: challenges and opportunities
Madabhushi, A., & Lee, G. (2016) · Medical Image Analysis · doi:10.1016/j.media.2016.06.037 · Cited by 1,093
Patch-Based Convolutional Neural Network for Whole Slide Tissue Image Classification
Hou, L., Samaras, D., Kurç, T., Gao, Y., Davis, J., & Saltz, J. H. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.266 · Cited by 921
Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015) · arXiv · doi:10.48550/arxiv.1505.00387 · Cited by 912
Systematic Analysis of Breast Cancer Morphology Uncovers Stromal Features Associated with Survival
Beck, A. H., Sangoi, A. R., Leung, S. C. Y., Marinelli, R. J., Nielsen, T. O., van de Vijver, M. J., West, R. B., van de Rijn, M., & Koller, D. (2011) · Science Translational Medicine · doi:10.1126/scitranslmed.3002564 · Cited by 780
Toward understanding and exploiting tumor heterogeneity
Alizadeh, A. A., Aranda, V., Bardelli, A., Blanpain, C., Bock, C., Borowski, C., Caldas, C., Califano, A., Doherty, M., Elsner, M., et al. (2015) · Nature Medicine · doi:10.1038/nm.3915 · Cited by 760
The pacific symposium on biocomputing
Dunker, A. K. (1998) · Journal of Molecular Graphics and Modelling · doi:10.1016/s1093-3263(98)80012-0 · Cited by 659
Unsupervised Person Re-identification
Fan, H., Zheng, L., Yan, C. C., & Yang, Y. (2018) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3243316 · Cited by 636
Dropout Training as Adaptive RegularizationPreprint
Wager, S., Wang, S. I., & Liang, P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.1493 · Cited by 498
Wang, S., Zhou, M., Liu, Z., Liu, Z., Gu, D., Zang, Y., Dong, D., Gevaert, O., & Tian, J. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.06.014 · Cited by 479
Self-citationGevaert, O., Xu, J., Hoang, C. D., Leung, A. N. C., Xu, Y., Quon, A. S. W., Rubin, D. L., Napel, S. A., & Plevritis, S. K. (2012) · Radiology · doi:10.1148/radiol.12111607 · Cited by 400
Self-citationGevaert, O., Smet, F. D., Timmerman, D., Moreau, Y., & Moor, B. D. (2006) · Intelligent Systems in Molecular Biology · doi:10.1093/bioinformatics/btl230 · Cited by 378
Self-citationGenomic, Pathway Network, and Immunologic Features Distinguishing Squamous Carcinomas
Campbell, J. D., Yau, C. S. F., Bowlby, R., Liu, Y., Brennan, K., Fan, H., Taylor, A. M., Wang, C., Walter, V., Akbani, R., et al. (2018) · Cell Reports · doi:10.1016/j.celrep.2018.03.063 · Cited by 341
Self-citationWSISA: Making Survival Prediction from Whole Slide Histopathological Images
Zhu, X., Yao, J., Zhu, F., & Huang, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.725 · Cited by 320
Kaiser, L., Gomez, A. N., Shazeer, N., Vaswani, A., Parmar, N., Jones, L., & Uszkoreit, J. (2017) · arXiv · doi:10.48550/arxiv.1706.05137 · Cited by 253
Unsupervised Person Re-identification: Clustering and Fine-tuningPreprint
Fan, H., Zheng, L., & Yang, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10444 · Cited by 146
MethylMix: an R package for identifying DNA methylation-driven genes
Gevaert, O. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv020 · Cited by 140
Self-citationDeep Learning for Patient-Specific Kidney Graft Survival AnalysisPreprint
Luck, M. M., Sylvain, T., Cardinal, H., Lodi, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10245 · Cited by 121
Bejnordi, B. E., Lin, J., Glass, B., Mullooly, M., Gierach, G. L., Sherman, M. E., Karssemeijer, N., van der Laak, J. A. W. M., & Beck, A. H. (2017) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2017.7950668 · Cited by 79
Tumor Heterogeneity and Therapeutic Resistance
Lovly, C. M., Salama, A. K. S., & Salgia, R. (2016) · American Society of Clinical Oncology Educational Book · doi:10.1200/edbk_158808 · Cited by 74
Novel image markers for non-small cell lung cancer classification and survival prediction
Wang, H., Xing, F., Su, H., Stromberg, A. J., & Yang, L. (2014) · BMC Bioinformatics · doi:10.1186/1471-2105-15-310 · Cited by 72
A kernel-based integration of genome-wide data for clinical decision support
Daemen, A., Gevaert, O., Ojeda, F., Debucquoy, A., Suykens, J. A. K., Sempoux, C. L., Machiels, J., Haustermans, K. M., & Moor, B. D. (2009) · Genome Medicine · doi:10.1186/gm39 · Cited by 70
Self-citationMicroRNA based Pan-Cancer Diagnosis and Treatment Recommendation
Cheerla, N., & Gevaert, O. (2017) · BMC Bioinformatics · doi:10.1186/s12859-016-1421-y · Cited by 70
Self-citationGevaert, O., Villalobos, V. M., Sikic, B. I., & Plevritis, S. K. (2013) · Interface Focus · doi:10.1098/rsfs.2013.0013 · Cited by 53
Self-citationIntegrated genomic analysis identifies subclasses and prognosis signatures of kidney cancer
Christinat, Y., & Krek, W. (2015) · Oncotarget · doi:10.18632/oncotarget.3294 · Cited by 45
Intra-tumor heterogeneity: lessons from microbial evolution and clinical implications
de Bruin, E. C., Taylor, T., & Swanton, C. (2013) · Genome Medicine · doi:10.1186/gm505 · Cited by 44
Pathway-Structured Predictive Model for Cancer Survival Prediction: A Two-Stage Approach
Zhang, X., Li, Y., Akinyemiju, T., Ojesina, A. I., Buckhaults, P., Liu, N., Xu, B., & Yi, N. (2016) · Genetics · doi:10.1534/genetics.116.189191 · Cited by 41
Daemen, A., Gevaert, O., Bie, T. D., Debucquoy, A., Machiels, J., Moor, B. D., & Haustermans, K. M. (2007) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789812776136_0018 · Cited by 38
Self-citationLitovkin, K., Joniau, S., Lerut, E., Laenen, A., Gevaert, O., Spahn, M., Kneitz, B., Isebaert, S. F., Haustermans, K. M., Beullens, M., et al. (2014) · Journal of Cancer Research and Clinical Oncology · doi:10.1007/s00432-014-1738-8 · Cited by 34
Self-citationA deep learning framework for imputing missing values in genomic dataPreprint
Qiu, Y. L., Zheng, H., & Gevaert, O. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/406066 · Cited by 29
Self-citationDeep Recurrent Attention Models for Histopathological Image AnalysisPreprint
Momeni, A., Thibault, M., & Gevaert, O. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/438341 · Cited by 26
Self-citationMiRNAs Predict the Prognosis of Patients with Triple Negative Breast Cancer: A Meta-Analysis
Liu, Y., Zhang, Y., Li, Q., Li, J., Ma, X., Jinfang, X., Rong, S., Wu, Z., Tian, Y., Li, J., et al. (2017) · PLoS ONE · doi:10.1371/journal.pone.0170088 · Cited by 24
Gevaert, O., & Plevritis, S. K. (2012) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789814447973_0013 · Cited by 23
Self-citationDropout-Enabled Ensemble Learning for Multi-scale Biomedical Data
Momeni, A., Thibault, M., & Gevaert, O. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11723-8_41 · Cited by 12
Self-citationZhu, X., Yao, J., Xiao, G., Xie, Y., Rodriguez‐Canales, J., Parra, E. R., Behrens, C., Wistuba, I. I., & Huang, J. (2016) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2016.7822559 · Cited by 8
Gevaert, O., Vooren, S. V., & Moor, B. D. (2007) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789812776136_0028 · Cited by 5
Self-citationA network analysis of methane and feed conversion genes in the rumen microbial community
Browne, F., Wang, H., Zheng, H., Roehe, R., Dewhurst, R. J., & Walsh, P. (2016) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2016.7822741 · Cited by 4
Flagged first, then by citations
Research works printed but not linked: 4
- Gevaert, 2013, Pacific Symposium on Biocomputing 2013, 123
- Gevaert, 2008, Pacific Symposium on Biocomputing 2008, 279
- Momeni, 2018, International MICCAI Brainlesion Workshop, 407
- Yao, 2016, International Conference on Medical Image Computing and Computer-Assisted Intervention, 649
Cited by
Sorted by citations
Deep learning in cancer diagnosis, prognosis and treatment selection
Tran, K. A., Kondrashova, O., Bradley, A., Williams, E. D., Pearson, J. V., & Waddell, N. (2021) · Genome Medicine · doi:10.1186/s13073-021-00968-x · Cited by 932
Artificial Intelligence in Cancer Research and Precision Medicine
Bhinder, B., Gilvary, C., Madhukar, N. S., & Elemento, O. (2021) · Cancer Discovery · doi:10.1158/2159-8290.cd-21-0090 · Cited by 781
Multimodal deep learning for biomedical data fusion: a review
Stahlschmidt, S. R., Ulfenborg, B., & Synnergren, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab569 · Cited by 766
“Different approaches have been proposed, such as multi-task learning [70], generative models [63] and multimodal dropout [78, 101].”
BackgroundMethodYang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
Harnessing multimodal data integration to advance precision oncology
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., & Shah, S. P. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-021-00408-3 · Cited by 662
Metadata
- Authors
- Cheerla, A., & Gevaert, O.
- Affiliations
- Monta Vista High School, Cupertino, CA, USA
- Stanford University (US)
- Corresponding author
- Gevaert, Olivier
- ORCID
- Gevaert, Olivier
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-07
- Volume
- 35
- Issue
- 14
- Pages
- i446-i454
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC6612862
- OpenAlex
- W2954499361
- Cited by
- 421
- References
- 52
- Citations by year
- 2026: 69 · 2025: 74 · 2024: 68 · 2023: 64 · 2022: 59 · 2021: 69 · 2020: 17 · 2019: 1
- FWCI
- 16.65
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion, cc-by-nc)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (R01EB020527)
- National Institute of Dental and Craniofacial Research (U01DE025188)
- National Cancer Institute (U01CA199241, U01CA217851)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- cancer prognosis prediction, pan-cancer analysis, multimodal data integration, overall survival prediction, clinical data, mRNA expression, microRNA expression, convolutional neural networks, missing data handling, precision oncology
- MeSH terms
- Deep Learning, Humans, Neoplasms, Neural Networks, Computer, Genome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
