The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Deep learning with multimodal representation for pancancer prognosis prediction
Open accessPublished version of a preprint
Abstract
Motivation: Estimating the future course of patients with cancer lesions is invaluable to physicians; however, current clinical methods fail to effectively use the vast amount of multimodal data that is available for cancer patients. To tackle this problem, we constructed a multimodal neural network-based model to predict the survival of patients for 20 different cancer types using clinical data, mRNA expression data, microRNA expression data and histopathology whole slide images (WSIs). We developed an unsupervised encoder to compress these four data modalities into a single feature vector for each patient, handling missing data through a resilient, multimodal dropout method. Encoding methods were tailored to each data type—using deep highway networks to extract features from clinical and genomic data, and convolutional neural networks to extract features from WSIs. Results: We used pancancer data to train these feature encodings and predict single cancer and pancancer overall survival, achieving a C-index of 0.78 overall. This work shows that it is possible to build a pancancer model for prognosis that also predicts prognosis in single cancer sites. Furthermore, our model handles multiple data modalities, efficiently analyzes WSIs and represents patient multimodal data flexibly into an unsupervised, informative representation. We thus present a powerful automated tool to accurately determine prognosis, a key step towards personalized treatment for cancer patients. Availability and implementation: https://github.com/gevaertlab/MultimodalPrognosis
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Different approaches have been proposed, such as multi-task learning [70], generative models [63] and multimodal dropout [78, 101].”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [78]
The cited source says what the claim attributes to it.
In the citing paper
“Cheerla and Gevaert (2019) constructed a multimodal neural network-based model to predict the survival of patients for 20 different cancer types using clinical data, mRNA expression data, microRNA expression data, and histopathology whole slide images (WSIs).”
From Navigating the landscape of multimodal AI in medicine: A scoping review on technical challenges and clinical applications (2025) · cites this paper as Cheerla and Gevaert, 2019
In the citing paper
“Indeed, it has been shown that training a model on multiple tissues or cancer variants can improve detection performance for each cancer53, as many morphological features are observed across cancers from different topographies54.”
From A foundation model for clinical-grade computational pathology and rare cancers detection (2024) · cites this paper as [53]
In the citing paper
“although we posit that increased diversity helps to learn robust and rare features. Indeed, it has been shown that training a model on multiple tissues or cancer variants can improve detection performance for each cancer53, as many morphological features are observed across cancers from different topographies54.”
From A foundation model for clinical-grade computational pathology and rare cancers detection (2024) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
(2008) · Journal of Machine Learning Research · Cited by 33,051
The cancer genome atlas pan-cancer analysis project
The Cancer Genome Atlas Research Network, Weinstein, J. N., Collisson, E. A., Mills, G. B., Shaw, K. R. M., Ozenberger, B. A., Ellrott, K., Shmulevich, I., Sander, C., & Stuart, J. M. (2013) · Nature Genetics · doi:10.1038/ng.2764 · Cited by 9,092
MicroRNA signatures in human cancers
Calin, G. A., & Croce, C. M. (2006) · Nature reviews. Cancer · doi:10.1038/nrc1997 · Cited by 7,668
Cox, D. R., & Oakes, D. (2018) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315137438 · Cited by 6,269
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201
Oncomirs—MicroRNAs with a role in cancer
Esquela-Kerscher, A., & Slack, F. J. (2006) · Nature Reviews Cancer · doi:10.1038/nrc1840 · Cited by 5,884
Learning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036
Flagged first, then by citations
Research works printed but not linked: 4
- Gevaert, 2013, Pacific Symposium on Biocomputing 2013, 123
- Gevaert, 2008, Pacific Symposium on Biocomputing 2008, 279
- Momeni, 2018, International MICCAI Brainlesion Workshop, 407
- Yao, 2016, International Conference on Medical Image Computing and Computer-Assisted Intervention, 649
Cited by
Sorted by citations
Deep learning in cancer diagnosis, prognosis and treatment selection
Tran, K. A., Kondrashova, O., Bradley, A., Williams, E. D., Pearson, J. V., & Waddell, N. (2021) · Genome Medicine · doi:10.1186/s13073-021-00968-x · Cited by 932
Artificial Intelligence in Cancer Research and Precision Medicine
Bhinder, B., Gilvary, C., Madhukar, N. S., & Elemento, O. (2021) · Cancer Discovery · doi:10.1158/2159-8290.cd-21-0090 · Cited by 781
Multimodal deep learning for biomedical data fusion: a review
Stahlschmidt, S. R., Ulfenborg, B., & Synnergren, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab569 · Cited by 766
“Different approaches have been proposed, such as multi-task learning [70], generative models [63] and multimodal dropout [78, 101].”
BackgroundMethodYang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
Harnessing multimodal data integration to advance precision oncology
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., & Shah, S. P. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-021-00408-3 · Cited by 662
Metadata
- Authors
- Cheerla, A., & Gevaert, O.
- Affiliations
- Monta Vista High School, Cupertino, CA, USA
- Stanford University (US)
- Corresponding author
- Gevaert, Olivier
- ORCID
- Gevaert, Olivier
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-07
- Volume
- 35
- Issue
- 14
- Pages
- i446-i454
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC6612862
- OpenAlex
- W2954499361
- Cited by
- 421
- References
- 52
- Citations by year
- 2026: 69 · 2025: 74 · 2024: 68 · 2023: 64 · 2022: 59 · 2021: 69 · 2020: 17 · 2019: 1
- FWCI
- 16.65
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion, cc-by-nc)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health (R01EB020527)
- National Institute of Dental and Craniofacial Research (U01DE025188)
- National Cancer Institute (U01CA199241, U01CA217851)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- cancer prognosis prediction, pan-cancer analysis, multimodal data integration, overall survival prediction, clinical data, mRNA expression, microRNA expression, convolutional neural networks, missing data handling, precision oncology
- MeSH terms
- Deep Learning, Humans, Neoplasms, Neural Networks, Computer, Genome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
