Deep learning-based survival prediction for multiple cancer types using histopathology images
Open accessThe DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Providing prognostic information at the time of cancer diagnosis has important implications for treatment and monitoring. Although cancer staging, histopathological assessment, molecular features, and clinical variables can provide useful prognostic insights, improving risk stratification remains an active research area. We developed a deep learning system (DLS) to predict disease specific survival across 10 cancer types from The Cancer Genome Atlas (TCGA). We used a weakly-supervised approach without pixel-level annotations, and tested three different survival loss functions. The DLS was developed using 9,086 slides from 3,664 cases and evaluated using 3,009 slides from 1,216 cases. In multivariable Cox regression analysis of the combined cohort including all 10 cancers, the DLS was significantly associated with disease specific survival (hazard ratio of 1.58, 95% CI 1.28-1.70, p<0.0001) after adjusting for cancer type, stage, age, and sex. In a per-cancer adjusted subanalysis, the DLS remained a significant predictor of survival in 5 of 10 cancer types. Compared to a baseline model including stage, age, and sex, the c-index of the model demonstrated an absolute 3.7% improvement (95% CI 1.0-6.5) in the combined cohort. Additionally, our models stratified patients within individual cancer stages, particularly stage II (p = 0.025) and stage III (p<0.001). By developing and evaluating prognostic models across multiple cancer types, this work represents one of the most comprehensive studies exploring the direct prediction of clinical outcomes using deep learning and histopathology images. Our analysis demonstrates the potential for this approach to provide significant prognostic information in multiple cancer types, and even within specific pathologic stages. However, given the relatively small number of cases and observed clinical events for a deep learning task of this type, we observed wide confidence intervals for model performance, thus highlighting that future work will benefit from larger datasets assembled for the purposes for survival modeling.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Automated image analysis, feature recognition, data extraction and deep learning models are everyday reality for the tech giants but have only partially reached precision pathology (51–53).”
From Artificial Intelligence: Present and Future Potential for Solid Organ Transplantation (2022) · cites this paper as [52]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In fact, DL has the ability to enhance human lives by providing additional accuracy in diagnosis, including estimating natural disasters [40], the discovery of new drugs [41], and cancer diagnosis [42–44].”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [43]
In the citing paper
“In addition, using multi-task learning 147 to train models to perform a variety of tasks rather than one narrowly defined task, such as multi-cancer detection from histopathology images 148 , makes them more generally useful and often more robust.”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [148]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Regression Models and Life-Tables
Cox, D. R. (1972) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1972.tb00899.x · Cited by 39,793
Regression Models and Life-Tables
Cox, D. R. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_37 · Cited by 26,297
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · cited 2×
“Our CNN consisted of layers of depth-wise separable convolution layers, similar to the MobileNet [23] CNN architecture.”
MethodInfluentialAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Nishimura, R., Osako, T., Nishiyama, Y., Tashima, R., Nakano, M., Fujisue, M., Toyozumi, Y., & Arima, N. (2013) · Oncology · doi:10.1159/000353099 · Cited by 13,166
The cancer genome atlas pan-cancer analysis project
The Cancer Genome Atlas Research Network, Weinstein, J. N., Collisson, E. A., Mills, G. B., Shaw, K. R. M., Ozenberger, B. A., Ellrott, K., Shmulevich, I., Sander, C., & Stuart, J. M. (2013) · Nature Genetics · doi:10.1038/ng.2764 · Cited by 9,092
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“In recent years, deep learning has been shown to recognize objects [6] and diagnose diseases from medical images [7,8] with impressive accuracy.”
BackgroundSelf-citationLiu, J., Lichtenberg, T., Hoadley, K. A., Poisson, L. M., Lazar, A. J., Cherniack, A. D., Kovatich, A. J., Benz, C. C., Levine, D. A., Lee, A. V., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.052 · Cited by 3,725 · cited 3×
“Clinical data (including approximated disease specific survival) were obtained from the TCGA Pan-Cancer Clinical Data Resource[21] and the Genomic Data Commons.”
MethodInfluentialSelf-citationEvaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490 · cited 1×
“In this analysis, prognosis prediction performance was measured using the c-index, an extension of the AUC for binary outcomes without censorship [30].”
MethodEhteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293 · cited 2×
“In recent years, deep learning has been shown to recognize objects[6] and diagnose diseases from medical images[7,8] with impressive accuracy.”
BackgroundGenomic classification of cutaneous melanoma
Akbani, R., Akdemir, K. C., Aksoy, B. A., Albert, M., Ally, A., Amin, S. B., Arachchi, H., Arora, A., Auman, J. T., Ayala, B., et al. (2015) · Cell · doi:10.1016/j.cell.2015.05.044 · Cited by 3,151
Adjuvant Chemotherapy Guided by a 21-Gene Expression Assay in Breast Cancer
Sparano, J. A., Gray, R. J., Makower, D. F., Pritchard, K. I., Albain, K. S., Hayes, D. F., Geyer, C. E., Dees, E. C., Goetz, M. P., Olson, J. A., et al. (2018) · New England Journal of Medicine · doi:10.1056/nejmoa1804710 · Cited by 2,397 · cited 1×
“Although TNM staging is useful and well-studied, there is room for improvement in some settings, with ongoing efforts to develop improved prediction strategies that incorporate information such as clinical variables [1,2], genetic information [3,4], and histomorphological features including tumor grade [5].”
BackgroundCOX, D. R. (1975) · Biometrika · doi:10.1093/biomet/62.2.269 · Cited by 2,292
Covariance Analysis of Censored Survival Data
Breslow, N. (1974) · Biometrics · doi:10.2307/2529620 · Cited by 1,763 · cited 1×
“In our implementation, we used Breslow's approximation[25] for handling tied event”
MethodPredicting cancer outcomes from histology and genomics using convolutional networks
Mobadersany, P., Yousefi, S., Amgad, M., Gutman, D. A., Barnholtz-Sloan, J. S., Velázquez Vega, J. E., Brat, D. J., & Cooper, L. A. D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1717139115 · Cited by 1,084
Kather, J. N., Krisam, J., Charoentong, P., Luedde, T., Herpel, E., Weis, C.-A., Gaiser, T., Marx, A., Valous, N. A., Ferber, D., et al. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002730 · Cited by 1,082 · cited 2×
“While other machine learning efforts have identified prognostic significance for non-tumor elements[17,32], our observations suggest that at least for our specific models, the morphologic features of the tumor appear to be more relevant than non-tumor regions.”
BackgroundMethodSaltz, J., Gupta, R., Hou, L., Kurc, T., Singh, P., Nguyen, V., Samaras, D., Shroyer, K. R., Zhao, T., Batiste, R., et al. (2018) · Cell Reports · doi:10.1016/j.celrep.2018.03.086 · Cited by 1,027
Self-citationYu, K.-H., Zhang, C., Berry, G. J., Altman, R. B., Ré, C., Rubin, D. L., & Snyder, M. (2016) · Nature Communications · doi:10.1038/ncomms12474 · Cited by 972 · cited 1×
“These prior works have used feature-engineering approaches [13,16], leveraged annotated regions of interest [12,18,19], focused on learning of known histologic features [17] and/ or developed models to directly predict survival for an individual cancer type.”
MethodCOOPERBERG, M. R., PASTA, D. J., ELKIN, E. P., LITWIN, M. S., LATINI, D. M., Du CHANE, J., & CARROLL, P. R. (2005) · Journal of Urology · doi:10.1097/01.ju.0000158155.33890.e7 · Cited by 772 · cited 1×
“Although TNM staging is useful and well-studied, there is room for improvement in some settings, with ongoing efforts to develop improved prediction strategies that incorporate information such as clinical variables[1,2], genetic information[3,4], and histomorphological features including tumor grade[5].”
BackgroundSystematic Analysis of Breast Cancer Morphology Uncovers Stromal Features Associated with Survival
Beck, A. H., Sangoi, A. R., Leung, S., Marinelli, R. J., Nielsen, T. O., van de Vijver, M. J., West, R. B., van de Rijn, M., & Koller, D. (2011) · Science Translational Medicine · doi:10.1126/scitranslmed.3002564 · Cited by 763 · cited 1×
“While other machine learning efforts have identified prognostic significance for non-tumor elements [17,32], our observations suggest that at least for our specific models, the morphologic features of the tumor appear to be more relevant than non-tumor regions.”
BackgroundDeep learning based tissue analysis predicts outcome in colorectal cancer
Bychkov, D., Linder, N., Turkki, R., Nordling, S., Kovanen, P. E., Verrill, C., Walliander, M., Lundin, M., Haglund, C., & Lundin, J. (2018) · Scientific Reports · doi:10.1038/s41598-018-21758-3 · Cited by 644 · cited 2×
“A similar approach has been explored [18] though only for tissue microarrays of a single cancer type and using image features from a frozen model that was trained on ImageNet.”
BackgroundMethodPrognostic Significance of Nottingham Histologic Grade in Invasive Breast Carcinoma
Rakha, E. A., El-Sayed, M. E., Lee, A. H. S., Elston, C. W., Grainge, M. J., Hodi, Z., Blamey, R. W., & Ellis, I. O. (2008) · Journal of Clinical Oncology · doi:10.1200/jco.2007.15.5986 · Cited by 628 · cited 1×
“Although TNM staging is useful and well-studied, there is room for improvement in some settings, with ongoing efforts to develop improved prediction strategies that incorporate information such as clinical variables [1,2], genetic information [3,4], and histomorphological features including tumor grade [5].”
BackgroundClinical and Genomic Risk to Guide the Use of Adjuvant Therapy for Breast Cancer
Sparano, J. A., Gray, R. J., Ravdin, P. M., Makower, D. F., Pritchard, K. I., Albain, K. S., Hayes, D. F., Geyer, C. E., Dees, E. C., Goetz, M. P., et al. (2019) · New England Journal of Medicine · doi:10.1056/nejmoa1904819 · Cited by 532 · cited 1×
“Although TNM staging is useful and well-studied, there is room for improvement in some settings, with ongoing efforts to develop improved prediction strategies that incorporate information such as clinical variables [1,2], genetic information [3,4], and histomorphological features including tumor grade [5].”
BackgroundNagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 1×
“In pathology, prior studies have reported deep learning models with performance on par with human experts for diagnostic tasks such as tumor detection and histologic grading[8–10].”
BackgroundSelf-citationLiu, Y., Kohlberger, T., Norouzi, M., Dahl, G. E., Smith, J. L., Mohtashamian, A., Olson, N., Peng, L. H., Hipp, J. D., & Stumpe, M. C. (2019) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2018-0147-oa · Cited by 394
Self-citationEvaluating the yield of medical tests
Harrell, F. E. (1982) · JAMA · doi:10.1001/jama.247.18.2543 · Cited by 310
WSISA: Making Survival Prediction from Whole Slide Histopathological Images
Zhu, X., Yao, J., Zhu, F., & Huang, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.725 · Cited by 292
PanCancer insights from The Cancer Genome Atlas: the pathologist's perspective
Cooper, L. A., Demicco, E. G., Saltz, J. H., Powell, R. T., Rao, A., & Lazar, A. J. (2018) · The Journal of Pathology · doi:10.1002/path.5028 · Cited by 228
Predicting Survival After Curative Colectomy for Cancer: Individualizing Colon Cancer Staging
Weiser, M. R., Gönen, M., Chou, J. F., Kattan, M. W., & Schrag, D. (2011) · Journal of Clinical Oncology · doi:10.1200/jco.2011.36.5080 · Cited by 177 · cited 1×
“Although TNM staging is useful and well-studied, there is room for improvement in some settings, with ongoing efforts to develop improved prediction strategies that incorporate information such as clinical variables [1,2], genetic information [3,4], and histomorphological features including tumor grade [5].”
BackgroundBenz, C. C., Hoadley, K. A., Thorsson, V., Lazar, A. J., Levine, D. A., Lee, A. V., Cherniack, A. D., Wolf, D. M., Poisson, L. M., Omberg, L., et al. (2021) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/e4tc-a065 · Cited by 169
CamDavidsonPilon/lifelines: 0.15.3
Davidson-Pilon, C., Kalderstam, J., Kuhn, B., Zivich, P., Fiore-Gartland, A., Moneda, L., Parij, A., Stark, K., Anton, S., Besson, L., et al. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.805993 · Cited by 167
Association of Omics Features with Histopathology Patterns in Lung Adenocarcinoma
Yu, K.-H., Berry, G. J., Rubin, D. L., Ré, C., Altman, R. B., & Snyder, M. (2017) · Cell Systems · doi:10.1016/j.cels.2017.10.014 · Cited by 118 · cited 1×
“These prior works have used feature-engineering approaches [13,16], leveraged annotated regions of interest [12,18,19], focused on learning of known histologic features [17] and/ or developed models to directly predict survival for an individual cancer type.”
MethodWang, C., Pécot, T., Zynger, D. L., Machiraju, R., Shapiro, C. L., & Huang, K. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001538 · Cited by 47
CamDavidsonPilon/lifelines: v0.24.8
Davidson-Pilon, C., Kalderstam, J., Jacobson, N. T., Zivich, P. N., Kuhn, B., Williamson, M., sean-reed, AbdealiJK, Fiore-Gartland, A., Datta, D., et al. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3833188 · Cited by 32
CamDavidsonPilon/lifelines: v0.21.3
Davidson-Pilon, C., Kalderstam, J., Zivich, P. N., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3240536 · Cited by 10
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4<361::aid-sim168>3.0.co;2-4
Research works printed but not linked: 2
- Y Liu, 2018, Artificial Intelligence-Based Breast Cancer Nodal Metastasis Detection, Arch Pathol Lab Med
- T Tieleman, 2012, Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude, COURSERA: Neural networks for machine learning, 4, 26
Cited by
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Liu, J., Lichtenberg, T., Hoadley, K. A., Poisson, L. M., Lazar, A. J., Cherniack, A. D., Kovatich, A. J., Benz, C. C., Levine, D. A., Lee, A. V., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.052 · Cited by 3,725
Self-citationTowards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560
Deep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406
Self-citationDeep learning in histopathology: the path to the clinic
van der Laak, J., Litjens, G., & Ciompi, F. (2021) · Nature Medicine · doi:10.1038/s41591-021-01343-4 · Cited by 902
Metadata
- Authors
- Wulczyn, E., Steiner, D. F., Xu, Z., Sadhwani, A., Wang, H., Flament-Auvigne, I., Mermel, C. H., Chen, P.-H. C., Liu, Y., & Stumpe, M. C.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Liu, Yun
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-06-17
- Volume
- 15
- Issue
- 6
- Pages
- e0233678
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC7299324
- OpenAlex
- W3036901136
- Cited by
- 266
- References
- 36
- Citations by year
- 2026: 39 · 2025: 33 · 2024: 53 · 2023: 36 · 2022: 51 · 2021: 46 · 2020: 7
- FWCI
- 13.75
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- cancer survival prediction, histopathological images, disease-specific survival, prognostic model, weakly supervised learning, risk stratification, tumor staging, multiple cancer types, TCGA, deep learning
- MeSH terms
- Deep Learning, Adult, Age Factors, Feasibility Studies, Female, Humans, Image Processing, Computer-Assisted, Male, Middle Aged, Neoplasm Staging, Neoplasms, Prognosis, Risk Factors, Sex Factors, Survival Analysis, Risk Assessment, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.