CoChat
Sign Up
Cited by 217Radiomics and Machine Learning in Medical ImagingAI in cancer detectionProstate Cancer Diagnosis and Treatment

The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.

Prostate cancer therapy personalization via multi-modal deep learning on randomized phase III clinical trials

Open access
Esteva, Andre, Feng, Jean, van der Wal, Douwe, Huang, Shih-Cheng, Simko, Jeffry P., DeVries, Sandy, Chen, Emmalyn, Schaeffer, Edward M., Morgan, Todd M., Sun, Yilun, Ghorbani, Amirata, Naik, Nikhil, Nathawani, Dhruv, Socher, Richard, Michalski, Jeff M., Roach, Mack, Pisansky, Thomas M., Monson, Jedidiah M., Naz, Farah, Wallace, James, Ferguson, Michelle J., Bahary, Jean-Paul, Zou, James, Lungren, Matthew, Yeung, Serena, Ross, Ashley E., NRG Prostate Cancer AI Consortium, Kucharczyk, Michael, Souhami, Luis, Ballas, Leslie, Peters, Christopher A., Liu, Sandy, Balogh, Alexander G., Randolph-Jackson, Pamela D., Schwartz, David L., Girvigian, Michael R., Saito, Naoyuki G., Raben, Adam, Rabinovitch, Rachel A., Katato, Khalil, Sandler, Howard M., Tran, Phuoc T., Spratt, Daniel E., Pugh, Stephanie, Feng, Felix Y., Mohamad, Osama·2022·npj Digital Medicine
DOI 10.1038/s41746-022-00613-wPMC PMC9177850

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Prostate cancer is the most frequent cancer in men and a leading cause of cancer death. Determining a patient's optimal therapy is a challenge, where oncologists must select a therapy with the highest likelihood of success and the lowest likelihood of toxicity. International standards for prognostication rely on non-specific and semi-quantitative tools, commonly leading to over- and under-treatment. Tissue-based molecular biomarkers have attempted to address this, but most have limited validation in prospective randomized trials and expensive processing costs, posing substantial barriers to widespread adoption. There remains a significant need for accurate and scalable tools to support therapy personalization. Here we demonstrate prostate cancer therapy personalization by predicting long-term, clinically relevant outcomes using a multimodal deep learning architecture and train models using clinical data and digital histopathology from prostate biopsies. We train and validate models using five phase III randomized trials conducted across hundreds of clinical centers. Histopathological data was available for 5654 of 7764 randomized patients (71%) with a median follow-up of 11.4 years. Compared to the most common risk-stratification tool-risk groups developed by the National Cancer Center Network (NCCN)-our models have superior discriminatory performance across all endpoints, ranging from 9.2% to 14.6% relative improvement in a held-out validation set. This artificial intelligence-based tool improves prognostication over standard tools and allows oncologists to computationally predict the likeliest outcomes of specific patients to determine optimal treatment. Outfitted with digital scanners and internet access, any clinic could offer such capabilities, enabling global access to therapy personalization.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-022-00613-w

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Multimodal deep learning system and dataset. a The multimodal architecture is composed of two parts: a tower stack to parse a variable number of digital histopathology slides and another tower stack to merge the resultant features and predict binary outcomes. b The training of the self-supervised model of the image tower stack.
Fig. 1 Multimodal deep learning system and dataset. a The multimodal architecture is composed of two parts: a tower stack to parse a variable number of digital histopathology slides and another tower stack to merge the resultant features and predict binary outcomes. b The training of the self-supervised model of the image tower stack.CC BY · PubMed Central
Fig. 2 Pathologist interpretation of self-supervised model tissue clusters. The self-supervised model in the multimodal model was trained to identify whether or not augmented versions of small patches of tissue came from the same original patch, without ever seeing clinical data labels. After training, each image patch in the dataset of 10.05 M image patches was fed through this model to extract a 128-dimensional feature vector, and the UMAP algorithm27 was used to cluster and visualize the resultant vectors. A pathologist was then asked to interpret the 20 image patches closest to each of the 25 cluster centroids—the descriptions are shown next to the insets. For clarity, we only highlight 6 clusters (colored), and show the remaining clusters in gray. See Supplementary Fig. 2 for full pathologist annotation.
Fig. 2 Pathologist interpretation of self-supervised model tissue clusters. The self-supervised model in the multimodal model was trained to identify whether or not augmented versions of small patches of tissue came from the same original patch, without ever seeing clinical data labels. After training, each image patch in the dataset of 10.05 M image patches was fed through this model to extract a 128-dimensional feature vector, and the UMAP algorithm27 was used to cluster and visualize the resultant vectors. A pathologist was then asked to interpret the 20 image patches closest to each of the 25 cluster centroids—the descriptions are shown next to the insets. For clarity, we only highlight 6 clusters (colored), and show the remaining clusters in gray. See Supplementary Fig. 2 for full pathologist annotation.CC BY · PubMed Central
Fig. 3 Comparison of the multimodal deep learning system to NCCN risk groups across trials and outcomes. a Performance results reporting on the area under the curve (AUC) of time-dependent receiver operator characteristics of the MMAI (blue bars) vs. NCCN (gray bars) models, include 95% confidence intervals and two-sided p-values. Comparisons were made across 5-year and 10-year time points on the following binary outcomes: distant metastasis (DM), biochemical failure (BF), prostate cancer-specific survival (PCSS), and overall survival (OS). b Summary table of the relative improvement of the MMAI model over the NCCN model across the various outcomes and broken down by performance on the data from each trial in the validation set. Relative improvement is given by (AUCMMAI − AUCNCCN)/AUCNCCN. c Ablation study showing model performance when trained on a sequentially decreasing set of data inputs, including the pathology images only (path), pathology images + NCCN variables (path + NCCN), and pathology images + NCCN variables + age + Gleason primary + Gleason secondary (path + NCCN + 3). d–h Performance comparison on the individual clinical trial subsets of the validation set—together, these five comprise the entire validation set shown in (a).
Fig. 3 Comparison of the multimodal deep learning system to NCCN risk groups across trials and outcomes. a Performance results reporting on the area under the curve (AUC) of time-dependent receiver operator characteristics of the MMAI (blue bars) vs. NCCN (gray bars) models, include 95% confidence intervals and two-sided p-values. Comparisons were made across 5-year and 10-year time points on the following binary outcomes: distant metastasis (DM), biochemical failure (BF), prostate cancer-specific survival (PCSS), and overall survival (OS). b Summary table of the relative improvement of the MMAI model over the NCCN model across the various outcomes and broken down by performance on the data from each trial in the validation set. Relative improvement is given by (AUCMMAI − AUCNCCN)/AUCNCCN. c Ablation study showing model performance when trained on a sequentially decreasing set of data inputs, including the pathology images only (path), pathology images + NCCN variables (path + NCCN), and pathology images + NCCN variables + age + Gleason primary + Gleason secondary (path + NCCN + 3). d–h Performance comparison on the individual clinical trial subsets of the validation set—together, these five comprise the entire validation set shown in (a).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

Sorted by citations

See all 217

Related

See all

Metadata

Authors
Esteva, A., Feng, J., van der Wal, D., Huang, S.-C., Simko, J. P., DeVries, S., Chen, E., Schaeffer, E. M., Morgan, T. M., Sun, Y., Ghorbani, A., Naik, N., Nathawani, D., Socher, R., Michalski, J. M., Roach, M., Pisansky, T. M., Monson, J. M., Naz, F., Wallace, J., Ferguson, M. J., Bahary, J.-P., Zou, J., Lungren, M., Yeung, S., Ross, A. E., NRG Prostate Cancer AI Consortium, Kucharczyk, M., Souhami, L., Ballas, L., Peters, C. A., Liu, S., Balogh, A. G., Randolph-Jackson, P. D., Schwartz, D. L., Girvigian, M. R., Saito, N. G., Raben, A., Rabinovitch, R. A., Katato, K., Sandler, H. M., Tran, P. T., Spratt, D. E., Pugh, S., Feng, F. Y., & Mohamad, O.
Affiliations
  • University of California, San Francisco (US)
  • Stanford University (US)
  • NRG Oncology (US)
  • Northwestern University (US)
  • University of Michigan (US)
  • Case Western Reserve University (US)
  • Salesforce (United States) (US)
  • Washington University in St. Louis (US)
  • Mayo Clinic (US)
  • California Cancer Associates for Research and Excellence (US)
  • Saint John Regional Hospital (CA)
  • Ingalls Memorial Hospital (US)
  • Centre Hospitalier de l’Université de Montréal (CA)
  • Nova Scotia Cancer Centre (CA)
  • University of Southern California (US)
  • Northeast Radiation Oncology Center (US)
  • Ronald Reagan UCLA Medical Center (US)
  • VA NY Harbor Healthcare System (US)
  • Kaiser Permanente (US)
  • Indiana University School of Medicine (US)
  • Christiana Care Health System (US)
  • University of Colorado Hospital (US)
  • Hurley Medical Center (US)
  • Cedars-Sinai Medical Center (US)
  • University of Maryland, Baltimore (US)
  • University Hospitals Seidman Cancer Center (US)
Corresponding author
Esteva, Andre
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2022-06-08
Volume
5
Issue
1
Pages
71-71
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-022-00613-w
PMC
PMC9177850
OpenAlex
W4282942659
Cited by
217
References
39
Citations by year
2026: 58 · 2025: 85 · 2024: 44 · 2023: 26 · 2022: 4
FWCI
22.55
Type
journal article
Language
en
Update notices
correction (2023-02-22)
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Digital Commons@Becker (Washington University School of Medicine) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
Funders
  • NRG Oncology (U24CA196067, U10 CA180822, UG1CA189867, U10CA180868)
  • National Cancer Institute (UG1CA189867, U10CA180868, U24CA196067, U10CA180822)
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Urology › Prostate Cancer Diagnosis and Treatment
Keywords
prostate cancer, multimodal deep learning, digital histopathology, prostate biopsy, outcome prediction, risk stratification, NCCN risk classification, long-term clinical outcomes, treatment selection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review