CoChat
Sign Up
Cited by 579Radiomics and Machine Learning in Medical ImagingAI in cancer detectionLung Cancer Diagnosis and Treatment

The DOI matches its Crossref record; 96 of the 96 references it cites were found in a registry or an index.

Deep learning for lung cancer prognostication: A retrospective multi-cohort radiomics study

Open access
Hosny, Ahmed, Parmar, Chintan, Coroller, Thibaud P., Grossmann, Patrick, Zeleznik, Roman, Kumar, Avnish, Bussink, Johan, Gillies, Robert J., Mak, Raymond H., Aerts, Hugo J. W. L.·2018·PLOS Medicine
DOI 10.1371/journal.pmed.1002711PMC PMC6269088
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Non-small-cell lung cancer (NSCLC) patients often demonstrate varying clinical courses and outcomes, even within the same tumor stage. This study explores deep learning applications in medical imaging allowing for the automated quantification of radiographic characteristics and potentially improving patient stratification. METHODS AND FINDINGS: We performed an integrative analysis on 7 independent datasets across 5 institutions totaling 1,194 NSCLC patients (age median = 68.3 years [range 32.5-93.3], survival median = 1.7 years [range 0.0-11.7]). Using external validation in computed tomography (CT) data, we identified prognostic signatures using a 3D convolutional neural network (CNN) for patients treated with radiotherapy (n = 771, age median = 68.0 years [range 32.5-93.3], survival median = 1.3 years [range 0.0-11.7]). We then employed a transfer learning approach to achieve the same for surgery patients (n = 391, age median = 69.1 years [range 37.2-88.0], survival median = 3.1 years [range 0.0-8.8]). We found that the CNN predictions were significantly associated with 2-year overall survival from the start of respective treatment for radiotherapy (area under the receiver operating characteristic curve [AUC] = 0.70 [95% CI 0.63-0.78], p < 0.001) and surgery (AUC = 0.71 [95% CI 0.60-0.82], p < 0.001) patients. The CNN was also able to significantly stratify patients into low and high mortality risk groups in both the radiotherapy (p < 0.001) and surgery (p = 0.03) datasets. Additionally, the CNN was found to significantly outperform random forest models built on clinical parameters-including age, sex, and tumor node metastasis stage-as well as demonstrate high robustness against test-retest (intraclass correlation coefficient = 0.91) and inter-reader (Spearman's rank-order correlation = 0.88) variations. To gain a better understanding of the characteristics captured by the CNN, we identified regions with the most contribution towards predictions and highlighted the importance of tumor-surrounding tissue in patient stratification. We also present preliminary findings on the biological basis of the captured phenotypes as being linked to cell cycle and transcriptional processes. Limitations include the retrospective nature of this study as well as the opaque black box nature of deep learning networks. CONCLUSIONS: Our results provide evidence that deep learning networks may be used for mortality risk stratification based on standard-of-care CT images from NSCLC patients. This evidence motivates future research into better deciphering the clinical and biological basis of deep learning networks as well as validation in prospective data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pmed.1002711

Cites

96 of 96 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 General design of the analytical setup. 3D convolutional neural network is trained end-to-end on the radiotherapy dataset group. This is followed by a transfer learning approach, where the same network is fine-tuned on the surgery dataset group. The training, tuning, and testing of these networks are all carried out on independent datasets as illustrated. Four further experiments are carried out on the networks in order to benchmark their performance against random forest models, assess their stability, identify regions in images responsible for predictions, and finally, explore their biological basis. Numbers outside parentheses refer to the number of patients with survival follow-up per dataset. Numbers within parentheses refer to the number of patients with 2-year overall survival follow-up only. Refer to Methods for patient censoring information and S1 Table for further dataset breakdown and information.
Fig 1 General design of the analytical setup. 3D convolutional neural network is trained end-to-end on the radiotherapy dataset group. This is followed by a transfer learning approach, where the same network is fine-tuned on the surgery dataset group. The training, tuning, and testing of these networks are all carried out on independent datasets as illustrated. Four further experiments are carried out on the networks in order to benchmark their performance against random forest models, assess their stability, identify regions in images responsible for predictions, and finally, explore their biological basis. Numbers outside parentheses refer to the number of patients with survival follow-up per dataset. Numbers within parentheses refer to the number of patients with 2-year overall survival follow-up only. Refer to Methods for patient censoring information and S1 Table for further dataset breakdown and information.CC BY · PubMed Central
Fig 2 Illustration of the convolutional neural network. This network was used to predict overall 2-year survival of patients with non-small-cell lung cancer. The final classifier layer outputs normalized probabilities for both classes (0 = deceased and 1 = alive). Only the weights of the final fully connected layer were fine-tuned during transfer learning. The final convolutional layer (conv4) was used for activation mapping.
Fig 2 Illustration of the convolutional neural network. This network was used to predict overall 2-year survival of patients with non-small-cell lung cancer. The final classifier layer outputs normalized probabilities for both classes (0 = deceased and 1 = alive). Only the weights of the final fully connected layer were fine-tuned during transfer learning. The final convolutional layer (conv4) was used for activation mapping.CC BY · PubMed Central
Fig 3 Prognostic power (AUC) and Kaplan–Meier (KM) curves of deep learning features for both the radiotherapy and surgical networks. (A) AUC plot for the radiotherapy test dataset Maastro (n = 211). (B) KM plot for the Maastro dataset (n = 307). Patients who have been previously excluded for lack of 2-year survival follow-up have been reincluded (S1 Table). To ensure an independent evaluation, the median split is calculated on the radiotherapy tuning dataset Radboud (n = 147) and locked for evaluation on the radiotherapy test dataset Maastro. (C) AUC plot for the surgery test dataset M-SPORE (n = 97). (D) KM plot for the M-SPORE dataset (n = 101). The median split is calculated on the surgery tuning dataset MUMC (n = 90) and locked for evaluation on the surgery test dataset M-SPORE. AUC or ROC-AUC, area under the receiver operating characteristic curve.
Fig 3 Prognostic power (AUC) and Kaplan–Meier (KM) curves of deep learning features for both the radiotherapy and surgical networks. (A) AUC plot for the radiotherapy test dataset Maastro (n = 211). (B) KM plot for the Maastro dataset (n = 307). Patients who have been previously excluded for lack of 2-year survival follow-up have been reincluded (S1 Table). To ensure an independent evaluation, the median split is calculated on the radiotherapy tuning dataset Radboud (n = 147) and locked for evaluation on the radiotherapy test dataset Maastro. (C) AUC plot for the surgery test dataset M-SPORE (n = 97). (D) KM plot for the M-SPORE dataset (n = 101). The median split is calculated on the surgery tuning dataset MUMC (n = 90) and locked for evaluation on the surgery test dataset M-SPORE. AUC or ROC-AUC, area under the receiver operating characteristic curve.CC BY · PubMed Central
Fig 4 Activation mapping. Visual highlights of the most “important” regions within the input image—those with the most contribution to maximizing the outputs of the final prediction layer. The rows represent 4 randomly selected samples. From the left, the first column represents the central axial slice of the network input (150 × 150 mm) with tumor annotations. In the second column, a 50 × 50 mm patch is cropped around the tumor. In the third column, activation contours are overlaid, with blue and red showing the lowest and highest contributions (gradients), respectively. Column 4 represents the activation heatmaps for a better visual reference. While the heatmaps are 3D, only the central axial slice is shown. Therefore, the entire color spectrum might not be fully visualized.
Fig 4 Activation mapping. Visual highlights of the most “important” regions within the input image—those with the most contribution to maximizing the outputs of the final prediction layer. The rows represent 4 randomly selected samples. From the left, the first column represents the central axial slice of the network input (150 × 150 mm) with tumor annotations. In the second column, a 50 × 50 mm patch is cropped around the tumor. In the third column, activation contours are overlaid, with blue and red showing the lowest and highest contributions (gradients), respectively. Column 4 represents the activation heatmaps for a better visual reference. While the heatmaps are 3D, only the central axial slice is shown. Therefore, the entire color spectrum might not be fully visualized.CC BY · PubMed Central
Fig 5 Global gene set expression patterns—Moffitt dataset. The deep learning network predictions on the surgery training dataset Moffitt were linked to global gene expression patterns using a pre-ranked gene set enrichment analysis (GSEA). Negative and positive enrichments are shown in red and blue, respectively. The top 10 enrichments in each category are highlighted. See S1 File for full ranking and associated enrichment scores.
Fig 5 Global gene set expression patterns—Moffitt dataset. The deep learning network predictions on the surgery training dataset Moffitt were linked to global gene expression patterns using a pre-ranked gene set enrichment analysis (GSEA). Negative and positive enrichments are shown in red and blue, respectively. The top 10 enrichments in each category are highlighted. See S1 File for full ranking and associated enrichment scores.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 95 references

Research works printed but not linked: 1

  1. Maas AL, Hannun AY, Ng AY. Rectifier nonlinearities improve neural network acoustic models. 30th International Conference on Machine Learning; 2013 Jun 16–21; Atlanta, GA, US.

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Hosny, A., Parmar, C., Coroller, T. P., Grossmann, P., Zeleznik, R., Kumar, A., Bussink, J., Gillies, R. J., Mak, R. H., & Aerts, H. J. W. L.
Affiliations
  • Brigham and Women's Hospital (US)
  • Radboud University Medical Center (NL)
  • Moffitt Cancer Center (US)
Corresponding author
Aerts, Hugo J. W. L.
ORCID
Venue
PLOS Medicine
Publisher
Public Library of Science (PLoS)
Published
2018-11-30
Volume
15
Issue
11
Pages
e1002711
ISSN
  • 1549-1277 (ISSN-L)
  • 1549-1676
DOI
10.1371/journal.pmed.1002711
PMC
PMC6269088
OpenAlex
W2903150666
Cited by
579
References
95
Citations by year
2026: 30 · 2025: 61 · 2024: 78 · 2023: 93 · 2022: 95 · 2021: 103 · 2020: 84 · 2019: 33 · 2018: 1
FWCI
38.12
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS Medicine (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
Funders
National Institutes of Health (U24CA194354, U01CA190234)
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Pulmonary and Respiratory Medicine › Lung Cancer Diagnosis and Treatment
Keywords
non-small cell lung cancer, outcome prediction, radiomics, 3D convolutional neural networks, computed tomography, transfer learning, overall survival, radiotherapy, surgical resection, patient stratification, deep learning interpretability, random forest comparison, prognostic signature
MeSH terms
Clinical Decision-Making, Preliminary Data, Deep Learning, Adult, Aged, Aged, 80 and over, Carcinoma, Non-Small-Cell Lung, Diagnosis, Computer-Assisted, Female, Humans, Lung Neoplasms, Male, Middle Aged, Neoplasm Staging, Predictive Value of Tests, Radiographic Image Interpretation, Computer-Assisted, Retrospective Studies, Risk Factors, Tomography, X-Ray Computed, Reproducibility of Results, Risk Assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review