CoChat
Sign Up
Cited by 199Radiomics and Machine Learning in Medical ImagingAI in cancer detectionDomain Adaptation and Few-Shot Learning

The DOI matches its Crossref record; 49 of the 54 references it cites were found in a registry or an index; none of the 25 sources its claims cite is checked for support yet.

Foundation model for cancer imaging biomarkers

Open access
Pai, Suraj, Bontempi, Dennis, Hadzic, Ibrahim, Prudente, Vasco, Sokač, Mateo, Chaunzwa, Tafadzwa L., Bernatz, Simon, Hosny, Ahmed, Mak, Raymond H., Birkbak, Nicolai J., Aerts, Hugo J. W. L.·2024·Nature Machine Intelligence
DOI 10.1038/s42256-024-00807-9PMC PMC10957482
Open this paper in CoChat — free accountOpen PDF

Abstract

Foundation models in deep learning are characterized by a single large-scale model trained on vast amounts of data serving as the foundation for various downstream tasks. Foundation models are generally trained using self-supervised learning and excel in reducing the demand for training samples in downstream applications. This is especially important in medicine, where large labelled datasets are often scarce. Here, we developed a foundation model for cancer imaging biomarker discovery by training a convolutional encoder through self-supervised learning using a comprehensive dataset of 11,467 radiographic lesions. The foundation model was evaluated in distinct and clinically relevant applications of cancer imaging-based biomarkers. We found that it facilitated better and more efficient learning of imaging biomarkers and yielded task-specific models that significantly outperformed conventional supervised and other state-of-the-art pretrained implementations on downstream tasks, especially when training dataset sizes were very limited. Furthermore, the foundation model was more stable to input variations and showed strong associations with underlying biology. Our results demonstrate the tremendous potential of foundation models in discovering new imaging biomarkers that may extend to other clinical use cases and can accelerate the widespread translation of imaging biomarkers into clinical settings.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

A 3D ResNet50 foundation model was pretrained with a modified SimCLR objective on 11,467 CT lesions from 2,312 patients and evaluated on anatomical site classification, lung nodule malignancy, and NSCLC prognosis. It improved downstream imaging-biomarker performance, especially with limited training data, and showed stability and biological associations.

Key findings

  • Modified SimCLR pretraining outperformed auto-encoder, SimCLR, SwAV, and NNCLR on lesion anatomical site classification, achieving balanced accuracy 0.779 (95% CI 0.750–0.810) and mAP 0.847 (95% CI 0.750–0.810); SimCLR was second best with balanced accuracy 0.696 (95% CI 0.663–0.728) and mAP 0.779 (95% CI 0.749–0.811).
  • For lesion anatomical site classification (use case 1), Foundation (fine-tuned) reached mAP 0.857 (95% CI 0.828–0.886) and balanced accuracy 0.804 (95% CI 0.775–0.835), significantly outperforming baselines on mAP and most baselines on balanced accuracy; Foundation (features) was especially robust under reduced training data.
  • For lung nodule malignancy (use case 2), Foundation (fine-tuned) achieved AUC 0.944 (95% CI 0.907–0.972) and mAP 0.953 (95% CI 0.915–0.979); Foundation (features) dominated baselines at 50%, 20%, and 10% training data (n = 254, 101, and 51).
  • For NSCLC prognostication (use case 3), Foundation (features) achieved AUC 0.638 (95% CI 0.584–0.692) on LUNG1 and AUC 0.653 (95% CI 0.532–0.771) on RADIO; it was the only model with significant Kaplan–Meier stratification on RADIO (P = 0.009), had test–retest ICC 0.984 versus 0.966 for Supervised (fine-tuned), and showed significantly higher absolute gene-expression correlations (P = 0.008) with immune-associated pathway enrichment.

Methods

  • Design: self-supervised contrastive pretraining of a 3D ResNet50 encoder using a modified SimCLR framework on 11,467 unlabelled CT lesions from 5,513 scans across 2,312 patients; downstream tasks used either frozen features with a logistic regression classifier or full fine-tuning.
  • Data: five datasets—DeepLesion, LUNA16, HarvardRT, LUNG1, and RADIO—covering three use cases: anatomical site classification (3,830 train/tune lesions; 1,221 test), nodule malignancy (338 train, 169 tune, 170 test nodules), and NSCLC 2-year overall survival (HarvardRT n = 291 train/tune; LUNG1 n = 420 and RADIO n = 133 test).
  • Analysis: balanced accuracy and mAP for anatomical site, AUC-ROC for malignancy and prognosis, permutation tests with 1,000 permutations, bootstrap 95% CIs with 1,000 resamples, Kaplan–Meier/log-rank tests, stability analyses (ICC, MSE, perturbation trials), saliency maps, and gene-expression correlation plus gene-set enrichment analysis.

Limitations

  • Stated by the paper: the study is retrospective; reliability across diverse demographic groups, distribution shifts, and biomarker tasks remains to be evaluated; larger pretraining data and integration of clinical covariates may be needed; deep-learning models remain “black box,” and saliency attribution methods have technical limitations; the biological association analysis is preliminary.
  • Evident from the text: Models Genesis was excluded from use case 2 because it was pretrained on the same LUNA16 split, making comparison unfair due to overfitting; RADIO had a small sample size, so most model differences were not significant; reduced-training-data experiments were not performed for use case 3 because that use case already has inherently small sample sizes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s42256-024-00807-9

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 General overview of the study. a, Foundation model pretraining: a foundation model, specifically a deep convolutional encoder, was pretrained by contrasting volumes with and without lesions. b, Clinical application of the foundation model: the foundation model was used to extract biomarkers and subsequently evaluated on three classification tasks using diverse datasets. c, Foundation model implementation approaches: the foundation model was implemented on specific use cases by (1) training a linear classifier on extracted features or (2) through transfer learning by fine-tuning all model parameters. d, Performance evaluations: we compared the performance of the foundation model against supervised models, trained from random initialization and transfer-learned, through fine-tuning, from a different task. Publicly available state-of-the-art models, Med3D and Models Genesis, were also compared against our foundation model using identical implementation approaches. The comparison was made through several criteria for the different use cases, including quantitative performance, stability, biological and efficiency analysis.
Fig. 1 General overview of the study. a, Foundation model pretraining: a foundation model, specifically a deep convolutional encoder, was pretrained by contrasting volumes with and without lesions. b, Clinical application of the foundation model: the foundation model was used to extract biomarkers and subsequently evaluated on three classification tasks using diverse datasets. c, Foundation model implementation approaches: the foundation model was implemented on specific use cases by (1) training a linear classifier on extracted features or (2) through transfer learning by fine-tuning all model parameters. d, Performance evaluations: we compared the performance of the foundation model against supervised models, trained from random initialization and transfer-learned, through fine-tuning, from a different task. Publicly available state-of-the-art models, Med3D and Models Genesis, were also compared against our foundation model using identical implementation approaches. The comparison was made through several criteria for the different use cases, including quantitative performance, stability, biological and efficiency analysis.CC BY · PubMed Central
Fig. 2 Comparison of pretraining strategies and performance evaluation for lesion anatomical site (use case 1) and nodule malignancy classification (use case 2). We determined the best pretraining approach for our foundation model on their ability to extract features that can be linearly classified to best predict lesion anatomical site. a,b, Different pretraining approaches were evaluated using balanced accuracy (BA) (a) and mAP (b). c,d, After pretraining our foundation model using the best strategy, we adapted them to use case 1, lesion anatomical site classification, and compared them against baseline methods using balanced accuracy (c) and mAP (d). We show performance on these metrics aggregated across eight anatomical sites when trained on the full training set and when the training data percentage decreased to 50, 20 and 10%. e,f, Similar to use case 1, we implemented our foundation model on use case 2 and compared it against baseline methods using the AUC-ROC (e) and mAP (f). Both metrics were computed when trained on the full and 50, 20 and 10% of the dataset. In e,f, Models Genesis approaches are shaded and/or dotted as they were trained on the same data split of LUNA16 and therefore do not present a fair comparison due to overfitting. For use case 2, we also added a supervised model fine-tuned through transfer learning from use case 1. The error bars for a–f show 95% CIs of the estimates and the bar centre shows the mean estimate of the displayed metric. The estimates were computed by generating a bootstrap distribution with 1,000 resamples for datasets with n = 1,221 samples (a–d) and n = 170 samples (e,f).
Fig. 2 Comparison of pretraining strategies and performance evaluation for lesion anatomical site (use case 1) and nodule malignancy classification (use case 2). We determined the best pretraining approach for our foundation model on their ability to extract features that can be linearly classified to best predict lesion anatomical site. a,b, Different pretraining approaches were evaluated using balanced accuracy (BA) (a) and mAP (b). c,d, After pretraining our foundation model using the best strategy, we adapted them to use case 1, lesion anatomical site classification, and compared them against baseline methods using balanced accuracy (c) and mAP (d). We show performance on these metrics aggregated across eight anatomical sites when trained on the full training set and when the training data percentage decreased to 50, 20 and 10%. e,f, Similar to use case 1, we implemented our foundation model on use case 2 and compared it against baseline methods using the AUC-ROC (e) and mAP (f). Both metrics were computed when trained on the full and 50, 20 and 10% of the dataset. In e,f, Models Genesis approaches are shaded and/or dotted as they were trained on the same data split of LUNA16 and therefore do not present a fair comparison due to overfitting. For use case 2, we also added a supervised model fine-tuned through transfer learning from use case 1. The error bars for a–f show 95% CIs of the estimates and the bar centre shows the mean estimate of the displayed metric. The estimates were computed by generating a bootstrap distribution with 1,000 resamples for datasets with n = 1,221 samples (a–d) and n = 170 samples (e,f).CC BY · PubMed Central
Fig. 3 Performance of the foundation model for prognostication of NSCLC tumours (use case 3). We compared the foundation model implementation approaches against baseline methods using the AUC. a,c, Each implementation was adapted for 2 year overall survival classification, trained on the HarvardRT dataset and evaluated on LUNG1 (a) and RADIO (c) datasets. b,d, Kaplan–Meier curves for groups stratified by model predictions from the best performing among implementation approaches are shown for LUNG1 (b) and RADIO (d). To ensure a fair comparison, we calculated the threshold to split the risk groups on the HarvardRT tuning set for each implementation. Kaplan–Meier curves for all approaches can be found in Extended Data Fig. 6. The 95% CI of the estimates is shown by error bars in a,c and error bands in b,d. The measure of centre for the error bars is the mean estimate of AUC and the measure of centre for the error bands is the Kaplan–Meier estimate of the survival function. The estimates for the bar plots in a and c have been computed through a bootstrap distribution with 1,000 resamples using dataset sizes of n = 420 and n = 133, respectively.
Fig. 3 Performance of the foundation model for prognostication of NSCLC tumours (use case 3). We compared the foundation model implementation approaches against baseline methods using the AUC. a,c, Each implementation was adapted for 2 year overall survival classification, trained on the HarvardRT dataset and evaluated on LUNG1 (a) and RADIO (c) datasets. b,d, Kaplan–Meier curves for groups stratified by model predictions from the best performing among implementation approaches are shown for LUNG1 (b) and RADIO (d). To ensure a fair comparison, we calculated the threshold to split the risk groups on the HarvardRT tuning set for each implementation. Kaplan–Meier curves for all approaches can be found in Extended Data Fig. 6. The 95% CI of the estimates is shown by error bars in a,c and error bands in b,d. The measure of centre for the error bars is the mean estimate of AUC and the measure of centre for the error bands is the Kaplan–Meier estimate of the survival function. The estimates for the bar plots in a and c have been computed through a bootstrap distribution with 1,000 resamples using dataset sizes of n = 420 and n = 133, respectively.CC BY · PubMed Central
Fig. 4 Input and test–retest stability of the foundation model. We analysed input stability on the LUNG1 dataset and test–retest robustness on the RIDER dataset by comparing between Foundation (features) and Supervised (fine-tuned) (best performing, overall for LUNG1 and RADIO use cases). a, We compared ICC between test–retest model predictions on the RIDER dataset (n = 26). b, We further visualize the linearity between flattened features extracted from test and retest scans on the RIDER dataset. c, We show the sampling distribution for input perturbations that are used to simulate inter-reader variability. We perturbed across x, y and z axes, although the distribution is shown only for x and y perturbations for simplicity. d, We compared the stability of the features across models using mean-squared error (MSE) between feature values across all the trials. e, We demonstrated the prognostic stability of models when the input seed point is perturbed, estimated through calculating AUC for 2-year survival from model predictions. The error bars in a represent the 95% CI of the estimates and the bar centre is the mean estimate. For the box plots (d,e), the centre line shows the median, the box edges represent first and third quartiles and the whiskers extend to 1.5 times the inter-quartile range. The distribution of the data is shown alongside the box plot. Each AUC and MSE measure in the box plots (d,e) have been computed on a dataset with n = 422 samples and the distribution of the measures are obtained from 50 independent perturbation trials.
Fig. 4 Input and test–retest stability of the foundation model. We analysed input stability on the LUNG1 dataset and test–retest robustness on the RIDER dataset by comparing between Foundation (features) and Supervised (fine-tuned) (best performing, overall for LUNG1 and RADIO use cases). a, We compared ICC between test–retest model predictions on the RIDER dataset (n = 26). b, We further visualize the linearity between flattened features extracted from test and retest scans on the RIDER dataset. c, We show the sampling distribution for input perturbations that are used to simulate inter-reader variability. We perturbed across x, y and z axes, although the distribution is shown only for x and y perturbations for simplicity. d, We compared the stability of the features across models using mean-squared error (MSE) between feature values across all the trials. e, We demonstrated the prognostic stability of models when the input seed point is perturbed, estimated through calculating AUC for 2-year survival from model predictions. The error bars in a represent the 95% CI of the estimates and the bar centre is the mean estimate. For the box plots (d,e), the centre line shows the median, the box edges represent first and third quartiles and the whiskers extend to 1.5 times the inter-quartile range. The distribution of the data is shown alongside the box plot. Each AUC and MSE measure in the box plots (d,e) have been computed on a dataset with n = 422 samples and the distribution of the measures are obtained from 50 independent perturbation trials.CC BY · PubMed Central
Fig. 5 Saliency maps for fine-tuned foundation models. a–c, We generated gradient-based saliency maps for each of the fine-tuned foundation models from use cases 1 (a), 2 (b) and 3 (c) using smooth guided back-propagation and visualized salient regions on two samples from corresponding test datasets. The first and fourth columns show the central axial slice (50 × 50 mm) of the volume provided as input to the model. The second and fifth columns show isolines for saliency contours overlayed on the image. Finally, the third and sixth columns show saliency maps highlighting areas of the input volume that contribute the most to a change in the output prediction.
Fig. 5 Saliency maps for fine-tuned foundation models. a–c, We generated gradient-based saliency maps for each of the fine-tuned foundation models from use cases 1 (a), 2 (b) and 3 (c) using smooth guided back-propagation and visualized salient regions on two samples from corresponding test datasets. The first and fourth columns show the central axial slice (50 × 50 mm) of the volume provided as input to the model. The second and fifth columns show isolines for saliency contours overlayed on the image. Finally, the third and sixth columns show saliency maps highlighting areas of the input volume that contribute the most to a change in the output prediction.CC BY · PubMed Central
Fig. 6 Underlying biological basis of the foundation model. We compared the Foundation (features) and Supervised (fine-tuned) (best-performing models on the RADIO dataset) model predictions with gene expression profiles. a, Box plot of absolute correlation coefficients (y axis) of selected genes against model predictions (x axis) across n = 130 samples. Statistical significance between the two groups is determined through a two-sided Wilcoxon signed rank test. b, Gene-set enrichment analysis of genes with correlation coefficient greater than 0.1 revealed for the foundation (left) and supervised model predictions (right). Genetic pathways are shown on the y axis, and the gene ratio is shown on the x axis. Gene count and adjusted P values are also shown in the legend. False discovery rates are used to adjust the P values for multiple comparisons. The box plots in a are defined by the median as the centre line, first and third quartiles as the box edges and 1.5 times the inter-quartile range as the whiskers. MHC, major histocompatibility complex.
Fig. 6 Underlying biological basis of the foundation model. We compared the Foundation (features) and Supervised (fine-tuned) (best-performing models on the RADIO dataset) model predictions with gene expression profiles. a, Box plot of absolute correlation coefficients (y axis) of selected genes against model predictions (x axis) across n = 130 samples. Statistical significance between the two groups is determined through a two-sided Wilcoxon signed rank test. b, Gene-set enrichment analysis of genes with correlation coefficient greater than 0.1 revealed for the foundation (left) and supervised model predictions (right). Genetic pathways are shown on the y axis, and the gene ratio is shown on the x axis. Gene count and adjusted P values are also shown in the legend. False discovery rates are used to adjust the P values for multiple comparisons. The box plots in a are defined by the median as the centre line, first and third quartiles as the box edges and 1.5 times the inter-quartile range as the whiskers. MHC, major histocompatibility complex.CC BY · PubMed Central
Extended Data Fig. 1 Visual exploration of the features generated from the foundation and baseline models. Features from the foundation model and each of the baseline models are extracted on the independent test-set for identifying lesion anatomical sites and visualized using several different dimensionality reduction approaches. Approaches chosen aim to avoid biases from parameter selection, therefore, tSNE with different perplexity settings and PCA are used. The x-axis corresponds to dimension 1, and the y-axis to dimension 2 of the dimensionality reduction. The density contours of each class are underlaid to highlight separability between classes in the feature space. It is to be noted that the supervised model was trained with lesion anatomical site labels while all the other models (Foundation, Med3D, ModelsGenesis) were used merely as feature extractors without being trained for the particular label.
Extended Data Fig. 1 Visual exploration of the features generated from the foundation and baseline models. Features from the foundation model and each of the baseline models are extracted on the independent test-set for identifying lesion anatomical sites and visualized using several different dimensionality reduction approaches. Approaches chosen aim to avoid biases from parameter selection, therefore, tSNE with different perplexity settings and PCA are used. The x-axis corresponds to dimension 1, and the y-axis to dimension 2 of the dimensionality reduction. The density contours of each class are underlaid to highlight separability between classes in the feature space. It is to be noted that the supervised model was trained with lesion anatomical site labels while all the other models (Foundation, Med3D, ModelsGenesis) were used merely as feature extractors without being trained for the particular label.CC BY · PubMed Central
Extended Data Fig. 2 Time and memory efficiency of implementation approaches. We compare the two implementation approaches of our foundation model, 1) linear modelling on extracted features, which comprises a feature extraction step followed by the linear modelling step, and 2) transfer learning through a fine-tuning step. a, Training times (in minutes) for each of the three use cases and the three steps are shown. b, Memory usage (GPU VRAM and System RAM) are shown for the feature extraction, linear modelling and fine-tuning steps. Memory usage for each step across use-cases remains mostly constant due to batch processing and high feature dimensionality. All analyses were run with six cores on AMD EPYCTM 7402 P Processor 24-core @ 2.80 GHz. The GPU, which was only used for fine-tuning, was the Quadro RTX 8000. For both CPU and GPU runs, where batch processing was used, a batch size of 32 was chosen.
Extended Data Fig. 2 Time and memory efficiency of implementation approaches. We compare the two implementation approaches of our foundation model, 1) linear modelling on extracted features, which comprises a feature extraction step followed by the linear modelling step, and 2) transfer learning through a fine-tuning step. a, Training times (in minutes) for each of the three use cases and the three steps are shown. b, Memory usage (GPU VRAM and System RAM) are shown for the feature extraction, linear modelling and fine-tuning steps. Memory usage for each step across use-cases remains mostly constant due to batch processing and high feature dimensionality. All analyses were run with six cores on AMD EPYCTM 7402 P Processor 24-core @ 2.80 GHz. The GPU, which was only used for fine-tuning, was the Quadro RTX 8000. For both CPU and GPU runs, where batch processing was used, a batch size of 32 was chosen.CC BY · PubMed Central
Extended Data Fig. 3 Detailed comparison of the foundation model implementations against baseline methods for lesion anatomical site classification. Comparison of the balanced accuracy and mean average precision of the Foundation (Features) and Foundation (Finetuned) against all other methods when using 100%, 50%, 20%, and 10% percent of the training data. For each metric-percentage pair, a p-value heatmap (darker colours show non-significant values) is shown with the foundation models on the y-axis and all other models to compare on the x-axis. In each cell, the increase or decrease in metric value is shown along with the corresponding p-value. p-values between models were compared using the permutation test with N = 1000 permutations conducted for each pair-wise comparison.
Extended Data Fig. 3 Detailed comparison of the foundation model implementations against baseline methods for lesion anatomical site classification. Comparison of the balanced accuracy and mean average precision of the Foundation (Features) and Foundation (Finetuned) against all other methods when using 100%, 50%, 20%, and 10% percent of the training data. For each metric-percentage pair, a p-value heatmap (darker colours show non-significant values) is shown with the foundation models on the y-axis and all other models to compare on the x-axis. In each cell, the increase or decrease in metric value is shown along with the corresponding p-value. p-values between models were compared using the permutation test with N = 1000 permutations conducted for each pair-wise comparison.CC BY · PubMed Central
Extended Data Fig. 4 Anatomical site-wise breakdown of foundation model and baseline method performance. We compare the foundation model against baseline methods across different training data percentages using average precision scores for each anatomical site in the DeepLesion held-out test dataset. This allows us to show the generalizability of approaches across anatomical sites.
Extended Data Fig. 4 Anatomical site-wise breakdown of foundation model and baseline method performance. We compare the foundation model against baseline methods across different training data percentages using average precision scores for each anatomical site in the DeepLesion held-out test dataset. This allows us to show the generalizability of approaches across anatomical sites.CC BY · PubMed Central
Extended Data Fig. 5 Detailed comparison of the foundation model implementations against baseline methods for nodule malignancy classification and NSCLC prognostication. a, Comparison of the area-under-receiver operating curve (AUC) and mean average precision(mAP) of the Foundation (Features) and Foundation (Finetuned) against all other methods when using 100%, 50%, 20%, and 10% percent of the training data on use case 2. b, Comparison of the AUC of the Foundation (Features) and Foundation (Finetuned) against all other models for the LUNG1 (left) and RADIO (right) dataset for use-case 3. For each metric-percentage pair, a p-value heatmap (darker colours show non-significant values) is shown with the foundation models on the y-axis and all other models to compare on the x-axis. In each cell, the increase or decrease in metric value is shown along with the corresponding p-value. p-values between models were compared using the permutation test with N = 1000 permutations conducted for each pair-wise comparison.
Extended Data Fig. 5 Detailed comparison of the foundation model implementations against baseline methods for nodule malignancy classification and NSCLC prognostication. a, Comparison of the area-under-receiver operating curve (AUC) and mean average precision(mAP) of the Foundation (Features) and Foundation (Finetuned) against all other methods when using 100%, 50%, 20%, and 10% percent of the training data on use case 2. b, Comparison of the AUC of the Foundation (Features) and Foundation (Finetuned) against all other models for the LUNG1 (left) and RADIO (right) dataset for use-case 3. For each metric-percentage pair, a p-value heatmap (darker colours show non-significant values) is shown with the foundation models on the y-axis and all other models to compare on the x-axis. In each cell, the increase or decrease in metric value is shown along with the corresponding p-value. p-values between models were compared using the permutation test with N = 1000 permutations conducted for each pair-wise comparison.CC BY · PubMed Central
Extended Data Fig. 6 Survival analysis for all models implemented on NSCLC prognostication. a,b, Kaplan Meier curves on the LUNG1 (a) and RADIO (b) datasets for both the foundation model implementation approaches as well as the baseline comparisons are shown. c,d, In c and d, Hazard ratios (HR), computed through univariate Cox regression, for each of the implementation approaches on the LUNG1 and RADIO datasets are shown using forest plots. For both these analyses, groups are determined based on respective model predictions split on the median of the corresponding HarvardRT tuning set predictions. The error bands in (a,b) represent the 95% confidence interval of the Kaplan-Meier estimates of the survival function. The log-rank test is used to determine significant differences between the groups in the KM analysis. For (c, d), the error bars represent the 95% confidence interval of the hazard ratio, and the p-values are calculated using the Wald test. For the LUNG1 dataset, n = 420 and RADIO, n = 133 samples are used to compute each of the analyses in the plots above.
Extended Data Fig. 6 Survival analysis for all models implemented on NSCLC prognostication. a,b, Kaplan Meier curves on the LUNG1 (a) and RADIO (b) datasets for both the foundation model implementation approaches as well as the baseline comparisons are shown. c,d, In c and d, Hazard ratios (HR), computed through univariate Cox regression, for each of the implementation approaches on the LUNG1 and RADIO datasets are shown using forest plots. For both these analyses, groups are determined based on respective model predictions split on the median of the corresponding HarvardRT tuning set predictions. The error bands in (a,b) represent the 95% confidence interval of the Kaplan-Meier estimates of the survival function. The log-rank test is used to determine significant differences between the groups in the KM analysis. For (c, d), the error bars represent the 95% confidence interval of the hazard ratio, and the p-values are calculated using the Wald test. For the LUNG1 dataset, n = 420 and RADIO, n = 133 samples are used to compute each of the analyses in the plots above.CC BY · PubMed Central
Extended Data Fig. 7 Diameter distribution of DeepLesion. Distribution of diameters in the x and y axes for the DeepLesion training dataset based on RECIST bookmarks identified on key slices. Input dimensions of 50x50x50 mm3 were chosen as they covered 93% and 97% of the distribution in the x and y axes, respectively.
Extended Data Fig. 7 Diameter distribution of DeepLesion. Distribution of diameters in the x and y axes for the DeepLesion training dataset based on RECIST bookmarks identified on key slices. Input dimensions of 50x50x50 mm3 were chosen as they covered 93% and 97% of the distribution in the x and y axes, respectively.CC BY · PubMed Central
Extended Data Fig. 8 Stages of the implementation pipeline. a, We first pre-train using a modified version of the SimCLR on 11.467 lesions. The pre-training process consists of a positive contrastive and a negative contrastive loss component. In the positive contrastive loss, augmentations of the same lesion are made to learn similar features. At the same time, the negative contrastive loss learns different features for volumes with and without lesions. b, In the second stage, for each task, different implementation approaches are followed by adapting the pretrained model by either extracting features from a frozen model followed by linearly predicting a target or by fine-tuning all model weights for predicting a target.
Extended Data Fig. 8 Stages of the implementation pipeline. a, We first pre-train using a modified version of the SimCLR on 11.467 lesions. The pre-training process consists of a positive contrastive and a negative contrastive loss component. In the positive contrastive loss, augmentations of the same lesion are made to learn similar features. At the same time, the negative contrastive loss learns different features for volumes with and without lesions. b, In the second stage, for each task, different implementation approaches are followed by adapting the pretrained model by either extracting features from a frozen model followed by linearly predicting a target or by fine-tuning all model weights for predicting a target.CC BY · PubMed Central
Extended Data Fig. 9 Dataset breakdown. The first table shows the 6 different cohorts used in this study along with eligible scans and patients. A secondary table shows the outcome, sex, and age distribution of each of the cohorts.
Extended Data Fig. 9 Dataset breakdown. The first table shows the 6 different cohorts used in this study along with eligible scans and patients. A secondary table shows the outcome, sex, and age distribution of each of the cohorts.CC BY · PubMed Central

Verification

Claims the paper makes

50 Own result · 16 Not checked yet · 16 Own interpretation · 6 No basis given · 4 Own method

Verify all 25 references — free account

Show all 51 claims

  • It may state two separate claims as one. 25
  • Probably not a substantial claim. 18
  • The quote only partly states this claim. 2
41 of 41

  • The paper relies on this without support.

    Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.

    “However, access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.”

  • The paper relies on this without support.

    There is still limited evidence investigating whether SSL can help train foundation models that learn general, robust and transferrable representations that can act as imaging biomarkers, especially prognostic, for tasks of clinical relevance.

    “There is still limited evidence investigating whether SSL can help train foundation models that learn general, robust and transferrable representations that can act as imaging biomarkers, especially prognostic, for tasks of clinical relevance.”

  • The paper relies on this without support.

    The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.

    “The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.”

  • The paper relies on this without support.

    In scenarios with limited annotated data, many investigators fall back on traditional handcrafted or engineered approaches that limit the scope of discovery.

    “In such scenarios, many investigators fall back on traditional handcrafted or engineered approaches based on defined mathematical and statistical algorithms that analyse attributes such as the shape and texture of objects in images, which limit the scope of discovery.”

  • The paper relies on this without support.

    Following pretraining, foundation models can be applied to task-specific problems, improving generalization, especially in tasks with small datasets.

    “Following pretraining, foundation models can be applied to task-specific problems, improving generalization, especially in tasks with small datasets.”

  • The paper relies on this without support.

    Image-biomarker studies for predicting endpoints such as overall survival on small cohorts largely rely on statistical feature extraction (engineered radiomics) and classical machine learning-based modelling.

    “Image-biomarker studies for predicting endpoints, such as overall survival on small cohorts, largely rely on statistical feature extraction (engineered radiomics) and classical machine learning-based modelling.”

  • The paper relies on this without support.

    These statistical feature extraction approaches require precise 3D segmentations, increasing the annotation burden of these studies.

    “These require precise 3D segmentations for feature extraction, increasing the annotation burden of these studies.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Research works printed but not linked: 8

  1. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics (eds Burstein, J. et al.) 4171–4186 (ACL, 2019).
  2. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. In Proc. 37th International Conference on Machine Learning (eds III, H.D. & Singh, A.) 1597–1607 (PMLR, 2020).
  3. Kumar, D. et al. in Image Analysis and Recognition (eds Karray, F. et al.) 54–62 (Springer, 2017).
  4. Taleb, A. et al. 3D self-supervised methods for medical imaging. Adv. Neural Inf. Process. Syst. 33, 18158–18172 (2020).
  5. Adebayo, J. et al. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems 9505–9515 (Curran Associates, 2018).
  6. Sohn, K. Improved deep metric learning with multi-class N-pair loss objective. In Advances in Neural Information Processing Systems (eds Lee, D. et al.) 1857–1865 (Curran Associates, 2016).
  7. van der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).
  8. Springenberg, J. T., Dosovitskiy, A., Brox, T. & Riedmiller, M. A. Striving for simplicity: the all convolutional net. In 3rd International Conference on Learning Representations Workshop (ICLR, 2015).

Cited by

Sorted by citations

See all 199

Related

See all

Metadata

Authors
Pai, S., Bontempi, D., Hadzic, I., Prudente, V., Sokač, M., Chaunzwa, T. L., Bernatz, S., Hosny, A., Mak, R. H., Birkbak, N. J., & Aerts, H. J. W. L.
Affiliations
  • Brigham and Women's Hospital (US)
  • Aarhus University (DK)
  • Harvard University (US)
Corresponding author
Aerts, Hugo J. W. L.
ORCID
Venue
Nature Machine Intelligence
Publisher
Springer Science and Business Media LLC
Published
2024-03-15
Volume
6
Issue
3
Pages
354-367
ISSN
2522-5839 (ISSN-L)
DOI
10.1038/s42256-024-00807-9
PMC
PMC10957482
OpenAlex
W4392850630
Semantic Scholar
b0dc5a23b2f834660c53c436bd1c3dd9fa01d354
Cited by
199
References
54
Citations by year
2026: 81 · 2025: 98 · 2024: 19
FWCI
50.44
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, cc-by)
Funders
  • European Commission
  • Deutsche Forschungsgemeinschaft (502050303)
  • National Institutes of Health (R35CA22052)
  • National Cancer Institute (U01CA190234, U24CA194354, U54CA274516, U01CA209414)
  • H2020 European Research Council (866504)
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
foundation models, cancer imaging biomarkers, self-supervised learning, convolutional encoder, radiographic lesions, limited labeled data, transfer learning, biomarker discovery, radiology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review