CoChat
Sign Up
Cited by 199Radiomics and Machine Learning in Medical ImagingAI in cancer detectionDomain Adaptation and Few-Shot Learning

Foundation Model for Cancer Imaging Biomarkers

Open access
Pai, Suraj, Bontempi, Dennis, Hadzic, Ibrahim, Prudente, Vasco, Sokač, Mateo, Chaunzwa, Tafadzwa L., Bernatz, Simon, Hosny, Ahmed +3 more·2024·Nature Machine Intelligence
DOI 10.1038/s42256-024-00807-9PMC PMC10957482
Open this paper in CoChat — free accountOpen PDF

Abstract

Foundation models in deep learning are characterized by a single large-scale model trained on vast amounts of data serving as the foundation for various downstream tasks. Foundation models are generally trained using self-supervised learning and excel in reducing the demand for training samples in downstream applications. This is especially important in medicine, where large labelled datasets are often scarce. Here, we developed a foundation model for cancer imaging biomarker discovery by training a convolutional encoder through self-supervised learning using a comprehensive dataset of 11,467 radiographic lesions. The foundation model was evaluated in distinct and clinically relevant applications of cancer imaging-based biomarkers. We found that it facilitated better and more efficient learning of imaging biomarkers and yielded task-specific models that significantly outperformed conventional supervised and other state-of-the-art pretrained implementations on downstream tasks, especially when training dataset sizes were very limited. Furthermore, the foundation model was more stable to input variations and showed strong associations with underlying biology. Our results demonstrate the tremendous potential of foundation models in discovering new imaging biomarkers that may extend to other clinical use cases and can accelerate the widespread translation of imaging biomarkers into clinical settings.

AI snapshot

Written from the paper’s full text.

TL;DR

A 3D ResNet50 foundation model was pretrained with a modified SimCLR objective on 11,467 CT lesions from 2,312 patients and evaluated on anatomical site classification, lung nodule malignancy, and NSCLC prognosis. It improved downstream imaging-biomarker performance, especially with limited training data, and showed stability and biological associations.

Key findings

  • Modified SimCLR pretraining outperformed auto-encoder, SimCLR, SwAV, and NNCLR on lesion anatomical site classification, achieving balanced accuracy 0.779 (95% CI 0.750–0.810) and mAP 0.847 (95% CI 0.750–0.810); SimCLR was second best with balanced accuracy 0.696 (95% CI 0.663–0.728) and mAP 0.779 (95% CI 0.749–0.811).
  • For lesion anatomical site classification (use case 1), Foundation (fine-tuned) reached mAP 0.857 (95% CI 0.828–0.886) and balanced accuracy 0.804 (95% CI 0.775–0.835), significantly outperforming baselines on mAP and most baselines on balanced accuracy; Foundation (features) was especially robust under reduced training data.
  • For lung nodule malignancy (use case 2), Foundation (fine-tuned) achieved AUC 0.944 (95% CI 0.907–0.972) and mAP 0.953 (95% CI 0.915–0.979); Foundation (features) dominated baselines at 50%, 20%, and 10% training data (n = 254, 101, and 51).
  • For NSCLC prognostication (use case 3), Foundation (features) achieved AUC 0.638 (95% CI 0.584–0.692) on LUNG1 and AUC 0.653 (95% CI 0.532–0.771) on RADIO; it was the only model with significant Kaplan–Meier stratification on RADIO (P = 0.009), had test–retest ICC 0.984 versus 0.966 for Supervised (fine-tuned), and showed significantly higher absolute gene-expression correlations (P = 0.008) with immune-associated pathway enrichment.

Methods

  • Design: self-supervised contrastive pretraining of a 3D ResNet50 encoder using a modified SimCLR framework on 11,467 unlabelled CT lesions from 5,513 scans across 2,312 patients; downstream tasks used either frozen features with a logistic regression classifier or full fine-tuning.
  • Data: five datasets—DeepLesion, LUNA16, HarvardRT, LUNG1, and RADIO—covering three use cases: anatomical site classification (3,830 train/tune lesions; 1,221 test), nodule malignancy (338 train, 169 tune, 170 test nodules), and NSCLC 2-year overall survival (HarvardRT n = 291 train/tune; LUNG1 n = 420 and RADIO n = 133 test).
  • Analysis: balanced accuracy and mAP for anatomical site, AUC-ROC for malignancy and prognosis, permutation tests with 1,000 permutations, bootstrap 95% CIs with 1,000 resamples, Kaplan–Meier/log-rank tests, stability analyses (ICC, MSE, perturbation trials), saliency maps, and gene-expression correlation plus gene-set enrichment analysis.

Limitations

  • Stated by the paper: the study is retrospective; reliability across diverse demographic groups, distribution shifts, and biomarker tasks remains to be evaluated; larger pretraining data and integration of clinical covariates may be needed; deep-learning models remain “black box,” and saliency attribution methods have technical limitations; the biological association analysis is preliminary.
  • Evident from the text: Models Genesis was excluded from use case 2 because it was pretrained on the same LUNA16 split, making comparison unfair due to overfitting; RADIO had a small sample size, so most model differences were not significant; reduced-training-data experiments were not performed for use case 3 because that use case already has inherently small sample sizes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s42256-024-00807-9

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 25 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Features from the pretrained foundation model provided consistently interpretable and well-separated clusters across different settings.

    “We observed that features from the pretrained foundation model provided consistently interpretable and well-separated clusters across different settings.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.
  • Findingown data

    None of the deep-learning fine-tuned baselines significantly improve over linear classification for nodule malignancy prediction.

    “Notably, none of the deep-learning fine-tuned baselines significantly improve over linear classification.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.
  • Findingown data

    When considering only mAP, the simple linear classification of foundation model features significantly (P < 0.05) outperformed all other implementations.

    “Moreover, when considering only mAP, the simple linear classification significantly (P < 0.05) outperformed all other implementations.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.
  • Findingown data

    In use case 1, the saliency focus was mainly on areas surrounding the lesions, such as the parenchyma and bone regions in the lung and the trachea in mediastinal lesions.

    “Specifically, in use case 1 (Fig. 5a), the focus was mainly on areas surrounding the lesions, such as the parenchyma and bone regions in the lung and the trachea in mediastinal lesions.”
    Quote in the paperNot a substantial claim
  • Findingown data

    In the RADIO cohort, deep-learning implementations did not demonstrate superior performance.

    “Similar to the LUNG1 cohort, deep-learning implementations did not demonstrate superior performance (Fig. 3c).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.
    • Rests on an assumption: Image-biomarker studies for predicting endpoints such as overall survival on small cohorts largely rely on statistical feature extraction (engineered radiomics) and classical machine learning-based modelling.
    • Rests on an assumption: These statistical feature extraction approaches require precise 3D segmentations, increasing the annotation burden of these studies.
  • May combine two claims. 25
  • Classified as probably not a substantial claim. 18
  • The quote only partly states this claim. 2

  • May combine two claims.

    Findingown data

    The supervised model showed enrichment of individual pathways but no identifiable pattern.

    “Conversely, while the supervised model did show enrichment of individual pathways, no identifiable pattern was observed (Fig. 6b).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In use case 3, saliency primarily attributed areas surrounding the centre of mass of the tumour, with some contribution from high-density bone regions.

    “Use case 3 (Fig. 5c) primarily attributed areas surrounding the centre of mass of the tumour, with some contribution from high-density bone regions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Med3D (fine-tuned) achieved AUC = 0.917 (95% CI 0.871-0.957) and mAP = 0.9307 (95% CI 0.888-0.964), performing slightly worse than the foundation model but not significantly (P = 0.134).

    “The implementation Med3D (fine-tuned), with AUC = 0.917 (95% CI 0.871–0.957) and mAP = 0.9307 (95% CI 0.888–0.964), performs slightly worse than our model, but this is not significant (P = 0.134).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.

  • May combine two claims.

    Findingown data

    Using features from the foundation model provided the best performance on small cohorts in predicting prognosis and demonstrated significant stratification of patients by their associated risk for each of the LUNG1 and RADIO cohorts (P < 0.01).

    “Using these features provided the best performance on small cohorts in predicting prognosis and also demonstrated significant stratification of patients by their associated risk for each of the LUNG1 and RADIO cohorts (P < 0.01).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.
    • Rests on an assumption: Image-biomarker studies for predicting endpoints such as overall survival on small cohorts largely rely on statistical feature extraction (engineered radiomics) and classical machine learning-based modelling.
    • Rests on an assumption: These statistical feature extraction approaches require precise 3D segmentations, increasing the annotation burden of these studies.

  • May combine two claims.

    Findingown data

    The fine-tuned foundation model with balanced accuracy 0.804 (95% CI 0.775-0.835) showed significant (P < 0.01) improvement in balanced accuracy over all baselines except Med3D (fine-tuned), where improvement was borderline (P = 0.059).

    “With a balanced accuracy of 0.804 (95% CI 0.775–0.835), a significant (P < 0.01) improvement in balanced accuracy was also observed in comparison to all baselines except Med3D (fine-tuned), where the improvement was borderline (P = 0.059).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.

  • May combine two claims.

    Findingown data

    The modified SimCLR pretraining achieved a balanced accuracy of 0.779 (95% CI 0.750-0.810) and mAP of 0.847 (95% CI 0.750-0.810) on lesion anatomical site classification.

    “achieving a balanced accuracy of 0.779 (95% confidence interval (CI) 0.750–0.810) and mAP = 0.847 (95% CI 0.750–0.810).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Following pretraining, foundation models can be applied to task-specific problems, improving generalization, especially in tasks with small datasets.

  • May combine two claims.

    Findingown data

    In the LUNG1 cohort, all comparisons were significant (P < 0.05) except for Med3D (fine-tuned), where borderline significance was observed (P = 0.053).

    “All comparisons were significant (P < 0.05) except for Med3D (fine-tuned), where borderline significance was observed (P = 0.053).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.
    • Rests on an assumption: Image-biomarker studies for predicting endpoints such as overall survival on small cohorts largely rely on statistical feature extraction (engineered radiomics) and classical machine learning-based modelling.
    • Rests on an assumption: These statistical feature extraction approaches require precise 3D segmentations, increasing the annotation burden of these studies.

  • May combine two claims.

    Findingown data

    In the RADIO cohort, due to small sample size, none of the models showed significant differences from the rest (P > 0.05) except Foundation (features) improving over the Supervised model, which had near-random performance (AUC = 0.520).

    “Due to the small sample size, none of the models showed significant differences from the rest (P > 0.05) except for the Foundation (features) improving over the Supervised model, which had near-random performance (AUC = 0.520).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.
    • Rests on an assumption: Image-biomarker studies for predicting endpoints such as overall survival on small cohorts largely rely on statistical feature extraction (engineered radiomics) and classical machine learning-based modelling.
    • Rests on an assumption: These statistical feature extraction approaches require precise 3D segmentations, increasing the annotation burden of these studies.

  • May combine two claims.

    Findingown data

    The fine-tuned foundation model (Foundation (fine-tuned)) achieved AUC = 0.944 (95% CI 0.907-0.972) and mAP = 0.953 (95% CI 0.915-0.979) and showed significant (P < 0.01) superiority over most baseline implementations for nodule malignancy prediction.

    “The approach of fine-tuning the foundation model, Foundation (fine-tuned), with an area under the curve (AUC) = 0.944 (95% CI 0.907–0.972) and mAP = 0.953 (95% CI 0.915–0.979) resulted in significant (P < 0.01) superiority over most of the baseline implementations (Fig. 2e,f).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.

  • May combine two claims.

    Findingown data

    The method remained robust with the smallest decline in balanced accuracy and mAP of 9 and 12%, respectively, when reducing training data from 100 to 10%.

    “More importantly, it remained robust as evidenced by the smallest decline in balanced accuracy and mAP of 9 and 12%, respectively, when reducing training data from 100 to 10%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Following pretraining, foundation models can be applied to task-specific problems, improving generalization, especially in tasks with small datasets.

  • May combine two claims.

    Findingown data

    Foundation (fine-tuned) showed a larger drop in performance and failed to improve significantly over baseline implementations as training data were decreased, losing significance from 20% onward.

    “Foundation (fine-tuned) showed a larger drop in performance and failed to improve significantly over baseline implementations as training data were decreased (losing significance from 20% onward).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Feature-based implementations were computationally efficient when considering both time and memory.

    “Feature-based implementations were also computationally efficient when considering both time and memory.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.

  • The quote only partly states this claim.

    Findingown data

    For the RADIO cohort, Foundation (features) shows the best performance with an AUC of 0.653 (95% CI 0.532-0.771) for 2-year overall survival prediction.

    “For the RADIO cohort, Foundation (features) shows the best performance with an AUC of 0.653 (95% CI 0.532–0.771).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.
    • Rests on an assumption: Image-biomarker studies for predicting endpoints such as overall survival on small cohorts largely rely on statistical feature extraction (engineered radiomics) and classical machine learning-based modelling.
    • Rests on an assumption: These statistical feature extraction approaches require precise 3D segmentations, increasing the annotation burden of these studies.

  • May combine two claims.

    Findingown data

    Absolute correlation coefficients between gene expression profiles and model predictions were significantly higher (P = 0.008) for the foundation model than for the supervised model, indicating a stronger association with underlying tumour biology.

    “We found that absolute correlation coefficients between gene expression profiles and model predictions were significantly higher (P = 0.008) for the foundation model, indicating a stronger association with underlying tumour biology (Fig. 6a).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Modelling using features extracted from the foundation model was the most robust across these use cases when subjected to drops in training data, offering stable performance even when data sizes were considerably reduced, for example, using only 51 samples in use case 2.

    “Modelling using features extracted from the foundation model was the most robust across these use cases when subjected to drops in training data, offering stable performance even when data sizes were considerably reduced, for example, using only 51 samples in use case 2.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.

  • The quote only partly states this claim.

    Findingown data

    When limited data (50, 20 and 10%) was used for downstream task training, the modified SimCLR method demonstrated consistently improved performance.

    “When limited data (50, 20 and 10%) was used for downstream task training, our method demonstrated consistently improved performance.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Following pretraining, foundation models can be applied to task-specific problems, improving generalization, especially in tasks with small datasets.

  • May combine two claims.

    Findingown data

    SimCLR was the second best-performing pretraining approach, with balanced accuracy 0.696 (95% CI 0.663-0.728) and mAP 0.779 (95% CI 0.749-0.811).

    “As expected, the second best-performing approach was SimCLR (balanced accuracy 0.696 (95% CI 0.663–0.728); mAP = 0.779 (95% CI 0.749–0.811)).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Following pretraining, foundation models can be applied to task-specific problems, improving generalization, especially in tasks with small datasets.

  • May combine two claims.

    Findingown data

    Predictions from Foundation (features) and Supervised (fine-tuned) had high test-retest stability with ICC values of 0.984 and 0.966, respectively, on the RIDER dataset.

    “We found that predictions from the overall best-performing models on LUNG1 and RADIO: Foundation (features) and Supervised (fine-tuned) had high stability with intraclass correlation coefficient (ICC) values of 0.984 and 0.966, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Foundation (fine-tuned) showed superior performance over all baselines at 50% training data but showed large drops in performance from a 20% reduction onward for nodule malignancy prediction.

    “Foundation (fine-tuned) shows superior performance over all baselines at 50% but shows large drops in performance from a 20% reduction onward.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    The foundation model surpasses current deep-learning techniques in various tasks while fitting conveniently into existing radiomic research methods.

    “Furthermore, it surpasses current deep-learning techniques in various tasks while fitting conveniently into existing radiomic research methods.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The original temperature value of 0.1 for the NT-Xent loss was the most effective.

    “However, the original value of 0.1 proposed by the original paper was the most effective.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The baseline Models Genesis implementation was excluded from nodule malignancy analysis because it was pretrained on the same dataset and therefore does not indicate a fair comparison.

    “The baseline Models Genesis implementation was excluded in this analysis as this model was pretrained on the same dataset and, therefore, does not indicate a fair comparison.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Our approach is generalizable to multiple use cases, and for nodule malignancy, we obtain high performance using significantly lesser training data, only 338 nodules (due to our more stringent exclusion criteria).

    “By contrast, our approach is generalizable to multiple use cases, and for nodule malignancy, we obtain high performance using significantly lesser training data, only 338 nodules (due to our more stringent exclusion criteria).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.
    • Rests on an assumption: The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The tumour microenvironment influences cancer development and prognosis.

    “which is consistent with research demonstrating the tumour microenvironment’s influence on cancer development22 and prognosis.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Hinshaw, D. C. & Shevde, L. A. The tumor microenvironment innately modulates cancer progression. Cancer Res. 79, 4557–4566 (2019). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Expressing saliency maps over 4,096-dimensional outputs remains challenging in practice.

    “as expressing saliency maps over 4,096-dimensional outputs remains challenging in practice.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The approach of extracting patches centered around seed points allows for significantly higher throughput than manual segmentation.

    “This approach allows for significantly higher throughput than manual segmentation, which can be more tedious.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The foundation model offers a powerful and reliable framework for discovering cancer imaging biomarkers, especially in small datasets.

    “In conclusion, our foundation model offers a powerful and reliable framework for discovering cancer imaging biomarkers, especially in small datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Existing literature has suggested several strategies, such as image reconstruction, to pretrain networks to learn generalized, task-agnostic representations.

    “Existing literature12 has suggested several strategies, such as image reconstruction, to pretrain networks to learn these representations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] Balestriero, R. et al. A cookbook of self-supervised learning. Preprint at https://arxiv.org/abs/2304.12210 (2023).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The saturation point for pretraining dataset size largely depends on the nature and sizes of training data in the downstream task.

    “This point largely depends on the nature and sizes of training data in the downstream task.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Cole et al.38 Cole, E., Yang, X., Wilber, K., Aodha, O. M. & Belongie, S. When does contrastive visual representation learning work? In Proc. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 14755–14764 (IEEE, 2022). source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Deep-learning methods are robust to differences in acquisition and segmentation variability and provide improved performance.

    “Deep-learning methods, in comparison, are robust to differences in acquisition and segmentation variability and provide improved performance10.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10] Hosny, A. et al. Deep learning for lung cancer prognostication: a retrospective multi-cohort radiomics study. PLoS Med. 15, e1002711 (2018). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    It is typical to have inherently small sample sizes in use case 3 when compared to task complexity due to study-specific inclusion criteria.

    “as it is typical to have inherently small sample sizes in such use cases when compared to task complexity due to study-specific inclusion criteria.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    A significant challenge with deep-learning models, including the study's model, is their 'black box' nature, which limits interpretability and explainability.

    “Third, a significant challenge with deep-learning models, including ours, is their ‘black box’ nature, which limits interpretability and explainability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The study's pretraining dataset, comprising 11,467 lesion volumes from 5,513 unique CT scans, is one of the largest lesion-specific datasets and one of the largest pretraining 3D CT datasets.

    “In our study, we pretrained on 11,467 lesion volumes and randomly sampled volumes, from 5,513 unique CT scans, leveraging not only one of the largest lesion-specific datasets but also one of the largest pretraining 3D CT datasets.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Many studies investigating imaging-based biomarkers incorporate supervised deep-learning algorithms, but they are typically applied in scenarios where large datasets are available for training and testing.

    “While many studies investigating imaging-based biomarkers incorporate supervised deep-learning algorithms into their models9–11, they are typically applied in scenarios where large datasets are available for training and testing.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [9] Mahajan, A. et al. Deep learning-based predictive imaging biomarker model for EGFR mutation status in non-small cell lung cancer from CT imaging. J. Clin. Orthod. 38, 3106 (2020).

      Not checked in the free run.

    • cites [10] Hosny, A. et al. Deep learning for lung cancer prognostication: a retrospective multi-cohort radiomics study. PLoS Med. 15, e1002711 (2018). source

      Not checked in the free run.

    • cites [11] Braghetto, A., Marturano, F., Paiusco, M., Baiesi, M. & Bettinelli, A. Radiomics and deep learning methods for the prediction of 2-year overall survival in LUNG1 dataset. Sci. Rep. 12, 14132 (2022). source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Cole et al. showed a saturation point associated with pretraining dataset size and diminishing returns beyond this point.

    “They show a saturation point associated with pretraining dataset size and diminishing returns beyond this point.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Cole et al.38 Cole, E., Yang, X., Wilber, K., Aodha, O. M. & Belongie, S. When does contrastive visual representation learning work? In Proc. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 14755–14764 (IEEE, 2022). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    In the comparative analysis, uniformity in batch sizes and dataset parameters was maintained across all methods, while optimizer and loss-specific settings were aligned with each method's original configuration.

    “In our comparative analysis, we maintained uniformity in batch sizes and dataset parameters across all methods, while optimizer and loss-specific settings were aligned with each method’s original configuration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationnone given

    The initial biological association analysis is preliminary and requires more rigorous investigation for a concrete understanding.

    “Furthermore, our initial biological association analysis, aimed at explaining the model’s decisions, is preliminary and requires more rigorous investigation for a concrete understanding.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    A high proportion of negatives relative to positives results in consistently low false-positive rates and might overestimate the AUC.

    “due to the high proportion of negatives relative to positives, which results in consistently low false-positive rates and might overestimate the AUC.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several studies have demonstrated the efficacy of SSL in medicine where only limited data might be available for training deep-learning networks.

    “Several studies23–25 have demonstrated the efficacy of SSL in medicine where only limited data might be available for training deep-learning networks.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [23] Azizi, S. et al. Big self-supervised models advance medical image classification. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) 3458–3468 (IEEE, 2021). source

      Not checked in the free run.

    • cites [24] Krishnan, R., Rajpurkar, P. & Topol, E. J. Self-supervised learning in medicine and healthcare. Nat. Biomed. Eng. 6, 1346–1352 (2022). source

      Not checked in the free run.

    • cites [25] Ghesu, F. C. et al. Contrastive self-supervised learning from 100 million medical images with optional supervision. J. Med. Imaging 9, 064503 (2022). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The RIDER dataset is routinely used for test-retest robustness analysis in tumour imaging.

    “We used scans from 26 patients from the RIDER dataset19, routinely used for test–retest robustness analysis in tumour imaging19–21.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [19] Zhao, B. et al. Evaluating variability in tumor measurements from same-day repeat CT scans of patients with non-small cell lung cancer. Radiology 252, 263–272 (2009). source

      Not checked in the free run.

    • cites [20] Aerts, H. J. W. L. et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat. Commun. 5, 4006 (2014). source

      Not checked in the free run.

    • cites [21] Hosny, A. et al. Clinical validation of deep learning algorithms for radiotherapy targeting of non-small-cell lung cancer: an observational study. Lancet Digit. Health 4, e657–e666 (2022). source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Experiments involving further data reduction in use case 3 do not provide any valuable insights.

    “Therefore, experiments involving further data reduction in this case do not provide any valuable insights.”
    Quote in the paperSubstantial claim
41 of 41

  • The paper relies on this without support.

    Access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.

    “However, access to large amounts of annotated data for specialized applications is often challenging and demands expertise, time and labour.”

  • The paper relies on this without support.

    There is still limited evidence investigating whether SSL can help train foundation models that learn general, robust and transferrable representations that can act as imaging biomarkers, especially prognostic, for tasks of clinical relevance.

    “There is still limited evidence investigating whether SSL can help train foundation models that learn general, robust and transferrable representations that can act as imaging biomarkers, especially prognostic, for tasks of clinical relevance.”

  • The paper relies on this without support.

    The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.

    “The quantity and quality of annotated data are strongly linked to the robustness of deep-learning models.”

  • The paper relies on this without support.

    In scenarios with limited annotated data, many investigators fall back on traditional handcrafted or engineered approaches that limit the scope of discovery.

    “In such scenarios, many investigators fall back on traditional handcrafted or engineered approaches based on defined mathematical and statistical algorithms that analyse attributes such as the shape and texture of objects in images, which limit the scope of discovery.”

  • The paper relies on this without support.

    Following pretraining, foundation models can be applied to task-specific problems, improving generalization, especially in tasks with small datasets.

    “Following pretraining, foundation models can be applied to task-specific problems, improving generalization, especially in tasks with small datasets.”

  • The paper relies on this without support.

    Image-biomarker studies for predicting endpoints such as overall survival on small cohorts largely rely on statistical feature extraction (engineered radiomics) and classical machine learning-based modelling.

    “Image-biomarker studies for predicting endpoints, such as overall survival on small cohorts, largely rely on statistical feature extraction (engineered radiomics) and classical machine learning-based modelling.”

  • The paper relies on this without support.

    These statistical feature extraction approaches require precise 3D segmentations, increasing the annotation burden of these studies.

    “These require precise 3D segmentations for feature extraction, increasing the annotation burden of these studies.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Transformer-based methods support clinical text processing, whole-slide and radiologic image interpretation, and genomic sequence modeling in oncology.

    “In oncology, transformer-based methods support clinical text processing, whole-slide and radiologic image interpretation (vision transformers), and genomic sequence modeling [24,25,26,27].”

    From Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery (2025) · cites this paper as [24]

  • Foundation models have shown impressive performance across medical fields such as ophthalmology, radiology and pathology.

    “Built on rich knowledge representations, these models have shown impressive performance across medical fields such as ophthalmology20, radiology21 and pathology22–25.”

    From A Multimodal Vision Foundation Model for Clinical Dermatology (2025) · cites this paper as [21]

  • Specialty-specific foundation models typically focus on single imaging modalities.

    “Specialty-specific foundation models20,21,23 typically focus on single imaging modalities”

    From A Multimodal Vision Foundation Model for Clinical Dermatology (2025) · cites this paper as [21]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Stability analysis reports more LUNG1 samples than were selected

    Minor

    Figure 4 says its perturbation analyses used 422 samples, whereas the Methods say 420 LUNG1 patients were selected for the prognostication test set. The stability analysis is described as using LUNG1.

  • Reported mAP lies outside its 95% confidence interval

    Minor

    The reported mAP of 0.847 exceeds the upper bound of its stated 95% confidence interval, 0.810. This affects the reported performance of the selected pretraining strategy.

2 of 2

Cites

Flagged first, then by citations

6 of the 54 references couldn’t be matched to a known paper.

Research works printed but not linked: 8

  1. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics (eds Burstein, J. et al.) 4171–4186 (ACL, 2019).
  2. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. In Proc. 37th International Conference on Machine Learning (eds III, H.D. & Singh, A.) 1597–1607 (PMLR, 2020).
  3. Kumar, D. et al. in Image Analysis and Recognition (eds Karray, F. et al.) 54–62 (Springer, 2017).
  4. Taleb, A. et al. 3D self-supervised methods for medical imaging. Adv. Neural Inf. Process. Syst. 33, 18158–18172 (2020).
  5. Adebayo, J. et al. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems 9505–9515 (Curran Associates, 2018).
  6. Sohn, K. Improved deep metric learning with multi-class N-pair loss objective. In Advances in Neural Information Processing Systems (eds Lee, D. et al.) 1857–1865 (Curran Associates, 2016).
  7. van der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).
  8. Springenberg, J. T., Dosovitskiy, A., Brox, T. & Riedmiller, M. A. Striving for simplicity: the all convolutional net. In 3rd International Conference on Learning Representations Workshop (ICLR, 2015).

Cited by

Sorted by citations

Related

Authors
Pai, S., Bontempi, D., Hadzic, I., Prudente, V., Sokač, M., Chaunzwa, T. L., Bernatz, S., Hosny, A., Mak, R. H., Birkbak, N. J., & Aerts, H. J. W. L.
Affiliations
  • Brigham and Women's Hospital (US)
  • Aarhus University (DK)
  • Harvard University (US)
Corresponding author
Aerts, Hugo J. W. L.
ORCID
Venue
Nature Machine Intelligence
Publisher
Springer Science and Business Media LLC
Published
2024-03-15
Volume
6
Issue
3
Pages
354-367
ISSN
2522-5839 (ISSN-L)
DOI
10.1038/s42256-024-00807-9
PMC
PMC10957482
OpenAlex
W4392850630
Semantic Scholar
b0dc5a23b2f834660c53c436bd1c3dd9fa01d354
Cited by
199
References
54
Citations by year
2026: 81 · 2025: 98 · 2024: 19 · 2023: 1
FWCI
50.44
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, cc-by)
Funders
  • European Commission
  • Deutsche Forschungsgemeinschaft (502050303)
  • National Institutes of Health (R35CA22052)
  • National Cancer Institute (U01CA190234, U24CA194354, U54CA274516, U01CA209414)
  • H2020 European Research Council (866504)
Topics
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
foundation models, cancer imaging biomarkers, self-supervised learning, convolutional encoder, radiographic lesions, limited labeled data, transfer learning, biomarker discovery, radiology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review