CoChat
Sign Up
Cited by 479Domain Adaptation and Few-Shot LearningArtificial Intelligence in Healthcare and EducationMedical Image Segmentation Techniques

The DOI matches its Crossref record; 124 of the 130 references it cites were found in a registry or an index; 6 of the 71 sources its claims cite were checked for support.

Self-supervised learning for medical image classification: a systematic review and implementation guidelines

Open access
Huang, Shih-Cheng, Pareek, Anuj, Jensen, Malte, Lungren, Matthew P., Yeung, Serena, Chaudhari, Akshay S.·2023·npj Digital Medicine
DOI 10.1038/s41746-023-00811-0PMC PMC10131505
Open this paper in CoChat — free accountOpen PDF

Abstract

Advancements in deep learning and computer vision provide promising solutions for medical image analysis, potentially improving healthcare and patient outcomes. However, the prevailing paradigm of training deep learning models requires large quantities of labeled training data, which is both time-consuming and cost-prohibitive to curate for medical images. Self-supervised learning has the potential to make significant contributions to the development of robust medical imaging models through its ability to learn useful insights from copious medical datasets without labels. In this review, we provide consistent descriptions of different self-supervised learning strategies and compose a systematic review of papers published between 2012 and 2022 on PubMed, Scopus, and ArXiv that applied self-supervised learning to medical imaging classification. We screened a total of 412 relevant studies and included 79 papers for data extraction and analysis. With this comprehensive effort, we synthesize the collective knowledge of prior work and provide implementation guidelines for future researchers interested in applying self-supervised learning to their development of medical imaging classification models.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review synthesizes 79 studies applying self-supervised learning (SSL) to medical image classification and finds that SSL pre-training generally improves downstream performance over purely supervised or non-SSL baselines. It also provides terminology, implementation guidelines, and highlights that optimal SSL strategies remain uncertain due to heterogeneous studies and limited direct comparisons.

Key findings

  • From 412 identified studies, 148 full-text screened, and 79 included; most SSL models improved performance, with relative increases of 0.216–32.6% AUROC, 0.440–29.2% accuracy, and 0.137–14.3% F1 over the same architecture without SSL pre-training, while five studies reported slight decreases of 0.980–4.51%.
  • Contrastive learning was the most common strategy (44/79), with SimCLR, MoCo, and BYOL used in 13, 8, and 3 papers, respectively; 36 studies comparing contrastive SSL to supervised pre-training reported an average improvement of 6.35%.
  • Innate relationship SSL was used in 15/79 studies; clinically relevant pre-text tasks achieved greater average improvement over supervised methods than transformation-based tasks (13.7% vs. 5.03%). Generative SSL appeared in 3/79, self-prediction in 6/79, and combined strategies in 11/79; all 8 combined-strategy studies that compared with purely supervised approaches reported improvement (0.140–8.29%).
  • End-to-end fine-tuning tended to outperform training a classifier on extracted features; of 11 studies with quantitative comparisons, 8 favored end-to-end fine-tuning. Radiology was the most represented clinical domain (47/79), followed by pathology (12/79).

Methods

  • PRISMA-based systematic review searching PubMed, Scopus, and ArXiv for papers published between January 2012 and May 2021, using search terms combining “self-supervised learning” and “medical imaging.”
  • Inclusion required English, peer-reviewed original research applying SSL to a downstream clinical medical image classification task; exclusions included semi-supervised learning, any manual labels during self-supervision, regression- or segmentation-only downstream tasks, and non-human medical imaging.
  • Screening used Covidence by three independent researchers, with disagreements resolved by discussion and a fourth arbitrator if needed. Data extraction recorded SSL strategy, modality, clinical domain, task, framework, fine-tuning strategy, metrics, SSL and supervised performance, and relative differences; AUROC was prioritized, then accuracy, F1, and sensitivity.

Limitations

  • Stated by the paper: publication bias may overestimate SSL benefits; the post-2012 search excluded pre-deep-learning SSL work; studies could not be aggregated or statistically compared because of different modalities, metrics, and objectives; categorizing SSL strategies introduced subjectivity; and the review was limited to classification, excluding segmentation, regression, and registration.
  • Evident from the text: only 33/79 studies compared with another SSL strategy, only 18/79 used natural-image pre-trained weights, and only 13 compared feature extraction with end-to-end fine-tuning (two without numerical results), indicating limited baselines and ablation experiments.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-023-00811-0

Cites

124 of 124 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Timeline showing the number of publications on deep learning for medical image classification per year, found by using the same search criteria on PubMed, Scopus and, ArXiv. The figure shows that self-supervised learning is a rapidly growing subset of deep learning for medical imaging literature.
Fig. 1 Timeline showing the number of publications on deep learning for medical image classification per year, found by using the same search criteria on PubMed, Scopus and, ArXiv. The figure shows that self-supervised learning is a rapidly growing subset of deep learning for medical imaging literature.CC BY · PubMed Central
Fig. 2 Illustration of different self-supervised learning and fine-tuning strategies. During Stage 1 a model is pre-trained using one or more of the following self-supervised learning strategies: (a) Innate relationship SSL pre-trains a model on a hand-crafted task by leveraging the internal structure of the data, (b) Generative SSL learns the distribution of training data, enabling reconstruction of the original input (c) Contrastive SSL forms positive pairs between different augmentations of the same image and minimizes representational distances of positive samples (d) Self-prediction augments or masks out random portions of an image, and reconstructs the original image based on the unaltered parts of the original image. During Stage 2, the pre-trained model can be fine-tuned using one of the following strategies: (e) end-to-end fine-tuning of the pre-trained model and classifier, or (f) train a classifier that uses extracted features from the SSL pre-trained model.
Fig. 2 Illustration of different self-supervised learning and fine-tuning strategies. During Stage 1 a model is pre-trained using one or more of the following self-supervised learning strategies: (a) Innate relationship SSL pre-trains a model on a hand-crafted task by leveraging the internal structure of the data, (b) Generative SSL learns the distribution of training data, enabling reconstruction of the original input (c) Contrastive SSL forms positive pairs between different augmentations of the same image and minimizes representational distances of positive samples (d) Self-prediction augments or masks out random portions of an image, and reconstructs the original image based on the unaltered parts of the original image. During Stage 2, the pre-trained model can be fine-tuned using one of the following strategies: (e) end-to-end fine-tuning of the pre-trained model and classifier, or (f) train a classifier that uses extracted features from the SSL pre-trained model.CC BY · PubMed Central
Fig. 3 The PRISMA diagram for this review. The authors independently screened all records for eligibility. Out of 412 studies identified from PubMed, Scopus, and ArXiv, 79 studies were included in the systematic review.
Fig. 3 The PRISMA diagram for this review. The authors independently screened all records for eligibility. Out of 412 studies identified from PubMed, Scopus, and ArXiv, 79 studies were included in the systematic review.CC BY · PubMed Central
Fig. 4 Summary of extracted data from studies in our system review. a Prevalence of different medical specialties split by self-supervised learning strategy. b Prevalence of different medical imaging modalities split by self-supervised learning strategy. c Relative performance difference between different types of self-supervised learning strategies on the same task. d Performance comparison between end-to-end fine-tuning vs. training a classifier using extracted features from pre-trained self-supervised models. e Relative difference in downstream task performance between self-supervised and non-self-supervised models.
Fig. 4 Summary of extracted data from studies in our system review. a Prevalence of different medical specialties split by self-supervised learning strategy. b Prevalence of different medical imaging modalities split by self-supervised learning strategy. c Relative performance difference between different types of self-supervised learning strategies on the same task. d Performance comparison between end-to-end fine-tuning vs. training a classifier using extracted features from pre-trained self-supervised models. e Relative difference in downstream task performance between self-supervised and non-self-supervised models.CC BY · PubMed Central
Fig. 5 Examples of augmentations and transformations that alter the semantic meaning of medical images144 but not natural images145. a The image shows a T2-weighted brain MRI with a cavernoma in the right parietal lobe (bounded in red). b and c Masking and cropping operations can obscure the cavernoma and alter the semantic meaning of the image, as the MRI-scan no longer exhibits any abnormality. d Image of a dog (bounded in red), (e) and (f) Masking and cropping operations do not obscure the dog nor alter the semantic meaning of the image.
Fig. 5 Examples of augmentations and transformations that alter the semantic meaning of medical images144 but not natural images145. a The image shows a T2-weighted brain MRI with a cavernoma in the right parietal lobe (bounded in red). b and c Masking and cropping operations can obscure the cavernoma and alter the semantic meaning of the image, as the MRI-scan no longer exhibits any abnormality. d Image of a dog (bounded in red), (e) and (f) Masking and cropping operations do not obscure the dog nor alter the semantic meaning of the image.CC BY · PubMed Central

Verification

Claims the paper makes

37 Own result · 34 Not checked yet · 21 Own interpretation · 8 Own method · 7 No basis given · 2 Partly supported · accuracy not yet measured

Verify all 65 references — free account

Show all 65 claims

  • Probably not a substantial claim. 21
  • It may state two separate claims as one. 28
44 of 44

  • The paper relies on this without support.

    Annotating medical imaging datasets requires domain expertize, making large-scale annotations cost-prohibitive and time-consuming.

    “However, the prevailing paradigm of training deep learning models requires large quantities of labeled training data, which is both time-consuming and cost-prohibitive to curate for medical images.”
    “However, annotating medical imaging datasets requires domain expertize, making large-scale annotations cost-prohibitive and time-consuming, which fundamentally limits building effective medical imaging models across varying clinical use cases.”

  • The paper relies on this without support.

    The current success of deep learning has been largely dominated by supervised learning frameworks, which typically require large-scale labeled datasets to achieve high performance.

    “While deep learning provides promising solutions for improving medical image interpretation, the current success has been largely dominated by supervised learning frameworks, which typically require large-scale labeled datasets to achieve high performance.”

  • The paper relies on this without support.

    The inability to generalize can be largely due to the process of supervised learning, which encourages the model to mainly learn features heavily correlated with specific labels rather than general features representative of the whole data distribution.

    “The inability to generalize can be largely due to the process of supervised learning, which encourages the model to mainly learn features heavily correlated with specific labels rather than general features representative of the whole data distribution.”

  • The paper relies on this without support.

    In a healthcare system where a myriad of opportunities and possibilities for automation exist, it is practically impossible to curate labeled datasets for all tasks, modalities, and outcomes for training supervised models.

    “In a healthcare system where a myriad of opportunities and possibilities for automation exist, it is practically impossible to curate labeled datasets for all tasks, modalities, and outcomes for training supervised models.”

  • The paper relies on this without support.

    SSL can create generalist models that can be fine-tuned for many downstream tasks without large-scale labeled datasets.

    “Unlike supervised learning, SSL can create generalist models that can be fine-tuned for many downstream tasks without large-scale labeled datasets.”

  • The paper relies on this without support.

    The most suitable strategies and best practices for medical images have not been sufficiently investigated.

    “the most suitable strategies and best practices for medical images have not been sufficiently investigated.”

  • The paper relies on this without support.

    Many clinical tasks are based on classification.

    “We focus on medical image classification tasks because many clinical tasks are based on classification, and thus our research may be directly applicable to deep learning models for clinical workflows.”

  • The paper relies on this without support.

    SSL is task agnostic, and the same SSL pre-trained model can be fine-tuned for different types of downstream tasks, including classification, segmentation, and object detection.

    “It is worth emphasizing that SSL is task agnostic, and the same SSL pre-trained model can be fine-tuned for different types of downstream tasks, including classification, segmentation, and object detection.”

  • The paper relies on this without support.

    Current limited knowledge and understanding of possible imaging features arising in psychiatric diseases constitutes a major clinical challenge to making local annotations such as bounding boxes or segmentations on brain scans.

    “Current limited knowledge and understanding of possible imaging features arising in psychiatric diseases constitutes a major clinical challenge to making local annotations such as bounding boxes or segmentations on brain scans.”

  • The paper relies on this without support.

    Curating large-scale medical image datasets from multiple institutions is often challenged by the difficulty of sharing patient data while preserving patient privacy.

    “Curating large-scale medical image datasets from multiple institutions is often challenged by the difficulty of sharing patient data while preserving patient privacy.”

  • The paper relies on this without support.

    Many medical imaging modalities have corresponding radiology reports that contain detailed descriptions of the medical conditions observed by radiologists.

    “Furthermore, many medical imaging modalities have corresponding radiology reports that contain detailed descriptions of the medical conditions observed by radiologists.”

  • The paper relies on this without support.

    Currently there are very few publicly available medical image datasets with corresponding radiology reports, largely due to the difficulties in preserving patient privacy.

    “However, currently there are very few publicly available medical image datasets with corresponding radiology reports, largely due to the difficulties in preserving patient privacy.”

  • The paper relies on this without support.

    Many models pre-trained in a self-supervised manner on large-scale natural image datasets are publicly available.

    “Although self-supervised pre-training can be computationally demanding, many models pre-trained in a self-supervised manner on large-scale natural image datasets are publicly available and should be utilized.”
13 of 13

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 130 references

Research works printed but not linked: 21

  1. Brown, T., Mann, B. & Ryder, N. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. (2020).
  2. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A Simple Framework for Contrastive Learning of Visual Representations. Proc. 37th Int. Conf. Mach. Learn., PMLR 119, 1597–1607 (2020).
  3. Lilian Weng, J. W. K. Self-Supervised Learning: Self-Prediction and Contrastive Learning. Adv. Neural Inf. Proc. Syst. (2021).
  4. Goodfello w, I. J. et al. Generative Adversarial Networks. Preprint at https://community.unix.com/uploads/short-url/oXsmq2VZ9hc2X6hwPRXZRMGbV20.pdf (2014).
  5. Donahue J. & Simonyan K. Large scale adversarial representation learning. Adv. Neural Inf. Process. Syst. (2019).
  6. Grill J. B., Strub F., Altché F. & Tallec C. Bootstrap your own latent-a new approach to self-supervised learning. Adv. Neural Inf. Proc. Syst. 33, (2020).
  7. Chen, X. & He, K. Exploring Simple Siamese Representation Learning. Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021, 15750–15758 (2021).
  8. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T. & Efros, A. A. Context Encoders: Feature Learning by Inpainting. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) 2016, 2536–2544 (2016).
  9. He, K. et al. Masked Autoencoders Are Scalable Vision Learners. Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) 2022, 16000–16009 (2022).
  10. Vu, Y. N. T., Tsue, T., Su, J. & Singh, S. An improved mammography malignancy model with self-supervised learning. Med. Imaging 2021: Comput-Aided Diagn. 11597, 210–216 (2021).
  11. Hao, H., Didari, S., Woo, J. O., Moon, H. & Bangert, P. Highly Efficient Representation and Active Learning Framework for Imbalanced Data and its Application to COVID-19 X-Ray Classification. Conference on Neural Information Processing Systems (NeurIPS 2021) (2021).
  12. Dehaene, O., Camara, A., Moindrot, O., de Lavergne, A. & Courtiol, P. Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology. in ML4H 2020 NeurIPS workshop (2020).
  13. Kotar, K., Ilharco, G. & Schmidt, L. Contrasting contrastive self-supervised representation learning pipelines. Proc. Estonian Acad. Sci. Biol. Ecol.
  14. Cole, E., Yang, X., Wilber, K., Aodha, O. M. & Belongie, S. When Does Contrastive Visual Representation Learning Work? Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) 2022, 14755–14764 (2022).
  15. Azizi, S., Mustafa, B., Ryan, F. & Beaver, Z. Big self-supervised models advance medical image classification. Proc. Estonian Acad. Sci. Biol. Ecol.
  16. Shi, Y., Siddharth, N., Torr, P. H. S. & Kosiorek, A. R. Adversarial Masking for Self-Supervised Learning. In International Conference on Machine Learning, 20026–20040. (2022).
  17. Van Gansbeke, X. & Vandenhende, S. Revisiting contrastive methods for unsupervised learning of visual representations. Adv. Neural Inf. Process. Syst. 34, 16238–16250 (2021).
  18. Sowrirajan, H., Yang, J., Ng, A. Y. & Rajpurkar, P. MoCo-CXR: MoCo Pre-training Improves Representation and Transferability of Chest X-ray Models. Proc. Mach. Learn. Res. 143, 727–743 (2021).
  19. Bao, W., Jin, Y., Huang, C. & Peng, W. CT Image Classification of Invasive Depth of Gastric Cancer based on 3D-DPN Structure. In The 11th International Workshop on Computer Science and Engineering (WCSE 2021) 115–121 (2021).
  20. Kaku, A., Upadhya, S. & Razavian, N. Intermediate layers matter in momentum contrastive self supervised learning. In 35th Conference on Neural Information Processing Systems (NeurIPS 2021). (2021).
  21. Liu, Q. et al. SimTriplet: Simple Triplet Representation Learning with a Single GPU. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 102–112 (2021). · DOI · Paper page

Cited by

Sorted by citations

See all 479

Related

Metadata

Authors
Huang, S.-C., Pareek, A., Jensen, M., Lungren, M. P., Yeung, S., & Chaudhari, A. S.
Affiliations
  • Stanford University (US)
  • Stanford Medicine (US)
Corresponding author
Huang, Shih-Cheng
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2023-04-26
Volume
6
Issue
1
Pages
74-74
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-023-00811-0
PMC
PMC10131505
OpenAlex
W4367055910
Cited by
479
References
130
Citations by year
2026: 145 · 2025: 208 · 2024: 102 · 2023: 24
FWCI
10.03
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institutes of Health (75N92020C00021, 75N92020C00008, AR077604, EB002524)
  • National Heart, Lung, and Blood Institute (75N92020C00008, 75N92020C00021, R01HL155410)
  • National Institute of Arthritis and Musculoskeletal and Skin Diseases (R01AR077604, R01AR079431)
  • National Institute of Biomedical Imaging and Bioengineering (P41EB027060, R01EB002524)
  • U.S. National Library of Medicine (R01LM012966)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
Keywords
self-supervised learning, medical image classification, diagnostic imaging, unlabeled data, implementation guidelines, deep learning, limited labeled data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review