CoChat
Sign Up
Cited by 150Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMachine Learning and Algorithms

Survey on Self-Supervised Learning: Auxiliary Pretext Tasks and Contrastive Learning Methods in Imaging

Open access
Albelwi, Saleh·2022·Entropy
DOI 10.3390/e24040551PMC PMC9029566
Open this paper in CoChat — free accountOpen PDF

Abstract

Although deep learning algorithms have achieved significant progress in a variety of domains, they require costly annotations on huge datasets. Self-supervised learning (SSL) using unlabeled data has emerged as an alternative, as it eliminates manual annotation. To do this, SSL constructs feature representations using pretext tasks that operate without manual annotation, which allows models trained in these tasks to extract useful latent representations that later improve downstream tasks such as object classification and detection. The early methods of SSL are based on auxiliary pretext tasks as a way to learn representations using pseudo-labels, or labels that were created automatically based on the dataset’s attributes. Furthermore, contrastive learning has also performed well in learning representations via SSL. To succeed, it pushes positive samples closer together, and negative ones further apart, in the latent space. This paper provides a comprehensive literature review of the top-performing SSL methods using auxiliary pretext and contrastive learning techniques. It details the motivation for this research, a general pipeline of SSL, the terminologies of the field, and provides an examination of pretext tasks and self-supervised methods. It also examines how self-supervised methods compare to supervised ones, and then discusses both further considerations and ongoing challenges faced by SSL.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/e24040551

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Self-supervised pretext tasks used in DINO's training offer unique advantages over traditional supervised approaches.

    “The self-supervised pretext tasks used in DINO's training process also offer unique advantages over traditional supervised approaches (Albelwi 2022; Batutin 2024).”

    From A Semi-Supervised Learning Method for the Identification of Bad Exposures in Large Imaging Surveys (2026) · cites this paper as Albelwi 2022

    The Introduction (page unavailable) says self-supervised learning uses unlabeled data, can compete with supervised methods, and offers data-efficiency advantages. Section 3.2 (page unavailable) describes benefits of pretext tasks, but the retrieved passages do not discuss DINO or establish that the tasks used in its training have those advantages specifically. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Self-supervised pretext tasks enable the model to learn nuanced features that extend beyond simple classification boundaries and that might otherwise be overlooked, unlike supervised learning constrained by rigid label definitions.

    “While supervised learning methods are constrained by rigid label definitions, self-supervised pretext tasks (such as image augmentation recognition) enable the model to learn nuanced features that extend beyond simple classification boundaries and that otherwise might be overlooked.”

    From A Semi-Supervised Learning Method for the Identification of Bad Exposures in Large Imaging Surveys (2026) · cites this paper as Albelwi 2022

    The Discussion (section 8) says pretext tasks produce complicated feature representations, and the Introduction (section 1) describes their use in classification, detection, and segmentation. The paper does not establish that supervised learning is constrained by rigid label definitions or that pretext tasks recover features supervised learning would overlook. Its narrower point in section 2.1 is that labels used in transfer learning may not generalize to a target task. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Manual labeling of data is time-consuming and labor-intensive.

    “Manually labeling data is a time-consuming and labor-intensive process [1].”

    From Self-supervised learning: A succinct review (2023) · cites this paper as [1]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 55 · Cited by 150

Research works printed but not linked: 26

  1. [1]Simonyan K. Zisserman A. Very deep convolutional networks for large-scale image recognition arXiv 2014 1409.1556
  2. [5]He K. Gkioxari G. Dollár P. Girshick R. Mask r-cnn Proceedings of the IEEE International Conference on Computer Vision Honolulu, HI, USA 21–26 July 2017 2961 2969
  3. [11]Kingma D.P. Welling M. Auto-encoding variational bayes arXiv 2013 1312.6114
  4. [12]Goodfellow I. Pouget-Abadie J. Mirza M. Xu B. Warde-Farley D. Ozair S. Courville A. Bengio Y. Generative adversarial nets Adv. Neural Inf. Process. Syst. 2014 27
  5. [13]Mirza M. Osindero S. Conditional generative adversarial nets arXiv 2014 1411.1784
  6. [14]Donahue J. Krähenbühl P. Darrell T. Adversarial feature learning arXiv 2016 1605.09782
  7. [15]Grill J.-B. Strub F. Altché F. Tallec C. Richemond P.H. Buchatskaya E. Doersch C. Pires B.A. Guo Z.D. Azar M.G. Bootstrap your own latent: A new approach to self-supervised learning arXiv 2020 2006.07733
  8. [18]Chen X. Fan H. Girshick R. He K. Improved baselines with momentum contrastive learning arXiv 2020 2003.04297
  9. [19]Tao L. Wang X. Yamasaki T. Pretext-Contrastive Learning: Toward Good Practices in Self-supervised Video Representation Leaning arXiv 2020 2010.15464
  10. [29]Chen T. Kornblith S. Swersky K. Norouzi M. Hinton G. Big self-supervised models are strong semi-supervised learners arXiv 2020 2006.10029
  11. [38]Dumoulin V. Belghazi I. Poole B. Mastropietro O. Lamb A. Arjovsky M. Courville A. Adversarially learned inference arXiv 2016 1606.00704
  12. [44]Yang C. An Z. Cai L. Xu Y. Mutual Contrastive Learning for Visual Representation Learning arXiv 2021 2104.12565
  13. [45]Kalantidis Y. Sariyildiz M.B. Pion N. Weinzaepfel P. Larlus D. Hard negative mixing for contrastive learning arXiv 2020 2010.01028
  14. [46]Khosla P. Teterwak P. Wang C. Sarna A. Tian Y. Isola P. Maschinot A. Liu C. Krishnan D. Supervised contrastive learning arXiv 2020 2004.11362
  15. [47]Tian Y. Chen X. Ganguli S. Understanding self-supervised learning dynamics without contrastive pairs arXiv 2021 2102.06810
  16. [52]Balestriero R. Misra I. LeCun Y. A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Moments arXiv 2022 2202.08325
  17. [53]Lee H. Hwang S.J. Shin J. Rethinking data augmentation: Self-supervision and self-distillation arXiv 2019 1910.05872
  18. [54]Tomasev N. Bica I. McWilliams B. Buesing L. Pascanu R. Blundell C. Mitrovic J. Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet? arXiv 2022 2201.05119
  19. [56]Appalaraju S. Zhu Y. Xie Y. Fehérvári I. Towards Good Practices in Self-supervised Representation Learning arXiv 2020 2012.00868
  20. [59]Oord A.v.d. Li Y. Vinyals O. Representation learning with contrastive predictive coding arXiv 2018 1807.03748
  21. [68]Asano Y.M. Rupprecht C. Vedaldi A. Self-labelling via simultaneous clustering and representation learning arXiv 2019 1911.05371
  22. [69]Li J. Zhou P. Xiong C. Hoi S.C. Prototypical contrastive learning of unsupervised representations arXiv 2020 2005.04966
  23. [70]Goyal P. Caron M. Lefaudeux B. Xu M. Wang P. Pai V. Singh M. Liptchinsky V. Misra I. Joulin A. Self-supervised pretraining of visual features in the wild arXiv 2021 2103.01988
  24. [76]Caron M. Misra I. Mairal J. Goyal P. Bojanowski P. Joulin A. Unsupervised learning of visual features by contrasting cluster assignments arXiv 2020 2006.09882
  25. [79]Keshav V. Delattre F. Self-supervised visual feature learning with curriculum arXiv 2020 2001.05634
  26. [80]Jing L. Vincent P. LeCun Y. Tian Y. Understanding dimensional collapse in contrastive self-supervised learning arXiv 2021 2110.09348

Authors
Albelwi, S.
Affiliations
University of Tabuk (SA)
Corresponding author
Albelwi, Saleh
ORCID
Albelwi, Saleh
Venue
Entropy
Publisher
MDPI AG
Published
2022-04-14
Volume
24
Issue
4
Pages
551
ISSN
1099-4300 (ISSN-L)
DOI
10.3390/e24040551
PMC
PMC9029566
OpenAlex
W4223948957
Cited by
150
References
55
Citations by year
2026: 27 · 2025: 29 · 2024: 49 · 2023: 37 · 2022: 8
FWCI
2.14
Type
journal article
Language
en
Open access
Licence
Locations
  • Entropy (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
University of Tabuk
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
self-supervised learning, contrastive learning, image representation learning, pseudo-labeling, unlabeled data, object classification, object detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review