CoChat
Sign Up
Cited by 295Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMachine Learning and Algorithms

Self-supervised learning: A succinct review

Open access
Rani, Veenu, Nabi, Syed Tufael, Kumar, Munish, Mittal, Ajay, Kumar, Krishan·2023·Archives of Computational Methods in Engineering
DOI 10.1007/s11831-023-09884-2PMC PMC9857922
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine learning has made significant advances in the field of image processing. The foundation of this success is supervised learning, which necessitates annotated labels generated by humans and hence learns from labelled data, whereas unsupervised learning learns from unlabeled data. Self-supervised learning (SSL) is a type of un-supervised learning that helps in the performance of downstream computer vision tasks such as object detection, image comprehension, image segmentation, and so on. It can develop generic artificial intelligence systems at a low cost using unstructured and unlabeled data. The authors of this review article have presented detailed literature on self-supervised learning as well as its applications in different domains. The primary goal of this review article is to demonstrate how images learn from their visual features using self-supervised approaches. The authors have also discussed various terms used in self-supervised learning as well as different types of learning, such as contrastive learning, transfer learning, and so on. This review article describes in detail the pipeline of self-supervised learning, including its two main phases: pretext and downstream tasks. The authors have shed light on various challenges encountered while working on self-supervised learning at the end of the article.

AI snapshot

Written from the paper’s full text.

TL;DR

This review summarizes self-supervised learning (SSL), covering learning paradigms, pretext and downstream tasks, contrastive versus non-contrastive methods, applications, datasets, and challenges. It argues SSL can exploit unlabeled data to learn useful representations for downstream vision, NLP, and healthcare tasks, while noting difficulties in designing pretext tasks and applying SSL to unstructured or multi-object domains.

Key findings

  • SSL is presented as a bridge between supervised and unsupervised learning: it auto-generates labels from unlabeled data and uses two main phases, pretext tasks and downstream tasks, for representation learning.
  • The review distinguishes contrastive learning, which uses positive and negative samples, from non-contrastive learning, which uses only positive sample pairs; Table 1 states contrastive learning is prone to dimensional collapse and uses complex neural networks, while non-contrastive learning has lesser dimensional collapse and simpler networks.
  • Applications are reviewed across NLP, healthcare, and computer vision. Cited examples include BERT/T5, ERNIE 2.0, and ALBERT in NLP; Jamaludin et al. using 1016 subjects for spinal MRI; Kwasigroch et al. using 2000 training, 150 validation, and 600 testing images for skin lesion classification; Ghesu et al. using 100 million medical images; and Nguyen et al. using StructSeg with CT images of 60 lung cancer patients and RSNA with 17,079 images.
  • The critical analysis states that SSL performance improves as dataset size grows, but SSL struggles when images contain multiple objects, as in self-driving scenes, and when images lack structure, as in satellite and medical images, where relative patch prediction and jigsaw puzzles are described as useless.

Methods

  • Design: a succinct narrative literature review; no new experiments or primary empirical study are reported.
  • Data/sample: synthesizes prior SSL literature and lists datasets used in computer vision, medical imaging, and NLP, including CIFAR-10 with 60,000 images, PASCAL VOC 2012 with 11,530 images, MS COCO with 328,000 images, and Kinetics 400 with 400 video clips per human action.
  • Main analysis: qualitative organization and discussion of learning algorithms, SSL tasks, SSL learning techniques, applications, taxonomy, datasets, critical analysis, and future directions.

Limitations

  • Stated in the paper: SSL pre-trained models often rely on ImageNet-style images with a single dominant object; multi-object scenes are hard to distinguish; satellite and medical images lack structure, making relative patch prediction and jigsaw puzzles ineffective; different augmentation methods are needed for less structured datasets; and designing a useful pretext task is the most difficult aspect of SSL.
  • Evident from the text: as a succinct narrative review, it does not describe a systematic search or selection protocol, risk-of-bias assessment, or quantitative meta-analysis.
  • Evident from the text: many numerical details are reported from cited secondary studies rather than independently verified by the review authors.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11831-023-09884-2

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 37 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    As dataset size grows, system performance grows.

    “After reviewing a large amount of literature, we discovered that as the size of the dataset grows, so does the system's performance.”
    Quote in the paperNot a substantial claim
  • Interpretationreasoning

    Self-supervised learning does not require expensive manual annotations.

    “Self-Supervised Learning (SSL) is a solution to the aforementioned issues that has emerged as one of the most promising techniques that does not necessitate any expensive manual annotations.”
    Quote in the paperSubstantial claim
  • Premisecitation

    Manual labeling of data is time-consuming and labor-intensive.

    “Manually labeling data is a time-consuming and labor-intensive process [1].”
    • Also stated in: § Motivation Behind the Work · plain
      “Labeling each data point manually or with data labelling software is a costly and time-consuming task.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [1] Albelwi S (2022) Survey on self-supervised learning: auxiliary pretext tasks and contrastive learning methods in imaging. Entropy 24(4):551. https://doi.org/10.3390/e24040551 source

      Not checked in the free run.

  • Premisecitation

    Doersch et al. proposed using SSL to predict a missing patch by selecting a random patch and predicting the relative position of an image's second and central patches.

    “Doersch et al. [18] proposed using SSL to predict a missing patch. They chose a random patch and predicted the relative position of an image's second and central patches, as shown in Fig. 9b.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] Doersch C, Gupta A, Efros AA (2015) Unsupervised visual representation learning by context prediction. In: 2015 IEEE International Conference on Computer Vision (ICCV). https://doi.org/10.1109/iccv.2015.167 source

      Not checked in the free run.

  • Premisecitation

    HAR has enabled using unlabeled data to build reliable recognition systems with only a few labeled training samples.

    “HAR has enabled the field to address one of its most pressing issues: using unlabeled data to build reliable recognition systems with only a few labeled training samples [30].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] Haresamudram H, Essa I, Plötz T (2022) Assessing the state of self-supervised human activity recognition using wearables. arXiv preprint arXiv:2202.12938 source

      Not checked in the free run.

  • Classified as probably not a substantial claim. 51
  • May combine two claims. 21

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Self-supervised approaches have dominated supervised learning.

    “Self-supervised approaches have dominated supervised learning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    SSL has achieved remarkable success compared to supervised models in computer vision, NLP, and healthcare.

    “When compared to supervised models, SSL has achieved remarkable success in the computer vision, NLP, and healthcare domains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    SSL approaches have been shown effective in difficult downstream tasks such as image classification, object detection, and image segmentation with little labeled input.

    “Self-supervised learning approaches have been shown to be effective in difficult downstream tasks such as image classification, object detection, image segmentation, and other tasks with little labeled input.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The basic idea of transfer learning is to transfer information from a trained model on a data-rich task to an objective task with less data.

    “The basic idea behind using transfer learning is to transfer the information contained in a trained model on a task with a large amount of data to an objective task with less data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Chen et al. used a hybrid SSL approach to regularize training of a text classification task.

    “Chen et al. [35, 36] used a hybrid SSL approach to regularize the training of a text classification task.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [35] Chen T, Liu S, Chang S, Cheng Y, Amini L, Wang Z (2020a) Adversarial robustness: from self-supervised pre-training to fine-tuning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr42600.2020.00078 source

      Not checked in the free run.

    • cites [36] Chen T, Kornblith, Norouzi M, Hinton G (2020b) A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002. 05709

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The major categories of SSL learning are contrastive learning and non-contrastive learning.

    “The major categories of SSL learning are Contrastive learning and non-contrastive learning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    You and Wang proposed VEJP to recognize 3D human actions, which captures multi-view data and forces the encoder to obtain view-independent high-level features from the human skeleton.

    “You and Wang [31] proposed using a view-enhanced jigsaw puzzle (VEJP) to recognize 3D human actions. VEJP captures multi-view data and forces the encoder to obtain view independent high-level features from the human skeleton.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31] You W, Wang X (2022) View enhanced jigsaw puzzle for self-supervised feature learning in 3D human action recognition. IEEE Access 10:36385–36396. https://doi.org/10.1109/access.2022.3165040 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Pretext tasks allow the model to learn useful feature representations that are used to complete downstream tasks.

    “Pretext tasks, also known as secondary tasks, allow the model to learn useful feature representation information that is used to complete downstream tasks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Semi-supervised algorithms do not guarantee that adding only unlabeled data improves prediction performance, because unlabeled data helps only if it contains information useful for label prediction.

    “These algorithms do not guarantee that including only unlabeled data will improve prediction performance because unlabeled data is only useful when it contains information useful for label prediction.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Image classification is used as a downstream task to estimate the quality of an image's features.

    “Image classification is used as a downstream task to estimate the quality of an image's features.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    In contrastive learning, one sample is used as an anchor, its augmented form is positive, and remaining examples in the training batch are negative.

    “In contrastive learning one sample from the training data is used as an anchor, its augmented form is labeled as a positive sample, and the remaining examples in the training batch are labeled as negative samples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    The ConvNet model is the foundation of SSL, and its architecture influences the quality of visual representations learned through pretext tasks.

    “The ConvNet model serves as the foundation of SSL, and its architecture influences the quality of visual representations learned through the pretext task.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    SSL requires labels but does not require human labeling, which distinguishes it from unsupervised learning.

    “SSL differs from unsupervised learning in that it requires labels but does not require human labeling [2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2] Ohri K, Kumar M (2021) Review on self-supervised image recognition using deep neural networks. Knowl-Based Syst 224:7090. https://doi.org/10.1016/j.knosys.2021.107090 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Lan et al. proposed ALBERT, a two-parameter reduction technique using fewer parameters than traditional BERT.

    “Lan et al. [39] proposed a two-parameters reduction technique called ALBERT (A lite BERT) that use fewer parameters than traditional BERT.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R (2019) Albert: a lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The SSL model trains itself using one part of input data to learn another part.

    “The SSL model trains itself using one part of the input data to learn the other part of the input data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Downstream tasks can be accomplished by fine-tuning or by using a linear classifier.

    “Down streaming can be accomplished in two ways: fine-tuning or using a linear classifier [25].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] Isravel DP, Silas S, Rajsingh EB (2021) Self-supervised learning approaches for traffic engineering in software-defined networks. In: Advances in Intelligent Systems and Computing, pp 511–522. https://doi.org/10.1007/978-981-33-6984-9_41 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    TAPT and DAPT were proposed by Gururangan et al.; TAPT continues training on target task training data after pre-training RoBERTa, while DAPT pre-trains RoBERTa on datasets with minor domain differences from target tasks.

    “Task adaptive pre-training (TAPT) and domain adaptive pre-training (DAPT) have been proposed by Gururangan et al. [37]. TAPT continues to train it on the target task's training dataset, after pre-training RoBERTa on large-scale corpora. DAPT is still pre-training RoBERTa on datasets with minor domain differences from datasets in target tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37] Gururangan S, Marasović A, Swayamdipta S, Lo K, Beltagy I, Downey D, Smith NA (2020) Don’t stop pretraining: adapt language models to domains and tasks. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.740 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    HAR is frequently used to evaluate the quality of video features learned using self-supervised methods.

    “HAR is frequently used to evaluate the quality of video features learned using self-supervised methods.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Designing an appropriate pretext task requires domain knowledge.

    “Designing an appropriate pretext task necessitates domain knowledge.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Solving a jigsaw puzzle requires knowledge of a single patch and relationships between different patches of the same image.

    “Solving a jigsaw puzzle requires not only knowledge of a single patch, but also knowledge of the relationships between different patches of the same image, as shown in Fig. 9d [22].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Noroozi M, Favaro P (2016) Unsupervised learning of visual representations by solving jigsaw puzzles. In: European Conference on Computer Vision, pp 69–84. Springer, Cham. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Incremental learning aims to learn new knowledge from new samples and solve new tasks continuously without forgetting previous tasks.

    “the goal of incremental learning is to learn new knowledge from new samples [14] and solve new tasks on a continuous basis without forgetting previous tasks by using new data.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Luo Y, Yin L, Bai W, Mao K (2020) An appraisal of incremental learning methods. Entropy (Basel, Switzerland) 22(11):1190. https://doi.org/10.3390/e22111190 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Instagram is an example of weakly-supervised learning, where users' hash tags are used to label data.

    “Instagram is an example of weakly-supervised learning, in which users' hash tags are used to label data [2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2] Ohri K, Kumar M (2021) Review on self-supervised image recognition using deep neural networks. Knowl-Based Syst 224:7090. https://doi.org/10.1016/j.knosys.2021.107090 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The goal of contrastive learning is to bring semantically related samples closer together while keeping dissimilar samples separate.

    “The goal of contrastive learning is to bring semantically related samples closer together while keeping dissimilar samples separate [32].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] Bhattacharjee A, Karami M, Liu H (2022) Text transformations in contrastive self-supervised learning: a review. arXiv preprint arXiv:2203.12000 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    The SSL algorithm can auto-generate labels for unlabelled data, converting an unsupervised model to a supervised model.

    “The SSL algorithm has the ability to auto-generate the labels for un-labelled data, which converts the un-supervised model to a supervised model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    SSL's most fruitful results in NLP are BERT and T5.

    “SSL's most fruitful results in the field of NLP are BERT and T5.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Sun et al. proposed ERNIE 2.0, a framework for learning multiple tasks incrementally, and used GLUE to assess its performance.

    “Sun et al. [38] proposed ERNIE 2.0, a framework for learning multiple tasks in incremental increments. They created several tasks and trained the ERNIE 2.0 model to extract lexical, syntactic, and semantic information from the training dataset. They used General Language Understanding Evaluation (GLUE) to assess the model's performance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [38] Sun Y, Wang S, Li Y, Feng S, Tian H, Wu H, Wang H (2020) Ernie 2.0: a continual pre-training framework for language understanding. Proc AAAI Conf Artifi Intell 34(05):8968–8975. https://doi.org/10.1609/aaai.v34i05.6428 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In contrastive learning, three types of encoders are used: image encoders, momentum encoders, and dictionaries.

    “In contrastive learning, three types of encoders are used: image encoders, momentum encoders, and dictionaries [33].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] He K, Fan H, Wu F, Xie S, Girshick R (2020) Momentum contrast for unsupervised visual representation learning. IEEE/CVF Conf Comput Vis Pattern Recogn 2020:9726–9735. https://doi.org/10.1109/CVPR42600.2020.00975 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Contrastive learning distinguishes between augmented image features.

    “Contrastive learning, on the other hand, distinguishes between augmented image features.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Transfer learning refers to transferring knowledge from one domain to another.

    “Transfer learning refers to the transfer of knowledge from one domain to another [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] Weiss K, Khoshgoftaar TM, Wang D (2016) A survey of transfer learning. J Big Data 3:9. https://doi.org/10.1186/s40537-016-0043-6 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    BYOL and SimSiam proved that using stop-gradient and extra-predictor operations, non-contrastive learning does not suffer representation collapse.

    “BYOL and SimSiam proved that using these operations, the non-contrastive learning model does not suffer representation collapse.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    SSL-Reg uses a self-supervised task to regularize RoBERTa fine-tuning, whereas TAPT and DAPT use separate tasks.

    “The difference between the SSL-Reg technique and TAPT and DAPT is that SSL-Reg uses a self-supervised task (for example, mask token prediction) to regularize RoBERTa fine-tuning, whereas TAPT and DAPT use separate tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    FCN is a watershed moment in semantic segmentation because it employs a fully convolutional network.

    “FCN is a watershed moment in semantic segmentation because it employs a fully convolutional network to solve the problem.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    SSL employs two main concepts: auxiliary pretext tasks and contrastive learning.

    “SSL employs two main concepts: auxiliary pretext tasks and contrastive learning [4].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] Tao L, Wang X, Yamasaki T (2022) An improved inter-intra contrastive learning framework on self-supervised video representation. IEEE Trans Circ Syst Video Technol. https://doi.org/10.1109/tcsvt.2022.3141051 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Zhou et al. used SSL regularization in NLP for text classification, with prediction and self-supervision tasks on encoded text.

    “Zhou et al. [34] used the SSL regularization technique in NLP for text classification. They defined text classification as a key concept in NLP. Authors provided training texts that were encoded using a text encoder as input. In the encoded text, the authors defined two tasks: prediction and self-supervision.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [34] Zhou M, Li Z, Xie P (2021) Self-supervised regularization for text classification. Trans Assoc Comput Linguist 9:641–656. https://doi.org/10.1162/tacl_a_00389 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    In pretext tasks, the hidden portion of data is predicted using the visible portion.

    “In pretext tasks, the hidden portion of data is predicted using the visible portion.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    SSL tasks are divided into pretext and downstream tasks.

    “SSL tasks are divided into two categories: pretext and downstream tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Pretext tasks can be applied to any type of data, such as images, audio, and video.

    “The pretext task can be applied to any type of data, such as images, audio, video, and so on [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] Jaiswal A, Babu AR, Zadeh MZ, Banerjee D, Makedon F (2020) A survey on contrastive self-supervised learning. Technologies 9(1):2. https://doi.org/10.3390/technologies9010002 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Data augmentation can alter an image's visual appearance without changing its semantics.

    “Data augmentation can alter an image's visual appearance without changing its semantics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Supervised learning necessitates manual labelling of data, which slows system performance.

    “Supervised learning necessitates manual labelling of data, which slows the system's performance [8].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [8] Alloghani M, Al-Jumeily D, Mustafina J, Hussain A, Aljaaf AJ (2019) A systematic review on supervised and unsupervised machine learning algorithms for Data Science. In: Unsupervised and Semi-Supervised Learning, pp 3–21. https://doi.org/10.1007/978-3-030-22475-2_1 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    SSL is considered a bridge between supervised and unsupervised learning.

    “SSL is considered to be the bridge between supervised learning and unsupervised learning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    In pretext tasks, pseudo labels are generated automatically from dataset attributes and are used for classification, detection, and segmentation.

    “In pretext tasks, pseudo labels are used for representations that were generated automatically by taking into account the attributes of the dataset. These pseudo labels are then used for classification, detection, and segmentation [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] Jaiswal A, Babu AR, Zadeh MZ, Banerjee D, Makedon F (2020) A survey on contrastive self-supervised learning. Technologies 9(1):2. https://doi.org/10.3390/technologies9010002 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Contrastive learning learns a representation or feature space that attracts and repels representations from similar images.

    “Contrastive learning is used to learn a representation or feature space that attracts and repels representations from similar images.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Auxiliary pretext tasks are primarily used to fill in missing image parts, convert images to grayscale, and predict hidden parts.

    “The auxiliary pretext is primarily used to fill in missing parts of an image [6], convert it to gray scale [7], predict hidden parts, and many other tasks.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6] Pathak D, Krähenbühl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: feature learning by inpainting. IEEE Conf Comput Vis Pattern Recogn 2016:2536–2544. https://doi.org/10.1109/CVPR.2016.278 source

      Not checked in the free run.

    • cites [7] Larsson G, Maire M, Shakhnarovich G (2016) Learning representations for automatic colorization. In: Computer Vision—ECCV 2016, pp 577–593. https://doi.org/10.1007/978-3-319-46493-0_35 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    SSL became a hot research topic since 2018, when Google introduced the NLP model.

    “SSL has become a hot research topic for researchers since 2018, when Google introduced the natural language processing (NLP) model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Semi-supervised learning uses both labelled and unlabeled data to perform specific tasks.

    “Semi-supervised learning uses both labelled and unlabeled data to perform specific tasks [9].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] Engelen JEV, Hoos HH (2019) A survey on semi-supervised learning. Mach Learn 109(2):373–440. https://doi.org/10.1007/s10994-019-05855-6 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Li et al. solved jigsaw puzzles using a GAN model with a multitask pipeline for unpaired image jigsaw puzzles.

    “Li et al. [24] solved Jigsaw puzzles using the GAN model. They created a multitask pipeline for solving unpaired image jigsaw puzzles.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] Li R, Liu S, Wang G, Liu G, Zeng B (2022) Jigsawgan: auxiliary learning for solving jigsaw puzzles with generative adversarial networks. IEEE Trans Image Process 31:513–524. https://doi.org/10.1109/tip.2021.3120052 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Catastrophic forgetting is the primary issue with incremental learning.

    “This is referred to as catastrophic forgetting (CF), and it is the primary issue with incremental learning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Weak supervision is classified into incomplete, inexact, and inaccurate supervision.

    “Weak supervision is classified into three types: incomplete supervision, inexact supervision, and inaccurate supervision [10].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] Zhou ZH (2017) A brief introduction to weakly supervised learning. Natl Sci Rev 5(1):44–53. https://doi.org/10.1093/nsr/nwx106 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Non-contrastive learning techniques depend only on positive sample pairs.

    “Non-contrastive learning techniques only depends on positive sample pairs which means that the training data contains only related representations”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    SSL can address problems caused by supervised learning.

    “The findings explain in their article that SSL has the ability to solve the problems caused by supervised learning.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Chowdhury et al. [40] Chowdhury A, Rosenthal J, Waring J, Umeton R (2021) Applying self-supervised learning to medicine: review of the State of the art and medical implementations. Informatics 8(3):59. https://doi.org/10.3390/informatics8030059 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    SSL has been widely applied in computer vision tasks including object detection, image classification, graph classification, and visual question answering.

    “SSL has been widely used in computer vision applications such as object detection, image classification, graph classification, visual question answering, and so on.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    When data from multiple sensors or modalities are available, their mutual constraints can train networks to learn features.

    “If multiple data modalities from other sensors are available, the constraints between them can be used to train networks to learn features.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Few studies have investigated learning multiple pretext tasks for self-supervised feature learning.

    “Only a few studies have examined learning multiple pretext tasks for self-supervised feature learning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Various pretext tasks generate different supervision signals that can help a network learn more typical characteristics.

    “Various pretext tasks generate different supervision signals, which can help the network learn more typical characteristics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Most current self-supervised visual feature learning algorithms train ConvNet on a single pretext task.

    “In most existing self-supervised visual feature learning algorithms, ConvNet is trained to solve one pretext task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Downstream SSL tasks use fine-tuned models to achieve the highest performance.

    “To achieve the highest level of performance, downstream tasks in SSL use fine-tuned models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    SSL is attractive for its data efficiency, using fewer labels and smaller samples while incorporating a neural network.

    “Self-supervised learning has attracted many researchers for its excellent data efficiency. In this method, fewer labels and smaller samples are used to learn more by incorporating a neural network.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    In NLP, a few words can complete an incomplete sentence by using the meaning of the preceding sentence.

    “Later words can be completed by understanding the semantics of the previous sentence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Most current self-supervised visual feature learning methods learn features for only one modality.

    “The majority of self-supervised visual feature learning approaches currently available are focused on learning features for a single modality.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Datasets for self-supervised training are collected without human-annotated labels.

    “These datasets are collected for self-supervised training without the use of human-annotated labels.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Object detection requires dense labeling, which is time-consuming.

    “Object detection necessitates dense labeling, which is a time-consuming process,the FSOD method attempts to recognize previously unseen object classes based on a few labels.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Huang et al. [48] Huang G, Laradji I, Vazquez D, Lacoste-Julien S, Rodriguez P (2021) A survey of self-supervised and few-shot object detection. arXiv preprint arXiv:2110.14711 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Relative patch prediction and jigsaw puzzles are useless for satellite and medical images.

    “As a result, approaches like relative patch prediction and jigsaw puzzles are useless when dealing with such images.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Satellite and medical images lack structure, making context finding extremely difficult.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    When combined with other learning methods, SSL can achieve greater success.

    “When combined with other learning methods, SSL can achieve greater success.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Automating data curation is necessary to cut manual input, and fewer labels should be needed for good performance.

    “The data curation process must be automated to reduce manual input, and the number of labels required for good performance must be reduced [57].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [57] Taherkhani F, Dabouei A, Soleymani S, Dawson J, Nasrabadi NM (2021) Self-supervised Wasserstein pseudo-labeling for semi-supervised image classification. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 12262–12272. https://doi.org/10.1109/CVPR46437.2021.01209. source

      Not checked in the free run.

64 of 64

  • The paper relies on this without support.

    Unlabeled data is abundant and easily available.

    “Unlabeled data, on the other hand, is available in plenty and easily.”

  • The paper relies on this without support.

    Most artificial intelligence techniques require labelled datasets to make predictions.

    “Most artificial intelligence techniques require labelled datasets to make predictions.”

  • The paper relies on this without support.

    A large dataset is required to train machines to predict.

    “A large dataset is required to train machines to predict.”

  • The paper relies on this without support.

    Humans can learn faster than machines by observing and sensing their surroundings.

    “One important distinction between humans and machines is that humans can learn faster than machines by observing and sensing their surroundings.”

  • The paper relies on this without support.

    Machines can take hours or even days to simulate.

    “Machines, on the other hand, can take hours or even days to simulate.”

  • The paper relies on this without support.

    The quality of features increases as the number of layers in a network increases.

    “The quality of features increases as the number of layers in a network increases.”

  • The paper relies on this without support.

    When the domain gap between self-supervised pre-training and the downstream task is smaller, performance on the downstream task is usually better.

    “When the domain gap between self-supervised pre-training and the downstream task is smaller, performance on the downstream task is usually better.”

  • The paper relies on this without support.

    The vast majority of self-supervised pre-trained models, such as those trained on ImageNet, are trained on images with a single dominant object.

    “The vast majority of self-supervised pre-trained models, such as those in the ImageNet dataset, are trained on images with a single dominant object.”

  • The paper relies on this without support.

    Scenes in applications such as self-driving cars contain several items, making it difficult to distinguish between two similar scenes.

    “The scene in applications such as self-driving cars contains several items, making it difficult to distinguish between two similar scenes.”

  • The paper relies on this without support.

    Satellite and medical images lack structure, making context finding extremely difficult.

    “Finding context in satellite and medical images is extremely difficult due to their lack of structure.”

  • The paper relies on this without support.

    Unlabeled data is freely available in vast amounts.

    “They use the vast amounts of unlabeled data that are freely available.”
11 of 11

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • HMDB51 clip count does not match the stated per-category count

    Minor

    The table says there are 51 categories with 101 clips each, which gives 5,151 clips, not the reported 6,849.

    Reported 6849 · recomputed 5151

  • The same Chen citations are assigned different research topics

    Minor

    The NLP section describes references [35, 36] as regularizing text classification, whereas Table 2 describes those same references as a visual-representation framework evaluated on ImageNet.

  • PASCAL VOC 2007 test counts exceed its stated total size

    Minor

    Table 2 lists two test-image counts for PASCAL VOC 2007, each larger than the 9,963 images Table 3 gives as the dataset's total size.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Orhan, AE, Gupta VV, Lake BM (2007) Self-supervised learning through the eues of a child 2020, arXiv e-prints, arXiv-2007
  2. Chen T, Kornblith, Norouzi M, Hinton G (2020b) A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002. 05709

Cited by

Sorted by citations

Related

Authors
Rani, V., Nabi, S. T., Kumar, M., Mittal, A., & Kumar, K.
Affiliations
  • Maharaja Ranjit Singh Punjab Technical University (IN)
  • Panjab University (IN)
Corresponding author
Kumar, Munish
ORCID
Venue
Archives of Computational Methods in Engineering
Publisher
Springer Science and Business Media LLC
Published
2023-01-20
Volume
30
Issue
4
Pages
2761-2775
ISSN
  • 1134-3060 (ISSN-L)
  • 1886-1784
DOI
10.1007/s11831-023-09884-2
PMC
PMC9857922
OpenAlex
W4317542939
Semantic Scholar
0a1c21e12d9dea630ae10459a8ca8a25dc8bfe8c
Cited by
295
References
52
Citations by year
2026: 77 · 2025: 133 · 2024: 71 · 2023: 14
FWCI
6.17
Type
journal article
Language
en
Open access
Licence
Locations
  • Archives of Computational Methods in Engineering (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
self-supervised learning, contrastive learning, transfer learning, pretext tasks, downstream tasks, unsupervised learning, image learning, visual features, computer vision, object detection, image segmentation, image understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review