Unsupervised Representation Learning by Predicting Image Rotations

“In recent years, the widespread adoption of deep convolutional neural networks (LeCun et al., 1998) (ConvNets) in computer vision, has lead to a tremendous progress in the field.”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“Specifically, our self-supervised approach manages to drastically improve the state-of-the-art results on unsupervised feature learning for ImageNet classification, PASCAL classification, PASCAL detection, PASCAL segmentation, and CIFAR-10 classification, surpassing prior approaches by a significant margin and thus drastic…”

“…(Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al. (2014); Donahue et al. (2016); Radford et al. (2015).”

“Specifically, we first train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Colorization (Zhang et al.,…”

“…feature learning are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al.…”

“…59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et al., 2016) 41.9 32.2 NAT (Bojanowski & Joulin,…”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“InspiredbyRotNet[24],welearnthesemanticfeaturesof imagerotationinadata-drivenwaytoestimatethegeometric transformations.”

“For example, some previous work predicts image rotation (Gidaris et al. 2018), images colorization (Zhang et al. 2016), and puzzle solving (Noroozi and Favaro 2016).”

“21 (2) The model is trained with very few labeled data during the downstream task.”

“Common pretext tasks include predicting image rotations [17] or patch positions [13], coloring images [66] and solving jigsaw puzzles [37, 57], etc .”

“masked modeling [8], rotation parameters prediction [14], .”

“, rotation recognition [16]) on Du to pre-train the encoder G(·), the classifier F(·), and the rotation predictor H(·).”

“Other simple methods are used, like the rotation of the image [26], inpainting [27], or coloring gray-scale images [28] or more complex techniques like contrastive learning [29].”

“E.g., images are rotated and the machine learning model is tasked with predicting the degree of rotation [30].”

“By pretraining the model on a pretext task [24, 32, 67], the model adapts to the structure of the target domain, i.e., herein Computerized Tomography Pulmonary Angiograms.”

“While early work on SSL focused on designing various pretext tasks such as solving jigsaw puzzles [36], colouring [51, 33, 52] and predicting rotation [18], more recent techniques focused on learning augmentation invariant representation [49] [46] to replace the loss functions that require manual supervision.”

“Geometric transformations as self-supervision have obtained tremendous success on image unsupervised learning tasks [13, 18, 19, 47, 48] and the core idea is that DNNs are considered to be able to excavate useful information of image data if they are able to recognize their transformed patterns [18].”

“(2015); Noroozi and Favaro (2016) proposed to transform input images into patches and attempted to solve context prediction and jigsaw-puzzles respectively, while others found it useful to predict a prior rotation transformation (Gidaris et al., 2018).”

“The projection head is used to isolate the influence of pseudo labels on generated representations because high-layer features are found to be overly discriminative to the auxiliary task [29], which is different from the final task of condition monitoring.”

“The paper also compared the proposed algorithm with other self-supervised methods, including Rotation prediction method [50], instance discrimination method [44], Moco V1 [51], and SimCLR [52].”

“Some early, but influential 53 ones include: predicting image rotations [7], solving a jigsaw 54 puzzle [8], gray scale image colorization [9], automatic image 55 inpainting [10], spatio-temporal geometric deformations in a 56 video sequence [11], or utilizing 2D heatmaps as a supervision 57 signal [12].”

“Early techniques involved pretext tasks , e.g. colorization, inpainting, or spatial reasoning (Doersch et al., 2015; Zhang et al., 2016; Pathak et al., 2016; Noroozi & Favaro, 2016; Gidaris et al., 2018).”

“Prominent examples of pretext tasks for image understanding include solving jigsaw puzzles (Noroozi and Favaro, 2016), learning image rotations (Gidaris et al., 2018), predicting color channels from grayscale images and vice-versa (Zhang et al.”

“Recently, self-supervised learning, a branch of the unsupervised learning technique, has raised a lot of interest and shown great success in multiple fields such as CV [24], [25], [26] and NLP [27], [28], [29].”

“Inspired by self-supervised pretraining techniques in natural language processing [40, 41, 46], other SSL methods design pretext tasks ( e.g. colorization, rotation prediction, and patch reconstruction) on unlabeled images [5, 6, 14, 15, 20, 29, 33, 35].”

“…learning methods focus on designing image-based pretext tasks with pseudo-labels as supervision signals for learning representations such as image colorization [37, 38], object counting [46, 50], predicting rotations [24], solving puzzles [5, 31], image inpainting [28], and image clustering [6, 7].”

“Distinguished from traditional supervised learning, SSL capitalizes on unlabeled data, deriving proxy tasks from the data itself to train models without human annotation (18).”

“2) Generative representation learning (GRL) methods (Denoising [89], In-painting [69], Models Genesis [68], Rotation [90]) provide baseline performance for classic approaches.”

“Typically, after pre-training in either a supervised [18, 46] or self-supervised [47–51] manner, fine-tuning is performed on smaller datasets for downstream tasks.”

“Pioneer works for SSL pro-pose pretext tasks (Doersch et al., 2015; Pathak et al., 2016; Zhang et al., 2016; Noroozi & Favaro, 2016; Gidaris et al., 2018), and recently, contrastive learning (Chen et al., 2020b; He et al., 2020; Chen et al., 2021b; Caron et al., 2021) and masked image modeling (Bao…”

“Early self-supervised pretext tasks include predicting rotations [10], colorizing grayscale images [11], and solving image jigsaw puzzle [12].”

“(3): Random 0.588(.07) 0.782(.03) 0.840(.03) 0.928(.00) Rotation [16] 0.572(.08) 0.809(.03) 0.868(.02) 0.925(.00) PIRL [10] 0.492(.03) 0.823(.04) 0.865(.01) 0.927(.00) SimCLR [9] 0.352(.06) 0.725(.08) 0.824(.04) 0.927(.00) GCL [17] 0.636(.05) 0.803(.04) 0.872(.01) 0.927(.01) PCL [12] 0.671(.06)…”

“To acquire robust visual representations, some methods [41], [43] introduce image rotation as a straightforward yet effective pretext task.”

“Unlike the other equivariance methods where the transformation parameters are estimated directly [10], [21], the TINC loss (4) enforces it with the margin in the loss function as a soft constraint.”

“This includes tasks like rotation prediction [18], transitioning between grayscale and color images [28,69], and inpainting [46].”

“RotNet[21] randomly rotates the labeled images and trains the model on the rotated images to generate improved representations for the mask task of object recognition and image classification.”

“(Chen et al., 2020; Gidaris et al., 2018; Oord et al., 2018; Wu et al., 2018; Tian et al., 2020a).”

“When dealing with image data, possible pretext tasks include predicting the relative positions of two randomly selected patches of a same image [4] and predicting which rotation angle has been applied to an image [5].”

“Examples of pretext tasks include cross-channel prediction [11], image context restoration [12], image rotation prediction [13], image colorization [14], image super-resolution [15], image inpainting [16], and resolution sequence prediction (RSP) [17].”

“Common pretext tasks for images include context prediction, 22,25 jigsaw puzzles, 26 rotation prediction, 27 and colorization; 28 all of which are not directly applicable to microscopy images.”

“These signals were based on some independent tasks like verifying the correct rotation [11], the correct sequence of frames [13], or the correct placement of tiles [12].”

“Rotation prediction [6] trains a model to predict the rotation degree of an image in various orientations.”

“RotNet [49] trains a model to predict the rotation of examples, while the other approaches [32], [50] focus on instance discrimination.”

“To this end, various pretext tasks have been designed, including jigsaw [12], hole-fill [30] and transformation invariance [16].”

“Self supervised learning (SSL) has become an increasingly popular research area in recent years [11, 15, 23, 65, 104].”

“Early work on unsupervised representation learning has focused on designing pretext tasks and training the network to predict their pseudo labels [17], [18].”

“Early SSL approaches employed simple augmentation tasks such as predicting the transformation of images [7] or determining the spatial arrangement of image patches [8] for pretraining the network backbone.”

“(Hendrycks et al., 2019; Gidaris et al., 2018; Tack et al., 2020) need to train the model with self-supervision first and then perform OOD detection using the performance of the self-supervision task.”

“Recently, many self-supervised approaches have been proposed in the vision domain, such as pre-text based methods (Doersch et al., 2015; Zhang et al., 2016; Gidaris et al., 2018), contrastive learning with Siamese networks (Oord et al.”

“Predicting the enhancing methods utilized for images, predicting the relative placements of image blocks, recolor-ing sample grayscale maps [21], and restoring the missing parts of images to complete them [22] are some of the generative methods proposed by previous studies.”

“Forcomputervision, someworksusepretextssuchascolorization[43],rotation prediction[44],andpatchpositionprediction[45],[46]tolearn representativefeaturesforimagedata.”

“Remarkable methods are generating surrogate classes with data augmentation [18], predicting the relative position of patches [16], solving jigsaw puzzle [9, 33,46,49,50], and predicting the degree of rotation [22].”

“Early self-supervised approaches attempted to learn useful representations from unlabeled data via solving handcrafted pretext tasks such as Jigsaw puzzle [27], colorization [40], transformation prediction [1, 14], and inpainting [31].”

“analysis, such as in-painting [7], colorization [8] or predicting transformations [9].”

“Later self-supervised approaches used the pretext tasks such as solving jigsaw puzzles [33], image rotation estimation [18], super-resolution [20], colorization [53], and inpainting [34].”

“Without access to any data annotations, self-supervised learning (SSL) approaches impose additional constraints or hypothesis among distinct views of the original input data [2, 7, 13, 14, 32, 33, 39, 41, 44, 52].”

“Early SSL methods design hand-crafted pretext tasks [13, 67, 17], which rely on somewhat ad-hoc heuristics.”

“Arguably, the most important design choice is the pretext task, with prominent families including structural alignment [65], [66], [67], [68], clustering [69], contrastive learning (e.g., instance discrimination [70], [71] and multi-view learning [72], [73]), masked prediction [74], [75], [76],…”

“There are many classic self-supervised pretext tasks, such as relative position prediction [19], spatial transformation prediction [20], image reconstruction [21], and image colorization [22].”

“…by learning pretext tasks, e.g., colorizing grayscale images [57], image inpainting [58], solving jigsaw puzzle [59], predicting rotations [60], etc. SSL has been applied to many fields, such as representation learning [61], [62], semi-supervised learning [63], few-shot learning [64],…”

“To further improve performance, each base classifier is first trained on the whole training set by semi-supervised learning, focusing on predicting image rotation angles [20], then is fine-tuned on its dedicated partition through supervised learning.”

“In the computer vision domain, SSL is a method used to learn generalizable visual representations, often employing various pretext tasks for pretraining [17, 23, 48].”

“Recently, advances have been made to develop techniques [9], [15], [16], [17], [18], [19], [20], [21], [22], which can learn meaningful representations of the underlying data without human annotations.”

“…learning (SSL) has demonstrated remarkable capability in acquiring robust and generalized vi-ImageConditionedGenerativeModel sual representations from abundant unlabeled data sources [2, 15, 25, 26, 29, 33, 34, 38, 50, 62, 63, 92, 102], which can be transferred or leveraged in downstream tasks.”

“Meticulously designed pretext tasks, such as relative location prediction (Doersch et al., 2015), colorization (Zhang et al., 2016), jigsaw puzzle solving (Noroozi & Favaro, 2017) and rotation prediction (Gidaris et al., 2018), are empirically demonstrated to learn meaningful visual representations.”

“In earlier works, labels were generated by diverse pre-text tasks such as prediction of rotation [41], colorization [85], pathces positions [24] and others [38].”

“…contrast loss, so that similar samples are closer, and heterogeneous samples are far away; context-instance self-supervision [8], [11], [12], [13], [14], where an auxiliary task is designed to generate transformed data from the original unlabeled data according to certain rules, and assign…”

“Early pre-text tasks based pretraining designs objective that does not require additional labels, such as rotation prediction [18], context restoration [48], reconstruction [32], among others [24, 42, 15].”

“(4) Pretext task pre-training, including Rotation [23], Inpainting [28], and Jigsaw [26].”

“In the most common form of self-supervised learning, a model is trained on an annotation-free pretext task, e.g., [24, 35, 45, 56, 57, 59], and then fine-tuned on a downstream task of interest with fewer annotated samples.”

“inpainting [8], or rotation prediction [5]) and tasks a neural network to solve it, with the aim of generating latent representations that capture high-level image semantics.”

“The common practice in computer vision is to design proxy tasks to allow visual representation learning from unlabeled images [7, 23, 27, 9].”

“The auxiliary objectives can be handcrafted pretext tasks, such as relative patch prediction (Doersch, Gupta, and Efros 2015), solving jigsaw puzzles (Noroozi and Favaro 2016) and rotation prediction (Komodakis and Gidaris 2018).”

“Several data augmentations, including appearance transformation, such as ColorJitter [33], and spatial geometric transformation, such as horizontal flipping, rotation, cropping, and resizing [34] were applied in our study to increase data diversity.”

“Mainstream approaches either manually design specific pre-text tasks to assimilate the intrinsic data structure [8], [12], [13], [14], [15], or encode data similarities with a contrastive self-supervised learning (CSL) paradigm [9], [16], [17], [18].”

“, 2019), jigsaw (Gidaris et al., 2018) and rotation (Carlucci et al.”

“Example of such frameworks include relative patch prediction [11], solving jigsaw puzzles [33], colorization [51] and rotation prediction [7, 23].”

“For the cases of ManyShotLinear, ManyShotFinetune, and FewShotKornblith, we recover geometric properties that were previously considered important for classification [49, 59].”

“SSL has been widely applied in NLP [1, 2, 3, 4, 5] and computer vision [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16].”

“Typical works included patch ordering [18], solving jigsaw puzzles [43], and rotation prediction [23].”

“Approaches in CV include rotating an image and predicting the rotation angle [75, 222], splitting an image into smaller segments and solving a jigsaw puzzle from the shuffled segments [146], predicting the relative spatial location of two segments of the same image [52], and using triplet loss…”

“, Fréchet distance (FD) [8, 18], Maximum Mean Discrepancy [2]); and 3) model performance on related pretext task [7, 13], such as rotation prediction accuracy.”

“Examples include solving puzzles [31], image colorisation [32], and predicting image rotations [33].”

“auxiliary tasks to learn features relevant to the downstream task, and has emerged as a crucial component when DNNs are trained on difficult tasks with insufficient labeled data [6], [7], [8], [9].”

“Incremental learning: Prototype classifier [28] is a popular approach feature extractor 𝜙 (·) trained by the base session remains fixed feature c 𝑘 𝑛𝑒𝑤 Δ 𝑐 𝑛𝑒𝑤 𝑏,𝑛𝑒𝑤 𝑛𝑒𝑤 𝑏 𝑛𝑒𝑤 𝑛𝑒𝑤”

“The SOTA methods include well-known approaches, such as Rotation [26], InstFeat [27], MoCov2 [14], SimCLR [19], Fix-Match [28], SWAV [29], BYOL [30], SimSiam [20], MPL [31], and SimMatch [32].”

“Several works attempt to transfer instead mutual relations mined amongst the network outputs extracted from a batch of training instances.”

“…requires an appropriate pretext task which replaces a data-defined objective, and over the years, a plethora of such tasks have been proposed [28, 32, 44, 71], with Joint-Embedding Architectures (JEA) [15, 16, 18–21, 34, 36, 45, 69, 72], and Masked image modeling (MIM) [59, 60] gaining the most…”

“Self-supervised learning has been utilized for many vision tasks by formulating diverse pre-text tasks, including predicting context [56] or image rotation [57].”

“Contrastive learning that pulls positive samples closer and pushes negative samples apart is widely believed to be a prominent I-SSL paradigm, leading to rapid progress in recent years [13, 50, 5, 80, 46, 30, 3, 32, 33, 60, 49, 14, 37, 15, 77, 64, 81].”

“Then, some researchers began to design specific pretext tasks to train the models, which were used for other downstream tasks [30], [31], [32], [33], [34], [35], [36], [37], [38].”

“Early SSL approaches rely on several pretext tasks, such as rotation prediction (Gidaris et al., 2018), jigsaw puzzle (Noroozi & Favaro, 2016), and colorization (Zhang et al., 2016).”

“Specifically, in the design stage of the agent task, firstly, according to the study of method [41], the image is processed for Angle rotation and color channel ranking.”

“Numerous self-supervised pretext tasks have been proposed, including relative position prediction [60], spatial transformation prediction [61], image inpainting [62], image colorization [63], and jigsaw puzzle [64].”

“Existing SSL methods mainly construct pretext tasks with spatial relationships through jigsaw puzzles [52], inpaint [53], colorization [54], rotation [55], etc., which can guide the model to learn the spatial relationship of the target.”