Unsupervised Representation Learning by Predicting Image Rotations

“In recent years, the widespread adoption of deep convolutional neural networks (LeCun et al., 1998) (ConvNets) in computer vision, has lead to a tremendous progress in the field.”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“Specifically, our self-supervised approach manages to drastically improve the state-of-the-art results on unsupervised feature learning for ImageNet classification, PASCAL classification, PASCAL detection, PASCAL segmentation, and CIFAR-10 classification, surpassing prior approaches by a significant margin and thus drastic…”

“…(Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al. (2014); Donahue et al. (2016); Radford et al. (2015).”

“Specifically, we first train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Colorization (Zhang et al.,…”

“…feature learning are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al.…”

“…59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et al., 2016) 41.9 32.2 NAT (Bojanowski & Joulin,…”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“An example of it is RotNet [27], where the model has to predict the rotation that was applied to an image from the possible 0, 90, 180 and 270 degrees values.”

“In [30]–[33], self-supervised regularization is applied, where the difference of images before and after the affine/projective transform is compared.”

“SSPSF method is compared with well-known image classification methods and self-supervised learning methods: Resnet18, VGG16 [49], Rotation [50], and SimCLR [51].”

“Recently, with the advancement of research on pretext tasks [13,12,18,33,34,44], self-supervised learning (SSL) presents extraordinary potential in computer vision, pushing the frontier of transfer learning.”

“ARViT is pre-trained on the LSVRC-2012 ImageNet dataset with 1.3M images using a self-supervised approach, with rotation estimation [10] as a pretext task, then fine-tuned on smaller supervised classification down-stream tasks.”

“[38] combined self-supervised learning [44] with the manifold mixup [45] technique (S2M2) in training to improve the representation ability of the transfer-based feature extractor.”

“…of color channels (Zhang, Isola, and Efros 2016) or image patches (Pathak et al. 2016), or the recognition of various image transformations (Gidaris, Singh, and Komodakis 2018; Jenni and Favaro 2018), While prior patch-based methods inspired our approach of detecting wrongly placed image…”

“Representations learned from supervised learning [18, 33, 52], disentangled learning [8, 10, 19, 28, 54], and most recently self-supervised learning [9, 14, 16, 42, 43] now typify modern image representations.”

“Self-supervised representation learning (SSRL) [25] is used to extract meaningful representations for solving various visual tasks, such as image classification [26], object detection [27], and visual tracking [28], to name”

“5 Unsupervised - RotNet Linear (Gidaris et al., 2018) 81 .”

“Different from the previous methods by predicting rotations [8], colorization [19] and cluster assignments [2], contrastive learning makes positive and negative samples from original data by data augmentation [1,15].”

“With few exceptions (Gidaris et al., 2018; Hundman et al., 2018; Shen et al., 2020), there is much less work on other domains such as time series.”

“SSL has a rich history of leveraging various pretext transformations for representation learning, such as rotation [15], patch shuffling [11, 43], and inpainting [44] in images, as well as 3D rotation [27], clip shuffling [59], and acceleration [2, 63] in videos.”

“Early versions of SSL models learned features by solving pretext tasks such as predicting rotations [9], vanilla contrastive learning [10], masked auto-encoding [11], and forms of self-distillation [12, 13].”

“To the best of our knowledge, ours is the first work to explore test-time training applied to VICL models, where previous self-supervised pretext tasks [9, 12, 18, 22, 25, 34, 54, 55, 60] cannot be employed due to their constraints on the single-image context.”

“Automatic chromosome classification studies are greater in number than chromosome orientation correction studies [40], [41].”

“…Ima-geNet linear probing (c, left) , this correlation is much weaker than that observed among high-level vision tasks ( e.g ., Caltech-256 [27] vs. ImageNet classification) (c, right) , as indicated by the R 2 statistics. computer vision has made significant strides [5–8, 25, 43, 46, 47, 66, 71].”

“One type of visual pre-training method learns features based on the image itself using pretext tasks such as predicting transformations [15], inpainting [31, 41] or recovering masked regions [17, 21, 57].”

“In recent years, many pre-training tasks have been proposed in the literature, including contrastive learning [20], predictive coding [21], and rotation prediction [22].”

“The advancement of self-supervised learning has been illustrated in previous work [1], which involve generating pre-text tasks, including inpainting [10], rotation [9] and contrastive coding [2], to embed intrinsic features.”

“Self-supervised learning has evolved from more specialized pretext tasks such as solving rotations (Gidaris et al., 2018), jigsaw puzzles (Noroozi & Favaro, 2016), and relative positions (Doersch et al., 2015), to a predominant set of tasks based on instance discrimination (He et al., 2020; Chen et…”

“…(Devlin et al., 2019; He et al., 2022), contrastive SSL objectives (Chen et al., 2020b; Giorgi et al., 2021) and context-based SSL objectives (Larsson et al., 2016; Gidaris et al., 2018)), generative paradigms have achieved impressive performance in vision, language, and multi-modal domains.”

“…savings with a higher upper bound compared to probing with MLP. (c) Normalizing manufacturability regression tasks statically without incorporating heuristic estimation yields robust performance, while dynamic normalization is advantageous when reliable task-dependent input is available.”

“Concretely, TTP builds a correlation between human pose estimation and an auxiliary task ( i.e . image rotation prediction [8] or unsupervised landmark detection [13]) by a shared encoder.”

“Early works explored pretext tasks such as predicting the rotation angle [11] and recovering the original image from random per-muted patches [34, 35].”

“…representations has made remarkable progress in recent years (Chen et al., 2020a; Bardes et al., 2022; Zhou et al., 2022a; Larsson et al., 2016; Gidaris et al., 2018; Bachman et al., 2019; Gidaris et al., 2021; Grill et al., 2020; Shwartz-Ziv et al., 2022; Misra and Maaten, 2020; Chen et al.,…”

“SSL offers various pretraining options, where one approach involves designing hand-crafted pretext tasks [36], [37], [38], such as predicting image rotations [39] or contextually consistent patches, solving a jigsaw puzzle [40], image colorization [41] etc.”

“For example, early self-supervised learning attempts are based on devising classification tasks that try to predict the properties of a transformation (e.g. rotations [47], colorization [48], [49], orderings or relative positions [50], [51]) applied on the input data.”

“…methods of learning representation, and do not discuss Masked-image-models (Bao et al., 2021) or other methods based on proxy task such as rotation (Gidaris et al., 2018), colorization (Zhang et al., 2016), jigsaw (Noroozi & Favaro, 2016), relative patch location (Doersch et al., 2015) etc., as…”

“Due to their wide variety, we do not address all SSL algorithms, in particular those with regression-style auxiliary tasks, e.g. reconstructing data from perturbed versions (He et al., 2022; Xie et al., 2022); or predicting perturbations, e.g. rotation angle (Gidaris et al., 2018).”

“Another example of a PR task is image rotation prediction [39].”

“This is achieved using pre-designed pretext tasks, such as inpainting patches [1], rotation prediction [2], contrastive learning [3, 4], and non-contrastive learning [5, 6], which do not require labeled data to train the model but instead rely on a proxy task.”

“Early methods mainly focus on designing different pretext tasks, including color restoration [55], image rotation prediction [56],”

“For anomaly detection, there is a risk of spurious false positives when correctly classified examples are close to the decision boundary.”

“We deem the rotation prediction task compelling to evaluate since it is a common task in the literature for self-supervised pre-training (Gidaris et al., 2018).”

“This includes predicting rotation types [15], predicting translation and scale types [41], solving jigsaw puzzles [40], etc. Alternative self-supervised methods utilize positive and negative samples to train semantic information from their similarity and dissimilarity [1, 6, 19], which is the main…”

“In self-supervised task augmentation, the image is pre-processed with some image transformation like jigsaw scrambling (Noroozi & Favaro, 2016) or image rotation (Gidaris et al., 2018).”

“These pretext tasks include predicting image rotations [27], relative position of image patches [28], solving jigsaw puzzles [29], etc that have shown to learn good representations.”

“Examples of valid pretext task are image Colorization [18], predicting of relative patch off-sets [20], completing the Jigsaw puzzles [21], cross-Channel prediction [22], spotting artifacts [41], recovering missing image patches [42], learning to count [43], and predicting image rotations [23].”

“Unsupervised methods are relative location [23], rotation prediction [30], PIRL [61], DeepCluster-v2 [11], and 4 instance discrimination methods (NPID [83], SimCLR [14], MoCo v2 [15], and SimSiam [16]).”

“Unsupervised approaches for representation learning [23, 37, 42, 48, 81], especially contrastive learning, such as SimCLR [8], MoCo-v2 [10], ProtoNCE [49], and PaCo [13], similarly aim for learning generic features from a given dataset, but typically necessitate sophisticated training algorithms.”

“Seminal SSL works for images focused on devising sophisticated pretext tasks to extract transformation covariance representations, such as solving jigsaw puzzles [52], classifying rotation angles [39], and predicting relative locations of patches [18].”

“By contrast, discriminative methods learn representations by solving different pretext tasks, such as recognizing the rotation of the image [51], solving jigsaw puzzles [52], and contrastive prediction task [53].”

“The pretext tasks explored in previous works include rotation prediction [11], relative location prediction [12], jigsaw puzzle solving [13], colorization [14], [15], etc. Recently, contrastive learning shows its extraordinary ability to learn discriminative representation without labels.”

“RotNet [19], context encoders [22], SimCLR [21] and BYOL [20] are examples of successful self-supervised representation learning algorithms.”

“In the presence of unlabeled data, however, the surrogate model can be trained by defining a self-supervised task, Ts, such as predicting rotation [15], solving jigsaw puzzle [36] or by matching different views of the same input sample [8].”

“, rotation with large angle as in [15] and jigsaw transformation as in [16].”

“Early works rely on ad-hoc heuristics to design pretext tasks [9,50,34,25], which limits the generality of learned representations.”

“, contrastive learning [10] or rotation prediction [32]) for model pretraining, followed by standard training on long-tailed data.”

“Specifically, TTT trains a classification model using both a supervised learning objective and selfsupervisions (rotation prediction [7]).”

“For example, context prediction [24], jigsaw puzzles [67], and rotation angle prediction [36].”

“The most popular self-supervised learning technology is contrastive learning, which has been successfully employed in many fi elds [15 – 18].”

“These methods have shown great promise across many domains, including computer vision [22], natural language processing [16, 54, 55], and speech processing [11–13].”

“As a concrete example, in the image rotation prediction task [33], the authors define a collection of K discrete geometric transformations, denoted as T = { t ( ·| r ) } = 0 .”

“The particular self-supervised task used in [61] is rotation prediction [21].”