Unsupervised Representation Learning by Predicting Image Rotations

“In recent years, the widespread adoption of deep convolutional neural networks (LeCun et al., 1998) (ConvNets) in computer vision, has lead to a tremendous progress in the field.”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“Specifically, our self-supervised approach manages to drastically improve the state-of-the-art results on unsupervised feature learning for ImageNet classification, PASCAL classification, PASCAL detection, PASCAL segmentation, and CIFAR-10 classification, surpassing prior approaches by a significant margin and thus drastic…”

“…(Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al. (2014); Donahue et al. (2016); Radford et al. (2015).”

“Specifically, we first train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Colorization (Zhang et al.,…”

“…feature learning are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al.…”

“…59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et al., 2016) 41.9 32.2 NAT (Bojanowski & Joulin,…”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“Our idea is inspired by ConvNets, which learn image classi-fication features by predicting the 2D image rotation that is applied to the input image [37].”

“, rotation prediction [15], Jigsaw puzzle solving [36] and objects tracking [52].”

“Early works such as (Doersch, Gupta, and Efros 2015; Doso-vitskiy et al. 2014; Gidaris, Singh, and Komodakis 2018; Noroozi and Favaro 2016) on self-supervised learning were based on the idea that a network could learn important image features and semantic representation of the scenes when trained…”

“In computer vision, the pretext tasks can be predicting angles of rotated images [55], learning the relative position of segmented regions in an image [56], placing shuffled patches back [57], or colorizing grayscale input images [58].”

“As fvcore does not support all operations of ViL, the FLOPS are not 100% accurate, but nevertheless serve as a good reverence point to compare the relative computational effort between block designs.”

“, jigsaw puzzle solving [48], rotation prediction [18], instance discrimination [21,66].”

“…We compare our method with state-of-the-art methods, including PT-based methods: Colorization’16 ECCV [47], Jigpuz’16 ECCV [27], Rotation’18 ICLR [26]; CL-based methods: MoCo’20 CVPR [11], SimCLR’20 ICML [5], MoCo-v2’20 Arxiv [1], PCL’21 ICLR [48], BYOL’20 NIPS [9], SwAV’20 NIPS [8], SCRL’21…”

“Unlike [44], we only consider ResNet-18 with MoCo-v2 [12] and BYOL [23] since we find that older methods like Jigsaw [39] and RotNet [20] have significantly worse clean accuracies compared to MoCo-v2 and BYOL.”

“To achieve this, early approaches utilize hand-crafted pretext tasks, including colorization [64], rotation prediction [14], position prediction [45], and Selfie [55].”

“In recent years, many self-supervised learning methods have been proposed [8, 10, 30, 34, 58, 60, 67, 72, 85].”

“In this paper, we are mainly concerned with the works of using pre-text tasks to improve the performance of few-shot classification [11,12,46,18,5].”

“…layers of networks has been considered before, e.g., for training object detectors (Lin et al., 2017) and image classification models (Lee et al., 2015), or evaluating the transferability of individual layers (Zhang et al., 2016; Gidaris et al., 2018) or groups of layers (Evci et al., 2022).”

“…and DenseFormer without and with MAE pre-training as well as the DenseFormer pre-training with different self-supervised learning methods: Rotation [37], MoCo [24] and DINO [26], which are denoted as “Densenet”, “ViT”, “Densenet + MAE”, “ViT + MAE”, “DenF + MAE”, “DenF + Rota”, “DenF + MoCo” and…”

“the model to predict the rotation angle [21].”

“Without the need for GT labels annotated by the human, early SSL approaches leverage handcrafted “pretext” tasks, like solving Jigsaw puzzle [20] or predicting image rotation [40], while recent methods seek to learn augmentation-invariant representation [2,26,10,8,24].”

“In fields like computer vision, early self-supervised learning methods aimed to capture invariant features by predicting image transformations [9], [10].”

“Our work adopts rotation-based self-supervision task [12].”

“Generativelearningaimstoexploitcompactsemanticrep-resentations [35], such as inpainting [36], jigsaw puzzle [37], colorization [38], [39], image rotation [40], [41] and relative patch location prediction [42] in computer vision tasks.”

“Previous successful visual representation learning frameworks based on denoising autoencoders use global transformations such as rotation [20] and Jigsaw [24] and are outperformed by siamese networks.”

“Some pretext tasks employed were image colorization (Zhang, Isola, and Efros 2016), jigsaw puzzle solving (Noroozi and Favaro 2016), patch ordering (Doersch, Gupta, and Efros 2015), rotation prediction (Gidaris, Singh, and Komodakis 2018) among others.”

“…[2] 19.80 19.64 19.68 LAV [4] 23.89 23.65 23.56 VAVA (ours) 29.58 28.74 28.48 IKEA ASM Background SAL [5] 14.28 14.04 14.10 TCN [6] 17.37 17.03 16.96 TCC [2] 18.03 17.53 17.20 LAV [4] 20 [6] 27.05 26.92 26.59 TCC [2] 28.58 28.05 28.34 LAV [4] 29.39 28.48 28.20 VAVA (ours) 36.53 34.71 34.63 Table 1.”

“While this work primarily focuses on jigsaw puzzles as the pretext task, numerous alternatives exist and worth exploration (Gidaris et al., 2018; Gui et al., 2024).”

“Gidaris et al. [25] supposed that the Convolutional Neural Networks (CNN) can be more efficient if they can perceive the rotation angle of the main objects in the images.”

“For example, some tasks include predicting the rotation of an image (Gidaris et al., 2018), or predicting masked words in a sentence (Devlin et al.”

“More recently, “connectivity-driven ” fiber clustering is based on the connectivity of the voxels through which fibers pass (Tunç et al., 2016, 2014, 2013), and AnatomiCuts clusters fibers based on their position relative to anatomical regions (Siless et al., 2020, 2018).”

“To avoid such reliance, various learning frameworks have been proposed and investigated: Self-supervised learning aims to learn meaningful representations with heuristic proxy visual tasks, such as rotation prediction [16] and the more prevalent instance discrimination task, the latter, being widely applied in self-su…”

“For example, rotations [51], colorization [52], and jigsaw puzzle solving [53] are efficient pretext tasks that can explore the inner patterns in images.”

“Several different Self-SL pre-training strategies were developed based on different tasks s.a. generative models or clustering [12, 13, 21, 25], with contrastive training being currently the de facto standard in image classification.”

“PTC further improves its generalizability by fusing the RGB and optical flow features through a self-attention weight mechanism and selection of training videos at the boundary of action classes through a sphere boundary sample-selecting scheme.”

“For example, early attempts use different surrogate tasks whose labels can be naturally generated, like pixel value inpainting [28][17], cross-channel feature regression [29], or rotation prediction [30].”

“[3] introduced existing self-supervised techniques [18, 31] to use unlabeled data but resulted in approximately the same performance.”

“Some disrupt the arrangement order after cutting pictures (Santa Cruz et al. 2017), while the others rotate images (Gidaris, Singh, and Komodakis 2018; Li et al. 2020).”

“Entropy minimization is ideal for domain adaptation: it can be used on a trained model, without retraining, and doesn’t require balancing a proxy loss with a classification loss, as in (Gidaris et al., 2018; Sun et al., 2020; Gandelsman et al., 2022).”

“Given a test sample, TTT first fine-tunes the model via rotation classification [17] and then makes a prediction using the updated model.”

“Alternatively, it can predict the rotation degrees of images, which de fi ne four degrees for prediction, i.e. , 0, 90, 180, and 270 [104].”

“Pretext Tasks and Losses for Self-Supervision: Several pretext tasks have been used in the past to provide training signals for image-based self-supervision, including predicting the position of an image patch relative to the first [17], solving jigsaw puzzles with shuffled image patches [40], performing colorization…”

“This led to the development of prompt-based learning works [23, 25] aimed at helping these pre-trained models generalize to downstream tasks instead of fine-tuning their internal parameters [21] for better transfer learning.”

“Given their cost-effectiveness and efficiency, computer vision methods have been developed to leverage weak annotations while addressing their inherent inaccuracies [351].”

“The model is pre-trained using a combination of cross-entropy and rotation prediction [2], and then fine-tuned with Manifold Mixup [7].”

“Overall, both learning biases of general dataset statistics and instance discrimination have been shown to improve transfer performance on classification downstream tasks [27, 29, 71] which is consistent with the boost in classification observed for η = 1.”

“While earlier SSL methods rely on transformation-based [20], [21], reconstruction [22], [23] and generative [24] approaches, the most recent state-of-the-art SSL frameworks are based on the contrastive learning paradigm.”

“Drawing motivation from self-supervised pretext task literature [47,16], recent DA works [6,43] have adopted subsidiary tasks as side-objectives to improve the adaptation performance.”

“A prevalent strategy is to design pretext tasks and acquire pseudo labels as self-supervision [7,24] or employ contrastive learning to discriminate instances [12,29,8].”

“SSL has been widely exploited in different domains, including computer vision [21–24], audio/speech processing [25], and time-series analysis [26,27].”

“The basic idea of SSL is to train neural networks on a pre-text task such as context encoder [40], jigsaw puzzle [41], [42], image rotation [43], and colorization [44].”

“One major bottleneck in deep learning is the scarcity of labeled data, and much attention has been paid to unsupervised learning [15, 17, 30, 33] and self-supervised learning [4, 12, 14, 16, 25, 40].”

“More types of pretext tasks have also been investigated, such as image rotation [8], image inpainting [23], image colorization and motion segmentation prediction Pathak et al.”

“Self-supervised learning (SSL) is a framework for learning representations of data [6, 14, 20, 7, 5, 61, 50, 37, 28, 39].”

“For the RotNet training with augmentation, The augmentations used in both Stage 1 and Stage”

“Learning general features has been extensively studied in the context of unsupervised and contrastive learning [3, 9, 12, 15, 32, 51] when labels are not available.”

“…the pre-training process, researchers design variant pretext tasks, i.e., jigsaw puzzles [41], [42], colorization [68], predicting rotation [13], masked patch reconstruction [15], [24], [57], contrast similarity between samples [7], [16], [43], [45], etc. Recently, masked patches…”

“Restrictions apply. and self-supervised learning [58], [59], [60].”

“Early self-supervised methods defined pretext tasks such as predicting by which amount an image was rotated [12] or solving a jigsaw puzzle [28].”

“[25] employ rotation prediction [8] as an auxiliary task for TTT.”

“Similar to other MIM-based SSL work [19, 45], we reconstruct knowledge for those masked patches of input sample 𝑥 .”

“Though model-free augmentations are efficient, the difficulty of these augmentations seems to be inadequate for deep model [21, 66].”

“For example, Gidaris et al. [10] learn image representation by training an image rotation aware network.”

“[25], rotation prediction [41], jigsaw puzzle [42], denoising”

“Typically, discriminative approaches are implemented with contrastive learning frameworks [3, 7, 16, 17] or through novel pre-text tasks [12, 29].”

“It can include predicting image rotation (Gidaris et al., 2018), solving jigsaw puzzles (Noroozi and Favaro, 2016), re-coloring gray-scale images (Larsson et al., 2017), to mention a few.”

“Following the previous work [5], we also adopt a self-supervised rotation loss [24] to facilitate learning.”

“…benchmark by: 1) fine-tuning with matching architecture and checkpoint (ViT L/16 i21k) reported by Steiner et al. (2021), 2) the Sup-rotation method (Gidaris et al., 2018) that achieved the best result in the VTAB-full leaderboard (Zhai et al., 2019b), 3-4) µ 2Net results after 2 task iterations,…”

“[13] which learns image features by learning to predict image rotations for the particular dataset, and we refer to this model as RotNet.”

“Given a randomly chosen image sample, augmentations of the image are generated using common image transformations such as cropping and resizing a smaller region of the image, color transformations (hue, saturation, contrast), rotations etc. (Chen et al., 2020a; Gidaris et al., 2018).”

“Recently, semi-supervised and especially self-supervised learning methods [101–103] have become popular in many machine learning application domains including human activity recognition using wearables [13,43,44].”

“Early studies such as test-time training (TTT) and its variants [22, 23], introduce auxiliary tasks (e.g., rotation prediction task [25]) into the supervised training objective to improve the model generalization at the training stage, and then adapt the pre-trained model to test samples via…”

“In the realm of computer vision, a prevalent transfer learning approach involves pre-training on an extensive and diverse dataset such as ImageNet [184], MS-COCO [23], or Open Images Dataset [185], followed by fine-tuning on a narrower target dataset.”

“Data augmentation methods like image colorization [21] and rotation [22] in CV, or masking [7] and synonym replacement [23] in NLP, may introduce biases unsuitable for the target domain.”

“In recent years, self-supervised learning in computer vision has evolved from simple pretext tasks like Jigsaw Puzzles [22], Rotation Prediction [17], and Patch Position Prediction [10] to the current SotA methods such as restorative autoen-coders [13] and contrastive [8] or non-contrastive [9]…”

“Prior works [21,45,56] verified that self-supervised auxiliary tasks are helpful for domain adaptation, especially in stereo matching task [43].”

“, image colorization [41], image inpainting [28], and rotation [24] prediction.”

“Rotation [9] transformed the images by the four image rotations and trained the network on the classification of recognizing the correct rotation.”

“For instance, Gidaris et al. (2018) predict labels of random image rotations, Doersch et al. (2015) predict the relative pixel positions, Dosovitskiy et al. (2014) learn to discriminate a set of surrogate classes, and approaches (Gidaris et al., 2019; Su et al., 2019) improve the few-shot…”

“Self-supervised learning methods operate on an unlabeled dataset by explicitly defining pretext tasks such as solving jigsaw puzzle [48], patch location prediction [16], inpainting [51], and image rotation [25] prediction.”

“Self-supervised learning designs pretext tasks [19, 58, 61, 91] to produce supervision signals derived from the data itself, while contrastive learning is its subset that aims to group similar samples closer and diverse samples away from each other [7,9–11, 20, 24,27,28,45, 74 Extending contrastive…”

“, rotation prediction [17], solving jigsaw puzzle [38], image colorization [62], and image inpainting [40].”

“6 in Appendix C for a detailed summary of all RotNet transfer learning evaluations.”

“Pretraining can be conducted even without any human annotations on artificiallydesigned self-supervised pretext tasks [9,22,23,24,26,28] with impressive results on a variety of downstream tasks.”

“The evaluation metrics includes: Information theory-based metrics: FMI Image structural similarity-based metrics: SSIM [68], Q Y [46]; Human perception inspired metrics: VIF [69], Q CV [73].”

“Early approaches used hand-crafted objectives such as context prediction [22], patch re-ordering [57], colorization [88, 89], inpainting [63], geometric transformation prediction [28], and instance discrimination [23, 85].”

“By developing pretext tasks with the objective of learning generalisable brain representations, we leverage this architecture for self-supervised learning from unlabelled data, hoping to replicate similar successes in computer vision [21, 8].”

“ReMixMatch also conducts Mixup regularization [2, 48] and self-supervised learning with rotation [14].”