Unsupervised Representation Learning by Predicting Image Rotations

“In recent years, the widespread adoption of deep convolutional neural networks (LeCun et al., 1998) (ConvNets) in computer vision, has lead to a tremendous progress in the field.”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“Specifically, our self-supervised approach manages to drastically improve the state-of-the-art results on unsupervised feature learning for ImageNet classification, PASCAL classification, PASCAL detection, PASCAL segmentation, and CIFAR-10 classification, surpassing prior approaches by a significant margin and thus drastic…”

“…(Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al. (2014); Donahue et al. (2016); Radford et al. (2015).”

“Specifically, we first train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Colorization (Zhang et al.,…”

“…feature learning are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al.…”

“…59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et al., 2016) 41.9 32.2 NAT (Bojanowski & Joulin,…”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“Early works used pretext tasks like patch permutation [27] or rotation prediction [10].”

“…of image patches (Doer-sch, Gupta, and Efros 2015; Noroozi and Favaro 2016), the colorization of gray-scale images (Zhang, Isola, and Efros 2016, 2017), or the classification of sets of image transformations (Gidaris, Singh, and Komodakis 2018; Jenni and Favaro 2018; Jenni, Jin, and Favaro 2020).”

“Specifically, the additional classes could be either borrowed from other real databases or synthesized using some simple approaches, such as image rotation [14] and between-class interpolation (e.g., random interpolations of two samples drawn from different classes).”

“Imagerotation prediction[10],jigsawpuzzlerecovery[11],andsentenceorder prediction [12] are utilized as pretext task to capture generalized representations for the downstream tasks.”

“The main auxiliary tasks in the image domain include Jigsaw Puzzles [44], Image Colorization [45], image rotation [46], image restoration [47], image fusion [48] and so on.”

“For example, the study [10] plays a jigsaw game on images, and the study [11] predicts the rotation degrees of images.”

“Other methods uses various different tasks, for example, colorization [24], [25], geometric distortion [26], [27], noise as targets [28], clustering [29], contrast learning [30], [31].”

“Our work is inspired by the use of clustering to obtain labels in computer vision (Gidaris et al., 2018; Kolesnikov et al., 2019).”

“Self-supervised learning has become the leading paradigm for visual learning, leveraging large unlabeled datasets to learn representations without annotations [24, 42, 47, 69, 77].”

“Rotation prediction [26] is a simple SSL pretext task where the DNN must predict the rotation that has been artificially applied to a given input image.”

“Common transformation-based predictive tasks involve aspects such as assessing relative position [72], solving jigsaw puzzles [73], and determining rotation angles [74], among others.”

“Typically ground truth in the form of annotations or labels is not used for pre-training, pretext tasks like solving jigsaw puzzles [30] or predicting rotation [31] are used to extract meaningful information.”

“For example, RotNet [16] predicts the 2D rotation applied to the input image.”

“Since our method modifies the input space, it complements established test-time model weight adaptation methods (e.g., TENT [62], BN [55], and MEMO [71]) and can be generalized to multiple self-supervised objectives, such as contrastive learning [4], rotation [14], and masked autoencoder (MAE) [15].”

“, 2013) architecture, to be trained with the set of hyperparameters from (Gidaris et al., 2018).”

“For example, [10,28] proposed studies that played a jigsaw game on images and predicted the rotation degrees of images for learning good representations, respectively.”

“To capture the complex patterns of visual structures, we utilize groups of downsampling degradation transformations under different downsampling rate, noise and degradation kernel as the self-supervised signal [64,16].”

“…et al., 2015), the task of inpainting (Pathak et al., 2016), solving Jigsaw Puzzles (Noroozi & Favaro, 2016), image colorization (Zhang et al., 2016; Larsson et al., 2016), rotation prediction (Gidaris et al., 2018), and other tasks (Zhang et al., 2017; Caron et al., 2018; Kolesnikov et al., 2019).”

“We evaluate four SSL methods on all the semantic-2 related and temporal-related tasks for a comprehensive study.”

“The first one is how to select the pretext tasks, and the conventional solution is to predict colors [26, 58], angle [11, 24] of images or the locations of image patches [8, 37].”

“Self-supervised representation learning methods [21, 9] have been adopted to build deep one-class classifiers [22, 25, 2, 57, 56], showing improvement in anomaly detection [4, 67, 36].”

“Techniques widely employed in self-supervised learning for image data encompass geometric transformations [16], [17], [18], patching [19], [20], and color permutations [21], [22].”

“Gidaris et al. (2018) introduced RotaNet [25], which employs a pretext task where the model predicts the rotation angle applied to an image, enhancing feature learning for various orientations.”

“To address this, machine/deep learning models [12], [13], [14], [15], [16] are employed to differentiate unknown or illegal drones from the prior whitelist.”

“To alleviate this issue, recent works [20], [21], [22] have opted for the lane of SSL.”

“Early works design various pretext tasks to generate pseudo labels as self-supervision [49, 30, 47, 10, 1, 62].”

“Also, the relevance of other self-supervised tasks such as rotation prediction [14], context prediction [5], colorization [15], noise prediction [1], feature clustering [3] for adapting IQA models is not clear.”

“Self-supervised learning (SSL) has gone a long and successful way since its beginning using carefully hand-crafted proxy tasks such as colorization [26], jigsaw puzzle solving [32], or image rotations prediction [14].”

“SSL [8, 9, 16, 23, 26, 29, 30, 58, 63] automatically derives a supervisory signal for the training of high-quality feature representations, preventing the need of large labeled datasets.”

“6b are inspired by self-supervised learning [6,8,12,13,39].”

“Initially pretext tasks were derived from ad-hoc heuristics such as predicting image rotation (Komodakis and Gidaris, 2018) and colourisation (Zhang et al., 2016) but these were sur-passed by more general context matching tasks such as solving jigsaw puzzles (Noroozi and Favaro, 2016) and…”

“Most of the early self-supervised learning methods rely on carefully designed heuristic tasks, such as jigsaw puzzles [11], patch localization [12], image inpainting [13] and rotation prediction [14].”

“Representation learning for visual (Gidaris et al., 2018; Chen et al., 2020b; Oord et al., 2018; Kolesnikov et al., 2019; Han et al., 2019) and trajectory data (Sun et al., 2021b; Zhan et al., 2021) has been applied to a variety of tasks, such as for image classification (Chen et al., 2020b),…”

“Instead, the image representations are learnt via pretext tasks [7,22] which commonly apply a transformation to the image and then enforce the neural network to predict the properties of the transformation.”

“Among the former, some early works try to design auxiliary handcrafted prediction tasks to learn semantic representation, such as jigsaw puzzle [29] and rotation prediction [15].”

“Some commonly used tasks are: 90° rotation prediction [27], jigsaw puzzle [28], distortions [29], colorization [30], image inpainting [31] or relative patches prediction [32].”

“In the image domain, a myriad of SSL pretext tasks have been proposed such as predicting perturbations incurred into an image [19, 20, 82], contrastive learning by separating similar from dissimilar views [15, 16, 26, 55], learning by clustering [10, 11], self-distillation [2, 5, 12, 21, 23], or…”

“In the field of image self-supervised representation learning, many pretext tasks have been designed to learn effective features, such as jigsaw puzzles [29], [30], rotation prediction [31], etc.”

“Contrastive Learning: Recently, the most compelling meth-ods for learning representations without labels have been unsupervised contrastive learning [59] , [60], [61], [62], [63], [64], which significantly outperform other pretext task-based alternatives [65], [66], [67].”

“These span from predicting the relative location of extracted image patches [3], solving jigsaw puzzles [13] or predicting a random rotation artificially applied to an image [10].”

“This is usually achieved by defining different pretext tasks, such as context prediction [21], solving jigsaw puzzles [22], image colourisation [23] and rotation prediction [24].”

“While initial efforts in computer vision relied on creating artificial pretext tasks [16, 21, 64], the field is moving towards contrastive-based approaches [6,9,74].”

“, rotation [13], colorization [27], jigsaw [10, 37], to learn certain attributes.”

“[17] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.”

“These auxiliary pretexts task included solving jigsaw puzzles [44], colorization from grayscale [39, 60], inpainting [46], relative patch prediction [17], predicting rotation angle [23], or a combination of tasks [18].”

“The paradigm of self-supervised learning is to design auxiliary task(s) with the supervision of the data itself, for example, predicting spatial context [10] or image rotation [16], solving jigsaw puzzles [40], and many others [5, 11, 19, 27, 42, 73].”

“Self-supervised learning leverages unlabeled data to learn meaningful representations without explicit supervision, demonstrating significant advancements in pattern recognition [2, 10, 12, 28, 29, 39, 43, 47].”

“Examples of such self-supervised tasks include predicting the relative position of image patches [12], colorizing grayscale images [34], recognizing the rotation of an image [20], or inpainting removed patches [51].”

“Examples of pretext tasks include image colorization [49], image inpainting [29], rotation prediction [8], and instance-level discrimination [3, 10].”

“Emerging from the image domain, the pretext tasks rely on the context understanding, e . g ., rotation prediction [28].”

“Instead of the standard ImageNet-based pre-training, the pre-text task strategy presented by Gidaris et al. [13] was adopted to speed up the training of the classification model and enhance its performance.”

“This is typically achieved through pretext tasks [8,36,37,43], enabling the learned representations to be applicable to a broad set of downstream tasks [12,15,56], either by fine-tuning the learned models for specific tasks, or by freezing the weights and training a linear classifier or an MLP on…”

“Past researchers [15], [16] have found that compared to the latest self-supervised learning methods such as SimCLR [34] and SimSiam [35], rotation prediction [37] is better at self-supervised task learning.”

“In computer vision, a variety of pretext tasks have emerged in order to learn similar visual representations for different views of the same data.”

“Recently, contrastive learning has made great strides in the fields of CV [36, 37], NLP [38, 39], and recommendation systems [19, 40].”

“For example, self-supervision strategies such as masked image reconstruction [12, 21, 18], jigsaw puzzles [27], multi-view contrast [2, 45, 5] and angle recognition [11] are widely adopted in the vision domain.”

“SSL implementation involves utilizing attributes such as image rotation prediction [27], patch localization [28], and image reconstruction [29], which can be accessed without manual labeling.”

“2) Contextual methods (Gidaris et al., 2018) design pretext tasks (denoising auto-encoders (Vincent et al., 2008), context auto encoders (Zhang et al., 2016), etc).”

“, rotation prediction [15], reconstruction [20,55], colorization [30,59, 72] and contrastive learning [7, 8, 21, 32, 57, 63].”

“Image rotation [37], Image inpainting [38], and Jigsaw puzzle [39–41] are examples of pretext tasks.”

“nals from the data themselves by adopting various pretext tasks, such as contrastive learning [28, 6], masked autoencoding [11, 27, 70], rotation estimation [18, 43], jigsaw puzzles [39] and so on [1, 21].”

“Early methods included image inpainting [43], and rotation prediction [44].”

“In high-level vision, such pretext tasks include predicting colors from a grayscale image [19], inpainting pixels given the spatial context [20] or through autoregression [21], predicting the orientation of a rotated image [22], solving a jigsaw puzzle given shuffled image patches [23], and others [24, 25].”

“Early works for self-supervised representation learning have focused on pretext tasks exploiting some property of the data [3, 8, 10, 28, 35, 48] without any human supervision.”

“, image rotation [20], inpainting [21], colorizing [22], jigsaw [23], etc.”

“RS [11, 12], NCL [45] and OpenMix [46] first use RotNet [22] to self-supervise on the labeled and unlabeled data.”

“…model zoo contains 15 models trained on ImageNet [39] by various algorithms, e.g., supervised learning (Sup-100%), semi-supervised learning (Semi-rotation-10% and Semi-exemplar-10% [40]), self-supervised learning (Rotation [41] and Jigsaw [42]), generative method (Cond-biggan [43]), and VAEs [44].”

“The former focuses on learning representations by generating pixels in the input space [18, 19, 27], while the latter involves modeling data to discover intrinsic patterns or relationships within it [1, 16, 36].”

“SSL has demonstrated impressive performance in representation learning for various data types including images [15, 16, 17], videos [18, 19, 20], and text [21, 22, 23].”

“For example, in [53], the authors rotated each training image with four different angles and trained the neural network to predict the angle.”

“Initially, these methods addressed some well-defined pretext tasks, such as predicting rotation [21], determining patch position [17], or solving jigsaw puzzles in images [43], and labels for these discriminative pretext tasks can be automatically computed to enable learning of meaningful feature…”

“Favaro, 2016) or patch permutations (Gidaris et al., 2018).”

“In contrast, discriminative approaches generate labels for a proxy task from the unlabeled data, and the encoder is trained on this proxy task (Gidaris et al., 2018; Noroozi and Favaro, 2016; Pathak et al., 2016).”

“Examples of pretext tasks include predicting image rotations [27], the relative position of patches within an image [28], or the next word in a natural language sequence (e.”

“…the expensive labeled data [47], such as completion [45], colorization [48], patch position [49], jigsaw puzzle [50], counting [51], rotation [52], instance discrimination [44], etc. Recently, contrastive learning by constructing pairs of positive and negative samples has made remarkable…”

“In previous work, RotNet [45] used rotational transformations to train better networks based on the idea of self-supervised learning, and this approach was used in few-shot learning [46] to train a better feature extractor.”

“…magnitude can indicate discrimination confidence of the CL model, then the alignment-adaptive temperature dynamically controls penalty strength (arrow length) to negative samples to balance uniformity and tolerance for samples. labeling (Gidaris et al., 2018; He et al., 2022; Grill et al., 2020).”

“, 2016), different objective functions (Giradis et al., 2018; Doersch et al., 2015; Zhang et al., 2016), or different data modalities (Radford et al.”

“Rotation of images, change of colorization, etc, have been popularly used for performing self-supervision in computer vision [33, 141].”

“Early works utilized hand-crafted pretext tasks, e.g., coloration [27], [28], jigsaw puzzles [29], rotation prediction [30], autoencoder [31], [32], image inpainting [33] and counting [34] to learn representations based on heuristic cues [19], but only achieved limited generalization ability.”

“To reduce the need for large labelled training datasets, self-supervised learning models [12]–[20] derive a compact representation by mapping longitudinal MRIs into a latent space using label-independent information [5], [21].”

“Self-supervised representation learning (SSL): Earlier works on SSL ([8, 14, 18, 36, 37, 43]) learn effective representations by solving pretext tasks that do not require supervised labels.”

“While early works proposed tasks such as image colorization [45], orientation prediction [11], jigsaw puzzle tasks [29], and inpainting [32], constrastive learning [1, 5–7, 10, 14, 17, 31, 40, 43] has emerged as the leading technique.”

“Historically, early work on self-supervised learning focused on hand-crafted pretext tasks such as predicting the rotation of an image (Gidaris et al., 2018), the relative position of patches (Doersch et al., 2015) or the color of a grayscale image (Zhang et al., 2016).”

“[83], [84], [85] using pretext tasks [86], [87], [88], [89], [90], contrastive learning [91], [92], [93], [94], [95], [96], [97], or mask modeling [98], [99], [100], [101].”

“In 2018, the community observed a jump in performance when evaluating on Pascal VOC 2007 image classification [8, 22], the implementation of both works leveraging RandomResizedCrop instead of RandomCrop .”

“Early SSL methods achieved this through handcrafted pretext tasks that act as inductive biases in the representation learning process (Komodakis & Gidaris, 2018; Noroozi & Favaro, 2016; Kim et al., 2018; Doersch et al., 2015; Larsson et al., 2016; Zhang et al., 2016).”