Unsupervised Representation Learning by Predicting Image Rotations

“In recent years, the widespread adoption of deep convolutional neural networks (LeCun et al., 1998) (ConvNets) in computer vision, has lead to a tremendous progress in the field.”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“Specifically, our self-supervised approach manages to drastically improve the state-of-the-art results on unsupervised feature learning for ImageNet classification, PASCAL classification, PASCAL detection, PASCAL segmentation, and CIFAR-10 classification, surpassing prior approaches by a significant margin and thus drastic…”

“…(Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al. (2014); Donahue et al. (2016); Radford et al. (2015).”

“Specifically, we first train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Colorization (Zhang et al.,…”

“…feature learning are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al.…”

“…59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et al., 2016) 41.9 32.2 NAT (Bojanowski & Joulin,…”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“One school of works leverages different pretext tasks for pre-training [19,31,57,61,77,86].”

“For example, (Gidaris et al., 2018) pre-trains the model by predicting the rotation angle of the image.”

“Recently, to get rid of the expensive cost of the labeled data, self-supervised learning has risen to be a promising paradigm for learning meaningful representations by designing various pretraining tasks, including context-based tasks [14], contrastive learning [20] and masked image modeling [21].”

“The semi-supervised learning method applied in our research was inspired mainly from the rotation degree Self-Supervised Learning [42] and the SESEMI method [43], and partially from FixMatch method [44].”

“Different methods define different pretext tasks to realize representation learning [30], [31], [32], [33], [34], [35].”

“One is appearance transformation, such as sharpness, brightness, contrast, saturation, gray processing, Gaussian blur, and elast ic i ty (22), another involves spat ia l geometric transformation, such as horizontal flipping, rotation, cropping, and resizing (23).”

“Pretext-based approaches mainly explore the context features of images or videos such as context similarity [35,9], spatial structure [14], clustering property [2], temporal structure [26], etc.”

“The advantage of these techniques is that they leverage unlabeled images from the target data domain during a pretraining phase that learns relevant representations of the data, which boosts the performance of various learning tasks [3], [4], [13]–[16].”

“These could include reconstruction, denoising, color augmentation and rotation prediction [11,22,30,34].”

“Rotation prediction (e.g. RotNet [63]) and contrastive learning (e.g. SimCLR [64] and MoCo [65]) further guide models to understand geometric transformations and learn robust, invariant representations.”

“Meanwhile, the Rotation game draws inspiration from rotation-angle prediction as a supervised pre-text task in self-supervised learning [26].”

“Early SSL methods focused on pretext tasks, such as predicting image rotations [12] or solving jigsaw puzzles [13], to generate pseudo-labels for training.”

“Notably, common pretext tasks such as unrotating an image (Gidaris et al., 2018) or reassembling jigsaw puzzles (Noroozi and Favaro, 2016) from image tiles are not expected to translate well to histopathology due to the inherently complex orderings and rotations in histopathology images, which lack…”

“We add to list this several more self-supervised methods including iBOT (Zhou et al., 2021), DINOv2 (Oquab et al., 2023), MoCo (Chen et al., 2021), CISSL/DISSL (Dubois et al., 2022), and pretext tasks like jigsaw puzzle solving (Noroozi & Favaro, 2016) and rotation prediction (Gidaris et al., 2018).”

“Technically, inspired by [96], [97], we simply learn a unified model by augmenting the current class based on self-supervised transformation (SST).”

“This is achieved by utilizing various pretext tasks, such as prediction of image colorization [28], image rotation [29], and several others [30].”

“Unlike traditional meth-ods [5, 8, 25, 48, 57], we operate unsupervised, learning deformable canonical layers to model vessel motion more effectively.”

“Recently, self-supervision has attracted significant attention across various domains, including vision, where alternative tasks were proposed for learning representations from diverse data types, such as still images, videos, sound, and robotics [47], [48], [49].”

“Previous methods [6, 56, 11, 27] conduct representation learning by using auto-encoders to reconstruct the input itself.”

“Self-supervised learning has a long history in computer vision research community [62, 63, 64, 65, 66, 7].”

“The learned representations by RotNet when used as a feature extractor yield a strong performance similar to the first experiment.”

“SSL requires defining a pretext task such as predicting the future in latent space (Oord et al., 2018), predicting the rotation of an image (Gidaris et al., 2018), or solving a jigsaw puzzle (Noroozi and Favaro, 2016).”

“2) STL10: Table III also reports the compared results on STL-10 between our method and other contrastive representation learning methods, such as CPC [22], Embedding [24], SimCLR [3], Debiased [25], IP-IRM [30], and RotNet [10], which predicts the rotate degree.”

“Self-supervised learning Self-supervised learning methods learn data representations using pretext tasks such as rotation [23], jigsaw puzzle [24], clustering [25], contrastive learning [26, 27], and so on.”

“As the recent self-supervised learning techniques (He et al. 2020; Chen et al. 2020b,a; Gidaris, Singh, and Komodakis 2018) arising, image visual representations can be learned without labels in an unsupervised way.”

“SSGAN-LA [17] leveraged a similar approach but is limited to categorical self-supervision such as rotation recognition [18] to mitigate catastrophic forgetting of”

“To comprehensively evaluate the proposed methods, the raw images and high-level features extracted by rotation prediction [43] are utilized as the algorithms’ input, respectively.”

“Early methods relied on pretext tasks related to high-level image understanding, including reconstruction tasks such as denoising [53], inpainting [36] or colorizing [20, 59, 60], or transformation prediction tasks such as jigsaw puzzles [10, 35] or image rotations [12].”

“Self-supervised auxiliary tasks are demonstrated to be beneficial for aligning domains in high-level tasks (Gidaris et al. 2018; Xu et al. 2019).”

“The basic S4L technique is simple, a self-supervised loss (eg. predicting rotations [18]) is computed on every image (labeled and unlabeled), and an additional cross-entropy loss is computed only on the labeled image.”

“SSL methods in general have shown great promise across a range of domains, including computer vision [31], natural language processing [32]–[34], and speech processing [35]–[37].”

“Initial self-supervised approaches employ image-level pre-text tasks [5, 6, 8, 14, 19, 23, 35, 45], transferring suboptimally to pixel-level prediction [66, 67].”

“One of the most effective approaches is self-supervised learning (SSL [7,9–12,17,21–23,37, 38,56]), which has been developed in order to alleviate the manual annotations on large-amount of images.”

“Inrecentyears,self-supervisedlearning(SSL)techniques have emerged as a transformative approach across various domains, including computer vision [7], [8], [9], [10], [11], [12], natural language processing [13], [14], [15], [16], and time series [17], [18], [19], [20], [21].”

“Some early works are devoted to designing special pretext tasks, e.g., Colorization [28], In-painting [37], Jigsaw [36], Rotate prediction [17], etc. Con-trastive learning based on instance discrimination [45] has become the mainstream in the community in recent years, whose core idea is to…”

“Utilizing pretext tasks such as rotation [60], colorization [61], or jigsaw solving [62] is a popular self-supervised learning approach for computer vision tasks.”

“Recently, self-supervised learning has also been demonstrated effective for one-class classification and anomaly detection by solving proxy tasks, such as jigsaw puzzle [54], rotation prediction [55], clustering [56] and contrastive learning [57].”

“They includes one-class methods [42, 44, 48] and self-supervised learning [6, 17, 18, 22].”

“, rotation prediction [33] and automatic colorization [34].”

“Some early pretext tasks like Exemplar-CNN [38] or predicting image rotations [39] are based on the idea that applying some transformations to an image does not change its semantic meaning.”

“Inspired by the rotation ConvNets (20,21), in this paper, we simply rotate the CT scan at certain angles and design a rotation classification network as the pretext task to distinguish the rotation angle of each CT scan.”

“A wide range of predictive and generative tasks have been proposed to learn representation like rotation prediction [13], jigsaw puzzle [11], context encoders [16], split-brain autoencoders [17] and colorization [14].”

“However, these strategies are multiple-shot strategies, which means unbearably high computational burden in semi-supervised learning scenarios.”

“In recent years, designing different kinds of pretext tasks (e.g. jigsaw puzzle[46], rotation angle prediction[14], temporal order prediction[43]) as a self-supervised pre-training method became a popular trend in this community.”

“’s [5] work and presents an algorithm to combine hierarchical clustering with unsupervised feature learning using the rotation prediction pretext task [18].”

“A variety of different self-defined pretext tasks (Komodakis and Gidaris 2018; Feng, Xu, and Tao 2019; Zhang, Isola, and Efros 2016; Noroozi and Favaro 2016; Zhang et al. 2019) have been proposed.”

“…approaches typically involved a pre-defined image manipulation with a Deep Learning model tasked to detect a given random transformation, for example, image reconstruction (Doersch et al., 2015; Noroozi and Favaro, 2016), colourisation (Zhang et al., 2016) and rotations (Gidaris et al., 2018).”

“Self-supervised learning leverages the intrinsic data structure to design pretext tasks, which effectively utilizes large-scale unlabeled data to capture generic representations [15, 24, 38, 44, 73, 84] These representations are important for a wide range of downstream tasks [21, 30, 39].”

“For instance, [27] rotates an image by 0 , 90 , 180 , or 270 and tasks the model with predicting the rotation.”

“Early efforts in unsupervised learning emphasized pretext tasks designed to predict pseudo labels, such as solving jigsaw puzzles (Noroozi & Favaro, 2016), restoring missing patches (Pathak et al., 2016), and predicting image rotations (Gidaris et al., 2018).”

“Moreover, common augmentation methods designed for computer vision tasks, like colour distortion or rotation, typically do not align well with the nature of CSI data [27].”

“This method highlights the distinctions between samples, equipping the model with robust discriminative features [23, 24], Li et al [25] studied the impact of unsupervised representation learning on unlabeled data for speech emotion recognition.”

“For example, Gidaris et al. [41] propose to predict the rotated picture’s rotation angle; to accomplish this, the model should learn some intrinsic knowledge about the picture.”

“Self-Supervised Learning Early work in self-supervised learning used simple pretext tasks, such as relative patch prediction [19, 20], image colorization [67], image rotation prediction [24], or solving jigsaw puzzles [44] to train feature extractors in absence of annotations.”

“The self-supervised contrastive loss projects the features ( z i , z j ) into the clustering embedding space.”

“Test-time training [38] can provide generalization under distribution shifts using a self-supervised auxiliary task to update model parameters at test time, where self-supervised learning studies how to create labels from the data, such as context prediction [8], and rotation prediction [1, 14].”

“Various meth-ods have used this approach to learn representations from unsupervised data [24, 51, 63, 71], achieving superior performance compared to other approaches [14, 20].”

“example, the typical pretext task, image rotation [23], may not benefit activity segmentation, because the sequence with the start-state label will be quite similar to the one with the end-state label when the former rotates 180 (referring to Fig.”

“Spatial tasks include solving jigsaws [17], predicting patch rotations [6], or classifying relative patch positions from a grid-like pattern [5].”

“Since the labeled data is limited and expensive to obtain, one option would be to use large-scale unlabeled data and apply self-supervised learning (Gidaris et al., 2018; Noroozi and Favaro, 2016), an approach that trains the model on unlabeled data in a supervised manner by utilizing self-generated labels from the da…”

“This self-supervised representation learning involves contrastive visual representation learning [19] or pre-text tasks, such as relative patch prediction [20], jigsaw puzzle completion [21], image colorization [22], and image rotation prediction [23].”

“…the noisy or perturbed image via autoencoding [8], denoising autoencoder [9], predicting the order of the patches [10], recovering color of grayscale image [11], predicting missing region via image inpainting [12], predicting rotation angle of image [13], adversarial loss for masked patches [14].”

“re-ordering perturbed image patches [29], counting virtual primitives [30], and classifying image rotations [19].”

“However, these EEG signals can be used for representation learning under self-supervision [17] as unlabeled data, which can be helpful to train the deep model under the condition of less training data related to a specific task.”

“…in feature space become meaningless for high-dimensional data, deep learning researchers have turned to similarity in a representation space learned via self-supervised contrastive tasks [53, 21, 29, 12], or similarity in a feature space learned end-to-end for a clustering task [10, 11, 57, 72].”

“RotNet [22] is a widely-used method, which predicts the rotation of images as a pretext task.”

“For example, Doersch et al. (2015), Gidaris et al. (2018), and Noroozi and Favaro (2016) propose hand-crafted tasks such as solving a jigsaw.”

“In the realm of images, positive samples are often created through various augmentation such as rotation, cropping, and random adjustments in grayscale and color [38], [39], [40], [41], [42].”

“For discriminative methods, early approaches utilize heuristics-based pretext tasks (e.g. solving jigsaw puzzles [26] and rotation prediction [16]).”

“With the rapid advancement of self-supervised learning in the fields of Natural Language Processing (NLP) [30, 38] and CV [4, 15, 21], due to the similarities between CTR prediction models based on user behavior sequences and NLP, they initially incorporate CL [23, 57, 73].”

“Rotation [12], [29], Location [15], and SSL ++ [23] try to capture the low-level positional and texture features to obtain general models.”

“To do so, they propose Character Contrastive Learning ( ChaCo ), which uses the Character Unit Crop module to generate two random crops of an image ensuring that they contain overlapping parts.”

“The process automatically creates pseudo labels based on the pretext task assigned to the network to solve which removes the need for external supervision [33].”

“Examples of such tasks include predicting the context of a patch (Doersch et al., 2015) or its rotation (Gidaris et al., 2018), image inpainting (Pathak et al., 2016), and “solving” jigsaw puzzles (Noroozi & Favaro, 2016).”

“This leads us to ask the question: can we train visual encoders that extract only navigation-relevant features using self-supervision?”

“These include techniques such as rotation prediction [Gidaris et al., 2018], colourisation [Zhang et al., 2016], inpainting [Pathak et al., 2016], and image jigsaw puzzles [Noroozi and Favaro, 2016].”

“Thus, we establish a self-supervised framework to integrate the expert knowledge, utilizing this feature vector as the proxy label [28], which provides a comprehensive representation of various fault types.”

“SSL learns representations from unlabeled data through pre-text tasks [26], [27] or domain-specific data augmentation.”