Unsupervised Representation Learning by Predicting Image Rotations

“In recent years, the widespread adoption of deep convolutional neural networks (LeCun et al., 1998) (ConvNets) in computer vision, has lead to a tremendous progress in the field.”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“Specifically, our self-supervised approach manages to drastically improve the state-of-the-art results on unsupervised feature learning for ImageNet classification, PASCAL classification, PASCAL detection, PASCAL segmentation, and CIFAR-10 classification, surpassing prior approaches by a significant margin and thus drastic…”

“…(Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al. (2014); Donahue et al. (2016); Radford et al. (2015).”

“Specifically, we first train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Colorization (Zhang et al.,…”

“…feature learning are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al.…”

“…59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et al., 2016) 41.9 32.2 NAT (Bojanowski & Joulin,…”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“Specifically, prediction-based approaches involve defining pretext tasks that require the model to predict specific properties of the input data, such as image rotation [34] or patch relative position prediction [35].”

“Gidaris et al.[65] offered image rotation classification.”

“With the rapid advancement of self-supervised learning in the fields of Natural Language Processing (NLP) [25, 30] and Computer Vision (CV) [3, 12, 18], due to the similarities between click-through rate prediction models based on user behavior sequences and NLP, they initially incorporate…”

“Predictive self-supervised learning aims to learn data representations from unlabeled data by treating the pretext task as a classification or regression task such as Jigsaw puzzle [37], Rotation prediction [38] and Relative position prediction [39].”

“For natural images, random rotation, flipping, and noise injection are the most common augmentation strategies [14, 23, 42].”

“A promising unsupervised learning subclass is self-supervised learning (SSL) that can be used to initialize DL models used for specific downstream tasks with only few labeled data [32], [33].”

“Another effective approach in relation prediction involves rotating different parts of an input and inferring the degree of rotation for each part [51].”

“In CIR+, we empirically find that the contrastive learning performs worse than the rotation recognition (Komodakis and Gidaris 2018), so in this paper, we introduce the rotation recognition as an auxiliary task for feature learning enhancement.”

“Earlier self-supervised learning approaches such as rotation, exemplar pretext tasks, or contrastive learning (Gidaris, Singh, and Komodakis 2018; He et al. 2020) focused on extracting semantic properties from images in order to enhance the performance of downstream tasks such as classi fi cation,…”

“State-of-the-art SSL techniques include contrastive methods utilizing positive-negative pairs (SIMCLR [8] and MoCo [9]); contrastive learning based on asymmetry (BYOL [7] and SIMSIAM [10]); learning visual pretext tasks ([13] and [14]); and learning via redundancyreduction (Barlow Twins [11] and W-MSE [12]).”

“To name a few, Sun et al. [42] propose the pioneering work, which predicts the rotation angle [53] as the auxiliary task.”

“Competing SSL approaches We compared our method with other self-supervised methds, including predictive-based methods (e.g., Rotation Prediction idaris et al., 2018 , Puzzle Solving Noroozi and Favaro, 2016 ), econstructive-based methods (e.g., MLM Lan et al., 2019 ), and conrastive based methods (e.g., Moco v1 He et…”

“We can simply classify the exiting pretext tasks into two categories according to the task type such as recovering the input image under preset corruptions [54] and forming the pseudo label for input image [2, 3, 7, 12, 18, 27].”

“Examples include colorization [26], contextual autoencoders [27], inpainting [21], spatial puzzles [28], and discriminative orientation [29].”

“It has been applied to great effect within natural language processing (Deng et al., 2013; Van Den Oord et al., 2017) and computer vision (Ciregan et al., 2012; Gidaris et al., 2018) and encompasses both supervised and unsupervised models ( f : X → Z ).”

“Rotation [51] is a self-supervised learning approach for pre-training by predicting the rotation angles of images.”

“Various pretext tasks have been explored, such as, patch position [12], image colorization [45], image inpainting [31], rotation [15], and predictive coding [24].”

“For example, learning color from grayscale images (colorization) [9,10], shuffling and predicting image tile permutations (Jigsaw) [11], removing and re-generating image regions (inpainting) [12,13] rotation prediction (RotNet) [14] and classifying augmented positive and negative pairs (contrastive learning) [15,16].”

“These pretexts can be mainly separated by two categories: reconstruction-based which concludes colorization [48], spatial jigsaw puzzles [39], inpainting [45] and discriminant-based which contains rotation predict [26], instance-level contrastive [10,19,20], and fine-grained contrastive [38, 42].”

“In [11], semantic features have been learned with self-supervision in order to recognise the rotation that has been applied to an image given four possible transformations as multiples of 90 degrees.”

“Examples of such pretext tasks are context prediction [2], image colourisation [3], jigsaw puzzle [4], and rotation prediction [5].”

“It would allow us to check whether and when fine-tuning, using [35], provides better performance for a small training set.”

“, colorization [78, 79], rotations of inputs [22], modeling image similarity and dissimilarity among multiple views [4, 5, 8–10, 31, 76], to name a few), and transfer them to target domains, such as object detection [23, 24, 32, 55], action recognition [6, 56], or various fine-grained recognition tasks [11, 18, 58, 64…”

“Self-supervised learning learns representations by designing appropriate surrogate tasks, like rotation prediction (Komodakis and Gidaris, 2018; Wang et al., 2021), masked image reconstruction (He et al., 2022; Zhang et al., 2022; Du et al., 2024), and instance discrimination (He et al., 2020;…”

“Early SSL methods relied on pretext tasks, such as rotation prediction [36], context prediction [33], solving jigsaw puzzle [59], and colorization [101], to extract meaningful representations.”

“Since no constraints are placed on label use during training, supervised contrastive learning typically captures richer knowledge compared to contrastive learning [11], [12], [14].”

“Besides contrastive learning, one can also train a self-supervised model using a pre-training task that does not require detailed patch-level or region-level label information, such as predicting rotation [36], stain color [37], weak slide-level labels [38], or tissue phenotypes [39].”

“Pre-training can be conducted even without any human annotations on artificially-designed self-supervised pretext tasks (Caron et al., 2018; Gidaris et al., 2021, 2018; Grill et al., 2020; He et al., 2020; Hénaff et al., 2021) with impressive results on a variety of down-stream tasks.”

“For example, RotNet [25] predicts the 2D rotation applied to the input image.”

“Early SSL pretraining works [46], [47] mainly concentrate on joint-embedding designs.”

“Common pretext tasks include solving jigsaw puzzles [38], recognizing image orientation [39], learning to count [40], and image colorization [41].”

“Gidaris et al. [17] first rotate the images to fixed angles and train a model to predict the rotation angles, then construct classifiers based on the angle predictor.”

“However, large-scale pre-trained models [4,17,21,59] and efficient parameter tuning methods [6,18,27,28,32,61,62,85,85] have been recently proposed to address this issue for effectively adapting to new tasks.”

“With the great success of SSL in computer vision [28] and natural languageprocessing[11], researchers areincreasinglyinterested in applying SSL to graph data analysis.”

“…& Ranzato, 2017; Chaudhry et al., 2019; Buzzega et al., 2020; Prabhu et al., 2020; Cha et al., 2021; Boschini et al., 2022), leveraging knowledge distillation (Hinton et al., 2015) and self-supervised learning (Doersch et al., 2015; Gidaris et al., 2018) to retain context from previous tasks.”

“For example, the model in [39] predicts the image rotation to learn the image representation for the image classification task.”

“Recently, self-supervised learning has demonstrated significant success across diverse domains, including natural language processing [7], [8] and computer vision [9], [10].”

“Such learning strategy promotes models to capture the intrinsic structure and relationships in data by designing innovative pretext tasks, such as image reconstruction (e.g., Context encoder [17]), contrastive learning (e.g., SimCLR [18]), or prediction tasks (e.g., rotation prediction [19]).”

“For instance, Gidaris et al. [40] proposed to learn image features by training deep networks to recognize the 2D rotation angles of images.”

“Following the common practice of classic SSL [23], [26], our pipeline consists of two steps.”

“Gidaris [10] and Lee [11] introduced rotation and RGB channel transformations, respectively, to enable the model to comprehend object properties, such as position and color.”

“Another line of work relies on incorporating beneficial inductive biases such as image rotation [49], solving Jigsaw puzzles [46] or by introducing sequential information that comes from video [45].”

“RotNet [16] rotates the original image with a special angle, and learns a latent embedding through predicting the rotation angle of the input image.”

“Early SSL methods focus on proposing novel pretext tasks, such as relative location prediction [10] and rotation prediction [21].”

“For instance, [54] and [55] learned disentangled components according to different categories, [56] emphasized different viewpoints, and [57] captured different components in videos via disentanglement.”

“1) Distortion, for example, predicting rotation [19].”

“Training a model to predict the rotation angle of a given image was proposed in [13] for self-supervised learning.”

“47,48 In the proposed GSAL , an auxiliary rotation loss is introduced into discriminator network D to eliminate discriminator forgetting during model training.”

“This often involves training with auxiliary loss functions along with the standard cross entropy loss, such as knowledge-distillation [38], mixup-based data augmentation [25] and self-supervision , such as predicting rotations [10] and contrastive learning [42].”

“In the CV field, data augmentations include colorization [35], rotation [36], and cropping [37]; and in the NLP field, data augmentations include term masking [38] and”

“Self-supervised learning [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [42], [43], [44], [45] builds general representations by exploiting the internal priors or structures…”

“One line of works seek for designing self-supervised pre-text tasks, including predicting the expected pixel values of an output image [19, 38, 39], solving a surrogate proxy task such as predicting rotations [12], scale and tiling [23], and patch orderings [11, 22, 31].”

“In this work, we empirically show that combining instance-discriminative tasks with well-posed handcrafted pretext tasks such as Rotation prediction [13] can indeed lead to more effective and efficient learning of visual representations.”

“…devise numerous pretext tasks — colorization, predicting image rotations or solving image puzzles — to learn useful representations from unlabeled images (Gidaris et al., 2018; Pathak et al., 2016; Noroozi and Favaro, 2016; Doersch et al., 2015; Larsson et al., 2016; Zhang et al., 2017; 2016).”

“The signal may come from predicting missing, corrupted, or future information or by establishing correspondence between inputs [24, 25, 26] or even from comparison between multiple views of an input image [10, 11, 15, 16].”

“Another emerging direction is to leverage self-supervised learning algorithms such as RotNet [14], Jig-Saw [29], SimCLR [10], or MOCO [16] for unsupervised pretraining and then fine-tune with the limited labeled set [11, 19].”

“In this context, representation learning has been applied to various areas such as computer vision [10,15,17], natural language processing [7,12], and sound signal processing [4,39].”

“Additionally, rotation prediction [14] was used as an unsuar X iv :2 20 2.”

“A prevalent approach for representation learning is to formulate pretext tasks with pseudo labels generated from the unlabeled data (Doersch et al., 2015; Noroozi & Favaro, 2016; Gidaris et al., 2018).”

“Pre-training vision models on datasets of images without supervision has been a fruitful area of research in recent years [10, 27, 28, 34, 54, 87, 88].”

“Early methods often train from scratch using self-supervised strategies such as RotNet [201] and MoCo [202].”

“This objective can be direct prediction of parameters corresponding to augmented views (Lee et al., 2021; Scherr et al., 2022; Gidaris et al., 2018; Gupta et al., 2024), or additional transformations such as rotation (Dangovski et al., 2022).”

“Self-supervised learning was first used in the image domain to teach models to learn useful data representations by augmenting the data through techniques such as rotation, cropping, and collage [31, 32].”

“To obviate the need for clean ground-truth image, self-supervised image denoising methods have emerged as a powerful alternative [45–47 ].”

“2 (e), was applied to provide semantic information for the model to learn [29,82].”

“Another category of the methods is based on the semantic context of the images [16], [17].”

“Gidaris et al. (2018) employed the spatial structural properties of an image to artificially construct a self-supervised task of predicting the angle of rotation of an image to train a network.”

“Self-supervised learning [Gidaris et al. , 2018] has achieved remarkable success in the field of images, excelling in var-Proceedings .”

“Early methods trained these representations by solving pretext tasks e.g. , relative position prediction (Doersch et al., 2015), image recol-orization (Zhang et al., 2016), jigsaw puzzles (Noroozi and Favaro, 2016) or rotation prediction (Gidaris et al., 2018).”

“Innate relationship Image rotation prediction [16]; Image context prediction [17] Generative Autoencoder [18]; GAN [19] Contrastive SimCLR [20]; BYOL [10]; DINO [11] Self-prediction MAE [12] However, a notable limitation of self-supervised learning is its dependency on large datasets.”

“(P3) Computationally cheap: minimal computational overhead to be practically usable.”

“We further explore task variations by setting A as the first 50 classes of CIFAR-100, B as the last 50 classes of CIFAR-100, and C as a rotation classification (Gidaris et al., 2018) task.”

“For instance, Komodakis et al. [21] trained a network to predict the rotation angle of images, enabling the network to comprehend image content.”

“We also scale the updates for a few initial samples in the reset window to reduce their impact on incorrect layer selection.”

“Finally, training tricks include training a larger encoder, including a pre-training step [4], and adding a pre-text task [11] in order to help boost the overall performance without compromising either stability or adaptability.”

“For transfer tasks within a dataset, without using information from other large-scale datasets (e.g., ImageNet-21K [44]), we adopt the RotNet [45] to pre-train an AlexNet model on each dataset as the backbone network, as RotNet is a self-supervised method and can be integrated into any pipeline…”

“Later on, multiple SSL works found that utilizing the domain-specific inductive biases can effectively learn representations in a self-supervised manner, including, colorization [107], solving jigsaw puzzles [64], counting the number of objects [65], and rotation prediction [27], to name a few.”

“The emergence of self-supervised learning begins from unsupervised representation learning works [10, 25, 33], in which pretext tasks such as rotation prediction [10], jigsaw puzzle solving [25] or image colorization [33] are designed to obtain supervision through data manipulation.”

“Prior to the success of contrastive learning, a variety of pretext learning techniques had been developed [61, 62, 51, 50].”

“For example, in early SSL, Doersh et al. [8] and Noroozi Favaro [15] predict the relative position of image patches, and Gidaris et al. [9] uses the rotation transformation of the image to learn visual representation.”

“Methods like MoCo [15], SimCLR [7], jigsaw puzzles [35], colorization [54], orientation discrimination [13], and inpainting [37] are pioneer works of self-supervised learning.”

“However, the formation of such pretext tasks is usually based on heuristic inspiration [11,12].”

“Recently, self-supervised learning [24, 13, 16, 4, 15] has shown remarkable results in representation learning.”

“Instead, the adversarial pretraining aligns with unsupervised representation learning without access to label y as in the above formulation [1, 14, 24, 30, 44, 49, 59, 60, 75, 83], and has separate pretext and downstream tasks that are often more complex than the adversarial training.”

“The idea of transferring a model trained for a given task to a related one has become central to computer vision [55,53], and appears in many research fields such as task transfer [67], domain adaptation [14] or self-supervised learning [20,42,9].”