Unsupervised Representation Learning by Predicting Image Rotations

“In recent years, the widespread adoption of deep convolutional neural networks (LeCun et al., 1998) (ConvNets) in computer vision, has lead to a tremendous progress in the field.”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“Specifically, our self-supervised approach manages to drastically improve the state-of-the-art results on unsupervised feature learning for ImageNet classification, PASCAL classification, PASCAL detection, PASCAL segmentation, and CIFAR-10 classification, surpassing prior approaches by a significant margin and thus drastic…”

“…(Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al. (2014); Donahue et al. (2016); Radford et al. (2015).”

“Specifically, we first train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Colorization (Zhang et al.,…”

“…feature learning are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al.…”

“…59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et al., 2016) 41.9 32.2 NAT (Bojanowski & Joulin,…”

“For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”

“, 2017; Donahue and Simonyan, 2019), pretext based (Zhang et al., 2016; Pathak et al., 2016; Noroozi and Favaro, 2016; Gidaris et al., 2018; Lee et al., 2020; Teng and Huang, 2021) and contrastive based (Hjelm et al.”

“SSL has been found to be effective for different domains, such as images [39, 86, 91, 131, 143, 189], natural language [84, 143, 177], graphs [113], and point-clouds [56, 73, 153].”

“For images, pretext tasks typically include solving jigsaw puzzles [41], coloring images [62], and predicting relative patches [10] or image rotations [15], etc .”

“Self-Supervised Learning Early approaches for unsupervised representation learning leverages handcrafted tasks, like prediction rotation [38] and solving the Jigsaw puzzle [39, 40], geometry prediction [41] and Selfie [42].”

“Auxiliary pretext tasks can include predicting the rotation degree [20], filling in a missing part of an image [21], colorizing a grayscale image [22–24], predicting the relative position of a patch [25], and more.”

“Different pretext tasks correspond to different pre-training paradigms, the typical ones including rotation prediction (Gidaris et al., 2018), contrastive learning (He et al., 2020), and masked image modeling (He et al., 2022), etc. Related works generally employed the loss of pretext task for AL.”

“Rotation prediction [134, 135] involves rotating the input image and predicting the rotation angle, enabling the model to learn the rotational invariance of the image.”

“Inspired by the 2D method that predicts image rotations [127], Poursaeed et al.”

“In addition, some studies [161], [162] employed self-supervised learning as an auxiliary task to help the primary task learn better by training the auxiliary task alongside the primary task.”

“Early discriminative approaches relied on heuristics when defining the pretext tasks[18,14,90,55,22], limiting the generality of the resulting representations.”

“Earlier self-supervised works used “pretext tasks” such as rotation prediction [15], solving jigsaw puzzles [28], colorization [44], or missing data completion [17].”

“Representation-based methods Self-supervised learning has been used in the past to learn image features [29, 31, 14], often solving auxiliary tasks.”

“supervision learning [29], [30], which uses the curvature and density of the point cloud to generate sharp edge points or points in sparse regions.”

“Discriminative SSL methods facilitate learning with proxy tasks (Pathak et al., 2016; Noroozi & Favaro, 2016; Zhang et al., 2016; Gidaris et al., 2018).”

“SupCon [21] converted to “class discrimination” task, leveraging the label information to decide positive pairs.”

“Inspired by its success in CV [6, 9, 36] and NLP [7, 8, 23] domains, researchers are exploring contrastive learning for recommendation [40, 46, 47].”

“Specifically, we optimizes a rotation prediction loss [19].”

“Early image-based self-supervised learning work focused on pretext tasks that require useful representations to solve [24,25,48, 58,61,78,90].”

“Early attempts use pretext tasks, e.g. colorization (Zhang et al., 2016), rotation prediction (Gidaris et al., 2018), jigsaw (Noroozi & Favaro, 2016), as the self-supervised objective.”

“Being a dominant self-supervised learning approach, contrastive learning has recently made significant progress in a wide range of domains such as computer vision [23, 28, 43, 67] and natural language processing [21, 27, 42, 59, 77].”

“audio [34] or text [38]), as well as colorization of artificially gray-scaled images [40], jigsaw tasks [20] or rotation prediction [12] in image data.”

“, rotation prediction (Gidaris et al., 2018), jigsaw puzzle (Noroozi & Favaro, 2016)) or requires a large batch of negative sample pairs as is the case for contrastive learning (Chen et al.”

“…tasks for self-supervised learning have been proposed, including the task of predicting the relative positions of neighboring patches within an image [23], the task of inpainting [66], solving Jigsaw Puzzles [62], image colorization [53,99], rotation prediction [34], and other tasks [8,52,100].”

“Common approaches include reconstruction tasks (Vincent et al., 2008; Erhan et al., 2010; Devlin et al., 2019; Gidaris et al., 2018; Lewis et al., 2020), and contrastive learning (He et al., 2020; Chen et al., 2020b; Caron et al., 2020; Radford et al., 2021b), and recent work has shown that…”

“…models that were trained with a non-contrastive learning objective (no negative examples), BarlowTwins (Zbontar et al., 2021), SwAV (Caron et al., 2020), and VICReg (Bardes et al., 2022), as well as earlier SSL, non-Siamese models, Jigsaw (Noroozi & Favaro, 2016), and RotNet (Gidaris et al., 2018).”

“Here we briefly introduce some of the most related state-of-the-art source-free methods: SHOT [24] and CPGA [32] exploit pseudo label prediction to achieve class-wise adaptation, NRC [48] and G-SFDA [47] strengthen the local structure by neighborhood information from source model, and SHOT++ [25] is a two-stage extens…”

“Proxy Task is referred to as learning with free labels generated from the data itself, such as solving Jigsaw puzzles [19], predicting rotations [6] or reconstruction [5].”

“Self-supervised learning is a way to learn generalizable visual representations often using different pretext tasks for pre-training [55–59].”

“This design choice stems from the fact that orientation in natural images is often important for preserving object context—predicting the “correct” orientation of rotated images is even an effective pretext task for pre-training (Gidaris et al., 2018).”

“A few pretext tasks for learning image representations include image colourization [41, 42], object counting [48, 55], solving jigsaw puzzles [5, 36], predicting image rotations [20, 37], and reconstructing input images [29] from noise [71].”

“image color [97], patch ordering [57] or image orientation [25]) and trying to recover it.”

“4) Self-Supervision-Based Approaches: Recently, self-supervised learning has been widely used as it benefits many downstream tasks like classification [52], [53], detection [34], [54], segmentation [53], and tracking [55].”

“These methods encompass a variety of techniques including, but not limited to, rotation prediction (Gidaris et al., 2018), contrastive learning (Bartler et al., 2022; Liu et al., 2021a), entropy minimization (Wang et al., 2021), etc.”

“These methods have shown great promise across many domains, including computer vision [22], natural language processing [16, 54, 55], and speech processing [11–13].”

“Existing pretrain pretext tasks can be divided into discriminative tasks [4, 9, 14, 48] and generative tasks [3, 11, 19, 26, 27, 33, 51, 54].”

“Early explorations range from low-level pixel-wise reconstruction tasks that include denoising [66], inpainting [57], and cross-channel prediction [83] to higher-level instance discrimination [17], rotation prediction [22], context prediction [16], jigsaw puzzle [54], counting [55], and colorization [82].”

“Without ground-truth labels, a discrimiantive pretext task can be designed as solving jigsaw puzzles [17] or predicting rotation [18].”

“Representative methods include: solving a jigsaw puzzle [39], rotation prediction [14], Instance Discrimination (ID) [10,17,65], contrasting cluster assignments [7], self knowledge distillation [8], and masked image modeling [2].”

“These approaches learn without labels using pretext tasks, which can be discriminative [23, 37], contrastive [11, 12, 24, 26], or generative, where the model predicts a degraded version of its input [27, 51].”

“Self-supervised learning [14, 35, 43] approaches have been explored to minimise external supervision.”

“Another method is to use discriminative models with contrastive learning [5,27,40].”

“Here, inspired by existing methods [5, 12, 34, 47], we design the task of distinguishing the types of stance features as a pretext task in order to provide a surrogate supervised signal for contrastive learning.”

“In recent years, self-supervised learning has become popular in computer vision, and many methods, such as pretext-task-based self-supervised learning algorithms [19], [20], [21], [22] and contrastive-learning-based models [23], [24], have been proposed.”

“prediction task to construct the signal [117], [118],”

“Our conjecture is that this is because RotNet mainly encourages learning global representation, and heavily removing the global information from the input via privacy removal branch leads to drop in action recognition performance as well.”

“A rich collection of such intuitions and objectives was further developed, ranging from pixel-wise reconstructionbased approaches, such as denoising [82], inpainting [67], colorization [50, 94], and more [23, 95], to higher-level pretext tasks, such as predicting spatial layouts [21, 62, 63], orientation [30], egomoti…”

“Whereas early self-supervised learners such as RotNet [26] or DeepCluster [10] substantially lagged their supervised counterparts in vision pre-training, more recent approaches have become quite competitive.”

“Self-supervised learning aims to learn from unlabeled visual data by a pre-text task that is constructed by image/patch operations ( e.g ., [14, 51, 75, 79, 26, 4]) and spatial-temporal operations ( e.g ., [28, 48, 25, 52, 73]).”

“…predictive self-supervised auxiliary tasks widely applied in natural image processing are relative position prediction ( Doersch et al 2015 ) , solving Jigsaw puzzles ( Noroozi and Favaro 2016 ) , and rotation angle prediction ( Gidaris et al 2018 ) , etc. Study in pathological image analysis.”

“Other approaches [55,41] propose to jointly optimize a main task(1) and a selfsupervised task, such as rotation prediction [15] or instance discrimination [20,7], during pre-training in the source domain, and then update the model using only the self-supervised task during test time.”

“One common optimization objective is to learn invariant representation against various augmentations, such as rotation [20], colorization [66], mixup [41] and random erasing [71].”

“Self-supervisedcontrastivelearning[28],[69],[70],[71],[72] has recently emerged as the most competitive method for learning representations without labels, significantly outperforming other pretext task-based alternatives [73], [74], [75], [76].”

“Based on the nature of the discrimination task, the instantiation of Ldis can be crossentropy [22,26,35,58], contrastive [3,8,12,27], redundancy reduction [21, 53], etc.”

“While prediction tasks alone have been realized successfully before in SSL (Agrawal et al., 2015; Doersch et al., 2015; Zhang et al., 2016; Misra et al., 2016; Noroozi & Favaro, 2016; Zamir et al., 2016; Lee et al., 2017; Mundhenk et al., 2018; Gidaris et al., 2018; Zhang et al., 2019; Zhang, 2020), to our knowledge w…”

“Initially focused on discriminative tasks [52, 53, 54], recent self-supervised approaches for images can be categorized as contrastive or generative.”

“AAACAnicbVC7TsMwFHV4lvIKMCEWi7ZSWaKkA2WsYGEsj7aR0qhyXKe16jiR7SBVUcXCr7AwgBArX8HG3+C2GaDlSJaOzrlH1/cECaNS2fa3sbK6tr6xWdgqbu/s7u2bB4dtGacCkxaOWSzcAEnCKCctRRUjbiIIigJGOsHoaup3HoiQNOb3apwQP0IDTkOKkdJSzzwu18+terdShtU7hXTqliTQc13/rGeWbMueAS4TJyclkKPZM7+6/RinEeEKMySl59iJ8jMkFMWMTIrdVJIE4REaEE9TjiIi/Wx2wgRWtNKHYSz04wrO1N+JDEV…”

“One can interpret our co-attention classifier from a view of auxiliary-task learning [86], [87], which is investigated in the self-supervised learning field to improve data efficiency and robustness, by exploring auxiliary tasks from inherent data structures.”

“Among all the proposed methods (Noroozi & Favaro, 2016; Zhang et al., 2016b; Gidaris et al., 2018; Caron et al., 2018; 2020), a particularly effective one is the contrastive learning, introduced by SimCLR Chen et al. (2020a).”

“Previous self-supervised methods utilize different priors of images to design clever pretext tasks, such as predicting the patch positions [18], inpainting [31], colorization [39], and rotation prediction [22].”

“Generative approaches predict the original data based on the partially observed inputs [59, 49, 38], predict the transformation applied to the input [46, 18], or model pixels in the input space [36, 20, 33, 27].”

“rotation and distortion), patches, or frame sequence [14, 9, 8, 25, 6, 7, 41].”

“While early efforts in vision models focused on simple pre-text tasks [12, 16, 32, 35, 52], the field has evolved through stages like contrastive learning [5, 17] and masked image modeling [2, 18, 49].”

“These included predicting spatial context (Doersch et al., 2015), image rotation (Gidaris et al., 2018), artificial classes based on geometric transformations (Dosovitskiy et al., 2014a), and image patch arrangement (Noroozi and Favaro, 2016).”

“In the image domain, various selfsupervised learning techniques have been developed for learning high-level image representations, including predicting the relative locations of image patches [10], jigsaw puzzle [36], image rotation [13], image clustering [4], image inpainting [38], image colorization [51] and segment…”

“in [46] suggested a method for learning image features by training CNNs to recognize the two-dimensional rotation that is applied to the picture that it receives as input.”

“As a self-supervised pretext task, next token prediction [4] and masked token prediction [8] are commonly used in natural language processing, while solving jigsaw puzzles [35] and rotation prediction [22] are proposed in computer vision.”

“In addition, in contrast to hand-crafted auxiliary tasks [27, 65], which may require extensive domain knowledge and hyper-parameter tuning, contrastive methods are task and domain-agnostic.”

“Early SSL methods were based on engineered pretext tasks to encourage the model to learn distinct features by solving auxiliary problems such as image colorization [71], jigsaw puzzles [48], or rotation angle prediction [21].”

“Early methods mostly focused on direct estimation of the transformation applied to images [22, 30, 60, 63, 92].”

“Many these kind of proxy features are accessible by simple image manipulation, including image inpainting [37], colorization [62], jigsaw puzzles [33], rotate prediction [18], and etc.”

“Therefore, an image along with all its possible image transformations can be treated as one class (Dosovitskiy et al., 2014; Gidaris et al., 2018).”

“In contrast to the works [7,11,41] that introduce self-supervised pretext tasks such as rotation [13] and jigsaw [29] into FSL as auxiliary losses, recent approaches [9,25,26,32] have explored contrastive learning of instance discrimination in different parts of the two-stage training pipeline of FSL.”

“To generate individual augmentations, images usually use spatial [55], [56] and appearance transformation [57], [58], and natural language texts mostly use backtranslation approach [53] and spans technique [54].”

“One branch of methods optimize the network to solve pretext tasks e.g., image coloring [62], inpainting [41], transformation prediction [19], and patch ordering [35].”

“Pre-training a CNN on ImageNet using recent unsupervised [43] or self-supervised learning [24] approaches, then fine-tune such network for the intrinsic decomposition problem, could provide a solu-”

“…pretext tasks have been proposed to learn these representations, such as image in-painting (He et al. 2022), colorization (Zhang, Isola, and Efros 2016), or prediction of the rotation (Gidaris, Singh, and Komodakis 2018) or position of patches (Doersch, Gupta, and Efros 2015; Carlucci et al. 2019).”

“Test Time Training (TTT) [36, 55] adapts the model during the test phase via the self-supervise task, such as rotation classification [15].”

“Pretext tasks for the image domain include solving Jigsaw-puzzles (Noroozi & Favaro, 2016), predicting rotations or affine transformations (Gidaris et al., 2018; Zhang et al., 2019b) or discriminating instances (Wu et al., 2018; van den Oord et al., 2018).”

“There is a wide range of tasks that have been designed to learn useful visual representations, including colorization (Zhang, Isola, and Efros 2016), inpainting (Pathak et al. 2016), spatial jigsaw puzzles (Noroozi and Favaro 2016), and discriminate orientation (Gidaris, Singh, and Komodakis 2018).”