CoChat
Sign Up
PreprintCited by 2,477Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsAdvanced Image and Video Retrieval Techniques

The DOI matches its DataCite record; 30 of the 35 references it cites were found in a registry or an index.

Unsupervised Representation Learning by Predicting Image Rotations

Open access
Gidaris, Spyros, Singh, Praveer, Komodakis, Nikos·2018·arXiv
DOI 10.48550/arxiv.1803.07728arXiv 1803.07728
Open this paper in CoChat — free accountOpen PDF

Abstract

Over the last years, deep convolutional neural networks (ConvNets) have transformed the field of computer vision thanks to their unparalleled capacity to learn high level semantic image features. However, in order to successfully learn those features, they usually require massive amounts of manually labeled data, which is both expensive and impractical to scale. Therefore, unsupervised semantic feature learning, i.e., learning without requiring manual annotation effort, is of crucial importance in order to successfully harvest the vast amount of visual data that are available today. In our work we propose to learn image features by training ConvNets to recognize the 2d rotation that is applied to the image that it gets as input. We demonstrate both qualitatively and quantitatively that this apparently simple task actually provides a very powerful supervisory signal for semantic feature learning. We exhaustively evaluate our method in various unsupervised feature learning benchmarks and we exhibit in all of them state-of-the-art performance. Specifically, our results on those benchmarks demonstrate dramatic improvements w.r.t. prior state-of-the-art approaches in unsupervised representation learning and thus significantly close the gap with supervised feature learning. For instance, in PASCAL VOC 2007 detection task our unsupervised pre-trained AlexNet model achieves the state-of-the-art (among unsupervised methods) mAP of 54.4% that is only 2.4 points lower from the supervised case. We get similarly striking results when we transfer our unsupervised learned features on various other tasks, such as ImageNet classification, PASCAL classification, PASCAL segmentation, and CIFAR-10 classification. The code and models of our paper will be published on: https://github.com/gidariss/FeatureLearningRotNet .

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1803.07728

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Gradient-based learning applied to document recognition

    Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×

    “In recent years, the widespread adoption of deep convolutional neural networks (LeCun et al., 1998) (ConvNets) in computer vision, has lead to a tremendous progress in the field.”
    Background

  • Fully convolutional networks for semantic segmentationPreprint

    Long, J., Shelhamer, E., & Darrell, T. (2014) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1605.06211 · Cited by 43,601 · cited 1×

    “For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”
    Background

  • ImageNet Large Scale Visual Recognition Challenge

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 10×

    “Specifically, our self-supervised approach manages to drastically improve the state-of-the-art results on unsupervised feature learning for ImageNet classification, PASCAL classification, PASCAL detection, PASCAL segmentation, and CIFAR-10 classification, surpassing prior approaches by a significant margin and thus drastic…”
    BackgroundMethodInfluential

  • Unsupervised representation learning with deep convolutional generative adversarial networks

    Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235 · cited 1×

    “…(Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al. (2014); Donahue et al. (2016); Radford et al. (2015).”
    Method

  • Deep Visual-Semantic Alignments for Generating Image DescriptionsPreprint

    Karpathy, A., & Fei-Fei, L. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.48550/arxiv.1412.2306 · Cited by 6,303

  • Context Encoders: Feature Learning by Inpainting

    Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · cited 3×

    “Specifically, we first train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Colorization (Zhang et al.,…”
    Influential

  • Deep visual-semantic alignments for generating image descriptions

    Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902

  • Greedy layer-wise training of deep networks

    Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841 · cited 2×

    “…feature learning are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al.…”

  • Colorful Image Colorization

    Zhang, R., Isola, P., & Efros, A. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,417 · cited 9×

    “…59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et al., 2016) 41.9 32.2 NAT (Bojanowski & Joulin,…”
    BackgroundInfluential

  • Fast R-CNNPreprint

    Girshick, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.08083 · Cited by 2,781 · cited 1×

    “For instance, the image features learned by ConvNets in this supervised manner have achieved excellent results when they are transferred to other vision tasks, such as object detection (Girshick, 2015), semantic segmentation (Long et al., 2015), or image captioning (Karpathy & Fei-Fei, 2015).”
    Background

  • Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles

    Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · cited 9×

    “ImageNet labels from (Bojanowski & Joulin, 2017) 59.7 59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et…”
    BackgroundMethodInfluential

  • Unsupervised Visual Representation Learning by Context Prediction

    Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · cited 6×

    “…on other vision tasks, such as object recognition, object detection, and semantic segmentation (Zhang et al., 2016a; Larsson et al., 2016; Zhang et al., 2016b; Larsson et al., 2017; Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2017; Pathak et al., 2016a; Doersch & Zisserman, 2017).”
    BackgroundMethodInfluential

  • Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

    Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · cited 1×

    “…our AlexNet model trains in around 2 days using a single Titan X GPU), and can trivially adopt the efficient parallelization schemes devised for supervised learning (Goyal et al., 2017), making it an ideal candidate for unsupervised learning on internet-scale data (i.e., billions of images).”
    Method

  • Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction

    Masci, J., Meier, U., Ciresan, D. C., & Schmidhuber, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21735-7_7 · Cited by 2,137 · cited 1×

    “…methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al. (2014); Donahue et al. (2016); Radford et…”
    Method

  • Adversarial Feature LearningPreprint

    Donahue, J., Krähenbühl, P., & Darrell, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09782 · Cited by 1,662 · cited 5×

    “…23.4 21.9 18.4 Colorization (Zhang et al., 2016a) 16.0 25.7 29.6 30.3 29.7 Jigsaw Puzzles (Noroozi & Favaro, 2016) 23.0 31.9 35.0 34.2 29.3 BIGAN (Donahue et al., 2016) 22.0 28.7 31.8 31.3 29.7 Split-Brain (Zhang et al., 2016b) 21.3 30.7 34.0 34.1 32.5 Counting (Noroozi et al., 2017) 23…”
    MethodInfluential

  • Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition

    Ranzato, M., Huang, F. J., Boureau, Y.-L., & LeCun, Y. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383157 · Cited by 1,130 · cited 2×

    “…are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al., 2007; Huang et al., 2007; Masci et al., 2011), and methods that involve learning generative probabilistic models Goodfellow et al. (2014); Donahue et al.…”

  • Discriminative Unsupervised Feature Learning with Convolutional Neural NetworksPreprint

    Dosovitskiy, A., Springenberg, J., Riedmiller, M. A., & Brox, T. (2014) · Neural Information Processing Systems · doi:10.48550/arxiv.1406.6909 · Cited by 1,080 · cited 3×

    “Dosovitskiy et al. (2014) train a ConvNet model to yield representations that are discriminative between images and at the same time invariant on geometric and chromatic transformations.”
    BackgroundMethod

  • Learning Representations for Automatic Colorization

    Larsson, G., Maire, M., & Shakhnarovich, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_35 · Cited by 946 · cited 2×

    “For example, in order to learn features, Zhang et al. (2016a) and Larsson et al. (2016) train ConvNets to colorize gray scale images, Doersch et al. (2015) and Noroozi & Favaro (2016) predict the relative position of image patches, and Agrawal et al. (2015) predict the egomotion (i.e., self-motion)…”
    Background

  • Joint Unsupervised Learning of Deep Representations and Image Clusters

    Yang, J., Parikh, D., & Batra, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.556 · Cited by 879 · cited 2×

    “Other successful cases of unsupervised feature learning are clustering based methods (Dosovitskiy et al., 2014; Liao et al., 2016; Yang et al., 2016), reconstruction based methods (Bengio et al.”

  • Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction

    Zhang, R., Isola, P., & Efros, A. A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.76 · Cited by 649 · cited 3×

    “Specifically, we first train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Colorization (Zhang et al.,…”
    BackgroundInfluential

  • Multi-task Self-Supervised Visual Learning

    Doersch, C., & Zisserman, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.226 · Cited by 637 · cited 1×

    “…on other vision tasks, such as object recognition, object detection, and semantic segmentation (Zhang et al., 2016a; Larsson et al., 2016; Zhang et al., 2016b; Larsson et al., 2017; Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2017; Pathak et al., 2016a; Doersch & Zisserman, 2017).”
    Background

  • Colorization as a Proxy Task for Visual Understanding

    Larsson, G., Maire, M., & Shakhnarovich, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.96 · Cited by 481 · cited 1×

    “…on other vision tasks, such as object recognition, object detection, and semantic segmentation (Zhang et al., 2016a; Larsson et al., 2016; Zhang et al., 2016b; Larsson et al., 2017; Doersch et al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2017; Pathak et al., 2016a; Doersch & Zisserman, 2017).”
    Background

  • Learning Features by Watching Objects Move

    Pathak, D., Girshick, R., Dollár, P., Darrell, T., & Hariharan, B. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.638 · Cited by 458 · cited 3×

    “Specifically, we first train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Colorization (Zhang et al.,…”
    BackgroundInfluential

  • Learning to See by Moving

    Agrawal, P., Carreira, J., & Malik, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.13 · Cited by 438 · cited 3×

    “…(2016a) and Larsson et al. (2016) train ConvNets to colorize gray scale images, Doersch et al. (2015) and Noroozi & Favaro (2016) predict the relative position of image patches, and Agrawal et al. (2015) predict the egomotion (i.e., self-motion) of a moving vehicle between two consecutive frames.”
    BackgroundMethod

  • Representation Learning by Learning to Count

    Noroozi, M., Pirsiavash, H., & Favaro, P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.628 · Cited by 338 · cited 3×

    “Specifically, we first train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Colorization (Zhang et al.,…”
    BackgroundInfluential

  • Deep roto-translation scattering for object classification

    Oyallon, E., & Mallat, S. (2014) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2015.7298904 · Cited by 304

  • Network In Network

    Lin, M., Chen, Q., & Yan, S. (2013) · arXiv · doi:10.48550/arxiv.1312.4400 · Cited by 252 · cited 1×

    “In our CIFAR-10 experiments we implement the RotNet models with Network-In-Network (NIN) architectures (Lin et al., 2013).”
    Method

  • Unsupervised Learning by Predicting Noise

    Bojanowski, P., & Joulin, A. (2017) · arXiv · doi:10.48550/arxiv.1704.05310 · Cited by 228 · cited 3×

    “ImageNet labels from (Bojanowski & Joulin, 2017) 59.7 59.7 Random from (Noroozi & Favaro, 2016) 27.1 12.0 Tracking Wang & Gupta (2015) 38.8 29.8 Context (Doersch et al., 2015) 45.6 30.4 Colorization (Zhang et al., 2016a) 40.7 35.2 Jigsaw Puzzles (Noroozi & Favaro, 2016) 45.3 34.6 BIGAN (Donahue et…”
    Influential

  • Data-dependent Initializations of Convolutional Neural NetworksPreprint

    Krähenbühl, P., Doersch, C., Donahue, J., & Darrell, T. (2015) · International Conference on Learning Representations · doi:10.48550/arxiv.1511.06856 · Cited by 211

  • Scaling the Scattering Transform: Deep Hybrid Networks

    Oyallon, E., Belilovsky, E., & Zagoruyko, S. (2017) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2017.599 · Cited by 176

Research works printed but not linked: 6

  1. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
  2. Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
  3. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
  4. Renjie Liao, Alex Schwing, Richard Zemel, and Raquel Urtasun. Learning deep parsimonious representations. In Advances in Neural Information Processing Systems, pp. 5076-5084, 2016.
  5. Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2794-2802, 2015.
  6. Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Sys- tems 27, pp. 487-495.

Cited by

Sorted by citations

See all 2,477

Metadata

Authors
Gidaris, S., Singh, P., & Komodakis, N.
Affiliations
École nationale des ponts et chaussées (FR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
DOI
10.48550/arxiv.1803.07728
arXiv
1803.07728
OpenAlex
W2785325870
Cited by
2,477
References
35
Citations by year
2026: 7 · 2025: 99 · 2024: 298 · 2023: 433 · 2022: 443 · 2021: 678 · 2020: 398 · 2019: 109 · 2018: 12
FWCI
122.22
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • HAL (submittedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • Nvidia
  • Agence Nationale de la Recherche
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
Keywords
unsupervised feature learning, semantic feature learning, convolutional neural networks, feature transfer, object detection, image classification, image segmentation, AlexNet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review