CoChat
Sign Up
Cited by 1,435Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsAdvanced Image and Video Retrieval Techniques

Self-Supervised Learning of Pretext-Invariant Representations

Open access
Misra, Ishan, van der Maaten, Laurens·2020·2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr42600.2020.00674
Open this paper in CoChat — free accountOpen PDF

Abstract

The goal of self-supervised learning from images is to construct image representations that are semantically meaningful via pretext tasks that do not require semantic annotations. Many pretext tasks lead to representations that are covariant with image transformations. We argue that, instead, semantic representations ought to be invariant under such transformations. Specifically, we develop Pretext-Invariant Representation Learning (PIRL, pronounced as `pearl') that learns invariant representations based on pretext tasks. We use PIRL with a commonly used pretext task that involves solving jigsaw puzzles. We find that PIRL substantially improves the semantic quality of the learned image representations. Our approach sets a new state-of-the-art in self-supervised learning from images on several popular benchmarks for self-supervised learning. Despite being unsupervised, PIRL outperforms supervised pre-training in learning image representations for object detection. Altogether, our results demonstrate the potential of self-supervised representations with good invariance properties.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr42600.2020.00674

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 59

  1. Unaiza Ahsan, Rishi Madhok, and Irfan Essa. Video jigsaw: Unsupervised learning of spatiotemporal context for video action recognition. In WACV, 2019. 8
  2. Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, 2017. 8
  3. Relja Arandjelovic and Andrew Zisserman. Objects that sound. In ECCV, 2018. 8
  4. Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.00910, 2019. 5, 8 · arXiv
  5. Fabio M Carlucci, Antonio D'Innocente, Silvia Bucci, Bar- bara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR, 2019. 8
  6. Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018. 5, 6, 8
  7. Aditya Deshpande, Jason Rock, and David Forsyth. Learn- ing large-scale automatic image colorization. In ICCV, 2015. 8
  8. Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsuper- vised visual representation learning by context prediction. In ICCV, pages 1422-1430, 2015. 2, 8
  9. Carl Doersch and Andrew Zisserman. Multi-task self- supervised visual learning. In ICCV, 2017. 8, 13
  10. J. Donahue, P. Krahenbühl, and T. Darrell. Adversarial fea- ture learning. In ICLR, 2016. 8
  11. Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. arXiv preprint arXiv:1907.02544, 2019. 5, 8 · arXiv
  12. Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springen- berg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. TPAMI, 38(9), 2016. 1, 8
  13. Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle- consistency learning. In CVPR, 2019. 8
  14. Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learn- ing with odd-one-out networks. In CVPR, 2017. 8
  15. Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In ECCV, 2018. 8
  16. Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self-supervised visual rep- resentation learning. In ICCV, 2019. 3, 4, 5, 6, 7, 8, 12, 13, 14
  17. Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems, pages 529-536, 2005. 5
  18. Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth Inter- national Conference on Artificial Intelligence and Statistics, pages 297-304, 2010. 2
  19. Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimension- ality reduction by learning an invariant mapping. In CVPR, 2006. 2, 8
  20. Tengda Han, Weidi Xie, and Andrew Zisserman. Video rep- resentation learning by dense predictive coding. In ICCV Workshop, pages 0-0, 2019. 8
  21. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Let there be color!: joint end-to-end learning of global and lo- cal image priors for automatic image colorization with si- multaneous classification. ACM Transactions on Graphics, 35(4):110, 2016. 8
  22. X. Ji, J.F. Henriques, and A. Vedaldi. Invariant information clustering for unsupervised image classification and segmen- tation. In ICCV, 2019. 1, 8
  23. Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In ECCV, 2016. 5, 6
  24. Angjoo Kanazawa, David W Jacobs, and Manmohan Chan- draker. Warpnet: Weakly supervised matching for single- view reconstruction. In CVPR, pages 3253-3261, 2016. 1
  25. Dahun Kim, Donghyeon Cho, and In So Kweon. Self- supervised video representation learning with space-time cu- bic puzzles. In AAAI, volume 33, 2019. 8
  26. Bruno Korbar, Du Tran, and Lorenzo Torresani. Coopera- tive learning of audio and video models from self-supervised synchronization. In NeurIPS, 2018. 8
  27. Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR, 2017. 5, 8
  28. Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming- Hsuan Yang. Unsupervised representation learning by sort- ing sequences. In CVPR, 2017. 8
  29. Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In NeurIPS, 2019. 4
  30. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV. 2014. 1, 4
  31. Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018. 1, 5
  32. J. Masci, U. Meier, D. Cires, and J. Schmidhuber. Stacked convolutional auto-encoders for hierarchical feature extrac- tion. In ICANN, pages 52-59, 2011. 8
  33. L. Mescheder, S. Nowozin, and A. Geiger. Adversarial vari- ational Bayes: Unifying variational autoencoders and gener- ative adversarial networks. In ICML, 2017. 8
  34. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regulariza- tion method for supervised and semi-supervised learning. TPAMI, 41(8), 2018. 5
  35. Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016. 1, 2, 3, 7, 8
  36. Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Rep- resentation learning by learning to count. In ICCV, 2017. 8
  37. Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash. Boosting self-supervised learning via knowledge transfer. In CVPR, 2018. 8
  38. David Novotny, Samuel Albanie, Diane Larlus, and Andrea Vedaldi. Self-supervised learning of geometrically stable features through probabilistic introspection. In CVPR, pages 3637-3645, 2018. 1
  39. B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natu- ral images. Nature, 381(6583):607, 1996. 8
  40. Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018. 8
  41. Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In ECCV, 2016. 8
  42. Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching ob- jects move. In CVPR, 2017. 8
  43. MarcAurelio Ranzato, Fu-Jie Huang, Y-Lan Boureau, and Yann LeCun. Unsupervised learning of invariant feature hi- erarchies with applications to object recognition. In CVPR, 2007. 8
  44. Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. Convo- lutional neural network architecture for geometric matching. In CVPR, pages 6148-6157, 2017. 1
  45. R. Salakhutdinov and G. Hinton. Deep Boltzmann machines. In AI-STATS, pages 448-455, 2009. 8
  46. Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In ICRA, 2018. 8
  47. Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743, 2019. 8 · arXiv
  48. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhi- nav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017. 5
  49. Yonglong Tian, Dilip Krishnan, and Phillip Isola. Con- trastive multiview coding. arXiv preprint arXiv:1906.05849, 2019. 5, 8 · arXiv
  50. P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising au- toencoders. In ICML, 2008. 8
  51. Xiaolong Wang, Kaiming He, and Abhinav Gupta. Tran- sitive invariance for self-supervised visual representation learning. In ICCV, pages 1329-1338, 2017. 8
  52. Donglai Wei, Joseph Lim, Andrew Zisserman, and William T. Freeman. Learning and using the arrow of time. In IEEE Conference on Computer Vision and Pattern Recog- nition, 2018. 8
  53. Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation. arXiv preprint arXiv:1904.12848, 2019. 8 · arXiv
  54. Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In CVPR, 2019. 8
  55. Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lu- cas Beyer. S4l: Self-supervised semi-supervised learning. arXiv preprint arXiv:1905.03670, 2019. 5 · arXiv
  56. Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016. 8
  57. Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel pre- diction. In CVPR, 2017. 3, 4, 8
  58. Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Tor- ralba, and Aude Oliva. Learning deep features for scene recognition using places database. In NIPS, 2014. 4
  59. Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In ICCV, pages 6002-6012, 2019. 5, 13

Cited by

Sorted by citations

Authors
Misra, I., & van der Maaten, L.
Affiliations
Meta (United States) (US)
ORCID
Misra, Ishan
Venue
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2020-06
Pages
6706-6716
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr42600.2020.00674
OpenAlex
W2994536315
Cited by
1,435
References
25
Citations by year
2026: 46 · 2025: 162 · 2024: 221 · 2023: 265 · 2022: 323 · 2021: 298 · 2020: 117 · 2019: 3
FWCI
95.67
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
Keywords
self-supervised learning, image representation learning, invariant representations, pretext tasks, image transformation, semantic representation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review