CoChat
Sign Up
PreprintCited by 405Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMachine Learning and Data Classification

With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations

Open access
Dwibedi, Debidatta, Aytar, Yusuf, Tompson, Jonathan, Sermanet, Pierre, Zisserman, Andrew·2021·arXiv
DOI 10.48550/arxiv.2104.14548arXiv 2104.14548
Open this paper in CoChat — free accountOpen PDF

Abstract

Self-supervised learning algorithms based on instance discrimination train encoders to be invariant to pre-defined transformations of the same instance. While most methods treat different views of the same image as positives for a contrastive loss, we are interested in using positives from other instances in the dataset. Our method, Nearest-Neighbor Contrastive Learning of visual Representations (NNCLR), samples the nearest neighbors from the dataset in the latent space, and treats them as positives. This provides more semantic variations than pre-defined transformations. We find that using the nearest-neighbor as positive in contrastive losses improves performance significantly on ImageNet classification, from 71.7% to 75.6%, outperforming previous state-of-the-art methods. On semi-supervised learning benchmarks we improve performance significantly when only 1% ImageNet labels are available, from 53.8% to 56.5%. On transfer learning benchmarks our method outperforms state-of-the-art methods (including supervised learning with ImageNet) on 8 out of 12 downstream datasets. Furthermore, we demonstrate empirically that our method is less reliant on complex data augmentations. We see a relative reduction of only 2.1% ImageNet Top-1 accuracy when we train using only random crops.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2104.14548

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • NNCLR uses a queue of examples Q and defines the positive as the nearest-neighbour of the image in the queue, instead of an augmented version of the image.

    “Instead of defining positive pairs as ( xi , x + i ) where x + i is an augmented version of image xi , NNCLR uses a queue of examples Q , and defines x + i as the nearest-neighbour of xi in the queue.”

    From Detecting galaxy tidal features using self-supervised representation learning (2024) · cites this paper as Dwibedi et al. 2021

  • NNCLR methods aim to resolve the issue of limited positive pairs by creating a more diverse set of positive pairs.

    “NNCLR methods aim to resolve this issue by creating a more diverse set of positive pairs.”

    From Detecting galaxy tidal features using self-supervised representation learning (2024) · cites this paper as Dwibedi et al. 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 73 · Cited by 405

Research works printed but not linked: 45

  1. [4]4. Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L. Alexander, David W. Jacobs, and Peter N. Belhumeur. Bird- snap: Large-scale fine-grained visual categorization of birds. In Proc. Conf. Computer Vision and Pattern Recognition (CVPR), June 2014. 5
  2. [5]5. Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 -mining discriminative components with random forests. In European Conference on Computer Vision, 2014. 5
  3. [6]6. Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pages 132-149, 2018. 2
  4. [7]7. Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Ar- mand Joulin. Unsupervised pre-training of image features on non-curated data. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 2959-2968, 2019. 2
  5. [11]11. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on ma- chine learning, pages 1597-1607. PMLR, 2020. 1, 2, 3, 4, 5, 6
  6. [15]15. Ondrej Chum, James Philbin, Josef Sivic, Michael Isard, and Andrew Zisserman. Total recall: Automatic query expan- sion with a generative feature model for object retrieval. In 2007 IEEE 11th International Conference on Computer Vi- sion, pages 1-8. IEEE, 2007. 3
  7. [16]16. M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014. 5
  8. [17]17. Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsuper- vised visual representation learning by context prediction. In Proceedings of the IEEE international conference on com- puter vision, pages 1422-1430, 2015. 2
  9. [18]18. Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, and Alexei Efros. What makes paris look like paris? ACM Transactions on Graphics, 31(4), 2012. 3
  10. [19]19. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. 13
  11. [21]21. Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle- consistency learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1801-1810, 2019. 3, 7
  12. [22]22. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal- network.org/challenges/VOC/voc2007/workshop/index.html. 5
  13. [23]23. Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition Workshop, 2004.
  14. [24]24. Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. An- alyzing and improving representations with the soft near- est neighbor loss. In International Conference on Machine Learning, pages 2012-2020. PMLR, 2019. 7
  15. [25]25. Dedre Gentner and Laura L Namy. Comparison in the devel- opment of categories. Cognitive Development, 14(4):487- 513, 1999. 1
  16. [26]26. Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6928-6938, 2020. 2
  17. [29]29. Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensional- ity reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pat- tern Recognition (CVPR'06), volume 2, pages 1735-1742. IEEE, 2006. 2
  18. [30]30. Tengda Han, Weidi Xie, and Andrew Zisserman. Self- supervised co-training for video representation learning. In NeurIPS, 2020. 2, 3, 7
  19. [31]31. James Hays and Alexei A Efros. Scene completion using millions of photographs. ACM Transactions on Graphics (ToG), 26(3):4-es, 2007. 3
  20. [32]32. James Hays and Alexei A Efros. Im2gps: estimating geo- graphic information from a single image. In 2008 ieee con- ference on computer vision and pattern recognition, pages 1-8. IEEE, 2008. 3
  21. [33]33. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020. 1, 2, 3, 4, 6, 7
  22. [34]34. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 4, 13
  23. [35]35. Olivier Henaff. Data-efficient image recognition with con- trastive predictive coding. In International Conference on Machine Learning, pages 4182-4192. PMLR, 2020. 5
  24. [36]36. Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu. Unsupervised deep learning by neighbourhood discovery. In International Conference on Machine Learning, pages 2849-2858. PMLR, 2019. 2
  25. [37]37. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. In International conference on machine learn- ing, pages 448-456. PMLR, 2015. 4
  26. [39]39. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS, 2020. 2
  27. [40]40. Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pages 491-507. Springer, 2020. 13
  28. [41]41. Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013. 5
  29. [45]45. Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6707-6717, 2020. 5
  30. [46]46. M-E. Nilsback and A. Zisserman. Automated flower classi- fication over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008. 5
  31. [48]48. O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012. 5
  32. [49]49. Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Dar- rell, and Bharath Hariharan. Learning features by watch- ing objects move. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2701- 2710, 2017. 2
  33. [51]51. Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clus- tering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015. 2
  34. [52]52. Saurabh Singh, Abhinav Gupta, and Alexei A Efros. Unsu- pervised discovery of mid-level discriminative patches. In European Conference on Computer Vision, pages 73-86. Springer, 2012. 3
  35. [53]53. Kihyuk Sohn. Improved deep metric learning with multi- class n-pair loss objective. In Proceedings of the 30th Inter- national Conference on Neural Information Processing Sys- tems, pages 1857-1865, 2016. 3
  36. [54]54. James Thewlis, Samuel Albanie, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of landmarks by descrip- tor vector exchange. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 6361-6371, 2019. 3
  37. [57]57. Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through at- tention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021. 13, 14
  38. [58]58. Xiaolong Wang and Abhinav Gupta. Unsupervised learn- ing of visual representations using videos. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2015. 2
  39. [59]59. Zhirong Wu, Alexei A Efros, and Stella X Yu. Improving generalization via scalable neighborhood component analy- sis. In Proceedings of the European Conference on Computer Vision (ECCV), pages 685-701, 2018. 2, 3
  40. [60]60. Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733- 3742, 2018. 1, 2, 3, 5
  41. [61]61. J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Tor- ralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485- 3492, June 2010. 5
  42. [62]62. Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pages 478-487. PMLR, 2016. 2
  43. [63]63. Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadi- yaram, and Dhruv Mahajan. Clusterfit: Improving gen- eralization of visual representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6509-6518, 2020. 2
  44. [64]64. Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsuper- vised learning of deep representations and image clusters. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5147-5156, 2016. 2
  45. [66]66. Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6002-6012, 2019. 2, 3 ResNet50 [34] 224 - - 76.2 ResNet50 [40] 224 - JFT-300M 79.0 ResNet50 224 NNCLR - 79.1

Authors
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A.
Affiliations
  • Google (United States) (US)
  • Google DeepMind (United Kingdom) (GB)
ORCID
Zisserman, Andrew
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
9568-9577
DOI
10.48550/arxiv.2104.14548
arXiv
2104.14548
OpenAlex
W3158714121
Cited by
405
References
73
Citations by year
2026: 21 · 2025: 74 · 2024: 121 · 2023: 123 · 2022: 56 · 2021: 10
FWCI
35.85
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • IEEE/CVF International Conference on Computer Vision (ICCV)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
self-supervised learning, instance discrimination, visual representation learning, semi-supervised learning, transfer learning, data augmentation, ImageNet classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review