CoChat
Sign Up
PreprintCited by 115Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsAdvanced Image and Video Retrieval Techniques

What makes instance discrimination good for transfer learning?

Open access
Zhao, Nanxuan, Wu, Zhirong, Lau, Rynson W. H., Lin, Stephen·2020·arXiv
DOI 10.48550/arxiv.2006.06606arXiv 2006.06606
Open this paper in CoChat — free accountOpen PDF

Abstract

Contrastive visual pretraining based on the instance discrimination pretext task has made significant progress. Notably, recent work on unsupervised pretraining has shown to surpass the supervised counterpart for finetuning downstream applications such as object detection and segmentation. It comes as a surprise that image annotations would be better left unused for transfer learning. In this work, we investigate the following problems: What makes instance discrimination pretraining good for transfer learning? What knowledge is actually learned and transferred from these models? From this understanding of instance discrimination, how can we better exploit human annotation labels for pretraining? Our findings are threefold. First, what truly matters for the transfer is low-level and mid-level representations, not high-level representations. Second, the intra-category invariance enforced by the traditional supervised model weakens transferability by increasing task misalignment. Finally, supervised pretraining can be strengthened by following an exemplar-based approach without explicit constraints among the instances within the same category.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.06606

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Traditional cross-entropy based methods assume that all instances within each category should be close in feature distribution while ignoring the unique information of each sample.

    “Specifically, traditional cross-entropy based methods assume that all instances within each category should be close in feature distribution while ignoring the unique information of each sample.”

    From Dual Contrastive Learning for General Face Forgery Detection (2022) · cites this paper as Zhao et al. 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. Pulkit Agrawal, Ross Girshick, and Jitendra Malik. Analyzing the performance of multilayer neural networks for object recognition. In European conference on computer vision, pp. 329-344. Springer, 2014.
  2. Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. From generic to specific deep representations for visual recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 36-45, 2015.
  3. Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017. · arXiv
  4. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020a. · arXiv
  5. Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
  6. Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b. · arXiv
  7. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009.
  8. Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060, 2017.
  9. Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, 2015.
  10. Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with exemplar convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence, 2015.
  11. Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy & transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12387- 12396, 2019.
  12. Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. In Advances in neural information processing systems, pp. 262-270, 2015.
  13. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2414-2423, 2016.
  14. Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728, 2018. · arXiv
  15. Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6928-6938, 2020.
  16. Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014.
  17. Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self-supervised visual representation learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6391-6400, 2019.
  18. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
  19. Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4918-4927, 2019.
  20. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  21. Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272, 2019. · arXiv
  22. Derek Hoiem, Yodsawalai Chodpathumwan, and Qieyun Dai. Diagnosing error in object detectors. In European conference on computer vision, pp. 340-353. Springer, 2012.
  23. Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? arXiv preprint arXiv:1608.08614, 2016. · arXiv
  24. Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. arXiv preprint arXiv:1901.09005, 2019. · arXiv
  25. Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2661-2671, 2019.
  26. Fayao Liu, Chunhua Shen, and Guosheng Lin. Deep convolutional neural fields for depth estimation from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5162-5170, 2015.
  27. Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
  28. Tomasz Malisiewicz, Abhinav Gupta, and Alexei A Efros. Ensemble of exemplar-svms for object detection and beyond. In 2011 International conference on computer vision, pp. 89-96. IEEE, 2011.
  29. Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
  30. Romain Mormont, Pierre Geurts, and Raphaël Marée. Comparison of deep transfer learning strategies for digital pathology. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2262-2271, 2018.
  31. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, 2015.
  32. Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Inte- grated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013. · arXiv
  33. Daniel L Silver and Kristin P Bennett. Guest editor's introduction: special issue on inductive transfer learning. Machine Learning, 73(3):215, 2008.
  34. Daniel L Silver, Qiang Yang, and Lianghao Li. Lifelong machine learning systems: Beyond learning algorithms. In 2013 AAAI spring symposium series, 2013.
  35. Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? arXiv preprint arXiv:1905.07553, 2019. · arXiv
  36. Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019. · arXiv
  37. Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1395-1405, 2019.
  38. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446-9454, 2018.
  39. Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool. Branched multi-task networks: deciding what layers to share. arXiv preprint arXiv:1904.02920, 2019. · arXiv
  40. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
  41. Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11293-11302, 2019.
  42. Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskon- omy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3712-3722, 2018.
  43. Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision. Springer, 2016.
  44. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018.
  45. Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In European conference on computer vision, pp. 94-108. Springer, 2014.
  46. Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. Distilling localization for self-supervised representation learning. arXiv preprint arXiv:2004.06638, 2020. · arXiv
  47. Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
  48. Published as a conference paper at ICLR 2021 MoCo aeroplane Loc: 78% Sim: 8% Oth: 4% BG: 10% bicycle Loc: 60% Sim: 13% Oth: 17% BG: 11% boat Loc: 63% Sim: 5% Oth: 3% BG: 29% bus Loc: 50% Sim: 19% Oth: 12% BG: 19% car Loc: 45% Sim: 11% Oth: 4% BG: 40% motorbike Loc: 54% Sim: 27% Oth: 11% BG: 9% train Loc: 70% Sim: 13% Oth: 1% BG: 16% bird Loc: 52% Sim: 14% BG: 34% cat Loc: 55% Sim: 43% BG: 2% cow Loc: 22% Sim: 69% Oth: 1% BG: 8% dog Loc: 48% Sim: 45% Oth: 1% BG: 6% horse Loc: 49% Sim: 43% Oth: 1% BG: 6% sheep Loc: 23% Sim: 47% BG: 30% bottle Loc: 31% Oth: 9% BG: 61% chair Loc: 44% Sim: 7% Oth: 9% BG: 40% diningtable Loc: 25% Sim: 30% Oth: 25% BG: 20% pottedplant Loc: 40% Oth: 10% BG: 49% sofa Loc: 30% Sim: 33% Oth: 26% BG: 12% tvmonitor Loc: 12% Oth: 22% BG: 66% person Loc: 63% Sim: 2% Oth: 3% BG: 32% Supervised aeroplane Loc: 89% Sim: 3% Oth: 3% BG: 5% bicycle Loc: 73% Sim: 9% Oth: 6% BG: 13% boat Loc: 64% Sim: 1% Oth: 2% BG: 33% bus Loc: 46% Sim: 19% Oth: 10% BG: 25% car Loc: 45% Sim: 6% Oth: 4% BG: 45% motorbike Loc: 74% Sim: 12% Oth: 6% BG: 8% train Loc: 72% Sim: 8% Oth: 3% BG: 18% bird Loc: 55% Sim: 7% Oth: 2% BG: 36% cat Loc: 67% Sim: 28% Oth: 1% BG: 4% cow Loc: 36% Sim: 51% Oth: 1% BG: 12% dog Loc: 73% Sim: 22% Oth: 1% BG: 4% horse Loc: 56% Sim: 39% Oth: 3% BG: 2% sheep Loc: 40% Sim: 31% BG: 29% bottle Loc: 30% Oth: 8% BG: 63% chair Loc: 43% Sim: 10% Oth: 9% BG: 38% diningtable Loc: 37% Sim: 23% Oth: 24% BG: 16% pottedplant Loc: 44% Oth: 8% BG: 48% sofa Loc: 34% Sim: 26% Oth: 28% BG: 12% tvmonitor Loc: 25% Oth: 9% BG: 66% person Loc: 62% Sim: 1% Oth: 3% BG: 35% Figure 7: Distribution of four types of false positives for each category.

Cited by

Sorted by citations

Authors
Zhao, N., Wu, Z., Lau, R. W. H., & Lin, S.
Affiliations
  • City University of Hong Kong (HK)
  • Microsoft (United States) (US)
  • Microsoft Research Asia (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.06606
arXiv
2006.06606
OpenAlex
W3034748927
Cited by
115
References
7
Citations by year
2025: 8 · 2024: 21 · 2023: 26 · 2022: 24 · 2021: 32 · 2020: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
Keywords
instance discrimination, contrastive pre-training, unsupervised pre-training, transfer learning, visual representation learning, object detection, semantic segmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review