Distilling Localization for Self-Supervised Representation Learning
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Recent progress in contrastive learning has revolutionized unsupervised representation learning. Concretely, multiple views (augmentations) from the same image are encouraged to map to close embeddings, while views from different images are pulled apart.In this paper, through visualizing and diagnosing classification errors, we observe that current contrastive models are ineffective at localizing the foreground object, limiting their ability to extract discriminative high-level features. This is due to the fact that view generation process considers pixels in an image uniformly.To address this problem, we propose a data-driven approach for learning invariance to backgrounds. It first estimates foreground saliency in images and then creates augmentations by copy-and-pasting the foreground onto a variety of back-grounds. The learning still follows an instance discrimination approach, so that the representation is trained to disregard background content and focus on the foreground. We study a variety of saliency estimation methods, and find that most methods lead to improvements for contrastive learning. With this approach, significant performance is achieved for self-supervised learning on ImageNet classification, and also for object detection on PASCAL VOC and MSCOCO.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
Unsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664
Self-citationUnsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629
Tian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997
Self-Supervised Learning of Pretext-Invariant Representations
Misra, I., & van der Maaten, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00674 · Cited by 1,435
AutoAugment: Learning Augmentation Policies from Data
Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., & Le, Q. V. (2018) · arXiv · doi:10.48550/arxiv.1805.09501 · Cited by 1,106
Object Detectors Emerge in Deep Scene CNNs
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2014) · arXiv · doi:10.48550/arxiv.1412.6856 · Cited by 971
Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
Dosovitskiy, A., Fischer, P., Springenberg, J. T., Riedmiller, M., & Brox, T. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2496141 · Cited by 762
Tian, Y., Krishnan, D., & Isola, P. (2019) · arXiv · doi:10.48550/arxiv.1906.05849 · Cited by 465
InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting
Fang, H.-S., Sun, J., Wang, R., Gou, M., Li, Y.-L., & Lu, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00077 · Cited by 228
Cited by
What makes for good views for contrastive learning
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., & Isola, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10243 · Cited by 773
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends
Gui, J., Chen, T., Zhang, J., Cao, Q., Sun, Z., Luo, H., & Tao, D. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3415112 · Cited by 552
A Cookbook of Self-Supervised Learning
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12210 · Cited by 182
“…learning algorithms, express seemingly contradictory findings, namely that existing self-supervised learning strategies are or are not effective for downstream dense prediction tasks [Goyal et al., 2019, Purushwalkam and Gupta, 2020, Zhao et al., 2021, Ericsson et al., 2021b, Shwartz-Ziv et al.].”
BackgroundWhat makes instance discrimination good for transfer learning?
Zhao, N., Wu, Z., Lau, R. W. H., & Lin, S. (2020) · arXiv · doi:10.48550/arxiv.2006.06606 · Cited by 115
Self-citationOn Robustness and Transferability of Convolutional Neural Networks
Djolonga, J., Yung, J., Tschannen, M., Romijnders, R., Beyer, L., Kolesnikov, A., Puigcerver, J., Minderer, M., D'Amour, A., Moldovan, D., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01619 · Cited by 98
Metadata
- Authors
- Zhao, N., Wu, Z., Lau, R. W. H., & Lin, S.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2021-05-18
- Volume
- 35
- Issue
- 12
- Pages
- 10990-10998
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W3175156984
- Cited by
- 37
- References
- 12
- FWCI
- 3.33
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.