CoChat
Sign Up
PreprintCited by 266Domain Adaptation and Few-Shot LearningAdvanced Image and Video Retrieval TechniquesAdvanced Neural Network Applications

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

Contrastive Learning with Stronger Augmentations

Open access
Wang, Xiao, Qi, Guo-Jun·2021·arXiv
DOI 10.48550/arxiv.2104.07713arXiv 2104.07713
Open this paper in CoChat — free accountOpen PDF

Abstract

Representation learning has significantly been developed with the advance of contrastive learning methods. Most of those methods have benefited from various data augmentations that are carefully designated to maintain their identities so that the images transformed from the same instance can still be retrieved. However, those carefully designed transformations limited us to further explore the novel patterns exposed by other transformations. Meanwhile, as found in our experiments, the strong augmentations distorted the images' structures, resulting in difficult retrieval. Thus, we propose a general framework called Contrastive Learning with Stronger Augmentations~(CLSA) to complement current contrastive learning approaches. Here, the distribution divergence between the weakly and strongly augmented images over the representation bank is adopted to supervise the retrieval of strongly augmented queries from a pool of instances. Experiments on the ImageNet dataset and downstream datasets showed the information from the strongly augmented images can significantly boost the performance. For example, CLSA achieves top-1 accuracy of 76.2% on ImageNet with a standard ResNet-50 architecture with a single-layer classifier fine-tuned, which is almost the same level as 76.5% of supervised results. The code and pre-trained models are available in https://github.com/maple-research-lab/CLSA.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2104.07713

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Stronger data augmentations can benefit representation learning by introducing abundant semantic information that improves the generalizability of learned representations and can bridge the gap with fully supervised methods.

    “studies (Tian et al. 2020; Wang and Qi 2021) have shown that data augmentation design is crucial, and the abundant semantic information introduced by stronger data augmentations can significantly improve the generalizability of learned representations and eventually bridge the gap with the fully supervised methods.”

    From Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-Supervised Action Recognition (2022) · cites this paper as Wang and Qi 2021

  • CLSA found that the normally augmented query and the extremely augmented query share similar distribution for a randomly initialized network.

    “Fortunately, CLSA (Wang and Qi 2021) found that the normally augmented query and the extremely augmented query share similar distribution for a randomly initialized network.”

    From Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-Supervised Action Recognition (2022) · cites this paper as Wang and Qi 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 59

  1. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248-255.
  2. R. Hadsell, S. Chopra, and Y. LeCun, "Dimensionality reduction by learning an invariant mapping," in 2006 IEEE Computer Society Con- ference on Computer Vision and Pattern Recognition (CVPR'06), vol. 2. IEEE, 2006, pp. 1735-1742.
  3. I. Misra and L. v. d. Maaten, "Self-supervised learning of pretext- invariant representations," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6707-6717.
  4. X. Chen, H. Fan, R. Girshick, and K. He, "Improved baselines with mo- mentum contrastive learning," arXiv preprint arXiv:2003.04297, 2020. · arXiv
  5. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, "Momentum contrast for unsupervised visual representation learning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9729-9738.
  6. M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, "Unsupervised learning of visual features by contrasting cluster assign- ments," arXiv preprint arXiv:2006.09882, 2020. · arXiv
  7. O. Maron and T. Lozano-Pérez, "A framework for multiple-instance learning," Advances in neural information processing systems, pp. 570- 576, 1998.
  8. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple frame- work for contrastive learning of visual representations," arXiv preprint arXiv:2002.05709, 2020. · arXiv
  9. Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, "Unsupervised feature learning via non-parametric instance discrimination," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3733-3742.
  10. R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bach- man, A. Trischler, and Y. Bengio, "Learning deep representations by mutual information estimation and maximization," arXiv preprint arXiv:1808.06670, 2018. · arXiv
  11. A. v. d. Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," arXiv preprint arXiv:1807.03748, 2018. · arXiv
  12. P. Bachman, R. D. Hjelm, and W. Buchwalter, "Learning representations by maximizing mutual information across views," in Advances in Neural Information Processing Systems, 2019, pp. 15 535-15 545.
  13. C. Zhuang, A. L. Zhai, and D. Yamins, "Local aggregation for unsu- pervised learning of visual embeddings," in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 6002-6012.
  14. Y. Tian, D. Krishnan, and P. Isola, "Contrastive multiview coding," arXiv preprint arXiv:1906.05849, 2019. · arXiv
  15. O. J. Hénaff, A. Srinivas, J. De Fauw, A. Razavi, C. Doersch, S. Eslami, and A. v. d. Oord, "Data-efficient image recognition with contrastive predictive coding," arXiv preprint arXiv:1905.09272, 2019. · arXiv
  16. Q. Hu, X. Wang, W. Hu, and G.-J. Qi, "Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries," arXiv preprint arXiv:2011.08435, 2020. · arXiv
  17. Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, "What makes for good views for contrastive learning," arXiv preprint arXiv:2005.10243, 2020. · arXiv
  18. E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, "Randaugment: Practical automated data augmentation with a reduced search space," in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 702-703.
  19. E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, "Au- toaugment: Learning augmentation policies from data," arXiv preprint arXiv:1805.09501, 2018. · arXiv
  20. X. Wang, D. Kihara, J. Luo, and G.-J. Qi, "Enaet: Self-trained ensem- ble autoencoding transformations for semi-supervised learning," arXiv preprint arXiv:1911.09265, 2019. · arXiv
  21. K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel, "Fixmatch: Simplifying semi- supervised learning with consistency and confidence," arXiv preprint arXiv:2001.07685, 2020. · arXiv
  22. J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar et al., "Bootstrap your own latent: A new approach to self-supervised learning," arXiv preprint arXiv:2006.07733, 2020. · arXiv
  23. X. Wang, Z. Liu, and S. X. Yu, "Unsupervised feature learning by cross-level discrimination between instances and groups," arXiv preprint arXiv:2008.03813, 2020. · arXiv
  24. D. P. Kingma and M. Welling, "Auto-encoding variational bayes," arXiv preprint arXiv:1312.6114, 2013. · arXiv
  25. J. Donahue, P. Krähenbühl, and T. Darrell, "Adversarial feature learning," arXiv preprint arXiv:1605.09782, 2016. · arXiv
  26. M. Caron, P. Bojanowski, A. Joulin, and M. Douze, "Deep clustering for unsupervised learning of visual features," in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 132-149.
  27. X. Chen and K. He, "Exploring simple siamese representation learning," arXiv preprint arXiv:2011.10566, 2020. · arXiv
  28. B. Roh, W. Shin, I. Kim, and S. Kim, "Spatially consistent representation learning," arXiv preprint arXiv:2103.06122, 2021. · arXiv
  29. C. Wei, H. Wang, W. Shen, and A. Yuille, "{CO}2: Consistent contrast for unsupervised visual representation learning," in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=U4XLJhqwNF1
  30. J. Mitrovic, B. McWilliams, J. Walker, L. Buesing, and C. Blun- dell, "Representation learning via invariant causal mechanisms," arXiv preprint arXiv:2010.07922, 2020. · arXiv
  31. G.-J. Qi, L. Zhang, and X. Wang, "Learning generalized transformation equivariant representations via autoencoding transformations," arXiv preprint arXiv:1906.08628, 2019. · arXiv
  32. M. Noroozi and P. Favaro, "Unsupervised learning of visual representa- tions by solving jigsaw puzzles," in European conference on computer vision. Springer, 2016, pp. 69-84.
  33. C. Doersch, A. Gupta, and A. A. Efros, "Unsupervised visual repre- sentation learning by context prediction," in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1422-1430.
  34. L. Zhang, G.-J. Qi, L. Wang, and J. Luo, "Aet vs. aed: Unsupervised representation learning by auto-encoding transformations rather than data," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 2547-2555.
  35. L. Jing and Y. Tian, "Self-supervised visual feature learning with deep neural networks: A survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
  36. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
  37. D. Ciregan, U. Meier, and J. Schmidhuber, "Multi-column deep neural networks for image classification," in 2012 IEEE conference on computer vision and pattern recognition. IEEE, 2012, pp. 3642-3649.
  38. P. Y. Simard, D. Steinkraus, J. C. Platt et al., "Best practices for convolutional neural networks applied to visual document analysis." in Icdar, vol. 3, no. 2003, 2003.
  39. L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus, "Regularization of neural networks using dropconnect," in International conference on machine learning. PMLR, 2013, pp. 1058-1066.
  40. A. Antoniou, A. Storkey, and H. Edwards, "Data augmentation generative adversarial networks," arXiv preprint arXiv:1711.04340, 2017. · arXiv
  41. S. Mun, S. Park, D. K. Han, and H. Ko, "Generative adversarial network based acoustic scene training set augmentation and selection using svm hyper-plane," Proc. DCASE, pp. 93-97, 2017.
  42. L. Perez and J. Wang, "The effectiveness of data augmentation in image classification using deep learning," arXiv preprint arXiv:1712.04621, 2017. · arXiv
  43. S. Ravuri and O. Vinyals, "Classification accuracy score for conditional generative models," arXiv preprint arXiv:1905.10887, 2019. · arXiv
  44. E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, "Autoaug- ment: Learning augmentation strategies from data," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 113-123.
  45. D. Ho, E. Liang, X. Chen, I. Stoica, and P. Abbeel, "Population based augmentation: Efficient learning of augmentation policy schedules," in International Conference on Machine Learning. PMLR, 2019, pp. 2731-2741.
  46. S. Lim, I. Kim, T. Kim, C. Kim, and S. Kim, "Fast autoaugment," in Advances in Neural Information Processing Systems, 2019, pp. 6665- 6675.
  47. R. Hataya, J. Zdenek, K. Yoshizoe, and H. Nakayama, "Faster autoaug- ment: Learning augmentation strategies using backpropagation," arXiv preprint arXiv:1911.06987, 2019. · arXiv
  48. D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, "Mixmatch: A holistic approach to semi-supervised learning," in Advances in Neural Information Processing Systems, 2019, pp. 5049- 5059.
  49. H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," arXiv preprint arXiv:1710.09412, 2017. · arXiv
  50. S. Gidaris, A. Bursuc, N. Komodakis, P. Pérez, and M. Cord, "Learning representations by predicting bags of visual words," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6928-6938.
  51. S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, "Cutmix: Reg- ularization strategy to train strong classifiers with localizable features," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6023-6032.
  52. R. Hataya, J. Zdenek, K. Yoshizoe, and H. Nakayama, "Faster au- toaugment: Learning augmentation strategies using backpropagation," in European Conference on Computer Vision. Springer, 2020, pp. 1-16.
  53. G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," arXiv preprint arXiv:1503.02531, 2015. · arXiv
  54. J. Li, P. Zhou, C. Xiong, R. Socher, and S. C. Hoi, "Prototypical contrastive learning of unsupervised representations," arXiv preprint arXiv:2005.04966, 2020. · arXiv
  55. J. Donahue and K. Simonyan, "Large scale adversarial representation learning," in Advances in Neural Information Processing Systems, 2019, pp. 10 542-10 552.
  56. Y. M. Asano, C. Rupprecht, and A. Vedaldi, "Self-labelling via simultaneous clustering and representation learning," arXiv preprint arXiv:1911.05371, 2019. · arXiv
  57. S. Gidaris, P. Singh, and N. Komodakis, "Unsupervised repre- sentation learning by predicting image rotations," arXiv preprint arXiv:1803.07728, 2018. · arXiv
  58. L. Bottou, "Large-scale machine learning with stochastic gradient de- scent," in Proceedings of COMPSTAT'2010. Springer, 2010, pp. 177- 186.
  59. I. Loshchilov and F. Hutter, "Sgdr: Stochastic gradient descent with warm restarts," arXiv preprint arXiv:1608.03983, 2016. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Wang, X., & Qi, G.-J.
Affiliations
  • Purdue University West Lafayette (US)
  • Futurewei Technologies (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
45
Issue
5
Pages
1-12
ISSN
0162-8828 (ISSN-L)
DOI
10.48550/arxiv.2104.07713
arXiv
2104.07713
OpenAlex
W4294068600
Cited by
266
References
2
Citations by year
2026: 31 · 2025: 79 · 2024: 75 · 2023: 51 · 2022: 20 · 2021: 9
FWCI
18.52
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
contrastive learning, data augmentation, representation learning, distribution divergence, image retrieval, image classification, ImageNet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review