CoChat
Sign Up
PreprintCited by 10,857Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMachine Learning and Algorithms

A simple framework for contrastive learning of visual representations

Open access
Chen, Ting, Kornblith, Simon, Norouzi, Mohammad, Hinton, Geoffrey·2020·arXiv
DOI 10.48550/arxiv.2002.05709arXiv 2002.05709
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2002.05709

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Multi-task objectives may be more effective than single-task ones because they impose more task constraints on the representation, leading to a smaller and higher-quality solution space.

    “Such multi-task objectives may be more effective than single-task ones (e.g., ImageNet classification) due to the fact that they impose more task constraints on the representation, leading to a smaller and higher-quality solution space (Chen et al., 2020;He et al., 2020;Radford et al., 2017;2019).”

    From The Platonic Representation Hypothesis (2024) · cites this paper as Chen et al., 2020

    Section 3 (pp. 4–5) says that composing data augmentations defines broader contrastive prediction tasks and is crucial for learning good representations. This lends support to the idea that combining tasks can help, but the paper does not say that multi-task objectives outperform single-task objectives because additional constraints produce a smaller, higher-quality solution space. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Using a projection head improves greatly the accuracy of our method.

    “Similarly to other self-supervised frameworks, using a projection head [12] improves greatly the accuracy of our method.”

    From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [12]

  • The described contrastive learner setting is common for self-supervised contrastive learners with NCE objectives, including SimCLR and SimCSE.

    “We note that this is a common setting for self-supervised contrastive learners with NCE objectives (Gutmann & Hyvärinen, 2010;Oord et al., 2018), including SimCLR (Chen et al., 2020) and SimCSE (Gao et al., 2021).”

    From The Platonic Representation Hypothesis (2024) · cites this paper as Chen et al., 2020

  • The original SimCLR framework demonstrates significant performance differences among choices of augmentation functions.

    “Since the choice of augmentation function encodes the learning objective and invariance that the model should capture, the original SimCLR framework demonstrates significant performance differences among the choices of augmentation functions [28].”

    From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [28]

  • Contrastive learning has resulted in excellent representation learning performance across domains such as computer vision, natural language, speech recognition, and time-series data.

    “This framework has resulted in excellent representation learning performance across domains such as computer vision [12, 33, 35], natural language [49], speech recognition [6, 58] and time-series data [39].”

    From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [12]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. Asano, Y. M., Rupprecht, C., and Vedaldi, A. A critical analysis of self-supervision, or what we can learn from a single image. arXiv preprint arXiv:1904.13132 , 2019. · arXiv
  2. Bachman, P., Hjelm, R. D., and Buchwalter, W. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems , pp. 15509-15519, 2019.
  3. Becker, S. and Hinton, G. E. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature , 355 (6356):161-163, 1992.
  4. Berg, T., Liu, J., Lee, S. W., Alexander, M. L., Jacobs, D. W., and Belhumeur, P. N. Birdsnap: Large-scale fine-grained visual categorization of birds. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 2019-2026. IEEE, 2014.
  5. Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C. A. Mixmatch: A holistic approach to semisupervised learning. In Advances in Neural Information Processing Systems , pp. 5050-5060, 2019.
  6. Bossard, L., Guillaumin, M., and Van Gool, L. Food-101-mining discriminative components with random forests. In European conference on computer vision , pp. 446-461. Springer, 2014.
  7. Chen, T., Sun, Y., Shi, Y., and Hong, L. On sampling strategies for neural network-based collaborative filtering. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pp. 767-776, 2017.
  8. Chen, T., Zhai, X., Ritter, M., Lucic, M., and Houlsby, N. Selfsupervised gans via auxiliary rotation loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 12154-12163, 2019.
  9. Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 113-123, 2019.
  10. DeVries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017. · arXiv
  11. Doersch, C. and Zisserman, A. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision , pp. 2051-2060, 2017.
  12. Doersch, C., Gupta, A., and Efros, A. A. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision , pp. 1422-1430, 2015.
  13. Donahue, J. and Simonyan, K. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems , pp. 10541-10551, 2019.
  14. Dosovitskiy, A., Springenberg, J. T., Riedmiller, M., and Brox, T. Discriminative unsupervised feature learning with convolutional neural networks. In Advances in neural information processing systems , pp. 766-774, 2014.
  15. Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. The pascal visual object classes (voc) challenge. International Journal of Computer Vision , 88(2):303-338, 2010.
  16. Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop on Generative-Model Based Vision , 2004.
  17. Gidaris, S., Singh, P., and Komodakis, N. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 , 2018. · arXiv
  18. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., WardeFarley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in neural information processing systems , pp. 2672-2680, 2014.
  19. Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017. · arXiv
  20. Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) , volume 2, pp. 1735-1742. IEEE, 2006.
  21. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722 , 2019. · arXiv
  22. Hénaff, O. J., Razavi, A., Doersch, C., Eslami, S., and Oord, A. v. d. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272 , 2019. · arXiv
  23. Hinton, G. E., Osindero, S., and Teh, Y.-W. A fast learning algorithm for deep belief nets. Neural computation , 18(7):15271554, 2006.
  24. Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 , 2018. · arXiv
  25. Howard, A. G. Some improvements on deep convolutional neural network based image classification. arXiv preprint arXiv:1312.5402 , 2013. · arXiv
  26. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015. · arXiv
  27. Ji, X., Henriques, J. F., and Vedaldi, A. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference on Computer Vision , pp. 9865-9874, 2019.
  28. Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. · arXiv
  29. Kolesnikov, A., Zhai, X., and Beyer, L. Revisiting self-supervised visual representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pp. 1920-1929, 2019.
  30. Kornblith, S., Shlens, J., and Le, Q. V . Do better ImageNet models transfer better? In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 2661-2671, 2019.
  31. Krause, J., Deng, J., Stark, M., and Fei-Fei, L. Collecting a large-scale dataset of fine-grained cars. In Second Workshop on Fine-Grained Visual Categorization , 2013.
  32. Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016. · arXiv
  33. Maaten, L. v. d. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research , 9(Nov):2579-2605, 2008.
  34. Misra, I. and van der Maaten, L. Self-supervised learning of pretext-invariant representations. arXiv preprint arXiv:1912.01991 , 2019. · arXiv
  35. Noroozi, M. and Favaro, P. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision , pp. 69-84. Springer, 2016.
  36. Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 3498-3505. IEEE, 2012.
  37. Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. · arXiv
  38. Sohn, K. Improved deep metric learning with multi-class n-pair loss objective. In Advances in neural information processing systems , pp. 1857-1865, 2016.
  39. Sohn, K., Berthelot, D., Li, C.-L., Zhang, Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., and Raffel, C. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07685 , 2020. · arXiv
  40. Szegedy, C., Liu, W., Jia, Y ., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 1-9, 2015.
  41. Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. arXiv preprint arXiv:1906.05849 , 2019. · arXiv
  42. Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., and Lucic, M. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625 , 2019. · arXiv
  43. Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 3485-3492. IEEE, 2010.
  44. Xie, Q., Dai, Z., Hovy, E., Luong, M.-T., and Le, Q. V. Unsupervised data augmentation. arXiv preprint arXiv:1904.12848 , 2019. · arXiv
  45. Ye, M., Zhang, X., Yuen, P. C., and Chang, S.-F. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 6210-6219, 2019.
  46. You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 , 2017. · arXiv
  47. Zhai, X., Oliver, A., Kolesnikov, A., and Beyer, L. S4l: Selfsupervised semi-supervised learning. In The IEEE International Conference on Computer Vision (ICCV) , October 2019.
  48. Zhang, R., Isola, P., and Efros, A. A. Colorful image colorization. In European conference on computer vision , pp. 649-666. Springer, 2016.
  49. Zhuang, C., Zhai, A. L., and Yamins, D. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE International Conference on Computer Vision , pp. 6002-6012, 2019.

Cited by

Sorted by citations

Authors
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2002.05709
arXiv
2002.05709
OpenAlex
W3005680577
Cited by
10,857
References
11
Citations by year
2026: 80 · 2025: 1296 · 2024: 2446 · 2023: 2914 · 2022: 2085 · 2021: 1588 · 2020: 439 · 2019: 7 · 2018: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
contrastive learning, self-supervised learning, visual representation learning, SimCLR, batch size, semi-supervised learning, ImageNet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review