CoChat
Sign Up
PreprintCited by 185Domain Adaptation and Few-Shot LearningMachine Learning and AlgorithmsTopic Modeling

The DOI matches its DataCite record; 283 of the 313 references it cites were found in a registry or an index.

A Cookbook of Self-Supervised Learning

Open access
Balestriero, Randall, Ibrahim, Mark, Sobal, Vlad, Morcos, Ari, Shekhar, Shashank, Goldstein, Tom, Bordes, Florian, Bardes, Adrien, Mialon, Gregoire, Tian, Yuandong, Schwarzschild, Avi, Wilson, Andrew Gordon, Geiping, Jonas, Garrido, Quentin, Fernandez, Pierre, Bar, Amir, Pirsiavash, Hamed, LeCun, Yann, Goldblum, Micah·2023·arXiv
DOI 10.48550/arxiv.2304.12210arXiv 2304.12210
Open this paper in CoChat — free accountOpen PDF

Abstract

Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.12210

Cites

283 of 283 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Attention Is All You Need

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×

    “Since transformers [Vaswani et al., 2017] are employed in almost all domains of computer science, many works aim to reduce the compute and memory requirements of the attention mechanism.”
    Background

  • ImageNet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×

    “SSL methods for computer vision have been able to match or in some cases surpass models trained on labeled data, even on highly competitive benchmarks like ImageNet [Tomasev et al., 2022, He et al., 2020a, Deng et al., 2009].”
    Method

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×

    “Specifically, next token prediction [Radford et al., 2018, 2019, Brown et al., 2020] is akin to masking the last token in a string, while bidirectional encoders mask tokens anywhere in the string [Devlin et al., 2018] or fill larger spans of missing text [Raffel et al., 2020, Tay et al., 2022].”
    Background

  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472

  • Distributed Representations of Words and Phrases and their Compositionality

    Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 2×

    “depends on X, Ma and Collins [2018] considers conditional distribution X|Y , Dyer [2014] compares NCE and Negative Sampling [Mikolov et al., 2013] (the latter being a special case of the former) both extending Importance Sampling estimation [Bengio and Senécal, 2003] of the partition function (normalization factor)”
    Background

  • Language Models are Few-Shot Learners

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×

    “Specifically, next token prediction [Radford et al., 2018, 2019, Brown et al., 2020] is akin to masking the last token in a string, while bidirectional encoders mask tokens anywhere in the string [Devlin et al., 2018] or fill larger spans of missing text [Raffel et al., 2020, Tay et al., 2022].”
    Background

  • Auto-Encoding Variational Bayes

    Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · cited 1×

    “As SSL established a distinct paradigm from traditional reconstruction-based unsupervised learning methods such as (denoising, variational) Autoencoders [Vincent et al., 2008, 2010, Kingma and Welling, 2013], our vocabulary for understanding SSL in a unified view is limited.”
    Method

  • A fast learning algorithm for deep belief nets

    Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1×

    “An analogous approach from the time used Restricted Boltzman Machines (RBMs), which could be trained layer-wise and stacked to create deep belief nets [Hinton et al., 2006].”
    Method

  • Learning Transferable Visual Models From Natural Language Supervision

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 4×

    “…a result, image encoders arising from such a joint pre-training are especially robust to visual changes that leave semantic meaning unchanged, such as sketches of objects as evaluated in ImageNet-Sketch [Wang et al., 2019, Radford et al., 2021], and are strong on out-of-domain generalization tasks.”
    BackgroundInfluential

  • FaceNet: A unified embedding for face recognition and clustering

    Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · cited 3×

    “…examples from different classes should be larger than m. Similar to the contrastive loss, the Triplet loss [Weinberger and Saul, 2009, Chechik et al., 2010, Schroff et al., 2015] shares a similar spirit, but is composed of triplets: a query, a positive example, and a negative example (see eq.”
    BackgroundInfluential

Show all 313 references

Research works printed but not linked: 43

  1. K. K. Agrawal, A. K. Mondal, A. Ghosh, and B. A. Richards. α-req : Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=ii9X4vtZGTZ.
  2. G. Andrew, R. Arora, J. Bilmes, and K. Livescu. Deep canonical correlation analysis. In International conference on machine learning, pages 1247-1255. PMLR, 2013.
  3. S. P. Arunachalam, I. Güzey, S. Chintala, and L. Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. (arXiv:2210.06463), Oct 2022. URL http://arxiv.org/abs/ 2210.06463. arXiv:2210.06463 [cs]. · arXiv
  4. M. Assran, M. Caron, I. Misra, P. Bojanowski, F. Bordes, P. Vincent, A. Joulin, M. Rabbat, and N. Ballas. Masked siamese networks for label-efficient learning. In Computer Vision -ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI, page 456-473, Berlin, Heidelberg, 2022c. Springer-Verlag. ISBN 978-3-031-19820-5. doi: 10.1007/978-3-031-19821-2_26. URL https://doi.org/ 10.1007/978-3-031-19821-2_26. · DOI
  5. A. Bardes, J. Ponce, and Y. LeCun. Vicregl: Variance-invariance-covariance regularization for self-supervised learning. Advances in neural information processing systems, 2022. 22, 33, 44
  6. Y. Bengio and J.-S. Senécal. Quick training of probabilistic neural nets by importance sampling. In International Workshop on Artificial Intelligence and Statistics, pages 17-24. PMLR, 2003. 8, 9
  7. M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018. 6, 12, 44
  8. M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691- 1703. PMLR, 2020a.
  9. X. Chen and K. He. Exploring simple siamese representation learning. (arXiv:2011.10566), Nov 2020. URL http://arxiv.org/abs/2011.10566. arXiv:2011.10566 [cs]. · arXiv
  10. K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020. · arXiv
  11. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. · arXiv
  12. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ a4d92e2cd541fca87e4620aba658316d-Abstract.html.
  13. B. Eysenbach, T. Zhang, R. Salakhutdinov, and S. Levine. Contrastive learning as goal- conditioned reinforcement learning. (arXiv:2206.07568), Jun 2022. URL http://arxiv. org/abs/2206.07568. number: arXiv:2206.07568 arXiv:2206.07568 [cs]. · arXiv
  14. Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao. EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. arxiv:2211.07636[cs], Dec. 2022b. doi: 10.48550/arXiv.2211.07636. URL http://arxiv.org/abs/2211.07636. · DOI
  15. J. Goldberger, G. E. Hinton, S. Roweis, and R. R. Salakhutdinov. Neighbourhood compo- nents analysis. Advances in neural information processing systems, 17, 2004.
  16. I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio. Deep learning, volume 1. MIT Press, 2016.
  17. M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation princi- ple for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010. 9, 40
  18. B. He and M. Ozay. Exploring the gap between collapsed & whitened features in self- supervised learning. In International Conference on Machine Learning, pages 8613-8634. PMLR, 2022.
  19. K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
  20. K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735, 2020b. 18, 34, 43
  21. D.-H. Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896, 2013.
  22. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
  23. A. Mnih and K. Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. Advances in neural information processing systems, 26, 2013.
  24. R. Ni, M. Goldblum, A. Sharaf, K. Kong, and T. Goldstein. Data augmentation for meta- learning. In International Conference on Machine Learning, pages 8152-8161. PMLR, 2021a.
  25. R. Ni, M. Shu, H. Souri, M. Goldblum, and T. Goldstein. The close relationship be- tween contrastive learning and meta-learning. In International Conference on Learning Representations, 2021b. 7, 21
  26. A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 7, 10, 17, 18, 38, 41 · arXiv
  27. N. Parikh, S. Boyd, et al. Proximal algorithms. Foundations and trends® in Optimization, 1(3):127-239, 2014.
  28. A. Piché, J. Marino, G. M. Marconi, C. Pal, and M. E. Khan. Beyond target networks: Im- proving deep q-learning with functional regularization. arXiv preprint arXiv:2106.02613, 2021. · arXiv
  29. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  30. S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-maron, M. Giménez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A Generalist Agent. Transactions on Machine Learning Research, Nov. 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=1ikK0kHjvj.
  31. M. C. Schiappa, Y. S. Rawat, and M. Shah. Self-supervised learning for videos: A survey. ACM Computing Surveys, 2022b.
  32. K. Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016. 7, 10, 18
  33. Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, D. Bahri, T. Schuster, H. S. Zheng, N. Houlsby, and D. Metzler. Unifying Language Learning Paradigms. arxiv:2205.05131[cs], May 2022. URL http://arxiv.org/abs/2205.05131. 15, 39 · arXiv
  34. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/ paper/2019/hash/3eefceb8087e964f89c2d59e8a249915-Abstract.html.
  35. W. Wang, R. Arora, K. Livescu, and J. Bilmes. On deep multi-view representation learning. In International conference on machine learning, pages 1083-1092. PMLR, 2015. 6, 14
  36. X. Wang and A. Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2015.
  37. L. Xu, J. Neufeld, B. Larson, and D. Schuurmans. Maximum margin clustering. Advances in neural information processing systems, 17, 2004.
  38. D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Reinforcement learning with prototypical representations. arXiv:2102.11271 [cs], Jul 2021. URL http://arxiv.org/abs/2102. 11271. arXiv: 2102.11271. · arXiv
  39. A. YM., R. C., and V. A. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations, 2020. URL https: //openreview.net/forum?id=Hyx-jyBFPr.
  40. J. Yoon, Y. Zhang, J. Jordon, and M. van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:11033-11043, 2020.
  41. Curran Associates, Inc., 2014. URL https://proceedings.neurips.cc/paper/2014/file/ 375c71349b295fbe2dcdca9206f20a06-Paper.pdf.
  42. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In NeurIPS, 2014.
  43. J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022a. 12, 16, 22, 29, 30, 32, 33

Cited by

Sorted by citations

Metadata

Authors
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., Schwarzschild, A., Wilson, A. G., Geiping, J., Garrido, Q., Fernandez, P., Bar, A., Pirsiavash, H., LeCun, Y., & Goldblum, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2304.12210
arXiv
2304.12210
OpenAlex
W6960139023
Semantic Scholar
6bfafb32b423c3f0456a10984814f89046def489
Cited by
182
References
313
Citations by year
2026: 7 · 2025: 46 · 2024: 96 · 2023: 33
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion, other-oa)
  • arXiv (Cornell University)
  • Digital Repository at the University of Maryland (University of Maryland College Park)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
self-supervised learning, pretext tasks, hyperparameter optimization, research reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review