A Cookbook of Self-Supervised Learning
Open accessAbstract
Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SSL demonstrates remarkable potential for uncovering subtle, previously unknown features and removing human subjectivity from labeling.
“SSL demonstrates remarkable potential for uncovering subtle, previously unknown features within data, as well as for removing human subjectivity from the labeling process.”
From A Semi-Supervised Learning Method for the Identification of Bad Exposures in Large Imaging Surveys (2026) · cites this paper as Balestriero et al. 2023
The opening section says SSL learns generic representations from unlabeled data, including when a specific task is not known in advance. That supports learning useful features without labels, but the paper does not establish that those features are subtle or previously unknown, or that SSL removes human subjectivity from the labeling process. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“Since transformers [Vaswani et al., 2017] are employed in almost all domains of computer science, many works aim to reduce the compute and memory requirements of the attention mechanism.”
BackgroundImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1× · see citing works
“SSL methods for computer vision have been able to match or in some cases surpass models trained on labeled data, even on highly competitive benchmarks like ImageNet [Tomasev et al., 2022, He et al., 2020a, Deng et al., 2009].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 3× · see citing works
“Specifically, next token prediction [Radford et al., 2018, 2019, Brown et al., 2020] is akin to masking the last token in a string, while bidirectional encoders mask tokens anywhere in the string [Devlin et al., 2018] or fill larger spans of missing text [Raffel et al., 2020, Tay et al., 2022].”
BackgroundGenerative Adversarial Networks
Cohen, G., & Giryes, R. (2021) · International Conference on Computing Communication and Networking Technologies · doi:10.1109/icccnt56998.2023.10306417 · Cited by 28,188 · cited 1× · see citing works
“Generative Adversarial Networks (GANs) [Goodfellow et al., 2014] consist of an image generator and a discriminator that differentiates real images from generated images.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 2× · see citing works
“depends on X, Ma and Collins [2018] considers conditional distribution X|Y , Dyer [2014] compares NCE and Negative Sampling [Mikolov et al., 2013] (the latter being a special case of the former) both extending Importance Sampling estimation [Bengio and Senécal, 2003] of the partition function (normalization factor)”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3× · see citing works
“Specifically, next token prediction [Radford et al., 2018, 2019, Brown et al., 2020] is akin to masking the last token in a string, while bidirectional encoders mask tokens anywhere in the string [Devlin et al., 2018] or fill larger spans of missing text [Raffel et al., 2020, Tay et al., 2022].”
BackgroundAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · cited 1× · see citing works
“As SSL established a distinct paradigm from traditional reconstruction-based unsupervised learning methods such as (denoising, variational) Autoencoders [Vincent et al., 2008, 2010, Kingma and Welling, 2013], our vocabulary for understanding SSL in a unified view is limited.”
MethodA fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · cited 1× · see citing works
“An analogous approach from the time used Restricted Boltzman Machines (RBMs), which could be trained layer-wise and stacked to create deep belief nets [Hinton et al., 2006].”
MethodLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 4× · see citing works
“…a result, image encoders arising from such a joint pre-training are especially robust to visual changes that leave semantic meaning unchanged, such as sketches of objects as evaluated in ImageNet-Sketch [Wang et al., 2019, Radford et al., 2021], and are strong on out-of-domain generalization tasks.”
BackgroundInfluentialFaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · cited 3× · see citing works
“…examples from different classes should be larger than m. Similar to the contrastive loss, the Triplet loss [Weinberger and Saul, 2009, Chechik et al., 2010, Schroff et al., 2015] shares a similar spirit, but is composed of triplets: a query, a positive example, and a negative example (see eq.”
BackgroundInfluentialDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1× · see citing works
“For example, self-distillation [Hinton et al., 2015, Furlanello et al., 2018], Deep Q Network in reinforcement learning [Mnih et al., 2013], Mean Teacher in semisupervised learning [Tarvainen and Valpola, 2017], and even model average in supervised and generative modeling [Jean et al., 2014].”
BackgroundMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · cited 5× · see citing works
“SSL methods for computer vision have been able to match or in some cases surpass models trained on labeled data, even on highly competitive benchmarks like ImageNet [Tomasev et al., 2022, He et al., 2020a, Deng et al., 2009].”
BackgroundMethodInfluentialExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 3× · see citing works
“Specifically, next token prediction [Radford et al., 2018, 2019, Brown et al., 2020] is akin to masking the last token in a string, while bidirectional encoders mask tokens anywhere in the string [Devlin et al., 2018] or fill larger spans of missing text [Raffel et al., 2020, Tay et al., 2022].”
BackgroundMethodA simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · cited 10× · see citing works
“This key element of data-augmentation had also been largely used in the following SSL works [Chen et al., 2020d, Bardes et al., 2021, Zbontar et al., 2021] without significant changes.”
BackgroundMethodInfluentialPlaying Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · cited 2× · see citing works
“, 2018], Deep Q Network in reinforcement learning [Mnih et al., 2013], Mean Teacher in semisupervised learning [Tarvainen and Valpola, 2017], and even model average in supervised and generative modeling [Jean et al.”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 1× · see citing works
“Additionally, some methods for both contrastive and reconstructive pretraining corrupt the input with several other augmentations including document rotation, sentence permutation, and token deletion [Lewis et al., 2020, Raffel et al., 2020, Wu et al., 2018].”
MethodMasked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 18× · see citing works
“This idea evolved into masked auto-encoding methods [He et al., 2022], in which the masked region is a union of image patches that can be predicted using a transformer.”
BackgroundMethodInfluentialExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · cited 5× · see citing works
“Later advancements improved on the representation learning ability of auto-encoders, including denoising autoencoders [Vincent et al., 2008], cross-channel prediction [Zhang et al., 2017], and deep canonically correlated autoencoders [Wang et al., 2015].”
BackgroundMethodInfluentialRelations Between Two Sets of Variates
Hotelling, H. (1936) · Springer series in statistics · doi:10.1007/978-1-4612-4380-9_14 · Cited by 7,029 · cited 9× · see citing works
“These ideas were extended to deep learning in Deep Canonically Correlated Autoencoders (DCCAE) an autoencoder regularized via CCA.”
BackgroundMethodInfluentialImproved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965 · cited 1× · see citing works
“Early GANs papers [Salimans et al., 2016] experimented with downstream image classification using GAN components.”
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061 · see citing works
mixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831 · cited 3× · see citing works
“Borrowing from vision systems, a combination of CutMix [Yun et al., 2019] in input space and mixup [Zhang et al., 2018] in embedding space is also an effective augmentation for tabular data [Somepalli et al., 2021].”
MethodContext Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · cited 3× · see citing works
“Context encoders instead mask out large portions of an image and replace their pixel values with white, teaching an autoencoder to inpaint the white patches [Pathak et al., 2016].”
BackgroundMethodEmerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 16× · see citing works
“, 2020], SimSIAM [Chen and He, 2021], DINO [Caron et al., 2021], along with their variants rely on a simple mechanism: feeding two different views to two encoders, and mapping one to the other by means of a predictor.”
BackgroundMethodResultInfluentialCutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., & Choe, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00612 · Cited by 5,400 · cited 2× · see citing works
“Borrowing from vision systems, a combination of CutMix [Yun et al., 2019] in input space and mixup [Zhang et al., 2018] in embedding space is also an effective augmentation for tabular data [Somepalli et al., 2021].”
MethodDimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336 · cited 1× · see citing works
“Contrastive loss was first introduced in [Bromley et al., 1993] then more formally defined in [Chopra et al., 2005, Hadsell et al., 2006].”
BackgroundSelf-citationWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 1× · see citing works
“For example convolutions can be useful [Oord et al., 2016, Schneider et al., 2019, Baevski et al., 2021].”
BackgroundGreedy layer-wise training of deep networks
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0024 · Cited by 4,841 · cited 1× · see citing works
“Generative models: An early influential SSL method is greedy layer-wise pretraining [Bengio et al., 2006], in which layers of a deep network are trained one-at-a-time using an autoencoder loss.”
MethodDistance Metric Learning for Large Margin Nearest Neighbor Classification
Weinberger, K. Q., & Saul, L. K. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1577078 · Cited by 4,651 · cited 1× · see citing works
“…m, imposing the distance between examples from different classes should be larger than m. Similar to the contrastive loss, the Triplet loss [Weinberger and Saul, 2009, Chechik et al., 2010, Schroff et al., 2015] shares a similar spirit, but is composed of triplets: a query, a positive…”
BackgroundTarvainen, A., & Valpola, H. (2017) · arXiv · doi:10.48550/arxiv.1703.01780 · Cited by 4,275 · cited 2× · see citing works
“, 2013], Mean Teacher in semisupervised learning [Tarvainen and Valpola, 2017], and even model average in supervised and generative modeling [Jean et al.”
Background(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051 · cited 1× · see citing works
“As SSL established a distinct paradigm from traditional reconstruction-based unsupervised learning methods such as (denoising, variational) Autoencoders [Vincent et al., 2008, 2010, Kingma and Welling, 2013], our vocabulary for understanding SSL in a unified view is limited.”
MethodLearning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036 · cited 1× · see citing works
“Contrastive loss was first introduced in [Bromley et al., 1993] then more formally defined in [Chopra et al., 2005, Hadsell et al., 2006].”
BackgroundSelf-citationBootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 10× · see citing works
“Self-distillation methods such as BYOL [Grill et al., 2020], SimSIAM [Chen and He, 2021], DINO [Caron et al.”
BackgroundMethodInfluentialZhang, R., Isola, P., & Efros, A. A. (2016) · European Conference on Computer Vision · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,803 · cited 3× · see citing works
“A number of prominent early self-supervised pre-training algorithms for computer vision applied degradations to training images, such as decolorization [Zhang et al., 2016], noise [Vincent et al.”
BackgroundMethodMixMatch: A Holistic Approach to Semi-Supervised LearningPreprint
Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., & Raffel, C. A. (2019) · Neural Information Processing Systems · doi:10.48550/arxiv.1905.02249 · Cited by 3,744 · cited 1× · see citing works
“Important early methods in this category include MixMatch [Berthelot et al., 2019], which chooses pseudolabels by averaging outputs of a network on several different random augmentations of the training images, resulting in labels that are augmentation invariant.”
MethodExploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01549 · Cited by 3,725 · cited 12× · see citing works
“, 2020], SimSIAM [Chen and He, 2021], DINO [Caron et al.”
MethodResultInfluentialUnsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664 · cited 1× · see citing works
“Additionally, some methods for both contrastive and reconstructive pretraining corrupt the input with several other augmentations including document rotation, sentence permutation, and token deletion [Lewis et al., 2020, Raffel et al., 2020, Wu et al., 2018].”
Methodfairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · cited 2× · see citing works
“These speed-ups are available in open-source libraries: Fairseq [Ott et al., 2019], FairScale [FairScale, 2021], XFormers [Lefaudeux et al., 2022], Apex [NVidia, 2021], etc.”
MethodDepth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442 · cited 2× · see citing works
“For specialized applications like depth mapping, self-supervised methods have been proposed that learn monocular depth models from unlabeled image pairs [Eigen et al., 2014] and later the frames from a single-camera video [Zhou et al., 2017].”
MethodUnsupervised Learning of Depth and Ego-Motion from Video
Zhou, T., Brown, M., Snavely, N., & Lowe, D. G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.700 · Cited by 2,910 · cited 2× · see citing works
“For specialized applications like depth mapping, self-supervised methods have been proposed that learn monocular depth models from unlabeled image pairs [Eigen et al., 2014] and later the frames from a single-camera video [Zhou et al., 2017].”
BackgroundMethodVirtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
Miyato, T., Maeda, S., Koyama, M., & Ishii, S. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858821 · Cited by 2,860 · see citing works
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851 · cited 1× · see citing works
“They can also be generated using dropout so that one input has two different latent representations [Gao et al., 2021].”
BackgroundUnsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · cited 2× · see citing works
“, 2008], or shuffling image patches [Noroozi and Favaro, 2016], and taught models to undo these degradations.”
BackgroundViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767 · cited 1× · see citing works
“The frame features can also be used directly, by appending a linear layer on top of the features [Radford et al., 2019], or by using more complex heads [Ni et al., 2022, Arnab et al., 2021].”
MethodUnsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
Improved Baselines with Momentum Contrastive LearningPreprint
Chen, X., Fan, H., Girshick, R., & He, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04297 · Cited by 2,579 · cited 10× · see citing works
“This key element of data-augmentation had also been largely used in the following SSL works [Chen et al., 2020d, Bardes et al., 2021, Zbontar et al., 2021] without significant changes.”
BackgroundMethodInfluentialUnsupervised Learning of Visual Features by Contrasting Cluster Assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09882 · Cited by 2,499 · cited 11× · see citing works
“For instance, Feichtenhofer et al. [2021] have generalized SimCLR, MoCo, SwAV and BYOL to space-time video data.”
BackgroundMethodInfluentialUnsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477 · cited 2× · see citing works
“Another line of work tries to retain style information about the augmentations to improve downstream performance on tasks requiring style information such as color by predicting style information [Xiao et al., 2020, Dangovski et al., 2021, Gidaris et al., 2018, Scherr et al., 2022].”
BackgroundAccurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · cited 1× · see citing works
“A learning rate warmup period of 10k iterations [Goyal et al., 2017, Dosovitskiy et al.] also improves training stability.”
BackgroundDeep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104 · cited 2× · see citing works
“Stochastic depth [Huang et al., 2016] originated from NLP and was subsequently used in vision models [Touvron et al.”
MethodSignature Verification Using A "Siamese" Time Delay Neural Network
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1993) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001493000339 · Cited by 2,077 · cited 2× · see citing works
“Contrastive loss was first introduced in [Bromley et al., 1993] then more formally defined in [Chopra et al., 2005, Hadsell et al., 2006].”
BackgroundSelf-citationTian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997 · see citing works
Self-citationLearning deep representations by mutual information estimation and maximization
Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., & Bengio, Y. (2018) · arXiv · doi:10.48550/arxiv.1808.06670 · Cited by 1,991 · see citing works
Adversarial Feature LearningPreprint
Donahue, J., Krähenbühl, P., & Darrell, T. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1605.09782 · Cited by 1,982 · cited 2× · see citing works
“, 2017], or learn additional mappings from image to latent space [Donahue et al., 2017] to improve transfer learning.”
BackgroundScaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · cited 2× · see citing works
“, 2021], and ALIGN [Jia et al., 2021] are self-supervised learning approaches that use image-caption pairs to learn a joint embedding space for images and captions.”
BackgroundConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders
Woo, S., Debnath, S., Hu, R., Chen, X., Liu, Z., Kweon, I. S., & Xie, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01548 · Cited by 1,791 · cited 4× · see citing works
“More recently, masked autoencoding pre-training strategies have been adapted for convolutional architectures to great effect, where they achieve competitive performance on downstream object detection and instance segmentation [Woo et al., 2023].”
BackgroundMethodInfluentialLi, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741 · cited 3× · see citing works
“…have been introduced such as the entropy of the singular value distribution [Garrido et al., 2022a], the classical rank estimator [Jing et al., 2022], fitting a power law to the singular value distribution [Ghosh et al., 2022] or the AUC of the singular value distribution [Li et al., 2022a].”
BackgroundEstimating Optimal Transformations for Multiple Regression and Correlation
Breiman, L., & Friedman, J. H. (1985) · Journal of the American Statistical Association · doi:10.1080/01621459.1985.10478157 · Cited by 1,713 · see citing works
Deep Metric Learning via Lifted Structured Feature Embedding
Song, H. O., Yu, X., Jegelka, S., & Savarese, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.434 · Cited by 1,687 · see citing works
An Empirical Study of Training Self-Supervised Vision Transformers
Chen, X., Xie, S., & He, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00950 · Cited by 1,648 · cited 3× · see citing works
“One approach is with low-rank and/or sparse approximation mechanisms [Kitaev et al., 2020, Choromanski et al., 2020, Wang et al., 2020a, Chen et al., 2021a, Zaheer et al., 2020].”
BackgroundMethodBig Self-Supervised Models are Strong Semi-Supervised LearnersPreprint
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., & Hinton, G. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10029 · Cited by 1,480 · cited 10× · see citing works
“This key element of data-augmentation had also been largely used in the following SSL works [Chen et al., 2020d, Bardes et al., 2021, Zbontar et al., 2021] without significant changes.”
BackgroundMethodInfluentialThe iNaturalist Species Classification and Detection Dataset
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., & Belongie, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00914 · Cited by 1,451 · cited 2× · see citing works
“, 2014] and iNaturalist18 [Horn et al., 2018] without changing the augmentations strategy but tuning heavily loss related coefficients.”
BackgroundMethodReformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · cited 1× · see citing works
“One approach is with low-rank and/or sparse approximation mechanisms [Kitaev et al., 2020, Choromanski et al., 2020, Wang et al., 2020a, Chen et al., 2021a, Zaheer et al., 2020].”
BackgroundSelf-Supervised Learning of Pretext-Invariant Representations
Misra, I., & van der Maaten, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00674 · Cited by 1,435 · cited 4× · see citing works
“SSL approaches which rely on augmented views or jigsaw transformations, such as MoCo [He et al., 2020b] and PIRL [Misra and Maaten, 2020], learn occlusion invariance since they are trained with random crops on ImageNet where foreground objects are often large so that different crops contain different parts of the same…”
BackgroundMethodInfluentialBEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392 · cited 3× · see citing works
“In SSL vision models, LayerDecay increases performance when fine-tuning on downstream tasks [Bao et al., 2021b, Zhou et al., 2022a, He et al., 2022].”
BackgroundRecent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · see citing works
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357 · cited 1× · see citing works
“These modern SSL models are capable of representing both vision and text, and can be used in a number of applications that are multimodal, from visual-question answering to multimodal generation [Alayrac et al., 2022, Li et al., 2022c, Nichol et al., 2022, Rao et al., 2022].”
BackgroundBig Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · cited 1× · see citing works
“One approach is with low-rank and/or sparse approximation mechanisms [Kitaev et al., 2020, Choromanski et al., 2020, Wang et al., 2020a, Chen et al., 2021a, Zaheer et al., 2020].”
BackgroundSimMIM: a Simple Framework for Masked Image Modeling
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., & Hu, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00943 · Cited by 1,348 · cited 2× · see citing works
“[Xie et al., 2022] • Muse reaches state-of-the-art text conditional image generation with a masked transformer approach [Chang et al.”
BackgroundMethodBarlow Twins: Self-Supervised Learning via Redundancy ReductionPreprint
Žbontar, J., Jing, L., Misra, I., LeCun, Y., & Deny, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03230 · Cited by 1,329 · cited 11× · see citing works
“This key element of data-augmentation had also been largely used in the following SSL works [Chen et al., 2020d, Bardes et al., 2021, Zbontar et al., 2021] without significant changes.”
BackgroundMethodInfluentialSelf-citationGoing deeper with Image Transformers
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jegou, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00010 · Cited by 1,259 · cited 4× · see citing works
“The per-layer drop-rate may depend linearly on the layer depth or uniformly as suggested in recent works [Touvron et al., 2021b].”
BackgroundMethodResultInfluentialFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · cited 2× · see citing works
“Another approach, is to resort to IO-aware optimizations [Ivanov et al., 2021], the most known one perhaps being FlashAttention [Dao et al., 2022].”
BackgroundMastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · Nature · doi:10.1038/s41586-020-03051-4 · Cited by 1,214 · cited 2× · see citing works
“The best-performing method doing this is EfficientZero [Ye et al., 2021], which modifies MuZero [Schrittwieser et al., 2020] by, among other modifications, adding the SimSiam [Chen and He, 2020] objective to train the encoder and the forward model, and sets the new state of the art on Atari100k.”
BackgroundMethodDiscriminative Unsupervised Feature Learning with Convolutional Neural NetworksPreprint
Dosovitskiy, A., Springenberg, J., Riedmiller, M. A., & Brox, T. (2014) · Neural Information Processing Systems · doi:10.48550/arxiv.1406.6909 · Cited by 1,080 · cited 2× · see citing works
“, 2020a], which extends the result of [Dosovitskiy et al., 2014].”
BackgroundDINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078 · cited 7× · see citing works
“, 2022a] and DINOV2 [Oquab et al., 2023] employ a mix of masked image modeling and more classical approaches such as self-distillation.”
BackgroundMethodResultInfluentialSelf-citationLAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022 · cited 2× · see citing works
“Recent work has pushed these vision-language systems to larger scales [Ding et al., 2021, Yuan et al., 2021, Singh et al., 2022, Wang et al., 2022d, Fang et al., 2022b], based on freely available image-caption pairs collected from the internet, such as in [Schuhmann et al., 2022].”
BackgroundData-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S. M. A., & Oord, A. V. D. (2019) · arXiv · doi:10.48550/arxiv.1905.09272 · Cited by 1,014 · cited 1× · see citing works
“CPC was extended to images in [Henaff, 2020].”
MethodLarge Batch Training of Convolutional NetworksPreprint
You, Y., Gitman, I., & Ginsburg, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.03888 · Cited by 992 · cited 1× · see citing works
“For ImageNet pretraining, VICREg, Barlow Twins, BYOL, and SimCLR use a base learning rate of 0.2 − 0.3 with the LARS optimizer [You et al., 2017].”
MethodInfluentialFlamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · cited 1× · see citing works
“These modern SSL models are capable of representing both vision and text, and can be used in a number of applications that are multimodal, from visual-question answering to multimodal generation [Alayrac et al., 2022, Li et al., 2022c, Nichol et al., 2022, Rao et al., 2022].”
BackgroundLearning Representations for Automatic Colorization
Larsson, G., Maire, M., & Shakhnarovich, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_35 · Cited by 946 · cited 1× · see citing works
“Colorization-based SSL methods convert an image to grayscale, and then train a network to predict the original RGB values [Zhang et al., 2016, Larsson et al., 2016, Vondrick et al., 2018].”
MethodWang, T., & Isola, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10242 · Cited by 921 · cited 1× · see citing works
“Initially, InfoNCE was suggested as a variational approximation to the mutual information between two views [Aitchison and Ganev, 2023, Wang and Isola, 2020, Oord et al., 2018].”
MethodLarge Scale Online Learning of Image Similarity Through Ranking
Chechik, G., Sharma, V., Shalit, U., et al. (2009) · Journal of machine learning research · doi:10.5555/1756006.1756042 · Cited by 916 · cited 1×
“…between examples from different classes should be larger than m. Similar to the contrastive loss, the Triplet loss [Weinberger and Saul, 2009, Chechik et al., 2010, Schroff et al., 2015] shares a similar spirit, but is composed of triplets: a query, a positive example, and a negative…”
BackgroundVideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.12602 · Cited by 909 · cited 1× · see citing works
“More recently, masked auto-encoding objectives for video have been built around the same idea as images, but by masking patches/ tubes of patches in the temporal axis as well [Feichtenhofer et al., 2022, Tong et al., 2022, Girdhar et al., 2022].”
BackgroundZhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · cited 2× · see citing works
“When it is not explicitly needed by the method, using the average of the patch tokens instead of the class token saves memory without much change on the accuracies of the network [Zhai et al., 2022a].”
BackgroundMethodVariational Autoencoders and Nonlinear ICA: A Unifying FrameworkPreprint
Khemakhem, I., Kingma, D. P., Monti, R. P., et al. (2019) · International Conference on Artificial Intelligence and Statistics · doi:10.48550/arxiv.1907.04809 · Cited by 874 · cited 3× · see citing works
“…that InfoNCE balances alignment of “positive” examples and uniformity of the overall feature representation [Wang and Isola, 2022], or that (under strong assumptions) it can identify the latent structure in a hypothesized data-generating process, akin to nonlinear ICA [Khemakhem et al., 2020].”
BackgroundLinformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · cited 3× · see citing works
“Similarly, while masking images is often done with random pixels, the two dimensions of a spectrogram represent time and frequency and masking with horizontal and/or vertical bands is more effective [Wang et al., 2020b].”
BackgroundOn Using Very Large Target Vocabulary for Neural Machine Translation
Jean, S., Cho, K., Memisevic, R., & Bengio, Y. (2015) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p15-1001 · Cited by 839 · cited 2× · see citing works
“, 2013], Mean Teacher in semisupervised learning [Tarvainen and Valpola, 2017], and even model average in supervised and generative modeling [Jean et al., 2014].”
BackgroundBorn Again Neural NetworksPreprint
Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L., & Anandkumar, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04770 · Cited by 798 · cited 1× · see citing works
“For example, self-distillation [Hinton et al., 2015, Furlanello et al., 2018], Deep Q Network in reinforcement learning [Mnih et al., 2013], Mean Teacher in semisupervised learning [Tarvainen and Valpola, 2017], and even model average in supervised and generative modeling [Jean et al., 2014].”
BackgroundDiscriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
Dosovitskiy, A., Fischer, P., Springenberg, J. T., Riedmiller, M., & Brox, T. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2496141 · Cited by 762 · see citing works
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., & Carion, N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00180 · Cited by 761 · cited 1× · see citing works
“These features are also leveraged by recent works for open-vocabulary object detection [Kamath et al., 2021, Gu et al., 2021, Zareian et al., 2021, Minderer et al., 2022].”
BackgroundSelf-citationBerton, A., Kaltenmeier, A., Haiber, U., et al. (1990) · Encyclopedia of Biometrics · doi:10.1002/j.2637-496x.1990.tb05934.x · Cited by 734
Touvron, H., Cord, M., & Jégou, H. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2204.07118 · Cited by 733
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
Zhang, R., Isola, P., & Efros, A. A. (2016) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2017.76 · Cited by 724 · cited 2× · see citing works
“, 2008], cross-channel prediction [Zhang et al., 2017], and deep canonically correlated autoencoders [Wang et al.”
BackgroundRethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1× · see citing works
“One approach is with low-rank and/or sparse approximation mechanisms [Kitaev et al., 2020, Choromanski et al., 2020, Wang et al., 2020a, Chen et al., 2021a, Zaheer et al., 2020].”
BackgroundDenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
Rao, Y., Zhao, W., Chen, G., Tang, Y., Zhu, Z., Huang, G., Zhou, J., & Lu, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01755 · Cited by 673 · cited 1× · see citing works
“These modern SSL models are capable of representing both vision and text, and can be used in a number of applications that are multimodal, from visual-question answering to multimodal generation [Alayrac et al., 2022, Li et al., 2022c, Nichol et al., 2022, Rao et al., 2022].”
BackgroundDense Contrastive Learning for Self-Supervised Visual Pre-Training
Wang, X., Zhang, R., Shen, C., Kong, T., & Li, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00304 · Cited by 660 · cited 2× · see citing works
“It is worth pointing out that some works have only noticed minor increases in performance [Wang et al., 2021a] where it only lead to a 0.3 point performance increase.”
MethodResultInfluentialScaleNet: An Unsupervised Representation Learning Method for Limited Information
Huang, H.-L., & Roozbahani, M. M. (2023) · German Conference on Pattern Recognition · doi:10.1007/978-3-030-92659-5_11 · Cited by 651 · cited 8× · see citing works
“Deep clustering alternates between assigning labels to images by performing k-means in the feature space, and updating the model to respect these assigned class labels [Caron et al., 2018].”
BackgroundMethodInfluentialAgrawal, P., Carreira, J., & Malik, J. (2015) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2015.13 · Cited by 639 · see citing works
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
Hendrycks, D., Mazeika, M., Kadavath, S., & Song, D. (2019) · arXiv · doi:10.48550/arxiv.1906.12340 · Cited by 634 · cited 1× · see citing works
“There’s also evidence SSL models can learn representations that are more robust to adversarial examples, label corruption, and input perturbations—and are more fair—compared to their supervised counterparts [Hendrycks et al., 2019, Goyal et al., 2022].”
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630 · see citing works
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised LearningPreprint
Bardes, A., Ponce, J., & LeCun, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04906 · Cited by 617 · cited 9× · see citing works
“From these origins, stems SSL methods such as VICReg [Bardes et al., 2021], Barlow Twins [Zbontar et al.”
MethodInfluentialSelf-citationPrototypical Contrastive Learning of Unsupervised Representations
Li, J., Zhou, P., Xiong, C., & Hoi, S. C. H. (2020) · arXiv · doi:10.48550/arxiv.2005.04966 · Cited by 611 · see citing works
Self-supervised learning in medicine and healthcare
Krishnan, R., Rajpurkar, P., & Topol, E. J. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00914-1 · Cited by 605 · cited 1× · see citing works
“SSL can be especially useful in domains such as medicine where labels are costly or the specific task can not be known a priori [Krishnan et al., 2022, Ciga et al., 2022].”
BackgroundFLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01519 · Cited by 597 · cited 1× · see citing works
“Recent work has pushed these vision-language systems to larger scales [Ding et al., 2021, Yuan et al., 2021, Singh et al., 2022, Wang et al., 2022d, Fang et al., 2022b], based on freely available image-caption pairs collected from the internet, such as in [Schuhmann et al., 2022].”
BackgroundVIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-TrainingPreprint
Ma, Y., Sodhani, S., Jayaraman, D., Bastani, O., Kumar, V., & Zhang, A. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.00030 · Cited by 591 · see citing works
Open-vocabulary Object Detection via Vision and Language Knowledge DistillationPreprint
Gu, X., Lin, T.-Y., Kuo, W., & Cui, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13921 · Cited by 581 · cited 1× · see citing works
“These features are also leveraged by recent works for open-vocabulary object detection [Kamath et al., 2021, Gu et al., 2021, Zareian et al., 2021, Minderer et al., 2022].”
BackgroundTime-Contrastive Networks: Self-Supervised Learning from Video
Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Stefan Schaal, A., Levine, S., & Brain, G. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8462891 · Cited by 579 · cited 1× · see citing works
“Multi-modal models, on the other hand, can consider a soundbite and its text [Sermanet et al., 2018, Chung et al., 2018] or some frames of a video and the corresponding sound clip [Zhao et al., 2018, Alwassel et al., 2020] as different views to be used as positive pairs for contrastive learning.”
BackgroundUnsupervised and Semi-supervised Learning with Categorical Generative Adversarial NetworksPreprint
Springenberg, J. T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06390 · Cited by 578 · cited 2× · see citing works
“Specialized feature learning routines have also been developed that modify the discriminator [Springenberg, 2015], add a generator [Dai et al.”
BackgroundMINE: Mutual Information Neural EstimationPreprint
Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A. C., & Hjelm, R. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04062 · Cited by 575 · cited 1× · see citing works
“, 2018] τ log( + x) ex/τ MINE [Belghazi et al., 2018] log(x) ex Triplet [Schroff et al.”
Expanding Language-Image Pretrained Models for General Video Recognition
Ni, B., Peng, H., Chen, M., et al. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2208.02816 · Cited by 574 · cited 1×
“The frame features can also be used directly, by appending a linear layer on top of the features [Radford et al., 2019], or by using more complex heads [Ni et al., 2022, Arnab et al., 2021].”
MethodLearning Features by Watching Objects Move
Pathak, D., Girshick, R. B., Dollár, P., Darrell, T., & Hariharan, B. (2016) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2017.638 · Cited by 549 · see citing works
UP-DETR: Unsupervised Pre-training for Object Detection with Transformers
Dai, Z., Cai, B., Lin, Y., & Chen, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00165 · Cited by 547 · cited 3× · see citing works
“Recently, UP-DETR [Dai et al., 2021] and DETReg [Bar et al.”
BackgroundEVA: Exploring the Limits of Masked Visual Representation Learning at Scale
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., & Cao, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01855 · Cited by 540 · cited 3× · see citing works
“Recent work has pushed these vision-language systems to larger scales [Ding et al., 2021, Yuan et al., 2021, Singh et al., 2022, Wang et al., 2022d, Fang et al., 2022b], based on freely available image-caption pairs collected from the internet, such as in [Schuhmann et al., 2022].”
BackgroundMethodCURL: Contrastive Unsupervised Representations for Reinforcement LearningPreprint
Srinivas, A., Laskin, M., & Abbeel, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.04136 · Cited by 521 · see citing works
SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionPreprint
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 517 · cited 3× · see citing works
“MLM, along with span-infilling techniques, remains popular as a SSL objective for large language models [Raffel et al., 2020, Wang et al., 2022a, Tay et al., 2022].”
BackgroundMethodGood Semi-supervised Learning That Requires a Bad GANPreprint
Dai, Z.-H., Yang, Z.-L., Yang, F., Cohen, W. W., & Salakhutdinov, R. (2017) · Neural Information Processing Systems · doi:10.48550/arxiv.1705.09783 · Cited by 513 · cited 2× · see citing works
“Specialized feature learning routines have also been developed that modify the discriminator [Springenberg, 2015], add a generator [Dai et al., 2017], or learn additional mappings from image to latent space [Donahue et al., 2017] to improve transfer learning.”
BackgroundContrastive Learning with Hard Negative SamplesPreprint
Robinson, J., Chuang, C.-Y., Sra, S., & Jegelka, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.04592 · Cited by 504 · cited 1× · see citing works
“Recently, some works have focused on putting more weight on hard samples [Robinson et al., 2020].”
BackgroundBEiT v2: Masked Image Modeling with Vector-Quantized Visual TokenizersPreprint
Peng, Z., Dong, L., Bao, H., Ye, Q.-X., & Wei, F. (2022) · arXiv.org · doi:10.48550/arxiv.2208.06366 · Cited by 468 · cited 1× · see citing works
“Other works [Bao et al., 2021b, Peng et al., 2022, Wang et al., 2022b] first fine-tune on ImageNet-21k for 60 epochs, and further fine-tune on ImageNet-1k, which represents between 1/5 to 2 times the cost of the pre-training phase.”
BackgroundModel-Based Reinforcement Learning for Atari
Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1903.00374 · Cited by 467 · cited 1× · see citing works
“Recently, SSL has been used to improve sample efficiency on a challenging Atari100k benchmark [Kaiser et al., 2020].”
MethodSelf-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., & Ballas, N. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01499 · Cited by 456 · see citing works
Self-citationMasked Siamese Networks for Label-Efficient Learning
Assran, M., Caron, M., Misra, I., et al. (2022) · European Conference on Computer Vision · doi:10.48550/arxiv.2204.07141 · Cited by 456 · cited 3×
“To this effect some works have explored pretraining on large uncurated datasets [Goyal et al., 2021], or on datasets that different from ImageNet such as COCO [El-Nouby et al., 2021], or iNaturalist [Assran et al., 2022a].”
BackgroundMethodSelf-citationCogView: Mastering Text-to-Image Generation via Transformers
Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.13290 · Cited by 450 · cited 1× · see citing works
“Recent work has pushed these vision-language systems to larger scales [Ding et al., 2021, Yuan et al., 2021, Singh et al., 2022, Wang et al., 2022d, Fang et al., 2022b], based on freely available image-caption pairs collected from the internet, such as in [Schuhmann et al., 2022].”
BackgroundSELF-LABELLING VIA SIMULTANEOUS CLUSTERING AND REPRESENTATION LEARNINGPreprint
Asano, Y. M., Rupprecht, C., & Vedaldi, A. (2019) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.1911.05371 · Cited by 450 · see citing works
MaskGIT: Masked Generative Image Transformer
Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01103 · Cited by 446 · cited 2× · see citing works
“However, masked image models can also achieve strong generative modeling [Chang et al., 2022], including text-conditional generation [Chang et al., 2023].”
BackgroundOpen-Vocabulary Object Detection Using Captions
Zareian, A., Rosa, K. D., Hu, D. H., & Chang, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01416 · Cited by 443 · cited 1× · see citing works
“These features are also leveraged by recent works for open-vocabulary object detection [Kamath et al., 2021, Gu et al., 2021, Zareian et al., 2021, Minderer et al., 2022].”
BackgroundDiscriminative clustering for image co-segmentation
Joulin, A., Bach, F., & Ponce, J. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539868 · Cited by 437 · see citing works
Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., McDermott, J. H., & Torralba, A. B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_35 · Cited by 435 · cited 1× · see citing works
“Multi-modal models, on the other hand, can consider a soundbite and its text [Sermanet et al., 2018, Chung et al., 2018] or some frames of a video and the corresponding sound clip [Zhao et al., 2018, Alwassel et al., 2020] as different views to be used as positive pairs for contrastive learning.”
BackgroundDeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Giorgi, J., Nitski, O., Wang, B., & Bader, G. D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.72 · Cited by 432 · cited 1× · see citing works
“For contrastive learning, positive pairs often come from masking and/or cropping input sequences [Meng et al., 2021, Giorgi et al., 2021].”
BackgroundHard Negative Mixing for Contrastive LearningPreprint
Kalantidis, Y., Sarıyıldız, M. B., Pion, N., Weinzaepfel, P., & Larlus, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01028 · Cited by 412 · see citing works
A Theoretical Analysis of Deep Q-LearningPreprint
Fan, J., Wang, Z., Xie, Y., & Yang, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00137 · Cited by 411 · see citing works
He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06870 · Cited by 406 · cited 2× · see citing works
“, 2021] leverages RoIAlign [He et al., 2017], an algorithm designed for object detectors which extracts features corresponding to a specific image patch.”
BackgroundWith a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00945 · Cited by 404 · cited 3× · see citing works
“While with NNCLR [Dwibedi et al., 2021] the matched positive crop is replaced by its nearestneighbour in latent space, in MSF [Koohpayegani et al.”
BackgroundInfluentialLiT: Zero-Shot Transfer with Locked-image text Tuning
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01759 · Cited by 402 · cited 2× · see citing works
“When it is not explicitly needed by the method, using the average of the patch tokens instead of the class token saves memory without much change on the accuracies of the network [Zhai et al., 2022a].”
BackgroundMethodTracking Emerges by Colorizing Videos
Vondrick, C., Shrivastava, A., Fathi, A., Guadarrama, S., & Murphy, K. P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_24 · Cited by 393 · cited 1× · see citing works
“Colorization-based SSL methods convert an image to grayscale, and then train a network to predict the original RGB values [Zhang et al., 2016, Larsson et al., 2016, Vondrick et al., 2018].”
MethodFlorence: A New Foundation Model for Computer Vision
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.11432 · Cited by 389 · cited 1× · see citing works
“Recent work has pushed these vision-language systems to larger scales [Ding et al., 2021, Yuan et al., 2021, Singh et al., 2022, Wang et al., 2022d, Fang et al., 2022b], based on freely available image-caption pairs collected from the internet, such as in [Schuhmann et al., 2022].”
BackgroundRepresentation Learning by Learning to Count
Noroozi, M., Pirsiavash, H., & Favaro, P. (2017) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2017.628 · Cited by 379 · cited 1× · see citing works
“A different spatial task is learning to count [Noroozi et al., 2017]: the model is trained to output the number of objects in an image in a self-supervised way.”
Self-citationScaling Autoregressive Models for Content-Rich Text-to-Image Generation
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.10789 · Cited by 371 · see citing works
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Dou, Z.-Y., Xu, Y., Gan, Z., Wang, J., Wang, S., Wang, L., Zhu, C., Zhang, P., Yuan, L., Peng, N., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01763 · Cited by 366 · cited 1× · see citing works
“While its advantages in vision-language downstream applications are evident [Shen et al., 2022, Dou et al., 2022], shared embedding spaces can also be constructed by training solely the vision encoder first, fixing it, and then training a matching language encoder, as described in [Zhai et al.,…”
Background(2019) · neural information processing systems · Cited by 350 · cited 1× · see citing works
“Around the same time, it was discovered that good SSL performance could be achieved by training a network to maximize the mutual information between the representations of an image under different views [Bachman et al., 2019].”
BackgroundMasked Visual Pre-training for Motor ControlPreprint
Xiao, T.-T., Radosavovic, I., Darrell, T., & Malik, J. (2022) · arXiv.org · doi:10.48550/arxiv.2203.06173 · Cited by 346 · see citing works
Masked Autoencoders As Spatiotemporal Learners
Feichtenhofer, C., Fan, H., Li, Y., & He, K. (2022) · arXiv · doi:10.48550/arxiv.2205.09113 · Cited by 345 · cited 2× · see citing works
“With ViT for instance, the patch embedding convolutional layer can be transferred from 2D to 3D by repeating the weights along the temporal axis [Feichtenhofer et al., 2022].”
BackgroundMethodScaling and Benchmarking Self-Supervised Visual Representation Learning
Goyal, P., Mahajan, D. K., Gupta, A., & Misra, I. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00649 · Cited by 344 · cited 2× · see citing works
“…learning algorithms, express seemingly contradictory findings, namely that existing self-supervised learning strategies are or are not effective for downstream dense prediction tasks [Goyal et al., 2019, Purushwalkam and Gupta, 2020, Zhao et al., 2021, Ericsson et al., 2021b, Shwartz-Ziv et al.].”
BackgroundOn Mutual Information Maximization for Representation LearningPreprint
Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., & Lučić, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.13625 · Cited by 340 · see citing works
Learning Robust Global Representations by Penalizing Local Predictive Power
Wang, H., Ge, S., Xing, E. P., & Lipton, Z. C. (2019) · arXiv · doi:10.48550/arxiv.1905.13549 · Cited by 337 · cited 1× · see citing works
“…a result, image encoders arising from such a joint pre-training are especially robust to visual changes that leave semantic meaning unchanged, such as sketches of objects as evaluated in ImageNet-Sketch [Wang et al., 2019, Radford et al., 2021], and are strong on out-of-domain generalization tasks.”
BackgroundRushworth, G. (1968) · Journal of Vascular and Interventional Radiology · doi:10.1136/bmj.4.5622.46-b · Cited by 333
Overview of Supervised Learning
Hastie, T., Friedman, J. H., & Tibshirani, R. J. (2001) · Springer series in statistics · doi:10.1007/978-0-387-21606-5_2 · Cited by 333 · cited 1× · see citing works
“Self-supervised learning defines a pretext task based on unlabeled inputs to produce descriptive and intelligible representations [Hastie et al., 2009, Goodfellow et al., 2016].”
BackgroundA Fast and Simple Algorithm for Training Neural Probabilistic Language Models
Mnih, A., & Teh, Y. W. (2012) · arXiv · doi:10.48550/arxiv.1206.6426 · Cited by 315 · cited 1× · see citing works
“If fθ is powerful enough, one can set c = 0 and the model will self-normalize [Mnih and Teh, 2012] • Ceylan and Gutmann [2018] extends NCE to nonindependent noise realization i.”
BackgroundSAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-TrainingPreprint
Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., & Goldstein, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01342 · Cited by 309 · cited 4× · see citing works
“, 2018] in embedding space is also an effective augmentation for tabular data [Somepalli et al., 2021].”
MethodInfluentialSelf-citationPopel, M., Tomková, M., Tomek, J., Kaiser, Ł., Uszkoreit, J., Bojar, O., & Žabokrtský, Z. (2020) · Nature Communications · doi:10.1038/s41467-020-18073-9 · Cited by 302 · cited 1× · see citing works
“Self-supervised learning (SSL) underpins deep learning’s success in natural language processing leading to advances from automated machine translation to large language models trained on web-scale corpora of unlabeled text [Brown et al., 2020, Popel et al., 2020].”
BackgroundSelf-Supervised Learning by Cross-Modal Audio-Video Clustering
Alwassel, H., Mahajan, D., Korbar, B., Torresani, L., Ghanem, B., & Tran, D. (2019) · arXiv · doi:10.48550/arxiv.1911.12667 · Cited by 298 · cited 1× · see citing works
“Multi-modal models, on the other hand, can consider a soundbite and its text [Sermanet et al., 2018, Chung et al., 2018] or some frames of a video and the corresponding sound clip [Zhao et al., 2018, Alwassel et al., 2020] as different views to be used as positive pairs for contrastive learning.”
BackgroundBoosting Self-Supervised Learning via Knowledge Transfer
Noroozi, M., Vinjimoor, A., Favaro, P., & Pirsiavash, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00975 · Cited by 298 · cited 1× · see citing works
“This strategy was superseded by “jigsaw” methods [Pathak et al., 2016, Noroozi et al., 2018] that break an image into an array of disjoint patches and predict the relative location of each.”
MethodSelf-citationSelf supervised contrastive learning for digital histopathology
Ciga, O., Xu, T., & Martel, A. L. (2022) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100198 · Cited by 294 · cited 1× · see citing works
“SSL can be especially useful in domains such as medicine where labels are costly or the specific task can not be known a priori [Krishnan et al., 2022, Ciga et al., 2022].”
BackgroundHow to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., & Beyer, L. (2021) · arXiv · doi:10.48550/arxiv.2106.10270 · Cited by 292 · cited 1× · see citing works
“Conversely, when training smaller models like ViT-B, such regularization usually hurts the performance [Steiner et al., 2021].”
BackgroundSLIP: Self-supervision Meets Language-Image Pre-training
Mu, N., Kirillov, A., Wagner, D., & Xie, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_30 · Cited by 285 · see citing works
How Much Can CLIP Benefit Vision-and-Language Tasks?Preprint
Shen, S., Li, L. H., Tan, H. X., Bansal, M., Rohrbach, A., Chang, K.-W., Yao, Z., & Keutzer, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06383 · Cited by 272 · cited 1× · see citing works
“While its advantages in vision-language downstream applications are evident [Shen et al., 2022, Dou et al., 2022], shared embedding spaces can also be constructed by training solely the vision encoder first, fixing it, and then training a matching language encoder, as described in [Zhai et al.,…”
BackgroundThe Unsurprising Effectiveness of Pre-Trained Vision Models for ControlPreprint
Parisi, S., Rajeswaran, A., Purushwalkam, S., & Gupta, A. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2203.03580 · Cited by 251 · see citing works
Unsupervised Learning by Predicting Noise
Bojanowski, P., & Joulin, A. (2017) · arXiv · doi:10.48550/arxiv.1704.05310 · Cited by 228 · see citing works
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senécal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223 · cited 1× · see citing works
“This is an alternative to sampling based estimation of the gradient that was found to be less stable [Bengio and Senécal, 2003, 2008].”
MethodA large-scale study on unsupervised spatiotemporal representation learning
Feichtenhofer, C., Fan, H., Xiong, B., Girshick, R., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00331 · Cited by 222 · see citing works
Balestriero, R., & LeCun, Y. (2022) · Neural Information Processing Systems · doi:10.52202/068431-1934 · Cited by 197 · cited 1×
“In fact, attempts at unifying SSL methods under a single viewpoint have only started to emerge in the last year [HaoChen et al., 2021, Balestriero and LeCun, 2022, Shwartz-Ziv et al., 2022, Garrido et al., 2022b].”
BackgroundSelf-citationDecoupled Contrastive Learning
Yeh, C.-H., Hong, C.-Y., Hsu, Y., Liu, T.-L., Chen, Y., & LeCun, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_38 · Cited by 192 · cited 4× · see citing works
“Furthermore, some works such as DCL [Yeh et al., 2021] show that you can reach top performance with a batch size of 256 or more for SimCLR, and a queue size of only 256 or more for MoCo, by simply removing the positive pair from the denominator of the softmax and with more careful hyperparameter tuning.”
MethodInfluentialSelf-citationUnsupervised Speech RecognitionPreprint
Baevski, A., Hsu, W.-N., Conneau, A., & Auli, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11084 · Cited by 191 · cited 1× · see citing works
“For example convolutions can be useful [Oord et al., 2016, Schneider et al., 2019, Baevski et al., 2021].”
BackgroundMa, Z., & Collins, M. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1405 · Cited by 186
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182 · cited 4× · see citing works
“The per-layer drop-rate may depend linearly on the layer depth or uniformly as suggested in recent works [Touvron et al., 2021b].”
BackgroundMethodResultInfluentialUnsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAPreprint
Hyvärinen, A., & Morioka, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06336 · Cited by 176 · cited 2× · see citing works
“Alternative theories suggest that InfoNCE balances alignment of “positive” examples and uniformity of the overall feature representation [Wang and Isola, 2022], or that (under strong assumptions) it can identify the latent structure in a hypothesized data-generating process, akin to nonlinear ICA [Khemakhem et al., 20…”
BackgroundAre Large-scale Datasets Necessary for Self-Supervised Pre-training?Preprint
El-Nouby, A., Izacard, G., Touvron, H., et al. (2021) · arXiv.org · doi:10.48550/arxiv.2112.10740 · Cited by 172 · cited 2× · see citing works
“, 2021], or on datasets that different from ImageNet such as COCO [El-Nouby et al., 2021], or iNaturalist [Assran et al.”
BackgroundBaevski, A., Babu, A., Hsu, W.-N., & Auli, M. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2212.07525 · Cited by 166 · cited 3×
“0 [Baevski et al., 2022] which use a context part of an image to predict missing small parts of the image.”
BackgroundInfluentialLearning Invariant Representations for Reinforcement Learning without Reconstruction
Zhang, A., McAllister, R., Calandra, R., Gal, Y., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2006.10742 · Cited by 165 · see citing works
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language TasksPreprint
Wang, W., Bao, H., Li, D., Björck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.10442 · Cited by 163 · cited 3× · see citing works
“MLM, along with span-infilling techniques, remains popular as a SSL objective for large language models [Raffel et al., 2020, Wang et al., 2022a, Tay et al., 2022].”
BackgroundMethodSelf-Supervised Learning of Object Parts for Semantic Segmentation
Ziegler, A., & Asano, Y. M. (2022) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.01410 · Cited by 160 · cited 1× · see citing works
“Just as clustering-based methods cluster related images, Leopart [Ziegler and Asano, 2022] fine-tunes a pre-trained model to cluster patch-level features.”
MethodMuse: Text-To-Image Generation via Masked Generative Transformers
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00704 · Cited by 157 · cited 3× · see citing works
“, 2022] • Muse reaches state-of-the-art text conditional image generation with a masked transformer approach [Chang et al., 2023].”
BackgroundTowards the Generalization of Contrastive Self-Supervised LearningPreprint
Huang, W., Yi, M.-Y., Zhao, X.-Y., & Jiang, Z.-H. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2111.00743 · Cited by 156 · see citing works
Self-supervised learning for videos: A survey
Schiappa, M. C., Rawat, Y. S., & Shah, M. (2023) · ACM Computing Surveys · doi:10.1145/3577925 · Cited by 155 · cited 1× · see citing works
“SSL has also been successfully applied across other modalities such as video, audio, and time series [Wickstrøm et al., 2022, Liu et al., 2022a, Schiappa et al., 2022a].”
MethodFrom Play to Policy: Conditional Behavior Generation from Uncurated Robot Data
Cui, Z. J., Wang, Y., Shafiullah, N. M. (., et al. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.10047 · Cited by 152
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature DistillationPreprint
Wei, Y., Hu, H., Xie, Z., Zhang, Z., Cao, Y., Bao, J., Chen, D., & Guo, B. (2022) · arXiv.org · doi:10.48550/arxiv.2205.14141 · Cited by 148 · cited 1× · see citing works
“It has been shown that contrasting methods show inferior performance than masked image modeling with regards to fine-tuning, because they are less “optimization friendly” [Wei et al., 2022] - which explains the overall interest over MIM.”
MethodWhitening for Self-Supervised Representation LearningPreprint
Ermolov, A., Siarohin, A., Sangineto, E., & Sebe, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06346 · Cited by 147 · cited 2× · see citing works
“Mialon et al. [2022] argue that the projector enforces pairwise independence of the features in the representation and provide a demonstration for random projectors in the context of VICReg, BarlowTwins and W-MSE [Bardes et al., 2021, Zbontar et al., 2021, Ermolov et al., 2021].”
MethodWhat Should Not Be Contrastive in Contrastive LearningPreprint
Xiao, T., Wang, X.-L., Efros, A. A., & Darrell, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05659 · Cited by 147 · cited 1× · see citing works
“Another line of work tries to retain style information about the augmentations to improve downstream performance on tasks requiring style information such as color by predicting style information [Xiao et al., 2020, Dangovski et al., 2021, Gidaris et al., 2018, Scherr et al., 2022].”
BackgroundDETReg: Unsupervised Pretraining with Region Priors for Object Detection
Bar, A., Wang, X., Kantorov, V., Reed, C., Herzig, R., Chechik, G., Rohrbach, A., Darrell, T., & Globerson, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01420 · Cited by 142 · cited 3× · see citing works
“In DETReg, detection targets are obtained using the Selective Search algorithm, which does not require human annotations.”
BackgroundSelf-citationSelf-supervised Pretraining of Visual Features in the Wild
Goyal, P., Caron, M., Lefaudeux, B., Xu, M., Wang, P., Pai, V., Singh, M., Liptchinsky, V., Misra, I., Joulin, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.01988 · Cited by 141 · see citing works
Nonlinear canonical correlation analysis by neural networks
Hsieh, W. W. (2000) · Neural Networks · doi:10.1016/s0893-6080(00)00067-8 · Cited by 141 · see citing works
Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset BiasesPreprint
Purushwalkam, S., & Gupta, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13916 · Cited by 137 · cited 3× · see citing works
“…such as MoCo [He et al., 2020b] and PIRL [Misra and Maaten, 2020], learn occlusion invariance since they are trained with random crops on ImageNet where foreground objects are often large so that different crops contain different parts of the same object [Purushwalkam and Gupta, 2020].”
BackgroundDexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play
Guzey, I., Evans, B., Chintala, S., et al. (2023) · Conference on Robot Learning · doi:10.48550/arxiv.2303.12076 · Cited by 135
COCO-LM: Correcting and Contrasting Text Sequences for Language Model PretrainingPreprint
Yu, M., Xiong, C., Bajaj, P., Tiwary, S., Bennett, P. N., Han, J., & Song, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08473 · Cited by 133 · cited 15× · see citing works
“For instance, Feichtenhofer et al. [2021] have generalized SimCLR, MoCo, SwAV and BYOL to space-time video data.”
BackgroundInfluentialUnderstanding self-supervised Learning Dynamics without Contrastive PairsPreprint
Tian, Y., Chen, X., & Ganguli, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06810 · Cited by 132 · cited 1× · see citing works
“The predictor acts as a whitening operator preventing collapse [Tian et al., 2021], and momentum network can be applied only to the projector [Pham et al.”
BackgroundSelf-citationTABBIE: Pretrained Representations of Tabular Data
Iida, H., Thai, D., Manjunatha, V., & Iyyer, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.270 · Cited by 130 · cited 1× · see citing works
“Furthermore, it is common with tabular data to predict mask vectors as a pretext task [Yoon et al., 2020, Iida et al., 2021].”
MethodEfficient Self-supervised Vision Transformers for Representation LearningPreprint
Li, C., Yang, J., Zhang, P., Gao, M., Xiao, B., Dai, X., Yuan, L., & Gao, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09785 · Cited by 129 · cited 3× · see citing works
“…have been introduced such as the entropy of the singular value distribution [Garrido et al., 2022a], the classical rank estimator [Jing et al., 2022], fitting a power law to the singular value distribution [Ghosh et al., 2022] or the AUC of the singular value distribution [Li et al., 2022a].”
BackgroundUnderstanding Dimensional Collapse in Contrastive Self-supervised LearningPreprint
Jing, L., Vincent, P., LeCun, Y., & Tian, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09348 · Cited by 128 · cited 2× · see citing works
“, 2022a], the classical rank estimator [Jing et al., 2022], fitting a power law to the singular value distribution [Ghosh et al.”
BackgroundSelf-citationOn Feature Decorrelation in Self-Supervised Learning
Hua, T., Wang, W., Xue, Z., Ren, S., Wang, Y., & Zhao, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00946 · Cited by 121 · see citing works
Simple Open-Vocabulary Object Detection with Vision TransformersPreprint
Minderer, M., Gritsenko, A. A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.06230 · Cited by 119 · cited 1× · see citing works
“These features are also leveraged by recent works for open-vocabulary object detection [Kamath et al., 2021, Gu et al., 2021, Zareian et al., 2021, Minderer et al., 2022].”
BackgroundInstance Localization for Self-supervised Detection Pretraining
Yang, C., Wu, Z., Zhou, B., & Lin, S. C.-F. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00398 · Cited by 116 · cited 4× · see citing works
“Instance Localization [Yang et al., 2021] leverages RoIAlign [He et al.”
BackgroundInfluentialAudio self-supervised learning: A survey
Liu, S., Mallol-Ragolta, A., Parada‐Cabaleiro, E., Qian, K., Jing, X., Kathan, A., Hu, B., & Schuller, B. W. (2022) · Patterns · doi:10.1016/j.patter.2022.100616 · Cited by 116 · cited 1× · see citing works
“SSL has also been successfully applied across other modalities such as video, audio, and time series [Wickstrøm et al., 2022, Liu et al., 2022a, Schiappa et al., 2022a].”
MethodMixing up contrastive learning: Self-supervised representation learning for time series
Wickstrøm, K., Kampffmeyer, M., Mikalsen, K. Ø., & Jenssen, R. (2022) · Pattern Recognition Letters · doi:10.1016/j.patrec.2022.02.007 · Cited by 113 · cited 1× · see citing works
“SSL has also been successfully applied across other modalities such as video, audio, and time series [Wickstrøm et al., 2022, Liu et al., 2022a, Schiappa et al., 2022a].”
MethodNotes on Noise Contrastive Estimation and Negative SamplingPreprint
Dyer, C. (2014) · arXiv.org · doi:10.48550/arxiv.1410.8251 · Cited by 111 · see citing works
Representation Learning via Invariant Causal MechanismsPreprint
Mitrovic, J., McWilliams, B., Walker, J., Buesing, L., & Blundell, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07922 · Cited by 106 · see citing works
Data-Efficient Reinforcement Learning with Self-Predictive RepresentationsPreprint
Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A. C., & Bachman, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.05929 · Cited by 103 · see citing works
Does Zero-Shot Reinforcement Learning Exist?Preprint
Touati, A., Rapin, J., & Ollivier, Y. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2209.14935 · Cited by 103 · cited 2× · see citing works
“, 2017], forward-backward representations [Touati et al., 2023].”
MethodDeep Metric Learning With Tuplet Margin Loss
Yu, B., & Tao, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00659 · Cited by 102 · see citing works
A Self-Supervised Descriptor for Image Copy Detection
Pizzi, E., Roy, S. D., Ravindra, S. N., Goyal, P., & Douze, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01413 · Cited by 101 · cited 1× · see citing works
“Another approach, SSCD builds on the contrastive objective to the task of copy detection outperforming copy detection models and other contrastive methods [Pizzi et al., 2022].”
MethodSCARF: Self-Supervised Contrastive Learning using Random Feature CorruptionPreprint
Bahri, D., Jiang, H., Tay, Y., & Metzler, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15147 · Cited by 99 · see citing works
Flagged first, then by citations
30 of the 313 references couldn’t be matched to a known paper.
Research works printed but not linked: 43
- K. K. Agrawal, A. K. Mondal, A. Ghosh, and B. A. Richards. α-req : Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=ii9X4vtZGTZ.
- G. Andrew, R. Arora, J. Bilmes, and K. Livescu. Deep canonical correlation analysis. In International conference on machine learning, pages 1247-1255. PMLR, 2013.
- S. P. Arunachalam, I. Güzey, S. Chintala, and L. Pinto. Holo-dex: Teaching dexterity with immersive mixed reality. (arXiv:2210.06463), Oct 2022. URL http://arxiv.org/abs/ 2210.06463. arXiv:2210.06463 [cs]. · arXiv
- M. Assran, M. Caron, I. Misra, P. Bojanowski, F. Bordes, P. Vincent, A. Joulin, M. Rabbat, and N. Ballas. Masked siamese networks for label-efficient learning. In Computer Vision -ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI, page 456-473, Berlin, Heidelberg, 2022c. Springer-Verlag. ISBN 978-3-031-19820-5. doi: 10.1007/978-3-031-19821-2_26. URL https://doi.org/ 10.1007/978-3-031-19821-2_26. · DOI
- A. Bardes, J. Ponce, and Y. LeCun. Vicregl: Variance-invariance-covariance regularization for self-supervised learning. Advances in neural information processing systems, 2022. 22, 33, 44
- Y. Bengio and J.-S. Senécal. Quick training of probabilistic neural nets by importance sampling. In International Workshop on Artificial Intelligence and Statistics, pages 17-24. PMLR, 2003. 8, 9
- M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018. 6, 12, 44
- M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691- 1703. PMLR, 2020a.
- X. Chen and K. He. Exploring simple siamese representation learning. (arXiv:2011.10566), Nov 2020. URL http://arxiv.org/abs/2011.10566. arXiv:2011.10566 [cs]. · arXiv
- K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020. · arXiv
- J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. · arXiv
- Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ a4d92e2cd541fca87e4620aba658316d-Abstract.html.
- B. Eysenbach, T. Zhang, R. Salakhutdinov, and S. Levine. Contrastive learning as goal- conditioned reinforcement learning. (arXiv:2206.07568), Jun 2022. URL http://arxiv. org/abs/2206.07568. number: arXiv:2206.07568 arXiv:2206.07568 [cs]. · arXiv
- Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao. EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. arxiv:2211.07636[cs], Dec. 2022b. doi: 10.48550/arXiv.2211.07636. URL http://arxiv.org/abs/2211.07636. · DOI
- J. Goldberger, G. E. Hinton, S. Roweis, and R. R. Salakhutdinov. Neighbourhood compo- nents analysis. Advances in neural information processing systems, 17, 2004.
- I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio. Deep learning, volume 1. MIT Press, 2016.
- M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation princi- ple for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010. 9, 40
- B. He and M. Ozay. Exploring the gap between collapsed & whitened features in self- supervised learning. In International Conference on Machine Learning, pages 8613-8634. PMLR, 2022.
- K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
- K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735, 2020b. 18, 34, 43
- D.-H. Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896, 2013.
- I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
- A. Mnih and K. Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. Advances in neural information processing systems, 26, 2013.
- R. Ni, M. Goldblum, A. Sharaf, K. Kong, and T. Goldstein. Data augmentation for meta- learning. In International Conference on Machine Learning, pages 8152-8161. PMLR, 2021a.
- R. Ni, M. Shu, H. Souri, M. Goldblum, and T. Goldstein. The close relationship be- tween contrastive learning and meta-learning. In International Conference on Learning Representations, 2021b. 7, 21
- A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 7, 10, 17, 18, 38, 41 · arXiv
- N. Parikh, S. Boyd, et al. Proximal algorithms. Foundations and trends® in Optimization, 1(3):127-239, 2014.
- A. Piché, J. Marino, G. M. Marconi, C. Pal, and M. E. Khan. Beyond target networks: Im- proving deep q-learning with functional regularization. arXiv preprint arXiv:2106.02613, 2021. · arXiv
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-maron, M. Giménez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A Generalist Agent. Transactions on Machine Learning Research, Nov. 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=1ikK0kHjvj.
- M. C. Schiappa, Y. S. Rawat, and M. Shah. Self-supervised learning for videos: A survey. ACM Computing Surveys, 2022b.
- K. Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016. 7, 10, 18
- Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, D. Bahri, T. Schuster, H. S. Zheng, N. Houlsby, and D. Metzler. Unifying Language Learning Paradigms. arxiv:2205.05131[cs], May 2022. URL http://arxiv.org/abs/2205.05131. 15, 39 · arXiv
- Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/ paper/2019/hash/3eefceb8087e964f89c2d59e8a249915-Abstract.html.
- W. Wang, R. Arora, K. Livescu, and J. Bilmes. On deep multi-view representation learning. In International conference on machine learning, pages 1083-1092. PMLR, 2015. 6, 14
- X. Wang and A. Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2015.
- L. Xu, J. Neufeld, B. Larson, and D. Schuurmans. Maximum margin clustering. Advances in neural information processing systems, 17, 2004.
- D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Reinforcement learning with prototypical representations. arXiv:2102.11271 [cs], Jul 2021. URL http://arxiv.org/abs/2102. 11271. arXiv: 2102.11271. · arXiv
- A. YM., R. C., and V. A. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations, 2020. URL https: //openreview.net/forum?id=Hyx-jyBFPr.
- J. Yoon, Y. Zhang, J. Jordon, and M. van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:11033-11043, 2020.
- Curran Associates, Inc., 2014. URL https://proceedings.neurips.cc/paper/2014/file/ 375c71349b295fbe2dcdca9206f20a06-Paper.pdf.
- B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In NeurIPS, 2014.
- J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022a. 12, 16, 22, 29, 30, 32, 33
Cited by
Sorted by citations
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560 · see citing works
Regulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466 · see citing works
“[28, 53] that aim at allowing them “to find patterns and relationships”
BackgroundFoundation Model for Cancer Imaging Biomarkers
Pai, S., Bontempi, D., Hadzic, I., Prudente, V., Sokač, M., Chaunzwa, T. L., Bernatz, S., Hosny, A., Mak, R. H., Birkbak, N. J., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00807-9 · Cited by 199 · see citing works
“Existing literature 12 has suggested several strategies, such as image reconstruction, to pretrain networks to learn these representations.”
SARATR-X: Toward Building a Foundation Model for SAR Target Recognition
Li, W., Yang, W., Hou, Y., Liu, L., Liu, Y., & Li, X. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3531988 · Cited by 124 · see citing works
“In the following sections, we introduce recent developments in remote sensing foundation models.”
Self-Supervised Multimodal Learning: A Survey
Zong, Y., Aodha, O. M., & Hospedales, T. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429301 · Cited by 122 · see citing works
“Rethinking Self-supervision in the Multimodal Context and Beyond: The core appeal of SSL is utilizing widely available unlabeled data [199].”
Triguero, I., Molina, D., Poyatos, J., Del Ser, J., & Herrera, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102135 · Cited by 90 · see citing works
Yang, F., Zuo, R., & Kreuzer, O. P. (2024) · Earth-Science Reviews · doi:10.1016/j.earscirev.2024.104941 · Cited by 90 · see citing works
Li, W., Yang, W., Liu, T., Hou, Y., Li, Y., Liu, Z., Liu, Y., & Liu, L. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.09.013 · Cited by 79 · see citing works
Sun, L., Lian, Z., Liu, B., & Tao, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102382 · Cited by 78 · see citing works
“Specifically, masked We present Pearson correlation coe ffi cient on Werewolf-XL [17] and weighted F1-score on AVCA ff e [18] and MER-MULTI [19].”
MethodArtificial intelligence in digital pathology — time for a reality check
Aggarwal, A., Bharadwaj, S., Corredor, G., Pathak, T. B., Badve, S., & Madabhushi, A. (2025) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-025-00991-6 · Cited by 78 · see citing works
AstroCLIP: a cross-modal foundation model for galaxies
Parker, L., Lanusse, F., Golkar, S., Sarra, L., Cranmer, M., Bietti, A., Eickenberg, M., Krawezik, G. P., McCabe, M., Morel, R., et al. (2024) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stae1450 · Cited by 75 · see citing works
Machine learning for anomaly detection in particle physics
Belis, V., Odagiu, P., & Aarrestad, T. K. (2024) · Reviews in Physics · doi:10.1016/j.revip.2024.100091 · Cited by 78 · see citing works
“Self-supervised learning 56 is a form of unsupervised learning where the data provides the supervision.”
BackgroundChen, X., Yang, R., Xue, Y., Song, B., & Wang, Z. (2024) · Control Engineering Practice · doi:10.1016/j.conengprac.2024.105900 · Cited by 66 · see citing works
A systematic review of generalization research in medical image classification
Matta, S. C., Lamard, M., Zhang, P., Guilcher, A. L., Borderie, L., Cochener, B., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109256 · Cited by 75 · see citing works
“This is achieved by putting their embedding close to each other (for the same class) and far apart (for di ff erent classes) [148, 147].”
BackgroundVisual cognition in multimodal large language models
Schulze Buschoff, L. M., Akata, E., Bethge, M., & Schulz, E. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00963-y · Cited by 83 · see citing works
“Note that this model does not currently feature an option for manually setting the temperature, and the provided documentation does not specify what the default temperature is.”
Visual Point Cloud Forecasting Enables Scalable Autonomous Driving
Yang, Z., Chen, L., Sun, Y., & Li, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01390 · Cited by 56 · see citing works
“Considering the expensive labeling workflow, pre-training [2, 18, 62] has emerged as a crucial approach to scale up downstream applications.”
MethodTransformers in biosignal analysis: A review
Anwar, A., Khalifa, Y., Coyle, J. L., & Sejdić, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102697 · Cited by 55 · see citing works
Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology
Kraus, O. Z., Kenyon-Dean, K., Saberian, S., Fallah, M., McLean, P., Leung, J., Sharma, V., Khan, A., Balakrishnan, J., Çelik, S., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01117 · Cited by 51 · see citing works
“We found that training dynamics and downstream performance benefited from large batch sizes of up to 16,384 image crops and using the Lion optimizer [16], versus the typical choices of batch size and AdamW optimizer [3].”
MethodBrain-Inspired Remote Sensing Foundation Models and Open Problems: A Comprehensive Survey
Jiao, L., Huang, Z., Lu, X., Liu, X., Yang, Y., Zhao, J., Zhang, J., Hou, B., Yang, S., Liu, F., et al. (2023) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2023.3316302 · Cited by 45 · see citing works
“SSL defines a pretext task based on unlabeled inputs to generate descriptive and interpretable representations [70], [72], [75], [76].”
BackgroundMa, J., Guo, Z., Zhou, F., Wang, Y., Xu, Y., Li, J., Yan, F., Cai, Y., Zhu, Z., Jin, C., et al. (2025) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01488-4 · Cited by 59 · see citing works
Applications of Self-Supervised Learning to Biomedical Signals: A Survey
Pup, F. D., & Atzori, M. (2023) · IEEE Access · doi:10.1109/access.2023.3344531 · Cited by 43 · see citing works
Helpless infants are learning a foundation model
Cusack, R., Ranzato, M., & Charvet, C. J. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.05.001 · Cited by 34 · see citing works
Sun, J., Zhu, T., Cui, Y., & Wu, B. (2025) · The Innovation · doi:10.1016/j.xinn.2024.100750 · Cited by 45 · see citing works
Pre-Trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness
Wang, S., Zhang, J., Yuan, Z., & Shan, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02313 · Cited by 31 · see citing works
“Inspired by the success of pre-trained language models like BERT [13] and GPT [7], the rise of Self-Supervised Learning (SSL) [2] in computer vision has started to produce task-agnostic models with foundational characteristics, which we refer to as ”foundational models.””
MethodDeep generative models for detector signature simulation: A taxonomic review
Hashemi, H., & Krause, C. (2024) · Reviews in Physics · doi:10.1016/j.revip.2024.100092 · Cited by 35 · see citing works
“In their work, Hashemi et al. [69] enhance the diversity and complexity [275, 282] of generated samples by introducing Self-Supervised Learning [283] techniques that can be incorporated in any DGMs.”
MethodMalSSL—Self-Supervised Learning for Accurate and Label-Efficient Malware Classification
Ismail, S. J. I., Hendrawan, H., Rahardjo, B., Juhana, T., & Musashi, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3392251 · Cited by 29 · see citing works
“Leveraging self-supervised learning, MalSSL processes unlabeled data to learn representations and construct a model [14].”
MethodCL-MAE: Curriculum-Learned Masked Autoencoders
Madan, N., Ristea, N.-C., Nasrollahi, K., Moeslund, T. B., & Ionescu, R. T. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00248 · Cited by 25 · see citing works
“Even though a lot of attention has been dedicated to refining the pretext task [1,2,13,23,32,33,55,57], comparatively less attention has been paid to the token selection strategy [6,30,31].”
BackgroundLarge-scale foundation models and generative AI for BigData neuroscience
Wang, R., & Chen, Z. S. (2024) · Neuroscience Research · doi:10.1016/j.neures.2024.06.003 · Cited by 25 · see citing works
Three Pillars Improving Vision Foundation Model Distillation for Lidar
Puy, G., Gidaris, S., Boulch, A., Siméoni, O., Sautier, C., Pérez, P., Bursuc, A., & Marlet, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02033 · Cited by 25 · see citing works
“The backbone is finetuned using a layer-wise learning rate decay [3, 15], a technique commonly used when finetuning pretrained ViT backbones [4, 24, 79].”
MethodWang, Z., Lu, Y. J., Deng, K., et al. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103503 · Cited by 24 · see citing works
Büttner, N., Stalder, S., Volpi, M., et al. (2025) · Habitat International · doi:10.1016/j.habitatint.2025.103403 · Cited by 23 · see citing works
Russell, M., & Wang, P. (2023) · Journal of manufacturing systems · doi:10.1016/j.jmsy.2023.09.008 · Cited by 24 · see citing works
“Each domain is di ff erent [48], and designing equivalent augmentations for 1D time series data unlocks the potential of invariance-based SSL for raw sensing signals.”
BackgroundCentrality-based Relation aware Heterogeneous Graph Neural Network
Li, Y., Fu, S., Zeng, Y., Feng, H., Peng, R., Wang, J., & Zhang, S. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111174 · Cited by 22 · see citing works
BenthicNet: A global compilation of seafloor images for deep learning applications
Lowe, S. C., Misiuk, B., Xu, I., et al. (2025) · Scientific Data · doi:10.1038/s41597-025-04491-1 · Cited by 20 · see citing works
A survey on recent trends in robotics and artificial intelligence in the furniture industry
Brunello, A., Fabris, G., Gasparetto, A., Montanari, A., Saccomanno, N., & Scalera, L. (2024) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2024.102920 · Cited by 19 · see citing works
Tian, J., Lei, J., Zhang, J., Xie, W., & Li, Y. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3371481 · Cited by 19 · see citing works
“Early SSL methods largely relied on various pretext tasks [18], such as jigsaw puzzles [19], patch localization [20], and image inpainting [21].”
MethodPhilEO Bench: Evaluating Geo-Spatial Foundation Models
Fibæk, C. S., Camilleri, L., Luyts, A., Dionelis, N., & Saux, B. L. (2024) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss53475.2024.10642780 · Cited by 19 · see citing works
“The latent representations of each of the FMs [7, 32] are fed to a similar decoder.”
BackgroundBootstrapping Autonomous Driving Radars with Self-Supervised Learning
Hao, Y., Madani, S., Guan, J., Alloulah, M., Gupta, S., & Hassanieh, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01422 · Cited by 21 · see citing works
“Radio signals represent another SSL data domain [10] that comes with a unique set of challenges and considerations.”
BackgroundData compression and inference in cosmology with self-supervised machine learning
Akhmetzhanova, A., Mishra-Sharma, S., & Dvorkin, C. (2023) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stad3646 · Cited by 18 · see citing works
Artificial intelligence as a catalyst for combustion science and engineering
Ihme, M., & Chung, W. T. (2024) · Proceedings of the Combustion Institute · doi:10.1016/j.proci.2024.105730 · Cited by 18 · see citing works
A self-supervised learning method for Raman spectroscopy based on masked autoencoders
Ren, P., Zhou, R., & Li, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128576 · Cited by 18 · see citing works
Unsupervised learning of mid-level visual representations
Matteucci, G., Piasini, E., & Zoccolan, D. (2023) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2023.102834 · Cited by 18 · see citing works
“…reconstructing hidden image parts (masked autoencoders), restoring corrupted data (denoising autoencoders and diffusion models), generating new data samples (variational autoencoders and generative adversarial networks), or predicting the next data sample (contrastive predictive coding) [85].”
BackgroundRaghu, S. T. P., MacIsaac, D., & Scheme, E. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109479 · Cited by 17 · see citing works
“A comprehensive overview of the different SSL approaches, including discussions on the critical role of augmentations and the challenges of representation collapse, is provided in[15].”
BackgroundZhang, Z., Liu, H., Shao, Y., Yang, J., Liu, S., & Yuan, G. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108835 · Cited by 15 · see citing works
Self-supervised learning unveils urban change from street-level images
Stalder, S., Volpi, M., Büttner, N., Law, S., Harttgen, K., & Süel, E. (2024) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2024.102156 · Cited by 15 · see citing works
“On the other hand, self-supervised representation learning methods are being increasingly investigated as a way to extract meaningful information from large sets of structured but unlabelled data [54].”
Literature review on deep learning for the segmentation of seismic images
Monteiro, B. A. A., Canguçu, G. L., Jorge, L. M. S., Vareto, R. H., Oliveira, B. S., Silva, T. H., Lima, L. A., Machado, A. M. C., Schwartz, W. R., & Vaz-de-Melo, P. O. S. (2024) · Earth-Science Reviews · doi:10.1016/j.earscirev.2024.104955 · Cited by 14 · see citing works
Atzberger, C., Immitzer, M., Hemes, K. S., et al. (2025) · Remote Sensing of Environment · doi:10.1016/j.rse.2025.114774 · Cited by 14 · see citing works
WavLM model ensemble for audio deepfake detection
Combei, D., Stan, A., Oneata, D., & Cucu, H. (2024) · The Automatic Speaker Verification Spoofing Countermeasures Workshop (ASVspoof 2024) · doi:10.21437/asvspoof.2024-25 · Cited by 15 · see citing works
“Self-supervised learning (SSL) [9] is a powerful paradigm that aims to produce transferable representations.”
BackgroundUnsupervised Contrastive Learning for Robust RF Device Fingerprinting Under Time-Domain Shift
Chen, J., Wong, W.-K., & Hamdaoui, B. (2024) · International Conference on Communications · doi:10.1109/icc51166.2024.10622173 · Cited by 14 · see citing works
“Our approach is based on contrastive learning [17]–[20], which has been shown to be a highly effective self-supervised learning approach [21] for computer vision and natural language processing.”
MethodReview of deep representation learning techniques for brain–computer interfaces
Guetschel, P., Ahmadi, S., & Tangermann, M. (2024) · Journal of Neural Engineering · doi:10.1088/1741-2552/ad8962 · Cited by 15 · see citing works
AI in architecture and engineering from misconceptions to game-changing prospects
Sourek, M. (2024) · Architectural Intelligence · doi:10.1007/s44223-023-00046-9 · Cited by 14 · see citing works
“As opposed to supervised learning, which is limited by the availability of labeled data, self-supervised approaches can learn from vast unlabeled data (Balestriero, 2023).”
Sanchez‐Fernandez, A. J., Moreno‐Álvarez, S., Rico‐Gallego, J., & Tabik, S. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3421622 · Cited by 13 · see citing works
“SSL is recognized as one of the most promising methodologies for acquiring data representations that exhibit robust generalization across downstream tasks [16], [20], [31].”
BackgroundMethodResultAnatomically constrained tractography of the fetal brain
Calixto, C. A., Jaimes, C. E., Soldatelli, M. D., Warfield, S. K., Gholipour, A., & Karimi, D. (2024) · NeuroImage · doi:10.1016/j.neuroimage.2024.120723 · Cited by 12 · see citing works
“From this viewpoint, our approach is similar to the standard approaches in self-supervised learning methods.”
Application of self-supervised learning in steel surface defect detection
Hu, S., Ma, X., Zhang, Y., & Xu, W. (2025) · Journal of Materials Informatics · doi:10.20517/jmi.2025.21 · Cited by 11 · see citing works
Single-pixel imaging based on self-supervised conditional mask classifier-free guidance
Li, Q., Yan, Q., Dong, J., et al. (2024) · Optics Express · doi:10.1364/oe.518455 · Cited by 11 · see citing works
Emre, T., Chakravarty, A., Rivail, A., Lachinov, D. A., Leingang, O., Riedl, S., Mai, J., Scholl, H., Sivaprasad, S., Rueckert, D., et al. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3391215 · Cited by 13 · see citing works
“It has been a common approach when the labels are scarce or noisy to stabilize the optimization by preventing over-fitting for the downstream task [7].”
BackgroundHyperedge Graph Contrastive Learning
Zhang, J., Zeng, W., Tang, J., & Zhao, X. H. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3435861 · Cited by 11 · see citing works
“Recently, graph Self-Supervised Learning (SSL) starts to arrest increasingly more interests, which can reduce the reliance on labels and learn a common representation for downstream tasks by automatically generating labels for data [16].”
BackgroundA self-supervised contrastive change point detection method for industrial time series
Bao, X., Chen, L., Zhong, J., Wu, D., & Zheng, Y. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108217 · Cited by 11 · see citing works
Mitrai, I., & Daoutidis, P. (2024) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2024.108686 · Cited by 11 · see citing works
Moody, J. B., Poitrasson-Rivière, A., Renaud, J. M., Hagio, T., Alahdab, F., Al-Mallah, M. H., Vanderver, M. D., Goonewardena, S. N., Ficaro, E. P., & Murthy, V. L. (2025) · NEJM AI · doi:10.1056/aioa2500164 · Cited by 11 · see citing works
Self-Supervised Marine Organism Detection From Underwater Images
Li, J., Yang, W., Qiao, S., Gu, Z., Zheng, B., & Zheng, H. (2024) · IEEE Journal of Oceanic Engineering · doi:10.1109/joe.2024.3455565 · Cited by 10 · see citing works
From lazy to rich to exclusive task representations in neural networks and neural codes
Farrell, M., Recanatesi, S., & Shea‐Brown, E. (2023) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2023.102780 · Cited by 11 · see citing works
LarvaTagger: manual and automatic tagging of Drosophila larval behaviour
Laurent, F., Blanc, A., May, L., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae441 · Cited by 10 · see citing works
Scaling Up Multivariate Time Series Pre-Training with Decoupled Spatial-Temporal Representations
Zha, R., Zhang, L., Li, S., et al. (2024) · Proceedings - International Conference on Data Engineering · doi:10.1109/icde60146.2024.00057 · Cited by 9 · see citing works
“However, these methods typically require a large number of negative samples [54], leading to substantial memory overhead and failure to adapt to high-dimensional multivariate time series, such as Yizhuang which encompasses thounds of nodes.”
BackgroundEfficient Image Pre-Training with Siamese Cropped Masked Autoencoders
Eymaël, A., Vandeghen, R., Cioppa, A., Giancola, S., Ghanem, B., & Droogenbroeck, M. V. (2024) · European Conference on Computer Vision · doi:10.1007/978-3-031-73337-6_20 · Cited by 25 · see citing works
“This is especially interesting in domains in which data labelling is costly, such as image segmentation or object detection, or when the exact task to solve is not known beforehand [1].”
Backgroundda Silva, M. P., Correa, S. P. L. P., Schaefer, M., Reis, J. C. S., Nunes, I., dos Santos, J. A., & Oliveira, H. N. (2025) · Computers & graphics · doi:10.1016/j.cag.2025.104434 · Cited by 11 · see citing works
The Importance of Human-Labeled Data in the Era of LLMs
Liu, Y. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/802 · Cited by 10 · see citing works
Data-Efficient Multimodal Fusion on a Single GPU
Vouitsis, N., Liu, Z., Gorti, S. K., Villecroze, V., Cresswell, J. C., Yu, G., Loaiza-Ganem, G., & Volkovs, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02572 · Cited by 9 · see citing works
“In the natural image domain, common augmentations include horizontal flips, random crops, and color jitter [6, 13], which were designed to leave semantic information unchanged.”
BackgroundEvaluating Superhuman Models with Consistency Checks
Fluri, L., Paleka, D., & Tramèr, F. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00017 · Cited by 14 · see citing works
“Many semi-supervised [21] and self-supervised [7] learning algorithms enforce an invariance or contra-variance in model outputs, e.g., invariant predictions under adversarial transformations [58] or contrastive learning of data augmentations [22].”
BackgroundSun, J., Zhu, T., Cui, Y., & Wu, B. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.08.09.552725 · Cited by 9 · see citing works
“In contrast to supervised learning, which is limited by labeled data, self-supervised learning (SSL) is able to learn from vast unlabeled data 26 .”
BackgroundHu, L., & Wu, W. Y. (2024) · IEEE Access · doi:10.1109/access.2024.3493628 · Cited by 8 · see citing works
A novel self-supervised method for in-field occluded apple ripeness determination
Zhao, Z., Hicks, Y., Sun, X., McGuinness, B. J., & Lim, H. S. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110246 · Cited by 8 · see citing works
Abdallah, A., Kasem, M. S., Abdelhalim, I., Alghamdi, N. S., & El-Baz, A. S. (2025) · IEEE Access · doi:10.1109/access.2025.3581582 · Cited by 8 · see citing works
“To address these challenges, we propose PatchCascade-ViT, a novel framework that combines the strengths of ViTs with self-supervised learning (SSL) [37].”
CycleCL: Self-supervised Learning for Periodic Videos
Destro, M., & Gygli, M. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00284 · Cited by 8 · see citing works
“For a more in-depth review of self-supervised learning methods, please refer to [2,24,28].”
BackgroundPinetsuksai, N., Kittichai, V., Jomtarak, R., et al. (2023) · doi:10.1109/icitee59582.2023.10317719 · Cited by 7 · see citing works
“According to “A Cookbook of Self-Supervised Learning” [9], SSL is gaining more and more attention, even though SSL itself still has some complicated models.”
BackgroundAn Experimental Comparison of Multi-View Self-Supervised Methods for Music Tagging
Meseguer-Brocal, G., Desblancs, D., & Hennequin, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447473 · Cited by 7 · see citing works
“Self-supervised approaches can be clustered into two main categories that are based on the definition of the pretext task [9].”
BackgroundAugmentation-aware self-supervised learning with conditioned projector
Przewięźlikowski, M., Pyla, M., Zieliński, B., Twardowski, B., Tabor, J., & Śmieja, M. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112572 · Cited by 6 · see citing works
Polkinghorne, M., Pearson, N. J., van Duivenvoorde, W., et al. (2024) · Marine Policy · doi:10.1016/j.marpol.2024.106074 · Cited by 7 · see citing works
Toward collaborative artificial intelligence development for animal well-being
Sun, J. (2025) · Journal of the American Veterinary Medical Association · doi:10.2460/javma.24.10.0650 · Cited by 7 · see citing works
“In self-supervised learn - ing, 24 pretext tasks are designed to encourage the model to learn relevant representations from unlabeled data without relying on explicit labels.”
InfluentialDenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning
Wang, X., He, J., Wang, P., Zhou, Y., Sun, T., & Qiu, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29886 · Cited by 7 · see citing works
Masked and Inverse Dynamics Modeling for Data-Efficient Reinforcement Learning
Lee, Y. J., Kim, J., Park, Y. J., Kwak, M. G., & Kim, S. B. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3439261 · Cited by 6 · see citing works
“This strategy gains from having an asymmetric architecture and updates through EMA [52].”
DenoMAE: A Multimodal Autoencoder for Denoising Modulation Signals
Faysal, A., Bouhsine, T., Rostami, M., Roshan, R. G., Wang, H., Muralidhar, N., Sahoo, A., & Yao, Y. (2025) · IEEE Communications Letters · doi:10.1109/lcomm.2025.3570602 · Cited by 8 · see citing works
“Self-supervised learning has emerged as an efficient solution that leverages abundant raw data during pretraining, drastically reducing labeled data requirements while maintaining strong downstream performance [2].”
Mixup Feature: A Pretext Task Self-Supervised Learning Method for Enhanced Visual Feature Learning
Xu, J., & Stirenko, S. (2023) · IEEE Access · doi:10.1109/access.2023.3301561 · Cited by 6 · see citing works
“The recent development of self-supervised learning (SSL) has accelerated the progress of artificial intelligence towards the era of large-scale models [44], [45], [46].”
BackgroundMethodZhou, Y., Zhu, H., Xu, C., et al. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3391512 · Cited by 6 · see citing works
“…mainly considering two properties: 1) contrastive learning has been widely used as pretraining to achieve robust representations for different data [14], [46], [47] and 2) it directly trains the feature extractor without labels while still allowing the model to learn features with good class…”
BackgroundMethodChen, T., Xiang, Y., & Wang, J. (2024) · Measurement Science and Technology · doi:10.1088/1361-6501/ad3fd9 · Cited by 5 · see citing works
Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning
Quélennec, A., Chouteau, P., Peeters, G. G., & Essid, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887666 · Cited by 6 · see citing works
uniDINO: Assay-independent feature extraction for fluorescence microscopy images
Morelli, F. M., Kim, V., Hecker, F. A., Geibel, S., & Zapata, P. A. M. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.02.020 · Cited by 5 · see citing works
“Following its success in natural images [11 – 16], SSL has gained traction in morphological profiling.”
InfluentialMachine learning approaches enable the discovery of therapeutics across domains
Chhibbar, P., & Das, J. (2025) · Molecular Therapy · doi:10.1016/j.ymthe.2025.04.001 · Cited by 6 · see citing works
“55”
Fan, Z., Zhang, Z., Wang, J., et al. (2025) · Journal of Vibration Engineering & Technologies · doi:10.1007/s42417-025-01990-8 · Cited by 5 · see citing works
“This method improves the generalization of the model in varied environments and proves particularly effective in industrial settings where acquiring enough labeled samples is difficult [20].”
Carneiro, G. A., Aubry, T., Cunha, A. M. T. D. S., Radeva, P. I., & Sousa, J. J. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.111134 · Cited by 6 · see citing works
Deep-learning methods for unveiling large-scale single-cell transcriptomes
Shen, X., & Li, X. (2024) · Cancer Biology and Medicine · doi:10.20892/j.issn.2095-3941.2023.0436 · Cited by 4 · see citing works
“Self-supervised learning represents a subtype of unsupervised learning, in which the model acquires knowledge by predicting specific elements of input data, frequently through the generation of pseudo-labels derived from the data 7 .”
BackgroundMachine Learning for Cybersecurity Frameworks in Smart Farming
Eleftheriadis, C., Andronikidis, G., Kyranou, K., et al. (2024) · doi:10.1109/it61232.2024.10475711 · Cited by 5 · see citing works
“Additionally, the exploration of self-supervised learning [33] is of interest, as it enables the constant adaptation to emerging security threats, facilitating dynamic updates to the application’s features.”
BackgroundLoFT: LoRA-Based Efficient and Robust Fine-Tuning Framework for Adversarial Training
Fu, J., Fang, J., Sun, J., Zhuang, S., Geng, L., & Liu, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651480 · Cited by 5 · see citing works
Wolf, D., Payer, T., Lisson, C. S., et al. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109242 · Cited by 5 · see citing works
PFML: Self-Supervised Learning of Time-Series Data Without Representation Collapse
Vaaras, E., Airaksinen, M., & Räsänen, O. J. (2025) · IEEE Access · doi:10.1109/access.2025.3556957 · Cited by 8 · see citing works
“…layer normalization after the last two convolutions to make the pre-training process more stable, the kernel size of the second convolutional layer of the CNN encoder was changed from [4 , 5] to [3 , 5], and the originally temporally asymmetrical padding was set to [1 , 2] to make it symmetric.”
InfluentialYousef, I., Shah, S. L., & Gopaluni, R. B. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110914 · Cited by 5 · see citing works
CLRiuS: Contrastive Learning for intrinsically unordered Steel Scrap
Schäfer, M., Faltings, U., & Glaser, B. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100573 · Cited by 5 · see citing works
MyriadAL: Active Few Shot Learning for Histopathology
Schiavone, N., Wang, J., Li, S., Zemp, R. J., & Li, X. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00177 · Cited by 5 · see citing works
DeepLCZChange: A REMOTE SENSING DEEP LEARNING MODEL ARCHITECTURE FOR URBAN CLIMATE RESILIENCECRediT
Sun, W., Sun, Y., Liu, C., et al. (2023) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss52108.2023.10281573 · Cited by 5 · see citing works
“In the language of self-supervised learning (SSL) [7] with recent successes in Earth observation [8], we pretrain a model ( D ◦ E ) to generate a (compressed) feature representation c from a remote sensing modality s .”
Method
- Authors
- Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., Schwarzschild, A., Wilson, A. G., Geiping, J., Garrido, Q., Fernandez, P., Bar, A., Pirsiavash, H., LeCun, Y., & Goldblum, M.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.12210
- OpenAlex
- W6960139023
- Semantic Scholar
- 6bfafb32b423c3f0456a10984814f89046def489
- Cited by
- 182
- References
- 313
- Citations by year
- 2026: 7 · 2025: 46 · 2024: 96 · 2023: 33
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, other-oa)
- Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Digital Repository at the University of Maryland (University of Maryland College Park)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- self-supervised learning, pretext tasks, hyperparameter optimization, research reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
