A Cookbook of Self-Supervised Learning

“Another approach, SSCD builds on the contrastive objective to the task of copy detection outperforming copy detection models and other contrastive methods [Pizzi et al., 2022].”

“More recently, masked auto-encoding objectives for video have been built around the same idea as images, but by masking patches/ tubes of patches in the temporal axis as well [Feichtenhofer et al., 2022, Tong et al., 2022, Girdhar et al., 2022].”

“Laplacian eigenmaps [Machado et al., 2017], forward-backward representations [Touati et al.”

“Recent works have modified BYOL [Grill et al., 2020] or Barlow Twins [Zbontar et al., 2021] by feeding images of consecutive timesteps’ observations to the two branches of the siamese network: SGI [Schwarzer et al., 2021b] and Barlow Balance [Zhang et al., 2022c] did this for offline pretraining, while SPR [Schwarzer…”

“Multi-modal models, on the other hand, can consider a soundbite and its text [Sermanet et al., 2018, Chung et al., 2018] or some frames of a video and the corresponding sound clip [Zhao et al., 2018, Alwassel et al., 2020] as different views to be used as positive pairs for contrastive learning.”

“It is worth noting that perfect invariance is not achieved thanks to the projector [Bordes et al., 2022a], which helps improve performance on tasks which are not entirely invariant.”

“Similarly, while masking images is often done with random pixels, the two dimensions of a spectrogram represent time and frequency and masking with horizontal and/or vertical bands is more effective [Wang et al., 2020b].”

“In MSF, the use of this k-NN graph only increases training time by 6%.”

“Another approach, is to resort to IO-aware optimizations [Ivanov et al., 2021], the most known one perhaps being FlashAttention [Dao et al., 2022].”

“The majority of works that followed [Bao et al., 2021b, Zhou et al., 2022a, Dong et al., 2021] focused on this type of evaluation (and sometimes do not report linear/MLP results).”

“Another line of work tries to retain style information about the augmentations to improve downstream performance on tasks requiring style information such as color by predicting style information [Xiao et al., 2020, Dangovski et al., 2021, Gidaris et al., 2018, Scherr et al., 2022].”

“…have been introduced such as the entropy of the singular value distribution [Garrido et al., 2022a], the classical rank estimator [Jing et al., 2022], fitting a power law to the singular value distribution [Ghosh et al., 2022] or the AUC of the singular value distribution [Li et al., 2022a].”

“The best-performing method doing this is EfficientZero [Ye et al., 2021], which modifies MuZero [Schrittwieser et al., 2020] by, among other modifications, adding the SimSiam [Chen and He, 2020] objective to train the encoder and the forward model, and sets the new state of the art on Atari100k.”

“MLM, along with span-infilling techniques, remains popular as a SSL objective for large language models [Raffel et al., 2020, Wang et al., 2022a, Tay et al., 2022].”

“DINOv2 further builds on iBOT and improves its performance significantly in both linear and k-NN evaluations by improving the training recipe, the architecture, and by introducing additional regularizers such as KoLeo [Sablayrolles et al., 2018].”

“It is worth pointing out that some works have only noticed minor increases in performance [Wang et al., 2021a] where it only lead to a 0.3 point performance increase.”

“, 2017], and deep canonically correlated autoencoders [Wang et al., 2015].”

“Unlike popular contrastive objectives which encourage augmented views of an image to produce similar features, SelfPatch encourages adjacent patches within a single image to produce similar features [Yun et al., 2022].”

“, 2021], SEN [Park et al., 2022], or [Marchetti et al.”

“But more specifically, they find that pretraining objectives that use the labels are best, implying that SSL for tabular data has yet to be the state of the art in its domain [Rubachev et al., 2022].”

“However, contrastive pretraining for both the SAINT model [Somepalli et al., 2021] and SubTab [Ucar et al., 2021] seems to work best when this is paired with a reconstructive loss term.”

“…optimization such as Mixed Integer Programming (MIP), objective function is used as the guidance to generate positive solution pairs with comparable objectives and negative solution pairs whose objective values drastically differs despite tiny changes of a few variables [Huang et al., 2023].”

“In fact, attempts at unifying SSL methods under a single viewpoint have only started to emerge in the last year [HaoChen et al., 2021, Balestriero and LeCun, 2022, Shwartz-Ziv et al., 2022, Garrido et al., 2022b].”

“Specifically, they propose using the InfoNCE loss [Gutmann and Hyvärinen, 2010, Ceylan and Gutmann, 2018] to compare the representations of the clean and corrupted inputs.”

“This idea of splitting representations as invariant and equivariant was also explored in SIE [Garrido et al., 2023] and using Lie group formalism in Ibrahim et al.”

“Recent works have modified BYOL [Grill et al., 2020] or Barlow Twins [Zbontar et al., 2021] by feeding images of consecutive timesteps’ observations to the two branches of the siamese network: SGI [Schwarzer et al., 2021b] and Barlow Balance [Zhang et al., 2022c] did this for offline pretraining, while SPR [Schwarzer…”

“[Tian, 2022] shows that contrastive learning with deep linear network is equivalent to Principal Component Analysis (PCA) and [Tian, 2023] further analyzes the role played by nonlinearity in the architecture if trained with contrastive loss, showing that nonlinearity leads to many local optima that can host diverse pa…”

“Recent works have modified BYOL [Grill et al., 2020] or Barlow Twins [Zbontar et al., 2021] by feeding images of consecutive timesteps’ observations to the two branches of the siamese network: SGI [Schwarzer et al., 2021b] and Barlow Balance [Zhang et al., 2022c] did this for offline pretraining, while SPR [Schwarzer…”

“There’s also evidence SSL models can learn representations that are more robust to adversarial examples, label corruption, and input perturbations—and are more fair—compared to their supervised counterparts [Hendrycks et al., 2019, Goyal et al., 2022].”

“…learning algorithms, express seemingly contradictory findings, namely that existing self-supervised learning strategies are or are not effective for downstream dense prediction tasks [Goyal et al., 2019, Purushwalkam and Gupta, 2020, Zhao et al., 2021, Ericsson et al., 2021b, Shwartz-Ziv et al.].”

“…learning algorithms, express seemingly contradictory findings, namely that existing self-supervised learning strategies are or are not effective for downstream dense prediction tasks [Goyal et al., 2019, Purushwalkam and Gupta, 2020, Zhao et al., 2021, Ericsson et al., 2021b, Shwartz-Ziv et al.].”

“In a similar spirit, ISD [Tejankar et al., 2021] compares a query distribution to anchors from the student distribution using KL-divergence that relaxes the binary distinction between positive and negative samples.”

“SSL has also been successfully applied across other modalities such as video, audio, and time series [Wickstrøm et al., 2022, Liu et al., 2022a, Schiappa et al., 2022a].”

“One failure mode of SSL objectives when applied to RL agents’ data is the proclivity to latch on ‘slow features’ [Sobal et al., 2022].”

“Existing SSL methods designed specifically for transformers confirm that the trained models are effective for downstream detection and segmentation tasks, especially when fine-tuned [Li et al., 2021b, He et al., 2022].”

“MLM, along with span-infilling techniques, remains popular as a SSL objective for large language models [Raffel et al., 2020, Wang et al., 2022a, Tay et al., 2022].”

“…have been introduced such as the entropy of the singular value distribution [Garrido et al., 2022a], the classical rank estimator [Jing et al., 2022], fitting a power law to the singular value distribution [Ghosh et al., 2022] or the AUC of the singular value distribution [Li et al., 2022a].”

“Similar approaches are also used in guided language generation [Yang et al., 2023].”

“In addition, to these procedures which require labels for the downstream task, more recently, RankMe [Garrido et al., 2022a] has appeared as a viable alternative to costly evaluations, and is used as an oracle to final accuracy without having to do any training.”

“Instead other factors such as the feature extractor and formualtion of the mutual information estimator are important and can lead to drastically different performance [Guo et al., 2022a].”

“One approach is with low-rank and/or sparse approximation mechanisms [Kitaev et al., 2020, Choromanski et al., 2020, Wang et al., 2020a, Chen et al., 2021a, Zaheer et al., 2020].”

“It is worth pointing out that some works have only noticed minor increases in performance [Wang et al., 2021a] where it only lead to a 0.3 point performance increase.”

“4https://github.com/vturrisi/solo-learn 5https://github.com/facebookresearch/FFCV-SSL FFCV-SSL [Bordes et al., 2023a] shows that one can train SimCLR on ImageNet in less than 2 days on a single GPU or in just a few hours using 8 GPUs (Figure 15).”

“In addition, to these procedures which require labels for the downstream task, more recently, RankMe [Garrido et al., 2022a] has appeared as a viable alternative to costly evaluations, and is used as an oracle to final accuracy without having to do any training.”

“It is worth noting that perfect invariance is not achieved thanks to the projector [Bordes et al., 2022a], which helps improve performance on tasks which are not entirely invariant.”

“SSL methods for computer vision have been able to match or in some cases surpass models trained on labeled data, even on highly competitive benchmarks like ImageNet [Tomasev et al., 2022, He et al., 2020a, Deng et al., 2009].”

“In fact, attempts at unifying SSL methods under a single viewpoint have only started to emerge in the last year [HaoChen et al., 2021, Balestriero and LeCun, 2022, Shwartz-Ziv et al., 2022, Garrido et al., 2022b].”

“This choice of unidirectional next-token prediction versus bidirectional approaches leads to meaningful differences in downstream text applications [Artetxe et al., 2022].”

“To this effect some works have explored pretraining on large uncurated datasets [Goyal et al., 2021], or on datasets that different from ImageNet such as COCO [El-Nouby et al., 2021], or iNaturalist [Assran et al., 2022a].”

“…have been introduced such as the entropy of the singular value distribution [Garrido et al., 2022a], the classical rank estimator [Jing et al., 2022], fitting a power law to the singular value distribution [Ghosh et al., 2022] or the AUC of the singular value distribution [Li et al., 2022a].”

“…learning algorithms, express seemingly contradictory findings, namely that existing self-supervised learning strategies are or are not effective for downstream dense prediction tasks [Goyal et al., 2019, Purushwalkam and Gupta, 2020, Zhao et al., 2021, Ericsson et al., 2021b, Shwartz-Ziv et al.].”

“10 [Coria et al., 2020] x sigmoid(cx) Triplet Contrastive Eqn.”

“Another line of work tries to retain style information about the augmentations to improve downstream performance on tasks requiring style information such as color by predicting style information [Xiao et al., 2020, Dangovski et al., 2021, Gidaris et al., 2018, Scherr et al., 2022].”

“Instead other factors such as the feature extractor and formualtion of the mutual information estimator are important and can lead to drastically different performance [Guo et al., 2022a].”

“, 2021], and momentum network can be applied only to the projector [Pham et al., 2022] • Chen and He [2021, SimSIAM] replaces the BYOL moving average encoder by a stop-gradient • Caron et al.”

“, 2022], or [Marchetti et al., 2022] which also aims at splitting the representations as class and pose.”

“Pairwise independence, or a soft notion thereof, can be more appropriate to learn unsupervised representations from “real world” datasets such as ImageNet than mutual independence [Li et al., 2019].”

“[Tian, 2022] shows that contrastive learning with deep linear network is equivalent to Principal Component Analysis (PCA) and [Tian, 2023] further analyzes the role played by nonlinearity in the architecture if trained with contrastive loss, showing that nonlinearity leads to many local optima that can host diverse pa…”

“4https://github.com/vturrisi/solo-learn 5https://github.com/facebookresearch/FFCV-SSL FFCV-SSL [Bordes et al., 2023a] shows that one can train SimCLR on ImageNet in less than 2 days on a single GPU or in just a few hours using 8 GPUs (Figure 15).”

“This strategy was further employed in UniVCL [Tang et al., 2022] which used augmentation strategies such as edge of node masking in combination with a k-NN graph in latent space.”

“Recent works have modified BYOL [Grill et al., 2020] or Barlow Twins [Zbontar et al., 2021] by feeding images of consecutive timesteps’ observations to the two branches of the siamese network: SGI [Schwarzer et al., 2021b] and Barlow Balance [Zhang et al., 2022c] did this for offline pretraining, while SPR [Schwarzer…”

“While uncommon, language modeling can be done with contrastive learning for word or or character representations [Chen et al., 2022].”

“Deep clustering alternates between assigning labels to images by performing k-means in the feature space, and updating the model to respect these assigned class labels [Caron et al., 2018].”

“[28, 53] that aim at allowing them “to find patterns and relationships”

“Existing literature 12 has suggested several strategies, such as image reconstruction, to pretrain networks to learn these representations.”

“In the following sections, we introduce recent developments in remote sensing foundation models.”

“Rethinking Self-supervision in the Multimodal Context and Beyond: The core appeal of SSL is utilizing widely available unlabeled data [199].”