Contrastive Representation Learning: A Framework and Review
Open accessThe DOI matches its Crossref record; 147 of the 162 references it cites were found in a registry or an index, and 15 are not checked yet.
Abstract
Contrastive Learning has recently received interest due to its success in self-supervised representation learning in the computer vision domain. However, the origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields and domains including Metric Learning and natural language processing. In this paper, we provide a comprehensive literature review and we propose a general Contrastive Representation Learning framework that simplifies and unifies many different contrastive learning methods. We also provide a taxonomy for each of the components of contrastive learning in order to summarise it and distinguish it from other forms of machine learning. We then discuss the inductive biases which are present in any contrastive learning system and we analyse our framework under different views from various sub-fields of Machine Learning. Examples of how contrastive learning has been applied in computer vision, natural language processing, audio processing, and others, as well as in Reinforcement Learning are also presented. Finally, we discuss the challenges and some of the most promising future research directions ahead.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“According to the suggestions of Le-Khac et al. [29], the main methods can be divided into the following categories: multisensory signals, data augmentation, localglobal consistency, and temporal consistency.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Contrastive learning [56-58] is in fact a training procedure used in recent artificial intelligence models as a way to efficiently learn semantic representations.”
From How Adversarial REM Dreams May Facilitate Creativity, and Why We Become Aware of Them (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As summarized in [29], commonly used contrastive loss functions generally include scoring functions (cosine similarity), energy-based margin functions (pair loss and triplet loss), probabilistic NCE-based functions, and mutual information based functions.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“In SimCLRv2 [17], empirical experiments show that the output of the second layer of a 3-layer MLP projection head is a better representation for transfer learning than the output of the base ResNet [42] encoder.”
BackgroundMethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“Recently, the Bidirectional Encoder Representation from Transformer (BERT) [24] model learns bidirectional word representations using the Transformer architecture’s decoder [103] and demonstrated great performance for transfer learning in multiple downstream tasks.”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Even though the BERT [24] text encoder and the Faster RCNN [87] object detection model are trained separately and not updated by the contrastive loss, the LSTM cells [48] and a linear project head can still map between words and object representations.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The discretised speech representation can then be used directly as a representation for other models that expect discrete input such as BERT [24].”
MethodInfluentialFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The discretised speech representation can then be used directly as a representation for other models that expect discrete input such as BERT [24].”
MethodInfluentialEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 2×
“In Contrastive Predictive Coding (CPC) [77] where similarity is defined from the past-present relationship, a GRU [19] head is applied over previous time steps to aggregate the past information into a contextualised embed-”
MethodSemi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853 · cited 1×
“volutional network [31], [59] that summarises a patch of the graph centered around some nodes.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · cited 1×
“Previously, most unsupervised representation learning methods belong to the class of generative models [32], [58].”
BackgroundLabaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Momentum Contrast for Unsupervised Visual Representation LearningPreprint
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. B. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.05722 · Cited by 16,321 · cited 8×
“Momentum Contrast (MoCo) [43] further reduces the need to store an offline representation of the entire dataset in the memory bank through the use of a dynamic memory queue.”
BackgroundMethodInfluentialFaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 1×
“Based on the analysis by Bengio, Courville, and Vincent [9], a good representation has the properties of local”
MethodSentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
node2vec: Scalable Feature Learning for Networks
Grover, A., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939754 · Cited by 10,926 · cited 1×
“Focusing on learning useful node representations from general graphs, node2vec [35] aims to learn a node representation that is similar between neighbour nodes.”
BackgroundA simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · cited 10×
“As analysed in SimCLR [16], many contrastive methods perform very poorly without proper augmentations (i.”
BackgroundMethodInfluentialScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · cited 11×
“In a different direction, Oord, Li, and Vinyals [77] proposed CPC to learn invariances between context-instance relationships instead.”
BackgroundMethodInfluentialwav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020) · neural information processing systems · Cited by 5,822
(2013) · Cited by 5,726
Dimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336 · cited 3×
“In 2005, Chopra, Hadsell, and LeCun [20], [39] created the foundation for the contrastive learning framework with the original contrastive pair loss for discriminative models to learn an invariant mapping for recognition and verification problems.”
BackgroundMethodInfluentialA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
Intelligence without representation
Brooks, R. A. (1991) · Artificial Intelligence · doi:10.1016/0004-3702(91)90053-m · Cited by 4,853 · cited 1×
“we know the task we want to be good at there are better ways to directly optimise for that task without explicitly dealing with the representation as a leaky abstraction [12].”
BackgroundDeep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · cited 2×
“In SimCLRv2 [17], empirical experiments show that the output of the second layer of a 3-layer MLP projection head is a better representation for transfer learning than the output of the base ResNet [42] encoder.”
BackgroundMethodDistance Metric Learning for Large Margin Nearest Neighbor Classification
Weinberger, K. Q., & Saul, L. K. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1577078 · Cited by 4,044 · cited 4×
“[11] Siamese network in metric learning setting Foundation Chopra, Hadsell, and LeCun [20] Learn similarity metric with contrastive pair loss Energy-based loss, Application Hadsell, Chopra, and LeCun [39] Learn invariant representation from pair loss Energy-based loss, Application Weinberger, Blitzer, and Saul [108] L…”
BackgroundMethodInfluentialLearning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 6×
“In Bootstrap Your Own Latent (BYOL) [34], the similarity constraint between different views are also enforced through a prediction problem, but from an online network to an offline momentum-updated network.”
BackgroundMethodInfluentialUnsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664
Categorical Reparameterization with Gumbel-Softmax
Jang, E., Gu, S., & Poole, B. (2016) · arXiv · doi:10.48550/arxiv.1611.01144 · Cited by 3,616 · cited 3×
“For example, wav2vec 2.0 [7] uses a Gumbel-softmax [52] quantisation head to map the continuous audio signal into a discrete set of latent vectors (i.e ‘‘code book’’).”
MethodInfluentialSupervised Contrastive Learning
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., & Krishnan, D. (2020) · arXiv · doi:10.48550/arxiv.2004.11362 · Cited by 3,569 · cited 7×
“A TAXONOMY OF SIMILARITY Contrastive Learning revolves around learning a mapping from different views of the same scene, or context into the same region of a representation space, which is formalised through the similarity distribution.”
MethodInfluentialSpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558 · cited 1×
“e time and frequency masking [79]), they use a pipeline similar to many image instance discrimination methods to a learned representation that is language agnostic and is shown to transfer well to an emotion classification task, regardless of the spoken language.”
MethodNeural Message Passing for Quantum Chemistry
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.01212 · Cited by 3,440 · cited 1×
“volutional network [31], [59] that summarises a patch of the graph centered around some nodes.”
BackgroundFormal Semantics for Kolmogorov-Arnold Network Representations of Operational Games
Zhukov, G. A. (2025) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.15206559 · Cited by 3,212
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. U., & Hyvärinen, A. (2010) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 2,992
ImageNet: A large-scale hierarchical image database
Deng, J. H., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
In Defense of the Triplet Loss for Person Re-Identification
Hermans, A., Beyer, L., & Leibe, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07737 · Cited by 2,687 · cited 4×
“While using hard negative and positive samples has been a common component in successfully applying contrastive methods, Wu et al. [109] and Hermans, Beyer, and Leibe [45] argue for the case that quality of data pairs used in training are also of paramount importance for pair and triplet losses in the metric learning…”
BackgroundMethodInfluentialDeep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_9 · Cited by 2,674
BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., & Jiang, P. (2019) · Proceedings of the 28th ACM International Conference on Information and Knowledge Management · doi:10.1145/3357384.3357895 · Cited by 2,558
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09882 · Cited by 2,499 · cited 3×
“Most clustering-based methods up to now are offline in the sense that they require multiple passes over the data to compute features and perform clustering, but Swapping Assignment between multiple Views (SwAV) [14] proposed an online clustering method to learn unsupervised visual representations.”
BackgroundMethodInfluentialSinkhorn Distances: Lightspeed Computation of Optimal TransportPreprint
Cuturi, M. (2013) · Cited by 2,385
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369 · cited 1×
“0 [7] uses a Gumbel-softmax [52] quantisation head to map the continuous audio signal into a discrete set of latent vectors (i.”
MethodSIGNATURE VERIFICATION USING A “SIAMESE” TIME DELAY NEURAL NETWORK
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1994) · Series in Machine Perception and Artificial Intelligence · doi:10.1142/9789812797926_0003 · Cited by 2,350
Deep Metric Learning Using Triplet Network
Hoffer, E., & Ailon, N. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-24261-3_7 · Cited by 2,134 · cited 3×
“While conceptually simple and widely adopted in multiple metric learning applications [49], [90], [105], the pair and triplet losses usually suffer from slow convergence because”
BackgroundMethodInfluentialDistance Metric Learning for Large Margin Nearest Neighbor Classification
Weinberger, K. Q., Blitzer, J. C., & Saul, L. K. (2005) · neural information processing systems · Cited by 2,132
Signature Verification Using A "Siamese" Time Delay Neural Network
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1993) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001493000339 · Cited by 2,077
Tian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997
Learning deep representations by mutual information estimation and maximization
Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., & Bengio, Y. (2018) · arXiv · doi:10.48550/arxiv.1808.06670 · Cited by 1,991 · cited 8×
“Figure 7a describes the approach taken in Deep InfoMax (DIM) [46], where an image is encoded into a global feature vector and also into a feature map corresponding to spatial patches of pixels in the original image.”
BackgroundMethodInfluentialDeep Metric Learning via Lifted Structured Feature Embedding
Song, H. O., Xiang, Y., Jegelka, S., & Savarese, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.434 · Cited by 1,630 · cited 2×
“[75] and Sohn [92] proposed loss functions that improve the number of comparisons for a query in an iteration.”
BackgroundMethod(2016) · neural information processing systems · Cited by 1,572
Deep Graph InfomaxPreprint
Veličković, P., Fedus, W., Hamilton, W. L., Lió, P., Bengio, Y., & Hjelm, R. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10341 · Cited by 1,508 · cited 3×
“Also inspired by the mutual information maximisation between global and local structure of DIM, but with some design choices different from DGI [104], InfoGraph [97] focuses on learning graph-level representations.”
MethodContrastive Learning for Unpaired Image-to-Image Translation
Park, T., Efros, A. A., Zhang, R., & Zhu, J.-Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58545-7_19 · Cited by 1,442
Self-Supervised Learning of Pretext-Invariant Representations
Misra, I., & van der Maaten, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00674 · Cited by 1,435 · cited 6×
“Recent successes in transfer learning by instance discrimination contrastive pre-training [69, 43, 16] have raised the question of “what representation is learned from contrastive methods and why is it better than supervised pre-training” [119, 101]? However from the view of the Contrastive Representation Learning fra…”
BackgroundInfluentialRethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification
Xie, S., Sun, C., Huang, J., Tu, Z., & Murphy, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01267-0_19 · Cited by 1,415 · cited 1×
“The ASR sequences are encoded using a pre-trained BERT [24] model while a pre-trained S3D [113] model is used to extract visual features which are then fed into a shallow Transformer [103] network to construct a video-level visual embedding.”
MethodLearning Fine-Grained Image Similarity with Deep RankingPreprint
Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J.-B., Philbin, J., Chen, B., & Wu, Y. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1404.4661 · Cited by 1,382
Contrastive Representation DistillationPreprint
Tian, Y., Krishnan, D., & Isola, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10699 · Cited by 1,328 · cited 4×
“Contrastive Representation Distillation (CDR) [100] uses a large, pre-trained teacher network as the encoder for both the positive and negative keys, while the queries are encoded by a small network learned to match the”
MethodInfluentialwav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
Learning Fine-Grained Image Similarity with Deep Ranking
Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., & Wu, Y. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.180 · Cited by 1,269
f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
Nowozin, S., Cseke, B., & Tomioka, R. (2016) · arXiv · doi:10.48550/arxiv.1606.00709 · Cited by 1,183
Discriminative Unsupervised Feature Learning with Convolutional Neural NetworksPreprint
Dosovitskiy, A., Springenberg, J. T., Riedmiller, M., & Brox, T. (2014) · arXiv (Cornell University) · Cited by 1,153
Sun, F.-Y., Hoffmann, J., Verma, V., & Tang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.01000 · Cited by 1,025 · cited 4×
“InfoGraph uses GIN [115] as the base encoder and uses sum over mean for the readout function, both of which are more suitable to learning representations at graph-level.”
InfluentialAsymptotic evaluation of certain markov process expectations for large time, II
Donsker, M. D., & Varadhan, S. R. S. (1975) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.3160280206 · Cited by 1,025
Data-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S. M. A., & Oord, A. V. D. (2019) · arXiv · doi:10.48550/arxiv.1905.09272 · Cited by 1,014
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · cited 1×
“Wang andGupta [107] extended this idea beyond supervised learning by learning visual representations from video with the help of an unsupervised tracking method.”
BackgroundA simple framework for contrastive learning of visual representations
Chen, T. (2024) · TIB Data Manager · doi:10.57702/iuuvgtaz · Cited by 973
Auto-Encoding Variational Bayes
Chen, Y., Liu, J., Peng, L., Wu, Y., Xu, Y., & Zhang, Z. (2024) · Cambridge Explorations in Arts and Sciences · doi:10.61603/ceas.v2i1.33 · Cited by 934
Wang, T., & Isola, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10242 · Cited by 921
Sampling Matters in Deep Embedding Learning
Wu, C.-Y., Manmatha, R., Smola, A. J., & Krahenbuhl, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.309 · Cited by 836 · cited 2×
“[109] and Hermans, Beyer, and Leibe [45] argue for the case that quality of data pairs used in training are also of paramount importance for pair and triplet losses in the metric learning setting.”
BackgroundExploring the Limits of Language Modeling
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02410 · Cited by 823
Arandjelovic, R., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 782 · cited 3×
“This is different from the verification setting from previous work [3], where an MLP fusion network takes the concatenation of the two representations and outputs the final decision on whether the signals correspond.”
BackgroundMethodGCC: Graph Contrastive Coding for Graph Neural Network Pre-Training
Qiu, J., Chen, Q., Dong, Y., Zhang, J., Yang, H., Ding, M., Wang, K., & Tang, J. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403168 · Cited by 777 · cited 1×
“Aiming to learn a structural representation of a graph without node attributes and labels, Graph Contrastive Coding (GCC) [85] simulates the augmentation-based instance discrimination task in computer vision.”
BackgroundWhat makes for good views for contrastive learning
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., & Isola, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10243 · Cited by 773 · cited 1×
“Recent successes in transfer learning by instance discrimination contrastive pre-training [16], [43], [69] have raised the question of ‘‘what representation is learned from contrastive methods and why is it better than supervised pre-training’’ [101], [119]? However from the view of the Contrastive Representation Lear…”
BackgroundLarge Scale Online Learning of Image Similarity through Ranking
Chechik, G., Sharma, V., Shalit, U., & Bengio, S. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02172-5_2 · Cited by 722
Asymptotic evaluation of certain markov process expectations for large time, I
Donsker, M. D., & Varadhan, S. R. S. (1975) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.3160280102 · Cited by 622
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
Zhang, R., Isola, P., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.76 · Cited by 616
Prototypical Contrastive Learning of Unsupervised Representations
Li, J., Zhou, P., Xiong, C., & Hoi, S. C. H. (2020) · arXiv · doi:10.48550/arxiv.2005.04966 · Cited by 611
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature
Ye, M., Zhang, X., Yuen, P. C., & Chang, S.-F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00637 · Cited by 601 · cited 4×
“[117] andMisra andMaaten [69] showed that pre-training with contrastive loss can outperform supervised-only training for a computer vision task.”
BackgroundMethodInfluentialGutmann, M. U., & Hyvärinen, A. (2012) · Edinburgh Research Explorer (University of Edinburgh) · doi:10.5555/2503308.2188396 · Cited by 581
Time-Contrastive Networks: Self-Supervised Learning from Video
Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., & Brain, G. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8462891 · Cited by 546 · cited 3×
“In the Time-Contrastive Network [91], a visual representation is learned by pulling the representation of two simultaneous views from different cameras of the same scene, while pushing apart frames taken from far away in time but from the same video.”
BackgroundMethodInfluentialDeep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · arXiv · doi:10.48550/arxiv.1807.05520 · Cited by 533
Self-organizing neural network that discovers surfaces in random-dot stereograms
Becker, S., & Hinton, G. E. (1992) · Nature · doi:10.1038/355161a0 · Cited by 495 · cited 4×
“The core idea of learning by comparing between separate but related data points, without any supervised signal, dates back to 1992 to work by Becker and Hinton [8] and by Bromley et al.”
BackgroundInfluentialOn Variational Bounds of Mutual Information
Poole, B., Ozair, S., Oord, A. V. D., Alemi, A. A., & Tucker, G. (2019) · arXiv · doi:10.48550/arxiv.1905.06922 · Cited by 480 · cited 1×
“Some recent work [83], [88] performed a review of different MI estimators and derived a new continuum of multi-”
BackgroundTian, Y., Krishnan, D., & Isola, P. (2019) · arXiv · doi:10.48550/arxiv.1906.05849 · Cited by 465 · cited 1×
“In the most obvious way, the ‘‘views’’ from Contrastive Multiview Coding (CMC) [99] is straightforward to extend to multiple modalities.”
BackgroundLearning word embeddings efficiently with noise-contrastive estimation
Mnih, A., & Kavukcuoglu, K. (2013) · neural information processing systems · Cited by 446
An efficient framework for learning sentence representations
Logeswaran, L., & Lee, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02893 · Cited by 427
Contrastive Multi-View Representation Learning on Graphs
Hassani, K., & Khasahmadi, A. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05582 · Cited by 423
Big Self-Supervised Models are Strong Semi-Supervised Learners
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., & Hinton, G. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10029 · Cited by 421 · cited 3×
“SimCLRv2 [17] performed a comprehensive study of contrastive self-supervised learning in semi-supervised settings where few labels are present, and demonstrated state-of-the-art results by contrastive pre-training in various downstream vision tasks.”
BackgroundMethodLocal Aggregation for Unsupervised Learning of Visual Embeddings
Zhuang, C., Zhai, A., & Yamins, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00610 · Cited by 417
Arandjelović, R., & Zisserman, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_27 · Cited by 403
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
Baevski, A., Schneider, S., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05453 · Cited by 388 · cited 2×
“0 [7] replaces the unidirectional context head from vq-wav2vec [6] with a bidirectional masked Transformer.”
MethodVideo Representation Learning by Dense Predictive Coding
Han, T., Xie, W., & Zisserman, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00186 · Cited by 377 · cited 1×
“Expanding CPC into learning representations from natural videos,Dense Predictive Coding (DPC) [40] contrasts between local patches of the feature maps extracted from the past context with the local patches”
Background(2019) · neural information processing systems · Cited by 350
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · arXiv · doi:10.48550/arxiv.1908.10084 · Cited by 343
How Powerful are Graph Neural Networks?
Xu, K., Hu, W., Leskovec, J., & Jegelka, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.00826 · Cited by 332 · cited 2×
“GIN [115] is used as the base encoder with a momentum encoder [43] for the keys and InfoNCE is used as the contrastive loss.”
MethodA Fast and Simple Algorithm for Training Neural Probabilistic Language Models
Mnih, A., & Teh, Y. W. (2012) · arXiv · doi:10.48550/arxiv.1206.6426 · Cited by 315 · cited 3×
“Later, Mnih and Teh [71] adapted NCE [38] and proposed a more efficient algorithm to learn a language model using a probabilistic contrastive loss, where the context query includes all the previous words, the positive key is the next word in a sequence and the negative keys are sampled from a unigram distribution of w…”
MethodWhat Should Not Be Contrastive in Contrastive LearningPreprint
Xiao, T., Wang, X.-L., Efros, A. A., & Darrell, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05659 · Cited by 267 · cited 2×
“LooC [111] is an example where contrastive learning is used to concurrently learn multiple embedding sub-spaces, each of which is invariant to all but one transformation as specified by the distribution of positive pairs.”
BackgroundCERT: Contrastive Self-supervised Learning for Language UnderstandingPreprint
Fang, H., Wang, S., Zhou, M., Ding, J., & Xie, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.12766 · Cited by 254
On Mutual Information Maximization for Representation Learning
Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., & Lučić, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.13625 · Cited by 253 · cited 2×
“[102] argue and provide empirical evidences that the success of contrastive losses can not be attributed to mutual information alone.”
BackgroundDemystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset BiasesPreprint
Purushwalkam, S., & Gupta, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13916 · Cited by 249
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Chi, Z., Dong, L., Wei, F., Yang, N., Singhal, S., Wang, W., Song, X., Mao, X.-L., Huang, H., & Zhou, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.280 · Cited by 249
Representation Learning: A Review and New PerspectivesPreprint
Bengio, Y., Courville, A. C., & Vincent, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.5538 · Cited by 235 · cited 1×
“Based on the analysis by Bengio, Courville, and Vincent [9], a good representation has the properties of local”
MethodWhitening for Self-Supervised Representation LearningPreprint
Ermolov, A., Siarohin, A., Sangineto, E., & Sebe, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06346 · Cited by 232
node2vec: Scalable Feature Learning for Networks
Grover, A., & Leskovec, J. (2016) · PubMed · doi:10.48550/arxiv.1607.00653 · Cited by 229 · cited 1×
“Focusing on learning useful node representations from general graphs, node2vec [35] aims to learn a node representation that is similar between neighbour nodes.”
BackgroundSelf-supervised Learning of Audio-Visual Objects from Video
Afouras, T., Owens, A., Chung, J. S., & Zisserman, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58523-5_13 · Cited by 217
CURL: Contrastive Unsupervised Representations for Reinforcement Learning
Srinivas, A., Laskin, M., & Abbeel, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.04136 · Cited by 213 · cited 3×
“To improve the representation for reinforcement learning (RL) tasks, CURL [93] applied the instance discrimination task with a momentum encoder from MoCo [43] to train model-free RL agents directly from the pixel observations.”
MethodInfluentialAudio-Visual Instance Discrimination with Cross-Modal Agreement
Morgado, P., Vasconcelos, N., & Misra, I. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01229 · Cited by 211 · cited 1×
“Similarly, Cross-modal Audio Visual Instance Discrimination (Cross-AVID) [72] jointly learn the general representation from video using corresponding image frames and audio segments.”
BackgroundContrastive Learning of Structured World ModelsPreprint
Kipf, T., van der Pol, E., & Welling, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.12247 · Cited by 200 · cited 2×
“C-SWM [60] State + Action Next state Random state CNN and GCN Addition Pair loss InfoGraph [97] Global graph Local graph Random local graph GIN Readout summation JSD MI estimator”
MethodMa, Z., & Collins, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1405 · Cited by 196
Unsupervised State Representation Learning in AtariPreprint
Anand, A., Racah, E., Ozair, S., Bengio, Y., Côté, M.-A., & Hjelm, R. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08226 · Cited by 196
Learning Video Representations using Contrastive Bidirectional Transformer
Sun, C., Baradel, F., Murphy, K. P., & Schmid, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05743 · Cited by 192
A Theoretical Analysis of Contrastive Unsupervised Representation Learning
Arora, S., Khandeparkar, H., Khodak, M., Plevrakis, O., & Saunshi, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09229 · Cited by 191 · cited 1×
“Some recent work [83], [88] performed a review of different MI estimators and derived a new continuum of multi-”
BackgroundCERT: Contrastive Self-supervised Learning for Language Understanding
Fang, H., & Xie, P. (2020) · doi:10.36227/techrxiv.12308378.v1 · Cited by 187
Putting An End to End-to-End: Gradient-Isolated Learning of Representations
Löwe, S., O'Connor, P., & Veeling, B. S. (2019) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.1905.11786 · Cited by 174 · cited 1×
“In this direction, Greedy InfoMax (GIM) [65] extends the approach of CPC [77] while Local Contrastive (LoCo) [114] improved the performance by extending SimCLR [16] with a modified overlapping architecture between local layers.”
MethodCERT: Contrastive Self-supervised Learning for Language UnderstandingPreprint
Fang, H., & Xie, P. (2020) · arXiv.org · doi:10.36227/techrxiv.12308378 · Cited by 171
Loss Functions for Discriminative Training of Energy-Based Models
LeCun, Y., & Huang, F. J. (2005) · International Conference on Artificial Intelligence and Statistics · Cited by 168
Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.01978 · Cited by 163 · cited 7×
“[110] decoupled the batch size from the number of negative pairs by storing a detached copy of representations of the entire dataset into a separate memory bank.”
BackgroundMethodInfluentialA Mutual Information Maximization Perspective of Language Representation LearningPreprint
Kong, L., d’Autume, C. D. M., Wang, L., Yu, L., Dai, Z., & Yogatama, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08350 · Cited by 149 · cited 1×
“[61] showed that BERT or XLnet also maximise global-local mutual information, whereas the next sentence prediction pre-training task can be seen as constructing similarity pairs using the sequential coherence property.”
MethodMulti-modal Self-Supervision from Generalized Data TransformationsPreprint
Patrick, M., Asano, Y. M., Кузнецова, П., Fong, R., Henriques, J. F., Zweig, G., & Vedaldi, A. (2021) · arXiv (Cornell University) · Cited by 149
Time-Contrastive Networks: Self-Supervised Learning from VideoPreprint
Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06888 · Cited by 136 · cited 3×
“In the Time-Contrastive Network [91], a visual representation is learned by pulling the representation of two simultaneous views from different cameras of the same scene, while pushing apart frames taken from far away in time but from the same video.”
BackgroundMethodInfluentialContrastive Code Representation Learning
Jain, P., Jain, A., Zhang, T., Abbeel, P., Gonzalez, J., & Stoica, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.482 · Cited by 129 · cited 3×
“For program code data, ContraCode [51] uses various source-to-source transformation methods from the compiler literature such as variable renaming, identifier mangling, reformatting, beautification, compression, dead-code”
MethodInfluentialContrastive Code Representation Learning
Jain, P., Jain, A., Zhang, T., Abbeel, P., Gonzalez, J. E., & Stoica, I. (2020) · arXiv · doi:10.48550/arxiv.2007.04973 · Cited by 129 · cited 3×
“For program code data, ContraCode [51] uses various source-to-source transformation methods from the compiler literature such as variable renaming, identifier mangling, reformatting, beautification, compression, dead-code”
MethodInfluentialContrastive Learning for Unpaired Image-to-Image TranslationPreprint
Park, T., Efros, A. A., Zhang, R., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.15651 · Cited by 126
Decoupling Representation Learning from Reinforcement Learning
Stooke, A., Lee, K., Abbeel, P., & Laskin, M. (2020) · arXiv · doi:10.48550/arxiv.2009.08319 · Cited by 123 · cited 1×
“[94] proposed the Augmented Temporal Contrast (ACT) for pre-training representations that are transferable to multiple RL tasks.”
What makes instance discrimination good for transfer learning?
Zhao, N., Wu, Z., Lau, R. W. H., & Lin, S. (2020) · arXiv · doi:10.48550/arxiv.2006.06606 · Cited by 115 · cited 1×
“Recent successes in transfer learning by instance discrimination contrastive pre-training [16], [43], [69] have raised the question of ‘‘what representation is learned from contrastive methods and why is it better than supervised pre-training’’ [101], [119]? However from the view of the Contrastive Representation Lear…”
BackgroundWatching the World Go By: Representation Learning from Unlabeled VideosPreprint
Gordon, D., Ehsani, K., Fox, D., & Farhadi, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.07990 · Cited by 99
Sampling Matters in Deep Embedding LearningPreprint
Wu, C.-Y., Manmatha, R., Smola, A. J., & Krähenbühl, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.07567 · Cited by 82 · cited 2×
“[109] and Hermans, Beyer, and Leibe [45] argue for the case that quality of data pairs used in training are also of paramount importance for pair and triplet losses in the metric learning setting.”
BackgroundLearning Actionable Representations from Visual Observations
Dwibedi, D., Tompson, J., Lynch, C., & Sermanet, P. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593951 · Cited by 79
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-TrainingPreprint
Chi, Z., Dong, L., Wei, F., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07834 · Cited by 75
LoCo: Local Contrastive Representation LearningPreprint
Xiong, Y., Ren, M., & Urtasun, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01342 · Cited by 75 · cited 1×
“In this direction, Greedy InfoMax (GIM) [65] extends the approach of CPC [77] while Local Contrastive (LoCo) [114] improved the performance by extending SimCLR [16] with a modified overlapping architecture between local layers.”
MethodNeural Predictive Belief RepresentationsPreprint
Guo, Z. D., Azar, M. G., Piot, B., Pires, B. Á., & Munos, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.06407 · Cited by 74 · cited 2×
“In addition to learning representations of observations in the environment, CPC|Action [36] is a variant of CPC that explored whether contrastive learning methods can also encode belief states (i.”
BackgroundDelving into Inter-Image Invariance for Unsupervised Visual Representations
Xie, J., Zhan, X., Liu, Z., Ong, Y.-S., & Loy, C. C. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01681-x · Cited by 73 · cited 1×
“InterCLR [112] also performed mini-batch clustering with a set of learned cluster centroids but instead of using a swapped prediction with no explicit negative samples, they modelled the instance-cluster relationship by assigning a pseudo-label for each instance.”
MethodLearning invariance through imitation
Taylor, G. W., Spiro, I., Bregler, C., & Fergus, R. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995538 · Cited by 52 · cited 1×
“While there have been approaches to using probabilistic approaches to learning metric embeddings [98], most successful applications up to now all use the energy-based pair or triplet loss due to the computational requirements to compute the normalisation constant in probabilistic loss.”
MethodA discriminative language model with pseudo-negative samples
Okanohara, D., & Tsujii, J. (2007) · Cited by 46
Look, Listen and LearnPreprint
Arandjelović, R., & Zisserman, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08168 · Cited by 39 · cited 3×
“This is different from the verification setting from previous work [3], where an MLP fusion network takes the concatenation of the two representations and outputs the final decision on whether the signals correspond.”
BackgroundMethodSelf-Supervised Contrastive Video-Speech Representation Learning for Ultrasound
Jiao, J., Cai, Y., Alsharid, M., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-59716-0_51 · Cited by 38
Probing Contextual Language Models for Common Ground with Visual Representations
Ilharco, G., Zellers, R., Farhadi, A., & Hajishirzi, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.422 · Cited by 37
Online Object Representations with Contrastive LearningPreprint
Pirk, S., Khansari, M., Bai, Y., Lynch, C., & Sermanet, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04312 · Cited by 31 · cited 1×
“These methods have been used in [82], [106] to learn”
MethodRepresentation Learning with Video Deep InfoMaxPreprint
Hjelm, R. D., & Bachman, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13278 · Cited by 30 · cited 2×
“VOLUME 8, 2020 193923 By combining the image data transformation, temporal coherence between frames and global-local correspondence between features, Video Deep InfoMax (VDIM) [47] learned effective spatio-temporal representations for downstream tasks on videos.”
BackgroundPattern Recognition and Image Analysis
Hutchison, D., Pandu Rangan, C., Kanade, T., et al. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02172-5 · Cited by 27
Learning Actionable Representations from Visual ObservationsPreprint
Dwibedi, D., Tompson, J., Lynch, C., & Sermanet, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.00928 · Cited by 23
COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations
Favory, X., Drossos, K., Virtanen, T., & Serra, X. (2020) · arXiv · doi:10.48550/arxiv.2006.08386 · Cited by 20 · cited 1×
“In the same spirit of learning representations from loosely aligned data,COALA [30] learns a shared embedding between audio and its tags, which aremore readily available than a corresponding audio-transcript.”
BackgroundPaccanaro, A., & Hinton, G. E. (2000) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2000.857906 · Cited by 17 · cited 1×
“The earliest approaches to learning representation from relational data that comes in the form of triplets (subject, relation, object) is Linear Relational Encoding (LRE) [78].”
BackgroundDelving into Inter-Image Invariance for Unsupervised Visual RepresentationsPreprint
Xie, J., Zhan, X., Liu, Z., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.11702 · Cited by 15 · cited 1×
“InterCLR [112] also performed mini-batch clustering with a set of learned cluster centroids but instead of using a swapped prediction with no explicit negative samples, they modelled the instance-cluster relationship by assigning a pseudo-label for each instance.”
MethodCSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation LearningPreprint
Khurana, S., Laurent, A., & Glass, J. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02814 · Cited by 12
Probing Text Models for Common Ground with Visual RepresentationsPreprint
Ilharco, G., Zellers, R., Farhadi, A., & Hajishirzi, H. (2020) · arXiv (Cornell University) · Cited by 11
Language Agnostic Speech Embeddings for Emotion Classification
Nandan, A. D. M., & Vepa, J. (2020) · Cited by 11
Ma, Z., & Collins, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.01812 · Cited by 9
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video ClassificationPreprint
Xie, S., Sun, C., Huang, J., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.04851 · Cited by 4 · cited 1×
“The ASR sequences are encoded using a pre-trained BERT [24] model while a pre-trained S3D [113] model is used to extract visual features which are then fed into a shallow Transformer [103] network to construct a video-level visual embedding.”
MethodSelf-supervised Contrastive Video-Speech Representation Learning for UltrasoundPreprint
Jiao, J., Cai, Y., Alsharid, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.06607 · Cited by 3 · cited 1×
“[53] applied the misalignment objective to learn joint embeddings for ultrasound audio and the corresponding doctor’s narrative speech.”
BackgroundChen, T., Wang, Z., Li, L., Li, D., Li, Z., Chang, X., Bian, P., Liang, G., Wang, Q., & Xie, T. (2025) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.ecoop.2025.6
On Compositions of Transformations in Contrastive Self-Supervised LearningPreprint
Patrick, M., Asano, Y. M., Kuznetsova, P., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04298 · cited 2×
“[81] performed visual audio crossmodal contrastive learning with a more principled approach”
BackgroundMethodSelf-Supervised Learning of Pretext-Invariant Representations
Misra, I., & van der Maaten, L. (2019) · arXiv · doi:10.48550/arxiv.1912.01991 · cited 6×
“Recent successes in transfer learning by instance discrimination contrastive pre-training [69, 43, 16] have raised the question of “what representation is learned from contrastive methods and why is it better than supervised pre-training” [119, 101]? However from the view of the Contrastive Representation Learning fra…”
BackgroundInfluentialSelf-Supervised Learning of Audio-Visual Objects from Video
Afouras, T., Owens, A., Chung, J. S., & Zisserman, A. (2020) · arXiv · doi:10.48550/arxiv.2008.04237 · cited 1×
“[1] used de-synchronisation to select negative samples by mis-aligning (shifting) the video and audio features.”
MethodProbing Contextual Language Models for Common Ground with Visual Representations
Ilharco, G., Zellers, R., Farhadi, A., & Hajishirzi, H. (2020) · arXiv · doi:10.48550/arxiv.2005.00619 · cited 2×
“[50] learned a probe to find the similarities between words and object images from a paired image captioning dataset.”
BackgroundMethodAudio-Visual Instance Discrimination with Cross-Modal Agreement
Morgado, P., Vasconcelos, N., & Misra, I. (2020) · arXiv · doi:10.48550/arxiv.2004.12943 · cited 1×
“Similarly, Cross-modal Audio Visual Instance Discrimination (Cross-AVID) [72] jointly learn the general representation from video using corresponding image frames and audio segments.”
Background
Research works printed but not linked: 1
- stratos, 2019, Noise contrastive estimation
Cited by
Natural language processing applied to mental illness detection: a narrative review
Zhang, T., Schoene, A. M., Ji, S., & Ananiadou, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00589-7 · Cited by 476
“…learning techniques, various learning techniques have emerged and accelerated NLP research, such as adversarial training 181 , contrastive learning 182 , joint learning 183 , reinforcement learning 184 and transfer learning 185 , which can also be utilized for the mental illness detection task.”
BackgroundSelf-Supervised Learning for Electroencephalography
Rafiei, M. H., Gauthier, L. V., Adeli, H., & Takabi, D. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3190448 · Cited by 317
“4(a)] to retrieve a pretext data point [69].”
BackgroundSimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models
Wang, L., Zhao, W., Wei, Z., & Liu, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.295 · Cited by 264
Self-supervised learning methods and applications in medical imaging analysis: a survey
Shurrab, S., & Duwairi, R. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1045 · Cited by 254
“4 Le-Khac et al. (2020) Review Contrastive representation learning: A framework and review IEEE Access”
BackgroundMethodSelf-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“As summarized in [29], commonly used contrastive loss functions generally include scoring functions (cosine similarity), energy-based margin functions (pair loss and triplet loss), probabilistic NCE-based functions, and mutual information based functions.”
BackgroundMethod
Metadata
- Authors
- Le-Khac, P. H., Healy, G., & Smeaton, A. F.
- Affiliations
- Dublin City University (IE)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 193907-193934
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3096655658
- Cited by
- 862
- References
- 144
- Citations by year
- 2026: 111 · 2025: 197 · 2024: 212 · 2023: 157 · 2022: 113 · 2021: 71 · 2020: 1
- FWCI
- 33.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- arXiv (Cornell University) (acceptedVersion)
- Dublin City University Open Access Institutional Repository (Dublin City University) (submittedVersion, cc-by)
- Arrow@dit (Dublin Institute of Technology) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Science Foundation Ireland through the Science Foundation Ireland (SFI) Centre for Research Training in Machine Learning (18/CRT/6183)
- Insight Centre for Data Analytics (SFI/12/RC/2289_P2)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- contrastive learning, representation learning, self-supervised learning, metric learning, inductive bias, computer vision, natural language processing, audio signal processing, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.