Contrastive Representation Learning: A Framework and Review
Open accessThe DOI matches its Crossref record; 147 of the 162 references it cites were found in a registry or an index, and 15 are not checked yet.
Abstract
Contrastive Learning has recently received interest due to its success in self-supervised representation learning in the computer vision domain. However, the origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields and domains including Metric Learning and natural language processing. In this paper, we provide a comprehensive literature review and we propose a general Contrastive Representation Learning framework that simplifies and unifies many different contrastive learning methods. We also provide a taxonomy for each of the components of contrastive learning in order to summarise it and distinguish it from other forms of machine learning. We then discuss the inductive biases which are present in any contrastive learning system and we analyse our framework under different views from various sub-fields of Machine Learning. Examples of how contrastive learning has been applied in computer vision, natural language processing, audio processing, and others, as well as in Reinforcement Learning are also presented. Finally, we discuss the challenges and some of the most promising future research directions ahead.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“According to the suggestions of Le-Khac et al. [29], the main methods can be divided into the following categories: multisensory signals, data augmentation, localglobal consistency, and temporal consistency.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Contrastive learning [56-58] is in fact a training procedure used in recent artificial intelligence models as a way to efficiently learn semantic representations.”
From How Adversarial REM Dreams May Facilitate Creativity, and Why We Become Aware of Them (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As summarized in [29], commonly used contrastive loss functions generally include scoring functions (cosine similarity), energy-based margin functions (pair loss and triplet loss), probabilistic NCE-based functions, and mutual information based functions.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“In SimCLRv2 [17], empirical experiments show that the output of the second layer of a 3-layer MLP projection head is a better representation for transfer learning than the output of the base ResNet [42] encoder.”
BackgroundMethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“Recently, the Bidirectional Encoder Representation from Transformer (BERT) [24] model learns bidirectional word representations using the Transformer architecture’s decoder [103] and demonstrated great performance for transfer learning in multiple downstream tasks.”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Even though the BERT [24] text encoder and the Faster RCNN [87] object detection model are trained separately and not updated by the contrastive loss, the LSTM cells [48] and a linear project head can still map between words and object representations.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The discretised speech representation can then be used directly as a representation for other models that expect discrete input such as BERT [24].”
MethodInfluentialFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The discretised speech representation can then be used directly as a representation for other models that expect discrete input such as BERT [24].”
MethodInfluentialEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Research works printed but not linked: 1
- stratos, 2019, Noise contrastive estimation
Cited by
Natural language processing applied to mental illness detection: a narrative review
Zhang, T., Schoene, A. M., Ji, S., & Ananiadou, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00589-7 · Cited by 476
“…learning techniques, various learning techniques have emerged and accelerated NLP research, such as adversarial training 181 , contrastive learning 182 , joint learning 183 , reinforcement learning 184 and transfer learning 185 , which can also be utilized for the mental illness detection task.”
BackgroundSelf-Supervised Learning for Electroencephalography
Rafiei, M. H., Gauthier, L. V., Adeli, H., & Takabi, D. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3190448 · Cited by 317
“4(a)] to retrieve a pretext data point [69].”
BackgroundSimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models
Wang, L., Zhao, W., Wei, Z., & Liu, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.295 · Cited by 264
Self-supervised learning methods and applications in medical imaging analysis: a survey
Shurrab, S., & Duwairi, R. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1045 · Cited by 254
“4 Le-Khac et al. (2020) Review Contrastive representation learning: A framework and review IEEE Access”
BackgroundMethodSelf-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“As summarized in [29], commonly used contrastive loss functions generally include scoring functions (cosine similarity), energy-based margin functions (pair loss and triplet loss), probabilistic NCE-based functions, and mutual information based functions.”
BackgroundMethodA comprehensive survey on contrastive learning
Hu, H., Wang, X., Zhang, Y., Chen, Q., & Guan, Q. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128645 · Cited by 230
Identifying multicellular spatiotemporal organization of cells with SpaceFlow
Ren, H., Walker, B. L., Cang, Z., & Nie, Q. (2022) · Nature Communications · doi:10.1038/s41467-022-31739-w · Cited by 237
“DGI adopts a contrastive learning strategy 68 to learn the encoder, where features are learned through teaching which data points from an unlabeled dataset are similar or distinct.”
BackgroundEvaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study
Hämäläinen, P., Tavast, M., & Kunnari, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580688 · Cited by 248
Chen, J., Liu, Y., Wei, S., Bian, Z., Subramanian, S., Carass, A., Prince, J. L., & Du, Y. (2025) · Medical Image Analysis · doi:10.1016/j.media.2024.103385 · Cited by 167
Self-Guided Contrastive Learning for BERT Sentence Representations
Kim, T., Yoo, K. M., & Lee, S.-G. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.197 · Cited by 157
“For extensive surveys on contrastive learning, refer to Le-Khac et al. (2020) and Jaiswal et al. (2020).”
BackgroundMai, Z., Li, R., Kim, H., & Sanner, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00398 · Cited by 181
“The general goal of contrastive learning is intuitive: the representation of “similar” samples should be mapped close together in the embedding space, while that of “dissimilar” samples should be further away [25, 30].”
BackgroundDeep learning for Alzheimer's disease diagnosis: A survey
Khojaste-Sarakhsi, M., Haghighi, S. S., Ghomi, S. M. T. F., & Marchiori, E. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102332 · Cited by 194
CARLA: Self-supervised contrastive representation learning for time series anomaly detection
Darban, Z. Z., Webb, G. I., Pan, S., Aggarwal, C. C., & Salehi, M. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110874 · Cited by 229
“Contrastive representation learning creates an embedding space where similar samples are close and dissimilar ones are distant, applicable in domains like natural language processing, computer vision, and time series anomaly detection [15].”
BackgroundBYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation
Niizumi, D., Takeuchi, D., Ohishi, Y., Harada, N., & Kashino, K. (2021) · 2021 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534474 · Cited by 141
“In the computer vision domain, contrastive learning, which leverages the instance discrimination pretext task, has become dominant in self-supervised learning [4].”
BackgroundA Long-Text Classification Method of Chinese News Based on BERT and CNN
Chen, X., Cong, P., & Lv, S. (2022) · IEEE Access · doi:10.1109/access.2022.3162614 · Cited by 190
Multi-Objective Interpolation Training for Robustness to Label Noise
Ortego, D., Arazo, E., Albert, P., O'Connor, N. E., & McGuinness, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00654 · Cited by 209
“However, supervised and unsupervised contrastive learning approaches that consider a high number of negatives have recently received significant attention due to their success in unsupervised learning [4, 12, 25].”
BackgroundOut-of-Distribution Generalization on Graphs: A Survey
Li, H., Wang, X. Y., Zhang, Z., & Zhu, W. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3593897 · Cited by 191
“Unlikegenerativemodels,contrastivelearningisaninstance-wisediscriminativeapproachthatmakessimilarinstancescloser anddissimilarinstancesfarfromeachotherinrepresenta-tionspace[106],[107],soitcaneliminatecomputationally expensivegraphreconstructionandlearninformativegraph representations.”
Li, Y., Wang, H., & Sun, M. (2023) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2023.109850 · Cited by 180
Federated learning for healthcare applications
Chaddad, A., Wu, Y., & Desrosiers, C. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3325822 · Cited by 136
“Recently, contrastive learning has become a popular research topic for learning unlabeled data representations [196].”
BackgroundMethodAudio self-supervised learning: A survey
Liu, S., Mallol-Ragolta, A., Parada‐Cabaleiro, E., Qian, K., Jing, X., Kathan, A., Hu, B., & Schuller, B. W. (2022) · Patterns · doi:10.1016/j.patter.2022.100616 · Cited by 188
“Besides, generative adversarial networks (GANs) are also seen as a kind of SSL framework in some works.(8,11,17) For instance, the generator creates data from a random vector by taking the real data as training targets.”
Contrastive learning based self-supervised time-series analysis
Pöppelbaum, J., Chadha, G. S., & Schwung, A. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2021.108397 · Cited by 131
Self-supervised Video Transformer
Ranasinghe, K., Naseer, M., Khan, S. H., Khan, F. S., & Ryoo, M. S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00289 · Cited by 165
“However, recently contrastive methods have dominated self-supervised learning [6, 13, 16, 17, 19, 27, 37, 38, 50, 55, 74, 75] .”
BackgroundDeep learning for cross-domain data fusion in urban computing: Taxonomy, advances, and outlook
Zou, X., Yan, Y., Hao, X., Hu, Y., Wen, H., Liu, E., Zhang, J., Li, Y., Li, T., Zheng, Y., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102606 · Cited by 177
Contrastive self-supervised learning: review, progress, challenges and future research directions
Kumar, P., Rawat, P., & Chauhan, S. (2022) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-022-00245-6 · Cited by 140
CODER: Knowledge-infused cross-lingual medical term embedding for term normalization
Yuan, Z., Zhao, Z., Sun, H., Li, J., Wang, F., & Yu, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103983 · Cited by 111
Self-Supervised Pretraining Improves Self-Supervised Pretraining
Reed, C. J., Yue, X., Nrusimha, A., Ebrahimi, S., Vijaykumar, V., Mao, R., Li, B., Zhang, S., Guillory, D., Metzger, S., et al. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00112 · Cited by 98
Membership inference attacks against synthetic health data
Zhang, Z., Yan, C., & Malin, B. A. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103977 · Cited by 91
Contrastive learning on protein embeddings enlightens midnight zone
(2022) · doi:10.1093/nargab/lqac043 · Cited by 105
“Instead, contrastive learning (63), optimizes a new 35 embedding space in which similar samples are pushed closer, 36”
BackgroundTask-specific contrastive learning for few-shot remote sensing image scene classification
Zeng, Q., & Geng, J. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.07.013 · Cited by 108
Broaden Your Views for Self-Supervised Video Learning
Recasens, A., Luc, P., Alayrac, J.-B., Wang, L., Strub, F., Tallec, C., Malinowski, M., Patraaucean, V., Altche, F., Valko, M., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00129 · Cited by 89
Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search
Wang, J., Liu, Y., & Wang, X. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.151 · Cited by 50
“Image search models in the first fashion are often trained under the contrastive learning framework (Le-Khac et al., 2020).”
BackgroundCorrect-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious CorrelationsPreprint
Zhang, M., Sohoni, N. S., Zhang, H. R., Finn, C., & Ré, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.01517 · Cited by 201
“which works by predicting whether two inputs are “similar” or “dissimilar” (Le-Khac et al., 2020).”
Unsupervised Deraining: Where Contrastive Learning Meets Self-similarity
Ye, Y., Yu, C., Chang, Y., Zhu, L., Zhao, X.-L., Yan, L., & Tian, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00573 · Cited by 113
“It has been recognized that for different CL tasks, the choice of the encoder would vastly influence the final performance [27].”
BackgroundSupervised contrastive learning over prototype-label embeddings for network intrusion detection
López‐Martín, M., Sánchez-Esguevillas, A., Arribas, J. I., & Carro, B. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.09.014 · Cited by 96
“They have achieved excellent results [1–4], but they also have some significant drawbacks: (a) long training periods, (b) the need to create complex sampling strategies to extract negative samples, and (c) difficulties in creating representative prototypes for each class/label in a supervised…”
BackgroundYuan, Y., Lin, L., Zhou, Z.-G., Jiang, H., & Liu, Q. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.11.020 · Cited by 89
Machine learning in computational histopathology: Challenges and opportunities
Cooper, M., Ji, Z., & Krishnan, R. G. (2023) · Genes, Chromosomes and Cancer · doi:10.1002/gcc.23177 · Cited by 79
“Two popular means of contrastive self-supervised learning include SimCLR(50) and MoCo.(211) Reference 207 finds that over”
BackgroundSelf-supervised learning for medical image analysis: a comprehensive review
Rani, V., Kumar, M., Gupta, A., Sachdeva, M., Mittal, A., & Kumar, K. (2024) · Evolving Systems · doi:10.1007/s12530-024-09581-w · Cited by 103
“Among these, contrastive learning has garnered considerable attention for training models to distinguish between positive and negative pairs of data samples (Le-Khac et al. 2020).”
A systematic review of generalization research in medical image classification
Matta, S. C., Lamard, M., Zhang, P., Guilcher, A. L., Borderie, L., Cochener, B., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109256 · Cited by 104
“This process is designed to enhance the e ff ectiveness of various downstream tasks by capturing the essential information embedded in the data [121].”
BackgroundNot All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification
Suresh, V., & Ong, D. C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.359 · Cited by 115
“Contrastive learning has been widely used in computer vision, especially in self-supervised settings (Le-Khac et al., 2020; Chen et al., 2020) where such learning guides the model based on similarities between the latent representation of the samples.”
BackgroundSensor Data Augmentation by Resampling in Contrastive Learning for Human Activity Recognition
Wang, J., Zhu, T., Gan, J., Chen, L. L., Ning, H., & Wan, Y. (2022) · IEEE Sensors Journal · doi:10.1109/jsen.2022.3214198 · Cited by 76
Object detection in power line infrastructure: A review of the challenges and solutions
Sharma, P., Saurav, S., & Singh, S. K. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107781 · Cited by 87
Han, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 103
Peng, T., Shen, C., Sun, S., & Wang, D. (2022) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2021.3111567 · Cited by 76
Identifying spatial domain by adapting transcriptomics with histology through contrastive learning
Zeng, Y., Yin, R., Luo, M., Chen, J., Pan, Z., Lu, Y., Yu, W., & Yang, Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad048 · Cited by 76
Zhang, T., Chen, J., Liu, S., & Liu, Z. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3240921 · Cited by 73
“Since contrastive learning models are trained by similarity contrast rather than data labels, they can exploit the value of small data more fully than traditional supervised learning models [18].”
BackgroundTang, K., Xu, F., Chen, X., Dong, Q., Yuan, Y., & Chen, Y. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.04.013 · Cited by 74
Wang, W., Song, H., Si, S., Lu, W., & Cai, Z. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110394 · Cited by 73
Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text
Wang, S., Zhong, W., Tang, D., Wei, Z., Fan, Z., Jiang, D., Zhou, M., & Duan, N. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.127 · Cited by 48
Anomaly detection in IR images of PV modules using supervised contrastive learning
Bommes, L., Hoffmann, M., Buerhop‐Lutz, C., Pickel, T., Hauch, J., Brabec, C. J., Maier, A., & Peters, I. M. (2022) · Progress in Photovoltaics Research and Applications · doi:10.1002/pip.3518 · Cited by 75
“Contrastive representation learning is a form of deep metric learning initially proposed by Hadsell et al,24 which succeeds the older triplet25 and N-pair losses.26 For a good review, see Le-Khac et al.27 Contrastive representation learning uses deep neural networks to learn a lowdimensional feature space of high-dime…”
BackgroundYan, K., Lv, H., Shao, J., Chen, S., & Liu, B. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4147-8 · Cited by 71
“Contrastive learning is a robust technology suitable for feature representation learning tasks and has been widely used in unsupervised learning tasks [55].”
BackgroundSelf-supervised semi-supervised nonnegative matrix factorization for data clustering
Chavoshinejad, J., Seyedi, S. A., Tab, F. A., & Salahian, N. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109282 · Cited by 68
Shot Contrastive Self-Supervised Learning for Scene Boundary Detection
Chen, S., Nie, X., Fan, D., Zhang, D., Bhat, V., & Hamid, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00967 · Cited by 91
“Contrastive Learning: As an important subset of SSL methods, contrastive learning algorithms attempt to learn data representations by contrasting similar data against dissimilar data while using contrastive loss functions [27].”
Machine learning for bridge wind engineering
Zhang, Z., Li, S., Feng, H. X., Zhou, X., Xu, N., Li, H., Laima, S., & Chen, W. (2024) · Advances in wind engineering. · doi:10.1016/j.awe.2024.100002 · Cited by 62
Analyzing Heterogeneous Networks With Missing Attributes by Unsupervised Contrastive Learning
He, D., Liang, C., Huo, C., Feng, Z., Jin, D., Yang, L., & Zhang, W. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3149997 · Cited by 67
“Contrastive learning becomes popular first in visual representation learning [23], [24].”
BackgroundDissecting Supervised Contrastive LearningPreprint
Gräf, F., Hofer, C. D., Niethammer, M., & Kwitt, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08817 · Cited by 159
“We focus on works closely linked to our theoretical results of Section 3; we refer the reader to (Khosla et al., 2020) (and references therein) for additional background on the supervised contrastive loss and to (Le-Khac et al., 2020) for a general survey on contrastive learning.”
Jiang, X., Zhang, R., Xu, Y., Qiu, R., Fang, Y., Wang, Z., Tang, J., Ding, H., Chu, X., Zhao, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.15883 · Cited by 76
“GTE Encoder follows a two-stage training process: initially using a large-scale dataset with weak supervision from text pairs, followed by fine-tuning with high-quality manually labeled data using Contrastive Learning [38, 41].”
MethodSelf-Supervised Learning on 3D Point Clouds by Learning Discrete Generative Models
Eckart, B., Yuan, W., Liu, C., & Kautz, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00815 · Cited by 76
“…the idea that a well-structured latent space can be trained in a metric learning sense by pushing together learned representations derived from inputs that represent the same semantic content and pushing away learned representations from inputs that have different semantics [28, 34, 22, 18, 31].”
BackgroundContrastive Self-Supervised Learning: A Survey on Different Architectures
Khan, A., Albarri, S., & Manzoor, M. A. (2022) · doi:10.1109/icai55435.2022.9773725 · Cited by 74
Zhou, T. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110282 · Cited by 67
A review of named entity recognition: from learning methods to modelling paradigms and tasks
Seow, W. L., Chaturvedi, I., Hogarth, A., Mao, R., & Cambria, E. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11321-8 · Cited by 75
Negative Selection by Clustering for Contrastive Learning in Human Activity Recognition
Wang, J., Zhu, T., Chen, L. L., Ning, H., & Wan, Y. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3239945 · Cited by 52
“cn To alleviate the problem of insufficient data annotation, contrastive learning, a paradigm of self-supervised learning, has achieved excellent performance in computer vision [14].”
BackgroundECG Biometric Authentication Using Self-Supervised Learning for IoT Edge Sensors
Wang, G., Shreejith, S., Nag, A., Lian, Y., & John, D. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3455803 · Cited by 64
“Contrastive learning is a self-supervised approach that learns the signal features by comparing the data with positive and negative samples in the feature space, respectively [23].”
BackgroundChen, Z., Ji, J., Yu, W., Ni, Q., Lu, G., & Chang, X. (2024) · Measurement · doi:10.1016/j.measurement.2024.114497 · Cited by 50
CLF-Net: Contrastive Learning for Infrared and Visible Image Fusion Network
Zhu, Z., Yang, X., Lu, R., Shen, T., Xie, X., & Zhang, T. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3203000 · Cited by 60
“Contrastive learning has attracted increasing attention in the field of computer vision due to its excellent performance [23].”
BackgroundImproving BERT Model Using Contrastive Learning for Biomedical Relation Extraction
Su, P., Peng, Y., & Vijay‐Shanker, K. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.1 · Cited by 53
Physics-driven self-supervised learning system for seismic velocity inversion
Liu, B., Jiang, P., Wang, Q., Ren, Y., Yang, S., & Cohn, A. G. (2022) · Geophysics · doi:10.1190/geo2021-0302.1 · Cited by 46
Fatima, H. S., Hassan, I. U., Hasan, S., Khurram, M., Stricker, D., & Afzal, M. Z. (2023) · Applied Sciences · doi:10.3390/app13063997 · Cited by 48
Varangot‐Reille, C., Bouvard, C., Ciancone, M., Gourru, A., Schaeffer, M., & Jacquenet, F. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.19801 · Cited by 58
“This technique ensures that queries with similar meanings are embedded close together in the representation space, while queries with different meanings are placed further apart [51].”
Engagement Detection and Its Applications in Learning: A Tutorial and Selective Review
Booth, B. M., Bosch, N., & D’Mello, S. K. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3309560 · Cited by 54
Affective Game Computing: A Survey
Yannakakis, G. N., & Melhárt, D. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3315689 · Cited by 60
“Complementary to the above computer vision methods, self-supervised learning (SSL) methods such as contrastive learning define a recent machine learning paradigm which has been widely and successfully employed for learning general representations of data [252].”
MethodEnd-to-end model for automatic seizure detection using supervised contrastive learning
Li, H., Dong, X., Zhong, X., Li, C., Cui, H., & Zhou, W. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108665 · Cited by 46
AOC-IDS: Autonomous Online Framework with Contrastive Learning for Intrusion Detection
Zhang, X., Zhao, R., Jiang, Z., Sun, Z., Ding, Y., Ngai, E. C. ‐., & Yang, S. (2024) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom52122.2024.10621346 · Cited by 58
“Contrastive learning has achieved state-of-the-art results in various domains, including computer vision [38], natural language processing [39], and mobile computing [40].”
BackgroundCountering Popularity Bias by Regularizing Score Differences
Rhee, W., Cho, S., & Suh, B. (2022) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3523227.3546757 · Cited by 52
“A related approach is contrastive learning which compares and contrasts training instances for efficient representation learning [18].”
BackgroundTransFoxMol: predicting molecular property with focused attention
Gao, J., Shen, Z., Xie, Y., Lu, J., Lu, Y., Chen, S., Bian, Q., Guo, Y., Shen, L., Wu, J. F., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad306 · Cited by 44
Gilakjani, S. S., & Osman, H. A. (2023) · IEEE Access · doi:10.1109/access.2023.3344476 · Cited by 42
“CL has been widely used for general data-representation learning [56], [57].”
MethodSelf-supervised learning in drug discovery
Chen, Y., Wang, Z., Wang, J., Chu, Y., Zhang, Q., Li, Z. A., & Zeng, X. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4453-4 · Cited by 42
García, G., del Amor, R., Colomer, A., Verdú‐Monedero, R., Morales‐Sánchez, J., & Naranjo, V. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102132 · Cited by 46
SGCL: Contrastive Representation Learning for Signed Graphs
Shu, L., Du, E., Chang, Y., Chen, C., Zheng, Z., Xing, X., & Shen, S. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482478 · Cited by 67
“Fortunately, graph contrastive representation learning [14, 22, 25] can help solve this problem, which defines a contrastive objec-tive on the graph to assist learning more robust and generalized representations for downstream tasks and has aroused a growing interest in recent years.”
Yang, Z. R., Song, J., Yang, M., Yao, L., Zhang, J., Shi, H., Ji, X., Deng, Y., & Wang, X. (2021) · Analytical Chemistry · doi:10.1021/acs.analchem.1c04307 · Cited by 53
Zhao, Q. P., Ding, Y., Lü, C., Wang, C., Ma, L., Tao, L., & Ma, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121001 · Cited by 41
Doubly contrastive representation learning for federated image recognition
Zhang, Y., Xu, Y., Wei, S., Wang, Y., Li, Y., & Shang, X. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109507 · Cited by 39
Mine Your Own vieW: Self-Supervised Learning Through Across-Sample PredictionPreprint
Azabou, M., Azar, M. G., Liu, R., Lin, C.-H., Johnson, E. C., Bhaskaran‐Nair, K., Dabagia, M., Avila-Pires, B., Kitchell, L. M., Hengen, K. B., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.10106 · Cited by 53
“Previous work in contrastive learning for sequential data often leverages a slowness assumption to use nearby samples as positive examples and farther samples as negative examples (Oord et al., 2018; Sermanet et al., 2018; Dwibedi et al., 2019; Le-Khac et al., 2020; Banville et al., 2020).”
Ferreira, R. E. P., Bresolin, T., Rosa, G. J. M., & Dórea, J. R. R. (2022) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2022.107272 · Cited by 40
Supervised contrastive learning for wafer map pattern classification
Bae, Y., & Kang, S. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107154 · Cited by 41
Kabir, M. S., Kurkin, S., Portnova, G. V., Martynova, O., Wang, Z., & Hramov, A. E. (2024) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2024.115123 · Cited by 43
Disentangled Graph Contrastive Learning With Independence Promotion
Li, H., Zhang, Z., Wang, X. Y., & Zhu, W. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3206875 · Cited by 54
“Unlike generative models, contrastive learning is an instance-wise discriminative approach that aims at making similar instances closer and dissimilar instances far from each other in representation space [20], [21].”
Liu, Y., Wen, W., Bai, Y., & Meng, Q. (2022) · Measurement · doi:10.1016/j.measurement.2023.112551 · Cited by 40
Yu, A., Cai, B., Wu, Q., García, M. M., Li, J., & Chen, X. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110181 · Cited by 36
Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis
Morano, J., Fazekas, B., Sükei, E., Fecso, R., Emre, T., Gumpinger, M., Faustmann, G., Oghbaie, M., Schmidt-Erfurth, U. M., & Bogunović, H. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01852-3 · Cited by 57
“SSL methods train models on unlabeled datasets using pretext tasks such as masked autoencoding (MAE) 18 , contrastive learning 19,20 , and self-distillation 21 .”
Yao, Y., Feng, J. Y., & Liu, Y. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3369704 · Cited by 35
“Recently, contrastive learning (CL) [21], which belongs to the family of self-supervised learning, has attracted much attention in the computer vision field.”
BackgroundOut-of-Distribution Detection and Radiological Data Monitoring Using Statistical Process Control
Zamzmi, G., Venkatesh, K., Nelson, B., Prathapan, S., Yi, P. H., Sahiner, B., & Delfino, J. G. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01212-9 · Cited by 39
“We chose contrastive learning as a training method for its proven effectiveness in learning discriminative features [47].”
Investigating the Role of Negatives in Contrastive Representation LearningPreprint
Ash, J. T., Goel, S., Krishnamurthy, A., & Misra, D. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09943 · Cited by 41
“…machine learning tasks, including computer vision [He et al., 2020, Huo et al., 2020, Khosla et al., 2020, Wang et al., 2020, Bachman et al., 2019], natural language processing [Fang and Xie, 2020, Fu et al., 2021, Le-Khac et al., 2020, Giorgi et al., 2020], and deep reinforcement learning.”
BackgroundRethinking 360° Image Visual Attention Modelling with Unsupervised Learning
Djilali, Y. A. D., Krishna, T., McGuinness, K., & O'Connor, N. E. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01513 · Cited by 39
Ahuja, C. K., & Sethia, D. (2024) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2024.1421922 · Cited by 41
Contrastive Coding for Active Learning under Class Distribution Mismatch
Du, P., Zhao, S., Chen, H., Chai, S., Chen, H., & Li, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00880 · Cited by 50
Contrastive learning of heart and lung sounds for label-efficient diagnosis
Soni, P. N., Shi, S., Sriram, P. R., Ng, A. Y., & Rajpurkar, P. (2021) · Patterns · doi:10.1016/j.patter.2021.100400 · Cited by 38
“This approach has been applied with success across domains, including for natural language processing 23–25 and computer vision 1,2,26,27 tasks.”
MethodMultimodal learning on graphs for disease relation extraction
Lin, Y., Lu, K., Yu, S., Cai, T., & Žitnik, M. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104415 · Cited by 45
NORESQA: A Framework for Speech Quality Assessment using Non-Matching ReferencesPreprint
Manocha, P., Xu, B., & Kumar, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08125 · Cited by 59
Lionakis, E., Κarampidis, K., & Papadourakis, G. M. (2023) · Multimodal Technologies and Interaction · doi:10.3390/mti7100095 · Cited by 36
Detection of Bone Metastases on Bone Scans through Image Classification with Contrastive Learning
Hsieh, T., Liao, C.-W., Lai, Y.-C., Law, K.-M., Chan, P.-K., & Kao, C. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11121248 · Cited by 37
“CRL CRL is a novel approach to learning representation, allowing models to learn to distinguish similar or dissimilar images by effectively exploiting the semantic relationships between groups of samples and mining higher-level information from input images [21].”
Background
Metadata
- Authors
- Le-Khac, P. H., Healy, G., & Smeaton, A. F.
- Affiliations
- Dublin City University (IE)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 193907-193934
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3096655658
- Cited by
- 862
- References
- 144
- Citations by year
- 2026: 111 · 2025: 197 · 2024: 212 · 2023: 157 · 2022: 113 · 2021: 71 · 2020: 1
- FWCI
- 33.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- arXiv (Cornell University) (acceptedVersion)
- Dublin City University Open Access Institutional Repository (Dublin City University) (submittedVersion, cc-by)
- Arrow@dit (Dublin Institute of Technology) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Science Foundation Ireland through the Science Foundation Ireland (SFI) Centre for Research Training in Machine Learning (18/CRT/6183)
- Insight Centre for Data Analytics (SFI/12/RC/2289_P2)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- contrastive learning, representation learning, self-supervised learning, metric learning, inductive bias, computer vision, natural language processing, audio signal processing, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.