Contrastive Representation Learning: A Framework and Review
Open accessThe DOI matches its Crossref record; 147 of the 162 references it cites were found in a registry or an index, and 15 are not checked yet.
Abstract
Contrastive Learning has recently received interest due to its success in self-supervised representation learning in the computer vision domain. However, the origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields and domains including Metric Learning and natural language processing. In this paper, we provide a comprehensive literature review and we propose a general Contrastive Representation Learning framework that simplifies and unifies many different contrastive learning methods. We also provide a taxonomy for each of the components of contrastive learning in order to summarise it and distinguish it from other forms of machine learning. We then discuss the inductive biases which are present in any contrastive learning system and we analyse our framework under different views from various sub-fields of Machine Learning. Examples of how contrastive learning has been applied in computer vision, natural language processing, audio processing, and others, as well as in Reinforcement Learning are also presented. Finally, we discuss the challenges and some of the most promising future research directions ahead.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“According to the suggestions of Le-Khac et al. [29], the main methods can be divided into the following categories: multisensory signals, data augmentation, localglobal consistency, and temporal consistency.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [29]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Contrastive learning [56-58] is in fact a training procedure used in recent artificial intelligence models as a way to efficiently learn semantic representations.”
From How Adversarial REM Dreams May Facilitate Creativity, and Why We Become Aware of Them (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“As summarized in [29], commonly used contrastive loss functions generally include scoring functions (cosine similarity), energy-based margin functions (pair loss and triplet loss), probabilistic NCE-based functions, and mutual information based functions.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“In SimCLRv2 [17], empirical experiments show that the output of the second layer of a 3-layer MLP projection head is a better representation for transfer learning than the output of the base ResNet [42] encoder.”
BackgroundMethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“Recently, the Bidirectional Encoder Representation from Transformer (BERT) [24] model learns bidirectional word representations using the Transformer architecture’s decoder [103] and demonstrated great performance for transfer learning in multiple downstream tasks.”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Even though the BERT [24] text encoder and the Faster RCNN [87] object detection model are trained separately and not updated by the contrastive loss, the LSTM cells [48] and a linear project head can still map between words and object representations.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“The discretised speech representation can then be used directly as a representation for other models that expect discrete input such as BERT [24].”
MethodInfluentialFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×
“The discretised speech representation can then be used directly as a representation for other models that expect discrete input such as BERT [24].”
MethodInfluentialEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Research works printed but not linked: 1
- stratos, 2019, Noise contrastive estimation
Cited by
Items 101–200 of 948
Rating Distribution Calibration for Selection Bias Mitigation in Recommendations
Liu, H., Tang, D. Q., Yang, J., Zhao, X., Liu, H., Tang, J., & Cheng, Y. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512078 · Cited by 51
“As an emerging technology for representation learning via unlabeled data, self-supervised learning (SSL) has drawn increasing attention from various communities [1, 12, 14, 21, 29, 30].”
BackgroundContrastive Learning-Based Embedder for the Representation of Tandem Mass Spectra
Guo, H., Xue, K., Sun, H., Jiang, W., & Pu, S. (2023) · Analytical Chemistry · doi:10.1021/acs.analchem.3c00260 · Cited by 38
Camouflaged Object Segmentation Based on Joint Salient Object for Contrastive Learning
Jiang, X., Cai, W., Ding, Y., Wang, X., Hong, D., Yang, Z., & Gao, W. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3306520 · Cited by 37
Logacjov, A., & Bach, K. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107478 · Cited by 41
Pre-trained Online Contrastive Learning for Insurance Fraud Detection
Zhang, R., Cheng, D., Yang, J., Ouyang, Y., Wu, X., Zheng, Y., & Jiang, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30259 · Cited by 49
“As illuminated in (Le-Khac, Healy, and Smeaton 2020), contrastive learning serves as a powerful tool in discovering complex patterns, subsequently improving the precision of fraud detection.”
BackgroundOn the use of contrastive learning for standard-plane classification in fetal ultrasound imaging
Migliorelli, G., Fiorentino, M. C., Cosmo, M. D., Villani, F. P., Mancini, A., & Moccia, S. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108430 · Cited by 36
Content-based Graph Reconstruction for Cold-start Item Recommendation
Kim, J., Kim, E., Yeo, K., Jeon, Y., Kim, C., Lee, S., & Lee, J. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657801 · Cited by 49
“We employ the contrastive loss [20, 34], a prevalent approach in representation learning.”
Wang, W., Li, C., Zhang, Z., Chen, J., He, S., & Feng, Y. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110650 · Cited by 34
Electricity frauds detection in Low-voltage networks with contrastive predictive coding
Fei, K., Li, Q., Zhu, C., Dong, M., & Li, Y. (2021) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2021.107715 · Cited by 34
Deep Autoencoder-like NMF with Contrastive Regularization and Feature Relationship Preservation
Salahian, N., Tab, F. A., Seyedi, S. A., & Chavoshinejad, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119051 · Cited by 31
Getting the bugs out of AI : Advancing ecological research on arthropods through computer vision
Schneider, S., Taylor, G. W., Kremer, S. C., & Fryxell, J. M. (2023) · Ecology Letters · doi:10.1111/ele.14239 · Cited by 34
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
Dixit, T., Paranjape, B., Hajishirzi, H., & Zettlemoyer, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.216 · Cited by 58
Unsupervised Deraining: Where Asymmetric Contrastive Learning Meets Self-Similarity
Chang, Y., Guo, Y., Ye, Y., Yu, C., Zhu, L., Zhao, X.-L., Yan, L., & Tian, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3321311 · Cited by 34
“It has been recognized that for different CL tasks, the choice of the encoder would vastly influence the final performance [42].”
BackgroundSundaram, S. S., Gurajada, S., Padmanabhan, D., Abraham, S. S., & Fisichella, M. (2024) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1534 · Cited by 35
“In the context of word problem solving, a few bespoke transformer based encoder-decoder models (Hong et al., 2021; Li, Zhang, et al., 2021) have been proposed; these seek to effectively leverage contrastive learning (Le-Khac et al., 2020).”
How Transferable Are Self-supervised Features in Medical Image Classification Tasks?Preprint
Truong, T., Mohammadi, S., & Lenga, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10048 · Cited by 66
“A detailed review and taxonomy of contrastive learning can be found in (Le-Khac et al., 2020).”
BackgroundLM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning
Abaskohi, A., Rothe, S., & Yaghoobzadeh, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.59 · Cited by 47
“Contrastive learning has been utilized for various tasks in deep learning (Le-Khac et al., 2020; Conde and Turgutlu, 2021; Abaskohi et al., 2022); however, most NLP data augmentation methods may influence semantics which re-sults in limited improvement.”
BackgroundA brief review of contrastive learning applied to astrophysics
Sarmiento, R., & Knapen, J. H. (2023) · RAS Techniques and Instruments · doi:10.1093/rasti/rzad028 · Cited by 38
Probing Contextual Language Models for Common Ground with Visual Representations
Ilharco, G., Zellers, R., Farhadi, A., & Hajishirzi, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.422 · Cited by 37
“4 This is done via InfoNCE (Oord et al., 2018), a loss function commonly used for retrieval and contrastive learning (Le-Khac et al., 2020).”
MethodSükei, E., Rumetshofer, E., Schmidinger, N., Mayr, A., Schmidt-Erfurth, U. M., Klambauer, G., & Bogunović, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-78515-y · Cited by 38
Yang, L., Allen, G., Zhang, Z., & Zhao, Y. (2024) · Buildings · doi:10.3390/buildings15010021 · Cited by 33
Hierarchical Consensus-Based Multi-Agent Reinforcement Learning for Multi-Robot Cooperation Tasks
Feng, P., Liang, J., Wang, S., Yu, X., Ji, X., Chen, Y.-T., Zhang, K., Shi, R., & Wu, W. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802212 · Cited by 45
“Adapting ideas from contrastive learning [12], we first map local observations into discrete latent spaces as forms of invariances using the consensus builder.”
Efficient Anomaly Detection Using Self-Supervised Multi-Cue Tasks
Jezequel, L., Vu, N., Beaudet, J., & Histace, A. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3231532 · Cited by 32
“More recently, the contrastive learning framework [33] has been extensively used for self-supervised representation learning.”
BackgroundRanking-based contrastive loss for recommendation systems
Tang, H., Zhao, G., He, Y., Wu, Y., & Qian, X. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110180 · Cited by 33
Contrastive Feature Disentangling for Partial Aspect Angles SAR Noncooperative Target Recognition
Wen, Z., Liu, J., Liu, Z., Li, S., & Pan, Q. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3265915 · Cited by 28
“As mentioned above, G (cid:50) acting as the contextualization head [46] will generate a blurred image G (cid:50) ( r j , f j ) defined as the imaginary counterpart of x j .”
BackgroundMethodInfluentialUnsupervised Structure-Adaptive Graph Contrastive Learning
Zhao, H., Yang, X., Deng, C. D., & Tao, D. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3271140 · Cited by 30
“Le-Khac Le-Khac et al. [36] proposes a general contrastive learning framework to unify numerous contrastive learning methods.”
A electricity theft detection method through contrastive learning in smart grid
Liu, Z., Ding, W., Chen, T., Sun, M., Cai, H., & Liu, C. (2023) · EURASIP Journal on Wireless Communications and Networking · doi:10.1186/s13638-023-02258-z · Cited by 30
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 27
“It has been successful in computer vision, natural language processing and reinforcement learning [136,165].”
MethodHeterogeneous Contrastive Learning for Foundation Models and Beyond
Zheng, L., Jing, B., Li, Z., Tong, H., & He, J. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671454 · Cited by 55
Clustering FunFams using sequence embeddings improves EC purity
Littmann, M., Bordin, N., Heinzinger, M., Schütze, K., Dallago, C., Orengo, C., & Rost, B. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab371 · Cited by 26
SCEHR: Supervised Contrastive Learning for Clinical Risk Prediction using Electronic Health Records
Zang, C., & Wang, F. (2021) · Proceedings · doi:10.1109/icdm51629.2021.00097 · Cited by 36
Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCEPreprint
Chen, J., Gan, Z., Li, X., Guo, Q., Chen, L., Gao, S., Chung, T., Xu, Y., Zeng, B., Lu, W., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.01152 · Cited by 35
“Fueled by the discovery of efficient algorithms and strong performance [1], extensive research has been devoted to this active topic [20].”
BackgroundGoals as reward-producing programs
Davidson, G., Todd, G., Togelius, J., Gureckis, T. M., & Lake, B. M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00981-4 · Cited by 35
“In this implementation, parameter learning of feature weights θ proceeds in a contrastive fashion 26,27 by optimizing for the difference in scores between our set of human-generated games and a substantially larger set of corrupted (that is, lower quality) games obtained through random tree…”
Drone Landing and Reinforcement Learning: State-of-Art, Challenges and Opportunities
Amendola, J., Cenkeramaddi, L. R., & Jha, A. (2024) · IEEE Open Journal of Intelligent Transportation Systems · doi:10.1109/ojits.2024.3444487 · Cited by 32
“In the context of RL, contrastive learning [82] involves training a model to discriminate between positive and negative pairs of state-action trajectories [83], [84].”
BackgroundShrivastava, A. D., & Kell, D. B. (2021) · Molecules · doi:10.3390/molecules26072065 · Cited by 25
Evaluating Contrastive Models for Instance-based Image Retrieval
Krishna, T., McGuinness, K., & O'Connor, N. E. (2021) · International Conference on Multimedia Retrieval · doi:10.1145/3460426.3463585 · Cited by 27
“This comparison is performed between positive pairs of “similar” and negative pairs of “dissimilar” inputs, which is achieved via a contrastive loss [8, 23] derived from Noise Contrastive Estimation (NCE) [16].”
BackgroundSelfPAB: large-scale pre-training on accelerometer data for human activity recognition
Logacjov, A., Herland, S., Ustad, A., & Bach, K. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05322-3 · Cited by 35
“The representations of "similar" samples (positive samples) need to be closer together than the representations of "dissimilar" samples (negative samples)[22].”
Contrastive Fine-tuning Improves Robustness for Neural Rankers
Ma, X., dos Santos, C. N., & Arnold, A. O. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.51 · Cited by 33
Imbalanced classification for protein subcellular localization with multilabel oversampling
Rana, P., Sowmya, A., Meijering, E. H. W., & Song, Y. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac841 · Cited by 26
An Overview of the Vision-Based Human Action Recognition Field
Camarena, F., González-Mendoza, M., Chang, L., & Cuevas-Ascencio, R. J. (2023) · Mathematical and Computational Applications · doi:10.3390/mca28020061 · Cited by 32
“On the other hand, contrastive learning for SSL [162] aims to identify differences between video samples by projecting them onto a shared feature space where clips from the same distribution are clustered together based on a distance metric.”
BackgroundTGC: Transaction Graph Contrast Network for Ethereum Phishing Scam Detection
Li, S., Gou, G., Liu, C., Xiong, G., Li, Z., Xiao, J., & Xing, X. (2023) · Annual Computer Security Applications Conference · doi:10.1145/3627106.3627109 · Cited by 36
Masked pre-training of transformers for histology image analysis
Jiang, S., Hondelink, L. M., Suriawinata, A. A., & Hassanpour, S. (2024) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2024.100386 · Cited by 29
“The second component is a modified version of the InfoNCE loss, which can minimize the similarity between an output instance and all other selected input feature sequences [39,40].”
BackgroundWang, M.-H., Li, Z., Wang, J., Zou, W., Zhou, J., & Gan, J. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112218 · Cited by 25
Lovell, C. C., Starkenburg, T., Ho, M., Anglés‐Alcázar, D., Davé, R., Gabrielpillai, A., Iyer, K. G., Matthews, A. E., Roper, W. J., Somerville, R. S., et al. (2025) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/staf1888 · Cited by 33
Lastilla, L., Ammirati, S., Firmani, D., Komodakis, N., Merialdo, P., & Scardapane, S. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.102875 · Cited by 29
Transfer Adaptation Learning for Target Recognition in SAR Images: A Survey
Yang, X., Jiao, L., & Pan, Q. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3434448 · Cited by 27
“Integratingmetric learning[180]andcontrastivelearning[181]toconstructa shared feature space while preserving distinguishing features between classes can be a solution to address the scarcity of discrimination in the future.”
BackgroundContrastive Self-Supervised Learning for Sensor-Based Human Activity Recognition: A Review
Chen, H., Gouin-Vallerand, C., Bouchard, K., Gaboury, S., Couture, M., Bier, N., & Giroux, S. (2024) · IEEE Access · doi:10.1109/access.2024.3480814 · Cited by 29
“Self-supervised learning (SSL) is an emerging paradigm that learns meaningful representations from massive unlabeled data, reducing the dependency on manual annotations [8].”
BackgroundEnriched music representations with multiple cross-modal contrastive learning
Ferraro, A., Favory, X., Drossos, K., Kim, Y., & Bogdanov, D. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3071082 · Cited by 23
“Contrastive learning has gained popularity in the last years [10].”
BackgroundUnsupervised Learning of Disentangled Representation via Auto-Encoding: A Survey
Eddahmani, I., Pham, C.-H., Napoléon, T., Badoc, I., Fouefack, J.-R., & Elbouz, M. (2023) · Sensors · doi:10.3390/s23042362 · Cited by 27
“Representation learning refers to finding a low-dimensional representation that captures true underlying factors of variation that explain the data [6].”
BackgroundVisible-to-Infrared Image Translation for Matching Tasks
Ma, D., Li, S., Su, J., Xian, Y., & Zhang, T. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3468456 · Cited by 30
“Contrastive learning is often used in self-supervised learning [58], aiming to, respectively, narrow and expand the distance between positive and negative examples [59].”
BackgroundGraphCLIP: Image-graph contrastive learning for multimodal artwork classification
Scaringi, R., Fiameni, G., Vessio, G., & Castellano, G. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112857 · Cited by 25
Hu, H., He, C., & Li, P. (2021) · IEEE International Test Conference (ITC) · doi:10.1109/itc50571.2021.00019 · Cited by 22
“Recently, a novel self-supervised learning framework called contrastive learning suggests extracting a good data representation by learning the similarity among samples [8]–[10].”
BackgroundMulti-Level Transitional Contrast Learning for Personalized Image Aesthetics Assessment
Yang, Z., Li, L., Yang, Y., Li, Y., & Lin, W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3290479 · Cited by 29
“Contrast learning [34] is a representation learning technique that learns by comparing among different samples.”
BackgroundZhang, R., Wu, C., Yang, Q., Liu, C., Wang, Y., Li, K., Huang, L., & Zhou, F. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae118 · Cited by 32
“The fundamental principle of contrastive learning involves guiding the model to generate proximal embed-dings for samples within the same class, while distancing those between different classes in the embedding space (Schroff et al. 2015, Le-Khac et al. 2020).”
BackgroundThe Promising Role of Representation Learning for Distributed Computing Continuum Systems
Donta, P. K., & Dustdar, S. (2022) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose55356.2022.00021 · Cited by 22
“When working with the unsupervised data, the contrastive ReL (CRL) can predict the labels and works like a self-supervised learning approach [21], [22].”
MethodCherqi, O., Moukafih, Y., Ghogho, M., & Benbrahim, H. (2023) · IEEE Access · doi:10.1109/access.2023.3299604 · Cited by 21
“This technique has primarily been used in the field of computer vision, where it has proven to be effective for tasks like image classification and object detection, [23] and in graph representation [24].”
MethodTu, L., Li, J., Huang, X., Gong, J., Xie, X., & Wang, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3392962 · Cited by 24
“The goal of contrastive learning is to pull together the representations of different views of the same image (i.e., positive samples) and push apart those of different images (i.e., negative samples) [25].”
BackgroundContrastive Learning Over Random Fourier Features for IoT Network Intrusion Detection
López‐Martín, M., Sánchez-Esguevillas, A., Arribas, J. I., & Carro, B. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3214758 · Cited by 26
“The similarity score is obtained with a contrastive learning scheme [6], where two embeddings will be compared (contrasted) and assigned a similarity value which can be latter used in classification tasks.”
BackgroundMethodBoosting Zero-Shot Learning via Contrastive Optimization of Attribute Representations
Du, Y. H., Shi, M., Wei, F., & Li, G. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3297134 · Cited by 25
“The goal of contrastive learning is to learn an embedding space in which similar samples are pushed close and dissimilar ones are pulled away [43].”
BackgroundUnderstanding Negative Samples in Instance Discriminative Self-supervised Representation LearningPreprint
Nozawa, K., & Sato, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06866 · Cited by 47
Towards Galaxy Foundation Models with Hybrid Contrastive LearningPreprint
Walmsley, M., Slijepcevic, I. V., Bowles, M., & Scaife, A. M. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.11927 · Cited by 29
Azher, Z. L., Suvarna, A., Chen, J.-Q., Zhang, Z., Christensen, B. C., Salas, L. A., Vaickus, L. J., & Levy, J. J. (2023) · BioData Mining · doi:10.1186/s13040-023-00338-w · Cited by 20
“Other unimodal pretraining strategies include the usage of variational autoencoders (VAEs) [24], transfer learning [25], and contrastive learning [26].”
BackgroundContrastive Learning with Heterogeneous Graph Attention Networks on Short Text Classification
Sun, Z., Harit, A., Cristea, A. I., Yu, J., Shi, L., & Moubayed, N. A. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892257 · Cited by 34
“Comprehensive information about contrastive learning can be found in [29] and [19].”
BackgroundChan, L., Kumar, R., Verdonk, M. L., & Poelking, C. R. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00564-7 · Cited by 21
Clinically Labeled Contrastive Learning for OCT Biomarker Classification
Kokilepersaud, K., Corona, S. T., Prabhushankar, M., AlRegib, G., & Wykoff, C. C. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3277789 · Cited by 26
“CONTRASTIVE learning [1] refers to a family of selfsupervised algorithms that leverages differences and similarities between data points in order to extract useful”
Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text ClassificationPreprint
Suresh, V., & Ong, D. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05427 · Cited by 32
“Contrastive learning has been widely used in computer vision, especially in self-supervised settings (Le-Khac et al., 2020; Chen et al., 2020) where such learning guides the model based on similarities between the latent representation of the samples.”
BackgroundLi, F., Li, H., Li, Y., Wu, H., Fu, B., Ji, Y., Wang, C., & Shi, G. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac6a7d · Cited by 19
Combs, K., Lu, H., & Bihl, T. (2023) · Algorithms · doi:10.3390/a16030146 · Cited by 24
GeoCLR: Georeference Contrastive Learning for Efficient Seafloor Image Interpretation
Yamada, T., Prügel‐Bennett, A., Williams, S. B., Pizarro, O. R., & Thornton, B. (2022) · Field Robotics · doi:10.55417/fr.2022037 · Cited by 19
“Contrastive learning is a form of self-supervision that has demonstrated robust performance gains across many application areas in the image representation learning domain (Chen et al., 2020; Jing and Tian, 2020; Le-Khac et al., 2020).”
End-to-End Simultaneous Speech Translation with Differentiable Segmentation
Zhang, S., & Feng, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.485 · Cited by 33
“…that the cosine similarity L cos is better than the L 2 distance to measure the difference between the segment representation and the word representation (Le-Khac et al., 2020; Ye et al., 2022), and the contrastive learning L ctr further improves DiSeg performance by introducing negative examples.”
BackgroundGong, L., Lin, Y., Guo, S., Yan, L., Wang, T., Zheng, E., Zhou, Z., & Wan, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25546 · Cited by 27
“Actually, representation learning is always one of the hottest topics in the fields of spatial-temporal data mining (STDM) (Lin et al. 2021) (Wan et al. 2021), natural language processing (NLP) and computer vision (CV) (Le-Khac, Healy, and Smeaton 2020).”
BackgroundSparse discriminant PCA based on contrastive learning and class-specificity distribution
Zhou, Q., Gao, Q., Wang, Q., Yang, M., & Gao, X. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.08.061 · Cited by 18
Huang, X., Schmelter, F., Irshad, M. T., Piet, A., Nisar, M. A., Sina, C., & Grzegorzek, M. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107501 · Cited by 20
Adversarial Domain Alignment With Contrastive Learning for Hyperspectral Image Classification
Liu, F., Gao, W., Liu, J., Tang, X., & Xiao, L. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3317079 · Cited by 19
Reassessing Java Code Readability Models with a Human-Centered Approach
Sergeyuk, A., Lvova, O., Titov, S., Serova, A., Bagirov, F., Kirillova, E., & Bryksin, T. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644435 · Cited by 28
Leiby, J. S., Hao, J., Kang, G. H., Park, J. W., & Kim, D. (2022) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc48229.2022.9871553 · Cited by 21
“Contrastive learning is a form of SSL in which a model is trained to map data to informative embeddings by maximizing the similarity between certain embeddings and minimizing the similarity between others [18].”
BackgroundReMeDi: Resources for Multi-domain, Multi-service, Medical Dialogues
Yan, G., Pei, J., Ren, P., Ren, Z., Xin, X., Liang, H., de Rijke, M., & Chen, Z. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531809 · Cited by 37
“CL compares similarity and dissimilarity by positive/negative data sampling, and defining contrastive training objectives [23, 27].”
BackgroundSupervised Contrastive Learning for Affect Modelling
Pinitas, K., Makantasis, K., Liapis, A., & Yannakakis, G. N. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556584 · Cited by 22
“As a self-supervised learning method, it aims to project data into a space in which different views of the same input have similar representations [21].”
MethodPace-Adaptive and Noise-Resistant Contrastive Learning for Multimodal Feature Fusion
Guo, X., Kot, A. C., & Kong, A. W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3252270 · Cited by 19
“…on unimodal representation learning, the discrimination between positive and negative pairs relies on the hypothesis that modalities from the same scenes contain identical or highly positive correlated semantic information while modalities from different scenes are negative correlated [20].”
BackgroundContrastNER: Contrastive-based Prompt Tuning for Few-shot NER
Layegh, A., Payberah, A. H., Soylu, A., Roman, D., & Matskin, M. (2023) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00038 · Cited by 21
“Recent works have studied contrastive learning for visual representation [29] [30], graph representations [31], and a”
BackgroundLiu, Y., Zhang, Z., Qin, S., Salim, F. D., & Yepes, A. J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43427-3_26 · Cited by 20
Multi-IMU with Online Self-consistency for Freehand 3D Ultrasound Reconstruction
Luo, M., Yang, X., Yan, Z., Li, J., Zhang, Y., Chen, J., Hu, X., Qian, J., Cheng, J., & Ni, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43907-0_33 · Cited by 17
Disambiguated Node Classification with Graph Neural Networks
Zhao, T., Zhang, X. D., & Wang, S. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645637 · Cited by 31
“Recently, remarkable progress has been made to adapt Contrastive Learning (CL) techniques [7, 28, 40] for the graph domain, by constructing multiple graph views via augmentations and maximizing the mutual information between instances with similar semantics (positive samples) [33, 75].”
BackgroundApplication of machine learning in LC-MS-based non-targeted analysis
Jin, Z.-L., Chen, L., Wang, Y., Shi, C.-T., Zhou, Y., & Xia, B. (2025) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2025.118243 · Cited by 20
NewsEmbed: Modeling News through Pre-trained Document Representations
Liu, J., Liu, T., & Yu, C. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467392 · Cited by 20
“Inspired by contrastive learning initiated in computer vision [36], we believe text representations can be learned by enforcing similar elements to be equal and dissimilar elements to be different.”
BackgroundCL-ETC: A Contrastive Learning Method for Encrypted Traffic Classification
Zhao, Z., Guo, Y., Wang, J. H., Wang, H., Zhang, C., & An, C. (2022) · IFIP Networking Conference (IFIP Networking) · doi:10.23919/ifipnetworking55013.2022.9829767 · Cited by 18
“Good representations should capture features that are invariant to augmentation, and representations extracted by CL can meet the requirement [15].”
InfluentialYou, Y., Chang, S., Yang, Z., & Sun, Q. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1167723 · Cited by 19
Wang, J., Wang, H., Deng, J., Wu, W., & Zhang, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04699 · Cited by 18
“Contrastive learning (Hadsell, Chopra, and LeCun 2006; Le-Khac, Healy, and Smeaton 2020) is a kind of representation learning by contrasting positive pairs against negative pairs.”
BackgroundGeng, Z., Wang, Y., Pan, W., Yu, C., Bai, Z., & Zhang, H. (2025) · Communications Earth & Environment · doi:10.1038/s43247-025-02003-y · Cited by 18
“Ontheotherhand,MCU-Quakeemploysacontrastive learningapproach 33,34 to maximize differences between signal sources.”
Hu, H., Ye, R., Thiyagalingam, J., Coenen, F., & Su, J. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04458-y · Cited by 18
“Contrastive learning [29] is a self-supervised learning approach whose basic idea is to make base models perform certain auxiliary tasks based on temporal correspondence [30], and cross-modal consistency [31].”
BackgroundMovies2Scenes: Using Movie Metadata to Learn Scene Representation
Chen, S., Liu, C.-H., Hao, X., Nie, X., Arap, M., & Hamid, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00632 · Cited by 31
Petelin, G., Cenikj, G., & Eftimov, T. (2023) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2023.101448 · Cited by 18
Self-clustered GAN for precipitation nowcasting
An, S., Oh, T.-J., Kim, S., & Jung, J. J. (2024) · Scientific Reports · doi:10.1038/s41598-024-60253-w · Cited by 20
“Recognizing the issue of collapsing, many researchers have chosen to cluster GAN architectures, typically as generators or discriminators 12–14 based on self-supervised learning (SSL).”
Unsupervised mutual transformer learning for multi-gigapixel Whole Slide Image classification
Javed, S., Mahmood, A., Qaiser, T., Werghi, N., & Rajpoot, N. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103203 · Cited by 21
HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights
Gökdemir, O., Siebenschuh, C., Brace, A., Wells, A. I., Hsu, B., Hippe, K., Setty, P. V., Ajith, A., Pauloski, J. G., Sastry, V. K., et al. (2025) · Platform for Advanced Scientific Computing Conference (PASC) · doi:10.1145/3732775.3733586 · Cited by 27
“In particular, contrastive learning [46] has improved retrieval accuracy by better estimating the relevance of document passages to queries.”
Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast
Wang, L., Zhao, W., & Liu, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.309 · Cited by 17
Self-Supervised Contrastive Representation Learning in Computer Vision
Baştanlar, Y., & Orhan, S. (2022) · Artificial intelligence · doi:10.5772/intechopen.104785 · Cited by 20
Contrastive Deep Knowledge Tracing
Dai, H., Yun, Y., Zhang, Y., Zhang, W., & Shang, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-11647-6_54 · Cited by 16
Gao, X., Sun, Y., Xiao, Y., Gu, Y., Chai, S., & Chen, B. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118275 · Cited by 16
Open set classification of sound event
You, J., Wu, W., & Lee, J. (2024) · Scientific Reports · doi:10.1038/s41598-023-50639-7 · Cited by 17
“2b) minimizes the three types of losses: cross-entropy loss, center loss, and supervised contrastive loss: In this section, we describe the implementation of the three types of losses in detail.”
Supervised Contrastive Learning for Vehicle Classification Based on the IR-UWB Radar
Li, X., Zhang, S., Zhu, Y., Xiao, Z., & Chen, S. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3203468 · Cited by 15
“We use 226 a framework similar to the SupCon series [25] and improve 227 it for the characteristics of IR-UWB echoes.”
Method
Metadata
- Authors
- Le-Khac, P. H., Healy, G., & Smeaton, A. F.
- Affiliations
- Dublin City University (IE)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 193907-193934
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3096655658
- Cited by
- 862
- References
- 144
- Citations by year
- 2026: 111 · 2025: 197 · 2024: 212 · 2023: 157 · 2022: 113 · 2021: 71 · 2020: 1
- FWCI
- 33.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- arXiv (Cornell University) (acceptedVersion)
- Dublin City University Open Access Institutional Repository (Dublin City University) (submittedVersion, cc-by)
- Arrow@dit (Dublin Institute of Technology) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Science Foundation Ireland through the Science Foundation Ireland (SFI) Centre for Research Training in Machine Learning (18/CRT/6183)
- Insight Centre for Data Analytics (SFI/12/RC/2289_P2)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- contrastive learning, representation learning, self-supervised learning, metric learning, inductive bias, computer vision, natural language processing, audio signal processing, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.