The DOI matches its DataCite record; 53 of the 53 references it cites were found in a registry or an index.
Representation Learning with Contrastive Predictive Coding
Open accessAbstract
While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Contrastive Predictive Coding (CPC) [80,143] utilizes a contrastive future prediction task, and it was found that predicting one or two timesteps in the future is (too) easy and results in poor performance, whereas predicting multiple timesteps into the future is much more complex, and leads to very effective representations.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [143]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Using this contrastive loss, self-supervised networks learn to make the representations similar for positive pairs, and dissimilar for negative pairs, and hence are able to cluster similar (or positive) samples together and push apart dissimilar (or negative) samples.”
From Detecting galaxy tidal features using self-supervised representation learning (2024) · cites this paper as Oord et al. 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It means that InfoNCE loss maximizes the agreement of two different augmented sequences' representations from the same sequence while minimizing the agreement with other negative sequences.”
From Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-Supervised Action Recognition (2022) · cites this paper as Oord, Li, and Vinyals 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We introduce the Contrastive Predictive Coding (CPC) framework to human activity recognition, which captures the long-term temporal structure of sensor data streams.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As mentioned in [58], either 𝑐 𝑡 (the output of 𝑔 𝑒𝑛𝑐 ) or 𝑧 𝑡 (the output of 𝑔 𝑎𝑟 ) can be utilized as the representation.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The framework for data augmentation and batch formation plays a critical role in multi-view similarity learning [12,16,44,115], where we start by following the popular SimCLR [16] method.”
From MARs: Multi-view Attention Regularizations for Patch-based Feature Recognition of Space Terrain (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We note that this is a common setting for self-supervised contrastive learners with NCE objectives (Gutmann & Hyvärinen, 2010;Oord et al., 2018), including SimCLR (Chen et al., 2020) and SimCSE (Gao et al., 2021).”
From The Platonic Representation Hypothesis (2024) · cites this paper as Oord et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“The key insight behind this approach is that predicting just the next future timestep involves exploiting the local smoothness of the signal, whereas predicting multiple subsequent timesteps requires inferring the global structure of time-series data.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“Haresamudram et al. [80] adapted Contrastive Predictive Coding (CPC [143]) for HAR, emphasizing the importance of temporal structure in sensor data representation.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [143]
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive predictive coding (CPC) proposed by [109] provides a contrastive learning framework to perform the prediction task using InfoNCE loss.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“It can be seen that CPC does not directly predict future observations x t + k . Instead, it tries to preserve the mutual information of c t and x t + k .”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“This contrastive loss, or InfoNCE loss (Oord et al. 2018), is minimised when the similarity is high for positive pairs and low for negative pairs.”
From Detecting galaxy tidal features using self-supervised representation learning (2024) · cites this paper as Oord et al. 2018
The cited source says what the claim attributes to it.
In the citing paper
“To evaluate the performance of our TS-TCC model, we follow the standard linear benchmarking evaluation scheme [Oord et al. , 2018; Chen et al. , 2020]. Particularly, we train a linear classifier (single MLP layer) on top of a frozen self-supervised pretrained encoder model.”
From Time-Series Representation Learning via Temporal and Contextual Contrasting (2021) · cites this paper as Oord et al. , 2018
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive-based methods usually use a contrastive similarity metric, such as InfoNCE loss [109].”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“This results in representations that encode the high-level information between temporally separated parts of the time-series signal [58], thereby leading to improved downstream recognition performance.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive Predictive Coding (CPC) [58], which is the focus of this work, performs contrastive learning over multiple future timestep predictions.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“This framework has resulted in excellent representation learning performance across domains such as computer vision [12, 33, 35], natural language [49], speech recognition [6, 58] and time-series data [39].”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“If extra context from the past is useful for classification, then 𝑐 𝑡 can be used; else 𝑧 𝑡 may be more effective.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“SimCLR aims to maximize the learned representation similarity between augmented views of the same input.”
From MARs: Multi-view Attention Regularizations for Patch-based Feature Recognition of Space Terrain (2024) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“More recent advancements in autoregressive modeling such as masked convolutional architectures [18, 19] or self-attention networks [20] could help improve results further.”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“These techniques made manually specified features largely redundant and have greatly improved state-of-the-art in several real-world applications [1, 2, 3].”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We use the Adam optimizer [32] with a learning rate of 2e-4, and use 8 GPUs each with a minibatch of 8 examples from which the negative samples in the contrastive loss are drawn.”
MethodImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“In our visual representation experiments we use the ILSVRC ImageNet competition dataset [34].”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1×
“We did not use Batch-Norm [38].”
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 2×
“In Word2Vec [9] neighbouring words are predicted using a contrastive loss.”
BackgroundLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“These techniques made manually specified features largely redundant and have greatly improved state-of-the-art in several real-world applications [1, 2, 3].”
BackgroundSelf-citationFaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · cited 1×
“For example, the techniques proposed by [21, 22, 23] were based on triplet losses using a max-margin approach to separate positive from negative examples.”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“[41] is the same skip-thought model with layer normalization trained for 1M iterations. evaluate with 10-fold cross-validation for MR, CR, Subj, MPQA and use the train/test split for TREC.”
Method
Research works printed but not linked: 4
- [12][12] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages 297-304, 2010. -
- [31][31] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding . IEEE Signal Processing Society, 2011. -
- [33][33] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(Nov):2579-2605, 2008. -
- [48][48] Sida Wang and Christopher D. Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2 , ACL '12, pages 90-94, 2012. -
Cited by
Sorted by citations
Items 501–600 of 1,066
ClusterFit: Improving Generalization of Visual Representations
Yan, X., Misra, I., Gupta, A. K., Ghadiyaram, D., & Mahajan, D. K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00654 · Cited by 109
CuCo: Graph Representation with Curriculum Contrastive Learning
Chu, G., Wang, X., Shi, C., & Jiang, X. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/317 · Cited by 102
Huang, X., Dong, M., Li, J., & Xian, G. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3202036 · Cited by 101
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual RepresentationsPreprint
Giorgi, J., Nitski, O., Wang, B., & Bader, G. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03659 · Cited by 85
The Conditional Entropy Bottleneck
Fischer, I. (2020) · Entropy · doi:10.3390/e22090999 · Cited by 102
CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding
Wang, D., Ding, N., Li, P., & Zheng, H.-T. (2021) · doi:10.18653/v1/2021.acl-long.181 · Cited by 111
Improved Text Classification via Contrastive Adversarial Training
Pan, L., Hang, C.-W., Sil, A., & Potdar, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21362 · Cited by 105
Contrastive Regression for Domain Adaptation on Gaze Estimation
Wang, Y., Jiang, Y., Li, J., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01877 · Cited by 95
BatchFormer: Learning to Explore Sample Relationships for Robust Representation Learning
Hou, Z., Yu, B., & Tao, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00711 · Cited by 109
PointContrast: Unsupervised Pre-training for 3D Point Cloud UnderstandingPreprint
Xie, S., Gu, J., Guo, D., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.10985 · Cited by 95
Pretrained Transformers as Universal Computation EnginesPreprint
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05247 · Cited by 95
Huang, J., Guan, D., Xiao, A., & Lu, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03374 · Cited by 144
Sanghi, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58526-6_37 · Cited by 91
Wang, J., Gao, Y., Li, K., Lin, Y., Ma, A. J., Cheng, H., Peng, P., Huang, F., Ji, R., & Sun, X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01163 · Cited by 99
Change-Aware Sampling and Contrastive Learning for Satellite Images
Mall, U., Hariharan, B., & Bala, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00509 · Cited by 110
Wasserstein Contrastive Representation Distillation
Chen, L., Wang, D., Gan, Z., Liu, J., Henao, R., & Carin, L. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01603 · Cited by 108
Blind Image Quality Assessment via Cross-View Consistency
Zhu, Y., Li, Y., Sun, W., Min, X., Zhai, G., & Yang, X. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3224319 · Cited by 98
Self-citationSiLK: Simple Learned Keypoints
Gleize, P., Wang, W., & Feiszli, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02056 · Cited by 111
“The probabilistic formulation, similar to InfoNCE [35], gives us a clean way to reason about matching, and the abundant supply of hard negative examples (from pixels in the same image) makes it an attractive choice.”
MethodLuo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Chen, X., & Zhou, M. (2020) · arXiv (Cornell University) · Cited by 82
Federated unsupervised representation learning
Zhang, F., Kuang, K., Chen, L., You, Z., Shen, T., Xiao, J., Zhang⋆, Y., Wu, C., Wu, F. Q., Zhuang, Y., et al. (2023) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2200268 · Cited by 166
Broaden Your Views for Self-Supervised Video Learning
Recasens, A., Luc, P., Alayrac, J.-B., Wang, L., Strub, F., Tallec, C., Malinowski, M., Pătrăucean, V., Altché, F., Valko, M., et al. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00129 · Cited by 98
Self-citationWu, H., Wang, Z., Song, Y., Yang, L., & Qin, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01137 · Cited by 95
A Self-Supervised Descriptor for Image Copy Detection
Pizzi, E., Roy, S. D., Ravindra, S. N., Goyal, P., & Douze, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01413 · Cited by 101
Progressive Instance-Aware Feature Learning for Compositional Action Recognition
Yan, R., Xie, L., Shu, X., Zhang, L., & Tang, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3261659 · Cited by 117
“For example, Oord et al. [48] proposed Contrastive Predictive Coding to learn effective video representations by predicting the future frame-level feature in latent space.”
BackgroundMethodUKSSL: Underlying Knowledge Based Semi-Supervised Learning for Medical Image Classification
Ren, Z., Kong, X., Zhang, Y., & Wang, S. (2023) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2023.3305190 · Cited by 94
“This loss function is termed by Chen et al. [28] as NT-Xent (the normalized temperature-scaled cross-entropy loss), and it has been widely used in lots of works [6], [7], [30].”
BackgroundMethodWhat Should Not Be Contrastive in Contrastive LearningPreprint
Xiao, T., Wang, X.-L., Efros, A. A., & Darrell, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05659 · Cited by 208
Understanding Dimensional Collapse in Contrastive Self-supervised LearningPreprint
Jing, L., Vincent, P., LeCun, Y., & Tian, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09348 · Cited by 128
Vector-quantized Image Modeling with Improved VQGANPreprint
Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., & Wu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04627 · Cited by 557
Cluster Attention Contrast for Video Anomaly Detection
Wang, Z., Zou, Y., & Zhang, Z. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413529 · Cited by 94
DivCo: Diverse Conditional Image Synthesis via Contrastive Generative Adversarial Network
Liu, R. Z. Z., Ge, Y., Choi, C. L., Wang, X., & Li, H. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01611 · Cited by 102
Cha, J., Mun, J., & Roh, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01074 · Cited by 108
“Existing methods learn models parametrized by θ to maximize the mutual information between paired images and texts [22, 23] as follows:”
Wang, W., Zhou, W., Bao, J., Chen, D., & Li, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01376 · Cited by 96
Self-supervised contrastive learning on agricultural images
Güldenring, R., & Nalpantidis, L. (2021) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2021.106510 · Cited by 95
Nguyen, V. N., Hu, Y., Xiao, Y., Salzmann, M., & Lepetit, V. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00665 · Cited by 93
Dive into the details of self-supervised learning for medical image analysis
Zhang, C., Zheng, H., & Gu, Y. (2023) · Medical Image Anal. · doi:10.1016/j.media.2023.102879 · Cited by 101
Large-Scale Unsupervised Semantic Segmentation
Gao, S., Li, Z.-Y., Yang, M.-H., Cheng, M.-M., Han, J., & Torr, P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3218275 · Cited by 88
COCOA: Cross Modality Contrastive Learning for Sensor Data
Deldari, S., Xue, H., Saeed, A., Smith, D. V., & Salim, F. D. (2022) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3550316 · Cited by 92
“During the last two years there was a rapid progress in contrastive learning methods in the eld of computer vision [11, 12, 32, 38, 85], audio and speech processing [32, 50, 54, 62, 82, 90], and natural language processing domain [9, 20, 23, 25, 28, 54, 89, 91].”
BackgroundMethodInfluentialTransferable Representation Learning in Vision-and-Language Navigation
Huang, H., Jain, V., Mehta, H., Ku, A., Magalhaes, G., Baldridge, J., & Ie, E. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00750 · Cited by 82
Deep clustering with a Dynamic Autoencoder: From reconstruction towards centroids construction
Mrabah, N., Khan, N., Ksantini, R., & Lachiri, Z. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.07.005 · Cited by 89
Siamese Image Modeling for Self-Supervised Vision Representation Learning
Tao, C., Zhu, X., Su, W., Huang, G. M., Li, B., Zhou, J., Qiao, Y., Wang, X., & Dai, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00212 · Cited by 118
Oops! Predicting Unintentional Action in Video
Epstein, D., Chen, B., & Vondrick, C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00100 · Cited by 82
Zhong, A., Li, X., Wu, D., Ren, H., Kim, K., Kim, Y.-G., Buch, V., Neumark, N., Bizzo, B. C., Tak, W. Y., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.101993 · Cited by 87
Recanatesi, S., Farrell, M., Lajoie, G., Denève, S., Rigotti, M., & Shea‐Brown, E. (2021) · Nature Communications · doi:10.1038/s41467-021-21696-1 · Cited by 104
Self-supervised Contrastive Learning for EEG-based Sleep Staging
Jiang, X., Zhao, J., Du, B., & Yuan, Z. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533305 · Cited by 95
Relating EEG to continuous speech using deep neural networks: a review
Puffay, C., Accou, B., Bollens, L., Monesi, M. J., Vanthornhout, J., hamme, H. V., & Francart, T. (2023) · Journal of Neural Engineering · doi:10.1088/1741-2552/ace73f · Cited by 80
“The main idea is that the negative samples should be what is called ”hard negatives” in the deep learning and contrastive learning literature (van den Oord et al., 2018).”
A Survey on Time-Series Pre-Trained Models
Ma, Q., Liu, Z., Zheng, Z., Huang, Z., Zhu, S., Yu, Z., & Kwok, J. T.-Y. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3475809 · Cited by 133
“For example, Oordet al. [108] proposedcontrastive predictive coding by employing model-predicted timesteps as positive samples and randomly-sampled timesteps as negative samples.”
BackgroundMethodCollaborative Unsupervised Visual Representation Learning from Decentralized Data
Zhuang, W., Gan, X., Wen, Y., Zhang, S., & Yi, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00487 · Cited by 96
SAM: Self-Supervised Learning of Pixel-Wise Anatomical Embeddings in Radiological Images
Yan, K., Cai, J., Jin, D., Miao, S., Guo, D., Harrison, A. P., Tang, Y., Xiao, J., Lu, J., & Lu, L. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3169003 · Cited by 95
MAE-AST: Masked Autoencoding Audio Spectrogram Transformer
Baade, A., Peng, P., & Harwath, D. F. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10961 · Cited by 125
Quality-aware Pretrained Models for Blind Image Quality Assessment
Zhao, K., Yuan, K., Sun, M. B., Li, M., & Wen, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02136 · Cited by 110
“Especially, InfoNCE [66] has become the most popular choice [5, 9, 27].”
BackgroundChaitanya, K., Erdil, E., Karani, N., & Konukoglu, E. (2020) · Neural Information Processing Systems · doi:10.3929/ethz-b-000443425 · Cited by 720
Contrastive Self-supervised Learning for Sensor-based Human Activity Recognition
Khaertdinov, B., Ghaleb, E., & Asteriadis, S. (2021) · 2021 IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb52358.2021.9484410 · Cited by 95
Brain-Inspired Computing: A Systematic Survey and Future Trends
Li, G., Deng, L., Tang, H., Pan, G., Tian, Y., Roy, K., & Maass, W. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3429360 · Cited by 113
“In addition, a second line of research turns to contrastive predictive coding for assistance [281], aiming to learn robust representations in a local self-supervised manner.”
Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction
Wang, W., Tang, Q., & Livescu, K. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053541 · Cited by 83
Semantically Coherent Out-of-Distribution Detection
Yang, J., Wang, H., Feng, L., Yan, X., Zheng, H., Zhang, W., & Liu, Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00819 · Cited by 90
Supervised contrastive learning over prototype-label embeddings for network intrusion detection
López‐Martín, M., Sánchez-Esguevillas, A., Arribas, J. I., & Carro, B. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.09.014 · Cited by 86
Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training
Chen, S., Wu, Y., Wang, C., Chen, Z., Chen, Z., Liu, S., Wu, J., Qian, Y., Wei, F., Li, J., et al. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747077 · Cited by 95
Liu, F., Vulić, I., Korhonen, A., & Collier, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.109 · Cited by 110
Self-Supervised Autoregressive Domain Adaptation for Time Series Data
Ragab, M., Eldele, E., Chen, Z., Wu, M., Kwoh, C. K., & Li, X. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3183252 · Cited by 86
FInfer: Frame Inference-Based Deepfake Detection for High-Visual-Quality Videos
Hu, J., Liao, X., Liang, J., Zhou, W., & Qin, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19978 · Cited by 97
“Since the GRU (Oord, Li, and Vinyals 2018) solves the problem of gradient disappearance and brings high efficiency in the case of extensive training data, we adopt GRU for the regressive part that utilizes various gate functions and states.”
MethodUnsupervised Deraining: Where Contrastive Learning Meets Self-similarity
Ye, Y., Yu, C., Chang, Y., Zhu, L., Zhao, X.-L., Yan, L., & Tian, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00573 · Cited by 91
HCSC: Hierarchical Contrastive Selective Coding
Guo, Y., Xu, M., Li, J., Ni, B., Zhu, X., Sun, Z., & Xu, Y. X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00948 · Cited by 101
Liu, S., Chen, J., He, S., Shi, Z., & Zhou, Z. (2022) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2022.110071 · Cited by 87
Cross-Image Pixel Contrasting for Semantic Segmentation
Zhou, T., & Wang, W. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367952 · Cited by 107
“A prevalent and highly successful loss function for contrastive learning, called InfoNCE [69], [112], takes the following form: where v + denotesafeatureembeddingofapositivesamplefor I , N I contains embeddings of all negative samples, ‘ · ’ denotes the inner (dot) product, and τ > 0 is a…”
BackgroundMethodMutual Contrastive Learning for Visual Representation Learning
Yang, C., An, Z., Cai, L., & Xu, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20211 · Cited by 95
Contrastive Collaborative Filtering for Cold-Start Item Recommendation
Zhou, Z., Zhang, L., & Yang, N. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583286 · Cited by 94
“When we introduce the contrastive learning, minimizing the contrastive loss defined by Equation (4) is equivalent to minimizing the mutual information between negative samples [20], i.”
BackgroundMethodInfluentialPedestrian Behavior Prediction Using Deep Learning Methods for Urban Scenarios: A Review
Zhang, C., & Berger, C. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3281393 · Cited by 84
“Social-NCE [38] applied LSTM model based on the noise-contrastive estimation (NCE) methods [105] by introducing a social contrastive loss, namely the InfoNCE loss [106].”
MethodProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingPreprint
Qi, W., Yan, Y., Gong, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.04063 · Cited by 84
WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-TrainingPreprint
Huo, Y., Zhang, M., Liu, G., Lu, H., Gao, Y., Yang, G., Wen, J., Zhang, H., Xu, B., Zheng, W., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06561 · Cited by 153
Self-citationX-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
Cho, J., Lu, J., Schwenk, D., Hajishirzi, H., & Kembhavi, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.707 · Cited by 99
Self-Supervised SAR-Optical Data Fusion of Sentinel-1/-2 Images
Chen, Y.-X., & Bruzzone, L. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3128072 · Cited by 88
Towards Open-Set Object Detection and Discovery
Zheng, J.-Y., Li, W.-H., Hong, J., Petersson, L., & Barnes, N. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00441 · Cited by 105
Supervised Adversarial Contrastive Learning for Emotion Recognition in Conversations
Hu, D., Bao, Y., Wei, L., Zhou, W. G., & Hu, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.606 · Cited by 116
Dual Contrast-Driven Deep Multi-View Clustering
Cui, J., Li, Y., Huang, H. P., & Wen, J. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3444269 · Cited by 95
“Contrastive learning is an unsupervised learning method, which performs instance-wise discrimination using the normalized cross-entropy as information (InfoNCE) loss [34], [35], [36].”
MethodSelf-citationLearning Predictive Representations for Deformable Objects Using Contrastive Estimation
Yan, W., Vangipuram, A., Abbeel, P., & Pinto, L. (2020) · arXiv · doi:10.48550/arxiv.2003.05436 · Cited by 122
Theoretical Analysis of Self-Training with Deep Networks on Unlabeled DataPreprint
Wei, C., Shen, K., Chen, Y., & Ma, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.03622 · Cited by 116
Contrastive Learning with Continuous Proxy Meta-data for 3D MRI Classification
Dufumier, B., Gori, P., Victor, J., Grigis, A., Wessa, M., Brambilla, P., Favre, P., Polosan, M., McDonald, C., Piguet, C. M., et al. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87196-3_6 · Cited by 82
Attention-Based Multi-Source Domain Adaptation
Zuo, Y., Yao, H., & Xu, C. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3065254 · Cited by 84
Thota, M., & Leontidis, G. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00250 · Cited by 83
Self-Contrastive Learning with Hard Negative Sampling for Self-supervised Point Cloud Learning
Du, B., Gao, X., Hu, W., & Li, X. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475458 · Cited by 82
Nunes, L., Marcuzzi, R., Chen, X., Behley, J., & Stachniss, C. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3142440 · Cited by 88
Wang, Y., Magar, R., Chen, L., & Farimani, A. B. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c00495 · Cited by 76
Human-to-Robot Imitation in the Wild
Bahl, S., Gupta, A., & Pathak, D. (2022) · Robotics: Science and Systems · doi:10.15607/rss.2022.xviii.026 · Cited by 86
“8, 41] have leveraged videos collected offline.”
Wan, W., Chen, J., Zhou, Z., & Shi, Z. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3209332 · Cited by 84
“Amounts of relevant algorithms have been proposed, such as momentum contrast (Moco) [12], contrastive predictive coding (CPC) [13], and a simple framework for contrastive learning of visual representations (SimCLR) [14].”
MethodLearning from Temporal Gradient for Semi-supervised Action Recognition
Xiao, J., Jing, L., Zhang, L., He, J., She, Q., Zhou, Z., Yuille, A., & Li, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00325 · Cited by 84
Self-citationCao, Y., Chen, L., Yuan, Y., & Sun, G. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.107993 · Cited by 90
Prototype-Based Semantic Segmentation
Zhou, T., & Wang, W. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387116 · Cited by 89
“Specifically, many instance-based approaches use the contrastive loss [129], [130] to explicitly compare pairs of image representations to bring samples (by different transformations) from the same image closer and push samples from different images apart [130], [131], [132], [133], [134].”
BackgroundFrom SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation
Kweon, H., & Yoon, K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01844 · Cited by 97
“We formalize this contrasting process with the InfoN-CELoss [42] in the following manner: P where N is the total number of segments in the SE map, and T is the temperature.”
MethodWhat makes instance discrimination good for transfer learning?
Zhao, N., Wu, Z., Lau, R. W. H., & Lin, S. (2020) · arXiv · doi:10.48550/arxiv.2006.06606 · Cited by 115
Pixel-Level Cycle Association: A New Perspective for Domain Adaptive Semantic SegmentationPreprint
Kang, G., Wei, Y., Yang, Y., Zhuang, Y., & Hauptmann, A. G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.00147 · Cited by 102
Efficient Self-supervised Vision Transformers for Representation LearningPreprint
Li, C., Yang, J., Zhang, P., Gao, M., Xiao, B., Dai, X., Yuan, L., & Gao, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09785 · Cited by 164
One Size Does Not Fit All: Generating and Evaluating Variable Number of Keyphrases
Yuan, X., Wang, T., Meng, R., Thaker, K. M., Brusilovsky, P. L., He, D., & Trischler, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.710 · Cited by 89
Generative Hierarchical Features from Synthesizing Images
Xu, Y., Shen, Y., Zhu, J., Yang, C., & Zhou, B. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00441 · Cited by 91
Self-Supervised Change Detection in Multiview Remote Sensing Images
Chen, Y., & Bruzzone, L. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3089453 · Cited by 90
Zhou, H.-Y., Lu, C., Chen, C., Yang, S., & Yu, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3234002 · Cited by 87
“Different from [3], [20], MoCo performs InfoNCE on top of global feature vectors only.”
InfluentialEgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone
Pramanick, S., Song, Y., Nag, S., Lin, K. Q., Shah, H., Shou, M. Z., Chellappa, R., & Zhang, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00487 · Cited by 106
“It makes two modifications over InfoNCE [70]: (i) Besides the matched video-text samples, all pairs that share at least one noun or one verb are treated as positives.”
MethodSelf-Supervised Contrastive Learning of Protein Representations By Mutual Information MaximizationPreprint
Lu, A. X., Zhang, H., Ghassemi, M., & Moses, A. M. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.09.04.283929 · Cited by 71
Spatially Consistent Representation Learning
Roh, B., Shin, W., Kim, I., & Kim, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00120 · Cited by 85
Instance Level Affinity-Based Transfer for Unsupervised Domain Adaptation
Sharma, A., Kalluri, T., & Chandraker, M. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr46437.2021.00532 · Cited by 53
Unified Pre-training with Pseudo Texts for Text-To-Image Person Re-identification
Shao, Z., Zhang, X., Ding, C., Wang, J., & Wang, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01026 · Cited by 95
“The objective function is the contrastive loss [55, 31, 32] on visual and textual feature representations.”
Background
Metadata
- Authors
- Oord, A. V. D., Li, Y., & Vinyals, O.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1807.03748
- OpenAlex
- W4297808394
- Cited by
- 6,242
- References
- 50
- Citations by year
- 2026: 77 · 2025: 660 · 2024: 1288 · 2023: 1444 · 2022: 1043 · 2021: 1173 · 2020: 454 · 2019: 88 · 2018: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Keywords
- contrastive predictive coding, unsupervised feature learning, autoregressive models, negative sampling, high-dimensional data, speech representation, image representation, text representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
