The DOI matches its DataCite record; 53 of the 53 references it cites were found in a registry or an index.
Representation Learning with Contrastive Predictive Coding
Open accessAbstract
While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Contrastive Predictive Coding (CPC) [80,143] utilizes a contrastive future prediction task, and it was found that predicting one or two timesteps in the future is (too) easy and results in poor performance, whereas predicting multiple timesteps into the future is much more complex, and leads to very effective representations.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [143]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Using this contrastive loss, self-supervised networks learn to make the representations similar for positive pairs, and dissimilar for negative pairs, and hence are able to cluster similar (or positive) samples together and push apart dissimilar (or negative) samples.”
From Detecting galaxy tidal features using self-supervised representation learning (2024) · cites this paper as Oord et al. 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It means that InfoNCE loss maximizes the agreement of two different augmented sequences' representations from the same sequence while minimizing the agreement with other negative sequences.”
From Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-Supervised Action Recognition (2022) · cites this paper as Oord, Li, and Vinyals 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We introduce the Contrastive Predictive Coding (CPC) framework to human activity recognition, which captures the long-term temporal structure of sensor data streams.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As mentioned in [58], either 𝑐 𝑡 (the output of 𝑔 𝑒𝑛𝑐 ) or 𝑧 𝑡 (the output of 𝑔 𝑎𝑟 ) can be utilized as the representation.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The framework for data augmentation and batch formation plays a critical role in multi-view similarity learning [12,16,44,115], where we start by following the popular SimCLR [16] method.”
From MARs: Multi-view Attention Regularizations for Patch-based Feature Recognition of Space Terrain (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We note that this is a common setting for self-supervised contrastive learners with NCE objectives (Gutmann & Hyvärinen, 2010;Oord et al., 2018), including SimCLR (Chen et al., 2020) and SimCSE (Gao et al., 2021).”
From The Platonic Representation Hypothesis (2024) · cites this paper as Oord et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“The key insight behind this approach is that predicting just the next future timestep involves exploiting the local smoothness of the signal, whereas predicting multiple subsequent timesteps requires inferring the global structure of time-series data.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“Haresamudram et al. [80] adapted Contrastive Predictive Coding (CPC [143]) for HAR, emphasizing the importance of temporal structure in sensor data representation.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [143]
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive predictive coding (CPC) proposed by [109] provides a contrastive learning framework to perform the prediction task using InfoNCE loss.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“It can be seen that CPC does not directly predict future observations x t + k . Instead, it tries to preserve the mutual information of c t and x t + k .”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“This contrastive loss, or InfoNCE loss (Oord et al. 2018), is minimised when the similarity is high for positive pairs and low for negative pairs.”
From Detecting galaxy tidal features using self-supervised representation learning (2024) · cites this paper as Oord et al. 2018
The cited source says what the claim attributes to it.
In the citing paper
“To evaluate the performance of our TS-TCC model, we follow the standard linear benchmarking evaluation scheme [Oord et al. , 2018; Chen et al. , 2020]. Particularly, we train a linear classifier (single MLP layer) on top of a frozen self-supervised pretrained encoder model.”
From Time-Series Representation Learning via Temporal and Contextual Contrasting (2021) · cites this paper as Oord et al. , 2018
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive-based methods usually use a contrastive similarity metric, such as InfoNCE loss [109].”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“This results in representations that encode the high-level information between temporally separated parts of the time-series signal [58], thereby leading to improved downstream recognition performance.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive Predictive Coding (CPC) [58], which is the focus of this work, performs contrastive learning over multiple future timestep predictions.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“This framework has resulted in excellent representation learning performance across domains such as computer vision [12, 33, 35], natural language [49], speech recognition [6, 58] and time-series data [39].”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“If extra context from the past is useful for classification, then 𝑐 𝑡 can be used; else 𝑧 𝑡 may be more effective.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“SimCLR aims to maximize the learned representation similarity between augmented views of the same input.”
From MARs: Multi-view Attention Regularizations for Patch-based Feature Recognition of Space Terrain (2024) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“More recent advancements in autoregressive modeling such as masked convolutional architectures [18, 19] or self-attention networks [20] could help improve results further.”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“These techniques made manually specified features largely redundant and have greatly improved state-of-the-art in several real-world applications [1, 2, 3].”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We use the Adam optimizer [32] with a learning rate of 2e-4, and use 8 GPUs each with a minibatch of 8 examples from which the negative samples in the contrastive loss are drawn.”
MethodImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“In our visual representation experiments we use the ILSVRC ImageNet competition dataset [34].”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1×
“We did not use Batch-Norm [38].”
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 2×
“In Word2Vec [9] neighbouring words are predicted using a contrastive loss.”
BackgroundLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“These techniques made manually specified features largely redundant and have greatly improved state-of-the-art in several real-world applications [1, 2, 3].”
BackgroundSelf-citationFaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · cited 1×
“For example, the techniques proposed by [21, 22, 23] were based on triplet losses using a max-margin approach to separate positive from negative examples.”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“[41] is the same skip-thought model with layer normalization trained for 1M iterations. evaluate with 10-fold cross-validation for MR, CR, Subj, MPQA and use the train/test split for TREC.”
Method
Research works printed but not linked: 4
- [12][12] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages 297-304, 2010. -
- [31][31] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding . IEEE Signal Processing Society, 2011. -
- [33][33] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(Nov):2579-2605, 2008. -
- [48][48] Sida Wang and Christopher D. Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2 , ACL '12, pages 90-94, 2012. -
Cited by
Sorted by citations
Items 301–400 of 1,066
One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation
Liu, Z., Qi, X., & Fu, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00177 · Cited by 156
Generative Pre-Training for Speech with Autoregressive Predictive Coding
Chung, Y.-A., & Glass, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054438 · Cited by 157
Zhang, X., Ge, Y., Qiao, Y., & Li, H. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00344 · Cited by 171
Self-supervised learning for videos: A survey
Schiappa, M. C., Rawat, Y. S., & Shah, M. (2023) · ACM Computing Surveys · doi:10.1145/3577925 · Cited by 155
“The NCE loss takes as input a positive pair of samples and a set of negative samples, minimizing the distance between the positive pair and maximizing the distance between the negative pairs [34, 57, 127, 133, 157].”
BackgroundMethodSource-free Domain Adaptation via Avatar Prototype Generation and Adaptation
Qiu, Z., Zhang, Y., Lin, H., Niu, S., Liu, Y., Du, Q., & Tan, M. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/402 · Cited by 170
Twin Contrastive Learning for Online Clustering
Li, Y., Yang, M., Peng, D., Li, T., Huang, J., & Peng, X. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01639-z · Cited by 164
“For text data, we employ the synonym replacement strategy (Zhang et al., 2021a) as the weak augmentation T and use the sentence operations (Wei and Zou, 2019) as the strong augmentation T (cid:48) .”
MethodSelf-citationDeep Graph InfomaxPreprint
Veličković, P., Fedus, W., Hamilton, W. L., Lió, P., Bengio, Y., & Hjelm, R. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10341 · Cited by 693
How to Represent Part-Whole Hierarchies in a Neural Network
Hinton, G. E. (2022) · Neural Computation · doi:10.1162/neco_a_01557 · Cited by 173
CARLA: Self-supervised contrastive representation learning for time series anomaly detection
Darban, Z. Z., Webb, G. I., Pan, S., Aggarwal, C. C., & Salehi, M. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110874 · Cited by 183
“Traditional models like InfoNCE loss [20], SimCLR [4], and MoCo [9] use positive-negative pairs to optimize this space, demonstrating strong performance and paving the way for advanced techniques.”
MethodCategory Contrast for Unsupervised Domain Adaptation in Visual Tasks
Huang, J., Guan, D., Xiao, A., Lu, S., & Shao, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00127 · Cited by 160
Interventional Video Grounding with Dual Contrastive Learning
Nan, G., Qiao, R., Xiao, Y., Liu, J., Leng, S., Zhang, H., & Lu, W. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00279 · Cited by 154
Exploring Vision Transformers for Polarimetric SAR Image Classification
Dong, H., Zhang, L., & Zou, B. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3137383 · Cited by 168
Contrastive Predictive Coding for Human Activity Recognition
Haresamudram, H., Essa, I., & Plötz, T. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3463506 · Cited by 150
“We adopt the Contrastive Predictive Coding (CPC) [58] framework for unsupervised representation learning of body-worn sensor data.”
BackgroundMethodInfluentialExploring wav2vec 2.0 on Speaker Verification and Language Identification
Fan, Z., Li, M., Zhou, S., & Xu, B. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1280 · Cited by 206
Self-Supervised Models are Continual Learners
Fini, E., da Costa, V. G. T., Alameda-Pineda, X., Ricci, E., Alahari, K., & Mairal, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00940 · Cited by 155
QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Tracking
Fischer, T., Huang, T., Pang, J., Qiu, L., Chen, H., Darrell, T., & Yu, F. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3301975 · Cited by 179
“For each training sample, we can use the non-parametric softmax [13], [16] with cross-entropy to optimize the feature embeddings where v , k + , k − are feature embeddings of the training sample, its positive target, and negative targets in K .”
BackgroundMethodPseudo-Supervised Deep Subspace Clustering
Lv, J., Kang, Z., Lu, X., & Xu, Z. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3079800 · Cited by 156
Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling
Filiot, A., Ghermi, R., Olivier, A., Jacob, P., Fidon, L., Camara, A., Mac Kain, A., Saillard, C., & Schiratti, J.-B. (2023) · doi:10.1101/2023.07.21.23292757 · Cited by 148
A survey on graph-based deep learning for computational histopathology
Ahmedt‐Aristizabal, D., Armin, M. A., Denman, S., Fookes, C., & Petersson, L. (2021) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2021.102027 · Cited by 165
Melas-Kyriazi, L., Rupprecht, C., Laina, I., & Vedaldi, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00818 · Cited by 164
Schirris, Y., Gavves, E., Nederlof, I., Horlings, H. M., & Teuwen, J. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102464 · Cited by 174
Augmentation Invariant and Instance Spreading Feature for Softmax Embedding
Ye, M., Shen, J., Zhang, X., Yuen, P. C., & Chang, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3013379 · Cited by 137
CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-Training
Huang, T., Dong, B., Yang, Y., Huang, X., Lau, R. W. H., Ouyang, W., & Zuo, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02025 · Cited by 191
“l intra(·) is based on InfoNCE [32] loss.”
BackgroundMethodInfluentialSequence-to-Sequence Contrastive Learning for Text Recognition
Aberdam, A., Litman, R., Tsiper, S., Anschel, O., Slossberg, R., Mazor, S., Manmatha, R., & Perona, P. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01505 · Cited by 152
Everything at Once – Multi-modal Fusion Transformer for Video Retrieval
Shvetsova, N., Chen, B., Rouditchenko, A., Thomas, S., Kingsbury, B. E. D., Feris, R., Harwath, D. F., Glass, J. R., & Kuehne, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01939 · Cited by 154
Learning Non-target Knowledge for Few-shot Semantic Segmentation
Liu, Y., Liu, N., Cao, Q., Yao, X., Han, J., & Shao, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01128 · Cited by 156
MFIR: Multimodal fusion and inconsistency reasoning for explainable fake news detection
Wu, L., Long, Y., Gao, C., Wang, Z., & Zhang, Y. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101944 · Cited by 156
BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation
Niizumi, D., Takeuchi, D., Ohishi, Y., Harada, N., & Kashino, K. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534474 · Cited by 173
Zhang, Y., Ren, Z., Zhou, S., Feng, K., Yu, K., & Liu, Z. (2022) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2022.3179289 · Cited by 143
“…n randomly sampled, SSL aims to learn the class-level alignment in embedding space by optimizing a contrastive loss function, such as InfoNCE loss [16], the loss takes as where the · symbol indicates the inner dot product, the index i is called the anchor, and i ∈ I ≡ { 1 , 2 , . . . , 2 N } is…”
BackgroundRe-IQA: Unsupervised Learning for Image Quality Assessment in the Wild
Saha, A., Mishra, S., & Bovik, A. C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00566 · Cited by 177
“To calculate the loss between the representation generated by the query and key encoder, we use the InfoNCE [23] loss function: Here q is the query image, k + is a positive sample (similar-quality), k − represnt negative samples (different-quality), and τ is a temperature hyper-parameter.”
MethodImplicit Sample Extension for Unsupervised Person Re-Identification
Zhang, X., Li, D., Wang, Z., Wang, J., Ding, E., Shi, Q., Zhang, Z., & Wang, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00722 · Cited by 154
FLAME: Free-Form Language-Based Motion Synthesis & Editing
Kim, J.-H., Kim, J., & Choi, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.25996 · Cited by 203
Deep Learning for Time-Series Prediction in IIoT: Progress, Challenges, and Prospects
Ren, L., Jia, Z., Laili, Y., & Huang, D. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3291371 · Cited by 158
“Contrastive learning is a typical discriminant self-supervised learning, thereby mining properties of the dataset itself to help the model learn without labels [71], [72].”
BackgroundGuiding Pseudo-labels with Uncertainty Estimation for Source-free Unsupervised Domain Adaptation
Litrico, M., Bue, A. D., & Morerio, P. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00738 · Cited by 168
“Accordingly, we optimise the following InfoNCELoss [59]: Lctrt = LInfoNCE = − log exp(q · k+/τ)∑ j∈Nq exp(q · kj/τ) (6) Nq = {j|ŷij ̸= ŷi, ∀j ∈ {1, ..., N},∀i ∈ {1, ..., T}}, where Nq is the set of indices of samples in Qe that never shared with the query sample the same pseudo-labels in the past T epochs.”
Adversarial Graph Augmentation to Improve Graph Contrastive LearningPreprint
Suresh, S., Pan, L., Hao, C., & Neville, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05819 · Cited by 208
Learning to Detect Objects with a 1 Megapixel Event CameraPreprint
Perot, E., de Tournemire, P., Nitti, D., Masci, J., & Sironi, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13436 · Cited by 145
CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations
Zolfaghari, M., Zhu, Y., Gehler, P., & Brox, T. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00148 · Cited by 153
Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning
Jin, M., Zheng, Y., Li, Y.-F., Gong, C., Zhou, C. D., & Pan, S. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/204 · Cited by 166
Self-citationDuplex Contextual Relation Network For Polyp Segmentation
Yin, Z., Liang, K., Ma, Z., & Guo, J. (2022) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi52829.2022.9761402 · Cited by 154
Hyperspectral Imagery Classification Based on Contrastive Learning
Hou, S., Shi, H., Cao, X., Zhang, X., & Jiao, L. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3139099 · Cited by 146
Guided Point Contrastive Learning for Semi-supervised Point Cloud Semantic Segmentation
Jiang, L., Shi, S., Tian, Z., Lai, X., Liu, S., Fu, C., & Jia, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00636 · Cited by 137
C 2 SLR: Consistency-enhanced Continuous Sign Language Recognition
Zuo, R., & Mak, B. K.-W. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00507 · Cited by 146
Wang, J., Ren, H., Shen, C., Huang, W., & Zhu, Z. (2023) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2023.109879 · Cited by 143
Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds
Rao, Y., Lu, J., & Zhou, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00542 · Cited by 134
Region-aware Contrastive Learning for Semantic Segmentation
Hu, H., Cui, J., & Wang, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01598 · Cited by 145
Graph Anomaly Detection via Multi-Scale Contrastive Learning Networks with Augmented View
Duan, J., Wang, S., Zhang, P., Zhu, E., Hu, J., Jin, H., Liu, Y., & Dong, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25907 · Cited by 149
Kim, M., Kim, S., Park, J., Park, S., & Sohn, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01786 · Cited by 152
“To this end, inspired by [34, 56], we adopt the contrastive regularization loss that jointly leverages positive and negative samples to regularize the model, and thus the model can overcome the limitation of strong reliance on supervision signal.”
Self-supervised Pretraining of Visual Features in the Wild
Goyal, P., Caron, M., Lefaudeux, B., Xu, M., Wang, P., Pai, V., Singh, M., Liptchinsky, V., Misra, I., Joulin, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.01988 · Cited by 141
CERT: Contrastive Self-supervised Learning for Language UnderstandingPreprint
Fang, H., & Xie, P. (2020) · arXiv.org · doi:10.36227/techrxiv.12308378 · Cited by 144
Contrastive Self-supervised Learning for Graph Classification
Zeng, J., & Xie, P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17293 · Cited by 156
Active Learning by Acquiring Contrastive Examples
Margatina, K., Vernikos, G., Barrault, L., & Αλέτρας, Ν. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.51 · Cited by 211
Wang, Y., Wang, Y., Cai, J., Lee, T. K., Miao, C., & Wang, Z. J. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102693 · Cited by 144
TweetNLP: Cutting-Edge Natural Language Processing for Social Media
Camacho-collados, J., Rezaee, K., Riahi, T., Ushio, A., Loureiro, D., Antypas, D., Boisson, J., Espinosa Anke, L., Liu, F., & Martínez Cámara, E. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2022.emnlp-demos.5 · Cited by 142
Multimodal Image Synthesis and Editing: The Generative AI Era
Zhan, F., Yu, Y., Wu, R., Zhang, J., Lu, S., Liu, L., Kortylewski, A., Theobalt, C., & Xing, E. P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3305243 · Cited by 197
Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding
Wang, Z., Wang, L., Wu, T., Li, T., & Wu, G. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20163 · Cited by 148
Contrastive learning based self-supervised time-series analysis
Pöppelbaum, J., Chadha, G. S., & Schwung, A. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2021.108397 · Cited by 131
Aligning Bag of Regions for Open-Vocabulary Object Detection
Wu, S., Zhang, W., Jin, S., Liu, W., & Loy, C. C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01464 · Cited by 155
“Specifically, given G bags of regions, the alignment InfoNCE loss [38] between bag-of-regions embeddings is calculated as The p kt,v and p kv,t are calculated as respectively, where τ (cid:48) is the temperature to re-scale the co-sine similarity.”
MethodDream to Control: Learning Behaviors by Latent Imagination
Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019) · arXiv · doi:10.48550/arxiv.1912.01603 · Cited by 558
SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data
Tang, C. I., Perez-Pozuelo, I., Spathis, D., Brage, S., Wareham, N., & Mascolo, C. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3448112 · Cited by 140
Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model
Chidambaram, M., Yang, Y., Cer, D. M., Yuan, S., Sung, Y.-H., Strope, B. P., & Kurzweil, R. (2019) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w19-4330 · Cited by 137
Efficient Visual Pretraining with Contrastive Detection
Hénaff, O. J., Koppula, S., Alayrac, J.-B., van den Oord, A., Vinyals, O., & Carreira, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00993 · Cited by 128
Self-citationBridging Video-text Retrieval with Multiple Choice Questions
Ge, Y., Ge, Y., Liu, X., Li, D., Shan, Y., Qie, X., & Luo, P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01569 · Cited by 138
Learning Representation for Clustering Via Prototype Scattering and Positive Sampling
Huang, Z., Chen, J., Zhang, J., & Shan, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3216454 · Cited by 144
Hard Sample Aware Network for Contrastive Deep Graph Clustering
Liu, Y., Yang, X., Zhou, S., et al. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.26071 · Cited by 128
Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset BiasesPreprint
Purushwalkam, S., & Gupta, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13916 · Cited by 137
COCO-LM: Correcting and Contrasting Text Sequences for Language Model PretrainingPreprint
Yu, M., Xiong, C., Bajaj, P., Tiwary, S., Bennett, P. N., Han, J., & Song, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08473 · Cited by 143
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training ParadigmPreprint
Li, Y., Liang, F., Zhao, L.-C., Cui, Y., Ouyang, W., Shao, J., Yu, F., & Yan, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05208 · Cited by 391
Self-citationContrastive Learning for Weakly Supervised Phrase Grounding
Gupta, T., Vahdat, A., Chechik, G., Yang, X., Kautz, J., & Hoiem, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_44 · Cited by 125
Learning to Contrast the Counterfactual Samples for Robust Visual Question Answering
Liang, Z., Jiang, W., Hu, H., & Zhu, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.265 · Cited by 137
Liu, S., Chen, J., He, S., et al. (2021) · ISA Transactions · doi:10.1016/j.isatra.2021.10.025 · Cited by 126
A Unifying Mutual Information View of Metric Learning: Cross-Entropy vs. Pairwise Losses
Boudiaf, M., Rony, J., Ziko, I. M., Granger, É., Pedersoli, M., Piantanida, P., & Ayed, I. B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58539-6_33 · Cited by 149
The Spatially-Correlative Loss for Various Image Translation Tasks
Zheng, C., Cham, T., & Cai, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01614 · Cited by 149
3D Self-Supervised Methods for Medical ImagingPreprint
Taleb, A., Loetzsch, W., Danz, N., Severin, J., Gaertner, T., Bergner, B., & Lippert, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03829 · Cited by 145
Contrastive Learning for Unpaired Image-to-Image TranslationPreprint
Park, T., Efros, A. A., Zhang, R., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.15651 · Cited by 126
Towards Learning a Universal Non-Semantic Representation of Speech
Shor, J., Jansen, A., Maor, R., Lang, O., Tuval, O., Quitry, F. D. C., Tagliasacchi, M., Shavitt, I., Emanuel, D., & Haviv, Y. A. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1242 · Cited by 128
An Unsupervised Sentence Embedding Method by Mutual Information Maximization
Zhang, Y., He, R., Liu, Z., Lim, K. H., & Bing, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.124 · Cited by 174
Zhou, W., Li, J., Zhihong, L. I. U., & Shen, L. (2021) · Chinese Journal of Aeronautics · doi:10.1016/j.cja.2021.09.008 · Cited by 125
Robot Learning From Randomized Simulations: A Review
Muratore, F., Ramos, F., Turk, G., Yu, W., Gienger, M., & Peters, J. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.799893 · Cited by 128
Zhang, Z., Zhao, T., Guo, Y., & Yin, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3449154 · Cited by 208
“Modality interaction was only through the InfoNCE loss [75].”
Structure-Preserving Image Super-Resolution
Ma, C., Rao, Y., Lu, J., & Zhou, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3114428 · Cited by 133
Divergence-Agnostic Unsupervised Domain Adaptation by Adversarial Attacks
Li, J., Du, Z., Zhu, L., Ding, Z., Lü, K., & Shen, H. T. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3109287 · Cited by 123
Crafting Better Contrastive Views for Siamese Representation Learning
Peng, X., Wang, K., Zhu, Z., Wang, M., & You, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01556 · Cited by 122
Neighbor Contrastive Learning on Learnable Graph Augmentation
Shen, X., Sun, D., Pan, S., Zhou, X., & Yang, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26168 · Cited by 134
SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-TrainingPreprint
Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., & Goldstein, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01342 · Cited by 536
Self-Training and Pre-Training are Complementary for Speech Recognition
Xu, Q., Baevski, A., Likhomanenko, T., Tomasello, P., Conneau, A., Collobert, R., Synnaeve, G., & Auli, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414641 · Cited by 152
Self-supervised Video Transformer
Ranasinghe, K., Naseer, M., Khan, S. H., Khan, F. S., & Ryoo, M. S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00289 · Cited by 128
Disentangling Long and Short-Term Interests for Recommendation
Zheng, Y., Gao, C., Chang, J., Niu, Y., Song, Y., Jin, D., & Li, Y. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512098 · Cited by 132
Self-citationA Multihead Attention Self-Supervised Representation Model for Industrial Sensors Anomaly Detection
Qiao, Y., Lü, J., Wang, T., Liu, K., Zhang, B., & Snoussi, H. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3280337 · Cited by 132
“The frequent loss function is In-foNCE [28], which seeks to bring the distance between positive sample pairs closer and the distance between negative sample pairs farther.”
BackgroundMultimodal Self-supervised Learning for Medical Image Analysis
Taleb, A., Lippert, C., Klein, T., & Nabi, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78191-0_51 · Cited by 148
Vector-Quantized Neural Networks for Acoustic Unit Discovery in the ZeroSpeech 2020 Challenge
van Niekerk, B., Nortje, L., & Kamper, H. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1693 · Cited by 112
How Useful Is Self-Supervised Pretraining for Visual Tasks?
Newell, A., & Deng, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00737 · Cited by 131
Learning Visual Representations with Caption Annotations
Sarıyıldız, M. B., Pérez, J., & Larlus, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58598-3_10 · Cited by 128
Towards Long-Form Video Understanding
Wu, C.-Y., & Krähenbühl, P. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00192 · Cited by 131
Rethinking the Truly Unsupervised Image-to-Image Translation
Baek, K., Choi, Y., Uh, Y., Yoo, J., & Shim, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01389 · Cited by 127
Contrastive Code Representation Learning
Jain, P., Jain, A., Zhang, T., Abbeel, P., Gonzalez, J., & Stoica, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.482 · Cited by 129
Bipartite Graph Embedding via Mutual Information Maximization
Cao, J., Lin, X., Guo, S., Liu, L., Liu, T., & Wang, B. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441783 · Cited by 132
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?
Hsu, W.-N., Tsai, Y.-H. H., Bolte, B., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414460 · Cited by 128
Hu, Q., Wang, X., Hu, W., & Qi, G.-J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00113 · Cited by 129
Class-Aware Contrastive Semi-Supervised Learning
Yang, F., Wu, K., Zhang, S., Jiang, G., Liu, Y., Zheng, F., Zhang, W., Wang, C., & Zeng, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01402 · Cited by 126
Contrastive Bidirectional Transformer for Temporal Representation Learning
Sun, C., Baradel, F., Murphy, K. P., & Schmid, C. (2019) · Cited by 109
Metadata
- Authors
- Oord, A. V. D., Li, Y., & Vinyals, O.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1807.03748
- OpenAlex
- W4297808394
- Cited by
- 6,242
- References
- 50
- Citations by year
- 2026: 77 · 2025: 660 · 2024: 1288 · 2023: 1444 · 2022: 1043 · 2021: 1173 · 2020: 454 · 2019: 88 · 2018: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Keywords
- contrastive predictive coding, unsupervised feature learning, autoregressive models, negative sampling, high-dimensional data, speech representation, image representation, text representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
