The DOI matches its DataCite record; 53 of the 53 references it cites were found in a registry or an index.
Representation Learning with Contrastive Predictive Coding
Open accessAbstract
While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Contrastive Predictive Coding (CPC) [80,143] utilizes a contrastive future prediction task, and it was found that predicting one or two timesteps in the future is (too) easy and results in poor performance, whereas predicting multiple timesteps into the future is much more complex, and leads to very effective representations.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [143]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Using this contrastive loss, self-supervised networks learn to make the representations similar for positive pairs, and dissimilar for negative pairs, and hence are able to cluster similar (or positive) samples together and push apart dissimilar (or negative) samples.”
From Detecting galaxy tidal features using self-supervised representation learning (2024) · cites this paper as Oord et al. 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It means that InfoNCE loss maximizes the agreement of two different augmented sequences' representations from the same sequence while minimizing the agreement with other negative sequences.”
From Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-Supervised Action Recognition (2022) · cites this paper as Oord, Li, and Vinyals 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We introduce the Contrastive Predictive Coding (CPC) framework to human activity recognition, which captures the long-term temporal structure of sensor data streams.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As mentioned in [58], either 𝑐 𝑡 (the output of 𝑔 𝑒𝑛𝑐 ) or 𝑧 𝑡 (the output of 𝑔 𝑎𝑟 ) can be utilized as the representation.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The framework for data augmentation and batch formation plays a critical role in multi-view similarity learning [12,16,44,115], where we start by following the popular SimCLR [16] method.”
From MARs: Multi-view Attention Regularizations for Patch-based Feature Recognition of Space Terrain (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We note that this is a common setting for self-supervised contrastive learners with NCE objectives (Gutmann & Hyvärinen, 2010;Oord et al., 2018), including SimCLR (Chen et al., 2020) and SimCSE (Gao et al., 2021).”
From The Platonic Representation Hypothesis (2024) · cites this paper as Oord et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“The key insight behind this approach is that predicting just the next future timestep involves exploiting the local smoothness of the signal, whereas predicting multiple subsequent timesteps requires inferring the global structure of time-series data.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“Haresamudram et al. [80] adapted Contrastive Predictive Coding (CPC [143]) for HAR, emphasizing the importance of temporal structure in sensor data representation.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [143]
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive predictive coding (CPC) proposed by [109] provides a contrastive learning framework to perform the prediction task using InfoNCE loss.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“It can be seen that CPC does not directly predict future observations x t + k . Instead, it tries to preserve the mutual information of c t and x t + k .”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“This contrastive loss, or InfoNCE loss (Oord et al. 2018), is minimised when the similarity is high for positive pairs and low for negative pairs.”
From Detecting galaxy tidal features using self-supervised representation learning (2024) · cites this paper as Oord et al. 2018
The cited source says what the claim attributes to it.
In the citing paper
“To evaluate the performance of our TS-TCC model, we follow the standard linear benchmarking evaluation scheme [Oord et al. , 2018; Chen et al. , 2020]. Particularly, we train a linear classifier (single MLP layer) on top of a frozen self-supervised pretrained encoder model.”
From Time-Series Representation Learning via Temporal and Contextual Contrasting (2021) · cites this paper as Oord et al. , 2018
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive-based methods usually use a contrastive similarity metric, such as InfoNCE loss [109].”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“This results in representations that encode the high-level information between temporally separated parts of the time-series signal [58], thereby leading to improved downstream recognition performance.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive Predictive Coding (CPC) [58], which is the focus of this work, performs contrastive learning over multiple future timestep predictions.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“This framework has resulted in excellent representation learning performance across domains such as computer vision [12, 33, 35], natural language [49], speech recognition [6, 58] and time-series data [39].”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“If extra context from the past is useful for classification, then 𝑐 𝑡 can be used; else 𝑧 𝑡 may be more effective.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“SimCLR aims to maximize the learned representation similarity between augmented views of the same input.”
From MARs: Multi-view Attention Regularizations for Patch-based Feature Recognition of Space Terrain (2024) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“More recent advancements in autoregressive modeling such as masked convolutional architectures [18, 19] or self-attention networks [20] could help improve results further.”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“These techniques made manually specified features largely redundant and have greatly improved state-of-the-art in several real-world applications [1, 2, 3].”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We use the Adam optimizer [32] with a learning rate of 2e-4, and use 8 GPUs each with a minibatch of 8 examples from which the negative samples in the contrastive loss are drawn.”
MethodImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“In our visual representation experiments we use the ILSVRC ImageNet competition dataset [34].”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1×
“We did not use Batch-Norm [38].”
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 2×
“In Word2Vec [9] neighbouring words are predicted using a contrastive loss.”
BackgroundLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“These techniques made manually specified features largely redundant and have greatly improved state-of-the-art in several real-world applications [1, 2, 3].”
BackgroundSelf-citationFaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · cited 1×
“For example, the techniques proposed by [21, 22, 23] were based on triplet losses using a max-margin approach to separate positive from negative examples.”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“[41] is the same skip-thought model with layer normalization trained for 1M iterations. evaluate with 10-fold cross-validation for MR, CR, Subj, MPQA and use the train/test split for TREC.”
Method
Flagged first, then by citations
Research works printed but not linked: 4
- [12][12] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages 297-304, 2010. -
- [31][31] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding . IEEE Signal Processing Society, 2011. -
- [33][33] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(Nov):2579-2605, 2008. -
- [48][48] Sida Wang and Christopher D. Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2 , ACL '12, pages 90-94, 2012. -
Cited by
Sorted by citations
Items 101–200 of 1,066
Deep Graph Contrastive Representation LearningPreprint
Zhu, Y., Xu, Y., Yu, F., Liu, Q., Wu, S., & Wang, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04131 · Cited by 1,003
Exploring Simple Siamese Representation Learning
Chen, X., & He, K. (2020) · arXiv · doi:10.48550/arxiv.2011.10566 · Cited by 381
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · arXiv · doi:10.48550/arxiv.2104.14548 · Cited by 405
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00945 · Cited by 404
Self-supervised learning for recommender systems: A survey
Yu, J., Yin, H., Xia, X., Chen, T., Li, J., & Huang, Z. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3282907 · Cited by 413
Rethinking Pre-training and Self-trainingPreprint
Zoph, B., Ghiasi, G., Lin, T.-Y., Cui, Y., Liu, H., Cubuk, E. D., & Le, Q. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06882 · Cited by 579
Multi-level Feature Learning for Contrastive Multi-view Clustering
Xu, J., Tang, H., Ren, Y., Peng, L., Zhu, X., & He, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01558 · Cited by 425
A review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389
Charting the Right Manifold: Manifold Mixup for Few-shot Learning
Mangla, P., Singh, M., Sinha, A., Kumari, N., Balasubramanian, V. N., & Krishnamurthy, B. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093338 · Cited by 402
Video Representation Learning by Dense Predictive Coding
Han, T., Xie, W., & Zisserman, A. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00186 · Cited by 393
Self-Supervised Learning of Graph Neural Networks: A Unified Review
Xie, Y., Zhao, X., Zhang, J., Wang, Z., & Ji, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3170559 · Cited by 452
LiT: Zero-Shot Transfer with Locked-image text Tuning
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01759 · Cited by 402
Zero-Shot Text-Guided Object Generation with Dream Fields
Jain, A., Mildenhall, B., Barron, J. T., Abbeel, P., & Poole, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00094 · Cited by 431
Learnable latent embeddings for joint behavioural and neural analysis
Schneider, S., Lee, J. H., & Mathis, M. W. (2023) · Nature · doi:10.1038/s41586-023-06031-6 · Cited by 406
Large Scale Adversarial Representation LearningPreprint
Donahue, J., & Simonyan, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.02544 · Cited by 418
Rethinking Semantic Segmentation: A Prototype View
Zhou, T., Wang, W., Konukoğlu, E., & Goo, L. V. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00261 · Cited by 426
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372
“As the CPC model, it uses the InfoNCE loss [56] to maximize the similarity between a contextualized representation and a localized representation.”
BackgroundMethodInfluentialImage-to-Image Translation: Methods and Applications
Pang, Y., Lin, J., Qin, T., & Chen, Z. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3109419 · Cited by 417
Structured Pruning for Deep Convolutional Neural Networks: A Survey
He, Y., & Xiao, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3334614 · Cited by 468
“Therefore, a similarity-based contrastive loss [256] is used for gradient-based learning without the labeled data.”
MethodDeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Giorgi, J., Nitski, O., Wang, B., & Bader, G. D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.72 · Cited by 491
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
Zhou, H., Sun, Y., Wu, W., Loy, C. C., Wang, X., & Liu, Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00416 · Cited by 391
Scaling and Benchmarking Self-Supervised Visual Representation Learning
Goyal, P., Mahajan, D. K., Gupta, A., & Misra, I. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00649 · Cited by 361
VirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01101 · Cited by 376
Self-Supervised Learning in Remote Sensing: A review
Wang, Y., Albrecht, C. M., Braham, N. A. A., Mou, L., & Zhu, X. X. (2022) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2022.3198244 · Cited by 364
“Contrastive Predictive Coding (CPC) [71].”
BackgroundMethodInfluentialMockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
Liu, A. T., Yang, S.-W., Chi, P.-H., Hsu, P.-C., & Lee, H.-Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054458 · Cited by 352
An Unsupervised Autoregressive Model for Speech Representation Learning
Chung, Y.-A., Hsu, W.-N., Tang, H., & Glass, J. R. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1473 · Cited by 410
Unsupervised neural network models of the ventral visual stream
Zhuang, C., Yan, S., Nayebi, A., Schrimpf, M., Frank, M. C., DiCarlo, J. J., & Yamins, D. L. K. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2014196118 · Cited by 367
ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training
Qi, W., Yan, Y., Gong, Y., Liu, D., Duan, N., Chen, J., Zhang, R., & Zhou, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.217 · Cited by 459
Self-Alignment Pretraining for Biomedical Entity Representations
Liu, F., Shareghi, E., Meng, Z., Basaldella, M., & Collier, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.334 · Cited by 479
Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-IDPreprint
Ge, Y., Zhu, F., Chen, D., Zhao, R., & Li, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02713 · Cited by 464
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
Hendrycks, D., Mazeika, M., Kadavath, S., & Song, D. (2019) · arXiv · doi:10.48550/arxiv.1906.12340 · Cited by 634
Single-Source Domain Expansion Network for Cross-Scene Hyperspectral Image Classification
Zhang, Y., Li, W., Sun, W., Tao, R., & Du, Q. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3243853 · Cited by 359
“loss [32] is generally constructed to correctly classify positive samples and negative samples,”
BackgroundDetCo: Unsupervised Contrastive Learning for Object Detection
Xie, E., Ding, J., Wang, W., Zhan, X., Xu, H., Sun, P., Li, Z., & Luo, P. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00828 · Cited by 361
Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data
Manas, O., Lacoste, A., Giro-i-Nieto, X., Vazquez, D., & Rodriguez, P. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00928 · Cited by 354
Cross-Modal Contrastive Learning for Text-to-Image Generation
Zhang, H., Koh, J. Y., Baldridge, J., Lee, H., & Yang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00089 · Cited by 375
Wang, C., Rivière, M., Lee, A., Wu, A., Talnikar, C., Haziza, D., Williamson, M., Pino, J., & Emmanuel Dupoux, A. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.80 · Cited by 383
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K., Liu, J., Bennett, P. N., Ahmed, J., & Overwikj, A. (2021) · International Conference on Learning Representations · Cited by 376
Self-citationYou, C., Zhou, Y., Zhao, R., Staib, L. H., & Duncan, J. S. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3161829 · Cited by 333
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
Liu, A. T., Li, S.-W. D., & Lee, H.-Y. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3095662 · Cited by 333
Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval
Jiang, D., & Ye, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00273 · Cited by 367
“1 2 and 3 denotes different CLIP models fine-tuned with the original CLIP loss (InfoNCE [34]).”
MethodAdvances and Challenges in Meta-Learning: A Technical Review
Vettoruzzo, A., Bouguelia, M.-R., Vanschoren, J., Rognvaldsson, T. S., & Santosh, K. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3357847 · Cited by 370
“Contrastive learning [116, 113] is commonly used in this context, aiming to learn features by bringing similar examples closer together while pushing differing examples apart.”
Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition
Shen, X., Liu, X., Hu, X., Zhang, D., & Song, S. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3164516 · Cited by 315
XSimGCL: Towards Extremely Simple Graph Contrastive Learning for Recommendation
Yu, J., Xia, X., Chen, T., Cui, L., Nguyen, Q. V. H., & Yin, H. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3288135 · Cited by 365
“The InfoNCE loss [22] is formulated as: where i and j are users/items in B , z i and z i are L 2 normalized representations learned from two different dropout-based graph augmentations (namely z i = e (cid:4) i e (cid:4) i (cid:5) 2 ), and τ > 0 (e.g., 0.2) is the temperature which controls the…”
BackgroundMethodInfluentialHierarchical motor control in mammals and machines
Merel, J., Botvinick, M., & Wayne, G. (2019) · Nature Communications · doi:10.1038/s41467-019-13239-6 · Cited by 274
Ssast: Self-supervised audio spectrogram transformer
Gong, Y., Lai, C.-I., Chung, Y.-A., & Glass, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21315 · Cited by 294
SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation
Xia, J., Wu, L., Chen, J., Hu, B., & Li, S. Z. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512156 · Cited by 327
Vision-Language Pre-Training with Triple Contrastive Learning
Yang, J., Duan, J., Tran, S., Xu, Y., Chanda, S., Chen, L., Zeng, B., Chilimbi, T., & Huang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01522 · Cited by 309
Parametric Contrastive Learning
Cui, J., Zhong, Z., Shu, L., Yu, B., & Jia, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00075 · Cited by 328
Self-Supervised Learning on Graphs: Contrastive, Generative, or Predictive
Wu, L., Lin, H., Tan, C., Gao, Z., & Li, S. Z. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3131584 · Cited by 345
Self-supervised Co-training for Video Representation Learning
Han, T., Xie, W., & Zisserman, A. (2020) · arXiv · doi:10.48550/arxiv.2010.09709 · Cited by 292
Contrastive Embedding for Generalized Zero-Shot Learning
Han, Z., Fu, Z., Chen, S., & Yang, J. J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00240 · Cited by 312
ICE: Inter-instance Contrastive Encoding for Unsupervised Person Re-identification
Chen, H., Lagadec, B., & Brémond, F. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01469 · Cited by 296
MURF: Mutually Reinforcing Multi-Modal Image Registration and Fusion
Xu, H., Yuan, J., & Ma, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3283682 · Cited by 303
“values for positive pairs while low values for negative ones, the contrastive loss function for learning f cl θ1(·) and f cl θ2(·) is defined as the InfoNCE loss [44]:”
BackgroundSelf-Supervised Pretraining of 3D Features on any Point-Cloud
Zhang, Z., Girdhar, R., Joulin, A., & Misra, I. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01009 · Cited by 310
Image Quality Assessment Using Contrastive Learning
Madhusudana, P. C., Birkbeck, N., Wang, Y., Adsumilli, B., & Bovik, A. C. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3181496 · Cited by 295
Learning to Diversify for Single Domain Generalization
Wang, Z., Luo, Y., Qiu, R., Huang, Z. H., & Baktashmotlagh, M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00087 · Cited by 310
Chung, Y.-A., Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang, R., & Wu, Y. (2021) · 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru51503.2021.9688253 · Cited by 293
CoMatch: Semi-supervised Learning with Contrastive Graph Regularization
Li, J., Xiong, C., & Hoi, S. C. H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00934 · Cited by 311
TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting
Ekambaram, V., Jati, A., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599533 · Cited by 379
“We categorize SOTA forecasting benchmarks into the following categories: (i) Standard Transformers: FEDformer [38], Autoformer[30] and Informer [37], (ii) Patch Transformers: PatchTST [22] and CrossFormer [3], (iii) MLPs and Non-Transformers: DLinear [34], LightTS [36] and S4 [12], (iv) Self-supervised models: BTSF [3…”
MethodInfluentialSpatio-Temporal Self-Supervised Learning for Traffic Flow Prediction
Ji, J., Wang, J., Huang, C., Wu, J., Xu, B., Wu, Z., Zhang, J., & Zheng, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25555 · Cited by 319
Contrastive Learning of Medical Visual Representations from Paired Images and TextPreprint
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., & Langlotz, C. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00747 · Cited by 465
Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts
Hou, J., Graham, B., Nießner, M., & Xie, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01533 · Cited by 299
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“LNT and TRL-CPC use the same structure as the original CPC [109] to build a representation learning model, and the purpose is to capture the local semantics across the time to detect the anomaly points.”
MethodETC: Encoding Long and Structured Inputs in Transformers
Ainslie, J., Ontañón, S., Alberti, C., Cvicek, V., Fisher, Z., Pham, P., Ravula, A., Sanghai, S. K., Wang, Q., & Li, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.19 · Cited by 371
Self-citationKostas, D., Aroca-Ouellette, S., & Rudzicz, F. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.653659 · Cited by 348
On Mutual Information Maximization for Representation LearningPreprint
Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., & Lučić, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.13625 · Cited by 427
REGTR: End-to-end Point Cloud Correspondences with Transformers
Yew, Z., & Lee, G. H. (2022) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00656 · Cited by 317
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
Xiao, B., Wu, H., Xu, W., Dai, X., Hu, H., Lu, Y., Zeng, M., Liu, C., & Yuan, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00461 · Cited by 459
“None semantic Rich semantic”
Contrastive Test-Time Adaptation
Chen, D., Wang, D., Darrell, T., & Ebrahimi, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00039 · Cited by 316
Big Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349
Multi-Task Self-Supervised Learning for Robust Speech Recognition
Ravanelli, M., Zhong, J., Pascual, S., Świętojański, P., Monteiro, J., Trmal, J., & Bengio, Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053569 · Cited by 287
CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
Tack, J., Mo, S., Jeong, J., & Shin, J. (2020) · arXiv · doi:10.48550/arxiv.2007.08176 · Cited by 351
Hard Negative Mixing for Contrastive LearningPreprint
Kalantidis, Y., Sarıyıldız, M. B., Pion, N., Weinzaepfel, P., & Larlus, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01028 · Cited by 423
Self-supervised learning methods and applications in medical imaging analysis: a survey
Shurrab, S., & Duwairi, R. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1045 · Cited by 299
Uncovering the structure of clinical EEG signals with self-supervised learning
Banville, H. J., Chehab, O., Hyvärinen, A., Engemann, D. A., & Gramfort, A. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/abca18 · Cited by 279
SelfReg: Self-supervised Contrastive Regularization for Domain Generalization
Kim, D., Yoo, Y. F., Park, S., Kim, J., & Lee, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00948 · Cited by 284
Layer-Wise Analysis of a Self-Supervised Speech Representation Model
Pasad, A., Chou, J.-C., & Livescu, K. (2021) · 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru51503.2021.9688093 · Cited by 262
Self-Supervised Learning for Multimedia Recommendation
Tao, Z., Liu, X., Xia, Y., Wang, X., Yang, L., Huang, X., & Chua, T. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3187556 · Cited by 287
Geography-Aware Self-Supervised Learning
Ayush, K., Uzkent, B., Meng, C., Tanmay, K., Burke, M., Lobell, D., & Ermon, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01002 · Cited by 288
Ding, Y., Zhuang, J., Ding, P., & Jia, M. (2021) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2021.108126 · Cited by 256
Neural Image Compression for Gigapixel Histopathology Image Analysis
Tellez, D., Litjens, G. J. S., van der Laak, J. A. W. M., & Ciompi, F. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2936841 · Cited by 246
A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives
Mathis, A., Schneider, S., Lauer, J., & Mathis, M. W. (2020) · Neuron · doi:10.1016/j.neuron.2020.09.017 · Cited by 231
Out-of-Distribution Generalization via Risk Extrapolation (REx)
Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Priol, R. L., & Courville, A. (2020) · arXiv · doi:10.48550/arxiv.2003.00688 · Cited by 407
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Chi, Z., Dong, L., Wei, F., Yang, N., Singhal, S., Wang, W., Song, X., Mao, X.-L., Huang, H., & Zhou, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.280 · Cited by 347
Deep semi-supervised learning for medical image segmentation: A review
Han, K., Sheng, V. S., Song, Y., Liu, Y., Qiu, C., Ma, S., & Liu, Z. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.123052 · Cited by 256
Lu, Y., Liu, J., Zhang, Y., Liu, Y., & Tian, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00514 · Cited by 285
An Overview of Deep Semi-Supervised LearningPreprint
Ouali, Y., Hudelot, C., & Tami, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05278 · Cited by 471
A Comprehensive Survey on Source-Free Domain Adaptation
Li, J., Yu, Z., Du, Z., Zhu, L., & Shen, H. T. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3370978 · Cited by 277
“One most typical contrastive loss is the InfoNCE loss [79], [153]: where V + , V − are the sets of positive and negative pairs about the same sample u respectively, and τ is the temperature parameter.”
BackgroundMethodDatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort
Zhang, Y., Ling, H., Gao, J., Yin, K., Lafleche, J.-F., Barriuso, A., Torralba, A., & Fidler, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01001 · Cited by 259
Scaling Language-Image Pre-Training via Masking
Li, Y., Fan, H., Hu, R., Feichtenhofer, C., & He, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02240 · Cited by 292
“The image/text encoders are trained to minimize a contrastive loss [48].”
BackgroundMethodSelf-citationPersonalized News Recommendation: Methods and Challenges
Wu, C., Wu, F., Huang, Y., & Xie, X. (2023) · ACM Transactions on Information Systems · doi:10.1145/3530257 · Cited by 240
Dual Contrastive Prediction for Incomplete Multi-View Representation Learning
Lin, Y., Gou, Y., Liu, X., Bai, J., Lv, J., & Peng, X. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3197238 · Cited by 255
“Since most existing contrastive learning studies [37], [40] have attributed their success to maximize the lower bound of mutual information, we propose to maximize the mutual information between the representations of different views”
Background3D Human Action Representation Learning via Cross-View Consistency Pursuit
Li, L., Wang, M., Ni, B., Wang, H., Yang, J., & Zhang, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00471 · Cited by 236
“Recently, contrastive methods [37, 55, 48, 18] based on instance discrimination have been proposed for representation learning.”
BackgroundInfluentialUnsupervised Semantic Segmentation by Contrasting Object Mask Proposals
Van Gansbeke, W., Vandenhende, S., Georgoulis, S., & Van Gool, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00990 · Cited by 231
Learning Problem-Agnostic Speech Representations from Multiple Self-Supervised Tasks
Pascual, S., Ravanelli, M., Serrà, J., Bonafonte, A., & Bengio, Y. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2605 · Cited by 222
Unsupervised Scalable Representation Learning for Multivariate Time Series
Franceschi, J.-Y., Dieuleveut, A., & Jaggi, M. (2019) · arXiv · doi:10.48550/arxiv.1901.10738 · Cited by 305
Transfer learning for ECG classification
Weimann, K., & Conrad, T. (2021) · Scientific Reports · doi:10.1038/s41598-021-84374-8 · Cited by 239
VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples
Tian, P., Song, Y., Yang, T., Jiang, W., & Liu, W. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01105 · Cited by 245
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Xiong, L., Xiong, C., Li, Y., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., & Overwijk, A. (2020) · arXiv · doi:10.48550/arxiv.2007.00808 · Cited by 204
Self-citationDual Contrastive Learning for Unsupervised Image-to-Image Translation
Han, J., Shoeiby, M., Petersson, L., & Armin, M. A. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00084 · Cited by 258
Metadata
- Authors
- Oord, A. V. D., Li, Y., & Vinyals, O.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1807.03748
- OpenAlex
- W4297808394
- Cited by
- 6,242
- References
- 50
- Citations by year
- 2026: 77 · 2025: 660 · 2024: 1288 · 2023: 1444 · 2022: 1043 · 2021: 1173 · 2020: 454 · 2019: 88 · 2018: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Keywords
- contrastive predictive coding, unsupervised feature learning, autoregressive models, negative sampling, high-dimensional data, speech representation, image representation, text representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
