The DOI matches its DataCite record; 53 of the 53 references it cites were found in a registry or an index.
Representation Learning with Contrastive Predictive Coding
Open accessAbstract
While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Contrastive Predictive Coding (CPC) [80,143] utilizes a contrastive future prediction task, and it was found that predicting one or two timesteps in the future is (too) easy and results in poor performance, whereas predicting multiple timesteps into the future is much more complex, and leads to very effective representations.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [143]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Using this contrastive loss, self-supervised networks learn to make the representations similar for positive pairs, and dissimilar for negative pairs, and hence are able to cluster similar (or positive) samples together and push apart dissimilar (or negative) samples.”
From Detecting galaxy tidal features using self-supervised representation learning (2024) · cites this paper as Oord et al. 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It means that InfoNCE loss maximizes the agreement of two different augmented sequences' representations from the same sequence while minimizing the agreement with other negative sequences.”
From Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-Supervised Action Recognition (2022) · cites this paper as Oord, Li, and Vinyals 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We introduce the Contrastive Predictive Coding (CPC) framework to human activity recognition, which captures the long-term temporal structure of sensor data streams.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As mentioned in [58], either 𝑐 𝑡 (the output of 𝑔 𝑒𝑛𝑐 ) or 𝑧 𝑡 (the output of 𝑔 𝑎𝑟 ) can be utilized as the representation.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The framework for data augmentation and batch formation plays a critical role in multi-view similarity learning [12,16,44,115], where we start by following the popular SimCLR [16] method.”
From MARs: Multi-view Attention Regularizations for Patch-based Feature Recognition of Space Terrain (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We note that this is a common setting for self-supervised contrastive learners with NCE objectives (Gutmann & Hyvärinen, 2010;Oord et al., 2018), including SimCLR (Chen et al., 2020) and SimCSE (Gao et al., 2021).”
From The Platonic Representation Hypothesis (2024) · cites this paper as Oord et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“The key insight behind this approach is that predicting just the next future timestep involves exploiting the local smoothness of the signal, whereas predicting multiple subsequent timesteps requires inferring the global structure of time-series data.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“Haresamudram et al. [80] adapted Contrastive Predictive Coding (CPC [143]) for HAR, emphasizing the importance of temporal structure in sensor data representation.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [143]
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive predictive coding (CPC) proposed by [109] provides a contrastive learning framework to perform the prediction task using InfoNCE loss.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“It can be seen that CPC does not directly predict future observations x t + k . Instead, it tries to preserve the mutual information of c t and x t + k .”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“This contrastive loss, or InfoNCE loss (Oord et al. 2018), is minimised when the similarity is high for positive pairs and low for negative pairs.”
From Detecting galaxy tidal features using self-supervised representation learning (2024) · cites this paper as Oord et al. 2018
The cited source says what the claim attributes to it.
In the citing paper
“To evaluate the performance of our TS-TCC model, we follow the standard linear benchmarking evaluation scheme [Oord et al. , 2018; Chen et al. , 2020]. Particularly, we train a linear classifier (single MLP layer) on top of a frozen self-supervised pretrained encoder model.”
From Time-Series Representation Learning via Temporal and Contextual Contrasting (2021) · cites this paper as Oord et al. , 2018
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive-based methods usually use a contrastive similarity metric, such as InfoNCE loss [109].”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [109]
The cited source says what the claim attributes to it.
In the citing paper
“This results in representations that encode the high-level information between temporally separated parts of the time-series signal [58], thereby leading to improved downstream recognition performance.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“Contrastive Predictive Coding (CPC) [58], which is the focus of this work, performs contrastive learning over multiple future timestep predictions.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“This framework has resulted in excellent representation learning performance across domains such as computer vision [12, 33, 35], natural language [49], speech recognition [6, 58] and time-series data [39].”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“If extra context from the past is useful for classification, then 𝑐 𝑡 can be used; else 𝑧 𝑡 may be more effective.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [58]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“SimCLR aims to maximize the learned representation similarity between augmented views of the same input.”
From MARs: Multi-view Attention Regularizations for Patch-based Feature Recognition of Space Terrain (2024) · cites this paper as [16]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“More recent advancements in autoregressive modeling such as masked convolutional architectures [18, 19] or self-attention networks [20] could help improve results further.”
BackgroundImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“These techniques made manually specified features largely redundant and have greatly improved state-of-the-art in several real-world applications [1, 2, 3].”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We use the Adam optimizer [32] with a learning rate of 2e-4, and use 8 GPUs each with a minibatch of 8 examples from which the negative samples in the contrastive loss are drawn.”
MethodImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“In our visual representation experiments we use the ILSVRC ImageNet competition dataset [34].”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1×
“We did not use Batch-Norm [38].”
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 2×
“In Word2Vec [9] neighbouring words are predicted using a contrastive loss.”
BackgroundLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“These techniques made manually specified features largely redundant and have greatly improved state-of-the-art in several real-world applications [1, 2, 3].”
BackgroundSelf-citationFaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · cited 1×
“For example, the techniques proposed by [21, 22, 23] were based on triplet losses using a max-margin approach to separate positive from negative examples.”
MethodLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · cited 1×
“[41] is the same skip-thought model with layer normalization trained for 1M iterations. evaluate with 10-fold cross-validation for MR, CR, Subj, MPQA and use the train/test split for TREC.”
MethodIdentity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741 · cited 1×
“We follow the same setup as [36] and use a ResNet v2 101 architecture [37] as the image encoder g enc to extract CPC representations (note that this encoder is not pretrained).”
MethodHinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · cited 2×
“These techniques made manually specified features largely redundant and have greatly improved state-of-the-art in several real-world applications [1, 2, 3].”
BackgroundMethodDistributed Representations of Sentences and DocumentsPreprint
Le, Q. V., & Mikolov, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4053 · Cited by 7,905 · cited 1×
“[40] is an unsupervised approach to learning sentence-level representations.”
BackgroundMining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793 · cited 1×
“For the classification tasks we used the following datasets: movie review sentiment (MR) [43], customer product reviews (CR) [44], subjectivity/objectivity [45], opinion polarity (MPQA) [46] and question-type classification (TREC) [47].”
MethodAsynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 1×
“Here, we take the standard batched A2C [52] agent as base model and add CPC as an auxiliary loss.”
MethodShow and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 2×
“…example, when pre-training a model to do image classification, the induced features transfer reasonably well to other image classification domains, but also lack certain information such as color or the ability to count that are irrelevant for classification but relevant for e.g. image captioning [4].”
BackgroundMethodSelf-citation“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · cited 1×
“tions, and in reinforcement learning we measure how the auxiliary CPC loss speeds up learning of the agent. 3.1 Audio For audio, we use a 100-hour subset of the publicly available LibriSpeech dataset [30]. Although the dataset does not provide labels other than the raw text, we obtained force-aligned phone sequences 4…”
MethodInfluentialRao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 1×
“In neuroscience, predictive coding theories suggest that the brain predicts observations at various levels of abstraction [7, 8].”
BackgroundWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 1×
“More recent advancements in autoregressive modeling such as masked convolutional architectures [18, 19] or self-attention networks [20] could help improve results further.”
BackgroundSelf-citationDistance Metric Learning for Large Margin Nearest Neighbor Classification
Weinberger, K. Q., & Saul, L. K. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1577078 · Cited by 4,652 · cited 1×
“For example, the techniques proposed by [21, 22, 23] were based on triplet losses using a max-margin approach to separate positive from negative examples.”
MethodA theory of cortical responses.
Friston, K. (2005) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2005.1622 · Cited by 4,578 · cited 2×
“In neuroscience, predictive coding theories suggest that the brain predicts observations at various levels of abstraction [7, 8].”
BackgroundMethodLearning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036 · cited 1×
“For example, the techniques proposed by [21, 22, 23] were based on triplet losses using a max-margin approach to separate positive from negative examples.”
MethodZhang, R., Isola, P., & Efros, A. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,417 · cited 4×
“Work shown useful [10, 11].”
BackgroundMethodInfluentialPang, B., & Lee, L. (2004) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1218955.1218990 · Cited by 3,376 · cited 1×
“nd the word embeddings learned by the model. For the classification tasks we used the following datasets: movie review sentiment (MR) [43], customer product reviews (CR) [44], subjectivity/objectivity [45], opinion polarity (MPQA) [46] and question-type classification (TREC) [47]. As in [26] we train a logistic regressi…”
MethodInfluentialLearning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · cited 2×
“The ImageNet dataset has been used to evaluate unsupervised vision models by many authors [28, 11, 35, 10, 29, 36].”
BackgroundMethodUnsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · cited 4×
“8 Relative Position [11] 30.”
BackgroundMethodInfluentialZhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · cited 1×
“We first learn our unsupervised model on the BookCorpus dataset [42], and evaluate the capability of our model as a generic feature extractor by using CPC representations for a set of classification tasks.”
MethodPang, B., & Lee, L. (2005) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1219840.1219855 · Cited by 2,213 · cited 1×
“For the classification tasks we used the following datasets: movie review sentiment (MR) [43], customer product reviews (CR) [44], subjectivity/objectivity [45], opinion polarity (MPQA) [46] and question-type classification (TREC) [47].”
MethodConditional Image Generation with PixelCNN DecodersPreprint
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 2,032 · cited 2×
“More recent advancements in autoregressive modeling such as masked convolutional architectures [18, 19] or self-attention networks [20] could help improve results further.”
BackgroundMethodSelf-citationAnnotating Expressions of Opinions and Emotions in Language
Wiebe, J., Wilson, T., & Cardie, C. (2005) · Computers and the Humanities · doi:10.1007/s10579-005-7880-9 · Cited by 1,794 · cited 1×
“For the classification tasks we used the following datasets: movie review sentiment (MR) [43], customer product reviews (CR) [44], subjectivity/objectivity [45], opinion polarity (MPQA) [46] and question-type classification (TREC) [47].”
MethodPixel Recurrent Neural NetworksPreprint
van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.06759 · Cited by 1,793 · cited 1×
“Next, we use a PixelCNN-style autoregressive model [19] (a convolutional row-GRU PixelRNN [39] gave similar results) to make predictions about the latent activations in following rows top-to-bottom, visualized in Figure 4.”
MethodSelf-citationAdversarial Feature LearningPreprint
Donahue, J., Krähenbühl, P., & Darrell, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09782 · Cited by 1,662 · cited 1×
“The ImageNet dataset has been used to evaluate unsupervised vision models by many authors [28, 11, 35, 10, 29, 36].”
MethodSlow Feature Analysis: Unsupervised Learning of Invariances
Wiskott, L., & B, T. J. S. A. (2002) · Neural Computation · doi:10.1162/089976602317318938 · Cited by 1,358 · cited 1×
“These ’slow features’ [13] that span many time steps are often more interesting (e.g., phonemes and intonation in speech, objects in images, or the story line in books.).”
BackgroundLi, X., & Roth, D. (2002) · International Conference on Computational Linguistics · doi:10.3115/1072228.1072378 · Cited by 1,308 · cited 1×
“For the classification tasks we used the following datasets: movie review sentiment (MR) [43], customer product reviews (CR) [44], subjectivity/objectivity [45], opinion polarity (MPQA) [46] and question-type classification (TREC) [47].”
MethodKiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293 · cited 6×
“As in [26] we train a logistic regression classifier and Table 5: Classification accuracy on five common NLP benchmarks.”
MethodInfluentialExploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02410 · Cited by 1,238 · cited 1×
“Although we cannot evaluate p ( x ) or p ( x | c ) directly, we can use samples from these distributions, allowing us to use techniques such as Noise-Contrastive Estimation [12, 14, 15] and Importance Sampling [16] that are based on comparing the target value with randomly sampled negative values.”
MethodSelf-citationIMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner ArchitecturesPreprint
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.01561 · Cited by 1,049 · cited 3×
“We refer to [50] for all other hyperparameter and implementation details.”
MethodUnsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · cited 2×
“In Computer Vision [28] use a triplet loss on tracked video patches so that patches from the same object at different timesteps are more similar to each other than to random patches.”
BackgroundMethodMINE: Mutual Information Neural EstimationPreprint
Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A. C., & Hjelm, R. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04062 · Cited by 702 · cited 1×
“InfoNCE is also related to MINE [54].”
Multi-task Self-Supervised Visual Learning
Doersch, C., & Zisserman, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.226 · Cited by 637 · cited 3×
“We follow the same setup as [36] and use a ResNet v2 101 architecture [37] as the image encoder g enc to extract CPC representations (note that this encoder is not pretrained).”
MethodTime-Contrastive Networks: Self-Supervised Learning from Video
Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., & Brain, G. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8462891 · Cited by 617
Learning to Generate Reviews and Discovering Sentiment
Radford, A., Jozefowicz, R., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1704.01444 · Cited by 330 · cited 2×
“Skip-thought vectors [26] and Byte mLSTM [27] are alternatives which go beyond word prediction with a Recurrent Neural Network, and use maximum likelihood over sequences of observations.”
BackgroundMethodA Fast and Simple Algorithm for Training Neural Probabilistic Language Models
Mnih, A., & Teh, Y. W. (2012) · arXiv · doi:10.48550/arxiv.1206.6426 · Cited by 315 · cited 1×
“Although we cannot evaluate p ( x ) or p ( x | c ) directly, we can use samples from these distributions, allowing us to use techniques such as Noise-Contrastive Estimation [12, 14, 15] and Importance Sampling [16] that are based on comparing the target value with randomly sampled negative values.”
MethodAdaptive Predictive Coding of Speech Signals
Atal, B. S., & Schroeder, M. R. (1970) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1970.tb04297.x · Cited by 256 · cited 1×
“This idea of predictive coding [5, 6] is one of the oldest techniques in signal processing for data compression.”
MethodAdaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senécal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223 · cited 1×
“though we cannot evaluate p(x) or p(xjc) directly, we can use samples from these distributions, allowing us to use techniques such as Noise-Contrastive Estimation [12, 14, 15] and Importance Sampling [16] that are based on comparing the target value with randomly sampled negative values. In the proposed model, either…”
MethodUnsupervised Learning of Visual Representations using VideosPreprint
Wang, X., & Gupta, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00687 · Cited by 194
Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAPreprint
Hyvärinen, A., & Morioka, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06336 · Cited by 193 · cited 1×
“In Time Contrastive Learning [25] a contrastive loss is used to predict the segment-ID of multivariate time-series as a way to extract features and perform nonlinear ICA.”
MethodSelf-Adaptive Hierarchical Sentence ModelPreprint
Zhao, H., Lu, Z., & Poupart, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.05070 · Cited by 143 · cited 1×
“We note that better [49, 27] results have been published on these target datasets, by transfer learning from a different source task.”
BackgroundTime-Contrastive Networks: Self-Supervised Learning from Multi-view Observation
Sermanet, P., Lynch, C., Hsu, J., & Levine, S. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.69 · Cited by 112 · cited 1×
“More recent work includes Time Contrastive Networks [24] which proposes to minimize distances between embeddings from multiple viewpoints of the same scene and whilst maximizing distances between embeddings extracted from different timesteps.”
BackgroundTime-Contrastive Networks: Self-Supervised Learning from VideoPreprint
Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06888 · Cited by 95
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., et al. (2016) · arXiv · doi:10.48550/arxiv.1612.03801 · Cited by 68
Negro, P. J. (2019) · BENTHAM SCIENCE PUBLISHERS eBooks · doi:10.2174/9781681087665119010016 · cited 2×
“This idea of predictive coding [5, 6] is one of the oldest techniques in signal processing for data compression.”
Method
Research works printed but not linked: 4
- [12][12] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages 297-304, 2010. -
- [31][31] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding . IEEE Signal Processing Society, 2011. -
- [33][33] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(Nov):2579-2605, 2008. -
- [48][48] Sida Wang and Christopher D. Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2 , ACL '12, pages 90-94, 2012. -
Cited by
Sorted by citations
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857
Exploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01549 · Cited by 3,725
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846
Metadata
- Authors
- Oord, A. V. D., Li, Y., & Vinyals, O.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1807.03748
- OpenAlex
- W4297808394
- Cited by
- 6,242
- References
- 50
- Citations by year
- 2026: 77 · 2025: 660 · 2024: 1288 · 2023: 1444 · 2022: 1043 · 2021: 1173 · 2020: 454 · 2019: 88 · 2018: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Keywords
- contrastive predictive coding, unsupervised feature learning, autoregressive models, negative sampling, high-dimensional data, speech representation, image representation, text representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
