The DOI matches its Crossref record; 94 of the 94 references it cites were found in a registry or an index.
Self-supervised Learning: Generative or Contrastive
Open accessAbstract
Deep supervised learning has achieved great success in the last decade. However, its defects of heavy dependence on manual labels and vulnerability to attacks have driven people to find other paradigms. As an alternative, self-supervised learning (SSL) attracts many researchers for its soaring performance on representation learning in the last several years. Self-supervised representation learning leverages input data itself as supervision and benefits almost all types of downstream tasks. In this survey, we take a look into new self-supervised learning methods for representation in computer vision, natural language processing, and graph learning. We comprehensively review the existing empirical methods and summarize them into three main categories according to their objectives: generative, contrastive, and generative-contrastive (adversarial). We further collect related theoretical analyses on self-supervised learning to provide deeper thoughts on why self-supervised learning works. Finally, we briefly discuss open problems and future directions for self-supervised learning. An outline slide for the survey is provided.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Research works printed but not linked: 22
- ganin, 2016, Domain-adversarial training of neural networks, Journal of Machine Learning Research, 17, 2096
- donahue, 2019, Large scale adversarial representation learning, Proc 33rd Int Conf Neural Inf Process Syst, 10541
- ho, 2019, Flow++: Improving flow-based generative models with variational dequantization and architecture design, Proc 36th Int Conf Mach Learn, 2722
- hu, 2019, Strategies for pre-training graph neural networks, Proc Int Conf Learn Representations
- zhu, 2017, Toward multimodal image-to-image translation, Proc 31st Int Conf Neural Inf Process Syst, 465
- goodfellow, 2014, Generative adversarial nets, Proc 27th Int Conf Neural Inf Process Syst, 2672
- arjovsky, 2017, Wasserstein generative adversarial networks, Proc 34th Int Conf Mach Learn, 214
- bachman, 2019, Learning representations by maximizing mutual information across views, Proc 33rd Int Conf Neural Inf Process Syst, 15509
- ballard, 1987, Modular learning in neural networks, Proc 6th Nat Conf Artif Intell, 279
- gutmann, 2010, Noise-contrastive estimation: A new estimation principle for unnormalized statistical models, Proc 13th Int Conf Artif Intell Statist, 297
- vaswani, 2017, Attention is all you need, Proc 31st Int Conf Neural Inf Process Syst, 5998
- van oord, 2016, Pixel recurrent neural networks, Proc 33rd Int Conf Mach Learn, 1747
- kingma, 2018, Glow: Generative flow with invertible 1x1 convolutions, Proc 32nd Int Conf Neural Inf Process Syst, 10215
- you, 2018, Graph convolutional policy network for goal-directed molecular graph generation, Proc 32nd Int Conf Neural Inf Process Syst, 6410
- you, 2018, GraphRNN: Generating realistic graphs with deep auto-regressive models, Proc 35th Int Conf Mach Learn, 5708
- krizhevsky, 2012, ImageNet classification with deep convolutional neural networks, Proc 25th Int Conf Neural Inf Process Syst, 1097
- shen, 2017, Style transfer from non-parallel text by cross-alignment, Proc 31st Int Conf Neural Inf Process Syst, 6830
- shen, 2017, Adversarial representation learning for domain adaptation, Stat, 1050
- sun, 2019, vGraph: A generative model for joint community detection and node representation learning, Proc 33rd Int Conf Neural Inf Process Syst, 512
- radford, 2019, Language models are unsupervised multitask learners, OpenAIRE blog, 1
- van den oord, 2016, WaveNet: A generative model for raw audio, Proc 9th ISCA Speech Synth Workshop, 125
- ng, 2011, Sparse autoencoder, Cs294a lecture notes, 72, 1
Cited by
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
Deep Neural Networks and Tabular Data: A Survey
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3229161 · Cited by 1,025
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021) · arXiv · doi:10.48550/arxiv.2103.10385 · Cited by 908
Self-citationMasked Autoencoders for Point Cloud Self-supervised Learning
Pang, Y., Wang, W., Tay, F. E. H., Liu, W., Tian, Y., & Yuan, L. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20086-1_35 · Cited by 605
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends
Gui, J., Chen, T., Zhang, J., Cao, Q., Sun, Z., Luo, H., & Tao, D. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3415112 · Cited by 552
Self-citationMachine learning for a sustainable energy future
Yao, Z., Lum, Y., Johnston, A., Mejia-Mendoza, L. M., Zhou, X., Wen, Y., Aspuru-Guzik, A., Sargent, E. H., & Seh, Z. W. (2022) · Nature Reviews Materials · doi:10.1038/s41578-022-00490-5 · Cited by 517
Graph Self-Supervised Learning: A Survey
Liu, Y., Jin, M., Pan, S., Zhou, C., Zheng, Y., Xia, F., & Yu, P. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3172903 · Cited by 510
Machine Learning Methods for Small Data Challenges in Molecular Science
Dou, B., Zhu, Z., Merkurjev, E., Ke, L., Chen, L., Jiang, J., Zhu, Y., Liu, J., Zhang, B., & Wei, G.-W. (2023) · Chemical Reviews · doi:10.1021/acs.chemrev.3c00189 · Cited by 499
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
Cheng, H., Zhang, M., & Shi, J. Q. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3447085 · Cited by 497
Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning
Liu, Y., Li, Z., Pan, S., Gong, C., Zhou, C., & Karypis, G. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3068344 · Cited by 492
A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources
Wang, X., Bo, D., Shi, C., Fan, S., Ye, Y., & Yu, P. S. (2023) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2022.3177455 · Cited by 438
Self-supervised learning for recommender systems: A survey
Yu, J., Yin, H., Xia, X., Chen, T., Li, J., & Huang, Z. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3282907 · Cited by 413
“WhileSSLhasbeenextensivelysurveyedinthefieldsof CV,NLP[9],[34]andgraphlearning[35],[36],[37],there hasnotbeenasystematicinvestigationofresearchendeavors onSSRdespitethegrowingnumberofpublications.”
BackgroundZhao, Z., Alzubaidi, L., Zhang, J., Duan, Y., & Gu, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122807 · Cited by 410
Self-citationSelf-Supervised Learning of Graph Neural Networks: A Unified Review
Xie, Y., Xu, Z., Zhang, J., Wang, Z., & Ji, S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3170559 · Cited by 389
Self-citationSelf-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372
“Due to the popularity of SSL, reviews have been published about the technology in general [16], [17], [18] as well as its application to natural language processing (NLP) [19], [20],”
BackgroundSelf-Supervised Learning in Remote Sensing: A review
Wang, Y., Albrecht, C. M., Braham, N. A. A., Mou, L., & Zhu, X. X. (2022) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2022.3198244 · Cited by 364
XSimGCL: Towards Extremely Simple Graph Contrastive Learning for Recommendation
Yu, J., Xia, X., Chen, T., Cui, L., Hung, N. Q. V., & Yin, H. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3288135 · Cited by 322
Self-Supervised Learning for Electroencephalography
Rafiei, M. H., Gauthier, L. V., Adeli, H., & Takabi, D. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3190448 · Cited by 317
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“The taxonomy is similar to the one proposed by Liu et al. [21] but specifically concentrated on time series data.”
MethodSelf-citationSelf-Supervised Learning on Graphs: Contrastive, Generative, or Predictive
Wu, L., Lin, H., Tan, C., Gao, Z., & Li, S. Z. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3131584 · Cited by 290
Semantic segmentation using Vision Transformers: A survey
Thisanke, H., Deshan, C., Chamith, K., Seneviratne, S., Vidanaarachchi, R., & Herath, D. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106669 · Cited by 288
Deep learning methods for molecular representation and property prediction
Li, Z., Jiang, M., Wang, S., & Zhang, S. (2022) · Drug Discovery Today · doi:10.1016/j.drudis.2022.103373 · Cited by 279
AMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103982 · Cited by 278
Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging
Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., Tomasev, N., Mitrović, J., Strachan, P., et al. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01049-7 · Cited by 250
Self-supervised learning methods and applications in medical imaging analysis: a survey
Shurrab, S., & Duwairi, R. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1045 · Cited by 254
Yang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2023) · arXiv · doi:10.48550/arxiv.2306.11489 · Cited by 239
Yang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3360454 · Cited by 238
A Review of Modern Audio Deepfake Detection Methods: Challenges and Future Directions
Almutairi, Z., & Elgibreen, H. (2022) · Algorithms · doi:10.3390/a15050155 · Cited by 187
“Following this, the available faked audio datasets were summarized, and the discussed methods were also compared.”
Self-supervised learning for videos: A survey
Schiappa, M. C., Rawat, Y. S., & Shah, M. (2023) · ACM Computing Surveys · doi:10.1145/3577925 · Cited by 155
Albelwi, S. (2022) · Entropy · doi:10.3390/e24040551 · Cited by 150
Contrastive learning based self-supervised time-series analysis
Pöppelbaum, J., Chadha, G. S., & Schwung, A. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2021.108397 · Cited by 131
Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series
Tipirneni, S., & Reddy, C. K. (2022) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3516367 · Cited by 130
A Mathematical Investigation of Hallucination and Creativity in GPT Models
Lee, M. (2023) · Mathematics · doi:10.3390/math11102320 · Cited by 105
Self-Supervised Contrastive Learning for Medical Time Series: A Systematic Review
Liu, Z., Alavi, A., Li, M., & Zhang, X. (2023) · Sensors · doi:10.3390/s23094221 · Cited by 90
Split Federated Learning for 6G Enabled-Networks: Requirements, Challenges, and Future Directions
Hafi, H., Brik, B., Frangoudis, P. A., Ksentini, A., & Bagaa, M. (2024) · IEEE Access · doi:10.1109/access.2024.3351600 · Cited by 90
Kotei, E., & Thirunavukarasu, R. (2023) · Information · doi:10.3390/info14030187 · Cited by 89
Spatially Consistent Representation Learning
Roh, B., Shin, W., Kim, I., & Kim, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00120 · Cited by 85
Hou, W., Lin, C., Yu, L., Qin, J., Yu, R., & Wang, L. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3253760 · Cited by 74
A review on the recent applications of deep learning in predictive drug toxicological studies
Sinha, K., Ghosh, N., & Sil, P. C. (2023) · Chemical Research in Toxicology · doi:10.1021/acs.chemrestox.2c00375 · Cited by 64
Practically Adopting Human Activity Recognition
Xu, H., Zhou, P., Tan, R., & Li, M. (2023) · Proceedings of the 29th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3570361.3613299 · Cited by 55
A Survey on Contrastive Self-Supervised Learning
Jaiswal, A., Babu, A. R., Zadeh, M. Z., Banerjee, D., & Makedon, F. (2020) · Technologies · doi:10.3390/technologies9010002 · Cited by 154
Yang, P., Yin, X., Lu, H., Hu, Z., Zhang, X., Jiang, R., & Lv, H. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102539 · Cited by 50
Breaking away from labels: The promise of self-supervised machine learning in intelligent health
Spathis, D., Perez-Pozuelo, I., Marques-Fernandez, L., & Mascolo, C. (2022) · Patterns · doi:10.1016/j.patter.2021.100410 · Cited by 39
Building towards Automated Cyberbullying Detection: A Comparative Analysis
Al-Harigy, L. M., Al-Nuaim, H. A., Moradpoor, N., & Tan, Z. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/4794227 · Cited by 32
Cheng, D., Zhang, L., Qin, L., Wang, S., Wu, H., & Song, A. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3373019 · Cited by 31
Qin, X., Wang, J., Ma, S., Lu, W., Zhu, Y., Xie, X., & Chen, Y. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599360 · Cited by 29
Learning cortical representations through perturbed and adversarial dreaming
Deperrois, N., Petrovici, M. A., Senn, W., & Jordan, J. (2022) · eLife · doi:10.7554/elife.76384 · Cited by 28
Deldari, S., Xue, H., Saeed, A., He, J., Smith, D. V., & Salim, F. D. (2022) · arXiv · doi:10.48550/arxiv.2206.02353 · Cited by 26
Shrivastava, A. D., & Kell, D. B. (2021) · Molecules · doi:10.3390/molecules26072065 · Cited by 25
“Of the various strategies available, those using contrastive learning [11,62,66,169–171] seem to be the most apposite.”
BackgroundSelf-Supervised Pre-training for Time Series Classification
Shi, P., Ye, W., & Qin, Z. (2021) · 2021 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533426 · Cited by 25
Text Transformations in Contrastive Self-Supervised Learning: A Review
Bhattacharjee, A., Karami, M., & Liu, H. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/757 · Cited by 20
Pre-trained Maldi Transformers improve MALDI-TOF MS-based prediction
(2025) · doi:10.1016/j.compbiomed.2025.109695 · Cited by 17
If only we had more data!: Sensor-Based Human Activity Recognition in Challenging Scenarios
Plötz, T. (2023) · 2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops) · doi:10.1109/percomworkshops56833.2023.10150267 · Cited by 16
“SSL has seen a surge in popularity and uptake in the community of sensor-based HAR, which is reasoned by the break-throughs w.r.t. deriving effective sensor data representations for improved downstream recognition tasks.”
BackgroundInfluentialA Label Efficient Semi Self-Supervised Learning Framework for IoT Devices in Industrial Process
Bharti, V., Kumar, A., Purohit, V., Singh, R., Singh, A. K., & Singh, S. K. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3289184 · Cited by 15
Wang, R., Liu, Z., Pang, J., Sun, J., Xiang, M., & Ning, X. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3579060 · Cited by 9
Learning beyond sensations: How dreams organize neuronal representations
Deperrois, N., Petrovici, M. A., Senn, W., & Jordan, J. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105508 · Cited by 9
Metadata
- Authors
- Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J.
- Venue
- IEEE Transactions on Knowledge and Data Engineering
- Published
- 2021
- Pages
- 1-1
- ISSN
- 1041-4347 (ISSN-L)
- 1558-2191
- 2326-3865
- OpenAlex
- W3035725276
- Cited by
- 1,087
- References
- 94
- FWCI
- 65.46
- Type
- journal article
- Language
- en
- Open access
- green
- Funders
- National Natural Science Foundation of China
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.