Self-Supervised Representation Learning: Introduction, advances, and challenges
Open accessThe DOI matches its Crossref record; 111 of the 125 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Self-supervised representation learning (SSRL) methods aim to provide powerful, deep feature learning without the requirement of large annotated data sets, thus alleviating the annotation bottleneck—one of the main barriers to the practical deployment of deep learning today. These techniques have advanced rapidly in recent years, with their efficacy approaching and sometimes surpassing fully supervised pretraining alternatives across a variety of data modalities, including image, video, sound, text, and graphs. This article introduces this vibrant area, including key concepts, the four main families of approaches and associated state-of-the-art techniques, and how self-supervised methods are applied to diverse modalities of data. We further discuss practical considerations including workflows, representation transferability, and computational cost. Finally, we survey major open challenges in the field, that provide fertile ground for future work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The various self-supervised pretexts are divided into five broad families: transformation prediction, masked prediction, instance discrimination, clustering, and contrastive instance discrimination.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [28]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Contrastive learning [56-58] is in fact a training procedure used in recent artificial intelligence models as a way to efficiently learn semantic representations.”
From How Adversarial REM Dreams May Facilitate Creativity, and Why We Become Aware of Them (2024) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“For example, Ericsson et al. [28] provides a very comprehensive review of pretext tasks for multiple modalities, including image, video, text, audio, time series, and graph.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [28]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“State of the art architectures usually start with CNN representation encoding hθ(·), with ResNet [1] being widely used, before appending task-specific”
InfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“long been used to model the context while recent works have moved to Transformer-based architectures with self-attention [2], which allow longer range connections to be made across words in a sentence, but require more data for training.”
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 9×
“Self-supervised strategies have been leveraged successfully to improve sample efficiency of learning across a variety of modalities, from image [5]–[7], video [8], [9], speech [10], [11], text [12], [13], and graphs [14], [15].”
InfluentialImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“As in much of computer vision, ImageNet [35] is the most typical source data set for self-supervised pretraining [5]–[7], [40], consisting of 1.28 million training images across 1,000 object categories, with the most commonly used resolution at 224 × 224.”
InfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 9×
“Self-supervised strategies have been leveraged successfully to improve sample efficiency of learning across a variety of modalities, from image [5]–[7], video [8], [9], speech [10], [11], text [12], [13], and graphs [14], [15].”
InfluentialAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 1×
“One notable version is a vision transformer [58], which is increasingly being used by recent self-supervised methods on image data [43].”
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · cited 2×
“These range from classic shallow approaches such as Gaussian mixtures [21] to deep methods such as variational autoencoders (VAEs) and generative adversarial networks (GANs) [18].”
Semi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853 · cited 1×
“Graph NNs (GNNs) [89] have advanced all these tasks significantly, especially where massive-labeled data are available.”
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 6×
“With data annotation being a major bottleneck, NLP was the first discipline to make major and successful use of self-supervision [22].”
Influential
Research works printed but not linked: 4
- 0, Youtube-8m A large and diverse labeled video dataset for video understanding research
- gokaslan, 2019, OpenWebText Corpus
- radford, 2019, Language Models are Unsupervised Multitask Learners
- goodfellow, 2016, Deep Learning
Cited by
SPot-the-Difference Self-supervised Pre-training for Anomaly Detection and Segmentation
Zou, Y., Jeong, J., Pemula, L., Zhang, D., & Dabeer, O. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20056-4_23 · Cited by 651
BockNet: Blind-Block Reconstruction Network With a Guard Window for Hyperspectral Anomaly Detection
Wang, D., Zhuang, L., Gao, L., Sun, X., Huang, M., & Plaza, A. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3335484 · Cited by 579
“Self-SL comprises two key steps: 1) constructing input samples and corresponding supervision information on unlabeled data and 2) designing the supporting pretext task according to specific target tasks, aiming to extract the intrinsic data features that are profitable to the target task [59].”
BackgroundZhao, Z., Alzubaidi, L., Zhang, J., Duan, Y., & Gu, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122807 · Cited by 410
A review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372
“Due to the popularity of SSL, reviews have been published about the technology in general [16], [17], [18] as well as its application to natural language processing (NLP) [19], [20],”
BackgroundSelf-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“For example, Ericsson et al. [28] provides a very comprehensive review of pretext tasksformultiplemodalities,includingimage,video,text,audio, timeseries,andgraph.”
BackgroundReview of multimodal machine learning approaches in healthcare
Krones, F., Marikkar, U., Parsons, G., Szmul, A., & Mahdi, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102690 · Cited by 235
Self-Supervised Multimodal Learning: A Survey
Zong, Y., Aodha, O. M., & Hospedales, T. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429301 · Cited by 221
“Recently, self-supervised learning (SSL) [9], [10] has beguntoalleviatethis issuebygeneratingsupervisionfrom readily available unannotated data.”
Self-citationBS3LNet: A New Blind-Spot Self-Supervised Learning Network for Hyperspectral Anomaly Detection
Gao, L., Wang, D., Zhuang, L., Sun, X., Huang, M. J., & Plaza, A. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3246565 · Cited by 217
Current AI technologies in cancer diagnostics and treatment
Tiwari, A., Mishra, S., & Kuo, T.-R. (2025) · Molecular Cancer · doi:10.1186/s12943-025-02369-9 · Cited by 207
A Comprehensive Survey of Deep Learning Approaches in Image Processing
Τρίγκα, Μ., & Dritsas, E. (2025) · Sensors · doi:10.3390/s25020531 · Cited by 207
“[115] Context Autoencoders: demonstrates their use for effective representation learning in image processing tasks.”
Hu, S. X., Li, D., Stuhmer, J., Kim, M., & Hospedales, T. M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00886 · Cited by 185
“Self-supervised & few-shot Our pipeline extends the typical unsupervised pre-train → supervised fine-tune workflow of the self-supervised research community [28, 38], which has recently demonstrated strong performance for low-shot supervised learning [15, 18, 27].”
Self-citationSliding Dual-Window-Inspired Reconstruction Network for Hyperspectral Anomaly Detection
Wang, D., Zhuang, L., Gao, L., Sun, X., Zhao, X., & Plaza, A. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3351179 · Cited by 168
“Self-supervised learning is similar to unsupervised learning in that only unlabeled HSIs are required, yet it generates supervision signals from the raw data for network training [59], [60].”
BackgroundLarge AI Model-Based Semantic Communications
Jiang, F., Peng, Y., Dong, L., Wang, K., Yang, K., Pan, C., & You, X. (2024) · IEEE Wireless Communications · doi:10.1109/mwc.001.2300346 · Cited by 161
“The multi-head attention mechanism develops a strong understanding of semantics and knowledge structures, hence LAMs can give high-quality semantic representation of input data.”
BackgroundResearch on multimodal techniques for arc detection in railway systems with limited data
Yan, J., Cheng, Y., Zhang, F., Li, M., Zhou, N., Jin, B., Wang, H., Yang, H., & Zhang, W. (2025) · Structural Health Monitoring · doi:10.1177/14759217251336797 · Cited by 142
“Semi-supervised learning 36 and self-supervised learning 37 have emerged as leading strategies for integrating unlabeled data.”
Wang, D., Zhuang, L., Gao, L., Sun, X., Huang, M. J., & Plaza, A. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3276175 · Cited by 133
“…pixels (indicated by the red arrow), thereby highlighting the significant differences between these pixels and shuffled backgrounds. designed according to the target task requirements, aiming at mining more purposeful features from unlabeled data and hopefully getting the best of both areas [49].”
BackgroundDexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic PlayPreprint
Guzey, I., Evans, B., Chintala, S., & Pinto, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.12076 · Cited by 128
“Feature Learning: To extract useful representations from the play data we employ SSL, which tries to learn a low dimensional representation from high-dimensional observations [18].”
MethodKong, W., Jiao, X., Xu, Y., Zhang, B., & Yang, Q. (2023) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2023.3264908 · Cited by 127
“CL is an individual discrimination approach of self-supervised learning in the computer vision (CV) field, and it has low complexity and fast convergence speed compared with pixel-level generative methods [18].”
BackgroundSelf-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works
Tao, C., Qi, J., Guo, M., Zhu, Q., & Li, H. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3276853 · Cited by 123
DDPM-CD: Denoising Diffusion Probabilistic Models as Feature Extractors for Change DetectionPreprint
Bandara, W. G. C., Nair, N. G., & Patel, V. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.11892 · Cited by 112
“Firstly, the model learns to predict certain features of the input images (pretext task) [33] and then leverages this pre-trained knowledge for downstream tasks like change detection.”
(2013) · Journal of Neural Engineering · doi:10.1088/issn.1741-2552 · Cited by 104
Self-supervised learning for medical image analysis: a comprehensive review
Rani, V., Kumar, M., Gupta, A., Sachdeva, M., Mittal, A., & Kumar, K. (2024) · Evolving Systems · doi:10.1007/s12530-024-09581-w · Cited by 103
Nicheformer: a foundation model for single-cell and spatial omics
Tejada-Lapuerta, A., Schaar, A. C., Gutgesell, R., Palla, G., Halle, L., Minaeva, M., Vornholz, L., Dony, L., Drummer, F., Richter, T., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02814-z · Cited by 100
“We combined all technology-specific datasets and pad missing genes.”
ProtoCLIP: Prototypical Contrastive Language Image Pretraining
Chen, D., Zhao, W., Liu, F., Yang, Z., Zheng, S., Tan, Y., & Zhou, E. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3335859 · Cited by 99
“Self-supervised learning (SSL) [35] aims at learning meaningful representations without human supervision.”
Self-supervised Learning for Electroencephalogram: A Systematic Survey
Weng, W., Gu, Y., Guo, S., Ma, Y., Yang, Z., Liu, Y., & Chen, Y. (2025) · ACM Computing Surveys · doi:10.1145/3736574 · Cited by 98
Cardiac Arrhythmia Classification Using Advanced Deep Learning Techniques on Digitized ECG Datasets
Sattar, S., Mumtaz, R., Qadir, M., Mumtaz, S., Khan, M. A., Waele, T. D., Poorter, E. D., Moerman, I., & Shahid, A. (2024) · Sensors · doi:10.3390/s24082484 · Cited by 98
“These techniques are helpful when we have a large amount of unlabeled data in addition to few labeled data [60].”
BackgroundDeep learning based single sample face recognition: a survey
Liu, F., Chen, D., Wang, F., Li, Z., & Xu, F. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10240-2 · Cited by 94
Transformers for Molecular Property Prediction: Lessons Learned from the Past Five Years
Sultan, A., Sieg, J., Mathea, M., & Volkamer, A. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00747 · Cited by 90
DAS-N2N: machine learning distributed acoustic sensing (DAS) signal denoising without clean data
Lapins, S., Butcher, A., Kendall, J. M., Hudson, T., Stork, A. L., Werner, M. J., Gunning, J. S., & Brisbourne, A. M. (2023) · Geophysical Journal International · doi:10.1093/gji/ggad460 · Cited by 89
Machine learning and Bayesian inference in nuclear fusion research: an overview
Pavone, A., Merlo, A., Kwak, S., & Svensson, J. (2023) · Plasma Physics and Controlled Fusion · doi:10.1088/1361-6587/acc60f · Cited by 85
SwitchTab: Switched Autoencoders Are Effective Tabular Learners
Wu, J., Chen, S., Zhao, Q., Sergazinov, R., Chen, L., Liu, S., Zhao, C., Xie, T., Guo, H., Ji, C., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29523 · Cited by 84
AI-Synthesized Voice Detection Using Neural Vocoder Artifacts
Sun, C., Jia, S., Hou, S., & Lyu, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00097 · Cited by 81
“This approach is similar to self-supervised representation learning [28].”
SELFormer: molecular representation learning via SELFIES language models
Yüksel, A., Ulusoy, E., Ünlü, A., & Doğan, T. (2023) · Machine Learning: Science and Technology · doi:10.1088/2632-2153/acdb30 · Cited by 81
“Overall, representation learning has emerged as an important subfield of artificial learning, as it improves performance and applicability across a broad range of tasks (Ericsson et al., 2022).”
BackgroundUniversal Time-Series Representation Learning: A Survey
Trirat, P., Shin, Y., Kang, J.-H., Nam, Y., Na, J., Bae, M., Kim, J., Kim, B., & Lee, J.-G. (2026) · ACM Computing Surveys · doi:10.1145/3817600 · Cited by 73
A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
Wang, W.-C., Ahn, E., Feng, D. D., & Kim, J. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1406-4 · Cited by 72
ART: A Unified Unsupervised Framework for Incident Management in Microservice Systems
Sun, Y., Shi, B., Mao, M., Ma, M., Xia, S., Zhang, S., & Pei, D. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695495 · Cited by 72
Li, H., Liu, H., Fu, H., Xu, Y., Shu, H., Niu, K., Hu, Y., & Liu, J. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102945 · Cited by 72
Cross-Domain HAR: Few-Shot Transfer Learning for Human Activity Recognition
Thukral, M., Haresamudram, H., & Plötz, T. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3704921 · Cited by 71
Deep Learning Methods for Semantic Segmentation in Remote Sensing with Small Data: A Survey
Yu, A., Quan, Y., Yu, R., Guo, W., Wang, X., Hong, D., Zhang, H., Chen, J., Hu, Q., & He, P. (2023) · Remote Sensing · doi:10.3390/rs15204987 · Cited by 70
Bandara, W. G. C., Nair, N. G., & Patel, V. M. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00513 · Cited by 70
“Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. tions by assuming concepts like the cluster [15], continuity/smoothness [50], and entropy [35] assumptions.”
Zhang, T., Gao, P., Dong, H., Zhuang, Y., Wang, G., Zhang, W., & Chen, H. (2022) · Remote Sensing · doi:10.3390/rs14225675 · Cited by 67
EEG foundation models: a critical review of current progress and future directions
Kuruppu, G., Wagh, N., Křemen, V., & Varatharajah, Y. (2026) · Journal of Neural Engineering · doi:10.1088/1741-2552/ae4455 · Cited by 65
Learning Clustering-based Prototypes for Compositional Zero-shot Learning
Qu, H., Wei, J., Shu, X., & Wang, W. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2502.06501 · Cited by 63
“Self-supervised representation learning (SSRL) meth-ods [62, 63, 64] aim to construct a well-structured embedding space without requiring extensively annotated datasets.”
AI-Driven Wireless Positioning: Fundamentals, Standards, State-of-the-Art, and Challenges
Pan, G., Gao, Y., Gao, Y., Yu, W., Zhong, Z., Yang, X., Guo, X., & Xu, S. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3648577 · Cited by 61
“…objective of a GAN can be formulated as 5) Other Advanced AI Techniques: Beyond the AI algorithms discussed, advanced techniques such as knowledge distillation [185], ensemble learning [186], and self-supervised learning [187] offer additional avenues to enhance wireless positioning systems.”
Time series classification based on convolutional network with a Gated Linear Units kernel
Liu, C., Zhen, J., & Shan, W. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106296 · Cited by 61
Holste, G., Oikonomou, E. K., Mortazavi, B. J., Wang, Z., & Khera, R. (2024) · Communications Medicine · doi:10.1038/s43856-024-00538-3 · Cited by 60
“…forms, including pretext tasks(e.g.,predictwhattransformationhasbeenappliedtoanimageorsolve ajigsawpuzzlebycorrectlyarrangingjumbledimagepatches) 12,13 , masked image modeling (remove patches of an image, then reconstruct the missing signal from the present patches) 14 – 16 , and contrastive…”
Collaborative contrastive learning for hypergraph node classification
Wu, H., Li, N., Zhang, J., Chen, S., Ng, M. K., & Long, J. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109995 · Cited by 59
Controllable Data Generation by Deep Learning: A Review
Wang, S.-Y., Du, Y., Guo, X., Pan, B., Qin, Z. S., & Zhao, L. (2024) · ACM Computing Surveys · doi:10.1145/3648609 · Cited by 58
“Deep representation learning has a huge impact on the downstream tasks, such as prediction [60, 61], classification [62, 63] and self-supervision [64, 65].”
BackgroundSenane, Z., Cao, L., Buchner, V., Tashiro, Y., You, L., Herman, P. A., Nordahl, M. G., Tu, R., & von Ehrenheim, V. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671673 · Cited by 58
Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis
Morano, J., Fazekas, B., Sükei, E., Fecso, R., Emre, T., Gumpinger, M., Faustmann, G., Oghbaie, M., Schmidt-Erfurth, U. M., & Bogunović, H. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01852-3 · Cited by 57
“DINOv2waspretrainedonaprivatedatasetof 142MnaturalimagesusinganSSLapproachbasedonself-distillation.”
InfluentialYazdani, E., Ziarati, N. K., Cheshmi, S. S., Sadeghi, M., Geramifar, P., Vosoughi, H., Kazemi-Jahromi, M., & Kheradpisheh, S. R. (2024) · Cancer Imaging · doi:10.1186/s40644-024-00675-x · Cited by 57
“Instead, it generates pseudo-labels from the data using pretext tasks, such as masking, predicting, or reconstructing parts of the data [30].”
MethodReConTab: Regularized Contrastive Representation Learning for Tabular DataPreprint
Chen, S., Wu, J., Hovakimyan, N., & Yao, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.18541 · Cited by 57
“In computer vision, deep representation learning methodologies have emerged as potent tools, capitalizing on self-and semi-supervised training paradigms [44, 22, 45].”
BackgroundBernabé, P., Gotlieb, A., Legeard, B., Marijan, D., Sem-Jacobsen, F. O., & Spieker, H. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3322690 · Cited by 56
“Self-supervision means to extract pseudo-labels from the unlabelled data set [7].”
BackgroundZhang, Z., Li, X., Li, H., Dunkin, F., Li, B., & Li, Z. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3436841 · Cited by 55
“Distinct from traditional self-supervised learning methods [25], [26] that solely focus on the features within the sample itself and external to the sample.”
BackgroundDelving into Inter-Image Invariance for Unsupervised Visual Representations
Xie, J., Zhan, X., Liu, Z., Ong, Y.-S., & Loy, C. C. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01681-x · Cited by 54
“A more detailed treat-ment is provided in Ericsson et al (2022).”
Self-citationXiao, W. J., & Hu, J. (2023) · Applied Sciences · doi:10.3390/app13158933 · Cited by 54
Unpaired Multi-Domain Causal Representation Learning
Sturma, N., Squires, C., Drton, M., & Uhler, C. (2023) · neural information processing systems · doi:10.52202/075280-1496 · Cited by 52
Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation
Marsocci, V., & Scardapane, S. (2023) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2023.3280029 · Cited by 51
“Recently, referring to CV in general, several innovations have been proposed [53], [54], reviewed in many surveys [24], [55], [56], [57], [58].”
Background6G WavesFM: A Foundation Model for Sensing, Communication, and Localization
Aboulfotouh, A., Mohammed, E., & Abou-Zeid, H. (2025) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2025.3600616 · Cited by 50
“In the SSL paradigm, the model learns from the structure of the data which provides the supervision [6], [7], [8].”
Decoupling Common and Unique Representations for Multimodal Self-supervised Learning
Wang, Y., Albrecht, C. M., Braham, N. A. A., Liu, C., Xiong, Z., & Zhu, X. X. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73397-0_17 · Cited by 49
“Self-supervised learning has achieved breakthroughs in machine learning [13] and many other communities [25, 43].”
Holo-Dex: Teaching Dexterity with Immersive Mixed Reality
Arunachalam, S. P., Güzey, I., Chintala, S., & Pinto, L. (2022) · arXiv · doi:10.48550/arxiv.2210.06463 · Cited by 48
“A. Visual Self-Supervised Learning Self-Supervised Learning (SSL) focuses on obtaining low-dimensional embeddings z from high-dimensional observations o [62].”
MethodA Critical Review of Methods and Challenges in Large Language Models
Asgari, R., Moradi, M., Yan, K., Colwell, D., & Samwald, M. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.061263 · Cited by 48
“LLMs are typically pre-trained in a self-supervised manner, where no labeled training samples are utilized to direct the training process [18].”
Li, W., Hacid, H., Almazrouei, E., & Debbah, M. A. (2023) · AI · doi:10.3390/ai4030039 · Cited by 47
Encrypted Network Traffic Classification using Self-supervised Learning
Towhid, M. S., & Shahriar, N. (2022) · IEEE International Conference on Network Softwarization (NetSoft) · doi:10.1109/netsoft54395.2022.9844044 · Cited by 47
“Usually, a self-supervised learning algorithm has two stages [10]–[12].”
BackgroundBIRB: A Generalization Benchmark for Information Retrieval in BioacousticsPreprint
Hamer, J., Triantafillou, E., van Merriënboer, B., Kahl, S., Klinck, H., Denton, T., & Dumoulin, V. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.07439 · Cited by 46
“Moreover, recent empirical results have shown the power of self-supervision in learning rich representations for down-stream tasks (Ericsson et al., 2022) and assist in labeling large unsupervised datasets.”
BackgroundWang, X., Fang, Y., Wang, Q., Yap, P., Zhu, H., & Liu, M. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103403 · Cited by 46
Cross-Domain HAR: Few Shot Transfer Learning for Human Activity Recognition
Thukral, M., Haresamudram, H., & Ploetz, T. (2023) · arXiv · doi:10.48550/arxiv.2310.14390 · Cited by 45
Holo-Dex: Teaching Dexterity with Immersive Mixed Reality
Arunachalam, S. P., Güzey, I., Chintala, S., & Pinto, L. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160547 · Cited by 45
“A. Visual Self-Supervised Learning Self-Supervised Learning (SSL) focuses on obtaining low-dimensional embeddings z from high-dimensional observations o [62].”
MethodUseful blunders: Can automated speech recognition errors improve downstream dementia classification?
Li, C., Xu, W., Cohen, T., & Pakhomov, S. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104598 · Cited by 45
“SSL is a subcategory of unsupervised learning that concerns approaches to extract information from the input data itself as the source of labels used to learn internal representations (for reviews, see Mohamed et al. [15], Ericsson et al. [16]), thus alleviating the manual annotation bottleneck.”
BackgroundSelf-supervised domain adaptation for machinery remaining useful life prediction
Xuan, Q. L., Munderloh, M., & Östermann, J. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110296 · Cited by 44
One-Shot Transfer of Affordance Regions? AffCorrs!Preprint
Hadjivelichkov, D., Zwane, S., Deisenroth, M. P., Agapito, L., & Kanoulas, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.07147 · Cited by 44
“Meanwhile, the self-supervision and un-supervised learning paradigms, present an alternative direction, which alleviates the data annotation bottleneck [5], and could enable robots to learn continually by themselves [6].”
BackgroundApplications of Self-Supervised Learning to Biomedical Signals: A Survey
Pup, F. D., & Atzori, M. (2023) · IEEE Access · doi:10.1109/access.2023.3344531 · Cited by 43
“Self-supervised learning attempts to address the issue of having limited annotated data by extracting general-purpose features from vast unlabeled data [36]; hence, it is usually referred to as an unsupervised technique.”
BackgroundLearning a deep language model for microbiomes: The power of large scale unlabeled microbiome data
Pope, Q., Varma, R., Tataru, C. A., David, M. M., & Fern, X. Z. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011353 · Cited by 43
Machine learning in cartography
Harrie, L., Touya, G., Oucheikh, R., Ai, T., Courtial, A., & Richter, K. (2024) · Cartography and Geographic Information Science · doi:10.1080/15230406.2023.2295948 · Cited by 42
“These methods can be used for a large variety of data and downstream tasks, for more information readers can refer to two recent reviews on the subject: one on general purposes (Ericsson et al., 2022) and one on geographic information-related tasks (Corcoran & Spasić, 2023).”
MethodA Survey of Methods for Automated Quality Control Based on Images
Diers, J., & Pigorsch, C. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01822-w · Cited by 41
“The research field of representation learning (Bengio et al., 2013; Ericsson et al., 2022; Jaiswal et al., 2021b) develops meth-ods to extract better embeddings for images.”
BackgroundA Comprehensive Survey on Self-Supervised Learning for Specific Emitter Identification
Liu, C., Gui, G., Wang, Y., Ohtsuki, T. O., Niyato, D. T., & Shen, X. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3588171 · Cited by 41
“Another problem within the instance discrimination framework is the lack of intraclass variability, which does not enable the model to learn the inherent variation within each class [106].”
Aruna Gladys, A., & Vetriselvi, V. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111553 · Cited by 40
Günther, F., Marelli, M., Tureski, S., & Petilli, M. A. (2023) · Psychological Review · doi:10.1037/rev0000392 · Cited by 38
PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction
Wenteler, A., Occhetta, M., Branson, N., Curean, V., Huebner, M., Dee, W., Connell, W., Chung, S. P., Hawkins-Hooker, A., Ektefaie, Y., et al. (2024) · doi:10.1101/2024.10.02.616248 · Cited by 37
“Among these, single-* cell foundation models (scFMs) leverage vast amounts of unlabeled transcriptomic single-cell RNA sequencing (scRNA-seq) data to learn contextualized representations through self-supervised pre-training (Ericsson et al., 2022).”
Contrastive Audio-Language Learning for Music
Manco, I., Benetos, E., Quinton, E., & Fazekas, G. (2022) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7316743 · Cited by 36
Zingman, I., Stierstorfer, B., Lempp, C., & Heinemann, F. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103067 · Cited by 35
Zhang, G., Zhang, Y., & Tan, Z. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612297 · Cited by 35
“Self-supervised learning (SSL) for vision aims to learn generalpurpose representations without human supervision [12].”
BackgroundMulti-level multi-type self-generated knowledge fusion for cardiac ultrasound segmentation
Yu, C., Li, S., Ghista, D. N., Gao, Z., Zhang, H., Ser, J. D., & Xu, L. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.004 · Cited by 35
Maven: a multimodal foundation model for supernova science
Zhang, G., Helfer, T., Gagliano, A. T., Mishra-Sharma, S., & Villar, V. A. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad990d · Cited by 34
Sinha, S., Dey, S., & Saha, G. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101599 · Cited by 33
Revisiting Learnable Affines for Batch Norm in Few-Shot Transfer Learning
Yazdanpanah, M., Rahman, A. A., Chaudhary, M., Desrosiers, C., Havaei, M., Belilovsky, E., & Kahou, S. E. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00890 · Cited by 33
“Progress in self-training [39] and self-supervised learning [6] meth-ods have led to promising solutions for CDFSL problems.”
BackgroundExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasets
Amiriparian, S., Packań, F., Gerczuk, M., & Schuller, B. W. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-280 · Cited by 33
A survey on self-supervised learning: Recent advances and open problems
Zhang, J., Yang, L., Mohammadabadi, S. M. S., & Yan, F. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131409 · Cited by 32
Zhao, S., Jin, M., Hou, Z., Yang, C., Li, Z., Wen, Q., & Wang, Y. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679741 · Cited by 32
Lui, C. F., Maged, A., & Xie, M. (2023) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-023-02232-y · Cited by 31
“In general, self-supervised learning methods aim to acquire generally transferable knowledge by pre-training machine learning model from unlabeled data with pseudo-labels (Ericsson et al., 2022).”
BenchMD: A Benchmark for Unified Learning on Medical Images and SensorsPreprint
Wantlin, K., Wu, C., Huang, S.-C., Banerjee, O., Dadabhoy, F. Z., Mehta, V. V., Han, R., Cao, F., Narayan, R. R., Colak, E., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08486 · Cited by 31
“Recent advances in transformers and self-supervised learning (SSL) have enabled state-of-the-art performance across many modalities, including text, images and videos [18, 41].”
BackgroundFrequency-Aware Masked Autoencoders for Multimodal Pretraining on BiosignalsPreprint
Liu, R., Zippi, E. L., Pouransari, H., Sandino, C., Nie, J., Goh, H., Azemi, E., & Moin, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05927 · Cited by 31
“While biosignals are crucial for diverse applications such as human-computer interaction, acquiring an ample amount of labeled biosignals is a labor-intensive process that requires the involvement of domain experts (Ericsson et al., 2022).”
BackgroundMeta-Learning for Wireless Communications: A Survey and a Comparison to GNNs
Zhao, B., Wu, J., Ma, Y., & Yang, C. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3380512 · Cited by 31
“…learning: This approach addresses the expensive data collection and manual annotation issue faced by supervised learning [45], [46], [47], [48], which employs a DNN to predict one part of the input data – or an automatically generated pseudo label – given another part of the input [45], [46].”
BackgroundMethodA survey of visual neural networks: current trends, challenges and opportunities
Feng, P., & Tang, Z. (2022) · Multimedia Systems · doi:10.1007/s00530-022-01003-8 · Cited by 30
“However, the IGPT has a higher computational cost.”
MethodInfluentialFew-Shot Deep Learning for AFM Characterization of Single Molecule Interactions
Waite, J. R., Tan, S. Y., Saha, H., Sarkar, S., & Sarkar, A. (2022) · Patterns · doi:10.1016/j.patter.2022.100672 · Cited by 30
“In contrast, few-shot learning is able to be trained with only a few labeled samples without the need for additional unlabeled samples.(31,33,34) By using a few-shot approach, we only need to determine whether a sample is similar or dissimilar to other samples rather than directly determining the class of a sample.”
BackgroundIntegrating Sustainable Big AI: Quantum Anonymous Semantic Broadcast
Tariq, S., Khalid, U., Arfeto, B. E., Duong, T. Q., & Shin, H. (2024) · IEEE Wireless Communications · doi:10.1109/mwc.007.2300503 · Cited by 29
Building 6G Radio Foundation Models with Transformer Architectures
Aboulfotouh, A., Eshaghbeigi, A., & Abou-Zeid, H. (2025) · International Conference on Communications · doi:10.1109/icc52391.2025.11161954 · Cited by 29
Wen, L., & Chen, X. (2024) · Frontiers in Energy Research · doi:10.3389/fenrg.2024.1355222 · Cited by 28
Sun, Y., Sun, Z., Du, Y., Zhang, J., & Ong, Y.-S. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671958 · Cited by 28
Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream TasksPreprint
Ericsson, L., Gouk, H., & Hospedales, T. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11398 · Cited by 28
“Please see [19, 29] for excellent surveys.”
Self-citation
Metadata
- Authors
- Ericsson, L., Gouk, H., Loy, C. C., & Hospedales, T. M.
- Affiliations
- University of Edinburgh (GB)
- Nanyang Technological University (SG)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2022-05
- Volume
- 39
- Issue
- 3
- Pages
- 42-62
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W3207924272
- Cited by
- 433
- References
- 125
- Citations by year
- 2026: 89 · 2025: 105 · 2024: 132 · 2023: 86 · 2022: 20
- FWCI
- 28.22
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by-nd)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/S000631/1, EP/L016427/1, EP/R026173/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- self-supervised representation learning, pre-training, deep feature learning, image representation, video representation, sound representation, text representation, graph representation, computational cost, open problems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.