Self-Supervised Representation Learning: Introduction, advances, and challenges
Open accessThe DOI matches its Crossref record; 111 of the 125 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Self-supervised representation learning (SSRL) methods aim to provide powerful, deep feature learning without the requirement of large annotated data sets, thus alleviating the annotation bottleneck—one of the main barriers to the practical deployment of deep learning today. These techniques have advanced rapidly in recent years, with their efficacy approaching and sometimes surpassing fully supervised pretraining alternatives across a variety of data modalities, including image, video, sound, text, and graphs. This article introduces this vibrant area, including key concepts, the four main families of approaches and associated state-of-the-art techniques, and how self-supervised methods are applied to diverse modalities of data. We further discuss practical considerations including workflows, representation transferability, and computational cost. Finally, we survey major open challenges in the field, that provide fertile ground for future work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The various self-supervised pretexts are divided into five broad families: transformation prediction, masked prediction, instance discrimination, clustering, and contrastive instance discrimination.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [28]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Contrastive learning [56-58] is in fact a training procedure used in recent artificial intelligence models as a way to efficiently learn semantic representations.”
From How Adversarial REM Dreams May Facilitate Creativity, and Why We Become Aware of Them (2024) · cites this paper as [58]
The cited source says what the claim attributes to it.
In the citing paper
“For example, Ericsson et al. [28] provides a very comprehensive review of pretext tasks for multiple modalities, including image, video, text, audio, time series, and graph.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [28]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“State of the art architectures usually start with CNN representation encoding hθ(·), with ResNet [1] being widely used, before appending task-specific”
InfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“long been used to model the context while recent works have moved to Transformer-based architectures with self-attention [2], which allow longer range connections to be made across words in a sentence, but require more data for training.”
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 9×
“Self-supervised strategies have been leveraged successfully to improve sample efficiency of learning across a variety of modalities, from image [5]–[7], video [8], [9], speech [10], [11], text [12], [13], and graphs [14], [15].”
InfluentialImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“As in much of computer vision, ImageNet [35] is the most typical source data set for self-supervised pretraining [5]–[7], [40], consisting of 1.28 million training images across 1,000 object categories, with the most commonly used resolution at 224 × 224.”
InfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 9×
“Self-supervised strategies have been leveraged successfully to improve sample efficiency of learning across a variety of modalities, from image [5]–[7], video [8], [9], speech [10], [11], text [12], [13], and graphs [14], [15].”
InfluentialAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 1×
“One notable version is a vision transformer [58], which is increasingly being used by recent self-supervised methods on image data [43].”
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · cited 2×
“These range from classic shallow approaches such as Gaussian mixtures [21] to deep methods such as variational autoencoders (VAEs) and generative adversarial networks (GANs) [18].”
Semi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853 · cited 1×
“Graph NNs (GNNs) [89] have advanced all these tasks significantly, especially where massive-labeled data are available.”
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 6×
“With data annotation being a major bottleneck, NLP was the first discipline to make major and successful use of self-supervision [22].”
Influential
Research works printed but not linked: 4
- 0, Youtube-8m A large and diverse labeled video dataset for video understanding research
- gokaslan, 2019, OpenWebText Corpus
- radford, 2019, Language Models are Unsupervised Multitask Learners
- goodfellow, 2016, Deep Learning
Cited by
Items 201–300 of 641
Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages
Zou, H., Lv, F., Zheng, D., Chng, E. S., & Rajan, D. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209040 · Cited by 10
“The evolution from LLMs to Multimodal Large Language Models (MM-LLMs) has revolutionized speech and audio processing.”
Pang, K., Liu, Y., Zhou, S., Liao, Y., Yin, Z., Zhao, L., & Chen, H. (2024) · Foods · doi:10.3390/foods13223598 · Cited by 10
TIMAR: Transition-informed representation for sample-efficient multi-agent reinforcement learning
Feng, M., Yang, Y., Zhou, W., & Li, H. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107081 · Cited by 10
Self-Supervised Framework Based on Subject-Wise Clustering for Human Subject Time Series Data
Seong, E., Lee, H., & Chae, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30240 · Cited by 10
Liu, Z., Zhang, F., Cheng, L., Deng, H., Yang, X., Zhang, Z., & Zhou, C.-C. (2026) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2026.112387 · Cited by 10
Exploiting Variational Inequalities for Generalized Change Detection on Graphs
Florez-Ospina, J. F., Jimenez-Sierra, D. A., Benítez-Restrepo, H. D., & Arce, G. R. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3322377 · Cited by 10
“Not only do these improve our understanding of change detection based on graph subspace prior models, but they also open the door to integrating elements of more recent learning paradigms such as self-supervised learning [45].”
BackgroundLearning beyond sensations: How dreams organize neuronal representations
Deperrois, N., Petrovici, M. A., Senn, W., & Jordan, J. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105508 · Cited by 9
Self-supervised learning for early detection of neurodegenerative diseases with small data
Jiang, H. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/166402 · Cited by 9
Mawgoud, R. A., & Posch, C. (2024) · JEADV Clinical Practice · doi:10.1002/jvc2.580 · Cited by 9
Fine-grained adaptive contrastive learning for unsupervised feature extraction
Yin, T., Wang, J., Zhao, Y., Wang, S., Ma, Y., & Liu, M. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129014 · Cited by 9
Sylvain, J., Drolet, G. G., Thiffault, É., & Anctil, F. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.103960 · Cited by 9
Fang, G., Dai, Y., Lin, Z., Zhou, C.-C., Song, J., Gu, Y., Guo, X., Mao, A., & Kong, X. (2025) · Astronomy and Astrophysics · doi:10.1051/0004-6361/202451734 · Cited by 9
Learning to Regrasp Using Visual–Tactile Representation-Based Reinforcement Learning
Zhang, Z., Sun, H. X., Zhou, Z., Wang, Y., Huang, H., Zhang, Z., & Cao, Q. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3470030 · Cited by 9
“The integration of self-supervised [26] objectives for joint representation learning with RL has been empirically demonstrated to enhance both the sample efficiency [27] and generalization [28] capabilities of RL in control tasks.”
SEML: Self-Supervised Information-Enhanced Meta-learning for Few-Shot Text Classification
Li, H., Huang, G., Li, Y., Zhang, X., Wang, Y., & Li, J. (2023) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-023-00287-6 · Cited by 9
Leveraging self-supervised learning for scene classification in child sexual abuse imagery
Valois, P. H. V., Macedo, J., Ribeiro, L. S. F., dos Santos, J. A., & Avila, S. (2025) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2025.301918 · Cited by 9
Demographic Attributes Prediction from Speech Using WavLM Embeddings
Yang, Y., Thebaud, T., & Dehak, N. (2025) · Conference on Information Sciences and Systems · doi:10.1109/ciss64860.2025.10944678 · Cited by 9
“Comparative experiments have shown that self-supervised embeddings frequently outperform manual feature sets for demographic profiling purposes [22].”
Gao, H., Huo, X., Zhu, C., He, C., & Meng, J. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.112509 · Cited by 9
On the Universality of Self-Supervised LearningPreprint
Qiang, W., Wang, J., Zheng, C., Xiong, H., & Hua, G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.01053 · Cited by 9
Contrastive Meta-Learning for Partially Observable Few-Shot LearningPreprint
Jelley, A., Storkey, A., Antoniou, A., & Devlin, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.13136 · Cited by 9
“Contrastive learning extracts features that are present in multiple views of a data item, by encouraging representations of related views to be close in an embedding space (Ericsson et al., 2021).”
BackgroundMethodQian, T., Li, J., Chen, Y., Cong, G., Sun, T., Wang, F., & Xu, Y. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3733747 · Cited by 9
Radio Signals Recognition With Unsupervised Deep Learning: A Survey
Milosheski, L., Bertalanič, B., Fortuna, C., & Mohorčič, M. (2025) · IEEE Access · doi:10.1109/access.2025.3647686 · Cited by 9
“For example, in [44], authors make a distinction based on the way the pseudo-labels are generated, while in [26], authors consider the final training phase only.”
Different Ventricular Fibrillation Types in Low-Dimensional Latent Spaces
Bernal, C. P., Melgarejo-Meseguer, F.-M., Carrera, E. V., Sánchez-Muñoz, J.-J., García-Alberola, A., & Rojo‐Álvarez, J. L. (2023) · Sensors · doi:10.3390/s23052527 · Cited by 9
Haresamudram, H., Tang, C. I., Suh, S., Lukowicz, P., & Plötz, T. (2024) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3675094.3677562 · Cited by 8
Gao, R., Peng, A., Duan, Y., Chen, M., Zheng, T., Zhang, M., Chen, L., & Sun, H. (2025) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.29734 · Cited by 8
“27 However, due to the unique content of medical images compared to natural images, designing effective pretext tasks for medical image analysis remains an open question.”
Interpreting human sleep activity through neural contrastive learning
Chen, Z., Zheng, H., Zhou, J., Zheng, L., Lin, P., Wang, H., Busche, M. A., Behrens, T. E. J., Dolan, R. J., & Liu, Y. (2026) · Neuron · doi:10.1016/j.neuron.2026.03.028 · Cited by 8
Gondhalekar, Y., Chies-Santos, A. L., de Souza, R. S., Queiroz, C., Lopes, A. R., Ferrari, F., Azevedo, G. M., Monteiro-Pereira, H., Overzier, R., Castelli, A. V. S., et al. (2024) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stae1410 · Cited by 8
Chen, M., Wang, H., & Zhang, F. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/ada8c7 · Cited by 8
Bhattachrjee, S., Li, H., Xia, J., & Xu, W. (2023) · Smart Health · doi:10.1016/j.smhl.2023.100396 · Cited by 8
Tan, X., Cheng, L., Qiu, X., Shi, S., Cheng, Y., Chu, W., Xu, Y., & Qi, Y. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671638 · Cited by 8
FedLID: Self-Supervised Federated Learning for Leveraging Limited Image Data
Psaltis, A., Kastellos, A., Patrikakis, C. Z., & Daras, P. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00111 · Cited by 8
Synthetic Echocardiograms Generation Using Diffusion ModelsPreprint
Pellicer, A. O., Yadav, A. K. S., Bhagtani, K., Xiang, Z., Pizlo, Z., Gradus‐Pizlo, I., & Delp, E. J. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.11.11.566718 · Cited by 8
“Self-supervised deep learning methods learn feature representations from unlabeled data by using pretext tasks such as reconstruction of input data from the learned feature representations [11].”
MethodSerialized EHR make for good text representationsPreprint
Chou, Z., Qin, Q., & Shi, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.13843 · Cited by 8
“These models excel in representing complex information and creating latent representations via extensive pre-training, typically using self-supervised methodologies [Ericsson et al., 2022].”
Deep Scene Flow Learning: From 2D Images to 3D Point Clouds
Xiang, X., Abdein, R., Li, W., & Saddik, A. E. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3319448 · Cited by 8
“Self-Supervised Learning: In self-supervised learning, unlabeled datasets are used for training, while a pretext task is needed to learn some visual representation from the dataset and use it as a supervision signal [87].”
MethodLiu, W., Pan, S., Chang, S., Huang, Q., & Jiang, N. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.112871 · Cited by 8
Liu, W., Pan, S., Chang, S., Huang, Q., & Jiang, N. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3487977 · Cited by 8
“Unlike LP experiments, FT experiments often use only a fraction of the training data in downstream tasks, evaluating model performances when training data are scarce [1].”
Isolated Channel Vision Transformers: From Single-Channel Pretraining to Multi-Channel FinetuningPreprint
Lian, W., Micke, P., Lindblad, J., & Sladoje, N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.09826 · Cited by 8
“Application of a SSL model typically consists of two stages: (1) pretraining on large-scale unlabeled data, to learn general representations; (2) fine-tuning on downstream datasets to effectively solve real and specific tasks [17, 46].”
Rehman, A., Heldal, I., & Lin, J. C. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825780 · Cited by 7
“Self Supervised Representation Learning (SSRepL) [1] captures useful representations of the input data that can potentially improve the model’s performance on down-stream tasks.”
Achieving high performance on sketch-based image retrieval without real sketches for training
Saavedra, J. M., Stears, C., & Campos, W. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.04.018 · Cited by 7
Furuya, T., Chen, Z., Ohbuchi, R., & Kuang, Z. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104025 · Cited by 7
SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Video
Jegorova, M., Petridis, S., & Pantić, M. (2023) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg57933.2023.10042638 · Cited by 7
“There is plenty of research showing the benefits of SSL for general image datasets [26], [27].”
BackgroundCCFC: Bridging Federated Clustering and Contrastive Learning
Liu, J., Yan, J., & Zhang, Z. (2026) · Symmetry · doi:10.3390/sym18081383 · Cited by 7
Sood, E., Dhar, P., Troiano, E., Southwell, R. V., & D’Mello, S. K. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-short.89 · Cited by 7
Zhang, H., Liu, W., Li, Z., Shi, J., Chang, S., Wang, H., He, J. L., & Huang, Q. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3336915 · Cited by 7
Self-Supervised Contrastive Learning for Disease Trajectory Prediction
Das, A. C., Shak, M. S., Rahman, N., Mahmud, F., Eva, A. A., & Hasan, M. N. (2025) · International Conference on Pervasive Computing and Social Networking (ICPCSN) · doi:10.1109/icpcsn65854.2025.11035472 · Cited by 7
“To address this challenge, self-supervised learning has emerged as a promising paradigm that leverages unlabeled data to learn meaningful representations, reducing the dependency on labeled examples [4], [5].”
Scalable Representation Learning in Linear Contextual Bandits with Constant Regret Guarantees
Tirinzoni, A., Papini, M., Touati, A., Lazaric, A., & Pirotta, M. (2022) · neural information processing systems · doi:10.52202/068431-0168 · Cited by 7
Srijiranon, K., Varisthanist, N., & Tanantong, T. (2026) · Technologies · doi:10.3390/technologies14020116 · Cited by 6
“Self-supervised learning (SSL) [15] has emerged as a potential alternative by enabling representation learning from unlabeled data.”
Yousef, I., Shah, S. L., & Gopaluni, R. B. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110914 · Cited by 6
Mixup Feature: A Pretext Task Self-Supervised Learning Method for Enhanced Visual Feature Learning
Xu, J., & Stirenko, S. (2023) · IEEE Access · doi:10.1109/access.2023.3301561 · Cited by 6
“The recent development of self-supervised learning (SSL) has accelerated the progress of artificial intelligence towards the era of large-scale models [44], [45], [46].”
BackgroundSuhas, M., Abisset‐Chavanne, E., & Rey, P.-A. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127513 · Cited by 6
An Efficient Lightning Classifier Using a Self‐Supervised Learning Neural Network
Lu, J., Li, J., Liu, Y., Yuan, S., Pu, Y., Bian, Q., Gong, Z., Sun, Z., Qie, X., & Jiang, R. (2025) · Geophysical Research Letters · doi:10.1029/2025gl115067 · Cited by 6
Sohail, M., Tanveer, M., & Kim, H. S. (2026) · Mathematics · doi:10.3390/math14111879 · Cited by 6
Lee, H., Kang, Y., Noh, S. M., Kim, J., & Lee, J. (2024) · Spatial Information Research · doi:10.1007/s41324-024-00597-7 · Cited by 6
“Self-supervised representation learning (SSRL) has emerged as a promising solution to this problem, garnering attention in natural language processing and computer vision [19, 29].”
BSoNet: Deep Learning Solution for Optimizing Image Quality of Portable Backscatter Imaging Systems
Lil, L., Wei, W., Lu, Y., Zhang, W., Zhang, Y., & Zhao, W. (2025) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2025.3567817 · Cited by 6
“Onthisbasis, self-supervisedlearning,astrategythatgeneratessupervision signalsfromtheinputdataitselfwithoutrelyingonhigh-quality labeleddata,hasgainedincreasingattention[5].”
MT-SLVR: Multi-Task Self-Supervised Learning for Transformation In(Variant) Representations
Heggan, C., Hospedales, T., Budgett, S., & Yaghoobi, M. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-1064 · Cited by 6
Self-citationThe growing impact of machine learning on drug formulation science
Bao, Z., Huynh, S., Dévédec, F. L., Nguyen, J., & Allen, C. J. (2025) · Advanced Drug Delivery Reviews · doi:10.1016/j.addr.2025.115761 · Cited by 6
Chen, D., Yao, L., Liu, M., Shen, Z., Hu, Y., Song, Z., Wang, Q., & Zhang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72120-5_57 · Cited by 6
“Recently, self-supervised learning (SSL) has emerged as a promising learning paradigm that can extract information from the underlying structure and features of limited data without specific task constraints [12,14].”
Yeboah, P. N., Kayes, A. S. M., Rahayu, W. J., Pardede, E., & Mahbub, S. (2026) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2026.104474 · Cited by 6
Kang, Y., Liu, A., Zhang, S., Wang, T., Dong, M., & Song, H. (2024) · Internet of Things · doi:10.1016/j.iot.2024.101198 · Cited by 6
Meta-Representational Predictive Coding: Neuroscience-Informed Self-Supervised LearningPreprint
Ororbia, A. G., Friston, K. J., & Rao, R. P. N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.21796 · Cited by 6
“Self-supervised learning has become an increasingly important paradigm in the domain of machine intelligence [21, 35].”
Towards the Sparseness of Projection Head in Self-Supervised LearningPreprint
Song, Z., Su, X., Wang, J., Qiang, W., Zheng, C., & Sun, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.08913 · Cited by 6
“The objective of self-supervised learning (SSL) lies in the extraction of valuable representations from unlabeled data[15, 5, 43].”
BackgroundOptimising Face Recognition System using Contrastive Learning and Contrastive Loss
Kumar, A. J., Manvitha, A., Poojitha, B., Mahitha, M., Ashesh, K., & Kiran, K. V. D. (2023) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc56507.2023.10084086 · Cited by 6
A survey on design choices for self-supervised learning in computer vision
Wittscher, L. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11506-9 · Cited by 5
Corponi, F., Li, B. M., Anmella, G., Valenzuela‐Pascual, C., Mas, A., Pacchiarotti, I., Valentí, M., Grande, I., Benabarre, A., Garriga, M., et al. (2024) · JMIR mhealth and uhealth · doi:10.2196/55094 · Cited by 6
“Specifically, we compared 2 main pretext task designs (ie, masked prediction [MP] and transformation prediction [TP]) [44], and for the best-performing routine, we studied its sensitivity to the unlabeled data availability in ablation analyses.”
Efficient Audio Deepfake Detection using WavLM with Early Exiting
Pimentel, A., Zhu, Y. M., Guimarães, H. R., & Falk, T. H. (2024) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs61860.2024.10810699 · Cited by 5
Skip-Step Contrastive Predictive Coding for Time Series Anomaly Detection
Zhang, K., Wen, Q., Zhang, C., Sun, L., & Liu, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447104 · Cited by 5
“Recently, self-supervised learning (SSL), which allows for learning representations without ground-truth labels, has exerted great power in the fields of computer vision, natural language processing, and signal processing [ 6].”
BackgroundDeep learning application for genomic data analysis
Jeong, C. B., Cho, H., & Park, D. (2025) · BMB Reports · doi:10.5483/bmbrep.2025-0020 · Cited by 6
Xie, Y., Li, J., Zhang, Z., Meng, B., Dai, S., Zhou, Y., Kennedy, E., Jiao, N., Chen, H. I., & Dong, Z. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf657 · Cited by 5
“Self-supervised learning does not require annotated data (Ericsson et al. 2022, Rafiei et al. 2024), and allows models to merely utilize the structure or intrinsic relationships of the data to discover potential feature representations in the data.”
Global-local coherency contrastive learning for context-aware time series forecasting
Ding, F., Chen, X., Liu, H., Lyu, C., Yang, G., Xiong, H., & Zhou, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114745 · Cited by 5
Liu, W., Pan, S., Li, Z., Chang, S., Huang, Q., & Jiang, N. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108452 · Cited by 5
RetFiner: A Vision-Language Refinement Scheme for Retinal Foundation Models
Fecso, R., Morano, J., Schmidt-Erfurth, U. M., & Bogunović, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04971-1_51 · Cited by 5
Teaching Masked Autoencoder With Strong Augmentations
Zhu, R., Bai, Y., Yao, T., Liu, J., Sun, Z., Mei, T., & Chen, C. W. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3419898 · Cited by 5
“To this end, most pretasks are proposed to create the supervised signal for the anchor image [20], [21] and then define the corresponding learning objectives, e.g., contrastive SSL [1], [6], [22] and generative SSL [3], [4], [23].”
Ashihara, T., Delcroix, M., Ijima, Y., & Kashino, M. (2024) · IEEE Access · doi:10.1109/access.2024.3428364 · Cited by 6
“Self-supervised learning (SSL) is a third class of learning scheme, which has rapidly grown in recent years [10], [11], [12] and driven many breakthroughs, especially in natural language processing (NLP) [13], [14], [15], [16], computer vision (CV) [17], [18], speech/audio [19], [20], [21], [22],…”
BackgroundRepresentations in Developing Statistical Thinking in Secondary Education
Selvakumar, P., Sharma, A., Roy, S. S., Mohit, M., & Manjunath, T. C. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-9934-7.ch005 · Cited by 5
Chen, K., Wichern, G., Germain, F. G., & Roux, J. L. (2023) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw59220.2023.10193575 · Cited by 5
“Self-supervised learning (SSL) is a rapidly growing deep learning approach that leverages unlabeled data to improve model training, resulting in better generalization capabilities and task performance [4].”
BackgroundSelf-Supervised Representation Learning for Geographical Data—A Systematic Literature Review
Corcoran, P., & Spasić, . (2023) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi12020064 · Cited by 5
“A broader introduction to the topic of SSRL can already be found in [9].”
BackgroundFastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed
Zhang, J., Wang, J., Sun, Z., Zou, J., & Balestriero, R. (2025) · neural information processing systems · doi:10.52202/085713-1147 · Cited by 5
Boosting Patient Representation Learning via Graph Contrastive Learning
Zhang, Z., Liu, Y., Bian, J., Yepes, A. J., Shen, J., Li, F., Long, G., & Salim, F. D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70378-2_21 · Cited by 5
Tied-Augment: Controlling Representation Similarity Improves Data AugmentationPreprint
Kurtuluş, E., Li, Z., Dauphin, Y., & Cubuk, E. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13520 · Cited by 5
“Self-supervised learning is a form of representation learning that usually makes use of pretext tasks to learn general representations (Ericsson et al., 2022).”
BackgroundAutomatic Speech Recognition for Documenting Endangered First Nations Languages
Sadeque, Z. A. (2023) · HARVEST (University of Saskatchewan) · doi:10.13140/rg.2.2.19379.20005 · Cited by 5
Chen, K., Wichern, G., Germain, F. G., & Roux, J. L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02160 · Cited by 5
“Self-supervised learning (SSL) is a rapidly growing deep learning approach that leverages unlabeled data to improve model training, resulting in better generalization capabilities and task performance [4].”
BackgroundApplications of Self-Supervised Learning to Biomedical Signals: where are we nowPreprint
Pup, F. D., & Atzori, M. (2023) · doi:10.36227/techrxiv.22567021 · Cited by 4
Applications of Self-Supervised Learning to Biomedical Signals: where are we nowPreprint
Pup, F. D., & Atzori, M. (2023) · doi:10.36227/techrxiv.22567021.v1 · Cited by 4
Masked self‐supervised pre‐training model for EEG‐based emotion recognition
Hu, X., Chen, Y., Yan, J., Wu, Y. L., Ding, L., Xu, J., & Cheng, J. T. (2024) · Computational Intelligence · doi:10.1111/coin.12659 · Cited by 4
Huang, Y., Chen, M., Chen, J., & Pan, Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129062 · Cited by 4
Audiovisual speech representation learning applied to emotion recognition
Sadok, S. (2024) · theses.fr (ABES) · doi:10.70675/0f15a36fzecf8z4cffz94e7z120e7d8fd39d · Cited by 4
Towards Unsupervised Domain-Specific Open-World Recognition
Alfarisy, G. A. F., Malik, O. A., & Ong, W.-H. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129141 · Cited by 4
Self-Supervised Representation Learning for Basecalling Nanopore Sequencing Data
Vintimilla, C., & Hwang, S. (2024) · IEEE Access · doi:10.1109/access.2024.3440882 · Cited by 4
“Self-supervised learning (SSL) aims to learn meaningful representations from unlabeled data by solving a pretext task and exploiting the underlying structures of data [12].”
BackgroundUnifying Variables in Neural Scaling for General Audio Representations via Embedding Effective Rank
Deng, X., Sun, Y., Dou, Y., & Xu, K. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3685874 · Cited by 4
“Recent advances have shown the potential of self-supervised learning to extract rich audio embeddings [17], [18], [19], [20], but there is still a lack of systematic empirical and theoretical investigation into how various factors jointly influence representation quality in the audio domain.”
Xu, H. J., Liang, Y., Xie, R., & Kong, Y. (2026) · Sensors · doi:10.3390/s26113609 · Cited by 4
“Research by Ericsson et al. [41] demonstrated that self-supervised learning, through the design of proxy tasks, allows deep networks to learn robust and generalized feature representations from vast amounts of unlabeled channel data, significantly reducing the dependence on costly labeled data.”
Fu, L.-W., Liu, C.-H., Jain, M., Chen, C. J., Wu, Y., Huang, S.-L. L., & Chen, H. H. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3327257 · Cited by 4
Prediction Consistency Regularization for Learning with Noise Labels Based on Contrastive Clustering
Sun, X., Zhang, S., & Ma, S. (2024) · Entropy · doi:10.3390/e26040308 · Cited by 4
“Contrastive learning relies on pretext tasks for supervision [38], broadly categorized into contrastive-instance and clustering-based categories.”
BackgroundChatterjee, M. J. V., Chan, A. D. C., & Komeili, M. (2026) · Scientific Reports · doi:10.1038/s41598-026-41549-5 · Cited by 4
“There have been previous attempts at SSRL on ECG data using Transformation Prediction 57,61 , Instance Discrimination 27,59 and Clustering 51 .”
CL-HOI: Cross-level human–object interaction distillation from multimodal large language models
Gao, J., Cai, C., Wang, R., Liu, W., Yap, K., Garg, K., & Han, B. S. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113561 · Cited by 4
Using Vocoder Artifacts For Audio Deepfakes Detection
Sun, C., AlBadawy, E. A., Davison, T. F., Robinson, S. R., Chang, M., & Lyu, S. (2023) · Advances in information security · doi:10.1007/978-3-031-49803-9_11 · Cited by 4
Self-Supervised Domain Adaptation for 6DoF Pose Estimation
Jin, J., Jeong, E., Cho, J., & Kim, Y.-G. (2024) · IEEE Access · doi:10.1109/access.2024.3430227 · Cited by 4
“Recent self-supervised learning with unlabeled data has emerged [29].”
BackgroundRen, L., Ding, Y., Zhang, Y., Teng, K.-H., Wang, L., Zhang, W.-Y., Zhu, Y., Xu, J., Wei, X., Wang, B., et al. (2025) · npj Precision Oncology · doi:10.1038/s41698-025-01186-z · Cited by 4
“Finally, the fl exibility of supervised learning supports integration with advanced architectures, including convolutional neural networks (CNNs) and graph neural networks (GNNs), which excel at capturing spatial and contextual dependencies within WSIs 16 .”
HamNava: A Dataset for Multi‑Label Instrument Classification
Mohseni, P., BabaAli, B., & Asadi, H. (2025) · Transactions of the International Society for Music Information Retrieval · doi:10.5334/tismir.257 · Cited by 4
Multi-Scale Contrastive Learning Networks for Graph Anomaly Detection
Zhe'ao, J. (2024) · Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS) · doi:10.1109/acctcs61748.2024.00114 · Cited by 4
“Contrastive self-supervised learning, a prominent subset within the realm of self-supervised learning [28], involves the utilization of handcrafted contrastive pretext tasks to train models by distinguishing pairs of instances representing positive examples from negative instance pairs [14].”
BackgroundKar, I., Mukhopadhyay, S., & Ralte, Z. (2024) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs60874.2024.10717196 · Cited by 4
“In particular [39] showed that these scores can be aggregated across multi attention heads, layers for a given image to obtain a single score for each solar PV images.”
Background
Metadata
- Authors
- Ericsson, L., Gouk, H., Loy, C. C., & Hospedales, T. M.
- Affiliations
- University of Edinburgh (GB)
- Nanyang Technological University (SG)
- Venue
- IEEE Signal Processing Magazine
- Published
- 2022-05
- Volume
- 39
- Issue
- 3
- Pages
- 42-62
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W3207924272
- Cited by
- 433
- References
- 125
- Citations by year
- 2026: 89 · 2025: 105 · 2024: 132 · 2023: 86 · 2022: 20
- FWCI
- 28.22
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Signal Processing Magazine (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by-nd)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/S000631/1, EP/L016427/1, EP/R026173/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- self-supervised representation learning, pre-training, deep feature learning, image representation, video representation, sound representation, text representation, graph representation, computational cost, open problems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.