Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
Open accessThe DOI matches its Crossref record; 204 of the 206 references it cites were found in a registry or an index.
Abstract
Large-scale labeled data are generally required to train deep neural networks in order to obtain better performance in visual feature learning from images or videos for computer vision applications. To avoid extensive cost of collecting and annotating large-scale datasets, as a subset of unsupervised learning methods, self-supervised learning methods are proposed to learn general image and video features from large-scale unlabeled data without using any human-annotated labels. This paper provides an extensive review of deep learning-based self-supervised general visual feature learning methods from images or videos. First, the motivation, general pipeline, and terminologies of this field are described. Then the common deep neural network architectures that used for self-supervised learning are summarized. Next, the schema and evaluation metrics of self-supervised learning methods are reviewed followed by the commonly used datasets for images, videos, audios, and 3D data, as well as the existing self-supervised visual feature learning methods. Finally, quantitative performance comparisons of the reviewed methods on benchmark datasets are summarized and discussed for both image and video feature learning. At last, this paper is concluded and lists a set of promising future directions for self-supervised visual feature learning.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Jing and Tian [18] summarize the self-supervised feature learning methods on image and video data, and four categories are discussed: generation-based, context-based, free semantic label-based, and cross modal-based, where cross-modal-based methods construct learning task using RGB frame sequence an optical flow sequence, which are unique features in the video.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [18]
In the citing paper
“Another hot research sub-area is self-supervised learning, which learns discriminative visual representation by solving predefined pretext tasks [194].”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [194]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 10×
“Figure is reproduced based on VGG [9].”
BackgroundMethodInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 8×
“Trained with accurate human-annotated labels, the supervised learning methods obtained break-through results on different computer vision applications [1], [4], [8], [16].”
MethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Compared to standard RNN [72], long short term memory (LSTM) uses memory cells to store, modify, and access internal states, to better model the long-term temporal relationships within video frames [73].”
BackgroundFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 2×
“DUE to the powerful ability to learn different levels of general visual features, deep neural networks have been used as the basic structure to many computer vision applications such as object detection [1], [2], [3], semantic segmentation [4], [5], [6], image captioning [7], etc.”
BackgroundMethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · cited 2×
“DUE to the powerful ability to learn different levels of general visual features, deep neural networks have been used as the basic structure to many computer vision applications such as object detection [1], [2], [3], semantic segmentation [4], [5], [6], image captioning [7], etc.”
BackgroundMethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 5×
“• ImageNet: The ImageNet dataset [13] contains 1.”
BackgroundInfluentialGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“• MNIST: The MNIST is a dataset of handwritten digits consisting of 70, 000 images while 60, 000 images belong to training set and the rest 10, 000 images are for testing [110].”
BackgroundDistinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · cited 1×
“Recently, many datasets such as MSCOCO [99] and OpenImage [14] have been proposed for object detection and many ConvNet-based models [1], [2], [3], [100], [101], [102], [103], [104] have been proposed and obtained great performance.”
Background
Cited by
Image segmentation using deep learning: A survey
Minaee, S., Boykov, Y. Y., Porikli, F., Plaza, A. J., Kehtarnavaz, N., & Terzopoulos, D. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3059968 · Cited by 3,796
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09882 · Cited by 2,499
“[27] for an exhaustive and detailed review of this literature.”
Self-supervised Learning: Generative or Contrastive
Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3090866 · Cited by 1,087
Model-Contrastive Federated Learning
Li, Q., He, B., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01057 · Cited by 1,534
“Self-supervised learning [18, 9, 3, 4, 12, 35] is a recent hot research direction, which tries to learn good data representations from unlabeled data.”
BackgroundDeep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406
Deep Neural Networks and Tabular Data: A Survey
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3229161 · Cited by 1,025
“These methods have also shown astonishing results in computer vision and natural language processing [215], [216].”
ResultDomain Generalization: A Survey
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195549 · Cited by 1,315
“Self-supervised learning is often referred to as learning with free labels generated from data itself (see [221] for a com-”
BackgroundRecent advances and clinical applications of deep learning in medical image analysis
Chen, X., Wang, X., Zhang, K., Fung, K.-M., Thai, T. C., Moore, K., Mannel, R. S., Liu, H., Zheng, B., & Qiu, Y. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102444 · Cited by 1,080
“In practice, self-supervision can be created either through pretext tasks (Misra and Maaten, 2020) or contrastive learning (Jing and Tian, 2020) as follows.”
BackgroundDeep learning classifiers for hyperspectral imaging: A review
Paoletti, M. E., Haut, J. M., Plaza, J., & Plaza, A. (2019) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2019.09.006 · Cited by 1,073
“This does not mean learning the inherent structure of data in the form of unsupervised learning (Liu et al., 2019; Jing and Tian, 2019).”
BackgroundA Survey on Deep Semi-Supervised Learning
Yang, X., Song, Z., King, I., & Xu, Z. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3220219 · Cited by 923
“Self-supervised learning [46], [47], [48] has gained popularity due to its ability to prevent the expense of annotating large-scale datasets.”
BackgroundGeneralizing to unseen domains: A survey on domain generalization
Wang, J., Lan, C., Liu, C., Ouyang, Y., Qin, T., Lu, W., Chen, Y., Zeng, W., & Yu, P. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3178128 · Cited by 1,046
“learning (SSL) is a recently popular learning paradigm that builds self-supervised tasks from the large-scale unlabeled data [213].”
BackgroundA foundation model for generalizable disease detection from retinal images
Zhou, Y., Chia, M. A., Wagner, S. K., Ayhan, M. S., Williamson, D. J., Struyven, R. R., Liu, T., Xu, M., Lozano, M. G., Woodward-Court, P., et al. (2023) · Nature · doi:10.1038/s41586-023-06555-x · Cited by 1,014
“Self-supervised learning (SSL) aims to alleviate data inefficiency by deriving supervisory signals directly from data, instead of resorting to expert knowledge by means of labels 8–11 .”
BackgroundData-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S. M. A., & Oord, A. V. D. (2019) · arXiv · doi:10.48550/arxiv.1905.09272 · Cited by 1,014
“C V ] 22 M a y 2019 sentations that are capable of predicting pixel-level details [34, 54, 12], and self-supervised methods that formulate alternative prediction tasks in which missing information is imputed from other parts of the image [33].”
BackgroundHands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users
Jospin, L. V., Laga, H., Boussaid, F., Buntine, W., & Bennamoun, M. (2022) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2022.3155327 · Cited by 879
“Self-supervised learning is a learning strategy where the data themselves provide the labels [71].”
BackgroundSalient Object Detection in the Deep Learning Era: An In-Depth Survey
Wang, W., Lai, Q., Fu, H., Shen, J., Ling, H., & Yang, R. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3051099 · Cited by 811
Bond-Taylor, S., Leach, A., Long, Y., & Willcocks, C. G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3116668 · Cited by 703
“Interrelated with generative models is the field of selfsupervised learning where the focus is on learning good intermediate representations that can be used for downstream tasks without supervision [105].”
BackgroundArtificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774
“Another hot research subarea is self-supervised learning, which learns discriminative visual representation by solving predefined pretext tasks [194].”
BackgroundDeep learning in optical metrology: a review
Zuo, C., Qian, J., Feng, S., Yin, W., Li, Y., Fan, P., Han, J., Qian, K., & Chen, Q. (2022) · Light: Science & Applications · doi:10.1038/s41377-022-00714-x · Cited by 711
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification
Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A. D., & van de Weijer, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3213473 · Cited by 740
“learning [118] is only little explored within the context of IL, and is expected to gain interest.”
BackgroundA Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends
Gui, J., Chen, T., Zhang, J., Cao, Q., Sun, Z., Luo, H., & Tao, D. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3415112 · Cited by 552
“Jing et al. [34] expanded the definition of SSL to encompass methods that operate without human-annotated labels.”
BackgroundA Survey on Deep Learning for Data-Driven Soft Sensors
Sun, Q., & Ge, Z. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3053128 · Cited by 665
“Self-supervised learning (SSL) is another feasible solution, which is a kind of unsupervised strategy [144].”
BackgroundGraph Self-Supervised Learning: A Survey
Liu, Y., Jin, M., Pan, S., Zhou, C., Zheng, Y., Xia, F., & Yu, P. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3172903 · Cited by 510
“In contrastive tasks, the dependency between nodes is non-negligible in graph SSL, while the samples in CV/NLP are independent.”
ResultInfluentialXia, F., Sun, K., Yu, S., Aziz, A., Wan, L., Pan, S., & Liu, H. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3076021 · Cited by 473
“which can be classified as a self-supervised model [151].”
BackgroundFine-Grained Image Analysis With Deep Learning: A Survey
Wei, X.-S., Song, Y.-Z., Aodha, O. M., Wu, J., Peng, Y., Tang, J., Yang, J., & Belongie, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3126648 · Cited by 434
“As a potential solution, constructing an unlabeled large-scale fine-grained database and employing unsupervised feature learning techniques (e.g., self-supervised learning [193]) could benefit discriminative features learning and further promote unsupervised downstream tasks [3].”
MethodArtificial intelligence and machine learning for medical imaging: A technology review
Barragán-Montero, A., Javaid, U., Valdés, G., Nguyen, D., Desbordes, P., Macq, B., Willems, S., Vandewinckele, L., Holmström, M., Löfman, F., et al. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.04.016 · Cited by 450
“It is gaining attention for computer vision tasks [70 – 73] and it could play an important role in future research directions for medical imaging applications.”
Zhao, Z., Alzubaidi, L., Zhang, J., Duan, Y., & Gu, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122807 · Cited by 410
An Overview of Deep Semi-Supervised Learning
Ouali, Y., Hudelot, C., & Tami, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05278 · Cited by 230
“Self-supervised learning [161, 74] is a form of unsupervised learning, where the model is trained using a standard supervised loss, but on a pretext task where the supervision comes from the data itself.”
MethodSelf-Supervised Representation Learning: Introduction, advances, and challenges
Ericsson, L., Gouk, H., Loy, C. C., & Hospedales, T. M. (2022) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3134634 · Cited by 433
“Compared to existing surveys [4], we provide a broader introduction to the field; a wider coverage of different modalities rather than focusing on images; highlight more practical considerations such as representation transferability, computation cost, and deployment strategies; and provide a…”
Prototype Augmentation and Self-Supervision for Incremental Learning
Zhu, F., Zhang, X.-Y., Wang, C., Yin, F., & Liu, C.-L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00581 · Cited by 427
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts
Liang, J., He, R., & Tan, T. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-024-02181-w · Cited by 271
“Self-supervised learning [84] is a learning paradigm that focuses on how to learn from unlabeled data by obtaining supervisory signals from the data itself through pretext tasks that leverage its underlying structure.”
Self-supervised learning for recommender systems: A survey
Yu, J., Yin, H., Xia, X., Chen, T., Li, J., & Huang, Z. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3282907 · Cited by 413
“WhileSSLhasbeenextensivelysurveyedinthefieldsof CV,NLP[9],[34]andgraphlearning[35],[36],[37],there hasnotbeenasystematicinvestigationofresearchendeavors onSSRdespitethegrowingnumberofpublications.”
A Decade Survey of Content Based Image Retrieval Using Deep Learning
Dubey, S. R. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3080920 · Cited by 356
“From a decade, a shift has been observed in feature representation from hand-engineering to learning-based after the emergence of deep learning [34], [35].”
BackgroundBiometrics recognition using deep learning: a survey
Minaee, S., Abdolrashidi, A., Su, H., Bennamoun, M., & Zhang, D. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10237-x · Cited by 245
“Self-supervised learning (Jing and Tian 2019) is also another recent popular topic in deep learning, which has not been explored enough for biometrics recognition.”
BackgroundZhou, Z., Sodha, V., Pang, J., Gotway, M. B., & Liang, J. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101840 · Cited by 328
“Specifically, Models Genesis can be utilized to initialize the encoder for the target classification tasks and to initialize the encoder-decoder for the target segmentation tasks, while the existing self-supervised approaches are largely focused on providing encoder models only (Jing and Tian, 2019).”
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372
“[21], [22] and computer vision (CV) [23].”
Self-Supervised Learning in Remote Sensing: A review
Wang, Y., Albrecht, C. M., Braham, N. A. A., Mou, L., & Zhu, X. X. (2022) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2022.3198244 · Cited by 364
“While there exist comprehensive surveys of SSL in computer vision [9]–[12], a didactic and up-to-date introduction to the remote sensing community is missing.”
BackgroundBootstrap Latent Representations for Multi-modal Recommendation
Zhou, X., Zhou, H., Liu, Y., Zeng, Z., Miao, C., Wang, P., You, Y., & Jiang, F. (2023) · Proceedings of the ACM Web Conference 2023 · doi:10.1145/3543507.3583251 · Cited by 298
“SSL-based methods have achieved competitive results in various CV and NLP tasks [12, 20].”
MethodSelFlow: Self-Supervised Learning of Optical Flow
Liu, P., Lyu, M., King, I., & Xu, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00470 · Cited by 323
“It is widely used for learning feature representations from unlabeled data [21].”
BackgroundDeep Gait Recognition: A Survey
Sepas-Moghaddam, A., & Etemad, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3151865 · Cited by 307
“In order to utilize unlabeled gait data to learn more efficient and generalizable gait representations, selfsupervised learning [198] can be exploited.”
MethodModels Genesis: Generic Autodidactic Models for 3D Medical Image Analysis
Zhou, Z., Sodha, V., Rahman Siddiquee, M. M., Feng, R., Tajbakhsh, N., Gotway, M. B., & Liang, J. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32251-9_42 · Cited by 327
“tches randomly sampled from a same brain [8], identifying whether two scans belong to the same person, predicting the level of vertebral bodies [3], and nally the systematic study by Tajbakhsh et al. [10] where individualized self-supervised schemes are studied for a set of target tasks. 3 Experiments and Results Expe…”
BackgroundMethodA survey on data‐efficient algorithms in big data era
Adadi, A. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00419-9 · Cited by 331
“Broadly speaking, self-supervised learning converts an unsupervised learning problem into a supervised one by creating surrogate labels from the unlabeled dataset, potentially greatly reducing the number of labeled examples required [107].”
BackgroundShrestha, Y. R., Krishna, V., & von Krogh, G. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2020.09.068 · Cited by 331
“Other popular approaches include semi-supervised learning (van Engelen & Hoos, 2020), which uses partially labeled datasets and self-supervised learning (where training data are automatically labeled by finding and exploiting relationships [Jing & Tian, 2020]).”
Generalizing to Unseen Domains: A Survey on Domain Generalization
Wang, J., Lan, C., Liu, C., Ouyang, Y., & Qin, T. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/628 · Cited by 308
Qi, G.-J., & Luo, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3031898 · Cited by 315
Federated learning for machinery fault diagnosis with dynamic validation and self-supervision
Zhang, W., Li, X., Ma, H., Luo, Z., & Li, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106679 · Cited by 301
SatMAE: Pre-Training Transformers for Temporal and Multi-Spectral Satellite Imagery
Cong, Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., Burke, M., Lobell, D., & Ermon, S. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-0015 · Cited by 303
Self-supervised learning methods and applications in medical imaging analysis: a survey
Shurrab, S., & Duwairi, R. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1045 · Cited by 254
“1 Jing and Tian (2020) Survey Self-supervised visual feature learning with deep neural networks: A survey IEEE Transactions on Pattern Analysis and Machine Intelligence 2 Ohri and Kumar (2021) Review Review on self-supervised image recognition using deep neural networks Knowledge-Based Systems”
BackgroundSelf-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“JingandTian[18]summarize theself-supervisedfeaturelearningmethodsonimageand videodata,andfourcategoriesarediscussed:generation-based, context-based,freesemanticlabel-based,andcrossmodal-based,wherecross-modal-basedmethodsconstructlearning task using RGB frame sequence an optical flow sequence, which…”
BackgroundResultSelf-supervised learning: A succinct review
Rani, V., Nabi, S. T., Kumar, M., Mittal, A., & Kumar, K. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-09884-2 · Cited by 295
“This task is extremely important in computer vision applications such as robotics, autonomous driving, scene text detection, and so on [27].”
BackgroundA Survey on Deep Learning Technique for Video Segmentation
Zhou, T., Porikli, F., Crandall, D. J., Van Gool, L., & Wang, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3225573 · Cited by 255
“trained with automatically-generated pseudo labels without any human annotations [20].”
BackgroundSun, L., Lian, Z., Liu, B., & Tao, J. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3274829 · Cited by 258
“…handcrafted pretext task, low-level feature reconstruction could fail to encourage the model to learn semantic information, as the model may find a shortcut to solve this task (i.e., only utilize local neighbor information to accomplish this task instead of inferring global semantics)[17],[58],[62].”
BackgroundArtificial intelligence in diabetes management: Advancements, opportunities, and challenges
Guan, Z., Li, H., Liu, R., Cai, C., Liu, Y., Li, J., Wang, X., Huang, S., Wu, L., Liu, D., et al. (2023) · Cell Reports Medicine · doi:10.1016/j.xcrm.2023.101213 · Cited by 412
ASE: large-scale reusable adversarial skill embeddings for physically simulated characters
Peng, X. B., Guo, Y., Halper, L., Levine, S., & Fidler, S. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530110 · Cited by 221
“…where large expressive models trained on massive datasets have been a central component of major advances [Brown et al. 2020; Deng et al. 2009; Doersch et al. 2015; He et al. 2016; Hinton and Salakhutdinov 2006; Jing and Tian 2021; Kingma and Welling 2014; Mikolov et al. 2013; Oord et al. 2018a].”
BackgroundA Close Look at Deep Learning with Small Data
Brigato, L., & Iocchi, L. (2021) · 2020 25th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr48806.2021.9412492 · Cited by 210
“Usually, a large pool of images is used to teach how to solve a pretext task to a CNN [10].”
BackgroundUncovering the structure of clinical EEG signals with self-supervised learning
Banville, H., Chehab, O., Hyvärinen, A., Engemann, D.-A., & Gramfort, A. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/abca18 · Cited by 245
“‘Self-supervised learning’ (SSL) is an unsupervised learning approach that learns representations from unlabeled data, exploiting the structure of the data to provide supervision [9].”
Modality specific U-Net variants for biomedical image segmentation: a survey
Punn, N. S., & Agarwal, S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10152-1 · Cited by 246
“In SSL strategies initially, pre-training is performed with un-annotated samples for some pretext task to learn feature representations such as predicting rotations, identifying the image patch, solving jigsaw puzzles, etc. and later model is fine-tuned to perform actual segmentation.”
BackgroundInfluentialContrastive Learning with Stronger Augmentations
Wang, X., & Qi, G.-J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3203630 · Cited by 266
“For more details about these works, please refer to the survey by [36].”
BackgroundContrastive Learning with Stronger Augmentations
Wang, X., & Qi, G.-J. (2021) · arXiv · doi:10.48550/arxiv.2104.07713 · Cited by 266
“For more details about these works, please refer to the survey by [36].”
BackgroundSource Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer
Liang, J., Hu, D., Wang, Y., He, R., & Feng, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3103390 · Cited by 241
“Self-supervised learning [66] offers great feasibility for effectively utilizing unlabeled data by generating and predicting labels from these data.”
BackgroundSemi-Supervised and Unsupervised Deep Visual Learning: A Survey
Chen, Y., Mancini, M., Zhu, X., & Akata, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3201576 · Cited by 220
“Our survey is related to other surveys on semi-supervised learning [8], [13], [17], self-supervised learning [18], or both topics [19].”
BackgroundWang, Y., Jiang, Z., Li, Y., Hwang, J.-N., Xing, G., & Liu, H. (2021) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2021.3058895 · Cited by 228
“posed in the machine learning community [42]–[45].”
A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions
Zhou, S., Xu, H., Zheng, Z., Chen, J., Li, Z., Bu, J., Wu, J. X., Wang, X. Y., Zhu, W., & Ester, M. (2024) · ACM Computing Surveys · doi:10.1145/3689036 · Cited by 339
“For detailed advantages of contrastive learning, please refer to the former contrastive learning surveys [85, 95].”
A systematic study of key elements underlying molecular property prediction
Deng, J., Yang, Z., Wang, H., Ojima, I., Samaras, D., & Wang, F. (2023) · Nature Communications · doi:10.1038/s41467-023-41948-6 · Cited by 213
“Compared to supervised learning, self-supervised learning does not require heavy human annotations 71 , which can be particularly expensive in drug discovery 1 .”
FCCDN: Feature constraint network for VHR image change detection
Chen, P., Zhang, B., Hong, D., Chen, Z., Yang, X., & Li, B. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.02.021 · Cited by 231
Górriz, J. M., Álvarez-Illán, I., Álvarez-Marquina, A., Arco, J. E., Atzmueller, M., Ballarini, F., Barakova, E., Bologna, G., Bonomini, P., Castellanos-Dominguez, G., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101945 · Cited by 214
Wang, Z., Yu, M., Wei, Y., Feris, R., Xiong, J., Hwu, W.-M., Huang, T. S., & Shi, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01265 · Cited by 242
A Survey on Semi-, Self- and Unsupervised Learning for Image Classification
Schmarje, L., Santarossa, M., Schroder, S.-M., & Koch, R. (2021) · IEEE Access · doi:10.1109/access.2021.3084358 · Cited by 246
“Often, self-supervision is achieved on a pretext task on the same or a different dataset and then the pretrained network is fine-tuned on a downstream task [19].”
BackgroundTen deep learning techniques to address small data problems with remote sensing
Safonova, A., Ghazaryan, G., Stiller, S., Main-Knorn, M., Nendel, C., & Ryo, M. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103569 · Cited by 196
“…2020), speech representation (Mohamed et al., 2022), RS (S. Wang et al., 2022), time-series analysis (Pöppelbaum et al., 2022), video processing (Jing and Tian, 2019; Z. Wang et al., 2022; Yan et al., 2020), speech processing (Hsu et al., 2021), and target tracking (Yuan et al., 2021) for…”
BackgroundImproving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals
He, G., Lan, Y., Jiang, J., Zhao, W. X., & Wen, J.-R. (2021) · Proceedings of the 14th ACM International Conference on Web Search and Data Mining · doi:10.1145/3437963.3441753 · Cited by 204
“ckward reasoning has been seldom considered in multi-hop KBQA task, especially its correspondence with forward reasoning. Such an idea is indeed related to recent progress in self-supervised learning [15], in which we leverage internal supervision signal to learn the model. 5 EXPERIMENT In this section, we perform the…”
InfluentialSpeedNet: Learning the Speediness in Videos
Benaim, S., Ephrat, A., Lang, O., Mosseri, I., Freeman, W. T., Rubinstein, M., Irani, M., & Dekel, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00994 · Cited by 232
Biological underpinnings for lifelong learning machines
Kudithipudi, D., Aguilar-Simon, M., Babb, J., Bazhenov, M., Blackiston, D., Bongard, J., Brna, A. P., Chakravarthi Raja, S., Cheney, N., Clune, J., et al. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00452-0 · Cited by 236
“As in self-supervised learning 34 , curiosity-driven reinforcement learning 35 , and works looking at open-ended learning 36 , there could be additional tasks (driven by particular objective or reward functions, for example, reducing uncertainty in predicting the future) that the agent may carry…”
BackgroundUnsupervised machine learning in urban studies: A systematic review of applications
Wang, J., & Biljecki, F. (2022) · Cities · doi:10.1016/j.cities.2022.103925 · Cited by 208
“The categorization of the applications is determined through summarizing common types from multiple related studies (Jing & Tian, 2019; Miller et al., 2018; Usama et al., 2019), while fine-tuning according to the contents in the literature pool.”
MethodGenerative and Contrastive Self-Supervised Learning for Graph Anomaly Detection
Zheng, Y., Jin, M., Liu, Y., Chi, L., Phan, K. T., & Chen, Y.-P. P. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3119326 · Cited by 220
“SSL has achieved great success on computer vision (CV) [17] and natural language processing (NLP) [16].”
BackgroundUnsupervised Semantic Segmentation by Contrasting Object Mask Proposals
Van Gansbeke, W., Vandenhende, S., Georgoulis, S., & Van Gool, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00990 · Cited by 231
“Unsupervised or self-supervised techniques [38] were recently being employed to learn rich and effective visual representations without external supervision.”
MethodChallenges and Opportunities in Machine-Augmented Plant Stress Phenotyping
Singh, A., Jones, S., Ganapathysubramanian, B., Sarkar, S., Mueller, D., Sandhu, K., & Nagasubramanian, K. (2021) · Trends in Plant Science · doi:10.1016/j.tplants.2020.07.010 · Cited by 222
“However, recent advances in ML methods (e.g., active learning , AL [28]; semi-supervised learning [29]; and self-supervised learning [30]) open up the possibility of settling with smaller labeled datasets.”
BackgroundSelf-Supervised Pretraining of Transformers for Satellite Image Time Series Classification
Yuan, Y., & Lin, L. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2020.3036602 · Cited by 204
“In this paradigm, models are trained on unlabeled data by leveraging the structure present in the data itself to create supervised tasks (such tasks are often referred as “pretext tasks”) [34].”
BackgroundMethodSelf-Supervised Contrastive Representation Learning for Semi-Supervised Time-Series Classification
Eldele, E., Ragab, M., Chen, Z., Wu, M., Kwoh, C.-K., Li, X., & Guan, C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3308189 · Cited by 224
“Self-supervised learning is gaining more attention recently, as it deals with the emerging problem of learning from unlabeled data [12], [13], [14].”
Vision Transformers for Image Classification: A Comparative Survey
Wang, Y., Deng, Y., Zheng, Y., Chattopadhyay, P., & Wang, L. (2025) · Technologies · doi:10.3390/technologies13010032 · Cited by 174
“The model is pre-trained on large datasets [3,35] or combinations of multiple datasets [36,37] using self-supervised learning techniques [38,39].”
Deep Comprehensive Correlation Mining for Image Clustering
Wu, J., Long, K., Wang, F., Qian, C., Li, C., Lin, Z., & Zha, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00824 · Cited by 198
“Self-supervised learning [24, 26] generally needs to design a pretext task, where a target objective can be computed without supervision.”
BackgroundEsteva, A., Feng, J., van der Wal, D., Huang, S.-C., Simko, J. P., DeVries, S., Chen, E., Schaeffer, E. M., Morgan, T. M., Sun, Y., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00613-w · Cited by 217
“Self-supervised learning 25 is a method recently popularized in the ML community for learning from datasets without annotations.”
Battery Safety: Machine Learning-Based Prognostics
Zhao, J., Feng, X., Pang, Q., Fowler, M., Lian, Y., Ouyang, M., & Burke, A. F. (2024) · Progress in Energy and Combustion Science · doi:10.1016/j.pecs.2023.101142 · Cited by 216
GMSS: Graph-Based Multi-Task Self-Supervised Learning for EEG Emotion Recognition
Li, Y., Chen, J., Li, F., Fu, B., Wu, H., Ji, Y., Zhou, Y., Niu, Y., Shi, G., & Zheng, W. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3170428 · Cited by 207
“Based on the different data attributes used in the design, there are four categories of pretext tasks: generation-based, context-based, free semantic label-based, and cross-modal-based [42].”
MethodConditional Contrastive Domain Generalization for Fault Diagnosis
Ragab, M., Chen, Z., Zhang, W., Eldele, E., Wu, M., Kwoh, C.-K., & Li, X. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3154000 · Cited by 202
“, negative pairs) are pushed away [14].”
Hu, S. X., Li, D., Stuhmer, J., Kim, M., & Hospedales, T. M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00886 · Cited by 185
“Self-supervised & few-shot Our pipeline extends the typical unsupervised pre-train → supervised fine-tune workflow of the self-supervised research community [28, 38], which has recently demonstrated strong performance for low-shot supervised learning [15, 18, 27].”
Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners
Zhang, R., Hu, X., Li, B., Huang, S., Deng, H., Qiao, Y., Gao, P., & Li, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01460 · Cited by 171
“To improve the representation capability by overcoming the constraints of annotation, self-supervised pre-training has attracted wide attention using large-scale unlabeled datasets [40].”
BackgroundTowards Faster and Stabilized GAN Training for High-fidelity Few-shot Image SynthesisPreprint
Liu, B., Zhu, Y., Song, K., & Elgammal, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.04775 · Cited by 357
“The auto-encoding approach we employ is a typical method for self-supervised learning, which has been well recognized to improve the model robustness and generalization ability (He et al., 2020; Hendrycks et al., 2019; Jing & Tian, 2020; Goyal et al., 2019).”
MethodArtificial intelligence powered Metaverse: analysis, challenges and future perspectives
Soliman, M. M., Ahmed, E., Darwish, A., & Hassanien, A. E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10641-x · Cited by 155
“An autoencoder is a neural network that has been trained to map its input to the corresponding output using its two primary functions of compression and decompression (Jing and Tian 2021).”
Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapés, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3243465 · Cited by 169
“Nevertheless, it is still unclear what these models are actually learning, so future research is needed for proper interpretation of SSL features, which currently are mostly evaluated based on their success on downstream performance [157], [186].”
BackgroundVision Foundation Models in Remote Sensing: A survey
Lu, S., Guo, J., Zimmer-Dauphinee, J. R., Nieusma, J. M., Wang, X., VanValkenburgh, P., Wernke, S. A., & Huo, Y. (2025) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2025.3541952 · Cited by 259
“The general pipeline of SSL [51].”
Machine learning in precision diabetes care and cardiovascular risk prediction
Oikonomou, E. K., & Khera, R. (2023) · Cardiovascular Diabetology · doi:10.1186/s12933-023-01985-3 · Cited by 190
“Building on these concepts, self-supervised learning (SSL) processes unlabeled data to create key representations that can facilitate downstream tasks [25].”
BackgroundLiu, R., Gao, J., Zhang, J., Meng, D., & Lin, Z. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3132674 · Cited by 316
“knowledge distillation [230], self-supervised learning [231], and transformer [232]) are more and more sophisticated.”
BackgroundPrototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation
Yue, X., Zheng, Z., Zhang, S., Gao, Y., Darrell, T., Keutzer, K., & Vincentelli, A. S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01362 · Cited by 159
“Self-supervised learning (SSL) is a subset of unsupervised learning methods where supervision is automatically generated from the data [36, 13, 76, 51, 22, 70].”
BackgroundSurvey of Deep Representation Learning for Speech Emotion Recognition
Latif, S., Rana, R., Khalifa, S., Jurdak, R., Qadir, J., & Schuller, B. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3114365 · Cited by 172
“Self-supervised representation learning-based models are getting tremendous interest in vision [178], NLP [179], and speech recognition [180], however, emotional representation”
BackgroundSelf-Path: Self-Supervision for Classification of Pathology Images With Limited Annotations
Koohbanani, N. A., Unnikrishnan, B., Khurram, S. A., Krishnaswamy, P., & Rajpoot, N. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3056023 · Cited by 181
Li, H., Li, Y., Zhang, G., Liu, R., Huang, H., Zhu, Q., & Tao, C. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3147513 · Cited by 170
“Self-supervised learning (SSL) [34]–[36] provides a new paradigm, as shown in Fig.”
BackgroundMethodInfluentialDeep Learning in Breast Cancer Imaging: A Decade of Progress and Future Directions
Luo, L., Wang, X., Lin, Y., Ma, X., Tan, A., Chan, R., Vardhanabhuti, V., Chu, C. W., Cheng, K.-T. T., & Chen, H. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3357877 · Cited by 227
“…methods, such as the generative adversarial network (GAN) [63], to model the data distribution and generate new samples, which is also quite often used in SSL. Self-supervised learning trains a neural network on the unlabeled images to learn representations for the supervised downstream tasks [64].”
Unsupervised Point Cloud Representation Learning With Deep Neural Networks: A Survey
Xiao, A., Huang, J., Guan, D., Zhang, X., Lu, S., & Shao, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3262786 · Cited by 224
“[24] introduced advances on unsupervised representation learning in 2D computer vision; Liu et al.”
Background
Metadata
- Authors
- Jing, L., & Tian, Y.
- Affiliations
- The Graduate Center, CUNY (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2021-11-01
- Volume
- 43
- Issue
- 11
- Pages
- 4037-4058
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3023371261
- Cited by
- 2,036
- References
- 206
- Citations by year
- 2026: 160 · 2025: 283 · 2024: 354 · 2023: 405 · 2022: 377 · 2021: 305
- FWCI
- 119.51
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (IIS-1400802)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- self-supervised learning, visual feature learning, imaging features, video features, deep learning architectures, unlabeled data, audio features, 3D data, benchmark dataset, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.