Self-Supervised Learning of Graph Neural Networks: A Unified Review
Open accessThe DOI matches its Crossref record; 96 of the 96 references it cites were found in a registry or an index.
Abstract
Deep models trained in supervised mode have achieved remarkable success on a variety of tasks. When labeled samples are limited, self-supervised learning (SSL) is emerging as a new paradigm for making use of large amounts of unlabeled samples. SSL has achieved promising performance on natural language and image learning tasks. Recently, there is a trend to extend such success to graph data using graph neural networks (GNNs). In this survey, we provide a unified review of different ways of training GNNs using SSL. Specifically, we categorize SSL methods into contrastive and predictive models. In either category, we provide a unified framework for methods as well as how these methods differ in each component under the framework. Our unified treatment of SSL methods for GNNs sheds light on the similarities and differences of various methods, setting the stage for developing new methods and algorithms. We also summarize different SSL settings and the corresponding datasets used in each setting. To facilitate methodological development and empirical comparison, we develop a standardized testbed for SSL in GNNs, including implementations of common baseline methods, datasets, and evaluation metrics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Inductive Representation Learning on Large Graphs
Hamilton, W. L., Ying, R., & Leskovec, J. (2017) · arXiv · doi:10.48550/arxiv.1706.02216 · Cited by 10,639
DeepWalk: online learning of social representations
Perozzi, B., Al-Rfou, R., & Skiena, S. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623732 · Cited by 10,123
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
Modeling Relational Data with Graph Convolutional Networks
Schlichtkrull, M., Kipf, T. N., Bloem, P., van den Berg, R., Titov, I., & Welling, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93417-4_38 · Cited by 5,570
BPR: Bayesian Personalized Ranking from Implicit Feedback
Rendle, S., Freudenthaler, C., Gantner, Z., & Schmidt-Thieme, L. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.2618 · Cited by 4,242
Research works printed but not linked: 38
- Yang, Rethinking the value of labels for improving class-imbalanced learning, 19290
- Xie, 2020, Noise2Same: Optimizing a self-supervised bound for image denoising, 20320
- Laine, 2019, High-quality self-supervised deep image denoising, Adv. Neural Inf. Process. Syst., 32, 6970
- Batson, Noise2Self: Blind denoising by self-supervision, 524
- Hassani, Contrastive multi-view representation learning on graphs, 4116
- Chen, A simple framework for contrastive learning of visual representations, 1597
- Kipf, Semi-supervised classification with graph convolutional networks
- Xu, 2019, How powerful are graph neural networks?, Int. Conf. Learn. Representations
- Yuan, StructPool: Structured graph pooling via conditional random fields
- Veličković, 2019, Deep graph infomax, Int. Conf. Learn. Representations
- Sun, Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization
- Hu, Strategies for pre-training graph neural networks
- You, Graph contrastive learning with augmentations, 5812
- 10.1109/ICCV.2019.00662 · DOI
- Hasanzadeh, Semi-implicit graph variational auto-encoders, 10 712
- Rong, Self-supervised graph transformer on large-scale molecular data, 12559
- Zhang, From canonical correlation analysis to self-supervised graph neural networks, 76
- Ganin, Unsupervised domain adaptation by backpropagation, 1180
- Hjelm, Learning deep representations by mutual information estimation and maximization
- Belghazi, Mutual information neural estimation, 531
- Gutmann, Noise-contrastive estimation: A new estimation principle for unnormalized statistical models, 297
- Sohn, 2016, Improved deep metric learning with multi-class n-pair loss objective, 1857
- Suresh, Adversarial graph augmentation to improve graph contrastive learning, 15920
- Xu, InfoGCL: Information-aware graph contrastive learning, 30414
- Jin, Junction tree variational autoencoder for molecular graph generation, 2323
- Li, Dirichlet graph variational autoencoder, 5274
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1
- Zbontar, Barlow twins: Self-supervised learning via redundancy reduction, 12310
- Tishby, The information bottleneck method, 368
- Morris, Tudataset: A collection of benchmark datasets for learning with graphs
- Yang, Revisiting semi-supervised learning with graph embeddings, 40
- Zeng, GraphSAINT: Graph sampling based inductive learning method
- 10.5555/3454287.3455008 · doi:10.5555/3454287.3455008
- Fey, Fast graph representation learning with PyTorch geometric
- You, GraphRNN: Generating realistic graphs with deep auto-regressive models, 5708
- Shi, Graphaf: A flow-based autoregressive model for molecular graph generation
- Luo, Graphdf: A discrete flow model for molecular graph generation, 7192
- 10.24963/ijcai.2021/371 · doi:10.24963/ijcai.2021/371
Cited by
Self-supervised learning in medicine and healthcare
Krishnan, R., Rajpurkar, P., & Topol, E. J. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00914-1 · Cited by 605
“Molecules can also be augmented by randomly removing some bonds, or by removing whole portions of atoms and bonds from parts of the molecule 25 .”
BackgroundGraph Self-Supervised Learning: A Survey
Liu, Y., Jin, M., Pan, S., Zhou, C., Zheng, Y., Xia, F., & Yu, P. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3172903 · Cited by 510
Machine Learning Methods for Small Data Challenges in Molecular Science
Dou, B., Zhu, Z., Merkurjev, E., Ke, L., Chen, L., Jiang, J., Zhu, Y., Liu, J., Zhang, B., & Wei, G.-W. (2023) · Chemical Reviews · doi:10.1021/acs.chemrev.3c00189 · Cited by 499
Self-supervised learning for recommender systems: A survey
Yu, J., Yin, H., Xia, X., Chen, T., Li, J., & Huang, Z. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3282907 · Cited by 413
“While SSL has been extensively surveyed in the fields of CV, NLP [34], [9] and graph learning [35], [36], [37], there has not been a systematic investigation of research endeavors on SSR despite the growing number of publications.”
BackgroundZhao, Z., Alzubaidi, L., Zhang, J., Duan, Y., & Gu, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122807 · Cited by 410
Self-citationA Comprehensive Survey on Deep Graph Representation Learning
Ju, W., Fang, Z., Gu, Y., Liu, Z., Long, Q., Qiao, Z., Qin, Y., Shen, J., Sun, F., Xiao, Z., et al. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106207 · Cited by 341
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“The SSL methods can be roughly divided into the following categories: generative-based, contrastive-based, and adversarial-based methods.”
BackgroundSelf-citationSelf-Supervised Learning on Graphs: Contrastive, Generative, or Predictive
Wu, L., Lin, H., Tan, C., Gao, Z., & Li, S. Z. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3131584 · Cited by 290
Self-supervised learning methods and applications in medical imaging analysis: a survey
Shurrab, S., & Duwairi, R. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1045 · Cited by 254
TS-IDS: Traffic-aware self-supervised learning for IoT Network Intrusion Detection
Nguyen, H., & Kashef, R. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110966 · Cited by 94
Metadata
- Authors
- Xie, Y., Xu, Z., Zhang, J., Wang, Z., & Ji, S.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-02-01
- Volume
- 45
- Issue
- 2
- Pages
- 2412-2429
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3129850062
- Cited by
- 389
- References
- 96
- FWCI
- 26.33
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- National Science Foundation (IIS-2006861, DBI-2028361)
- National Institutes of Health (1R21NS102828)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.