Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 1,087

The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index.

Self-supervised Learning: Generative or Contrastive

Open access
Liu, Xiao, Zhang, Fanjin, Hou, Zhenyu, Mian, Li, Wang, Zhaoyu, Zhang, Jing, Tang, Jie · 2021 · IEEE Transactions on Knowledge and Data Engineering

Abstract

Deep supervised learning has achieved great success in the last decade. However, its defects of heavy dependence on manual labels and vulnerability to attacks have driven people to find other paradigms. As an alternative, self-supervised learning (SSL) attracts many researchers for its soaring performance on representation learning in the last several years. Self-supervised representation learning leverages input data itself as supervision and benefits almost all types of downstream tasks. In this survey, we take a look into new self-supervised learning methods for representation in computer vision, natural language processing, and graph learning. We comprehensively review the existing empirical methods and summarize them into three main categories according to their objectives: generative, contrastive, and generative-contrastive (adversarial). We further collect related theoretical analyses on self-supervised learning to provide deeper thoughts on why self-supervised learning works. Finally, we briefly discuss open problems and future directions for self-supervised learning. An outline slide for the survey is provided.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tkde.2021.3090866

Cites

91 of 91 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 22

  1. ganin, 2016, Domain-adversarial training of neural networks, Journal of Machine Learning Research, 17, 2096
  2. donahue, 2019, Large scale adversarial representation learning, Proc 33rd Int Conf Neural Inf Process Syst, 10541
  3. ho, 2019, Flow++: Improving flow-based generative models with variational dequantization and architecture design, Proc 36th Int Conf Mach Learn, 2722
  4. hu, 2019, Strategies for pre-training graph neural networks, Proc Int Conf Learn Representations
  5. zhu, 2017, Toward multimodal image-to-image translation, Proc 31st Int Conf Neural Inf Process Syst, 465
  6. goodfellow, 2014, Generative adversarial nets, Proc 27th Int Conf Neural Inf Process Syst, 2672
  7. arjovsky, 2017, Wasserstein generative adversarial networks, Proc 34th Int Conf Mach Learn, 214
  8. bachman, 2019, Learning representations by maximizing mutual information across views, Proc 33rd Int Conf Neural Inf Process Syst, 15509
  9. ballard, 1987, Modular learning in neural networks, Proc 6th Nat Conf Artif Intell, 279
  10. gutmann, 2010, Noise-contrastive estimation: A new estimation principle for unnormalized statistical models, Proc 13th Int Conf Artif Intell Statist, 297
  11. vaswani, 2017, Attention is all you need, Proc 31st Int Conf Neural Inf Process Syst, 5998
  12. van oord, 2016, Pixel recurrent neural networks, Proc 33rd Int Conf Mach Learn, 1747
  13. kingma, 2018, Glow: Generative flow with invertible 1x1 convolutions, Proc 32nd Int Conf Neural Inf Process Syst, 10215
  14. you, 2018, Graph convolutional policy network for goal-directed molecular graph generation, Proc 32nd Int Conf Neural Inf Process Syst, 6410
  15. you, 2018, GraphRNN: Generating realistic graphs with deep auto-regressive models, Proc 35th Int Conf Mach Learn, 5708
  16. krizhevsky, 2012, ImageNet classification with deep convolutional neural networks, Proc 25th Int Conf Neural Inf Process Syst, 1097
  17. shen, 2017, Style transfer from non-parallel text by cross-alignment, Proc 31st Int Conf Neural Inf Process Syst, 6830
  18. shen, 2017, Adversarial representation learning for domain adaptation, Stat, 1050
  19. sun, 2019, vGraph: A generative model for joint community detection and node representation learning, Proc 33rd Int Conf Neural Inf Process Syst, 512
  20. radford, 2019, Language models are unsupervised multitask learners, OpenAIRE blog, 1
  21. van den oord, 2016, WaveNet: A generative model for raw audio, Proc 9th ISCA Speech Synth Workshop, 125
  22. ng, 2011, Sparse autoencoder, Cs294a lecture notes, 72, 1

Cited by

See all 1,087

Metadata

Authors
Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J.
Venue
IEEE Transactions on Knowledge and Data Engineering
Published
2021
Pages
1-1
ISSN
  • 1041-4347 (ISSN-L)
  • 1558-2191
  • 2326-3865
OpenAlex
W3035725276
Cited by
1,087
References
91
FWCI
65.46
Type
journal article
Language
en
Open access
green
Funders
National Natural Science Foundation of China

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.