Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 1,087

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.

Self-supervised Learning: Generative or Contrastive

Open access
Liu, Xiao, Zhang, Fanjin, Hou, Zhenyu, Mian, Li, Wang, Zhaoyu, Zhang, Jing, Tang, Jie·2021·IEEE Transactions on Knowledge and Data Engineering
DOI 10.1109/tkde.2021.3090866
Open this paper in CoChat — free account

Abstract

Deep supervised learning has achieved great success in the last decade. However, its defects of heavy dependence on manual labels and vulnerability to attacks have driven people to find other paradigms. As an alternative, self-supervised learning (SSL) attracts many researchers for its soaring performance on representation learning in the last several years. Self-supervised representation learning leverages input data itself as supervision and benefits almost all types of downstream tasks. In this survey, we take a look into new self-supervised learning methods for representation in computer vision, natural language processing, and graph learning. We comprehensively review the existing empirical methods and summarize them into three main categories according to their objectives: generative, contrastive, and generative-contrastive (adversarial). We further collect related theoretical analyses on self-supervised learning to provide deeper thoughts on why self-supervised learning works. Finally, we briefly discuss open problems and future directions for self-supervised learning. An outline slide for the survey is provided.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tkde.2021.3090866

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 45 references

Research works printed but not linked: 64

  1. ganin, 2016, Domain-adversarial training of neural networks, Journal of Machine Learning Research, 17, 2096
  2. 10.1145/3219819.3220025 · DOI · Paper page
  3. donahue, 2019, Large scale adversarial representation learning, Proc 33rd Int Conf Neural Inf Process Syst, 10541
  4. 10.1145/3269206.3271768 · DOI
  5. chongxuan, 2017, Triple generative adversarial nets, Proc 31st Int Conf Neural Inf Process Syst, 4088
  6. 10.18653/v1/P19-1285 · DOI · Paper page
  7. 10.1609/aaai.v32i1.11865 · DOI · Paper page
  8. de sa, 1994, Learning classification with unlabeled data, Proc 6th Int Conf Neural Inf Process, 112
  9. ho, 2019, Flow++: Improving flow-based generative models with variational dequantization and architecture design, Proc 36th Int Conf Mach Learn, 2722
  10. hu, 2019, Strategies for pre-training graph neural networks, Proc Int Conf Learn Representations
  11. 10.1145/3219819.3220052 · DOI
  12. 10.1109/ICCV.2019.00610 · DOI · Paper page
  13. zhu, 2017, Toward multimodal image-to-image translation, Proc 31st Int Conf Neural Inf Process Syst, 465
  14. 10.1609/aaai.v32i1.11782 · DOI · Paper page
  15. 10.1007/978-3-319-46487-9_40 · DOI · Paper page
  16. 10.1109/CVPR.2017.76 · DOI · Paper page
  17. iizuka, 2017, Globally and locally consistent image completion, ACM Trans Graph, 36, 1, 10.1145/3072959.3073659 · DOI · Paper page
  18. 10.1145/3366423.3380027 · DOI · Paper page
  19. 10.1145/3394486.3403237 · DOI · Paper page
  20. goodfellow, 2014, Generative adversarial nets, Proc 27th Int Conf Neural Inf Process Syst, 2672
  21. arjovsky, 2017, Wasserstein generative adversarial networks, Proc 34th Int Conf Mach Learn, 214
  22. bachman, 2019, Learning representations by maximizing mutual information across views, Proc 33rd Int Conf Neural Inf Process Syst, 15509
  23. ballard, 1987, Modular learning in neural networks, Proc 6th Nat Conf Artif Intell, 279
  24. 10.1145/3289600.3290967 · DOI · Paper page
  25. gutmann, 2010, Noise-contrastive estimation: A new estimation principle for unnormalized statistical models, Proc 13th Int Conf Artif Intell Statist, 297
  26. vaswani, 2017, Attention is all you need, Proc 31st Int Conf Neural Inf Process Syst, 5998
  27. van oord, 2016, Pixel recurrent neural networks, Proc 33rd Int Conf Mach Learn, 1747
  28. 10.1007/978-3-319-46493-0_41 · DOI
  29. 10.1609/aaai.v32i1.11536 · DOI
  30. 10.1609/aaai.v32i1.11872 · DOI · Paper page
  31. 10.1109/CVPR.2017.96 · DOI · Paper page
  32. 10.1007/978-3-031-02145-9 · DOI · Paper page
  33. 10.1109/CVPR.2019.00201 · DOI
  34. 10.1109/CVPR.2016.556 · DOI · Paper page
  35. kingma, 2018, Glow: Generative flow with invertible 1x1 convolutions, Proc 32nd Int Conf Neural Inf Process Syst, 10215
  36. 10.1109/WACV.2018.00092 · DOI · Paper page
  37. you, 2018, Graph convolutional policy network for goal-directed molecular graph generation, Proc 32nd Int Conf Neural Inf Process Syst, 6410
  38. you, 2018, GraphRNN: Generating realistic graphs with deep auto-regressive models, Proc 35th Int Conf Mach Learn, 5708
  39. krizhevsky, 2012, ImageNet classification with deep convolutional neural networks, Proc 25th Int Conf Neural Inf Process Syst, 1097
  40. 10.18653/v1/P18-1099 · DOI
  41. 10.1145/3292500.3330785 · DOI
  42. 10.24963/ijcai.2019/594 · DOI · Paper page
  43. 10.1145/3219819.3220077 · DOI · Paper page
  44. 10.1145/3394486.3403168 · DOI · Paper page
  45. 10.1145/2740908.2742839 · DOI · Paper page
  46. 10.1145/2623330.2623732 · DOI · Paper page
  47. shen, 2017, Style transfer from non-parallel text by cross-alignment, Proc 31st Int Conf Neural Inf Process Syst, 6830
  48. shen, 2017, Adversarial representation learning for domain adaptation, Stat, 1050
  49. 10.1109/ICCV.2019.00591 · DOI · Paper page
  50. 10.1145/3097983.3098061 · DOI · Paper page
  51. sun, 2019, vGraph: A generative model for joint community detection and node representation learning, Proc 33rd Int Conf Neural Inf Process Syst, 512
  52. 10.1609/aaai.v34i04.6048 · DOI · Paper page
  53. radford, 2019, Language models are unsupervised multitask learners, OpenAIRE blog, 1
  54. 10.1007/978-3-030-01264-9_9 · DOI · Paper page
  55. 10.1177/107769905303000401 · DOI · Paper page
  56. 10.1145/3097983.3098202 · DOI · Paper page
  57. 10.1109/CVPR42600.2020.00737 · DOI
  58. 10.1145/2736277.2741093 · DOI · Paper page
  59. van den oord, 2016, WaveNet: A generative model for raw audio, Proc 9th ISCA Speech Synth Workshop, 125
  60. van den oord, 2016, Conditional image generation with pixelcnn decoders, Proc 30th Int Conf Neural Inf Process Syst, 4790
  61. van den oord, 2017, Neural discrete representation learning, Proc 31st Int Conf Neural Inf Process Syst, 6306
  62. ng, 2011, Sparse autoencoder, Cs294a lecture notes, 72, 1
  63. 10.1007/978-3-319-46466-4_5 · DOI · Paper page
  64. 10.1109/CVPR.2018.00975 · DOI · Paper page

Cited by

See all 1,087

Metadata

Authors
Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J.
Venue
IEEE Transactions on Knowledge and Data Engineering
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2021
Pages
1-1
ISSN
  • 1041-4347 (ISSN-L)
  • 1558-2191
  • 2326-3865
DOI
10.1109/tkde.2021.3090866
OpenAlex
W3035725276
Cited by
1,087
References
45
FWCI
65.46
Type
journal article
Language
en
Open access
green
Funders
National Natural Science Foundation of China

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Self-supervised Learning: Generative or Contrastive. https://app.cochat.ai/papers/10.1109/tkde.2021.3090866
The paper
Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J. (2021). Self-supervised Learning: Generative or Contrastive. IEEE Transactions on Knowledge and Data Engineering. https://doi.org/10.1109/tkde.2021.3090866

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.