CoChat
Sign Up
PreprintCited by 574Topic ModelingDomain Adaptation and Few-Shot LearningMultimodal Machine Learning Applications

A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT

Open access
Zhou, Ce, Li, Qian, Li, Chen, Yu, Jun, Liu, Yixin, Wang, Guangjing, Zhang, Kai, Ji, Cheng +11 more·2023·arXiv
DOI 10.48550/arxiv.2302.09419arXiv 2302.09419
Open this paper in CoChat — free accountOpen PDF

Abstract

Pretrained Foundation Models (PFMs) are regarded as the foundation for various downstream tasks with different data modalities. A PFM (e.g., BERT, ChatGPT, and GPT-4) is trained on large-scale data which provides a reasonable parameter initialization for a wide range of downstream applications. BERT learns bidirectional encoder representations from Transformers, which are trained on large datasets as contextual language models. Similarly, the generative pretrained transformer (GPT) method employs Transformers as the feature extractor and is trained using an autoregressive paradigm on large datasets. Recently, ChatGPT shows promising success on large language models, which applies an autoregressive language model with zero shot or few shot prompting. The remarkable achievements of PFM have brought significant breakthroughs to various fields of AI. Numerous studies have proposed different methods, raising the demand for an updated survey. This study provides a comprehensive review of recent research advancements, challenges, and opportunities for PFMs in text, image, graph, as well as other data modalities. The review covers the basic components and existing pretraining methods used in natural language processing, computer vision, and graph learning. Additionally, it explores advanced PFMs used for different data modalities and unified PFMs that consider data quality and quantity. The review also discusses research related to the fundamentals of PFMs, such as model efficiency and compression, security, and privacy. Finally, the study provides key implications, future research directions, challenges, and open problems in the field of PFMs. Overall, this survey aims to shed light on the research of the PFMs on scalability, security, logical reasoning ability, cross-domain learning ability, and the user-friendly interactive ability for artificial general intelligence.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2302.09419

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 319 · Cited by 574

Research works printed but not linked: 407

  1. [8]8. G. G. Chowdhury, "Natural language processing," Annual review of information science and technol- ogy, 2003.
  2. [12]12. J. Li, T. Tang, W. X. Zhao, and J.-R. Wen, "Pretrained language models for text generation: A survey," arXiv, 2021.
  3. [13]13. K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, et al., "A survey on visual transformer," arXiv, 2020.
  4. [14]14. S. Sanchez, H. Romero, and A. Morales, "A review: Comparison of performance metrics of pretrained models for object detection using the tensorflow framework," in IOP Conference Series: Materials Science and Engineering.
  5. [15]15. W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec, "Pre-training graph neural networks," arXiv, 2019.
  6. [16]16. F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, "A comprehensive survey on transfer learning," Proceedings of the IEEE, 2020.
  7. [18]18. T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," in Proc. ICLR, 2013, 2013.
  8. [19]19. J. Devlin, M. Chang, K. Lee, and K. Toutanova, "BERT: pre-training of deep bidirectional transform- ers for language understanding," in NAACL-HLT, 2019.
  9. [20]20. Z. Yang, Z. Dai, Y. Yang, J. G. Carbonell, R. Salakhutdinov, and Q. V. Le, "Xlnet: Generalized autoregressive pretraining for language understanding," in NeurIPS.
  10. [23]23. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforcement learn- ing from human preferences," Advances in neural information processing systems, vol. 30, 2017.
  11. [24]24. N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Chris- tiano, "Learning to summarize with human feedback," Advances in Neural Information Processing Systems, vol. 33, pp. 3008-3021, 2020.
  12. [26]26. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., "Language models are few-shot learners," arXiv, 2020.
  13. [27]27. B. Lester, R. Al-Rfou, and N. Constant, "The power of scale for parameter-efficient prompt tun- ing," in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, 2021.
  14. [28]28. T. Schick and H. Schütze, "Exploiting cloze-questions for few-shot text classification and natural lan- guage inference," in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269, 2021.
  15. [29]29. Z. Zhang, A. Zhang, M. Li, and A. Smola, "Automatic chain of thought prompting in large language models," in International Conference on Learning Representations, 2023.
  16. [30]30. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, et al., "Chain- of-thought prompting elicits reasoning in large language models," in Advances in Neural Information Processing Systems, 2022.
  17. [34]34. A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela, "Flava: A foundational language and vision alignment model," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15638-15650, 2022.
  18. [36]36. K. Clark, M. Luong, Q. V. Le, and C. D. Manning, "ELECTRA: pre-training text encoders as dis- criminators rather than generators," in ICLR, 2020.
  19. [38]38. Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela, "Adversarial NLI: A new bench- mark for natural language understanding," in ACL.
  20. [39]39. T. Niven and H. Kao, "Probing neural network comprehension of natural language arguments," in ACL.
  21. [40]40. G. Wang, N. Ivanov, B. Chen, Q. Wang, and Q. Yan, "Graph learning for interactive threat detec- tion in heterogeneous smart home rule data," in 2023 ACM SIGMOD International Conference on Management of Data, ACM, 2023.
  22. [41]41. M. A. Gordon, K. Duh, and N. Andrews, "Compressing BERT: studying the effects of weight pruning on transfer learning," in RepL4NLP@ACL.
  23. [42]42. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "ALBERT: A lite BERT for self-supervised learning of language representations," in ICLR, 2020.
  24. [44]44. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," arXiv, 2017.
  25. [45]45. M.-H. Guo, T.-X. Xu, J.-J. Liu, Z.-N. Liu, P.-T. Jiang, T.-J. Mu, S.-H. Zhang, R. R. Martin, M.-M. Cheng, and S.-M. Hu, "Attention mechanisms in computer vision: A survey," Computational Visual Media, vol. 8, no. 3, pp. 331-368, 2022.
  26. [47]47. S. Yun, M. Jeong, R. Kim, J. Kang, and H. J. Kim, "Graph transformer networks," Advances in neural information processing systems, vol. 32, 2019.
  27. [50]50. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy, "Spanbert: Improving pre- training by representing and predicting spans," Trans. Assoc. Comput. Linguistics, 2020.
  28. [51]51. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettle- moyer, "BART: denoising sequence-to-sequence pre-training for natural language generation, trans- lation, and comprehension," in ACL.
  29. [52]52. K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, "Electra: Pre-training text encoders as discrim- inators rather than generators," arXiv, 2020.
  30. [53]53. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "Albert: A lite bert for self- supervised learning of language representations," arXiv, 2019.
  31. [54]54. J. Donahue, P. Krähenbühl, and T. Darrell, "Adversarial feature learning," arXiv, 2016.
  32. [55]55. M. Caron, P. Bojanowski, A. Joulin, and M. Douze, "Deep clustering for unsupervised learning of visual features," in ECCV.
  33. [59]59. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, "Deep contextualized word representations," arXiv, 2018.
  34. [60]60. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al., "Google's neural machine translation system: Bridging the gap between human and machine translation," arXiv, 2016.
  35. [61]61. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoy- anov, "Roberta: A robustly optimized BERT pretraining approach," CoRR, 2019.
  36. [62]62. R. Sennrich, B. Haddow, and A. Birch, "Neural machine translation of rare words with subword units," arXiv, 2015.
  37. [63]63. K. Song, X. Tan, T. Qin, J. Lu, and T. Liu, "Mpnet: Masked and permuted pre-training for language understanding," in NeurIPS, 2020.
  38. [64]64. Q. Li, H. Peng, J. Li, C. Xia, R. Yang, L. Sun, P. S. Yu, and L. He, "A survey on text classification: From traditional to deep learning," ACM Transactions on Intelligent Systems and Technology (TIST), vol. 13, no. 2, pp. 1-41, 2022.
  39. [65]65. K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu, "Mass: Masked sequence to sequence pre-training for language generation," arXiv, 2019.
  40. [66]66. L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H.-W. Hon, "Unified language model pre-training for natural language understanding and generation," arXiv, 2019.
  41. [67]67. Y. Sun, S. Wang, Y. Li, S. Feng, X. Chen, H. Zhang, X. Tian, D. Zhu, H. Tian, and H. Wu, "Ernie: Enhanced representation through knowledge integration," arXiv, 2019.
  42. [68]68. Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, and H. Wang, "Ernie 2.0: A continual pre-training framework for language understanding," in AAAI.
  43. [70]70. S. Diao, J. Bai, Y. Song, T. Zhang, and Y. Wang, "ZEN: pre-training chinese text encoder enhanced by n-gram representations," in EMNLP.
  44. [71]71. H. Tsai, J. Riesa, M. Johnson, N. Arivazhagan, X. Li, and A. Archer, "Small and practical bert models for sequence labeling," arXiv, 2019.
  45. [74]74. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, "Distributed representations of words and phrases and their compositionality," arXiv, 2013.
  46. [75]75. J. Pennington, R. Socher, and C. D. Manning, "Glove: Global vectors for word representation," in EMNLP.
  47. [76]76. A. M. Dai and Q. V. Le, "Semi-supervised sequence learning," arXiv, 2015.
  48. [77]77. P. Liu, X. Qiu, and X. Huang, "Recurrent neural network for text classification with multi-task learn- ing," arXiv, 2016.
  49. [78]78. P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, "Enriching word vectors with subword informa- tion," TACL, 2017.
  50. [79]79. B. McCann, J. Bradbury, C. Xiong, and R. Socher, "Learned in translation: Contextualized word vectors," arXiv, 2017.
  51. [80]80. Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov, "Transformer-xl: Attentive language models beyond a fixed-length context," arXiv, 2019.
  52. [81]81. L. Kong, C. d. M. d'Autume, W. Ling, L. Yu, Z. Dai, and D. Yogatama, "A mutual information maximization perspective of language representation learning," arXiv, 2019.
  53. [82]82. W. Wang, B. Bi, M. Yan, C. Wu, Z. Bao, J. Xia, L. Peng, and L. Si, "Structbert: Incorporating language structures into pre-training for deep language understanding," arXiv, 2019.
  54. [83]83. W. Xiong, J. Du, W. Y. Wang, and V. Stoyanov, "Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model," arXiv, 2019.
  55. [84]84. M. E. Peters, M. Neumann, R. L. Logan IV, R. Schwartz, V. Joshi, S. Singh, and N. A. Smith, "Knowledge enhanced contextual word representations," arXiv, 2019.
  56. [85]85. H. Huang, Y. Liang, N. Duan, M. Gong, L. Shou, D. Jiang, and M. Zhou, "Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks," arXiv, 2019.
  57. [86]86. J. M. Eisenschlos, S. Ruder, P. Czapla, M. Kardas, S. Gugger, and J. Howard, "Multifit: Efficient multi-lingual language model fine-tuning," arXiv, 2019.
  58. [87]87. I. Beltagy, K. Lo, and A. Cohan, "Scibert: A pretrained language model for scientific text," arXiv, 2019.
  59. [88]88. S. Sun, Y. Cheng, Z. Gan, and J. Liu, "Patient knowledge distillation for bert model compression," arXiv, 2019.
  60. [89]89. G. Lample and A. Conneau, "Cross-lingual language model pretraining," arXiv, 2019.
  61. [90]90. O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat, "Q8BERT: quantized 8bit BERT," in EMC2@NeurIPS.
  62. [91]91. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter," arXiv, 2019.
  63. [92]92. W. Liu, P. Zhou, Z. Zhao, Z. Wang, H. Deng, and Q. Ju, "Fastbert: a self-distilling bert with adaptive inference time," arXiv, 2020.
  64. [93]93. L. Martin, B. Müller, P. J. O. Suárez, Y. Dupont, L. Romary, É. de la Clergerie, D. Seddah, and B. Sagot, "Camembert: a tasty french language model," in ACL.
  65. [94]94. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, "Unsupervised cross-lingual representation learning at scale," in ACL.
  66. [95]95. N. Kitaev, L. Kaiser, and A. Levskaya, "Reformer: The efficient transformer," in ICLR, 2020.
  67. [96]96. S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer, "Q-bert: Hessian based ultra low precision quantization of bert," in AAAI.
  68. [97]97. Z. Chi, L. Dong, F. Wei, W. Wang, X.-L. Mao, and H. Huang, "Cross-lingual natural language gener- ation via pre-training," in AAAI.
  69. [98]98. W. Liu, P. Zhou, Z. Zhao, Z. Wang, Q. Ju, H. Deng, and P. Wang, "K-bert: Enabling language representation with knowledge graph," in AAAI.
  70. [99]99. Z. Jiang, W. Yu, D. Zhou, Y. Chen, J. Feng, and S. Yan, "Convbert: Improving BERT with span-based dynamic convolution," in NeurIPS, 2020.
  71. [100]100. W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, "Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers," in NeurIPS, 2020.
  72. [101]101. Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, "Mul- tilingual denoising pre-training for neural machine translation," Trans. Assoc. Comput. Linguistics, 2020.
  73. [102]102. T. Sun, Y. Shao, X. Qiu, Q. Guo, Y. Hu, X. Huang, and Z. Zhang, "Colake: Contextualized language and knowledge embedding," in COLING.
  74. [103]103. H. Le, L. Vial, J. Frej, V. Segonne, M. Coavoux, B. Lecouteux, A. Allauzen, B. Crabbé, L. Besacier, and D. Schwab, "Flaubert: Unsupervised language model pre-training for french," in LREC.
  75. [104]104. T. Shen, Y. Mao, P. He, G. Long, A. Trischler, and W. Chen, "Exploiting structured knowledge in text via graph-guided representation learning," in EMNLP.
  76. [105]105. X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, "Tinybert: Distilling BERT for natural language understanding," in EMNLP.
  77. [106]106. P. Delobelle, T. Winters, and B. Berendt, "Robbert: a dutch roberta-based language model," in EMNLP.
  78. [107]107. B. He, D. Zhou, J. Xiao, X. Jiang, Q. Liu, N. J. Yuan, and T. Xu, "Integrating graph contextualized knowledge into pre-trained language models," in EMNLP.
  79. [109]109. T. Schick and H. Schütze, "Exploiting cloze-questions for few-shot text classification and natural language inference," in EACL, 2021.
  80. [110]110. X. Wang, T. Gao, Z. Zhu, Z. Zhang, Z. Liu, J. Li, and J. Tang, "KEPLER: A unified model for knowledge embedding and pre-trained language representation," Trans. Assoc. Comput. Linguistics, 2021.
  81. [111]111. T. Gao, X. Yao, and D. Chen, "Simcse: Simple contrastive learning of sentence embeddings," CoRR, 2021.
  82. [112]112. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Fi- rat, et al., "Glam: Efficient scaling of language models with mixture-of-experts," in International Conference on Machine Learning, pp. 5547-5569, PMLR, 2022.
  83. [119]119. J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, "Finetuned language models are zero-shot learners," in International Conference on Learning Representations, 2022.
  84. [121]121. Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al., "Super-naturalinstructions: Generalization via declar- ative instructions on 1600+ nlp tasks," in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5085-5109, 2022.
  85. [122]122. S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi, "Cross-task generalization via natural language crowdsourcing instructions," in Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 3470-3487, 2022.
  86. [125]125. S. Kiegeland and J. Kreutzer, "Revisiting the weaknesses of reinforcement learning for neural ma- chine translation," in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1673-1681, 2021.
  87. [126]126. N. Jaques, J. H. Shen, A. Ghandeharioun, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Pi- card, "Human-centric dialog training via offline reinforcement learning," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3985-4003, 2020.
  88. [127]127. S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, "Self-critical sequence training for image captioning," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7008-7024, 2017.
  89. [128]128. R. Y. Pang and H. He, "Text generation by learning from demonstrations," in Proceedings of the international conference on learning representations, 2021.
  90. [129]129. M. Hausknecht, P. Ammanabrolu, M.-A. Côté, and X. Yuan, "Interactive fiction games: A colossal adventure," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 7903-7910, 2020.
  91. [132]132. V. Uc-Cetina, N. Navarro-Guerrero, A. Martin-Gonzalez, C. Weber, and S. Wermter, "Survey on reinforcement learning for language processing," Artificial Intelligence Review, pp. 1-33, 2022.
  92. [139]139. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, "Large language models are zero-shot rea- soners," in Advances in Neural Information Processing Systems (A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, eds.), 2022.
  93. [140]140. A. Dosovitskiy, J. T. Springenberg, M. Riedmiller, and T. Brox, "Discriminative unsupervised feature learning with convolutional neural networks," Advances in neural information processing systems, 2014.
  94. [141]141. A. Dosovitskiy, P. Fischer, J. T. Springenberg, M. Riedmiller, and T. Brox, "Discriminative unsuper- vised feature learning with exemplar convolutional neural networks," TPAMI, 2016.
  95. [142]142. C. Doersch, A. Gupta, and A. A. Efros, "Unsupervised visual representation learning by context prediction," in ICCV.
  96. [143]143. D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, "Context encoders: Feature learn- ing by inpainting," in CVPR.
  97. [144]144. R. Zhang, P. Isola, and A. A. Efros, "Colorful image colorization," in ECCV, 2016.
  98. [145]145. R. Zhang, P. Isola, and A. A. Efros, "Split-brain autoencoders: Unsupervised learning by cross- channel prediction," in CVPR.
  99. [146]146. M. Noroozi and P. Favaro, "Unsupervised learning of visual representations by solving jigsaw puz- zles," in ECCV.
  100. [147]147. D. Kim, D. Cho, D. Yoo, and I. S. Kweon, "Learning image representations by completing damaged jigsaw puzzles," in WACV.
  101. [148]148. M. Noroozi, H. Pirsiavash, and P. Favaro, "Representation learning by learning to count," in ICCV.
  102. [149]149. P. Bojanowski and A. Joulin, "Unsupervised learning by predicting noise," in ICML, 2017.
  103. [150]150. S. Gidaris, P. Singh, and N. Komodakis, "Unsupervised representation learning by predicting image rotations," arXiv, 2018.
  104. [151]151. A. v. d. Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," arXiv, 2018.
  105. [152]152. O. Henaff, "Data-efficient image recognition with contrastive predictive coding," in ICML, 2020.
  106. [153]153. J. Donahue and K. Simonyan, "Large scale adversarial representation learning," in NeurIPS.
  107. [154]154. V. Dumoulin, I. Belghazi, B. Poole, O. Mastropietro, A. Lamb, M. Arjovsky, and A. Courville, "Ad- versarially learned inference," arXiv, 2016.
  108. [155]155. M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever, "Generative pretraining from pixels," in ICML, 2020.
  109. [156]156. H. Bao, L. Dong, S. Piao, and F. Wei, "Beit: Bert pre-training of image transformers," in International Conference on Learning Representations, 2021.
  110. [157]157. M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, "Emerging prop- erties in self-supervised vision transformers," in Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 9650-9660, 2021.
  111. [158]158. A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, "Zero- shot text-to-image generation," in International Conference on Machine Learning, pp. 8821-8831, PMLR, 2021.
  112. [159]159. Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, "Unsupervised feature learning via non-parametric instance discrimination," in CVPR.
  113. [160]160. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, "Masked autoencoders are scalable vision learners," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
  114. [161]161. Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, "Simmim: A simple framework for masked image modeling," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
  115. [163]163. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
  116. [166]166. G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," arXiv, 2015.
  117. [167]167. C. Zhuang, A. L. Zhai, and D. Yamins, "Local aggregation for unsupervised learning of visual em- beddings," in ICCV.
  118. [168]168. I. Misra and L. v. d. Maaten, "Self-supervised learning of pretext-invariant representations," in CVPR.
  119. [169]169. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, "Momentum contrast for unsupervised visual repre- sentation learning," in CVPR.
  120. [170]170. X. Chen, H. Fan, R. Girshick, and K. He, "Improved baselines with momentum contrastive learning," arXiv, 2020.
  121. [171]171. J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al., "Bootstrap your own latent: A new approach to self-supervised learn- ing," arXiv, 2020.
  122. [172]172. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations," in ICML.
  123. [173]173. M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, "Unsupervised learning of visual features by contrasting cluster assignments," arXiv, 2020.
  124. [174]174. P. Goyal, M. Caron, B. Lefaudeux, M. Xu, P. Wang, V. Pai, M. Singh, V. Liptchinsky, I. Misra, A. Joulin, et al., "Self-supervised pretraining of visual features in the wild," arXiv, 2021.
  125. [175]175. I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Dollár, "Designing network design spaces," in CVPR.
  126. [176]176. X. Chen and K. He, "Exploring simple siamese representation learning," in CVPR.
  127. [177]177. J. Li, P. Zhou, C. Xiong, and S. C. H. Hoi, "Prototypical contrastive learning of unsupervised repre- sentations," in ICLR, OpenReview.net, 2021.
  128. [178]178. L. Zhang, G.-J. Qi, L. Wang, and J. Luo, "Aet vs. aed: Unsupervised representation learning by auto-encoding transformations rather than data," in CVPR.
  129. [179]179. P. Bachman, R. D. Hjelm, and W. Buchwalter, "Learning representations by maximizing mutual information across views," arXiv, 2019.
  130. [180]180. X. Yan, I. Misra, A. Gupta, D. Ghadiyaram, and D. Mahajan, "Clusterfit: Improving generalization of visual representations," in CVPR, 2020.
  131. [182]182. T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton, "Big self-supervised models are strong semi-supervised learners," arXiv, 2020.
  132. [183]183. Y. Tian, D. Krishnan, and P. Isola, "Contrastive multiview coding," arXiv, 2019.
  133. [184]184. E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, "Randaugment: Practical data augmentation with no separate search," arXiv, 2019.
  134. [185]185. Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, "What makes for good views for contrastive learning," arXiv, 2020.
  135. [186]186. X. Chen, S. Xie, and K. He, "An empirical study of training self-supervised vision transformers," arXiv, 2021.
  136. [187]187. J. Mitrovic, B. McWilliams, J. C. Walker, L. H. Buesing, and C. Blundell, "Representation learning via invariant causal mechanisms," in ICLR.
  137. [188]188. Y. Tian, X. Chen, and S. Ganguli, "Understanding self-supervised learning dynamics without con- trastive pairs," in ICML.
  138. [189]189. Z. Xie, Y. Lin, Z. Yao, Z. Zhang, Q. Dai, Y. Cao, and H. Hu, "Self-supervised learning with swin transformers," arXiv, 2021.
  139. [190]190. Z. Li, Z. Chen, F. Yang, W. Li, Y. Zhu, C. Zhao, R. Deng, L. Wu, R. Zhao, M. Tang, et al., "Mst: Masked self-supervised transformer for visual representation," Advances in Neural Information Pro- cessing Systems, vol. 34, 2021.
  140. [191]191. H. Bao, L. Dong, S. Piao, and F. Wei, "BEit: BERT pre-training of image transformers," in Interna- tional Conference on Learning Representations, 2022.
  141. [194]194. Y. You, T. Chen, Z. Wang, and Y. Shen, "When does self-supervision help graph convolutional net- works?," in ICML.
  142. [195]195. W. Jin, T. Derr, H. Liu, Y. Wang, S. Wang, Z. Liu, and J. Tang, "Self-supervised learning on graphs: Deep insights and new direction," CoRR, 2020.
  143. [196]196. W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. S. Pande, and J. Leskovec, "Strategies for pre-training graph neural networks," in ICLR, 2020.
  144. [197]197. B. Perozzi, R. Al-Rfou, and S. Skiena, "Deepwalk: online learning of social representations," in ACM SIGKDD.
  145. [198]198. A. Grover and J. Leskovec, "node2vec: Scalable feature learning for networks," in ACM SIGKDD.
  146. [200]200. T. N. Kipf and M. Welling, "Variational graph auto-encoders," CoRR, 2016.
  147. [201]201. J. Qiu, Q. Chen, Y. Dong, J. Zhang, H. Yang, M. Ding, K. Wang, and J. Tang, "GCC: graph contrastive coding for graph neural network pre-training," in KDD.
  148. [203]203. Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, "Graph contrastive learning with augmenta- tions," in NeurIPS, 2020.
  149. [206]206. P. Velickovic, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm, "Deep graph infomax," in ICLR, 2019.
  150. [207]207. K. Hassani and A. H. K. Ahmadi, "Contrastive multi-view representation learning on graphs," in ICML.
  151. [208]208. Y. Jiao, Y. Xiong, J. Zhang, Y. Zhang, T. Zhang, and Y. Zhu, "Sub-graph contrast for scalable self- supervised graph representation learning," in ICDM.
  152. [209]209. K. Sun, Z. Lin, and Z. Zhu, "Multi-stage self-supervised learning for graph convolutional networks on graphs with few labeled nodes," in AAAI.
  153. [210]210. Y. Rong, Y. Bian, T. Xu, W. Xie, Y. Wei, W. Huang, and J. Huang, "Self-supervised graph transformer on large-scale molecular data," in NeurIPS, 2020.
  154. [215]215. S. Wan, S. Pan, J. Yang, and C. Gong, "Contrastive and generative graph convolutional networks for graph-based semi-supervised learning," in AAAI.
  155. [216]216. J. Zhang, H. Zhang, C. Xia, and L. Sun, "Graph-bert: Only attention is needed for learning graph representations," arXiv, 2020.
  156. [218]218. Z. Hu, Y. Dong, K. Wang, K. Chang, and Y. Sun, "GPT-GNN: generative pre-training of graph neural networks," in KDD.
  157. [219]219. G. Wang, H. Guo, A. Li, X. Liu, and Q. Yan, "Federated iot interaction vulnerability analysis," in 2023 IEEE 39th International Conference on Data Engineering (ICDE), IEEE, 2023.
  158. [220]220. W. L. Hamilton, Z. Ying, and J. Leskovec, "Inductive representation learning on large graphs," in NIPS, 2017.
  159. [221]221. D. Hwang, J. Park, S. Kwon, K. Kim, J. Ha, and H. J. Kim, "Self-supervised auxiliary learning with meta-paths for heterogeneous graphs," in NeurIPS, 2020.
  160. [222]222. F. Sun, J. Hoffmann, V. Verma, and J. Tang, "Infograph: Unsupervised and semi-supervised graph- level representation learning via mutual information maximization," in ICLR, 2020.
  161. [223]223. C. Park, D. Kim, J. Han, and H. Yu, "Unsupervised attributed multiplex network embedding," in AAAI.
  162. [224]224. Y. You, T. Chen, Y. Shen, and Z. Wang, "Graph contrastive learning automated," CoRR, 2021.
  163. [225]225. J. Zeng and P. Xie, "Contrastive self-supervised learning for graph classification," in AAAI.
  164. [226]226. M. Xu, H. Wang, B. Ni, H. Guo, and J. Tang, "Self-supervised graph-level representation learning with local and global structure," CoRR, 2021.
  165. [228]228. J. Cao, X. Lin, S. Guo, L. Liu, T. Liu, and B. Wang, "Bipartite graph embedding via mutual informa- tion maximization," in WSDM, 2021.
  166. [229]229. X. Wang, N. Liu, H. Han, and C. Shi, "Self-supervised heterogeneous graph neural network with co-contrastive learning," KDD, 2021.
  167. [230]230. D. Kim and A. Oh, "How to find your friendly neighborhood: Graph attention design with self- supervision," in ICLR, 2021.
  168. [231]231. M. Sun, J. Xing, H. Wang, B. Chen, and J. Zhou, "Mocl: Contrastive learning on molecular graphs with multi-level domain knowledge," CoRR, 2021.
  169. [232]232. S. Schneider, A. Baevski, R. Collobert, and M. Auli, "wav2vec: Unsupervised pre-training for speech recognition," in INTERSPEECH.
  170. [233]233. A. Baevski, S. Schneider, and M. Auli, "vq-wav2vec: Self-supervised learning of discrete speech representations," in ICLR.
  171. [234]234. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," in NeurIPS, 2020.
  172. [235]235. Y. Chung and J. R. Glass, "Generative pre-training for speech with autoregressive predictive coding," in ICASSP.
  173. [236]236. X. Song, G. Wang, Y. Huang, Z. Wu, D. Su, and H. Meng, "Speech-xlnet: Unsupervised acoustic model pretraining for self-attention networks," in INTERSPEECH.
  174. [237]237. Y. Chung, Y. Wang, W. Hsu, Y. Zhang, and R. J. Skerry-Ryan, "Semi-supervised training for improv- ing data efficiency in end-to-end speech synthesis," in ICASSP, 2019.
  175. [238]238. P. Denisov and N. T. Vu, "Pretrained semantic speech embeddings for end-to-end spoken language understanding via cross-modal teacher-student learning," in Interspeech, 2020.
  176. [239]239. Y.-A. Chung, C. Zhu, and M. Zeng, "SPLAT: Speech-language joint pre-training for spoken language understanding," in ACL, 2021.
  177. [241]241. Y.-S. Huang and Y.-H. Yang, "Pop music transformer: Beat-based modeling and generation of ex- pressive pop piano compositions," in Proceedings of the 28th ACM International Conference on Mul- timedia, pp. 1180-1188, 2020.
  178. [243]243. B. Fernando, H. Bilen, E. Gavves, and S. Gould, "Self-supervised video representation learning with odd-one-out networks," in CVPR.
  179. [244]244. I. Misra, C. L. Zitnick, and M. Hebert, "Shuffle and learn: unsupervised learning using temporal order verification," in ECCV, 2016.
  180. [245]245. D. Kim, D. Cho, and I. S. Kweon, "Self-supervised video representation learning with space-time cubic puzzles," in AAAI.
  181. [246]246. L. Tao, X. Wang, and T. Yamasaki, "Self-supervised video representation learning using inter-intra contrastive framework," in ACM Multimedia.
  182. [247]247. G. Lorre, J. Rabarisoa, A. Orcesi, S. Ainouz, and S. Canu, "Temporal contrastive pretraining for video action recognition," in WACV.
  183. [248]248. T. Yao, Y. Zhang, Z. Qiu, Y. Pan, and T. Mei, "Seco: Exploring sequence supervision for unsupervised representation learning," arXiv, 2020.
  184. [249]249. L. H. Li, M. Yatskar, D. Yin, C. Hsieh, and K. Chang, "Visualbert: A simple and performant baseline for vision and language," CoRR, 2019.
  185. [250]250. G. Li, N. Duan, Y. Fang, M. Gong, and D. Jiang, "Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training," in AAAI.
  186. [251]251. W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai, "VL-BERT: pre-training of generic visual- linguistic representations," in ICLR, 2020.
  187. [252]252. J. Lu, D. Batra, D. Parikh, and S. Lee, "Vilbert: Pretraining task-agnostic visiolinguistic representa- tions for vision-and-language tasks," in NeurIPS.
  188. [254]254. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., "Learning transferable visual models from natural language supervision," in Interna- tional Conference on Machine Learning, pp. 8748-8763, PMLR, 2021.
  189. [257]257. Z. Ren and Y. J. Lee, "Cross-domain self-supervised multi-task feature learning using synthetic im- agery," in CVPR.
  190. [258]258. Y. Tian, D. Krishnan, and P. Isola, "Contrastive multiview coding," in ECCV.
  191. [259]259. A. Zlotchevski, D. Drain, A. Svyatkovskiy, C. B. Clement, N. Sundaresan, and M. Tufano, "Ex- ploring and evaluating personalized models for code generation," in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1500-1508, 2022.
  192. [263]263. Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, "Uniter: Univer- sal image-text representation learning," in European conference on computer vision, pp. 104-120, Springer, 2020.
  193. [264]264. X. Zhu, J. Zhu, H. Li, X. Wu, H. Li, X. Wang, and J. Dai, "Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16804-16815, 2022.
  194. [267]267. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter," CoRR, 2019.
  195. [268]268. D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits, "Is bert really robust? a strong baseline for natural language attack on text classification and entailment," in AAAI.
  196. [269]269. D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits, "Is BERT really robust? A strong baseline for natural language attack on text classification and entailment," in AAAI.
  197. [270]270. Y. Zang, F. Qi, C. Yang, Z. Liu, M. Zhang, Q. Liu, and M. Sun, "Word-level textual adversarial attacking as combinatorial optimization," in ACL.
  198. [271]271. E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh, "Universal adversarial triggers for at- tacking and analyzing NLP," in EMNLP-IJCNLP.
  199. [272]272. K. Kurita, P. Michel, and G. Neubig, "Weight poisoning attacks on pretrained models," in ACL.
  200. [273]273. R. Schuster, T. Schuster, Y. Meri, and V. Shmatikov, "Humpty dumpty: Controlling word meanings via corpus poisoning," in IEEE Symposium on Security and Privacy, 2020.

200 of 407 shown

Authors
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., Peng, H., Li, J., Wu, J., Liu, Z., Xie, P., Xiong, C., Pei, J., Yu, P. S., & Sun, L.
Affiliations
  • Michigan State University (US)
  • Beihang University (CN)
  • Lehigh University (US)
  • Macquarie University (AU)
  • Nanyang Technological University (SG)
  • University of California San Diego (US)
  • Salesforce (United States) (US)
  • Duke University (US)
  • University of Illinois Chicago (US)
Corresponding author
  • Zhou, Ce
  • Li, Qian
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
16
Issue
12
Pages
9851-9915
ISSN
1868-8071 (ISSN-L)
DOI
10.48550/arxiv.2302.09419
arXiv
2302.09419
OpenAlex
W4404658388
Semantic Scholar
3599a236f285af48782fc30b1341d13ec7320735
Cited by
574
References
319
Citations by year
2026: 88 · 2025: 252 · 2024: 150 · 2023: 83 · 2022: 1
FWCI
71.85
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • International Journal of Machine Learning and Cybernetics (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DR-NTU (Nanyang Technological University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
BERT, ChatGPT, GPT-4, large language models, multimodal foundation models, graph neural networks, model efficiency, privacy and security, logical reasoning, cross-domain learning, artificial general intelligence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review