Self-supervised Regularization for Text Classification
Open accessAbstract
Text classification is a widely studied problem and has broad applications. In many real-world problems, the number of texts for training classification models is limited, which renders these models prone to overfitting. To address this problem, we propose SSL-Reg, a data-dependent regularization approach based on self-supervised learning (SSL). SSL (Devlin et al., 2019a) is an unsupervised learning approach that defines auxiliary tasks on input data without using any human-provided labels and learns data representations by solving these auxiliary tasks. In SSL-Reg, a supervised classification task and an unsupervised SSL task are performed simultaneously. The SSL task is unsupervised, which is defined purely on input texts without using any human- provided labels. Training a model using an SSL task can prevent the model from being overfitted to a limited number of class labels in the classification task. Experiments on 17 text classification datasets demonstrate the effectiveness of our proposed method. Code is available at https://github.com/UCSD-AI4H/SSReg.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Zhou et al. used SSL regularization in NLP for text classification, with prediction and self-supervision tasks on encoded text.
“Zhou et al. [34] used the SSL regularization technique in NLP for text classification. They defined text classification as a key concept in NLP. Authors provided training texts that were encoded using a text encoder as input. In the encoded text, the authors defined two tasks: prediction and self-supervision.”
From Self-supervised learning: A succinct review (2023) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Momentum Contrast for Unsupervised Visual Representation LearningPreprint
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. B. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.05722 · Cited by 16,250 · see citing works
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
Nonlinear Dimensionality Reduction by Locally Linear Embedding
Roweis, S. T., & Saul, L. K. (2000) · Science · doi:10.1126/science.290.5500.2323 · Cited by 14,965 · see citing works
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · see citing works
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · see citing works
Zhang, R., Isola, P., & Efros, A. A. (2016) · European Conference on Computer Vision · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,803 · see citing works
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710 · see citing works
Unsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664 · see citing works
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601 · see citing works
Supervised Contrastive Learning
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., & Krishnan, D. (2020) · arXiv · doi:10.48550/arxiv.2004.11362 · Cited by 3,569 · see citing works
Unsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477 · see citing works
Recurrent Convolutional Neural Networks for Text Classification
Lai, S., Xu, L., Liu, K., & Zhao, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9513 · Cited by 2,365 · see citing works
Image-Based Recommendations on Styles and Substitutes
McAuley, J., Targett, C., Shi, Q., & van den Hengel, A. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767755 · Cited by 2,348 · see citing works
EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
Wei, J., & Zou, K. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1670 · Cited by 2,242 · see citing works
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103 · see citing works
Deep Learning--based Text Classification
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad-Khasmakhi, N., Asgari-Chenaghlu, M., & Gao, J. (2021) · ACM Computing Surveys · doi:10.1145/3439726 · Cited by 1,967 · see citing works
Fixing Weight Decay Regularization in Adam
Loshchilov, I., & Hutter, F. (2018) · Cited by 1,297 · see citing works
Recurrent Neural Network for Text Classification with Multi-Task LearningPreprint
Liu, P., Qiu, X., & Huang, X. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.05101 · Cited by 1,209 · see citing works
ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding
Sun, Y., Wang, S., Li, Y., Feng, S., Tian, H., Wu, H., & Wang, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6428 · Cited by 766 · see citing works
Luan, Y., He, L., Mari Ostendorf, E., & Hajishirzi, H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1360 · Cited by 714 · see citing works
A C-LSTM Neural Network for Text ClassificationPreprint
Zhou, C., Sun, C., Liu, Z., & Lau, F. C. M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08630 · Cited by 666 · see citing works
Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.-F., Wang, W. Y., & Zhang, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00679 · Cited by 522 · see citing works
CURL: Contrastive Unsupervised Representations for Reinforcement LearningPreprint
Srinivas, A., Laskin, M., & Abbeel, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.04136 · Cited by 521 · see citing works
Learning Correspondence From the Cycle-Consistency of Time
Wang, X., Jabri, A., & Efros, A. A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00267 · Cited by 452 · see citing works
Sample-Efficient Deep Learning for COVID-19 Diagnosis Based on CT ScansPreprint
He, X., Yang, X., Zhang, S., Zhao, J., Zhang, Y., Xing, E., & Xie, P. (2020) · medRxiv · doi:10.1101/2020.04.13.20063941 · Cited by 434 · see citing works
Self-citationBird, S., & Loper, E. (2004) · doi:10.3115/1219044.1219075 · Cited by 404 · see citing works
Deep Learning Based Text Classification: A Comprehensive ReviewPreprint
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Asgari-Chenaghlu, M., & Gao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03705 · Cited by 385 · see citing works
(2019) · neural information processing systems · Cited by 350 · see citing works
Combining Knowledge with Deep Convolutional Neural Networks for Short Text Classification
Wang, J., Wang, Z., Zhang, D., & Yan, J. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/406 · Cited by 313 · see citing works
Text Classification and Classifiers:A Survey
Korde, V. (2012) · International Journal of Artificial Intelligence & Applications · doi:10.5121/ijaia.2012.3208 · Cited by 250 · see citing works
Measuring the Evolution of a Scientific Field through Citation Frames
Jurgens, D., Kumar, S., Hoover, R., McFarland, D., & Jurafsky, D. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00028 · Cited by 212 · see citing works
CERT: Contrastive Self-supervised Learning for Language UnderstandingPreprint
Fang, H., & Xie, P. (2020) · doi:10.36227/techrxiv.12308378.v1 · Cited by 211 · see citing works
Self-citationSemEval-2019 Task 4: Hyperpartisan News Detection
Kiesel, J., Mestre, M., Shukla, R., Vincent, E., Adineh, P., Corney, D., Stein, B., & Potthast, M. (2019) · Proceedings of the 13th International Workshop on Semantic Evaluation · doi:10.18653/v1/s19-2145 · Cited by 193 · see citing works
Contrastive Self-supervised Learning for Graph Classification
Zeng, J., & Xie, P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17293 · Cited by 151 · see citing works
Self-citationChemProt-3.0: a global chemical biology diseases mapping
Kringelum, J. V., Kjærulff, S. K., Brunak, S., Lund, O., Oprea, T. I., & Taboureau, O. (2016) · Database · doi:10.1093/database/bav123 · Cited by 138 · see citing works
Improvements to Context Based Self-Supervised Learning
Mundhenk, T. N., Ho, D., & Chen, B. Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00973 · Cited by 122 · see citing works
Neural Networks for Joint Sentence Classification in Medical Paper Abstracts
Dernoncourt, F., Lee, J. Y., & Peter Szolovits, B. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2110 · Cited by 70 · see citing works
Pathological Visual Question AnsweringPreprint
He, X., Cai, Z., Wei, W., Zhang, Y., Mou, L., Xing, E. P., & Xie, P. (2020) · doi:10.36227/techrxiv.13127537 · Cited by 70 · see citing works
Self-citationContrastive Self-Supervised Learning for Commonsense Reasoning
Klein, T., & Nabi, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.671 · Cited by 58 · see citing works
Pathological Visual Question AnsweringPreprint
He, X., Cai, Z., Wei, W., Zhang, Y., Mou, L., Xing, E. P., & Xie, P. (2020) · doi:10.36227/techrxiv.13127537.v1 · Cited by 54 · see citing works
Self-citationJoint-task Self-supervised Learning for Temporal CorrespondencePreprint
Li, X., Liu, S., Mello, S. D., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11895 · Cited by 53 · see citing works
Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining ParadigmsPreprint
Yang, X., He, X., Liang, Y., Yang, Y., Zhang, S., & Xie, P. (2020) · doi:10.36227/techrxiv.12502298.v1 · Cited by 34 · see citing works
Self-citationImportance-Aware Learning for Neural Headline Editing
Wu, Q., Li, L., Zhou, H., Zeng, Y., & Yu, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6467 · Cited by 20 · see citing works
Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining ParadigmsPreprint
Yang, X., He, X., Liang, Y., Yang, Y., Zhang, S., & Xie, P. (2020) · doi:10.36227/techrxiv.12502298 · Cited by 19 · see citing works
Self-citationContrastive Self-supervised Learning for Graph ClassificationPreprint
Zeng, J., & Xie, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05923 · Cited by 9 · see citing works
Self-citationImportance-Aware Learning for Neural Headline EditingPreprint
Wu, Q., Li, L., Zhou, H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01114 · Cited by 7 · see citing works
Pathological Visual Question AnsweringPreprint
He, X., Cai, Z., Wei, W., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12435
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 3
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly op- timized BERT pretraining approach. arXiv: 1907.11692. · arXiv
- Yu Sun, Xiaolong Wang, Liu Zhuang, John Miller, Moritz Hardt, and Alexei A. Efros. 2020. Test-time training with self-supervision for generalization under distribution shifts. In ICML.
- Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NeurIPS.
Cited by
Sorted by citations
Self-supervised learning: A succinct review
Rani, V., Nabi, S. T., Kumar, M., Mittal, A., & Kumar, K. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-09884-2 · Cited by 295 · see citing works
- Authors
- Zhou, M., Li, Z., & Xie, P.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Northeastern University (US)
- University of California San Diego (US)
- Corresponding author
- Xie, Pengtao
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 641-656
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3180895563
- Cited by
- 14
- References
- 68
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 2 · 2023: 4 · 2021: 1
- FWCI
- 0.62
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- text classification, self-supervised learning, regularization, overfitting, limited training data, auxiliary tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
