CoChat
Sign Up
Cited by 53Topic ModelingSentiment Analysis and Opinion MiningMultimodal Machine Learning Applications

Contrastive Learning Models for Sentence Representations

Open access
Xu, Lingling, Xie, Haoran, Li, Zongxi, Wang, Fu Lee, Wang, Weiming, Li, Qing·2023·ACM Transactions on Intelligent Systems and Technology
DOI 10.1145/3593590
Open this paper in CoChat — free accountOpen PDF

Abstract

Sentence representation learning is a crucial task in natural language processing, as the quality of learned representations directly influences downstream tasks, such as sentence classification and sentiment analysis. Transformer-based pretrained language models such as bidirectional encoder representations from transformers (BERT) have been extensively applied to various natural language processing tasks, and have exhibited moderately good performance. However, the anisotropy of the learned embedding space prevents BERT sentence embeddings from achieving good results in the semantic textual similarity tasks. It has been shown that contrastive learning can alleviate the anisotropy problem and significantly improve sentence representation performance. Therefore, there has been a surge in the development of models that utilize contrastive learning to fine-tune BERT-like pretrained language models to learn sentence representations. But no systematic review of contrastive learning models for sentence representations has been conducted. To fill this gap, this article summarizes and categorizes the contrastive learning based sentence representation models, common evaluation tasks for assessing the quality of learned representations, and future research directions. Furthermore, we select several representative models for exhaustive experiments to illustrate the quantitative improvement of various strategies on sentence representations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3593590

Cites

127 of 127 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 Task 6: A pilot on seman- tic textual similarity. In Proceedings of the 1st Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the Main Conference and the Shared Task (*SEM'12), and Volume 2: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval'12) . 385-393.
  2. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In Proceedings of the 2nd Joint Conference on Lexical and Computational Semantics- Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity (*SEM'13) . 32-43. https://aclanthology.org/S13-1004 .
  3. Holger Schwenk, Loïc Barrault, Alexis Conneau, Douwe Kiela, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP'17) .
  4. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In Proceedings of the International Conference on Learning Representations (ICLR'17) .
  5. Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive tension. In Proceedings of the International Conference on Learning Representa- tions (ICLR'21) . https://openreview.net/forum?id=Ov _ sMNau-PF .
  6. Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. 2021. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR'21) . 9516-9525.
  7. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning (ICML'20) . 1597-1607.
  8. Tsai-Shien Chen, Wei-Chih Hung, Hung-Yu Tseng, Shao-Yi Chien, and Ming-Hsuan Yang. 2022. Incremental false negative detection for contrastive learning. In Proceedings of the International Conference on Learning Representations (ICLR'22) . https://openreview.net/forum?id=dDjSKKA5TP1 .
  9. Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. 2020. Debiased con- trastive learning. Advances in Neural Information Processing Systems 33 (2020), 8765-8775.
  10. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text en- coders as discriminators rather than generators. In Proceedings of the International Conference on Learning Represen- tations (ICLR'20) . https://openreview.net/pdf?id=r1xMH1BtvB .
  11. Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC'18) . https: //aclanthology.org/L18-1269 .
  12. Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljacic. 2022. Equivariant self-supervised learning: Encouraging equivariance in representations. In Pro- ceedings of the International Conference on Learning Representations (ICLR'22) . https://openreview.net/forum?id= gKLAAfiytI .
  13. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Pro- ceedings of the 3rd International Workshop on Paraphrasing (IWP'05) . https://aclanthology.org/I05-5002 .
  14. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation degeneration problem in train- ing natural language generation models. In Proceedings of the International Conference on Learning Representations (ICLR'19) . https://openreview.net/forum?id=SkEYojRqtm .
  15. Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. 2022. The power of contrast for feature learning: A theoretical analysis. In Proceedings of the International Conference on Learning Representations (ICLR'22) . https://openreview.net/pdf?id=yBY V UDj7yF .
  16. Yuxin Jiang and Wei Wang. 2022. Deep continuous prompt for contrastive learning of sentence embeddings. arXiv preprint arXiv:2203.06875 (2022). · arXiv
  17. Jeffrey Ling, Nicholas FitzGerald, Zifei Shan, Livio Baldini Soares, Thibault Févry, David Weiss, and Tom Kwiatkowski. 2020. Learning cross-context entity representations from text. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepLANLP'21) . 241-247. https://openreview.net/forum?id=HygwvC4tPH .
  18. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In Proceedings of the International Conference on Learning Representations (ICLR'18) . https://openreview.net/forum?id= rJvJXZb0W .
  19. J. MacQueen. 1967. Classification and analysis of multivariate observations. In Proceedings of the 5th Berkeley Sym- posium on Mathematical Statistics and Probability, Volume 1: Statistics . 281-297.
  20. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC'14) . 216-223. http://w w w.lrec-conf.org/ proceedings/lrec2014/pdf/363 _ Paper.pdf.
  21. Nils Reimers, Philip Beyer, and Iryna Gurevych. 2016. Task-oriented intrinsic evaluation of semantic textual similar- ity. In Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers (COLING'16) . 87-96.
  22. Abhishek Sinha, Kumar Ayush, Jiaming Song, Burak Uzkent, Hongxia Jin, and Stefano Ermon. 2021. Negative data augmentation. In Proceedings of the International Conference on Learning Representations (ICLR'21) . https: //openreview.net/forum?id=Ovp8dvB8IBH .
  23. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15, 1 (2014), 1929- 1958.
  24. Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the International Conference on Machine Learning (ICML'20) . 9929- 9939.
  25. Zixin Wen and Yuanzhi Li. 2021. Toward understanding the feature learning process of self-supervised contrastive learning. In Proceedings of the International Conference on Machine Learning (ICML'21) . 11112-11122.
  26. Mike Wu, Milan Mosse, Chengxu Zhuang, Daniel Yamins, and Noah Goodman. 2021. Conditional negative sam- pling for contrastive learning of visual representations. In Proceedings of the International Conference on Learning Representations (ICLR'21) . https://openreview.net/forum?id=v8b3e5jN66j .
  27. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In Proceedings of the International Conference on Learning Representations (ICLR'21) . https://openreview.net/forum?id=zeFrfgyZln .
  28. Ikuya Yamada, Hiroyuki Shindo, and Yoshiyasu Takefuji. 2018. Representation learning of entities and documents from knowledge base descriptions. In Proceedings of the 27th International Conference on Computational Linguistics . 190-201. https://aclanthology.org/C18-1016 .
  29. Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021. Barlow Twins: Self-supervised learning via redundancy reduction. In Proceedings of the International Conference on Machine Learning (ICML'21) . 12310-12320.

Cited by

Sorted by citations

Related

Authors
Xu, L., Xie, H., Li, Z., Wang, F. L., Wang, W., & Li, Q.
Affiliations
  • Hong Kong Metropolitan University (HK)
  • Lingnan University (HK)
  • Hong Kong Polytechnic University (HK)
ORCID
Venue
ACM Transactions on Intelligent Systems and Technology
Publisher
Association for Computing Machinery (ACM)
Published
2023-06-15
Volume
14
Issue
4
Pages
1-34
ISSN
  • 2157-6904 (ISSN-L)
  • 2157-6912
DOI
10.1145/3593590
OpenAlex
W4367680667
Cited by
50
References
119
Citations by year
2026: 9 · 2025: 23 · 2024: 16 · 2023: 2
FWCI
4.71
Type
journal article
Language
en
Open access
Licence
Locations
ACM Transactions on Intelligent Systems and Technology (publishedVersion, cc-by-nc-sa)
Funders
  • Research Grants Council of the Hong Kong Special Administrative Region, China (UGC/FDS16/E01/19)
  • Lam Woo Research Fund of Lingnan University (LWP20019)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
contrastive learning, sentence representation learning, BERT, semantic textual similarity, pre-trained language models, sentence embeddings, sentence classification, sentiment analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review