The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
Open accessAbstract
Recent advancements of sequential deep learning models such as Transformer and BERT have significantly facilitated the sequential recommendation. However, according to our study, the distribution of item embeddings generated by these models tends to degenerate into an anisotropic shape, which may result in high semantic similarities among embeddings. In this paper, both empirical and theoretical investigations of this representation degeneration problem are first provided, based on which a novel recommender model DuoRec is proposed to improve the item embeddings distribution. Specifically, in light of the uniformity property of contrastive learning, a contrastive regularization is designed for DuoRec to reshape the distribution of sequence representations. Given the convention that the recommendation task is performed by measuring the similarity between sequence representations and item embeddings in the same space via dot product, the regularization can be implicitly applied to the item embedding distribution. Existing contrastive learning methods mainly rely on data level augmentation for user-item interaction sequences through item cropping, masking, or reordering and can hardly provide semantically consistent augmentation samples. In DuoRec, a model-level augmentation is proposed based on Dropout to enable better semantic preserving. Furthermore, a novel sampling strategy is developed, where sequences having the same target item are chosen hard positive samples. Extensive experiments conducted on five datasets demonstrate the superior performance of the proposed DuoRec model compared with baseline methods. Visualization results of the learned representations validate that DuoRec can largely alleviate the representation degeneration problem.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
(2014) · Cited by 33,419
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857
Item-based collaborative filtering recommendation algorithms
Sarwar, B., Karypis, G., Konstan, J., & Riedl, J. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.372071 · Cited by 9,135
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
BPR: Bayesian Personalized Ranking from Implicit Feedback
Rendle, S., Freudenthaler, C., Gantner, Z., & Schmidt-Thieme, L. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.2618 · Cited by 4,242
The MovieLens Datasets: History and Context
Harper, F. M., & Konstan, J. A. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2827872 · Cited by 3,907
Exploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01549 · Cited by 3,725
Supervised Contrastive Learning
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., & Krishnan, D. (2020) · arXiv · doi:10.48550/arxiv.2004.11362 · Cited by 3,569
Self-Attentive Sequential Recommendation
Kang, W.-C., & McAuley, J. (2018) · 2018 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2018.00035 · Cited by 3,011
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851
Factorizing personalized Markov chains for next-basket recommendation
Rendle, S., Freudenthaler, C., & Schmidt-Thieme, L. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772773 · Cited by 2,692
BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., & Jiang, P. (2019) · Proceedings of the 28th ACM International Conference on Information and Knowledge Management · doi:10.1145/3357384.3357895 · Cited by 2,558
Image-Based Recommendations on Styles and Substitutes
McAuley, J., Targett, C., Shi, Q., & van den Hengel, A. (2015) · Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/2766462.2767755 · Cited by 2,280
Learning deep representations by mutual information estimation and maximization
Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., & Bengio, Y. (2018) · arXiv · doi:10.48550/arxiv.1808.06670 · Cited by 1,991
Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding
Tang, J., & Wang, K. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159656 · Cited by 1,975
Data-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S. M. A., & Oord, A. V. D. (2019) · arXiv · doi:10.48550/arxiv.1905.09272 · Cited by 1,014
Ethayarajh, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1006 · Cited by 841
Contrastive Learning for Sequential Recommendation
Xie, X., Sun, F., Liu, Z., Wu, S., Gao, J., Zhang, J., Ding, B., & Cui, B. (2022) · 2022 IEEE 38th International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00099 · Cited by 663
On the Sentence Embeddings from Pre-trained Language Models
Li, B., Zhou, H., He, J., Wang, M., Yang, Y., & Li, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.733 · Cited by 593
Gutmann, M. U., & Hyvärinen, A. (2012) · Edinburgh Research Explorer (University of Edinburgh) · doi:10.5555/2503308.2188396 · Cited by 581
Self-Supervised Hypergraph Convolutional Networks for Session-based Recommendation
Xia, X., Yin, H., Yu, J., Wang, Q., Cui, L., & Zhang, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16578 · Cited by 570
Self-citationSelf-Supervised Multi-Channel Hypergraph Convolutional Network for Social Recommendation
Yu, J., Yin, H., Li, J., Wang, Q., Hung, N. Q. V., & Zhang, X. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449844 · Cited by 519
Self-citationOn Sampled Metrics for Item Recommendation
Krichene, W., & Rendle, S. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403226 · Cited by 495
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Yan, Y., Li, R., Wang, S., Zhang, F., Wu, W., & Xu, W. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.393 · Cited by 474
Learning word embeddings efficiently with noise-contrastive estimation
Mnih, A., & Kavukcuoglu, K. (2013) · neural information processing systems · Cited by 446
Representation Degeneration Problem in Training Natural Language Generation ModelsPreprint
Gao, J., He, D., Tan, X., Qin, T., Wang, L.-W., & Liu, T.-Y. (2019) · International Conference on Learning Representations · doi:10.48550/arxiv.1907.12009 · Cited by 443
Exploring Simple Siamese Representation Learning
Chen, X., & He, K. (2020) · arXiv · doi:10.48550/arxiv.2011.10566 · Cited by 381
Self-supervised Co-training for Video Representation Learning
Han, T., Xie, W., & Zisserman, A. (2020) · arXiv · doi:10.48550/arxiv.2010.09709 · Cited by 292
Learning to Diversify for Single Domain Generalization
Wang, Z., Luo, Y., Qiu, R., Huang, Z., & Baktashmotlagh, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00087 · Cited by 284
Self-citationSelf-supervised Learning for Large-scale Item Recommendations
Yao, T., Yi, X., Cheng, D. Z., Yu, F., Chen, T., Menon, A., Hong, L., Chi, E. H., Tjoa, S., Kang, J. (., et al. (2021) · Proceedings of the 30th ACM International Conference on Information & Knowledge Management · doi:10.1145/3459637.3481952 · Cited by 253
Socially-Aware Self-Supervised Tri-Training for Recommendation
Yu, J., Yin, H., Gao, M., Xia, X., Zhang, X., & Viet Hung, N. Q. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467340 · Cited by 228
Self-citationDisentangled Self-Supervision in Sequential Recommenders
Ma, J., Zhou, C., Yang, H., Cui, P., Wang, X., & Zhu, W. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403091 · Cited by 225
CLEAR: Contrastive Learning for Sentence Representation
Wu, Z., Wang, S., Gu, J., Khabsa, M., Sun, F., & Ma, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15466 · Cited by 214
Memory-Augmented Dense Predictive Coding for Video Representation Learning
Han, T., Xie, W., & Zisserman, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58580-8_19 · Cited by 202
Exploiting Cross-session Information for Session-based Recommendation with Graph Neural Networks
Qiu, R., Huang, Z. H., Li, J., & Yin, H. (2020) · ACM Transactions on Information Systems · doi:10.1145/3382764 · Cited by 178
Self-citationCOCO-LM: Correcting and Contrasting Text Sequences for Language Model PretrainingPreprint
Yu, M., Xiong, C., Bajaj, P., Tiwary, S., Bennett, P. N., Han, J., & Song, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08473 · Cited by 143
Lightweight Self-Attentive Sequential Recommendation
Li, Y., Chen, T., Zhang, P.-F., & Yin, H. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482448 · Cited by 119
Self-citationGAG: Global Attributed Graph Neural Network for Streaming Session-based Recommendation
Qiu, R., Yin, H., Huang, Z. H., & Chen, T. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401109 · Cited by 117
Self-citationImproving Neural Language Generation with Spectrum Control
Wang, L., Huang, J., Huang, K., Hu, Z., Wang, G., & Gu, Q. (2020) · International Conference on Learning Representations · Cited by 105
Discovering Collaborative Signals for Next POI Recommendation with Iterative Seq2Graph Augmentation
Li, Y., Chen, T., Luo, Y., Yin, H., & Huang, Z. H. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/206 · Cited by 91
Self-citationDiscrete Deep Learning for Fast Content-Aware Recommendation
Zhang, Y., Yin, H., Huang, Z., Du, X.-Z., Yang, G., & Lian, D.-F. (2018) · Web Search and Data Mining · doi:10.1145/3159652.3159688 · Cited by 80
Self-citationMemory Augmented Multi-Instance Contrastive Predictive Coding for Sequential Recommendation
Qiu, R., Huang, Z. H., & Yin, H. (2021) · Proceedings · doi:10.1109/icdm51629.2021.00063 · Cited by 77
Self-citationComputing Crowd Consensus with Partial Agreement
Nguyen, Q. V. H., Viet, H. H., Nguyên, T. T., Weidlich, M., Yin, H., & Zhou, X. (2017) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2750683 · Cited by 75
Self-citationSequence-Aware Factorization Machines for Temporal Predictive Analytics
Chen, T., Yin, H., Nguyen, Q. V. H., Peng, W.-C., Li, X., & Zhou, X. (2020) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde48307.2020.00125 · Cited by 74
Self-citationToo Much in Common: Shifting of Embeddings in Transformer Language Models and its Implications
Biś, D., Podkorytov, M., & Liu, X. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.403 · Cited by 72
Exploiting Positional Information for Session-Based Recommendation
Qiu, R., Huang, Z. H., Chen, T., & Yin, H. (2021) · ACM Transactions on Information Systems · doi:10.1145/3473339 · Cited by 40
Self-citation
Cited by
Are Graph Augmentations Necessary?: Simple Graph Contrastive Learning for Recommendation
Yu, J., Yin, H., Xia, X., Chen, T., Cui, L., & Nguyen, Q. V. H. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531937 · Cited by 664
Self-citationBao, K., Zhang, J., Zhang, Y., Wang, W., Feng, F., & He, X. (2023) · Proceedings of the 17th ACM Conference on Recommender Systems · doi:10.1145/3604915.3608857 · Cited by 469
Self-supervised learning for recommender systems: A survey
Yu, J., Yin, H., Xia, X., Chen, T., Li, J., & Huang, Z. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3282907 · Cited by 413
Self-citationXSimGCL: Towards Extremely Simple Graph Contrastive Learning for Recommendation
Yu, J., Xia, X., Chen, T., Cui, L., Hung, N. Q. V., & Yin, H. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3288135 · Cited by 322
Self-citation
Metadata
- Authors
- Qiu, R., Huang, Z., Yin, H., & Wang, Z.
- Venue
- Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining
- Publisher
- ACM
- Published
- 2022-02-11
- Pages
- 813-823
- OpenAlex
- W3206127589
- Cited by
- 460
- References
- 50
- FWCI
- 84.02
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.