BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
Abstract
Modeling users' dynamic preferences from their historical behaviors is challenging and crucial for recommendation systems. Previous methods employ sequential neural networks to encode users' historical interactions from left to right into hidden representations for making recommendations. Despite their effectiveness, we argue that such left-to-right unidirectional models are sub-optimal due to the limitations including: \begin enumerate* [label=series\itshape\alph*\upshape)] \item unidirectional architectures restrict the power of hidden representation in users' behavior sequences; \item they often assume a rigidly ordered sequence which is not always practical. \end enumerate* To address these limitations, we proposed a sequential recommendation model called BERT4Rec, which employs the deep bidirectional self-attention to model user behavior sequences. To avoid the information leakage and efficiently train the bidirectional model, we adopt the Cloze objective to sequential recommendation, predicting the random masked items in the sequence by jointly conditioning on their left and right context. In this way, we learn a bidirectional representation model to make recommendations by allowing each item in user historical behaviors to fuse information from both left and right sides. Extensive experiments on four benchmark datasets show that our model outperforms various state-of-the-art sequential models consistently.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For recommendation, we perform an ablation study and compare Titans against a BERT4Rec baseline.
“As with MLM, we perform an ablation study, and compare Titans against a BERT4Rec (Sun et al., 2019) baseline.”
From Titans Revisited: A Lightweight Reimplementation and Critical Analysis of a Test-Time Memory Model (2025) · cites this paper as Sun et al., 2019
Language models like BERT have advanced recommender systems by enabling better understanding of natural language features and user sequential behaviors.
“Furthermore, the rise of language models like BERT has further advanced recommender systems by enabling a better understanding of both natural language features and user sequential behaviors [22].”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [22]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Geng, S., Liu, S., Fu, Z., Ge, Y., & Zhang, Y. (2022) · Proceedings of the 16th ACM Conference on Recommender Systems · doi:10.1145/3523227.3546767 · Cited by 556 · see citing works
Recommender Systems in the Era of Large Language Models (LLMs)
Zhao, Z., Fan, W., Li, J., Liu, Y., Mei, X., Wang, Y., Wen, Z., Wang, F., Zhao, X., Tang, J., et al. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3392335 · Cited by 481 · see citing works
Self-supervised learning for recommender systems: A survey
Yu, J., Yin, H., Xia, X., Chen, T., Li, J., & Huang, Z. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3282907 · Cited by 413 · see citing works
When large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367 · see citing works
How Can Recommender Systems Benefit from Large Language Models: A Survey
Lin, J., Dai, X., Xi, Y., Liu, W., Chen, B., Zhang, H., Liu, Y., Wu, C., Li, X., Zhu, C., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3678004 · Cited by 242 · see citing works
Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited
Yuan, Z., Yuan, F., Song, Y., Li, Y., Fu, J., Yang, F., Pan, Y., & Ni, Y. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591932 · Cited by 200 · see citing works
Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders
Hou, Y., He, Z., McAuley, J., & Zhao, W. X. (2023) · Proceedings of the ACM Web Conference 2023 · doi:10.1145/3543507.3583434 · Cited by 172 · see citing works
“In the pre-training stage, we optimize the parameters in the code embedding matrices and Transformer, while the BERT encoder is fxed and PQ is performed independent of pre-training.”
BackgroundMethodInfluentialData-efficient Fine-tuning for LLM-based Recommendation
Lin, X., Wang, W., Li, Y., Yang, S., Feng, F., Wei, Y., & Chua, T.-S. (2024) · arXiv · doi:10.48550/arxiv.2401.17197 · Cited by 165 · see citing works
Self-citationAdapting Large Language Models by Integrating Collaborative Semantics for Recommendation
Zheng, B., Hou, Y., Lu, H., Chen, Y., Zhao, W. X., Chen, M., & Wen, J.-R. (2024) · 2024 IEEE 40th International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00118 · Cited by 143 · see citing works
U-BERT: Pre-training User Representations for Improved Recommendation
Qiu, Z., Wu, X., Gao, J., & Fan, W. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16557 · Cited by 112 · see citing works
Membership Inference Attacks Against Sequential Recommender Systems
Zhu, Z., Wu, C., Fan, R., Lian, D., & Chen, E. (2023) · Proceedings of the ACM Web Conference 2023 · doi:10.1145/3543507.3583447 · Cited by 24 · see citing works
COSMO: A Large-Scale E-commerce Common Sense Knowledge Generation and Serving System at Amazon
Yu, C., Liu, X., Maia, J., Li, Y., Cao, T., Gao, Y., Song, Y., Goutam, R., Zhang, H., Yin, B., et al. (2024) · Companion of the 2024 International Conference on Management of Data · doi:10.1145/3626246.3653398 · Cited by 13 · see citing works
Zhao, P., Xiao, K., Zhang, Y., Bian, K., & Yan, W. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/290 · Cited by 7 · see citing works
On the Cost of Model-Serving Frameworks: An Experimental Evaluation
De Rosa, P., Bromberg, Y.-D., Felber, P., Mvondo, D., & Schiavoni, V. (2024) · 2024 IEEE International Conference on Cloud Engineering (IC2E) · doi:10.1109/ic2e61754.2024.00032 · Cited by 6 · see citing works
“Common applications of ML include price predictions of financial assets [36], [64], recommendation systems [79], [83], object detection [39], [42], speech recognition [38], [43], and more.”
A Slice and Dice Approach to Accelerate Compound Sparse Attention on GPU
Li, H., Choi, J., & Ahn, J. H. (2022) · 2022 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc55918.2022.00019 · Cited by 3 · see citing works
Titans Revisited: A Lightweight Reimplementation and Critical Analysis of a Test-Time Memory Model
Di Nepi, G., Siciliano, F., & Silvestri, F. (2025) · arXiv · doi:10.48550/arxiv.2510.09551
“Recommendation On MovieLens 1M, Titans failed to outperform the BERT4Rec baseline.”
Influential
- Authors
- Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., & Jiang, P.
- Affiliations
- Alibaba Group (China) (CN)
- Venue
- Proceedings of the 28th ACM International Conference on Information and Knowledge Management
- Publisher
- ACM
- Published
- 2019-11-03
- Pages
- 1441-1450
- ISSN
- 2155-0751 (ISSN-L)
- OpenAlex
- W2984100107
- Cited by
- 2,558
- References
- 3
- Citations by year
- 2026: 522 · 2025: 599 · 2024: 506 · 2023: 402 · 2022: 283 · 2021: 182 · 2020: 58 · 2019: 6
- FWCI
- 105.35
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Information and Knowledge Management (CIKM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Keywords
- BERT4Rec, sequential recommendation, user behavior sequences, dynamic user preferences, information leakage, user representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
