CoChat
Sign Up
Cited by 35Recommender Systems and TechniquesAdvanced Bandit Algorithms ResearchInformation Retrieval and Search Behavior

Large Language Models are Learnable Planners for Long-Term Recommendation

Open access
Shi, Wentao, He, Xiangnan, Zhang, Yang, Gao, Chongming, Li, Xinyue, Zhang, Jizhi, Wang, Qifan, Feng, Fuli·2024·Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
DOI 10.1145/3626772.3657683
Open this paper in CoChat — free accountOpen PDF

Abstract

Planning for both immediate and long-term benefits becomes increasingly important in recommendation. Existing methods apply Reinforcement Learning (RL) to learn planning capacity by maximizing cumulative reward for long-term recommendation. However, the scarcity of recommendation data presents challenges such as instability and susceptibility to overfitting when training RL models from scratch, resulting in sub-optimal performance. In this light, we propose to leverage the remarkable planning capabilities over sparse data of Large Language Models (LLMs) for long-term recommendation. The key to achieving the target lies in formulating a guidance plan following principles of enhancing long-term engagement and grounding the plan to effective and executable actions in a personalized manner. To this end, we propose a Bi-level Learnable LLM Planner framework, which consists of a set of LLM instances and breaks down the learning process into macro-learning and micro-learning to learn macro-level guidance and micro-level personalized recommendation policies, respectively. Extensive experiments validate that the framework facilitates the planning ability of LLMs for long-term recommendation. Our code and data can be found at https://github.com/jizhi-zhang/BiLLP.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3626772.3657683

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The 'Books' dataset from Amazon Review data has been used to evaluate Agent4Rec and BiLLP performance on large-scale tasks.

    “For instance, the 'Books' dataset (10.3M users, 4.4M items) from Amazon Review data (McAuley et al., 2015) has been used to evaluate Agent4Rec (Zhang et al., 2024a) and BiLLP (Shi et al., 2024) performance on largescale tasks”

    From A Survey on LLM-powered Agents for Recommender Systems (2025) · cites this paper as Shi et al., 2024

    Section 5.1.1 says BiLLP was evaluated using Amazon-Book, a subset of the Amazon Product Review dataset. However, the paper says it filtered that dataset, and Table 4 lists the experimental Amazon dataset as 3,109 users and 13,864 items. It does not support describing BiLLP's evaluation as using the cited 10.3M-user, 4.4M-item scale. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The Steam dataset contains 3.7M interactions between 334.7K users and 13K gaming items and has been extensively used by Agent4Rec, BiLLP, FLOW, and InteRecAgent to validate their effectiveness in game recommendation.

    “The Steam dataset, introduced by (Kang and McAuley, 2018), contains 3.7M interactions between 334.7K users and 13K gaming items, and has been extensively used by methods such as Agent4Rec (Zhang et al., 2024a), BiLLP (Shi et al., 2024), FLOW (Cai et al., 2024), and InteRecAgent (Huang et al., 2023) to validate their effectiveness in game recommendation.”

    From A Survey on LLM-powered Agents for Recommender Systems (2025) · cites this paper as Shi et al., 2024

  • BiLLP's planning mechanism employs a hierarchical structure with two levels: macro-learning (Planner and Reflector LLMs) generates high-level strategic plans and guidelines from experience, while micro-learning (ActorCritic) translates these plans into specific recommendations.

    “BiLLP (Shi et al., 2024) planning mechanism employs a hierarchical structure with two levels: macro-learning (Planner and Reflector LLMs) generates high-level strategic plans and guidelines from experience, while micro-learning (ActorCritic) translates these plans into specific recommendations.”

    From A Survey on LLM-powered Agents for Recommender Systems (2025) · cites this paper as Shi et al., 2024

  • BiLLP employs three key metrics adopted from reinforcement learning: trajectory length, average single-round reward, and cumulative trajectory reward.

    “BiLLP (Shi et al., 2024) employs three key metrics adopted from reinforcement learning: trajectory length, average singleround reward, and cumulative trajectory reward.”

    From A Survey on LLM-powered Agents for Recommender Systems (2025) · cites this paper as Shi et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 59

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). · arXiv
  2. Keqin Bao, Jizhi Zhang, Wenjie Wang, Yang Zhang, Zhengyi Yang, Yancheng Luo, Fuli Feng, Xiangnan He, and Qi Tian. 2023. A Bi-Step Grounding Paradigm for Large Language Models in Recommendation Systems. CoRR abs/2308.08434 (2023). · arXiv
  3. Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. arXiv preprint arXiv:2305.00447 (2023). · arXiv
  4. Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2020. Bias and Debias in Recommender System: A Survey and Future Directions. CoRR abs/2010.03240 (2020). · arXiv
  5. Jiawei Chen, Junkang Wu, Jiancan Wu, Xuezhi Cao, Sheng Zhou, and Xiangnan He. 2023. Adap-𝜏 : Adaptively Modulating Embedding Magnitude for Recom- mendation. In WWW. ACM, 1085-1096.
  6. Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H. Chi. 2019. Top-K Off-Policy Correction for a REINFORCE Recommender System. In WSDM. ACM, 456-464.
  7. Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxi- ang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering ChatGPT's Capabilities in Recommender Systems. arXiv preprint arXiv:2305.02182 (2023). · arXiv
  8. Thomas Degris, Martha White, and Richard S. Sutton. 2012. Off-Policy Actor- Critic. CoRR abs/1205.4839 (2012). · arXiv
  9. Gabriel Dulac-Arnold, Richard Evans, Peter Sunehag, and Ben Coppin. 2015. Reinforcement Learning in Large Discrete Action Spaces. CoRR abs/1512.07679 (2015). · arXiv
  10. Yue Feng, Shuchang Liu, Zhenghai Xue, Qingpeng Cai, Lantao Hu, Peng Jiang, Kun Gai, and Fei Sun. 2023. A Large Language Model Enhanced Conversational Recommender System. arXiv preprint arXiv:2308.06212 (2023). · arXiv
  11. Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. 2019. Benchmarking Batch Deep Reinforcement Learning Algorithms. CoRR abs/1910.01708 (2019). · arXiv
  12. Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-Policy Deep Reinforce- ment Learning without Exploration. In ICML (Proceedings of Machine Learning Research, Vol. 97). PMLR, 2052-2062.
  13. Chongming Gao, Kexin Huang, Jiawei Chen, Yuan Zhang, Biao Li, Peng Jiang, Shiqi Wang, Zhong Zhang, and Xiangnan He. 2023. Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation. In SIGIR. ACM, 238-248.
  14. Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2021. Advances and Challenges in Conversational Recommender Systems: A Survey. AI Open 2 (2021), 100-126.
  15. Chongming Gao, Shiqi Wang, Shijun Li, Jiawei Chen, Xiangnan He, Wenqiang Lei, Biao Li, Yuan Zhang, and Peng Jiang. 2024. CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender System. ACM Trans. Inf. Syst. 42, 1 (2024), 14:1-14:27.
  16. Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In IJCAI. ijcai.org, 1725-1731.
  17. Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2023. Large language models are zero-shot rankers for recommender systems. arXiv preprint arXiv:2305.08845 (2023). · arXiv
  18. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Lan- guage Models as Zero-Shot Planners: Extracting Actionable Knowledge for Em- bodied Agents. In ICML (Proceedings of Machine Learning Research, Vol. 162). PMLR, 9118-9147.
  19. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. 2022. Inner Monologue: Embodied Reasoning through Planning with Language Models. In CoRL (Proceedings of Machine Learning Research, Vol. 205). PMLR, 1769-1782.
  20. Xu Huang, Jianxun Lian, Yuxuan Lei, Jing Yao, Defu Lian, and Xing Xie. 2023. Recommender ai agent: Integrating large language models for interactive recom- mendations. arXiv preprint arXiv:2308.16505 (2023). · arXiv
  21. Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, and Craig Boutilier. 2019. SlateQ: A Tractable Decomposition for Reinforcement Learning with Recommendation Sets. In IJCAI. ijcai.org, 2592-2599.
  22. Wang-Cheng Kang and Julian J. McAuley. 2018. Self-Attentive Sequential Rec- ommendation. In ICDM. IEEE Computer Society, 197-206.
  23. Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng. 2023. Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction. arXiv preprint arXiv:2305.06474 (2023). · arXiv
  24. Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conserva- tive Q-Learning for Offline Reinforcement Learning. In NeurIPS.
  25. Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brah- man, Shiyu Huang, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2023. Swift- Sage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks. CoRR abs/2305.17390 (2023). · arXiv
  26. Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Xiangyang Li, Chenxu Zhu, Huifeng Guo, Yong Yu, Ruiming Tang, et al. 2023. How Can Recom- mender Systems Benefit from Large Language Models: A Survey. arXiv preprint arXiv:2306.05817 (2023). · arXiv
  27. Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, and Weinan Zhang. 2023. ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation. arXiv preprint arXiv:2308.11131 (2023). · arXiv
  28. Feng Liu, Ruiming Tang, Xutao Li, Yunming Ye, Haokun Chen, Huifeng Guo, and Yuzhou Zhang. 2018. Deep Reinforcement Learning based Recommendation with Explicit User-Item Interactions Modeling. CoRR abs/1810.12027 (2018). · arXiv
  29. Junling Liu, Chao Liu, Renjie Lv, Kang Zhou, and Yan Zhang. 2023. Is chatgpt a good recommender? a preliminary study. arXiv preprint arXiv:2304.10149 (2023). · arXiv
  30. Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timo- thy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asyn- chronous Methods for Deep Reinforcement Learning. In ICML (JMLR Workshop and Conference Proceedings, Vol. 48). JMLR.org, 1928-1937.
  31. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. 2013. Playing Atari with Deep Reinforcement Learning. CoRR abs/1312.5602 (2013). · arXiv
  32. Jianmo Ni, Jiacheng Li, and Julian J. McAuley. 2019. Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects. In EMNLP/IJCNLP (1). Association for Computational Linguistics, 188-197.
  33. Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. 2022. Planning with Large Language Models via Corrective Re-prompting. CoRR abs/2211.09935 (2022). · arXiv
  34. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. CoRR abs/1707.06347 (2017). · arXiv
  35. Junjie Sheng, Zixiao Huang, Chuyun Shen, Wenhao Li, Yun Hua, Bo Jin, Hongyuan Zha, and Xiangfeng Wang. 2023. Can language agents be alternatives to PPO? A Preliminary Empirical Study On OpenAI Gym. CoRR abs/2312.03290 (2023). · arXiv
  36. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366 [cs.AI] · arXiv
  37. Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. 2023. AdaPlanner: Adaptive Planning from Feedback with Language Models. CoRR abs/2305.16653 (2023). · arXiv
  38. Hugo Touvron et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). · arXiv
  39. Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Xiaojiang Huang, Yanbin Lu, and Yingzhen Yang. 2023. Rec- mind: Large language model powered agent for recommendation. arXiv preprint arXiv:2308.14296 (2023). · arXiv
  40. Yuyan Wang, Mohit Sharma, Can Xu, Sriraj Badam, Qian Sun, Lee Richardson, Lisa Chung, Ed H. Chi, and Minmin Chen. 2022. Surrogate for Long-Term User Experience in Recommender Systems. In KDD. ACM, 4100-4109.
  41. Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023. Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. CoRR abs/2302.01560 (2023). · arXiv
  42. Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh Merel, Jost Tobias Springen- berg, Scott E. Reed, Bobak Shahriari, Noah Y. Siegel, Çaglar Gülçehre, Nicolas Heess, and Nando de Freitas. 2020. Critic Regularized Regression. In NeurIPS.
  43. Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. 2023. Llmrec: Large language models with graph augmentation for recommendation. arXiv preprint arXiv:2311.00423 (2023). · arXiv
  44. Junda Wu, Zhihui Xie, Tong Yu, Handong Zhao, Ruiyi Zhang, and Shuai Li. 2022. Dynamics-Aware Adaptation for Reinforcement Learning Based Cross-Domain Interactive Recommendation. In SIGIR. ACM, 290-300.
  45. Likang Wu, Zhaopeng Qiu, Zhi Zheng, Hengshu Zhu, and Enhong Chen. 2023. Exploring large language model for graph data understanding in online job recommendations. arXiv preprint arXiv:2307.05722 (2023). · arXiv
  46. Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al. 2023. A Survey on Large Language Models for Recommendation. arXiv preprint arXiv:2305.19860 (2023). · arXiv
  47. Qingyun Wu, Hongning Wang, Liangjie Hong, and Yue Shi. 2017. Returning is believing: Optimizing long-term user engagement in recommender systems. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 1927-1936.
  48. Yunjia Xi, Weiwen Liu, Jianghao Lin, Jieming Zhu, Bo Chen, Ruiming Tang, Weinan Zhang, Rui Zhang, and Yong Yu. 2023. Towards Open-World Recom- mendation with Knowledge Augmentation from Large Language Models. arXiv preprint arXiv:2306.10933 (2023). · arXiv
  49. Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M. Jose. 2020. Self-Supervised Reinforcement Learning for Recommender Systems. In SIGIR. ACM, 931-940.
  50. Shuyuan Xu, Juntao Tan, Zuohui Fu, Jianchao Ji, Shelby Heinecke, and Yongfeng Zhang. 2022. Dynamic Causal Collaborative Filtering. In CIKM. ACM, 2301-2310.
  51. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR. OpenReview.net.
  52. Yuanqing Yu, Chongming Gao, Jiawei Chen, Heng Tang, Yuefeng Sun, Qian Chen, Weizhi Ma, and Min Zhang. 2024. EasyRL4Rec: A User-Friendly Code Library for Reinforcement Learning Based Recommender Systems. arXiv preprint arXiv:2402.15164 (2024). · arXiv
  53. Danyang Zhang, Lu Chen, Situo Zhang, Hongshen Xu, Zihan Zhao, and Kai Yu. 2023. Large Language Model Is Semi-Parametric Reinforcement Learning Agent. CoRR abs/2306.07929 (2023). · arXiv
  54. Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023. Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation. arXiv preprint arXiv:2305.07609 (2023). · arXiv
  55. Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. 2023. Recommendation as instruction following: A large language model empowered recommendation approach. arXiv preprint arXiv:2305.07001 (2023). · arXiv
  56. Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He. 2023. Collm: Integrating collaborative embeddings into large language models for recommendation. arXiv preprint arXiv:2310.19488 (2023). · arXiv
  57. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2023. ExpeL: LLM Agents Are Experiential Learners. CoRR abs/2308.10144 (2023). · arXiv
  58. Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2018. Deep reinforcement learning for page-wise recommendations. In RecSys. ACM, 95-103.
  59. Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018. DRN: A Deep Reinforcement Learning Framework for News Recommendation. In WWW. ACM, 167-176.

Cited by

Sorted by citations

Authors
Shi, W., He, X., Zhang, Y., Gao, C., Li, X., Zhang, J., Wang, Q., & Feng, F.
Affiliations
  • University of Science and Technology of China (CN)
  • Meta (United States) (US)
ORCID
Venue
Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Published
2024-07-10
Pages
1893-1903
DOI
10.1145/3626772.3657683
OpenAlex
W4392490033
Cited by
35
References
4
Citations by year
2026: 15 · 2025: 16 · 2024: 4
FWCI
13.33
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • the National Key Research and Development Program of China (2022YFB3104701)
  • the National Natural Science Foundation of China (62272437, 62121002)
  • the CCCD Key Lab of Ministry of Culture and Tourism
Topics
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
large language models, planning, personalized recommendation, long-term engagement, reinforcement learning for recommendation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review