CoChat
Sign Up
PreprintCited by 59Topic ModelingRecommender Systems and TechniquesInformation Retrieval and Search Behavior

Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders

Open access
Hou, Yupeng, Li, Jiacheng, Fu, Xiangjun, He, Zhankui, Yan, An, Chen, Xiusi, McAuley, Julian·2024·arXiv
DOI 10.48550/arxiv.2403.03952arXiv 2403.03952
Open this paper in CoChat — free accountOpen PDF

Abstract

Feature engineering has long been central to recommender systems, yet effectively leveraging textual item features remains challenging. Recent advances in large language models (LLMs) have enabled their use as semantic encoders for recommendation, but their roles and behaviors in this setting are still not well understood. Prior studies often rely on general-purpose embedding benchmarks (e.g., MTEB) when selecting LLMs, overlooking the unique characteristics of recommendation tasks. To address this gap, we introduce BLaIR, a comprehensive benchmark for evaluating LLMs as semantic encoders in recommendation scenarios. We contribute (1) a new large-scale Amazon Reviews 2023 dataset with over 570 million reviews and 48 million items, (2) a unified benchmark covering sequential recommendation, collaborative filtering, and product search, and (3) a new complex-query product search task featuring both semi-synthetic and real-world evaluation datasets. Experiments with 11 leading LLMs show that their rankings on BLaIR show little correlation with MTEB, highlighting the unique challenges of semantic encoding in recommendation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.03952

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • User preferences in the Amazon Review data cover many real-world applications, ranging from entertainment to healthcare and lifestyle.

    “Based on data analysis on Amazon-Review (Hou et al., 2024), we find that there is a large overlap in distinct domains, while they demonstrate the user preference covering many real-world applications, ranging from entertainment ( Books and Movies ) to healthcare ( Health and Personal Care ) and lifestyle ( Sports and Outdoors ).”

    From MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization (2026) · cites this paper as Hou et al., 2024

    Table 1 reports that Amazon Reviews 2023 spans 33 categories, and Appendix C.2 identifies Beauty, Games, and Baby as Amazon-related datasets. This supports broad coverage of product interests, but the passages do not establish Books, Movies, Health and Personal Care, or Sports and Outdoors as Amazon Review categories; C.2 lists book and movie data separately. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • For products, the paper uses the Amazon Reviews 2023 dataset, which ships ∼23M products across 33 categories and includes titles, feature bullets, free-text descriptions, and structured details.

    “For products, we use the Amazon Reviews 2023 dataset (Hou et al., 2024), which ships ∼ 23M products across 33 categories and includes titles, feature bullets, free-text descriptions, and structured details-rich textual content that GEO rewriters can operate on.”

    From E-GEO: A Testbed for Generative Engine Optimization in E-Commerce (2025) · cites this paper as Hou et al., 2024

    Table 1 reports 33 categories but lists 48,185,153 items and 35,393,189 item-metadata records for Amazon Reviews 2023, not approximately 23 million products. Section 3 describes structured JSON metadata with item descriptions and features, and Section C.2 confirms product titles; the copy does not establish the exact field list as worded. Section C.2 also says the paper generally used titles alone as item metadata for its benchmark. (LLM, conf=0.89)

    Experimental — accuracy not yet measured

  • MEMORYCD is constructed exclusively from the publicly released Amazon Reviews 2023 dataset.

    “MEMORYCD is constructed exclusively from the publicly released Amazon Reviews 2023 dataset (Hou et al., 2024), following all data usage rules and licensing conditions specified by the dataset maintainers.”

    From MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization (2026) · cites this paper as Hou et al., 2024

  • Hou et al. (2024) introduce Amazon Reviews 2023 together with a complex product search setting that emphasizes long, natural-language shopping contexts-the same regime E-GEO targets.

    “Domain-specific evidence reinforces this shift: Peng et al. (2024) show that instruction-tuning LLMs on large-scale e-commerce data meaningfully improves generalization to unseen products and tasks, and Hou et al. (2024) introduce Amazon Reviews 2023 together with a complex product search setting that emphasizes long, natural-language shopping contexts-the same regime E-GEO targets.”

    From E-GEO: A Testbed for Generative Engine Optimization in E-Commerce (2025) · cites this paper as Hou et al., 2024

  • There is a large overlap in distinct domains in the Amazon Review data.

    “Based on data analysis on Amazon-Review (Hou et al., 2024), we find that there is a large overlap in distinct domains, while they demonstrate the user preference covering many real-world applications, ranging from entertainment ( Books and Movies ) to healthcare ( Health and Personal Care ) and lifestyle ( Sports and Outdoors ).”

    From MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization (2026) · cites this paper as Hou et al., 2024

    Section 3 and Table 1 report the scale of Amazon Reviews 2023 and its 33 categories, but do not report overlap between domains. Searches for overlap and the named category examples found no analysis establishing the claimed large overlap. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 59

Research works printed but not linked: 56

  1. James Bennett, Stan Lanning, and 1 others. 2007. The netflix prize. In Proceedings of KDD cup and work- shop, volume 2007, page 35. New York.
  2. Tianqi Chen, Weinan Zhang, Qiuxia Lu, Kailong Chen, Zhao Zheng, and Yong Yu. 2012. Svdfea- ture: a toolkit for feature-based collaborative filtering. JMLR, 13(1):3619-3622.
  3. Ting Chen, Liangjie Hong, Yue Shi, and Yizhou Sun. 2017. Joint text embedding for personal- ized content-based recommendation. arXiv preprint arXiv:1706.01084. · arXiv
  4. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for con- trastive learning of visual representations. In ICML, pages 1597-1607.
  5. Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen An- derson, Greg Corrado, Wei Chai, Mustafa Ispir, and 1 others. 2016. Wide & deep learning for recom- mender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7-10.
  6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, and 1 others. 2023. Palm: Scaling language modeling with pathways. JMLR, 24(240):1- 113.
  7. Pierre Colombo, Nathan Noiry, Ekhine Irurozki, and Stéphan Clémençon. 2022. What are the best sys- tems? new perspectives on NLP benchmarking. In NeurIPS.
  8. Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, and 1 others. 2025. Mmteb: Massive multilingual text embedding benchmark. In ICLR.
  9. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence em- beddings. In EMNLP, pages 6894-6910.
  10. Prem Gopalan, Laurent Charlin, and David M Blei. 2014. Content-based recommendations with pois- son factorization. In NIPS, volume 27.
  11. F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context. Acm trans- actions on interactive intelligent systems (tiis), 5(4):1- 19.
  12. Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In WWW, pages 507-517.
  13. Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Majumder, Nathan Kallus, and Julian McAuley. 2023. Large language models as zero-shot conversational recom- menders. In CIKM.
  14. Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, and 1 others. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. · arXiv
  15. Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. 2022. Towards universal sequence representation learning for recom- mender systems. In KDD.
  16. Junjie Huang, Duyu Tang, Wanjun Zhong, Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, and Nan Duan. 2021. Whiteningbert: An easy unsuper- vised sentence embedding approach. In Findings of EMNLP, pages 238-244.
  17. Chumeng Jiang, Jiayin Wang, Weizhi Ma, Charles L. A. Clarke, Shuai Wang, Chuhan Wu, and Min Zhang. 2025. Beyond utility: Evaluating LLM as recom- mender. In WWW, pages 3850-3862.
  18. Yilun Jin, Zheng Li, Chenwei Zhang, Tianyu Cao, Yi- fan Gao, Pratik Jayarao, Mao Li, Xin Liu, Ritesh Sarkhel, Xianfeng Tang, Haodong Wang, Zhengyang Wang, Wenju Xu, Jingfeng Yang, Qingyu Yin, Xian Li, Priyanka Nigam, Yi Xu, Kai Chen, and 3 others. 2024. Shopping MMLU: A massive multi-task on- line shopping benchmark for large language models. In NeurIPS.
  19. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. CoRR, abs/2001.08361. · arXiv
  20. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham M. Kakade, Prateek Jain, and Ali Farhadi. 2022. Matryoshka representation learning. In NeurIPS.
  21. Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. 2021. Embedding-based product retrieval in taobao search. In KDD, pages 3181-3189.
  22. Jiao Liu, Zhu Sun, Shanshan Feng, and Yew-Soon Ong. 2024. Language model evolutionary algorithms for recommender systems: Benchmarks and algorithm comparisons. CoRR, abs/2411.10697. · arXiv
  23. Junling Liu, Chao Liu, Peilin Zhou, Qichen Ye, Dading Chong, Kang Zhou, Yueqi Xie, Yuwei Cao, Shoujin Wang, Chenyu You, and Philip S. Yu. 2023. Llmrec: Benchmarking large language models on recommen- dation task. CoRR, abs/2308.12241. · arXiv
  24. Qijiong Liu, Jieming Zhu, Lu Fan, Kun Wang, Hengchang Hu, Wei Guo, Yong Liu, and Xiao-Ming Wu. 2025. Benchmarking llms in recommendation tasks: A comparative evaluation with conventional recommenders. CoRR, abs/2503.05493. · arXiv
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. · arXiv
  26. Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recom- mendations on styles and substitutes. In SIGIR, pages 43-52.
  27. Raymond J Mooney and Loriene Roy. 2000. Content- based book recommending using learning for text categorization. In Proceedings of the fifth ACM con- ference on Digital libraries, pages 195-204.
  28. Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2025. Generative representational in- struction tuning. In ICLR.
  29. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. In EACL, pages 2014-2037.
  30. Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of ACL, pages 1864-1874.
  31. Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Jus- tifying recommendations using distantly-labeled re- views and fine-grained aspects. In EMNLP, pages 188-197.
  32. Yongxin Ni, Yu Cheng, Xiangyan Liu, Junchen Fu, Youhua Li, Xiangnan He, Yongfeng Zhang, and Fajie Yuan. 2023. A content-driven micro-video recommendation dataset at scale. arXiv preprint arXiv:2309.15379. · arXiv
  33. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, and 1 others. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. · arXiv
  34. Chandan K Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. 2022. Shopping queries dataset: A large-scale esci benchmark for improving product search. arXiv preprint arXiv:2206.06588. · arXiv
  35. Xubin Ren and Chao Huang. 2024. Easyrec: Simple yet effective language models for recommendation. arXiv preprint arXiv:2408.08821. · arXiv
  36. Xubin Ren, Wei Wei, Lianghao Xia, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. 2024. Representation learning with large language models for recommendation. In WWW, pages 3464- 3475.
  37. Steffen Rendle. 2010. Factorization machines. In ICDM, pages 995-1000. IEEE.
  38. Leheng Sheng, An Zhang, Yi Zhang, Yuxin Chen, Xiang Wang, and Tat-Seng Chua. 2024. Language models encode collaborative signals in recommendation. In ICLR.
  39. Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan Ö. Arik, Danqi Chen, and Tao Yu. 2025. BRIGHT: A realistic and challeng- ing benchmark for reasoning-intensive retrieval. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025.
  40. Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. CoRR, abs/2103.15316. · arXiv
  41. Jie Tang, Sen Wu, Jimeng Sun, and Hang Su. 2012. Cross-domain collaboration recommendation. In KDD, pages 1285-1293.
  42. Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
  43. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971. · arXiv
  44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, volume 30.
  45. Chong Wang and David M Blei. 2011. Collaborative topic modeling for recommending scientific articles. In KDD, pages 448-456.
  46. Hao Wang, Naiyan Wang, and Dit-Yan Yeung. 2015. Collaborative deep learning for recommender sys- tems. In KDD, pages 1235-1244.
  47. Jiangyuan Wang, Kejun Xiao, Qi Sun, Huaipeng Zhao, Tao Luo, Jiandong Zhang, and Xiaoyi Zeng. 2025. Shoppingbench: A real-world intent-grounded shopping benchmark for llm-based agents. CoRR, abs/2508.04266. · arXiv
  48. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Improving text embeddings with large language models. In ACL, pages 11897-11916.
  49. Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. 2024. Llmrec: Large language models with graph augmentation for recommendation. In WSDM, pages 806-815.
  50. An Yan, Zhankui He, Jiacheng Li, Tianyang Zhang, and Julian McAuley. 2023. Personalized showcases: Generating multi-modal explanations for recommen- dations. In SIGIR, pages 2251-2255.
  51. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real- world web interaction with grounded language agents. In NeurIPS.
  52. Guanghu Yuan, Fajie Yuan, Yudong Li, Beibei Kong, Shujie Li, Lei Chen, Min Yang, Chenyun Yu, Bo Hu, Zang Li, and 1 others. 2022. Tenrec: A large-scale multipurpose benchmark dataset for recommender systems. In NeurIPS, volume 35, pages 11480- 11493.
  53. Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. 2023. Where to go next for recommender systems? id-vs. modality-based recommender models revisited. In SIGIR.
  54. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. CoRR, abs/2506.05176. · arXiv
  55. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and 3 others. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223. · arXiv
  56. Cai-Nicolas Ziegler, Sean M McNee, Joseph A Konstan, and Georg Lausen. 2005. Improving recommenda- tion lists through topic diversification. In WWW, pages 22-32.

Authors
Hou, Y., Li, J., Fu, X., He, Z., Yan, A., Chen, X., & McAuley, J.
Affiliations
  • University of California San Diego (US)
  • University of California, Los Angeles (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
3251-3265
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.2403.03952
arXiv
2403.03952
OpenAlex
W4392576636
Cited by
59
References
0
Citations by year
2026: 35 · 2025: 22 · 2024: 2
FWCI
312.77
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (2432486)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
large language models, semantic encoding, item retrieval, recommender systems, benchmarking, cross-modal retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review