PreprintCited by 114Topic ModelingInformation Retrieval and Search BehaviorRecommender Systems and Techniques

LaMP: When Large Language Models Meet Personalization

Open access
Salemi, Alireza, Mysore, Sheshera, Bendersky, Michael, Zamani, Hamed·2023·arXiv

Abstract

This paper highlights the importance of personalization in large language models and introduces the LaMP benchmark -- a novel benchmark for training and evaluating language models for producing personalized outputs. LaMP offers a comprehensive evaluation framework with diverse language tasks and multiple entries for each user profile. It consists of seven personalized tasks, spanning three text classification and four text generation tasks. We additionally propose two retrieval augmentation approaches that retrieve personal items from each user profile for personalizing language model outputs. To this aim, we study various retrieval models, including term matching, semantic matching, and time-aware methods. Extensive experiments on LaMP for zero-shot and fine-tuned language models demonstrate the efficacy of the proposed retrieval augmentation approach and highlight the impact of personalization in various natural language tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.11406

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 19

  1. Bruce W. Croft, Stephen Cronen-Townsend, and Victor Lavrenko. 2001. Relevance feedback and personal- ization: A language modeling perspective. In DE- LOS Workshop: Personalisation and Recommender Systems in Digital Libraries.
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In North American Chapter of the Association for Computational Linguistics.
  3. Susan T. Dumais. 2016. Personalized search: Potential and pitfalls. In Proceedings of the 25th ACM Interna- tional on Conference on Information and Knowledge Management, CIKM '16, page 689, New York, NY, USA. Association for Computing Machinery. · DOI
  4. Manish Gupta, Rui Li, Zhijun Yin, and Jiawei Han. 2010. Survey on social tagging techniques. SIGKDD Explor. Newsl.
  5. Xiaolei Huang, Lucie Flek, Franck Dernoncourt, Charles Welch, Silvio Amir, Ramit Sawhney, and Diyi Yang. 2022. Usernlp'22: 2022 international workshop on user-centered natural language process- ing. In Companion Proceedings of the Web Confer- ence 2022, WWW '22, page 1176-1177, New York, NY, USA. Association for Computing Machinery. Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 188-197, Hong Kong, China. Association for Computational Lin- guistics. · DOI
  6. Oard, Douglas, Webber, William, Kirsch, David A., and Golitsynskiy, Sergey. 2015. Avocado research email collection. OpenAI. 2023. Gpt-4 technical report. · DOI
  7. Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10671-10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  8. Joan Plepi, Béla Neuendorf, Lucie Flek, and Charles Welch. 2022. Unifying data perspectivism and per- sonalization: An application to social norms. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 7391- 7402, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  9. Hongjin Qian, Xiaohe Li, Hanxun Zhong, Yu Guo, Yueyuan Ma, Yutao Zhu, Zhanliang Liu, Zhicheng Dou, and Ji-Rong Wen. 2021. Pchatbot: A large- scale dataset for personalized chatbot. In Proceed- ings of the 44th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, SIGIR '21, page 2470-2477, New York, NY, USA. Association for Computing Machinery. · DOI
  10. Omid Rafieian and Hema Yoganarasimhan. 2023. Ai and personalization. Artificial Intelligence in Market- ing, pages 77-102.
  11. Stephen Robertson, S. Walker, S. Jones, M. M. Hancock- Beaulieu, and M. Gatford. 1995. Okapi at trec-3. In Proceedings of the Third Text REtrieval Conference, TREC-3, pages 109-126.
  12. Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008. Arnetminer: Extraction and mining of academic social networks. In KDD'08, pages 990-998.
  13. Sebastian Vincent, Rowanne Sumner, Alice Dowek, Charlotte Blundell, Emily Preston, Chris Bayliss, Chris Oakley, and Carolina Scarton. 2023. Per- sonalised language modelling of screen characters using rich metadata annotations. arXiv preprint arXiv:2303.16618. · arXiv
  14. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Advances in Neural Information Processing Systems, volume 32.
  15. Joern Wuebker, Patrick Simianer, and John DeNero. 2018. Compact personalized models for neural ma- chine translation. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 881-886, Brussels, Belgium. As- sociation for Computational Linguistics. · DOI
  16. Jiajing Xu, Andrew Zhai, and Charles Rosenberg. 2022. Rethinking personalized ranking at pinterest: An end- to-end approach. In Proceedings of the 16th ACM Conference on Recommender Systems, RecSys '22, page 502-505, New York, NY, USA. Association for Computing Machinery. · DOI
  17. Hansi Zeng, Surya Kallumadi, Zaid Alibadi, Rodrigo Nogueira, and Hamed Zamani. 2023. A personalized dense retrieval framework for unified information ac- cess. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, New York, NY, USA. Association for Computing Machinery.
  18. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
  19. Jian Zhu and David Jurgens. 2021. Idiosyncratic but not arbitrary: Learning idiolects in online registers reveals distinctive yet consistent individual styles. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

Cited by

Sorted by citations

Authors
Salemi, A., Mysore, S., Bendersky, M., & Zamani, H.
Affiliations
  • University of Massachusetts Amherst (US)
  • Google (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
7370-7392
ISSN
0736-587X (ISSN-L)
OpenAlex
W4402671806
Cited by
109
References
0
Citations by year
2026: 42 · 2025: 47 · 2024: 11 · 2023: 9
FWCI
29.51
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2143434)
  • Office of Naval Research (N000142212688)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
personalized language models, retrieval augmentation, user profiles, text classification, text generation, semantic matching, zero-shot learning, fine-tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review