Cited by 109Topic ModelingInformation Retrieval and Search BehaviorRecommender Systems and Techniques

LaMP: When Large Language Models Meet Personalization

Open access
Salemi, Alireza, Mysore, Sheshera, Bendersky, Michael, Zamani, Hamed·2024·Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Abstract

This paper highlights the importance of personalization in large language models and introduces the LaMP benchmark -a novel benchmark for training and evaluating language models for producing personalized outputs.LaMP offers a comprehensive evaluation framework with diverse language tasks and multiple entries for each user profile.It consists of seven personalized tasks, spanning three text classification and four text generation tasks.We additionally propose two retrieval augmentation approaches that retrieve personal items from each user profile for personalizing language model outputs.To this aim, we study various retrieval models, including term matching, semantic matching, and time-aware methods.Extensive experiments on LaMP for zero-shot and fine-tuned language models demonstrate the efficacy of the proposed retrieval augmentation approach and highlight the impact of personalization in various natural language tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.acl-long.399

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Real-user memory benchmarks like LaMP and MSC are limited in length, offering only short sequences that do not stress the long-context windows available in modern LLMs.

    “Others with real user records, like LaMP (Salemi et al., 2024) and MSC (Xu et al., 2022), are limited in length, offering only short sequences that do not stress the long-context windows available in modern LLMs.”

    From MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalization (2026) · cites this paper as Salemi et al., 2024

    The Introduction says LaMP user profiles are often large enough to exceed LLM length limits, and Section 3 describes each profile as a potentially large collection of data points. That is contrary to the attributed claim that LaMP offers only short sequences, although the paper does not assess whether its profiles stress the context windows of modern LLMs. (LLM, conf=0.85)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured
  • A smaller line of work evaluates memory using real user interaction histories, including MSC and LaMP.

    “A smaller line of work evaluates memory using real user interaction histories, including MSC (Xu et al., 2022) and LaMP (Salemi et al., 2024).”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. Xiang Ao, Xiting Wang, Ling Luo, Ying Qiao, Qing He, and Xing Xie. 2021. PENS: A dataset and generic framework for personalized news headline genera- tion. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 82-92, Online. Association for Computational Linguistics. · DOI
  2. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
  3. Paul N. Bennett, Ryen W. White, Wei Chu, Susan T. Dumais, Peter Bailey, Fedor Borisyuk, and Xiaoyuan Cui. 2012. Modeling the impact of short-and long- term behavior on search personalization. In Proceed- ings of the 35th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, SIGIR '12, page 185-194, New York, NY, USA. Association for Computing Machinery. · DOI
  4. Bruce W. Croft, Stephen Cronen-Townsend, and Victor Lavrenko. 2001. Relevance feedback and personal- ization: A language modeling perspective. In DE- LOS Workshop: Personalisation and Recommender Systems in Digital Libraries.
  5. Abhinandan S. Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. 2007. Google news personalization: Scalable online collaborative filtering. In Proceed- ings of the 16th International Conference on World Wide Web, WWW '07, page 271-280, New York, NY, USA. Association for Computing Machinery. · DOI
  6. James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, and Dasarathi Sampath. 2010. The youtube video rec- ommendation system. In Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys '10, page 293-296, New York, NY, USA. Association for Computing Machinery. · DOI
  7. Shiran Dudy, Steven Bedrick, and Bonnie Webber. 2021. Refocusing on relevance: Personalization in NLG. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 5190-5202, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics. · DOI · Paper page
  8. Susan T. Dumais. 2016. Personalized search: Potential and pitfalls. In Proceedings of the 25th ACM Interna- tional on Conference on Information and Knowledge Management, CIKM '16, page 689, New York, NY, USA. Association for Computing Machinery. · DOI
  9. Peter S. Fader, Bruce G.S. Hardie, and Ka Lok Lee. 2005. Rfm and clv: Using iso-value curves for cus- tomer base analysis. Journal of Marketing Research, 42(4):415-430. · DOI · Paper page
  10. Michael Färber and Adam Jatowt. 2020. Citation rec- ommendation: Approaches and datasets. Int. J. Digit. Libr. · DOI
  11. Lucie Flek. 2020. Returning the N to NLP: Towards contextually personalized classification models. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7828- 7838, Online. Association for Computational Lin- guistics. · DOI
  12. Andrew Fowler, Kurt Partridge, Ciprian Chelba, Xiao- jun Bi, Tom Ouyang, and Shumin Zhai. 2015. Ef- fects of language modeling and its personalization on touchscreen typing performance. In Proceedings of the 33rd Annual ACM Conference on Human Fac- tors in Computing Systems, CHI '15, page 649-658, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
  13. Markus Freitag and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. In Proceedings of the First Workshop on Neural Ma- chine Translation, pages 56-60, Vancouver. Associa- tion for Computational Linguistics. · DOI
  14. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khy- athi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jham- tani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bod- hisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimo- rina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96-120, Online. Association for Computational Linguistics. · DOI · Paper page
  15. Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury learning: Integrat- ing dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Hu- man Factors in Computing Systems, CHI '22, New York, NY, USA. Association for Computing Machin- ery. · DOI
  16. F. Maxwell Harper and Joseph A. Konstan. 2015. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 5(4). · DOI
  17. M.A. Hearst, S.T. Dumais, E. Osuna, J. Platt, and B. Scholkopf. 1998. Support vector machines. IEEE Intelligent Systems and their Applications, 13(4):18- 28. · DOI · Paper page
  18. Xiaolei Huang, Lucie Flek, Franck Dernoncourt, Charles Welch, Silvio Amir, Ramit Sawhney, and Diyi Yang. 2022. Usernlp'22: 2022 international workshop on user-centered natural language process- ing. In Companion Proceedings of the Web Confer- ence 2022, WWW '22, page 1176-1177, New York, NY, USA. Association for Computing Machinery. · DOI
  19. Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Rai- son, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2775-2779, Brussels, Belgium. Association for Computational Linguistics. · DOI
  20. Fatemehsadat Mireshghallah, Vaishnavi Shrivastava, Milad Shokouhi, Taylor Berg-Kirkpatrick, Robert Sim, and Dimitrios Dimitriadis. 2022. UserIdentifier: Implicit user representations for simple and effective personalized sentiment analysis. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3449-3456, Seattle, United States. Association for Computational Linguistics. · DOI
  21. Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 188-197, Hong Kong, China. Association for Computational Lin- guistics. · DOI
  22. Oard, Douglas, Webber, William, Kirsch, David A., and Golitsynskiy, Sergey. 2015. Avocado research email collection. OpenAI. 2023. Gpt-4 technical report. · DOI
  23. Sheena Panthaplackel, Adrian Benton, and Mark Dredze. 2022. Updated headline generation: Cre- ating updated summaries for evolving news stories. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6438-6461, Dublin, Ireland. Association for Computational Linguistics. · DOI
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. · DOI · Paper page
  25. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics. · DOI · Paper page
  26. Hongjin Qian, Xiaohe Li, Hanxun Zhong, Yu Guo, Yueyuan Ma, Yutao Zhu, Zhanliang Liu, Zhicheng Dou, and Ji-Rong Wen. 2021. Pchatbot: A large- scale dataset for personalized chatbot. In Proceed- ings of the 44th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, SIGIR '21, page 2470-2477, New York, NY, USA. Association for Computing Machinery. · DOI
  27. Werner J. Reinartz and V. Kumar. 2000. On the prof- itability of long-life customers in a noncontractual setting: An empirical investigation and implications for marketing. Journal of Marketing, 64(4):17-35. · DOI · Paper page
  28. Werner J. Reinartz and V. Kumar. 2003. The impact of customer relationship characteristics on profitable lifetime duration. Journal of Marketing, 67(1):77- 99. · DOI
  29. Stephen Robertson, S. Walker, S. Jones, M. M. Hancock- Beaulieu, and M. Gatford. 1995. Okapi at trec-3. In Proceedings of the Third Text REtrieval Conference, TREC-3, pages 109-126.
  30. Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pier- rehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pages 175-190, Seattle, United States. Association for Computational Linguistics. · DOI
  31. Reza Shokri, Marco Stronati, Congzheng Song, and Vi- taly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE Sym- posium on Security and Privacy (SP), pages 3-18. · DOI · Paper page
  32. Nikita Soni, Matthew Matero, Niranjan Balasubrama- nian, and H. Andrew Schwartz. 2022. Human lan- guage modeling. In Findings of the Association for Computational Linguistics: ACL 2022, pages 622- 636, Dublin, Ireland. Association for Computational Linguistics. · DOI
  33. Shayan A. Tabrizi, Azadeh Shakery, Hamed Zamani, and Mohammad Ali Tavallaei. 2018. Person: Per- sonalized information retrieval evaluation based on citation networks. Information Processing & Man- agement, 54(4):630-656. · DOI
  34. Stojan Trajanovski, Chad Atalla, Kunho Kim, Vipul Agarwal, Milad Shokouhi, and Chris Quirk. 2021. When does text prediction benefit from additional context? an exploration of contextual signals for chat and email messages. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, pages 1-9, Online. Association for Computational Linguistics. · DOI
  35. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Advances in Neural Information Processing Systems, volume 32.
  36. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics. · DOI · Paper page
  37. Charles Welch, Chenxi Gu, Jonathan K. Kummerfeld, Veronica Perez-Rosas, and Rada Mihalcea. 2022. Leveraging similar users for personalized language modeling with limited data. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1742-1752, Dublin, Ireland. Association for Compu- tational Linguistics. · DOI
  38. Yuwei Wu, Xuezhe Ma, and Diyi Yang. 2021. Personal- ized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1956-1970, Online. Association for Computational Linguistics. · DOI
  39. Joern Wuebker, Patrick Simianer, and John DeNero. 2018. Compact personalized models for neural ma- chine translation. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 881-886, Brussels, Belgium. As- sociation for Computational Linguistics. · DOI
  40. Jiajing Xu, Andrew Zhai, and Charles Rosenberg. 2022. Rethinking personalized ranking at pinterest: An end- to-end approach. In Proceedings of the 16th ACM Conference on Recommender Systems, RecSys '22, page 502-505, New York, NY, USA. Association for Computing Machinery. · DOI
  41. Gui-Rong Xue, Jie Han, Yong Yu, and Qiang Yang. 2009. User language model for collaborative person- alized search. ACM Trans. Inf. Syst., 27(2). · DOI
  42. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics. · DOI · Paper page
  43. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
  44. Hanxun Zhong, Zhicheng Dou, Yutao Zhu, Hongjin Qian, and Ji-Rong Wen. 2022. Less is more: Learn- ing to refine dialogue history for personalized dia- logue generation. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pages 5808-5820, Seattle, United States. Association for Computational Lin- guistics. · DOI
  45. Wanjun Zhong, Duyu Tang, Jiahai Wang, Jian Yin, and Nan Duan. 2021. UserAdapter: Few-shot user learn- ing in sentiment analysis. In Findings of the Associa- tion for Computational Linguistics: ACL-IJCNLP 2021, pages 1484-1488, Online. Association for Computational Linguistics. · DOI
  46. Jianing Zhou and Suma Bhat. 2021. Paraphrase genera- tion: A survey of the state of the art. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 5075-5086, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI

Cited by

Sorted by citations

Authors
Salemi, A., Mysore, S., Bendersky, M., & Zamani, H.
Affiliations
  • University of Massachusetts Amherst (US)
  • Google (United States) (US)
Venue
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Published
2024
Pages
7370-7392
ISSN
0736-587X (ISSN-L)
OpenAlex
W4402671806
Cited by
109
References
11
Citations by year
2026: 42 · 2025: 47 · 2024: 11 · 2023: 9
FWCI
29.51
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2143434)
  • Office of Naval Research (N000142212688)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
personalized language models, retrieval augmentation, user profiles, text classification, text generation, semantic matching, zero-shot learning, fine-tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review