Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale
Open accessAbstract
Large Language Models (LLMs) have emerged as personalized assistants for users across a wide range of tasks -- from offering writing support to delivering tailored recommendations or consultations. Over time, the interaction history between a user and an LLM can provide extensive information about an individual's traits and preferences. However, open questions remain on how well LLMs today can effectively leverage such history to (1) internalize the user's inherent traits and preferences, (2) track how the user profiling and preferences evolve over time, and (3) generate personalized responses accordingly in new scenarios. In this work, we introduce the PERSONAMEM benchmark. PERSONAMEM features curated user profiles with over 180 simulated user-LLM interaction histories, each containing up to 60 sessions of multi-turn conversations across 15 real-world tasks that require personalization. Given an in-situ user query, i.e. query issued by the user from the first-person perspective, we evaluate LLM chatbots' ability to identify the most suitable response according to the current state of the user's profile. We observe that current LLMs still struggle to recognize the dynamic evolution in users' profiles over time through direct prompting approaches. As a consequence, LLMs often fail to deliver responses that align with users' current situations and preferences, with frontier models such as GPT-4.1, o4-mini, GPT-4.5, o1, or Gemini-2.0 achieving only around 50% overall accuracy, suggesting room for improvement. We hope that PERSONAMEM, along with the user profile and conversation simulation pipeline, can facilitate future research in the development of truly user-aware chatbots. Code and data are available at github.com/bowen-upenn/PersonaMem.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing benchmarks primarily operationalize long-term memory as shallow cross-session retrieval rather than sustained memory accumulation.
“Despite the growing interest in long-term memory, existing benchmarks (Maharana et al., 2024;Du et al., 2024;Jiang et al., 2025) primarily operationalize it as shallow cross-session retrieval rather than sustained memory accumulation.”
From From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents (2026) · cites this paper as Jiang et al., 2025
The PERSONAMEM Benchmark: Overview (pp. 3–4) describes multi-session interactions with evolving preferences and explicitly tests whether models track their full evolution, not just retrieve isolated facts. Although the paper includes fact-recall tasks, its benchmark is not characterized as primarily shallow cross-session retrieval. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredPersonaMem handles updates across no more than three sessions.
“PersonaMem (Jiang et al., 2025) handles updates across no more than three sessions.”
From From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents (2026) · cites this paper as Jiang et al., 2025
The Abstract (p. 1) says PersonaMem histories contain up to 60 sessions, contrary to the claimed three-session limit. The comparison table in §5.2 (p. 10) lists three as the maximum number of knowledge updates, not sessions. (LLM, conf=0.94)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredExisting long-term memory benchmarks predominantly operationalize long-term memory as fact retrieval from past conversations, with limited emphasis on memory consolidation and frequent memory mutation.
“Despite the growing interest in long-term memory, existing benchmarks (Maharana et al., 2024; Du et al., 2024; Jiang et al., 2025) primarily operationalize it as shallow cross-session retrieval rather than sustained memory accumulation.”
From From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents (2026) · cites this paper as Jiang et al., 2025
Jiang et al.'s PERSONAMEM benchmark is a counterexample to the characterization for which it is cited. The benchmark overview (p. 3) says it evaluates tracking the full evolution of user preferences and revisiting reasons for preference updates, not primarily retrieving static facts. The paper does not establish the separate claim about memory consolidation across benchmarks. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredBenchmarks like DiaSim, LongMemEval, PersonaMem, Multi-Session Chat, and Conversation Chronicles extend to multiple independent but related sessions, mirroring real-world scenarios, and assessing memory beyond the context window.
“Subsequently, benchmarks like DiaSim (Kim et al., 2024), LongMemEval (Wu et al., 2024), PersonaMem (Jiang et al., 2025), Multi-Session Chat (Xu et al., 2022a), and Conversation Chronicles (Jang et al., 2023) extend to multiple independent but related sessions, mirroring real-world scenarios, and assessing the memory beyond the context window.”
From EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memory (2026) · cites this paper as Jiang et al., 2025
The PERSONAMEM Benchmark Overview (pp. 3–4) describes interleaved conversation sessions across real-world topics, and §5.1 (p. 9) places the benchmark among evaluations of long-context memory. But §4.2 (p. 6) says the full interaction history is provided within the models’ context windows; the source does not support the claim that PERSONAMEM assesses memory beyond the context window. (LLM, conf=0.86)
Experimental — accuracy not yet measuredExisting dialogue benchmarks largely overlook non-declarative memory and lack a comprehensive set of assessment dimensions.
“Existing dialogue benchmarks (Kim et al., 2024; Wu et al., 2024; Jiang et al., 2025; Deshpande et al., 2025) largely overlook non-declarative memory and lack a comprehensive set of assessment dimensions, leaving important aspects of models' memory capabilities insufficiently evaluated.”
From EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memory (2026) · cites this paper as Jiang et al., 2025
Section 5.1 (p. 9) says earlier dialogue benchmarks cover narrower topics or assess general memory abilities rather than PERSONAMEM’s fine-grained personalization concerns. That supports a narrower claim about gaps in what those benchmarks assess, but the paper does not say they largely overlook non-declarative memory or lack a comprehensive set of assessment dimensions overall. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Existing benchmarks predominantly frame long-term memory evaluation as fact retrieval from past conversations, providing limited insight into agents' ability to consolidate memory over time or handle frequent knowledge updates.
“However, existing benchmarks predominantly frame long-term memory evaluation as fact retrieval from past conversations, providing limited insight into agents' ability to consolidate memory over time or handle frequent knowledge updates.”
From From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents (2026) · cites this paper as Jiang et al., 2025
Section 5.1 (p. 9) describes fact-finding as a common long-context test and contrasts prior memory benchmarks with PERSONAMEM’s finer-grained personalization evaluation. The Overview (p. 3) includes tracking evolving preferences, but the paper does not establish that existing benchmarks predominantly use fact retrieval from past conversations or specifically lack tests of memory consolidation and frequent knowledge updates. (LLM, conf=0.81)
Experimental — accuracy not yet measuredPersonaMem handles updates across no more than three sessions.
“LongMemEval (Wu et al., 2024) includes knowledge-update operations, but limits them to at most two sessions before evaluation, and PersonaMem (Jiang et al., 2025) handles updates across no more than three sessions.”
From From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents (2026) · cites this paper as Jiang et al., 2025
The PERSONAMEM Benchmark Overview (p. 3) says the benchmark tracks changing user preferences, so it does handle updates. But our copy does not establish a three-session limit: the Abstract (p. 1) describes histories of up to 60 sessions, and Appendix D.5.1 (p. 21) places reference information as far as 56–60 sessions from the query. Those figures do not specify how many sessions contain updates, so they do not by themselves prove the claimed limit false. (LLM, conf=0.82)
Experimental — accuracy not yet measuredPrior benchmarks place minimal stress on memory mutation.
“However, as shown in Table 1, prior benchmarks place minimal stress on memory mutation.”
From From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents (2026) · cites this paper as Jiang et al., 2025
Section 5.1 (p. 9) describes prior benchmarks as emphasizing retrieval, question answering, summarization, or general memory abilities, in contrast to PERSONAMEM’s fine-grained personalization focus. The benchmark overview (p. 3) confirms that PERSONAMEM tests changing preferences, but the paper does not establish that prior benchmarks place minimal stress on memory mutation specifically. (LLM, conf=0.74)
Experimental — accuracy not yet measuredExisting personalized-memory benchmarks primarily test factual retention or rely on weakly constrained model-generated trajectories, leaving event-driven preference adaptation underexplored.
“Existing personalized-memory benchmarks primarily test factual retention or rely on weakly constrained model-generated trajectories, leaving event-driven preference adaptation underexplored.”
From FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents (2026) · cites this paper as Jiang et al. 2025
Section 5.1 (p. 9) describes prior long-context benchmarks as testing fact retrieval and other general memory abilities, and says some use LLM-generated sessions. Section 5.2 (p. 10) distinguishes PERSONAMEM by its evolving preferences grounded in temporal events. The paper does not characterize earlier generated trajectories as weakly constrained, so that part of the claim is unsupported. (LLM, conf=0.85)
Experimental — accuracy not yet measuredExisting benchmarks emphasize factual retention, long-context recall, or explicit information updates.
“(G1) Limited event-conditioned state change.Existing benchmarks emphasize factual retention, long-context recall, or explicit information updates.”
From FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents (2026) · cites this paper as Jiang et al. 2025
Section 5.1 (p. 9) describes prior benchmarks centered on retrieving facts and evaluating memory over long contexts, supporting the first two parts of the claim. Section 5.2 (p. 10) contrasts PERSONAMEM with earlier personalization benchmarks, but does not establish that those benchmarks emphasize explicit information updates; the passages about preference updates describe PERSONAMEM itself. (LLM, conf=0.82)
Experimental — accuracy not yet measuredExisting benchmarks provide less control over how a user's latent preferences change after consequential events, making it difficult to separate recalling the past from reasoning with an updated user state.
“They provide less control over how a user's latent preferences change after consequential events, making it difficult to separate recalling the past from reasoning with an updated user state.”
From FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents (2026) · cites this paper as Jiang et al. 2025
Section 5.1 (p. 9) says earlier dialogue benchmarks focus on general memory, whereas PERSONAMEM evaluates fine-grained personalization. The benchmark overview (p. 3) describes preferences changing after life events, and Section 4.2 (p. 7) distinguishes recalling changes from using the latest preference. The paper does not explicitly establish that existing benchmarks offer less control over those changes or that this makes the two abilities difficult to separate. (LLM, conf=0.78)
Experimental — accuracy not yet measuredAll models score 4.5-11.2 points lower on anti-typical than typical personas, indicating a fallback to type-level stereotypes when individual evidence conflicts with a familiar profile.
“All models score 4.5-11.2points lower on anti-typical than typical personas, indicating a fallback to type-level stereotypes when individual evidence conflicts with a familiar profile (Jiang et al. 2025).”
From FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents (2026) · cites this paper as Jiang et al. 2025
The error analysis (Appendix F, p. 23) says GPT-4o often chose broadly reasonable responses over more personalized ones. It does not report a 4.5–11.2-point gap between anti-typical and typical personas, evaluate that comparison across all models, or identify type-level stereotypes as the cause. (LLM, conf=0.83)
Experimental — accuracy not yet measuredAnother line of work frames long-term memory primarily as personalization, aiming to adapt agents' behavior to individual users over extended interactions.
“In parallel, another line of work frames longterm memory primarily as personalization, aiming to adapt agents' behavior to the individual users over extended interactions.”
From From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents (2026) · cites this paper as Jiang et al., 2025
PersonaMem shifts toward personalized decision-making by testing whether models can infer evolving user states from long histories using multiple-choice questions.
“Per-sonaMem (Jiang et al., 2025) shifts toward personalized decision-making by testing whether models can infer evolving user states from long histories using multiple-choice questions.”
From From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents (2026) · cites this paper as Jiang et al., 2025
PersonaMem evaluates dynamic user profiling and personalized response selection across multi-session user-assistant interactions.
“PersonaMem [36] evaluates dynamic user profiling and personalized response selection across multi-session user-assistant interactions.”
From Beyond Similarity: Trustworthy Memory Search for Personal AI Agents (2026) · cites this paper as [36]
MemGate improves PreFEval and PersonaMem accuracy in most settings.
“MemGate improves PreFEval and PersonaMem accuracy in most settings, indicating that it does not weaken the agent's ability to use genuinely helpful user preferences.”
From Beyond Similarity: Trustworthy Memory Search for Personal AI Agents (2026) · cites this paper as [36]
Section 4.4 (p. 8) reports accuracy gains on PERSONAMEM from RAG and Mem0, not MemGate. Searches for MemGate and PreFEval found no passage assessing MemGate on either benchmark, so this copy cannot verify the claimed improvement. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 50
- Saleh Afzoon, Usman Naseem, Amin Beheshti, and Zahra Jamali. Persobench: Benchmarking personalized response generation in large language models. arXiv preprint arXiv:2410.03198 , 2024. · arXiv
- Abhishek Aggarwal, Cheuk Chi Tam, Dezhi Wu, Xiaoming Li, and Shan Qiao. Artificial intelligence-based chatbots for promoting health behavioral changes: systematic review. Journal of medical Internet research , 25:e40789, 2023.
- Tarek Ait Baha, Mohamed El Hajji, Youssef Es-Saady, and Hammou Fadili. The power of personalization: A systematic review of personality-adaptive chatbots. SN Computer Science , 4(5):661, 2023.
- David Castillo-Bolado, Joseph Davidson, Finlay Gray, and Marek Rosa. Beyond prompts: Dynamic conversational benchmarking of large language models. arXiv preprint arXiv:2409.20222 , 2024. · arXiv
- Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fr¨ anken, and Chelsea Finn. Persona: A reproducible testbed for pluralistic alignment. arXiv preprint arXiv:2407.17387 , 2024. · arXiv
- Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216 , 2024. · arXiv
- Sarah Dean and Jamie Morgenstern. Preference dynamics under personalized recommendations. In Proceedings of the 23rd ACM Conference on Economics and Computation , pp. 795-816, 2022.
- Yijiang River Dong, Tiancheng Hu, and Nigel Collier. Can llm be a personalized judge? arXiv preprint arXiv:2406.11657 , 2024. · arXiv
- Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094 , 2024. · arXiv
- Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. · arXiv
- Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025. · arXiv
- Kilem Li Gwet. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology , 61(1):29-48, 2008.
- Wenyue Hua, Lei Li, Shuyuan Xu, Li Chen, and Yongfeng Zhang. Tutorial on large language models for recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems , pp. 1281-1283, 2023.
- Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024. · arXiv
- Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo J Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. arXiv preprint arXiv:2406.11050 , 2024. · arXiv
- Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. Personallm: Investigating the ability of large language models to express personality traits. arXiv preprint arXiv:2305.02547 , 2023. · arXiv
- Jiho Kim, Woosog Chay, Hyeonji Hwang, Daeun Kyung, Hyunseung Chung, Eunbyeol Cho, Yohan Jo, and Edward Choi. Dialsim: A real-time simulator for evaluating long-term dialogue understanding of conversational agents. arXiv preprint arXiv:2406.13144 , 2024. · arXiv
- Hannah Rose Kirk, Alexander Whitefield, Paul R¨ ottger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. arXiv preprint arXiv:2404.16019 , 2024. · arXiv
- Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, et al. Longlamp: A benchmark for personalized long-form text generation. arXiv preprint arXiv:2407.11016 , 2024. · arXiv
- Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. In search of needles in a 10m haystack: Recurrent memory finds what llms miss. arXiv preprint arXiv:2402.10790 , 2024. · arXiv
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨ uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨ aschel, et al. Retrievalaugmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems , 33:9459-9474, 2020.
- Xintong Li, Jalend Bantupalli, Ria Dharmani, Yuwei Zhang, and Jingbo Shang. Toward multisession personalized conversation: A large-scale dataset and hierarchical tree framework for implicit reasoning. arXiv preprint arXiv:2503.07018 , 2025. · arXiv
- Ying-Chun Lin, Jennifer Neville, Jack Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, and Jaime Teevan. Interpretable user satisfaction estimation for conversational systems with large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1110011115, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.598. URL https://aclanthology.org/2024.acl-long.598/ . · DOI
- Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics , 12:157-173, 2024.
- Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of llm agents. arXiv preprint arXiv:2402.17753 , 2024. · arXiv
- Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longqi Yang, Bahareh Sarrafzadeh, Steve Menezes, Tina Baghaee, Emmanuel Barajas Gonzalez, Jennifer Neville, and Tara Safavi. Pearl: Personalizing large language model writing assistants with generation-calibrated retrievers. In Sachin Kumar, Vidhisha Balachandran, Chan Young Park, Weijia Shi, Shirley Anugrah Hayati, Yulia Tsvetkov, Noah Smith, Hannaneh Hajishirzi, Dongyeop Kang, and David Jurgens (eds.), Proceedings of the 1st Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U) , pp. 198-219, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.customnlp4u-1.16. URL https:// aclanthology.org/2024.customnlp4u-1.16/ . · DOI
- Elliot Nelson, Georgios Kollias, Payel Das, Subhajit Chaudhury, and Soham Dan. Needle in the haystack for memory based large language models. arXiv preprint arXiv:2407.01437 , 2024. · arXiv
- Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. Potato: The portable text annotation tool. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pp. 327-337, 2022.
- Filip Radlinski and Nick Craswell. A theoretical framework for conversational search. In Proceedings of the 2017 conference on conference human information interaction and retrieval , pp. 117-126, 2017.
- David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level googleproof q&a benchmark. In First Conference on Language Modeling , 2024.
- Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304.11406 , 2023. · arXiv
- Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning , pp. 29971-30004. PMLR, 2023.
- Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Jauhar, Sihao Chen, Shan Xia, et al. Wildfeedback: Aligning llms with in-situ user interactions and feedback. arXiv preprint arXiv:2408.15549 , 2024. · arXiv
- Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. Position: a roadmap to pluralistic alignment. In Proceedings of the 41st International Conference on Machine Learning , pp. 46280-46302, 2024.
- Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri` a Garriga-Alonso, et al. Beyond the imitation game: quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research , 2023(5):1-95, 2023.
- Juntao Tan, Liangwei Yang, Zuxin Liu, Zhiwei Liu, Rithesh Murthy, Tulika Manoj Awalgaonkar, Jianguo Zhang, Weiran Yao, Ming Zhu, Shirley Kokane, et al. Personabench: Evaluating ai models on understanding personal information through accessing (synthetic) private user data. arXiv preprint arXiv:2502.20616 , 2025. · arXiv
- Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 , 2024. · arXiv
- Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Yu-Ching Hsu, Jia-Yin Foo, Chao-Wei Huang, and Yun-Nung Chen. Two tales of persona in llms: A survey of role-playing and personalization. arXiv preprint arXiv:2406.01171 , 2024. · arXiv
- Yaqing Wang, Jiepu Jiang, Mingyang Zhang, Cheng Li, Yi Liang, Qiaozhu Mei, and Michael Bendersky. Automated evaluation of personalized text generation using large language models. arXiv preprint arXiv:2310.11593 , 2023. · arXiv
- Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Longmemeval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813 , 2024. · arXiv
- Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: a benchmark for real-world planning with language agents. In Proceedings of the 41st International Conference on Machine Learning , pp. 54590-54613, 2024a.
- Yangxinyu Xie, Bowen Jiang, Tanwi Mallick, Joshua David Bergerson, John K Hutchison, Duane R Verner, Jordan Branham, M Ross Alexander, Robert B Ross, Yan Feng, et al. Wildfiregpt: Tailored large language model for wildfire analysis. arXiv preprint arXiv:2402.07877 , 2024b. · arXiv
- J Xu. Beyond goldfish memory: Long-term open-domain conversation. arXiv preprint arXiv:2107.07567 , 2021. · arXiv
- Xiaoyue Xu, Qinyuan Ye, and Xiang Ren. Stress-testing long-context language models with lifelong icl and task haystack. arXiv preprint arXiv:2407.16695 , 2024. · arXiv
- Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9556-9567, 2024.
- Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. ∞ bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 15262-15277, 2024.
- Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do llms recognize your preferences? evaluating personalized preference following in llms. In The thirteenth international conference on learning representations , 2025.
- Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V Le, Ed H Chi, et al. Natural plan: Benchmarking llms on natural language planning. arXiv preprint arXiv:2406.04520 , 2024. · arXiv
- Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911 , 2023. · arXiv
- Thomas P Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li, and Hongseok Namkoong. Personalllm: Tailoring llms to individual preferences. arXiv preprint arXiv:2409.20296 , 2024. · arXiv
Cited by
Sorted by citations
From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
Uddin, M. N., Shubham, K., Blanco, E., Baral, C., & Wang, G. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1337 · Cited by 2 · see citing works
CAPTURE: Disentangling Preference Drift from Memory Poisoning in Personalized LLM Agents
Hossain, S. M. A., Shayoni, R. K., & Morol, M. K. (2026) · arXiv · doi:10.48550/arxiv.2609.02265
“…practical by paging information in and out of context (Packer et al., 2023), decaying it (Zhong et al., 2024), or reflecting on it (Park et al., 2023), and benchmarks now stress them at length (Maharana et al., 2024; Wu et al., 2025; Guo et al., 2026; Zhao et al., 2025; Jiang et al., 2025).”
FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents
Wang, B., Zhou, K., Guo, L., Chen, F., & Zhang, C. (2026) · arXiv · doi:10.48550/arxiv.2608.04095
From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
Uddin, M. N., Shubham, K., Blanco, E., Baral, C., & Wang, G. (2026) · arXiv · doi:10.48550/arxiv.2604.20006
“However, despite their apparent fluency, current PerLTQA (2024) 1.0 1.0 0.0 0.0 MemDaily (2024b) 3.3 8.0 0.0 0.0 LOCOMO (2024) 1.3 15.0 0.0 0.0 LongMemEval (2024) 1.9 6.0 2.0 2.0 PersonaMem (2025 be considered to answer a query, and memory mutation measures the number of updates or deletions applied across sessions be…”
InfluentialLongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues
Wu, D., Ji, Z., Kawatkar, A., Kwan, B., Gu, J.-C., Peng, N., & Chang, K.-W. (2026) · arXiv · doi:10.48550/arxiv.2605.12493
MemTrace: Probing What Final Accuracy Misses in Long-Term Memory
Long, X., Chen, Z., Zeng, S., Wang, S., Guo, K., & Tang, J. (2026) · arXiv · doi:10.48550/arxiv.2606.17328
Zhang, W., Wei, X., Huang, W.-C., Hui, Z., Wang, C., Gong, M., & Yu, P. S. (2026) · arXiv · doi:10.48550/arxiv.2603.25973
“(Zhao et al., 2025) 100K – ✗ ✓✗ ✓✗ ✗ LongMemEval (Wu et al., 2025a) 1.5M 500 ✓✗ ✓✗ ✗ ✗ PersonaMem (Jiang et al., 2025) 1M 60 ✓ ✓✗ ✓✗ ✗ MemBench (Tan et al., 2025a) 100K 0.5K-8K ✓✗ ✓✗ ✗ ✗ MemoryAgentBench (Hu et al., 2025) 1M – ✓✗ ✗ ✗ ✗ HaluMem (Chen et al., 2025) 1M 120 • We design an end-to-end…”
EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memory
Shen, Y., Pei, D., Guo, Y., Wang, J., Guo, Y., Zhang, Z., Jia, Q., Zhou, J., & Zhai, G. (2026) · arXiv · doi:10.48550/arxiv.2601.03543
“× PersonaMem (Jiang et al., 2025) ✓ ✓ Subj.”
InfluentialA Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052
Beyond Similarity: Trustworthy Memory Search for Personal AI Agents
Zhang, J., Chen, K., Ma, J., Hu, Y., He, L., Zhang, Y., Liu, J., Yang, X., Zhang, T., & Jia, R. (2026) · arXiv · doi:10.48550/arxiv.2606.06054
Transparent, Traceable, Deterministic: Agentic Memory via Knowledge Graphs
Gentile, A. L., An, S., & DeLuca, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-42029-9_16
- Authors
- Jiang, B., Hao, Z., Cho, Y.-M., Li, B., Yuan, Y., Chen, S., Ungar, L., Taylor, C. J., & Roth, D.
- ORCID
- Ungar, Lyle
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2504.14225
- OpenAlex
- W6891960188
- Cited by
- 1
- References
- 0
- Citations by year
- 2026: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- large language models, personalized responses, user profiling, dynamic user preferences, LLM benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
