CoChat
Sign Up
PreprintCited by 4Topic ModelingMultimodal Machine Learning ApplicationsMemory Processes and Influences

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

Episodic Memories Generation and Evaluation Benchmark for Large Language Models

Open access
Huet, Alexis, Houidi, Zied Ben, Rossi, Dario·2025·arXiv
DOI 10.48550/arxiv.2501.13121arXiv 2501.13121
Open this paper in CoChat — free accountOpen PDF

Abstract

Episodic memory -- the ability to recall specific events grounded in time and space -- is a cornerstone of human cognition, enabling not only coherent storytelling, but also planning and decision-making. Despite their remarkable capabilities, Large Language Models (LLMs) lack a robust mechanism for episodic memory: we argue that integrating episodic memory capabilities into LLM is essential for advancing AI towards human-like cognition, increasing their potential to reason consistently and ground their output in real-world episodic events, hence avoiding confabulations. To address this challenge, we introduce a comprehensive framework to model and evaluate LLM episodic memory capabilities. Drawing inspiration from cognitive science, we develop a structured approach to represent episodic events, encapsulating temporal and spatial contexts, involved entities, and detailed descriptions. We synthesize a unique episodic memory benchmark, free from contamination, and release open source code and datasets to assess LLM performance across various recall and episodic reasoning tasks. Our evaluation of state-of-the-art models, including GPT-4 and Claude variants, Llama 3.1, and o1-mini, reveals that even the most advanced LLMs struggle with episodic memory tasks, particularly when dealing with multiple related events or complex spatio-temporal relationships -- even in contexts as short as 10k-100k tokens.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2501.13121

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Systems like Xu et al. and Gutiérrez et al. attempt to address cross-session reasoning through linked structures, but still struggle with tracking changing preferences and resolving contradictions.

    “Although systems like Xu et al. (2025) and Gutiérrez et al. (2025) attempt to address this through linked structures, they still struggle with tracking changing preferences and resolving contradictions (Huet et al., 2025; Wu et al., 2025a).”

    From Pre-Storage Reasoning for Episodic Memory: Shifting Inference Burden to Memory for Personalized Dialogue (2025) · cites this paper as Huet et al., 2025

    The “Impact of cue type” section reports that the models tested struggled to track entity states over time. But the paper does not establish that Xu et al.’s or Gutiérrez et al.’s linked-structure systems struggle with changing preferences or resolving contradictions. Section A.1 explicitly leaves cognitive dissonance for future work. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The Episodic Memory Benchmark provides a principled framework for evaluating episodic memory capabilities.

    “For evaluation, we refer to the Episodic Memory Benchmark (Huet, Houidi, and Rossi 2025), which provides a principled framework for evaluating episodic memory capabilities.”

    From ARTEM: Enhancing Large Language Model Agents with Spatial-Temporal Episodic Memory (2026) · cites this paper as Huet, Houidi, and Rossi 2025

  • The Episodic Memory Benchmark reveals significant limitations in current large language models regarding temporal coherence and confabulation avoidance.

    “The Episodic Memory Benchmark (Huet, Houidi, and Rossi 2025) establishes standardized evaluation for episodic memory capabilities, revealing significant limitations in current LLMs regarding temporal coherence and confabulation avoidance.”

    From ARTEM: Enhancing Large Language Model Agents with Spatial-Temporal Episodic Memory (2026) · cites this paper as Huet, Houidi, and Rossi 2025

  • Huet et al. (2025) proposes a specialized benchmark to assess how LLMs generate and manage memories that capture specific events with contextual details, using synthetically created book chapters and events with LLM-based judge evaluation metrics to measure accuracy and relevance.

    “For episodic memory evaluation, (Huet et al., 2025) proposes a specialized benchmark to assess how LLMs generate and manage memories that capture specific events with contextual details. This benchmark utilizes synthetically created book chapters and events with LLMs-based judge evaluation metrics to measure accuracy and relevance.”

    From Survey on Evaluation of LLM-based Agents (2025) · cites this paper as Huet et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018, 2024. · arXiv
  2. Cristina M Atance and Daniela K O'Neill. Episodic future thinking. Trends in cognitive sciences, 5 (12):533-539, 2001.
  3. Helen C Barron, Raymond J Dolan, and Timothy EJ Behrens. Online evaluation of novel choices by simultaneous representation of multiple memories. Nature neuroscience, 16(10):1492-1498, 2013.
  4. Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. https://arxiv.org/pdf/2004.05150.pdf. · arXiv
  5. Alessio Benavoli, Giorgio Corani, and Francesca Mangili. Should we really use post-hoc tests based on mean-ranks? The Journal of Machine Learning Research, 17(1):152-161, 2016.
  6. Yael Benn, Ofer Bergman, Liv Glazer, Paris Arent, Iain D Wilkinson, Rosemary Varley, and Steve Whittaker. Navigating through digital folders uses the same brain structures as real world navigation. Scientific reports, 5(1):14719, 2015.
  7. Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. arXiv preprint arXiv:2304.11158, 2023. https://arxiv.org/abs/2304.11158. · arXiv
  8. Bernd Bohnet, Kevin Swersky, Rosanne Liu, Pranjal Awasthi, Azade Nova, Javier Snaider, Hanie Sedghi, Aaron T Parisi, Michael Collins, Angeliki Lazaridou, et al. Long-span question-answering: Automatic question generation and qa-system ranking via side-by-side evaluation. arXiv preprint arXiv:2406.00179, 2024. · arXiv
  9. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collabora- tively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1247-1250, 2008.
  10. Neil Burgess, Eleanor A Maguire, and John O'Keefe. The human hippocampus and spatial and episodic memory. Neuron, 35(4):625-641, 2002.
  11. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evalu- ating and testing unintended memorization in neural networks. In USENIX Security Symposium, volume 267, 2019.
  12. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021. https://arxiv.org/abs/2012.07805. · arXiv
  13. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. https://arxiv.org/abs/2202.07646. · arXiv
  14. Wenhu Chen, Xinyi Wang, and William Yang Wang. A dataset for answering time-sensitive questions. arXiv preprint arXiv:2108.06314, 2021. · arXiv
  15. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. https://arxiv.org/abs/1904. 10509. · arXiv
  16. Laura A Cushman, Karen Stein, and Charles J Duffy. Detecting navigational deficits in cognitive aging and alzheimer disease using virtual reality. Neurology, 71(12):888-895, 2008.
  17. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696, 2021. https://arxiv.org/ abs/2104.08696. · arXiv
  18. Payel Das, Subhajit Chaudhury, Elliot Nelson, Igor Melnyk, Sarath Swaminathan, Sihui Dai, Aurélie Lozano, Georgios Kollias, Vijil Chenthamarakshan, Soham Dan, et al. Larimar: Large language models with episodic memory control. arXiv preprint arXiv:2403.11901, 2024. · arXiv
  19. Felipe De Brigard, Sharda Umanath, and Muireann Irish. Rethinking the distinction between episodic and semantic memory: Insights from the past, present, and future. Memory & Cognition, 50(3): 459-463, 2022.
  20. Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
  21. Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257-273, 2022.
  22. Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486, 2023. https://arxiv.org/pdf/2307.02486.pdf. · arXiv
  23. Jeffery A Dusek and Howard Eichenbaum. The hippocampus and memory for orderly stimulus relations. Proceedings of the National Academy of Sciences, 94(13):7109-7114, 1997.
  24. Leon Festinger. A theory of cognitive dissonance, volume 2. Stanford university press, 1962.
  25. Zafeirios Fountas, Martin A Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lam- pouras, Haitham Bou-Ammar, and Jun Wang. Human-like episodic memory for infinite context llms. arXiv preprint arXiv:2407.09450, 2024. · arXiv
  26. Ankit Gupta and Jonathan Berant. Gmat: Global memory augmentation for transformers. arXiv preprint arXiv:2006.03274, 2020. https://arxiv.org/pdf/2006.03274.pdf. · arXiv
  27. Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard I Moser. Microstructure of a spatial map in the entorhinal cortex. Nature, 436(7052):801-806, 2005.
  28. Joses Ho, Tayfun Tumkaya, Sameer Aryal, Hyungwon Choi, and Adam Claridge-Chang. Moving beyond p values: data analysis with estimation graphics. Nature methods, 16(7):565-566, 2019.
  29. Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282, 2020. https://arxiv.org/ abs/2007.01282. · arXiv
  30. Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. Tem- pquestions: A benchmark for temporal question answering. In Companion Proceedings of the The Web Conference 2018, pp. 1057-1062, 2018.
  31. Marcia K Johnson and Carol L Raye. Reality monitoring. Psychological review, 88(1):67, 1981.
  32. Marcia K Johnson, Shahin Hashtroudi, and D Stephen Lindsay. Source monitoring. Psychological bulletin, 114(1):3, 1993.
  33. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. arXiv preprint arXiv:2211.08411, 2022. https: //arxiv.org/abs/2211.08411. · arXiv
  34. Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. Realtime qa: what's the answer right now? Advances in Neural Information Processing Systems, 36, 2024.
  35. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172, 2019. https://arxiv.org/abs/1911.00172. · arXiv
  36. Eugene Kharitonov, Marco Baroni, and Dieuwke Hupkes. How bpe affects memorization in transform- ers. arXiv preprint arXiv:2110.02782, 2021. https://arxiv.org/abs/2110.02782. · arXiv
  37. Michael D Kopelman. The autobiographical memory interview (ami) in organic and psychogenic am- nesia. Memory, 2(2):211-235, 1994. https://pubmed.ncbi.nlm.nih.gov/7584292/.
  38. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. arXiv preprint arXiv:2406.10149, 2024. · arXiv
  39. Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. Learning dense representations of phrases at scale. arXiv preprint arXiv:2012.12624, 2020. https://arxiv.org/abs/2012.12624. · arXiv
  40. Brian Levine, Eva Svoboda, Janine F Hay, Gordon Winocur, and Morris Moscovitch. Aging and autobiographical memory: dissociating episodic from semantic retrieval. Psychology and aging, 17(4):677, 2002.
  41. Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963, 2024. · arXiv
  42. Christopher J MacDonald, Kyle Q Lepage, Uri T Eden, and Howard Eichenbaum. Hippocampal "time cells" bridge the gap in memory for discontiguous events. Neuron, 71(4):737-749, 2011.
  43. George Mandler. Recognizing: The judgment of previous occurrence. Psychological review, 87(3): 252, 1980.
  44. Pedro Henrique Martins, Zita Marinho, and André FT Martins. ∞-former: Infinite memory trans- former. arXiv preprint arXiv:2109.00301, 2021. https://arxiv.org/pdf/2109.00301. pdf. · arXiv
  45. Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. Sources of hallucination by large language models on inference tasks. arXiv preprint arXiv:2305.14552, 2023. https://arxiv.org/abs/2305.14552. · arXiv
  46. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022a.
  47. Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. arXiv preprint arXiv:2306.07174, 2023. https://arxiv.org/abs/2306.07174. · arXiv
  48. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015. · arXiv
  49. Yuxiang Wu, Yu Zhao, Baotian Hu, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. An efficient memory-augmented transformer for knowledge-intensive nlp tasks. arXiv preprint arXiv:2210.16773, 2022. https://arxiv.org/abs/2210.16773. · arXiv
  50. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. https://arxiv. org/pdf/2309.17453.pdf. · arXiv
  51. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. Infinite bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15262-15277, 2024.
  52. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020. https://arxiv.org/pdf/2012.00363.pdf. Preprint Follow these guidelines: Structure and Information Reveal: 1. Divide the text into {nb_paragraphs} paragraph(s). Number each paragraph (1), (2), etc., while maintaining novel-appropriate paragraph lengths. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Huet, A., Houidi, Z. B., & Rossi, D.
Affiliations
Huawei Technologies (France) (FR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2501.13121
arXiv
2501.13121
OpenAlex
W4406785183
Cited by
3
References
2
Citations by year
2026: 1 · 2025: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
Keywords
episodic memory, large language models, spatio-temporal context, confabulation, event representation, spatio-temporal relations, GPT-4, Llama 3.1

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review