Memory-Augmented Architecture for Long-Term Context Handling in Large Language Models
Open accessPublished version of a preprint
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index; 8 of the 10 sources its claims cite were checked for support.
Abstract
Large Language Models struggle to maintain coherent, contextually-aware interactions over long dialogues due to the inherent limitations of fixed-length context windows. This leads to fragmented conversations, loss of essential information, and diminished user experience. We address this challenge by introducing a memory-augmented architecture with a dynamic, relevance-based pruning mechanism that evaluates and manages stored information based on its semantic importance to the ongoing conversation, rather than relying on simplistic temporal heuristics. Our approach enables intelligent retrieval, updating, and pruning of memory, ensuring that contextually vital information is retained. Extensive experiments across eight benchmarks demonstrate that our method improves long-term contextual coherence, achieves a 7.6% relative accuracy gain, and reduces memory overhead by 24.6% compared to state-of-the-art baselines.
Overview
CoChat Summary
TL;DR
The paper proposes a memory-augmented architecture for LLMs that retrieves past interactions by cosine similarity and manages the memory store with relevance-based pruning rather than LRU eviction. Across the 20 Questions game, Persona-Chat, and DailyDialog, the approach improves accuracy and contextual coherence while reducing memory overhead and latency relative to no pruning.
Key findings
- On the 20 Questions game, memory augmentation raised accuracy from 62.3% to 80.4% for LLAMA 3 8B and from 64.8% to 82.1% for Gemma 2 9B, with PTR of 35.2% and 36.8% respectively (baseline PTR 0.0%).
- Relevance-based pruning outperformed LRU eviction: for LLAMA 3 8B, 80.4% accuracy, 1022.7 MB memory overhead, and 1287.8 ms latency versus LRU's 76.5%, 1030.4 MB, and 1295.6 ms; for Gemma 2 9B, 82.1%, 1173.5 MB, and 1137.3 ms versus LRU's 78.2%, 1103.4 MB, and 1261.4 ms.
- Relevance-based pruning also beat no pruning on efficiency: LLAMA 3 8B went from 1250.0 MB / 1400.3 ms (no pruning, 78.1% accuracy) to 1022.7 MB / 1287.8 ms; Gemma 2 9B from 1320.5 MB / 1375.2 ms (no pruning, 82.6% accuracy) to 1173.5 MB / 1137.3 ms.
- Embedding choice mattered: GTE-large gave the highest accuracy (80.4% LLAMA 3 8B, 82.1% Gemma 2 9B), ahead of MiniLM-L6-v2 (75.8%, 77.3%) and Universal Sentence Encoder (73.2%, 74.9%). Gemma 2 9B reached the highest CCS (0.83) on Persona-Chat.
Methods
- Design: a modular memory-augmented framework that embeds each query with an encoder, retrieves the memory entry with highest cosine similarity to condition response generation, then appends the encoded query–response pair as a new memory entry; when the store exceeds size N, either LRU eviction or relevance-based pruning (removing the entry with lowest maximum similarity to the last T queries) is applied.
- Data: three tasks — a custom 20 Questions dataset of 100 entities across ten categories, Persona-Chat, and DailyDialog (over 13,000 dialogues).
- Analysis: baseline versus memory-augmented LLAMA 3 8B and Gemma 2 9B under identical hardware (Intel Xeon 3.4 GHz, 64 GB RAM, NVIDIA RTX 3090, PyTorch), using GTE-large as the embedding network; metrics were accuracy, Contextual Coherence Score, memory overhead (MB), latency (ms), and Positive Transferability Ratio.
Limitations
- The paper states no explicit limitations section; no limitations are acknowledged in the text.
- Evident from the text: baseline latency and memory overhead are reported only as dashes in Table I, so the efficiency comparison against the un-augmented baseline is not quantified; Table II is partially garbled and its DailyDialog rows are incomplete, making some CCS/PTR values hard to verify; no statistical significance testing or variance across runs is reported; evaluation is confined to two open-weight models and one hardware configuration, and the 20 Questions dataset is a custom 100-entity set.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 2 Not valid · 0 Not checked · 22 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“CAMELoT [4] introduces an associative memory module to improve context handling within LLMs.”
What the sources say
- Supportedcites [4] source
Section 3 describes CAMELoT as a base language model paired with an associative memory module that retrieves information to enhance the current context window. The Introduction explains that the module retains information from prior context beyond that window. (LLM, conf=0.94)
states a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, these approaches often rely on LRU eviction for memory management, which may not be optimal for retaining highly relevant information.”
What the sources say
- Could not verifycites [3] Barkley, What is long context — and why does it matter for your AI? sourceExperimental — accuracy not yet measured
Section 2.1 (p. 4) says LONGMEM removes the oldest sequences from its memory bank, but does not describe this as LRU eviction. The Introduction (pp. 1–2) discusses memory staleness instead. Searches for LRU, least-recently-used eviction, and cache replacement found no passage saying existing approaches often use LRU or that it fails to retain highly relevant information. (LLM, conf=0.74)
- Partly supportedcites [4] sourceExperimental — accuracy not yet measured
Section 4.1.1 (page not available) says the Memorizing Transformer replaces older caches with newer ones as its memory fills. Section 4.1.2 says this can overwrite older memories and limit long-term utility. The paper does not establish that memory-augmented approaches often use LRU eviction specifically; the example it describes uses a circular cache. (LLM, conf=0.84)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This limitation can result in responses that lack coherence, contextual relevance, and even lead to hallucinations in extended dialogues or complex tasks [1], [2].”
What the sources say
- Partly supportedcites [1] Howard, 2024, Context degradation syndrome: When large language models lose the plot sourceExperimental — accuracy not yet measured
The Main section and Abstract say that LLMs can hallucinate, producing false or unsubstantiated answers. They do not link those errors—or a lack of coherence or contextual relevance—to difficulty maintaining continuity in extended dialogues or complex tasks. The indexed passages provide section labels but no page numbers. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [2] Science, Ai chatbots (chatgpt) bad at remembering things — have scientists just cracked their terrible memory problem? source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MiniLM-L6-v2, while more computationally efficient as evidenced by lower latency times, shows a moderate decrease in accuracy (75.8% for LLAMA 3 8B and 77.3% for Gemma 2 9B).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The memory-augmented versions of LLAMA 3 8B and Gemma 2 9B show substantial accuracy gains over the baseline, with LLAMA 3 8B improving from 62.3% to 80.4% and Gemma 2 9B from 64.8% to 82.1%.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
13 more claims
Create a free account to see all 29 claims and every source verdict.
- It may state two separate claims as one. 10
- Probably not a substantial claim. 3
- The quote only partly states this claim. 1
Citations
Cites
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1×
“All experiments were conducted on NVIDIA GeForce RTX 5090 GPUs.”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 3×
“Retrieval-augmented generation (RAG) [11] augments LLMs with external retrieval, enhancing factuality but lacking dynamic, interaction-specific memory management.”
Universal Sentence Encoder for English
Cer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2029 · Cited by 1,750
Cer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · arXiv · doi:10.48550/arxiv.1803.11175 · Cited by 1,267 · cited 1×
“%) and simpler semantic encoders such as the Universal Sentence Encoder [22] (74.9%).”
InfluentialPersonalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177 · cited 1×
“These include LongMemEval [13] and the 20 Questions Game, which measure long-term memory recall and cumulative reasoning; Persona-Chat [14] and DailyDialog [15], Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
Detecting hallucinations in large language models using semantic entropy
Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024) · Nature · doi:10.1038/s41586-024-07421-0 · Cited by 934 · cited 1×
“The resulting “context horizon” problem manifests as fragmented responses, contradictions, and an inability to track user preferences or maintain persistent dialogue state.”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703
DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
Li, Y., Su, H., Shen, X., Li, W., Cao, Z., & Niu, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.03957 · Cited by 646 · cited 1×
“These include LongMemEval [13] and the 20 Questions Game, which measure long-term memory recall and cumulative reasoning; Persona-Chat [14] and DailyDialog [15], Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024) · arXiv · doi:10.48550/arxiv.2410.10813 · Cited by 493 · cited 1×
“These include LongMemEval [13] and the 20 Questions Game, which measure long-term memory recall and cumulative reasoning; Persona-Chat [14] and DailyDialog [15], Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
Cited by
In-Context Learning can Perform Continual Learning Like HumansPreprint
Kang, L., Wang, F., Liu, S., Chou, H.-C., Lin, C., & Ding, N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.22764 · Cited by 2
“Several recent studies have begun to explore directions related to ICCL, often by extending ICL with prompt modification (e.g., prompt tuning) or external augmentation (e.g., memory retrieval) [Wang et al., 2022, Gao et al., 2023, Momeni et al., 2024, Shinwari and Usama, 2025].”
Metadata
- Authors
- Shinwari, H. U. K., & Usama, M.
- Affiliations
- San Francisco Bay University (US)
- Korea Advanced Institute of Science and Technology (KR)
- Venue
- IEEE Transactions on Artificial Intelligence
- Published
- 2026-10
- Volume
- 7
- Issue
- 10
- Pages
- 5830-5837
- ISSN
- 2691-4581 (ISSN-L)
- OpenAlex
- W7164000559
- Cited by
- 0
- References
- 30
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, dialogue coherence, dynamic memory management, semantic memory retrieval, memory cost reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Memory-Augmented Architecture for Long-Term Context Handling in Large Language Models. https://app.cochat.ai/papers/10.1109/tai.2026.3701680
- The paper
- Shinwari, H. U. K., & Usama, M. (2026). Memory-Augmented Architecture for Long-Term Context Handling in Large Language Models. IEEE Transactions on Artificial Intelligence. https://doi.org/10.1109/tai.2026.3701680
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.