CoChat

Memory-Augmented Architecture for Long-Term Context Handling in Large Language Models

Open access
Shinwari, Haseeb Ullah Khan, Usama, Muhammad · 2026 · IEEE Transactions on Artificial Intelligence

Published version of a preprint

The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index; 8 of the 10 sources its claims cite were checked for support.

Abstract

Large Language Models struggle to maintain coherent, contextually-aware interactions over long dialogues due to the inherent limitations of fixed-length context windows. This leads to fragmented conversations, loss of essential information, and diminished user experience. We address this challenge by introducing a memory-augmented architecture with a dynamic, relevance-based pruning mechanism that evaluates and manages stored information based on its semantic importance to the ongoing conversation, rather than relying on simplistic temporal heuristics. Our approach enables intelligent retrieval, updating, and pruning of memory, ensuring that contextually vital information is retained. Extensive experiments across eight benchmarks demonstrate that our method improves long-term contextual coherence, achieves a 7.6% relative accuracy gain, and reduces memory overhead by 24.6% compared to state-of-the-art baselines.

Overview

CoChat Summary

TL;DR

The paper proposes a memory-augmented architecture for LLMs that retrieves past interactions by cosine similarity and manages the memory store with relevance-based pruning rather than LRU eviction. Across the 20 Questions game, Persona-Chat, and DailyDialog, the approach improves accuracy and contextual coherence while reducing memory overhead and latency relative to no pruning.

Key findings

  • On the 20 Questions game, memory augmentation raised accuracy from 62.3% to 80.4% for LLAMA 3 8B and from 64.8% to 82.1% for Gemma 2 9B, with PTR of 35.2% and 36.8% respectively (baseline PTR 0.0%).
  • Relevance-based pruning outperformed LRU eviction: for LLAMA 3 8B, 80.4% accuracy, 1022.7 MB memory overhead, and 1287.8 ms latency versus LRU's 76.5%, 1030.4 MB, and 1295.6 ms; for Gemma 2 9B, 82.1%, 1173.5 MB, and 1137.3 ms versus LRU's 78.2%, 1103.4 MB, and 1261.4 ms.
  • Relevance-based pruning also beat no pruning on efficiency: LLAMA 3 8B went from 1250.0 MB / 1400.3 ms (no pruning, 78.1% accuracy) to 1022.7 MB / 1287.8 ms; Gemma 2 9B from 1320.5 MB / 1375.2 ms (no pruning, 82.6% accuracy) to 1173.5 MB / 1137.3 ms.
  • Embedding choice mattered: GTE-large gave the highest accuracy (80.4% LLAMA 3 8B, 82.1% Gemma 2 9B), ahead of MiniLM-L6-v2 (75.8%, 77.3%) and Universal Sentence Encoder (73.2%, 74.9%). Gemma 2 9B reached the highest CCS (0.83) on Persona-Chat.

Methods

  • Design: a modular memory-augmented framework that embeds each query with an encoder, retrieves the memory entry with highest cosine similarity to condition response generation, then appends the encoded query–response pair as a new memory entry; when the store exceeds size N, either LRU eviction or relevance-based pruning (removing the entry with lowest maximum similarity to the last T queries) is applied.
  • Data: three tasks — a custom 20 Questions dataset of 100 entities across ten categories, Persona-Chat, and DailyDialog (over 13,000 dialogues).
  • Analysis: baseline versus memory-augmented LLAMA 3 8B and Gemma 2 9B under identical hardware (Intel Xeon 3.4 GHz, 64 GB RAM, NVIDIA RTX 3090, PyTorch), using GTE-large as the embedding network; metrics were accuracy, Contextual Coherence Score, memory overhead (MB), latency (ms), and Positive Transferability Ratio.

Limitations

  • The paper states no explicit limitations section; no limitations are acknowledged in the text.
  • Evident from the text: baseline latency and memory overhead are reported only as dashes in Table I, so the efficiency comparison against the un-augmented baseline is not quantified; Table II is partially garbled and its DailyDialog rows are incomplete, making some CCS/PTR values hard to verify; no statistical significance testing or variance across runs is reported; evaluation is confined to two open-weight models and one hardware configuration, and the 20 Questions dataset is a custom 100-entity set.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tai.2026.3701680

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

5 Valid · 2 Not valid · 0 Not checked · 22 Own results and interpretations

13 more claims

Create a free account to see all 29 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 10
  • Probably not a substantial claim. 3
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Citations

Cites

Show all 30 references

Cited by

  • In-Context Learning can Perform Continual Learning Like HumansPreprint

    Kang, L., Wang, F., Liu, S., Chou, H.-C., Lin, C., & Ding, N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.22764 · Cited by 2

    “Several recent studies have begun to explore directions related to ICCL, often by extending ICL with prompt modification (e.g., prompt tuning) or external augmentation (e.g., memory retrieval) [Wang et al., 2022, Gao et al., 2023, Momeni et al., 2024, Shinwari and Usama, 2025].”

See all 1

Metadata

Authors
Shinwari, H. U. K., & Usama, M.
Affiliations
  • San Francisco Bay University (US)
  • Korea Advanced Institute of Science and Technology (KR)
Venue
IEEE Transactions on Artificial Intelligence
Published
2026-10
Volume
7
Issue
10
Pages
5830-5837
ISSN
2691-4581 (ISSN-L)
OpenAlex
W7164000559
Cited by
0
References
30
FWCI
0.00
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, dialogue coherence, dynamic memory management, semantic memory retrieval, memory cost reduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Memory-Augmented Architecture for Long-Term Context Handling in Large Language Models. https://app.cochat.ai/papers/10.1109/tai.2026.3701680
The paper
Shinwari, H. U. K., & Usama, M. (2026). Memory-Augmented Architecture for Long-Term Context Handling in Large Language Models. IEEE Transactions on Artificial Intelligence. https://doi.org/10.1109/tai.2026.3701680

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.