CoChat

CAIM: Development and Evaluation of a Cognitive AI Memory Framework for Long-Term Interaction with Intelligent Agents

Open access
Westhäußer, Rebecca, Minker, Wolfgang, Zepf, Sebastian · 2026 · Proceedings of the 31st International Conference on Intelligent User Interfaces

The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index; 39 of the 58 sources its claims cite were checked for support.

Abstract

Large language models (LLMs) have advanced the field of artificial intelligence (AI) and are a powerful enabler for interactive systems. However, they still face challenges in long-term interactions that require adaptation towards the user as well as contextual knowledge and an understanding of the ever-changing environment. To overcome these challenges, holistic memory modeling is required to efficiently retrieve and store relevant information across sessions for accurate responses. Cognitive AI, which aims to simulate the human thought process in a computerized model, highlights interesting aspects, such as thoughts, memory mechanisms, and decision making, that can contribute towards improved memory modeling for LLMs. Inspired by these principles, we propose CAIM, a cognitive AI memory framework that models key aspects of human memory through a multi-agent architecture. Specialized LLM-based agents handle memory-related functions such as retrieval, contextual relevance evaluation, and memory maintenance. We compare CAIM against existing approaches, focusing on metrics such as retrieval accuracy, response correctness, and contextual coherence. The results demonstrate that CAIM outperforms baseline frameworks in different metrics, highlighting its context awareness and demonstrating its contribution to memory mechanisms for improving long-term human-AI interactions.

Overview

CoChat Summary

TL;DR

CAIM is a cognitive-AI-inspired memory framework that uses eight specialized LLM-based agents to decide when to retrieve, filter, and maintain long-term memory for LLM agents in long-term interaction. Across the GVD and LoCoMo benchmarks it outperforms MemoryBank and Think-in-Memory on retrieval accuracy, response correctness, and contextual coherence, though it struggles with detailed queries and relative time references.

Key findings

  • On GVD, CAIM reached 83.3% retrieval accuracy with GPT-3.5 turbo (vs. MemoryBank's 76.3%) and 88.7% with GPT-4o; with GLM it dropped to 67.6%, below TiM (82%) and MemoryBank (80.9%) because GLM produced Chinese tags and failed to follow the required tag format.
  • Response correctness improved across all models: GLM 60% (vs. TiM 45%, MemoryBank 43.8%), GPT-3.5 turbo 81.3% (vs. MemoryBank 71.6%), GPT-4o 87.5%. Contextual coherence also improved (GLM 86.5% vs. TiM 73.5% and MemoryBank 68%; GPT-3.5 turbo 98.3% vs. MemoryBank 91.2%).
  • On LoCoMo with GPT-4o, CAIM scored 66.8% retrieval accuracy, 63.8% response correctness, and 89.2% contextual coherence, versus MemoryBank's 53.9%, 42.6%, and 65.3%.
  • Ablation (GPT-4o, GVD): removing the Controller Agent left retrieval accuracy similar (85.5% vs. 87.5%) but dropped response correctness to 78.2%; removing the Filtering Agent cut retrieval accuracy to 64.3% and response correctness to 63.5%, while contextual coherence stayed at 90.2%.

Methods

  • Design: CAIM combines MemoryBank, Think-in-Memory, and Self-Controlled Memory concepts into a multi-agent architecture with STM/LTM, an LLM-generated three-level JSON ontology for tagging, and eight LLM-based agents (Controller, Tagging, Time, Filtering, Response Generator, Summary, Memory, Review).
  • Data: the public Generated Virtual Dataset (GVD; 15 users, ten days, 100 English test questions) compared against MemoryBank and TiM, and LoCoMo (ten long-term conversations) compared against MemoryBank; SCM was excluded due to incompatible datasets and metrics.
  • Analysis: fact-checking evaluation with three human annotators (GLM, GPT-3.5 turbo, GPT-4o; default parameters) using Retrieval Accuracy, Response Correctness, and Contextual Coherence (0/0.5/1 scales), with Percent Agreement and ICC for reliability; GPT-4o served as automatic evaluator for LoCoMo after validation against human ratings on GVD (93%/81%/94% agreement).

Limitations

  • Stated: latency and computational cost of the multi-agent architecture and ontology-based processing were intentionally not evaluated; no long-term user study was conducted (experiments are on synthetic data); tag-based retrieval is fragile across LLMs; memory conciseness limits detailed queries; relative time references and split information across inductive thoughts cause errors; occasional hallucinations when relevant information is missing.
  • Evident from the text: the GLM results show CAIM's effectiveness depends on a model's ability to follow predefined tag formats, and the low ICC values for Contextual Coherence on GPT models (5.8% and −0.6%) indicate that metric's reliability is questionable under low rating variance.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3742413.3789222

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

21 Valid · 3 Not valid · 7 Not checked · 104 Own results and interpretations

48 more claims · 3 with their cited sources checked

Create a free account to see all 135 claims and to verify the 5 references still open.

See all claims — free account
  • It may state two separate claims as one. 58
  • Probably not a substantial claim. 35

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 71 references

Research works printed but not linked: 2

  1. Shweta Bajpai Ram Bajpai and Himanshu Chaturvedi. 2015. Evaluation of inter-rater agreement and inter-rater reliability for observational data: an overview of concepts and methods. Journal of the Indian Academy of Applied Psychology 41 3 (2015) 20–27.
  2. 10.5555/3169338 · doi:10.5555/3169338

Cited by

See all 3

Metadata

Authors
Westhäußer, R., Minker, W., & Zepf, S.
Affiliations
  • Mercedes-Benz (Germany) (DE)
  • Universität Ulm (DE)
  • Human-Computer Interaction, Mercedes-Benz AG, Böblingen, Germany
Venue
Proceedings of the 31st International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2026-03-22
Pages
134-143
OpenAlex
W7133361058
Cited by
1
References
34
Citations by year
2026: 1
FWCI
8.44
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
Keywords
cognitive AI, memory modeling, large language models, multi-agent architecture, contextual relevance, memory retrieval, memory maintenance, context awareness, intelligent agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: CAIM: Development and Evaluation of a Cognitive AI Memory Framework for Long-Term Interaction with Intelligent Agents. https://app.cochat.ai/papers/10.1145/3742413.3789222
The paper
Westhäußer, R., Minker, W., & Zepf, S. (2026). CAIM: Development and Evaluation of a Cognitive AI Memory Framework for Long-Term Interaction with Intelligent Agents. Proceedings of the 31st International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3742413.3789222

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.