CoChat

From Organizational Knowledge to AI Agent Memory: Empirical Validation of the SECI Model on the LongMemEval Benchmark

Open access
Żatuchin, Dmitrij · 2026 · Research Square (preprint)

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 7 of the 27 sources its claims cite were checked for support.

Abstract

AI agent memory systems face a fundamental design question: should conversational history be stored verbatim or transformed into structured knowledge before retrieval? We propose that the SECI model of knowledge creation \citep{nonaka1995knowledge} provides empirically testable design principles for AI agent memory architecture. We adapt the four SECI phases to the AI agent context and evaluate structured extraction against raw storage baselines on LongMemEval \citep{wu2025longmemeval}. Across 500 questions spanning six memory task types, SECI-based extraction achieves 93.9% Recall@5 versus 92.1% for raw storage, both using the same embedding model and vector database. The dominant factor is embedding window truncation: the model processes 256 tokens maximum, while sessions average 10{,}000 characters. Raw storage loses approximately 90% of content at the embedding layer; extraction condenses sessions into representations that fit the window. Per-task analysis reveals extraction leads on four of six question types, while raw storage outperforms on knowledge-update questions. The empirical contribution is mechanistic: we identify embedding window truncation as a previously unreported confound in the long-term-memory retrieval literature and isolate the gain attributable to representation condensation alone, holding embedder, vector store, and retrieval depth fixed. Code and data are publicly available.

Overview

CoChat Summary

TL;DR

This preprint tests whether the SECI knowledge-creation model can improve AI agent memory retrieval on LongMemEval by structuring conversation history before embedding. On 470 evaluated questions, SECI extraction slightly outperformed raw user-only storage (R@5 0.937 vs 0.921), and the paper argues the main reason is embedding-window truncation rather than abstract representation quality.

Key findings

  • On LongMemEval S, across 470 evaluated questions (500 total minus 30 abstention instances), SECI extraction achieved R@5 = 0.937 and R@10 = 0.967; the SECI hybrid + keyword method reached R@5 = 0.939 and R@10 = 0.966.
  • The reproduced MemPalace-style raw user-only baseline scored R@5 = 0.921, while raw all turns scored 0.859; removing assistant turns improved raw retrieval by 6.2 percentage points.
  • The paper reports that the embedding model (all-MiniLM-L6-v2) has a 256-token maximum input, while LongMemEval sessions average 10,042 characters; raw sessions therefore lose about 90% of their content at the embedding layer.
  • Per-task analysis shows SECI extraction outperforms on four of six task types, but knowledge-update questions favor raw storage because original wording matters for changed facts.

Methods

  • Controlled benchmark study on LongMemEval S with 500 questions across six task types; 30 abstention questions were excluded from retrieval evaluation, leaving 470 evaluated questions.
  • Compared four retrieval approaches using the same infrastructure: raw all turns, raw user-only, SECI extraction (rule-based condensation), and SECI hybrid + keyword (RRF fusion plus keyword boost).
  • All methods used ChromaDB with all-MiniLM-L6-v2 embeddings and cosine similarity; evaluation used Recall@5/10 and NDCG@5/10 at session level.

Limitations

  • The paper states that extraction is rule-based rather than LLM-based; it says LLM extraction would likely produce higher-quality representations but at significant cost.
  • Results are tied to a single embedding model with a 256-token window, and the paper notes the truncation advantage would diminish with longer-context embedders.
  • The benchmark does not test SECI Combination or Internalization, because each question is evaluated independently; the paper says longitudinal evaluation would require a different design.
  • The paper also notes that its top-10 retrieval is more constrained than MemPalace’s top-50, which affects absolute recall numbers; this is explicitly stated as a comparability limitation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.21203/rs.3.rs-9801639/v1

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Recall@5 across four retrieval approaches on LongMemEval S (470 questions). SECI hybrid + keyword achieves the highest score (0.939), outperforming raw user-only by 1.8 percentage points. All approaches use the same embedding model (all-MiniLM-L6-v2, 256-token max sequence length).
Fig. 1 Recall@5 across four retrieval approaches on LongMemEval S (470 questions). SECI hybrid + keyword achieves the highest score (0.939), outperforming raw user-only by 1.8 percentage points. All approaches use the same embedding model (all-MiniLM-L6-v2, 256-token max sequence length).From the PDF

Verification

Claims the paper makes

5 Valid · 1 Not valid · 10 Not checked · 62 Own results and interpretations

23 more claims

Create a free account to see all 78 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 29
  • Probably not a substantial claim. 32
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Citations

Cites

Show all 33 references

Metadata

Authors
Żatuchin, D.
Affiliations
Estonian Entrepreneurship University of Applied Sciences (EE)
Corresponding author
Żatuchin, Dmitrij
Venue
Research Square (preprint)
Published
2026-07-07
OpenAlex
W7167607753
Cited by
0
References
23
Type
posted content
Language
en
Open access
Locations
Research Square (acceptedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
AI agent memory, SECI model, knowledge production, LongMemEval, conversational history, structured knowledge extraction, long-term memory retrieval, knowledge updating, vector databases

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: From Organizational Knowledge to AI Agent Memory: Empirical Validation of the SECI Model on the LongMemEval Benchmark. https://app.cochat.ai/papers/10.21203/rs.3.rs-9801639/v1
The paper
Żatuchin, D. (2026). From Organizational Knowledge to AI Agent Memory: Empirical Validation of the SECI Model on the LongMemEval Benchmark. Research Square (preprint). https://doi.org/10.21203/rs.3.rs-9801639/v1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.