From Organizational Knowledge to AI Agent Memory: Empirical Validation of the SECI Model on the LongMemEval Benchmark
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 7 of the 27 sources its claims cite were checked for support.
Abstract
AI agent memory systems face a fundamental design question: should conversational history be stored verbatim or transformed into structured knowledge before retrieval? We propose that the SECI model of knowledge creation \citep{nonaka1995knowledge} provides empirically testable design principles for AI agent memory architecture. We adapt the four SECI phases to the AI agent context and evaluate structured extraction against raw storage baselines on LongMemEval \citep{wu2025longmemeval}. Across 500 questions spanning six memory task types, SECI-based extraction achieves 93.9% Recall@5 versus 92.1% for raw storage, both using the same embedding model and vector database. The dominant factor is embedding window truncation: the model processes 256 tokens maximum, while sessions average 10{,}000 characters. Raw storage loses approximately 90% of content at the embedding layer; extraction condenses sessions into representations that fit the window. Per-task analysis reveals extraction leads on four of six question types, while raw storage outperforms on knowledge-update questions. The empirical contribution is mechanistic: we identify embedding window truncation as a previously unreported confound in the long-term-memory retrieval literature and isolate the gain attributable to representation condensation alone, holding embedder, vector store, and retrieval depth fixed. Code and data are publicly available.
Overview
CoChat Summary
TL;DR
This preprint tests whether the SECI knowledge-creation model can improve AI agent memory retrieval on LongMemEval by structuring conversation history before embedding. On 470 evaluated questions, SECI extraction slightly outperformed raw user-only storage (R@5 0.937 vs 0.921), and the paper argues the main reason is embedding-window truncation rather than abstract representation quality.
Key findings
- On LongMemEval S, across 470 evaluated questions (500 total minus 30 abstention instances), SECI extraction achieved R@5 = 0.937 and R@10 = 0.967; the SECI hybrid + keyword method reached R@5 = 0.939 and R@10 = 0.966.
- The reproduced MemPalace-style raw user-only baseline scored R@5 = 0.921, while raw all turns scored 0.859; removing assistant turns improved raw retrieval by 6.2 percentage points.
- The paper reports that the embedding model (all-MiniLM-L6-v2) has a 256-token maximum input, while LongMemEval sessions average 10,042 characters; raw sessions therefore lose about 90% of their content at the embedding layer.
- Per-task analysis shows SECI extraction outperforms on four of six task types, but knowledge-update questions favor raw storage because original wording matters for changed facts.
Methods
- Controlled benchmark study on LongMemEval S with 500 questions across six task types; 30 abstention questions were excluded from retrieval evaluation, leaving 470 evaluated questions.
- Compared four retrieval approaches using the same infrastructure: raw all turns, raw user-only, SECI extraction (rule-based condensation), and SECI hybrid + keyword (RRF fusion plus keyword boost).
- All methods used ChromaDB with all-MiniLM-L6-v2 embeddings and cosine similarity; evaluation used Recall@5/10 and NDCG@5/10 at session level.
Limitations
- The paper states that extraction is rule-based rather than LLM-based; it says LLM extraction would likely produce higher-quality representations but at significant cost.
- Results are tied to a single embedding model with a 256-token window, and the paper notes the truncation advantage would diminish with longer-context embedders.
- The benchmark does not test SECI Combination or Internalization, because each question is evaluated independently; the paper says longitudinal evaluation would require a different design.
- The paper also notes that its top-10 retrieval is more constrained than MemPalace’s top-50, which affects absolute recall numbers; this is explicitly stated as a comparability limitation.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
5 Valid · 1 Not valid · 10 Not checked · 62 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“SECI has been applied to software engineering (Rus and Lindvall 2002; Bjørnson and Dingsøyr 2008; Chikh 2011; Astorga-Vargas et al. 2017) and digital education ( ˙ Zatuchin 2024), but has not been applied to AI agent memory systems.”
- Also stated in: § 2.1 The SECI Model of Knowledge Creation · plain
“Neither, however, applies an organizational-knowledge framework to memory design. The present study extends the SECI line of work to AI agent memory systems.”
What the sources say
- Not checked — no full text of the sourcecites Rus and Lindvall 2002 source
- Not matchedcites Bjørnson and Dingsøyr 2008
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Chikh 2011 source
The citation is offered only for the clause that SECI has been applied to software engineering, and our copy states exactly that: Section 5.2 ('Applying the SECI Model to SRE', p. ~6) adapts the SECI framework to software requirements engineering, and the Conclusions (p. ~9) confirm 'we came up with the idea of applying SECI as a knowledge creation model to these processes' and report a developed KM framework in SRE built on the four SECI modes and their Ba. The negative clause about AI agent memory systems is the citing authors' own gap statement, not attributed to this source, and our copy (payroll virtual project case study, Section 6) does not address it. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.1 The SECI Model of Knowledge Creation · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Three concurrent systems published in 2025 report results on LongMemEval and constitute the most direct comparative baselines for our work.”
What the sources say
- Partly supportedcites Pan et al. 2025 sourceExperimental — accuracy not yet measured
The cited paper is a 2025 conversational-memory system, supporting its identification as a contemporary system. But its Datasets & Evaluation Metrics section (page unavailable) names LOCOMO and Long-MT-Bench+, and the Main Results section reports comparisons on those benchmarks; our copy does not show LongMemEval results or establish that it is one of three direct comparative baselines for the citing work. (LLM, conf=0.76)
- Supportedcites Tan et al. 2025 source
The cited paper is a 2025 system paper, and its Section 8.1 reports RMM results on LongMemEval alongside baseline methods. That supports citing it as one of the concurrent systems reporting results on the benchmark; the source does not need to identify the other two systems or assess which baselines are most direct for the citing authors’ work. (LLM, conf=0.90)
- Not matchedcites Xu et al. 2025
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SECI extraction alone scores 0.937.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Structured extraction outperforms raw storage on the same benchmark with the same embedding infrastructure.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Raw storage loses approximately 90% of content at the embedding layer; extraction condenses sessions into representations that fit the window.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
23 more claims
Create a free account to see all 78 claims and to verify the 2 references still open.
- It may state two separate claims as one. 29
- Probably not a substantial claim. 32
- The quote only partly states this claim. 3
Citations
Cites
The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation
Nonaka, I., & Takeuchi, H. (1995) · doi:10.1093/oso/9780195092691.001.0001 · Cited by 18,010
A Dynamic Theory of Organizational Knowledge Creation
Nonaka, I. (1994) · Organization Science · doi:10.1287/orsc.5.1.14 · Cited by 17,809
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Alavi, M., & Leidner, D. E. (2001) · MIS Quarterly · doi:10.2307/3250961 · Cited by 10,176
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., & Li, Q. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671470 · Cited by 827
Reciprocal rank fusion outperforms condorcet and individual rank learning methods
Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009) · Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1571941.1572114 · Cited by 797
Knowledge management in software engineering
Rus, I., & Lindvall, M. (2002) · IEEE Software · doi:10.1109/ms.2002.1003450 · Cited by 742
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024) · arXiv · doi:10.48550/arxiv.2410.10813 · Cited by 493
Metadata
- Authors
- Żatuchin, D.
- Affiliations
- Estonian Entrepreneurship University of Applied Sciences (EE)
- Corresponding author
- Żatuchin, Dmitrij
- ORCID
- Żatuchin, Dmitrij
- Venue
- Research Square (preprint)
- Published
- 2026-07-07
- OpenAlex
- W7167607753
- Cited by
- 0
- References
- 23
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- Research Square (acceptedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- AI agent memory, SECI model, knowledge production, LongMemEval, conversational history, structured knowledge extraction, long-term memory retrieval, knowledge updating, vector databases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: From Organizational Knowledge to AI Agent Memory: Empirical Validation of the SECI Model on the LongMemEval Benchmark. https://app.cochat.ai/papers/10.21203/rs.3.rs-9801639/v1
- The paper
- Żatuchin, D. (2026). From Organizational Knowledge to AI Agent Memory: Empirical Validation of the SECI Model on the LongMemEval Benchmark. Research Square (preprint). https://doi.org/10.21203/rs.3.rs-9801639/v1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.