CoChat
Sign Up
PreprintCited by 0Software Engineering ResearchSoftware System Performance and ReliabilitySoftware Engineering Techniques and Practices

Memory as Infrastructure: Reliability Engineering for Persistent Agent Memory in Months-Long LLM-Assisted Development

Open access
Helwig, Mike·2026·arXiv
DOI 10.48550/arxiv.2609.05510arXiv 2609.05510
Open this paper in CoChat — free accountOpen PDF

Abstract

LLM coding agents are crossing from task-scale to project-scale: single assisted efforts that run for months, across repeated context compactions, on codebases far larger than any context window. We report operational experience from one such effort (a research project under a single continuous Claude Code session line since January 2026, driving a 633,000-line codebase, with its memory subsystem continuously instrumented since July 2026) and describe SIx Harness, the open-source memory and continuity infrastructure that emerged from keeping it coherent. The harness combines per-project long-term memory (hybrid lexical-vector retrieval over SQLite, fully local), precision-gated context injection, anti-recurrence stores for decisions and dead-ends, conventions engineered to survive compaction, and, the part we argue is missing from operational practice, reliability engineering for the memory subsystem itself: a session-start health gate with discriminated failure modes, heartbeat telemetry designed so that no enumerated failure mode can pass unrecorded, and alert-fatigue budgeting borrowed from SRE practice. We present what we believe is the first months-scale instrumented operational record of a persistent agent-memory subsystem in production development use: 78,933 hook invocations; 85 recorded failures, none silent: 84 in the subsystem's first three weeks, one since, none in the final 20 days; an injection layer whose ten-day precision instrument shows zero false fires against an intact denominator; and three production incidents traced from instrument reading to structural fix. From the record we distill seven design principles, state our limitations plainly (N=1, no control arm, self-reported), and publish a tagged pre-registered ablation protocol that any team can run with the released MIT-licensed kit.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper reports an eight-month, single-project operational record of persistent agent memory in LLM-assisted development, arguing that memory should be treated as infrastructure with SRE-style reliability engineering. It presents SIx Harness, an MIT-licensed memory and continuity system, and distills seven design principles from 78,933 instrumented hook invocations, 85 recorded failures, and a ten-day precision window.

Key findings

  • Across 78,933 hook invocations from 2026-07-02 through 2026-08-30, the heartbeat store recorded 85 failures, none silent: 84 in the subsystem’s first three weeks, one since, and none in the final 20 days.
  • In the ten-day precision window (2026-08-19 to 08-28), there were 68 prompts, 14 intent-bearing, 3 real-use injections all judged topically justified, and 0 false injections on the 54 non-intent prompts; retrieval latency was median 0 ms across all prompts and median 2.5 s when retrieval ran.
  • Three production incidents were traced from instrument readings to structural fixes: a day-one latency stack (11.3 s hook wall time reduced to 1.6 s), an impossible fallback embedding model (768-dimensional vectors against a 1024-dimensional index), and injections fired on machine-generated turns with no reader.
  • The paper distills seven design principles, including “memory is infrastructure,” “instrument the denominator,” “precision beats recall for injected context,” and “fight both failure directions at once, with opposed rules.”

Methods

  • Design: an experience report and operational record from one continuous Claude Code session line since January 2026, driving a 633,446-line codebase across 1,936 code files plus a 448-file design corpus, with the memory subsystem continuously instrumented since July 2026.
  • System: SIx Harness combines per-project long-term memory (hybrid BM25 + vector retrieval over SQLite, local Ollama embeddings), anti-recurrence stores, precision-gated injection, a session-start health gate with heartbeats, and compaction-surviving conventions.
  • Analysis: numbers are re-derived by scripted queries against instrument stores; the ten-day precision instrument records every prompt, injection, latency, and gate verdict, and the paper publishes a tagged pre-registered ablation protocol (A1: injection off; A2: gate off) for causal testing.

Limitations

  • Stated in the paper: N=1, no control arm, self-reported by the system’s operator-builders (including the agent), confounding across model generations and tooling improvements, author-adjudicated labels without independent raters, and a pre-gate baseline whose sample denominator was not preserved.
  • Stated in the paper: the precision window was a light-duty period (about a sixth of lifetime average load) with an idle gap and an outage; unmeasured costs include triage minutes, injected tokens, and maintenance attention; scope is single-operator and local-only, leaving team-scale threats such as memory poisoning unaddressed.
  • Evident from the text: the quantities of greatest interest—rediscoveries prevented and knowledge not lost—are counterfactuals the observational record cannot count, and the causal ablation protocol is proposed but not yet run.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2609.05510

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 30 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In the project's early record, three consecutive sessions each independently re-proposed a subsystem that was already built and sitting dormant, with the relevant design documents present in context each time.

    “In our project's early record, three consecutive sessions each independently re-proposed a subsystem that was already built and sitting dormant, with the relevant design documents present in context each time.”
    • Also stated in: § 3.2 Anti-recurrence stores: curated, not scraped · plain
      “Early in the project, three consecutive sessions each re-proposed building a subsystem that already existed in the codebase in dormant form, with the design documentation injected into context each time.”
    Quote in the paperSubstantial claim
  • Findingown data

    The precision telemetry's first full-window read showed two of the three real-use injections had fired on machine-generated turns.

    “The precision telemetry's first full-window read showed two of the three real-use injections had fired on machine-generated turns”
    Quote in the paperSubstantial claim
  • Findingown data

    Before the relevance gates, the operations log records the stale-claim block firing on roughly four in five intent-bearing prompts.

    “Before the relevance gates, the operations log records the stale-claim block firing on roughly four in five intent-bearing prompts”
    Quote in the paperSubstantial claim
  • Findingown data

    On the precision instrument's first day, the prompt hook's wall time exceeded its 3-second budget nearly fourfold.

    “On the precision instrument's first day, the prompt hook's wall time exceeded its 3-second budget nearly fourfold, visible immediately in the invocation telemetry.”
    Quote in the paperSubstantial claim
  • Findingnone given

    The memory infrastructure is substantially built and daily-operated by the agent it serves, under the human operator's direction and review.

    “The memory infrastructure described here is not only for the agent; it is substantially built and daily-operated by the agent it serves, under the human operator's direction and review.”
    Quote in the paperSubstantial claim
  • May combine two claims. 62
  • Classified as probably not a substantial claim. 53

  • May combine two claims.

    Findingown data

    For those two injections, topical relevance was genuine but readership was zero and the spend was real.

    “Topical relevance was genuine; readership was zero; the spend was real.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The fallback existed only as an ~18-second model load plus a second timeout chain appended to every failure path.

    “it existed only as an ~18-second model load plus a second timeout chain appended to every failure path.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A configured fallback embedding model emitted 768-dimensional vectors against a 1024-dimensional index, so the dimension guard refused it every time and the fallback could never succeed.

    “a configured fallback embedding model emitted 768-dimensional vectors against a 1024-dimensional index: the dimension guard refused it every time, so the fallback could never succeed”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The window's remaining real-use injection fired on a human-authored intent-bearing turn, and DO-NOT-REBUILD, stale-claim, and lineage blocks cleared every gate and named existing production subsystems relevant to the discussion at hand.

    “The window's remaining real-use injection fired on a human-authored intent-bearing turn: DO-NOT-REBUILD, stale-claim, and lineage blocks cleared every gate and named existing production subsystems relevant to the discussion at hand.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    At single-project scale, the project's failures were never that the data model was insufficiently expressive and always that some part of the pipeline was quietly wrong.

    “at singleproject scale, our failures were never”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Under the old pre-gate behavior, the stale-claim block fired on roughly four in five intent-bearing prompts.

    “Under the old pre-gate behavior (the stale-claim block firing on roughly four in five intent-bearing prompts)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The client resolved localhost to IPv6 first while the local inference server binds IPv4-only, burning ~2 seconds per call on a blackholed connect.

    “the client resolved localhost to IPv6 first while the local inference server binds IPv4-only, burning ~2 seconds per call on a blackholed connect (the server was answering in 57 ms)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    As of this writing the codebase comprises 633,446 lines across 1,936 code files, plus a 448-file design corpus.

    “As of this writing it comprises 633,446 lines across 1,936 code files, plus a 448-file design corpus.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the ten-day precision window (2026-08-19 to 08-28), there were 2,121 hook invocations and 0 failures.

    “Hook invocations / failures | 2,121 / 0”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The machine-generated turns were completion callbacks that flow through the same prompt hook as human input and present no decision point to any reader, human or agent.

    “completion callbacks that flow through the same prompt hook as human input and present no decision point to any reader, human or agent.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The ten-day window was a light-duty period (2,121 hook invocations, roughly a sixth of the lifetime daily average), which bounds what it can show about load, though not about precision.

    “It was a light-duty period by the project's standards (2,121 hook invocations, roughly a sixth of the lifetime daily average, spanning a multi-day idle gap and a network outage), which bounds what it can show about load, though not about precision.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The instrumented window produced three injections on real-use turns, all judged topically justified.

    “The instrumented window produced three, all judged relevant”
    • Also stated in: § 9 Conclusion · plain
      “Injections on real-use turns | 3, each topically justified”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Findingown data

    The noise channel was remade so that silence is the default and every firing is accountable to a denominator.

    “a noise channel remade so that silence is the default and every firing is accountable to a denominator”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    All 85 recorded hook failures by day and family were classified from each row's stored error detail.

    “All 85 recorded hook failures by day and family, classified from each row's stored error detail.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The record reported here includes 78,933 invocations; 85 recorded failures, none silent, none in the final 20 days.

    “The record reported here (78,933 invocations; 85 recorded failures, none silent, none in the final 20 days”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Accumulated memory state at time of writing: 19,065 indexed chunks; 389 curated memory files; 154 captured decisions and 186 captured dead-ends, of which 59 and 47 respectively survived operator triage into the confirmed anti-recurrence record.

    “Accumulated memory state at time of writing: 19,065 indexed chunks; 389 curated memory files; 154 captured decisions and 186 captured dead-ends, of which 59 and 47 respectively survived operator triage into the confirmed anti-recurrence record”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The operator's triage decisions correctly identify which candidate decisions and dead-ends should enter the confirmed anti-recurrence record.

  • May combine two claims.

    Findingown data

    The failures cluster into four families: 20 Windows text-encoding failures on day one; 34 FTS5 virtual-table fault plus database-lock contention failures (July 2-6); 17 capture-pipeline failures (July 7-20); and 14 low-rate failures from control characters (July 8-August 10).

    “Day one, 2026-07-02: 20 Windows text-encoding failures across four hooks”
    • Also stated in: § 4.1 The long haul: 78,933 invocations and a maturation curve · plain
      “2026-07-02 → 07-06: 34 failures: an FTS5 virtual-table fault plus database-lock contention under concurrent hooks.”
    • Also stated in: § 4.1 The long haul: 78,933 invocations and a maturation curve · plain
      “July 8 → August 10: 14 low-rate failures from control characters in ingested text breaking FTS5 string parsing”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the ten-day precision window, there were 68 prompts, 14 intent-bearing.

    “Prompts / intent-bearing | 68 / 14”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The 2026-08-19 to 08-28 window adds denominator-complete injection telemetry.

    “The 2026-08-19 → 08-28 window (ten calendar days, bounded by the precision instrument's deployment date on one side and this draft on the other) adds denominator-complete injection telemetry.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The instrumented final two months record 90 boundary crossings, a rate implying a few hundred over the period.

    “Session boundaries before instrumentation went uncounted; the instrumented final two months alone record 90 boundary crossings, a rate implying a few hundred over the period.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Retrieval latency for all prompts had median 0 ms, with 54 of 68 prompts triggering no retrieval work.

    “Retrieval latency, all prompts | median 0 ms (54 of 68 prompts trigger no retrieval work)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Eighty-five hook invocations failed: 84 in the subsystem's first three weeks (July 2-21), one since (August 10), none in the final 20 days.

    “Eighty-five failed: 84 of them in the subsystem's first three weeks (July 2-21), one since (August 10), none in the final 20 days.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the ten-day window, 68 prompts were recorded, 14 intent-bearing, 3 injections (all judged relevant), 0 false fires on the 54 non-intent prompts.

    “68 prompts, 14 intent-bearing, 3 injections (all judged relevant), 0 false fires on the 54 non-intent prompts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All 85 hook failures in the instrumented lifetime were recorded; none was discovered by suspicion.

    “all 85 hook failures in the instrumented lifetime were recorded ( §4.1); none was discovered by suspicion.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The eleven health-gate checks and the heartbeat enumeration cover the failure modes that matter for a persistent agent-memory subsystem.

  • May combine two claims.

    Findingown data

    Structural fixes ended each failure family, except four July 15-17 duplicate-chunk collisions that stopped without an attributable fix; the last recorded failure is 2026-08-10.

    “Structural fixes ended each family, except four July 15-17 duplicate-chunk collisions that stopped without an attributable fix; the last recorded failure is 2026-08-10.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Serving systems treat context compaction as routine load-bearing infrastructure.

    “serving systems treat context compaction as routine load-bearing infrastructure [ParallelCompaction 2026]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites ParallelCompaction 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Injected blocks that are only sometimes relevant train the human operator (and plausibly the agent) to skim past them; once the channel is learned-noise, it is worse than absent.

    “Injected blocks that are only sometimes relevant train the human operator (and plausibly the agent) to skim past them; once the channel is learned-noise, it is worse than absent.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Vendors now publish engineering guidance for agents that outlive their context windows.

    “vendors now publish engineering guidance for agents that outlive their context windows [Anthropic 2025b]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Anthropic 2025b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    At tenure scale, the failures that matter most are memory failures, which come in four kinds: knowledge loss, repeated work, context degradation, and silent memory death.

    “At tenure scale, the failures that matter most are not reasoning failures.They are memory failures, and we found they come in four kinds”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Persistent agent memory is infrastructure, and infrastructure needs reliability engineering: health checks with actionable alarms, heartbeats, freshness objectives, and an economics of alerting that respects the operator's attention.

    “Our central claim is that the fourth failure re-frames the other three: persistent agent memory is infrastructure, and infrastructure needs reliability engineering: health checks with actionable alarms, heartbeats, freshness objectives, and an economics of alerting that respects the operator's attention [SRE 2016;Breck 2017]”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites SRE 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Breck 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: The eleven health-gate checks and the heartbeat enumeration cover the failure modes that matter for a persistent agent-memory subsystem.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Measured task horizons for AI agents have been doubling roughly every seven months.

    “Measured task horizons for AI agents have been doubling roughly every seven months [METR 2025]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites METR 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    The nearest neighbors to this work are a one-month qualitative action-research study of 391 consecutive LLM sessions on one project and org-scale production GenAI telemetry that is memory-blind; neither instruments the memory subsystem.

    “The nearest neighbors are a one-month qualitative action-research study of 391 consecutive LLM sessions on one project [WrittenByAI 2026] and org-scale production GenAI telemetry that is memory-blind [WhatsCode 2026]; neither instruments the memory subsystem.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites WrittenByAI 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites WhatsCode 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    Documentation research has long observed that artifacts go stale silently.

    “Documentation research has long observed that artifacts go stale silently (”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites OutdatedDocs 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    All numbers derive from the instrument stores by scripted queries, each documented in the artifact's numbers appendix.

    “All numbers derive from the instrument stores by scripted queries, each documented in the artifact's numbers appendix.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Three research streams converge on, and stop short of, this paper's ground.

    “Three research streams converge on, and stop short of, this paper's ground.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The related-work sweep behind §8 was web-verified on 2026-08-30, and the three most load-bearing comparisons were additionally re-verified against live sources at drafting time.

    “The related-work sweep behind §8 was web-verified on 2026-08-30, and the three most load-bearing comparisons were additionally re-verified against live sources at drafting time.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Relevance is not sufficient; the injection contract is with a reader, and a turn can have none.

    “relevance is not sufficient; the injection contract is with a reader, and a turn can have none.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Study design and conductnone given

    Whether delivery becomes prevention is a counterfactual an observational record cannot count.

    “Whether delivery becomes prevention (work not repeated that otherwise would have been) is precisely the counterfactual an observational record cannot count”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Five injection firings on the deployment day were authors' synthetic validation probes and are excluded from use-analysis.

    “Five injection firings on the deployment day were authors' synthetic validation probes and are excluded from use-analysis by that stated rule”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Every component sophisticated enough to fail interestingly will.

    “Every component sophisticated enough to fail interestingly will.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Every metric in the ablation protocol is a query against stores the kit creates.

    “every metric above is a query against stores the kit creates.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The kit's 580 offline tests did not catch either root cause and, being fully offline, could not have.

    “the kit's 580 offline tests did not catch either root cause and, being fully offline, could not have”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    A new memory subsystem in real production converges from a double-digit-failure first week to a single failure in its last five weeks and none in its last 20 days.

    “a new memory subsystem in real production converges 68 prompts (every one writes a telemetry row) 54 non-intent | 0 false injections 14 intent-bearing 3 injections (2 machine-turn, 1 human-turn), all judged relevant recorded pre-gate rate: ~4 of 5 intent-bearing prompts drew a noise block”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The kit installs on any Claude Code project in one command.

    “the kit installs on any Claude Code project in one command”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The record is produced by the system's own operator-builders, and one of those operators is the agent the system serves; the instrumentation is designed to resist motivated reading, but design intent is not independence.

    “Self-report: the record is produced by the system's own operatorbuilders, and one of those operators is the agent the system serves; the instrumentation is designed to resist motivated reading (denominators, append-only stores), but design intent is not independence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The agent implemented much of the infrastructure under the author's direction, operates its triage and health routines, and assisted substantially in drafting this manuscript, including deriving every reported number from the instrument stores.

    “The agent implemented much of the infrastructure under the author's direction, operates its triage and health routines, and assisted substantially in drafting this manuscript, including deriving every reported number from the instrument stores.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    SIx Harness v0.2.2 is released under the MIT license at github.com/mike-m6online/SIx_Harness.

    “SIx Harness v0.2.2 is released under the MIT license at github.com/mike-m6online/SIx_Harness”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    The eight months span model generations, tooling improvements, and operator learning, all of which improve outcomes independently of memory infrastructure.

    “Confounding: the eight months span model generations, tooling improvements, and operator learning, all of which improve outcomes independently of memory infrastructure.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    At task scale, agent memory is a convenience; at tenure scale it is load-bearing infrastructure, and it fails like infrastructure: quietly, at the joints, in both directions at once.

    “At task scale, agent memory is a convenience.At tenure scale it is load-bearing infrastructure, and it fails like infrastructure: quietly, at the joints, in both directions at once.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The record is reported by the system's own operator-builders, human and agent alike, which is a bias.

    “it is a bias disclosure: the record is reported by the system's own operator-builders, human and agent alike”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The binding constraint is the reader's trust in the channel, spent by every irrelevant firing and only slowly re-earned.

    “The binding constraint is the reader's trust in the channel, spent by every irrelevant firing and only slowly re-earned.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    N = 1: one project, one human operator with one agent, one stack (Claude Code, local Ollama, Windows), one domain.

    “N = 1: one project, one human operator with one agent, one stack (Claude Code, local Ollama, Windows), one domain.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A shared-memory deployment would need admission control and provenance we did not need.

    “a shared-memory deployment would need admission control and provenance we did not need.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The author's prior line placed a language model inside a deterministic verification harness, with provenance-carrying claims and evidence-class-bounded confidence, demonstrated on metadata reconstruction and distilled as design principles.

    “placing a language model inside a deterministic verification harness, with provenance-carrying claims and evidence-class-bounded confidence, demonstrated on metadata reconstruction [Rosetta 2026] and distilled as design principles [DetByDefault 2026].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Rosetta 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites DetByDefault 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    A rapid recent line argues deployed agent memory degrades and must be engineered for.

    “A rapid recent line argues deployed agent memory degrades and must be engineered for”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Aging 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites MemFail 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites STALE 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    A months-long agent session line is, in effect, a developer who turns over at every compaction; that is why turnover economics, not context-window engineering, is the right ancestral frame.

    “A months-long agent session line is, in effect, a developer who turns over at every compaction; that is why turnover economics, not context-window engineering, is the right ancestral frame.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    The released kit makes the causal questions testable, on our deployment or anyone's.

    “The released kit makes the causal questions testable, on our deployment or anyone's.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S4 (based).

    Study design and conductnone given

    Whether gated injection yields net benefit where their un-gated scaffolding cost net performance is an open question our A1 protocol is designed to answer.

    “whether gated injection yields net benefit where their un-gated scaffolding cost net performance is an open question our A1 protocol is designed to answer.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    An unactionable alarm does not merely waste attention; it trains the reader out of the channel that will one day carry the real one.

    “An unactionable alarm does not merely waste attention; it trains the reader out of the channel that will one day carry the real one.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Every hook invocation writes a row before doing work, including the boring ones.

    “Every hook invocation writes a row before doing work, including the boring ones.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    There the harness governs the model's claims about data; here the same stance governs the memory subsystem of a development agent.

    “There the harness governs the model's claims about data; here the same stance governs the memory subsystem of a development agent.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The pre-AI software-engineering literature stated our problem first: knowledge loss under developer turnover, measured and priced; documentation that goes stale silently; domain knowledge thinly spread as a root cause of failure; and developers relying on interruption and archaeology because knowledge lives in heads.

    “the pre-AI software-engineering literature stated our problem first: knowledge loss under developer turnover, measured and priced [Rigby 2016;Robillard 2021]; documentation that goes stale silently [OutdatedDocs 2023]; domain knowledge thinly spread as a root cause of failure [Curtis 1988]; and developers relying on interruption and archaeology because knowledge lives in heads [LaToza 2006].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Rigby 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Robillard 2021 source
    • cites OutdatedDocs 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Memory fails toward rediscovery (what exists is ignored) and toward decay (what is recorded is wrong).

    “Memory fails toward rediscovery (what exists is ignored) and toward decay (what is recorded is wrong).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    These reports are longitudinal but memory-blind.

    “These reports are longitudinal but memory-blind.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    A retrieval layer the reader has learned to skim is worse than none.

    “a retrieval layer the reader has learned to skim is worse than none.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The questions the record cannot settle can at least be made precise, pre-registered, and cheap for others to run.

    “the questions it cannot settle can at least be made precise, pre-registered, and cheap for others to run.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The pre-registered A1 ablation is designed to measure whether delivery becomes prevention.

    “it is the primary quantity the pre-registered A1 ablation ( §7) is designed to measure.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    A shakedown run by the system's builders changes the builders' behavior.

    “Observer effects: a shakedown run by the system's builders changes the builders' behavior.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Every number in this paper is re-derivable from the documented queries in the paper's numbers appendix run against the instrument stores; the stores themselves contain project-identifying content and are not released; the queries, schemas, and the instrument code that produces such stores on any deployment are.

    “Every number in this paper is re-derivable from the documented queries in the paper's numbers appendix run against our instrument stores; the stores themselves contain project-identifying content and are not released; the queries, schemas, and the instrument code that produces such stores on any deployment are.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The repository includes the 580-test offline suite, the one-command installer, and the tagged pre-registration (prereg-ablation-v1).

    “The repository includes the 580-test offline suite, the one-command installer, and the tagged pre-registration (prereg-ablation-v1).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The project was never run without the harness over a comparable period, so effect sizes are not causal claims.

    “No control arm: the project was never run without the harness over a comparable period; effect sizes are not causal claims.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The duality of memory failure has recently been formalized as a benchmark metric, and its decayed pole is now benchmarked directly.

    “The duality has recently been formalized as a benchmark metric [FAMA 2026], and its decayed pole is now benchmarked directly [STALE 2026]”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites FAMA 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites STALE 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    We regard this wave as the in-vitro companion to our in-vivo record: it names and measures, under controlled faults, the failure families our incidents exhibit under natural ones.

    “We regard this wave as the in-vitro companion to our in-vivo record: it names and measures, under controlled faults, the failure families our incidents exhibit under natural ones.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Within its enumerated failure modes, the system could not fail without leaving a record, and not one of the 85 recorded failures was discovered by suspicion or archaeology.

    “we claim that within its enumerated failure modes it could not fail without leaving a record, and that not one of the 85 recorded failures was discovered by suspicion or archaeology.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The eleven health-gate checks and the heartbeat enumeration cover the failure modes that matter for a persistent agent-memory subsystem.

  • May combine two claims.

    Study design and conductnone given

    The precision window is bounded by its instrument's deployment date rather than chosen, but it was a light-duty period (about a sixth of lifetime average load) including an idle gap and an outage.

    “Window representativeness: the precision window is bounded by its instrument's deployment date rather than chosen, but it was a light-duty period (about a sixth of lifetime average load) including an idle gap and an outage.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The second instantiation tempers the single-deployment concern for the principles, while the memory-subsystem operational record itself remains N=1.

    “that second instantiation tempers the single-deployment concern for the principles, while the memory-subsystem operational record itself remains N=1.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A 14-intent-prompt window under the old pre-gate behavior would have produced on the order of eleven noise firings.

    “a 14-intent-prompt window would have produced on the order of eleven noise firings.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The old pre-gate firing rate (roughly four in five intent-bearing prompts) is applicable to the instrumented window's intent-bearing prompts for the counterfactual estimate.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The genre of production experience with AI-assisted development runs from population-scale telemetry through org-scale deployment reports to RCTs whose disagreement about productivity underlines that throughput is not the only quantity worth measuring.

    “The genre runs from populationscale telemetry [Ziegler 2024;Peng 2023] through org-scale deployment reports (25 months of GenAI telemetry at WhatsApp [WhatsCode 2026], LLM-driven migrations at Google [GoogleMigrate 2025]) to RCTs whose disagreement about productivity [METR-RCT 2025] underlines that throughput is not the only quantity worth measuring.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Ziegler 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Peng 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites WhatsCode 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Study design and conductnone given

    The quantities we care most about (rediscoveries prevented, knowledge not lost) are counterfactuals observation cannot count.

    “Finally, the quantities we care most about (rediscoveries prevented, knowledge not lost) are counterfactuals observation cannot count.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Context injection is a delivery mechanism, not a guarantee of uptake.

    “Context injection is a delivery mechanism, not a guarantee of uptake.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The pre-gate baseline's sample denominator was not preserved and is reported as context only.

    “the pre-gate baseline's sample denominator was not preserved and is reported as context only.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    We did not instrument the harness's own overhead: triage minutes, injected tokens, maintenance attention; a net-cost accounting is required before claims of net benefit.

    “Unmeasured costs: we did not instrument the harness's own overhead: triage minutes, injected tokens, maintenance attention; a net-cost accounting is required before claims of net benefit”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The contemporaneous measurement's full sample denominator was not preserved.

    “This contemporaneous measurement's full sample denominator was not preserved; we report it as context, not as a comparison leg”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    SRE ideas have been applied to agents in general (safety SLOs, circuit breakers, health checks) and agent-observability tooling now traces memory operations as spans.

    “SRE ideas have been applied to agents in general (safety SLOs, circuit breakers, health checks [MsftSRE 2026, as described in its published summary]) and agent-observability tooling now traces memory operations as spans [Braintrust 2026]”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites MsftSRE 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Braintrust 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductnone given

    The full protocol is committed and tagged in the artifact repository (PREREGISTRATION.md, tag prereg-ablation-v1, 2026-08-30, registered before any ablation run).

    “The full protocol is committed and tagged in the artifact repository (PREREGISTRATION.md,tag prereg-ablation-v1, 2026-08-30, registered before any ablation run)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    This is an experience report from a single deployment, and its evidential limits are structural.

    “This is an experience report from a single deployment, and its evidential limits are structural.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Seven transferable design rules were distilled from traced failures, including the observation that memory fails in two opposite directions (rediscovering what exists and acting on decayed records), which demand deliberately opposed operational rules held in tension.

    “seven transferable design rules distilled from traced failures, including the observation that memory fails in two opposite directions (rediscovering what exists and acting on decayed records), which demand deliberately opposed operational rules held in tension.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The reliability-engineering toolkit transplants cleanly to agent memory, and transplants decision-deterministically, the model never adjudicating its own continuity.

    “the reliability-engineering toolkit transplants cleanly to agent memory (and transplants decision-deterministically, the model never adjudicating its own continuity)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductcitation

    The design stance behind §5 is not unique to this deployment: the same operator pair applied it to a second, independently published system, a metadata-reconstruction verification harness whose companion methodology paper articulates the shared stance.

    “Generality of the principles vs. the record: the design stance behind §5 is not unique to this deployment: the same operator pair applied it to a second, independently published system, a metadata-reconstruction verification harness [Rosetta 2026] whose companion methodology paper [DetByDefault 2026] articulates the shared stance”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Rosetta 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites DetByDefault 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    The regimes differ: their indicted scaffolding is standing symbolic constraint, accumulating in every prompt, and their failure mechanism is agent-side symbolic retreat; our injection layer is episodic, relevance-gated, silent by default, and health-monitored, and our observed failure economics concern the reader's attention.

    “The regimes differ: their indicted scaffolding is standing symbolic constraint, accumulating in every prompt, and their failure mechanism is agent-side symbolic retreat; our injection layer is episodic, relevance-gated, silent by default, and health-monitored (their report describes no gating or monitoring layer), and our observed failure economics concern the reader's attention.”
    Quote in the paperSubstantial claim
90 of 90

  • The paper relies on this without support.

    The operator's triage decisions correctly identify which candidate decisions and dead-ends should enter the confirmed anti-recurrence record.

    “the operator (in our deployment, usually the agent, batch-reviewed by the human) triages them (confirm, reject, retitle) before they enter the confirmed record”

  • The paper relies on this without support.

    The eleven health-gate checks and the heartbeat enumeration cover the failure modes that matter for a persistent agent-memory subsystem.

    “At every session start, eleven checks probe the stack end to end and print one line”

  • The paper relies on this without support.

    The old pre-gate firing rate (roughly four in five intent-bearing prompts) is applicable to the instrumented window's intent-bearing prompts for the counterfactual estimate.

    “Under the old pre-gate behavior (the stale-claim block firing on roughly four in five intent-bearing prompts), a 14-intent-prompt window would have produced on the order of eleven noise firings.”
3 of 3

Cites

Flagged first, then by citations

6 of the 53 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. Sculley et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS 2015.
  2. Beyer et al. (eds.) Site Reliability Engineering, Ch. 6, "Monitoring Distributed Systems." O'Reilly.
  3. Zhang et al. A Survey on the Memory Mechanism of LLM-based Agents. ACM TOIS 2025.

Authors
Helwig, M.
Corresponding author
Helwig, Mike
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2609.05510
arXiv
2609.05510
OpenAlex
W7212050121
Semantic Scholar
4cdafe2f1486935459188d3e3e6f5f958f8cdfcb
Cited by
0
References
53
Type
posted content
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
LLM coding agents, long-term memory, SQLite, context injection, alarm fatigue

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review