CoChat
Sign Up
PreprintCited by 48AI-based Problem Solving and PlanningTopic ModelingInformation Retrieval and Search Behavior

AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications

Open access
Zhao, Yujie, Yuan, Boqin, Huang, Junbo, Yuan, Haocheng, Yu, Zhongming, Xu, Haozhou, Hu, Lanxiang, Shankarampeta, Abhilash +4 more·2026·arXiv
DOI 10.48550/arxiv.2602.22769arXiv 2602.22769
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) are increasingly used as autonomous agents in complex, long-horizon applications, where effective memory is critical for sustained performance. Yet existing memory benchmarks are largely dialogue-centric, while real agent memory consists of continuous agent-environment interaction trajectories composed of states, actions, observations, and tool outputs. To address this gap, we introduce **AMA-Bench** (**A**gent **M**emory with **A**ny length), a benchmark for evaluating long-horizon memory in realistic agentic settings. AMA-Bench combines real-world agent trajectories from representative applications with expert-curated QA, as well as synthetic trajectories that scale to arbitrary horizons with rule-based QA. Our study shows that existing memory systems underperform because they fail to capture causal and objective information and rely heavily on lossy similarity-based retrieval. We further propose **AMA-Agent**, a memory system based on causality-graph construction and tool-augmented retrieval. AMA-Agent achieves **57.22%** accuracy on AMA-Bench, outperforming the strongest baseline by **11.16%**. Resources are available at: [https://ama-bench.github.io/](https://ama-bench.github.io/).

AI snapshot

Written from the paper’s full text.

TL;DR

AMA-Bench is a benchmark for long-horizon agent memory built from real-world and synthetic agentic trajectories, and AMA-Agent is proposed as a memory system using a causality graph plus tool-augmented retrieval. AMA-Agent reaches 57.22% average accuracy, surpassing the strongest memory system baselines by 11.16%, while existing memory systems often underperform long-context baselines.

Key findings

  • AMA-Bench contains a real-world subset with 2,496 QA pairs across six agent domains and a synthetic subset with 1,200 QA pairs stratified over 8K, 16K, 32K, 64K, and 128K tokens (240 samples per interval). GPT 5.2 achieves 72.26% accuracy on the real-world subset, but memory systems frequently underperform the long-context baseline.
  • AMA-Agent achieves 57.22% average accuracy on AMA-Bench, outperforming the strongest memory system baselines by 11.16%; under Qwen3-32B it scores Recall 0.6238, Causal Inference 0.6145, State Updating 0.5305, and State Abstraction 0.4719.
  • Ablations show removing the Causality Graph drops average score from 0.57 to 0.43, and removing tool-augmented retrieval drops it to 0.44. In BabyAI needle-protocol ablations, HippoRAG2 falls from 0.46 with full observation to 0.37 with constructed memory (-19.6%) and 0.21 end-to-end (-43.2%).
  • Scaling the backbone from 8B to 32B gives only marginal improvement (average 0.038), whereas varying memory architecture induces score ranges reaching 0.45, indicating memory design is the primary bottleneck.

Methods

  • The real-world subset is curated from six domains—Web, open-world QA, Text2SQL, Software Engineering, Gaming, and Embodied AI—with expert-curated QA; the synthetic subset uses programmatic TextWorld and BabyAI environments with rule-based QA and a needle-in-a-haystack protocol.
  • Evaluation compares long-context models, RAG methods, and memory agents using Qwen3-32B and Qwen3-8B backbones, reporting Accuracy and F1; accuracy is judged by Qwen3-32B as an LLM judge, validated against human annotations on 300 instances with 92.67% agreement.
  • AMA-Agent constructs a Causality Graph from turn pairs to capture state and causal dependencies, then retrieves via top-K embedding similarity plus self-evaluation and tool-augmented search using graph node traversal or keyword/code search.

Limitations

  • Stated: the study focuses on in-episode memory; future work should extend these standards to cross-task scenarios involving lifelong learning.
  • Evident: real-world environments are treated as black boxes, with only action-observation logs observed and no access to backend state; the synthetic subset is limited to TextWorld and BabyAI.
  • Evident: main accuracy relies on an LLM-as-judge, though calibrated on 300 instances; real-world QA annotation uses cross-review, but no inter-annotator agreement is reported in the text.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2602.22769

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 8 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Existing benchmarks for agent memory primarily focus on dialogue-centric, human-agent interactions.

    “existing benchmarks primarily focus on dialogue-centric, human-agent interactions.”
    • Also stated in: § (c) Agentic Applications · plain
      “Existing benchmarks typically evaluate these capabilities in dialoguecentric or synthetic retrieval tasks (Hsieh et al., 2024;Maharana et al., 2024)”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Hsieh et al., 2024

      The Related Work section (p. 3) mentions one benchmark for long-term conversations, but does not assess whether agent-memory benchmarks primarily focus on dialogue or human–agent interactions. The Introduction (p. 1) discusses long-context evaluations instead. Searches for agent-memory and dialogue-centric benchmarks found no passage establishing the claimed pattern. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites Maharana et al., 2024 source

      The Introduction (p. 2) says conventional evaluations of conversational agents assess responses against past dialogue history, supporting the dialogue-centric part. It does not establish that existing agent-memory benchmarks primarily involve human-agent interactions; the paper’s own benchmark uses conversations generated by LLM agents and edited by humans. (LLM, conf=0.81)

      Experimental — accuracy not yet measured
  • Premisecitation

    Long-context benchmarks evaluate static document-level reasoning, focusing on multi-hop comprehension over long inputs rather than interactive or incremental memory.

    “Long-context benchmarks such as QuALITY (Pang et al., 2022), RULER (Hsieh et al., 2024), and LongBench v2 (Bai et al., 2025) evaluate static document-level reasoning, focusing on multihop comprehension over long inputs rather than interactive”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Pang et al., 2022 source

      The Abstract and Introduction describe QuALITY as a benchmark for answering questions about long, fixed passages that require understanding the passage as a whole. The paper does not characterize QuALITY as focusing on multi-hop comprehension; its discussion of multi-hop reasoning describes other datasets instead. Its fixed-passage QA format supports the contrast with interactive or incremental memory tasks. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Hsieh et al., 2024

      The Introduction (pp. 1–2) and §3.2 (p. 5) show that RULER evaluates long inputs and includes a multi-hop tracing task. But RULER is a synthetic benchmark with four task categories, not a benchmark focused on static document-level, multi-hop comprehension; its paper does not make the claimed comparison with interactive or incremental memory. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites Bai et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    Removing the Causality Graph causes a substantial degradation, with the average score dropping from 0.57 to 0.43.

    “Removing the Causality Graph causes a substantial degradation, with the average score dropping from 0.57 to 0.43”
    Quote in the paperSubstantial claim
  • Findingown data

    Removing tool augmented retrieval reduces performance to 0.44.

    “removing tool augmented retrieval reduces performance to 0.44”
    Quote in the paperSubstantial claim
  • Findingown data

    GPT 5.2 achieves the highest average accuracy (0.73) on the real-world subset.

    “GPT 5.2 achieves the highest average accuracy (0.73)”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 26
  • May combine two claims. 26
  • The quote only partly states this claim. 3

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The Spider 2.0 Snow subset contains 547 examples, with gold answers provided for 120 examples to enable verification of generated SQL queries.

    “Spider 2.0 comprises three subsets (Snow, DBT, and Lite), with the Snow subset containing 547 examples.Among these, gold answers are provided for 120 examples to enable verification of generated SQL queries.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Lei et al., 2024 source

      The task-definition passage identifies Spider 2.0, Spider 2.0-snow, and Spider 2.0-lite, and §3.1 describes evaluating generated SQL using execution accuracy. However, the passages retrieved from our copy, including §2.3 Dataset Statistics, do not substantiate the specific counts of 547 Snow examples or gold answers for 120 of them. (LLM, conf=0.80)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Many existing agent memory designs underperform the long-context baseline, as errors introduced by lossy memory compression and similarity-based retrieval accumulate and compound over long-horizon agentic tasks.

    “many existing agent memory designs underperform the long-context baseline, as errors introduced by lossy memory compression and similarity-based retrieval accumulate and compound over long-horizon agentic tasks”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    AMA-agent exhibits superior scalability, maintaining robust and consistently high accuracy even at 128K.

    “AMA-agent exhibits superior scalability, maintaining robust and consistently high accuracy even at 128K.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingcitation

    Synthetic tasks are synthesized from TextWorld and BabyAI.

    “We synthesize tasks from two distinct environments characterized by long-range dependencies and partial observability: TextWorld (Côté et al., 2018), BabyAI (Chevalier-Boisvert et al., 2019).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Côté et al., 2018

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Chevalier-Boisvert et al., 2019

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Scaling from 8B to 32B provides only marginal improvements (average improvement 0.038), whereas varying memory architecture induces significantly higher variance, with score ranges reaching 0.45.

    “scaling from 8B to 32B provides only marginal improvements (avg.improvement is 0.038), whereas varying the memory architecture induces significantly higher variance, with score ranges reaching 0.45.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    Stepwise sequences are generated grounded in gold-standard transitions to resolve low-structural density.

    “We generate stepwise sequences {(a t ,o t )} T t=1 grounded in these gold-standard transitions to resolve the issue of low-structural density.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The Long Context approach maintains competitive accuracy at shorter scales, but its performance degrades significantly beyond 32K.

    “the Long Context approach maintains competitive accuracy at shorter scales, its performance degrades significantly beyond 32K”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The real-world subset contains 2,496 QA pairs.

    “with 2496 QA pairs”
    • Also stated in: § 5.1. Memory Construction: Causality Graph · plain
      “This subset comprising a total of 2,496 QA pairs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The long-context baseline is consistently strong and often achieves the best performance, whereas existing memory systems exhibit large variance across families and frequently underperform.

    “the long context baseline is consistently strong and often achieves the best performance, whereas existing memory systems exhibit large variance across families and frequently underperform”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    AMA-Agent achieves 57.22% average accuracy on AMA-Bench, surpassing the strongest memory system baselines by 11.16%.

    “AMA-Agent achieves 57.22% average accuracy on AMA-Bench, surpassing the strongest memory system baselines by 11.16%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    In the synthetic environment, the latent state and transition kernel are programmatically defined and machine-verifiable.

    “The latent state s t and transition kernel s t+1 = P ϕ (s t ,a t+1 ) are programmatically defined and machine-verifiable.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Existing agent memory techniques often outperform long-context LLM baselines on dialogue-centric benchmarks, but fall short of the baselines in many long-horizon agentic tasks.

    “While existing agent memory techniques often outperform long-context LLM baselines on dialogue-centric benchmarks, they fall short to the baselines in many long-horizon agentic tasks”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Each selected trajectory in the real-world subset is manually annotated with 12 memory-intensive QA pairs covering all categories.

    “we manually annotate each selected trajectory with 12 memory-intensive QA pairs that collectively cover all categories in Tab. 2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both the Causality Graph and tool augmented retrieval modules are necessary for strong performance.

    “The results in Tab.6 show that both modules are necessary for strong performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Agent memory remains challenging even for frontier commercial models, with GPT 5.2 achieving only 72.26% accuracy on AMA-Bench.

    “our systematic evaluation indicates that agent memory remains challenging even for frontier commercial models, with GPT 5.2 achieving only 72.26% accuracy.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    Two auxiliary perturbations are introduced: action stochasticity and observation verbosity.

    “we introduce two auxiliary perturbations beyond ϕ: (1) Action Stochasticity (ϵ): we inject random noise into π * to simulate sub-optimal action ratios, testing memory robustness under varying agent policies; and (2) Observation Verbosity (γ): we employ various symbolic representations with controllable descriptive granularity γ for o t .”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The real-world subset comprises multi-turn trajectories curated from six diverse domains of agent-environment interactions.

    “This section details the composition of the real-world subset, which comprises multi-turn trajectories curated from six diverse domains of agent-environment interactions (Tab.7).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    AMA-Agent under the Qwen3 32B backbone achieves Recall 0.6238, Causal Inference 0.6145, State Updating 0.5305, State Abstraction 0.4719, and average 0.5722 on the real-world subset.

    “Crucially, when controlled under the Qwen3 32B backbone (Tab.5), AMA-Agent establishes a new state-of-the-art across all dimensions-Recall (0.6238), Causal Inference (0.6145), State Updating (0.5305), and State Abstraction (0.4719)-reaching an average of 0.5722.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    For TextWorld, each trajectory contains on average 57 turns and 31,662 tokens.

    “On average, each trajectory contains 57 turns and 31,662 tokens.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the Web Task Execution domain, 31 trajectories comprising 372 QA pairs were selected, with an average of 25 turns and 34K tokens per trajectory, reaching up to 166K tokens for complex tasks.

    “We select 31 trajectories comprising 372 QA pairs, with an average of 25 turns and 34K tokens per trajectory, reaching up to 166K tokens for complex tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    For the Gaming domain, 30 trajectories totaling 360 QA pairs were selected, with an average of 150 turns per episode.

    “We select 30 trajectories totaling 360 QA pairs, with an average of 150 turns per episode.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    While scaling the backbone yields marginal gains, the choice of memory system accounts for the majority of performance variance.

    “While scaling the backbone yields marginal gains, the choice of memory system accounts for the majority of performance variance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    For the Open World Tool QA domain, 30 trajectories comprising 360 QA pairs were selected, with an average of 41 turns and 289K tokens, reaching up to 997K tokens.

    “We select 30 trajectories across all three difficulty levels from the validation set, comprising 360 QA pairs with an average of 41 turns and 289K tokens -the longest among all domains, reaching up to 997K tokens.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingown data

    For the Synthetic subset, 50 trajectories were collected, with an average length of 563 turns and 30,042 tokens per trajectory.

    “Each trajectory is paired with 12 questions by default, and we collect a total of 50 trajectories, with an average length of 563 turns and 30,042 tokens per trajectory.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    The LLM-as-judge achieves 92.67% accuracy on a human-annotated sample of 300 instances from GPT-5.2 results.

    “The judge achieves 92.67% accuracy, indicating reliable alignment with human judgment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Independent human annotation provides gold labels for evaluating the LLM-as-judge.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    For the Software Engineering domain, 36 trajectories totaling 432 QA pairs were selected, with an average of 103 turns and 19K tokens per trajectory.

    “We select 36 trajectories totaling 432 QA pairs, with an average of 103 turns and 19K tokens per trajectory.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the Text-to-SQL domain, 51 trajectories totaling 612 QA pairs were selected, with an average of 22 turns and 6K tokens per trajectory.

    “We sample 51 trajectories from these verified examples to ensure answer correctness can be validated.”
    • Also stated in: § Impact Statement · plain
      “Trajectories are generated using Claude Sonnet 4.5 with the Spider2-Agent framework, totaling 612 QA pairs with an average of 22 turns and 6K tokens per trajectory.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Existing lossy compression and similarity-based retrieval techniques are insufficient for the nuanced demands of agent memory, necessitating a paradigm shift toward agent-centric memory management strategies.

    “Existing lossy compression and similarity-based retrieval techniques are insufficient for the nuanced demands of agent memory, necessitating a paradigm shift toward”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Agent memory consists of a continuous stream of agent-environment interactions that are primarily composed of machine-generated representations.
    • Rests on an assumption: Agent trajectories are causally grounded, where each action induces a latent environment state transition that constrains subsequent observations.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Existing memory benchmarks can be categorized into Dialogue-Centric and Long-Context.

    “We categorize existing memory benchmarks into two primary classes (as shown in Tab.1): Dialogue-Centric and Long-Context.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    AMA-Agent introduces a Hybrid Tool-Augmented Retrieval mechanism that enables more efficient information extraction and synthesis in machine-generated representations.

    “we introduce a Hybrid Tool-Augmented Retrieval mechanism.This approach enables more efficient information extraction and synthesis in machine-generated representations.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationreasoning

    Memory compression and similarity-based retrieval perform poorly on agent memory tasks because most compression methods are designed for natural language, while agent trajectories contain dense, causally structured state transitions.

    “memory compression and similaritybased retrieval perform poorly on agent memory tasks for two main reasons.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Agent trajectories are causally grounded, where each action induces a latent environment state transition that constrains subsequent observations.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Each question in the real-world subset is formulated such that its answer is supported by explicit and unambiguous evidence within the trajectory, ensuring correctness can be verified from the log itself.

    “Each question is formulated such that its answer is supported by explicit and unambiguous evidence within the trajectory, ensuring that the correctness”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisereasoning

    Existing methods primarily rely on similarity-based or entity-centric retrieval, often neglecting the underlying causality within stored information.

    “existing methods primarily rely on similarity-based or entity-centric retrieval, often neglecting the underlying causality within stored information.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    The synthetic subset is constructed in programmatic agent environments with automatically generated QA pairs.

    “we construct a synthetic subset in programmatic agent environments with automatically generated QA pairs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Agent memories are often machine-generated representations, and similarity retrieval frequently fails to extract the required evidence.

    “agent memories are often machine-generated representations, and similarity retrieval frequently fails to extract the required evidence.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Agent memory consists of a continuous stream of agent-environment interactions that are primarily composed of machine-generated representations.

  • Classified as probably not a substantial claim.

    Premisenone given

    The real-world subset consists of expert-annotated and sanity-checked QA pairs sourced from six representative agent domains: Web, open-world QA, Text2SQL, Software Engineering, Gaming, and Embodied AI.

    “expert-annotated and sanity-checked Question-Answer (QA) pairs sourced from six representative agent domains: Web, open-world QA, Text2SQL, Software Engineering, Gaming, and Embodied AI”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    AMA-Bench comprises two complementary subsets: a real-world subset and a synthetic subset.

    “which comprises two complementary subsets: a real-world subset and a synthetic subset.”
    • Also stated in: § 3.3. Benchmark Construction · plain
      “AMA-Bench comprises two complementary components: (i) a real-world subset and (ii) a synthetic subset.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    AMA-Bench evaluates memory for agent applications, where the interaction trajectory is characterized by machine-generated representations, causal dependencies, and dense, objective information.

    “AMA-Bench evaluates memory for agent applications, where the interaction trajectory is characterized by machine-generated representations, causal dependencies, and dense, objective information.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Agent trajectories are causally grounded, where each action induces a latent environment state transition that constrains subsequent observations.

  • May combine two claims.

    Study design and conductown data

    QA pairs in the real-world subset are authored by graduate-level annotators with research experience in LLM agents, following shared guidelines, and each annotated trajectory undergoes a cross-review sanity check by a second annotator.

    “QA pairs are then authored by graduate-level annotators with research experience in LLM agents, following shared guidelines that standardize evidence grounding and category coverage across six domains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The final selection of QA pairs ensures that different reasoning types are represented proportionally to avoid data skew.

    “Balanced Distribution: The final selection ensures that different reasoning types (e.g., spatial reasoning vs. object state tracking) are represented proportionally to avoid data skew.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The needle-in-a-haystack QA generation pipeline produces QA pairs that evaluate long-context retrieval capabilities of memory-augmented agents.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The needle-in-a-haystack QA generation pipeline uses diversity-first sampling to prevent the LLM from exploiting positional biases.

    “Diversity-First Sampling: Instead of random sampling, we pick”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The needle-in-a-haystack QA generation pipeline produces QA pairs that evaluate long-context retrieval capabilities of memory-augmented agents.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The needle-in-a-haystack QA generation pipeline binds each QA pair to specific turn ids to verify whether the agent's retrieval mechanism identified the correct needle.

    “Ground Truth Traceability: By binding each QA pair to specific turn ids, we can verify whether the agent's retrieval mechanism successfully identified the correct”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The needle-in-a-haystack QA generation pipeline produces QA pairs that evaluate long-context retrieval capabilities of memory-augmented agents.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Human annotation for judge validation used at least two annotators per instance, with disagreements resolved by majority vote and a third annotator for adjudication when needed.

    “Each instance is labeled by at least two annotators with access to the question, reference answer, and predicted answer.”
    • Also stated in: § C.2. Human-Judge Agreement · plain
      “Disagreements are resolved by majority vote, with a third annotator used for adjudication when needed.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All data entries in AMA-Bench underwent human verification to ensure that no personally identifiable information (PII) or private data were included.

    “All data entries underwent human verification to ensure that no personally identifiable information (PII) or private data were included.”
    Quote in the paperNot a substantial claim
44 of 44

  • The paper relies on this without support.

    Long-horizon memory is critical for achieving strong performance in autonomous agents.

    “where enabling long-horizon memory is critical for achieving strong performance.”

  • The paper relies on this without support.

    Agent memory consists of a continuous stream of agent-environment interactions that are primarily composed of machine-generated representations.

    “In reality, agent memory consists of a continuous stream of agent-environment interactions that are primarily composed of machine-generated representations.”

  • The paper relies on this without support.

    Real-world agents mainly operate in machine-generated environments such as databases, code executors, and web interfaces.

    “Real-world agents mainly operate in machine generated environments such as databases, code executors, and web interfaces”

  • The paper relies on this without support.

    Agent trajectories are causally grounded, where each action induces a latent environment state transition that constrains subsequent observations.

    “agent trajectories are causally grounded, where each action induces a latent environment state transition that constrains subsequent observations”

  • The paper relies on this without support.

    Existing benchmarks follow unconstrained linguistic flow.

    “existing benchmarks follow unconstrained linguistic flow”

  • The paper relies on this without support.

    Dialog-centric benchmarks contain abundant redundant information such as phatic chit-chat.

    “dialog-centric benchmarks contain abundant redundant information such as phatic chit-chat”

  • The paper relies on this without support.

    The expert-level demonstrations from ALFRED ensure high-quality task completion in both familiar and novel environments.

    “These trajectories are generated using the expert-level demonstrations from ALFRED (Shridhar et al., 2020a) to ensure high-quality task completion in both familiar and novel environments.”

  • The paper relies on this without support.

    Independent human annotation provides gold labels for evaluating the LLM-as-judge.

    “We obtain gold labels via independent human annotation.”

  • The paper relies on this without support.

    The needle-in-a-haystack QA generation pipeline produces QA pairs that evaluate long-context retrieval capabilities of memory-augmented agents.

    “To evaluate the long-context retrieval capabilities of memory-augmented agents, we developed a structured pipeline to generate QA pairs where the answer is anchored to specific”

  • The paper relies on this without support.

    The four functional capabilities (Recall, Causal Inference, State Updating, State Abstraction) capture the essential mechanisms of agent memory.

    “Aligning these essential mechanisms with the specific requirements of agent-based tasks, we categorize memory capabilities into four functional capabilities.”

  • The paper relies on this without support.

    The six domains of agent-environment interactions in the real-world subset are representative of agentic applications.

    “This section details the composition of the real-world subset, which comprises multi-turn trajectories curated from six diverse domains of agent-environment interactions (Tab.7).”
11 of 11

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AMA-Bench evaluates memory over real and synthetic agentic trajectories with QA.

    “AMA-Bench [57] evaluates memory over real and synthetic agentic trajectories with QA”

    From MemGym: a Long-Horizon Memory Environment for LLM Agents (2026) · cites this paper as [57]

  • AMA-Bench focuses on understanding one trajectory, whereas LME-V2 focuses on environment knowledge induced across many past trajectories.

    “However, AMA-Bench focuses on understanding one trajectory, while LME-V2 focuses on environment knowledge induced across many past trajectories.”

    From LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues (2026) · cites this paper as Zhao et al., 2026b

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Qwen2.5-1M context length differs by a factor of ten

    Minor

    The text says the Qwen2.5 1M series extends to 100,000 tokens, while Table 9 lists 1,010,000 tokens for its long-context checkpoint.

  • Embodied AI trajectory count differs between tables

    Minor

    Table 7 lists 33 selected Embodied AI trajectories, whereas Table 8 lists 30 samples for that domain.

  • GPT 5.2 average does not round to 0.73

    Minor

    Table 4 reports an average accuracy of 0.7226, which rounds to 0.72 at two decimal places, not the stated 0.73.

  • Software Engineering trajectory count differs

    Minor

    The description says 36 Software Engineering trajectories were selected, but Table 7 lists 34 selected. Table 8 also lists 36.

4 of 4

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Zhao, Y., Yuan, B., Huang, J., Yuan, H., Yu, Z., Xu, H., Hu, L., Shankarampeta, A., Huang, Z., Ni, W., Tian, Y., & Zhao, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2602.22769
arXiv
2602.22769
OpenAlex
W7131825876
Cited by
0
References
48
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
agent memory, agent trajectories, large language models, causal graphs, synthetic trajectories

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review