MemGym: a Long-Horizon Memory Environment for LLM Agents

May combine two claims.

Measured on local trajectory data under Claude Sonnet 4.5, a single passthrough-memory episode costs $2.10 on SWE-Gym (median 55 turns and 700K cumulative input tokens), $1.50 on WebArena-Infinity hard tier, $0.40 on τ2-bench, and $0.15 on MEMGYM-DR.

“Measured on local trajectory data under Claude Sonnet 4.5, a single passthrough-memory episode costs $2.10 on SWE-Gym [32] (median 55 turns and 700K cumulative input tokens), $1.50 on WebArena-Infinity [60] hard tier, $0.40 on τ 2 -bench [5], and $0.15 on MEMGYM-DR at list pricing of $3/MTok input plus $15/MTok output.”

• cites [32] [32] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-Gym. arXiv preprint arXiv:2412.21139 , 2024. - sourceSupported
• cites [60] [60] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854 , 2023. sourceCould not verify

  Section 3 describes the original WebArena benchmark and its 812 test examples. Our copy does not address the cited WebArena-Infinity hard tier or its claimed $1.50 episode cost under Claude Sonnet 4.5 (searched for WebArena-Infinity, hard tier, and the pricing terms). (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [5] [5] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2 -Bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982 , 2025. - sourceSupported

May combine two claims.

The strongest non-A-Mem baselines are domain-dependent: rolling Summary on the coding QA axis and Naive RAG on retrieval.

“The strongest non-A-Mem baselines are domain-dependent (rolling Summary on the coding QA axis and Naive RAG on retrieval), both of which A-Mem beats by a comfortable margin under maximum pressure.”

• cites [44] [44] Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, and Li Guo. Recursively summarizing enables long-term dialogue memory in large language models. arXiv preprint arXiv:2308.15022 , 2023. - sourceCould not verify

  Table 2 and §5.1 evaluate long-term dialogue responses on the MSC dataset, comparing Chatgpt-Rsum with MemoryBank and MemoChat. Searches for A-Mem, coding QA, rolling Summary, and Naive RAG found no passage ranking those baselines by domain or reporting an A-Mem comparison under maximum pressure. The supplied copy gives no page numbers for these passages. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [25] [25] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems , 33:9459-9474, 2020. - sourceCould not verify

  Section 4.1 (pp. 5–6) compares RAG with open-domain QA models, and Section 4.5 (p. 7) examines retrieval ablations. Searches for A-Mem, rolling Summary, coding QA, Naive RAG, and maximum pressure found no passage comparing those baselines or reporting A-Mem’s performance. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

May combine two claims.

A-Mem is the best-performing strategy at the maximum-pressure point on MEMGYM-CODEQA (500k-token budget), reaching 0.75 accuracy, +0.55 over the no-memory baseline.

“A-Mem is the best-performing strategy at the maximum-pressure point on both benchmarks, reaching 0 . 75 on MEMGYM-CODEQA at the 500 k-token budget ( +0 . 55 vs. the nomemory baseline) and 0 . 518 on MEMGYM-DR at 5 / 6 -hop ( +0 . 509 vs. baseline).”

• cites [50] [50] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110 , 2025. - sourceCould not verify

  The empirical results (Section 4.3, p. 6) report A-MEM’s performance on LoCoMo and DialSim, not MEMGYM-CODEQA. Searches for MEMGYM-CODEQA, the 500k-token budget, 0.75 accuracy, and the no-memory baseline found no passage in our copy that verifies the claimed result. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A-Mem introduced an explicitly agentic memory mechanism: each new interaction is converted into a structured note, linked to related memories, and allowed to evolve existing memory representations.

“A-Mem [50] introduced an explicitly agentic memory mechanism: each new interaction is converted into a structured note, linked to related memories, and allowed to evolve existing memory representations.”

• cites [50] [50] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110 , 2025. - sourceSupported

May combine two claims.

Within the coding domain, the MEMRM gate transfers to memory strategies that did not appear in training (sliding-window, observation-masking, and structured memories), reaching AUROC ≈ 0.71 at ∼27% coverage on the cohorts retained by a pre-declared per-axis selection rule based on selective classification [12].

“Within the coding domain, the gate transfers to memory strategies that did not appear in training (sliding-window, observation-masking, and structured memories), reaching AUROC ≈ 0 . 71 at ∼ 27% coverage on the cohorts retained by a pre-declared per-axis selection rule based on selective classification [12].”

• cites [12] [12] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems , 2017. - sourceSupported

May combine two claims.

A-Mem is the best-performing strategy at the maximum-pressure point on MEMGYM-DR (5/6-hop), reaching 0.518 judge score, +0.509 over the no-memory baseline.

“A-Mem is the best-performing strategy at the maximum-pressure point on both benchmarks, reaching 0 . 75 on MEMGYM-CODEQA at the 500 k-token budget ( +0 . 55 vs. the nomemory baseline) and 0 . 518 on MEMGYM-DR at 5 / 6 -hop ( +0 . 509 vs. baseline).”

• cites [50] [50] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110 , 2025. - sourceCould not verify

  The empirical results (§4.3, p. 6) evaluate A-MEM on LoCoMo and DialSim, not MEMGYM-DR. Searches for MEMGYM-DR, 5/6-hop, and the reported 0.518 and 0.509 scores found no passage in our copy that verifies the claimed result or no-memory comparison. (LLM, conf=0.77)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

A-Mem evaluates on LoCoMo, a very-long-term conversational-memory benchmark built from 300-turn dialogues averaging 9K tokens across up to 35 sessions, with question answering, event summarization, and multimodal dialogue generation tasks.

“It evaluates on LoCoMo [29], a very-long-term conversational-memory benchmark built from 300-turn dialogues averaging 9K tokens across up to 35 sessions, with question answering, event summarization, and multimodal dialogue generation tasks.”

• cites [29] [29] Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of llm agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 13851-13870, 2024. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

LongMemEval adds six explicit memory competencies (information extraction, multi-session reasoning, knowledge update, temporal reasoning, single-session preference, abstention) over scalable conversational histories.

“LongMemEval [48] adds six explicit memory competencies (information extraction, multi-session reasoning, knowledge update, temporal reasoning, single-session preference, abstention) over scalable conversational histories.”

• cites [48] [48] Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Longmemeval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813 , 2024. - sourcePartly supported

  Section 3.2 (p. 4) defines five core memory abilities, not six: information extraction, multi-session reasoning, knowledge updates, temporal reasoning, and abstention. It separately lists single-session-preference as one of seven question types. The Introduction (p. 1) supports the claim that the benchmark uses freely extensible conversational histories. (LLM, conf=0.90)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

MemoryAgentBench adds incremental multi-turn ingestion to test how an agent updates memory as new information arrives.

“MemoryAgentBench [18] adds incremental multi-turn ingestion to test how an agent updates memory as new information arrives.”

• cites [18] [18] Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in llm agents via incremental multi-turn interactions. arXiv preprint arXiv:2507.05257 , 2025. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

REALTALK provides a 21-day real-world conversation dataset for long-term dialogue memory.

“REALTALK [23] provides a 21-day real-world conversation dataset for long-term dialogue memory.”

• cites [23] [23] Dong-Ho Lee et al. Realtalk: A 21-day real-world dataset for long-term conversation. arXiv preprint arXiv:2502.13270 , 2025. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Long-context evaluation is anchored by Needle-in-a-Haystack, RULER, LongBench, and LongBench v2, which probe what an LLM can recall from a single long input.

“Long-context evaluation is anchored by Needle-in-a-Haystack [21], RULER [17], LongBench [3], and LongBench v2 [4], which probe what an LLM can recall from a single long input.”

• cites [21] [21] Greg Kamradt. Needle in a haystack - pressure testing LLMs. https://github.com/ gkamradt/LLMTest\_NeedleInAHaystack , 2023. GitHub repository. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [17] [17] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 , 2024. -Partly supported

  The Introduction (p. 1) identifies RULER as a long-context benchmark and says its retrieval tasks extend Needle-in-a-Haystack, supporting the recall aspect of the claim. But the Abstract (p. 1) and Conclusion (p. 10) explicitly say RULER also tests multi-hop tracing, aggregation, and question answering—abilities beyond retrieving information from a long input. The characterization is therefore too narrow for RULER. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [3] [3] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages 3119-3137, 2024. -Partly supported

  The Introduction identifies LongBench as a benchmark for evaluating long-context understanding, supporting its inclusion in the list. But the Introduction and §3.2.1 describe tasks including summarization, code completion, and passage retrieval; characterizing the benchmark as probing only what an LLM can recall from a single long input is too narrow. This verdict concerns LongBench, not the other benchmarks cited separately in the sentence. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

WebArena provides fully functional web environments across e-commerce, forums, collaborative software development, and content management.

“WebArena [60] provides fully functional web environments across e-commerce, forums, collaborative software development, and content management”

• cites [60] [60] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854 , 2023. sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

OSGym extends this to a gym-style training surface.

“OSGym [36] extends this to a gym-style training surface.”

• cites [36] [36] Zengyi Qin, Jinyuan Chen, Yunze Man, Shengcao Cao, Ziqi Pang, Zhuoyuan Wang, Xin Sun, Gen Lin, Han Fang, Ling Zhu, et al. Osgym: Super-scalable distributed data engine for generalizable computer agents. arXiv preprint arXiv:2511.11672 , 2025. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

AutoCompressors learn soft summary vectors end-to-end so a long context can be replaced by a short sequence of compressed embeddings.

“AutoCompressors [7] learn soft summary vectors end-to-end so a long context can be replaced by a short sequence of compressed embeddings.”

• cites [7] [7] Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 3829-3846, 2023. -Supported

Probably not a substantial claim: may not pass S3 (load-bearing).

MemoryBench uses simulated user feedback to drive continual learning across sessions.

“MemoryBench [1] uses simulated user feedback to drive continual learning across sessions.”

• cites [1] [1] Qingyao Ai, Yichen Tang, Changyue Wang, Jianming Long, Weihang Su, and Yiqun Liu. Memorybench: A benchmark for memory and continual learning in llm systems. arXiv preprint arXiv:2510.17281 , 2025. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue) and long-document QA such as QuALITY test compositional retrieval and reasoning over a fixed corpus.

“Multi-hop QA benchmarks (HotpotQA [52], 2WikiMultiHopQA [16], MuSiQue [42]) and long-document QA such as QuALITY [33] test compositional retrieval and reasoning over a fixed corpus.”

• cites [52] [52] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing , pages 2369-2380, 2018. - sourceSupported
• cites [16] [16] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics , pages 6609-6625, 2020. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [42] [42] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics , 10:539-554, 2022. - sourceSupported

Classified as probably not a substantial claim.

SWE-Gym adds 2,438 Python training tasks with executable runtimes, unit tests, and released trajectories.

“SWE-Gym [32] adds 2,438 Python training tasks with executable runtimes, unit tests, and released trajectories.”

• cites [32] [32] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-Gym. arXiv preprint arXiv:2412.21139 , 2024. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The OpenHands framework provides the strategy-library lineage on which our memory-operation library is built.

“The OpenHands framework [45] provides the strategy-library lineage on which our memory-operation library is built.”

• cites [45] [45] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 , 2024. - sourcePartly supported

  Section 2.3 (p. 5) describes OpenHands’ extensible AgentSkills library, supporting the claim that it provides a library-based model for agent capabilities. The paper does not describe that library as a strategy library or establish the claimed lineage to the citing authors’ memory-operation library. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

OSWorld broadens computer-use evaluation to desktop and operating-system tasks under VM-style infrastructure with GUI grounding.

“OSWorld [49] broadens computer-use evaluation to desktop and operating-system tasks under VM-style infrastructure with GUI grounding”

• cites [49] [49] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems , 37:52040-52094, 2024. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

AMA-Bench evaluates memory over real and synthetic agentic trajectories with QA.

“AMA-Bench [57] evaluates memory over real and synthetic agentic trajectories with QA”

• cites [57] [57] Yujie Zhao, Boqin Yuan, Junbo Huang, Haocheng Yuan, Zhongming Yu, Haozhou Xu, Lanxiang Hu, Abhilash Shankarampeta, Zimeng Huang, Wentao Ni, Yuandong Tian, and Jishen Zhao. Ama-bench: Evaluating long-horizon memory for agentic applications. arXiv preprint arXiv:2602.22769 , 2026. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

AMemGym provides on-policy interactions with simulated users and structured latent-state evolution.

“AMemGym [6] provides on-policy interactions with simulated users and structured latent-state evolution.”

• cites [6] [6] Jiayang Cheng, Dongyu Ru, Lin Qiu, Yiyang Li, Xuezhi Cao, Yangqiu Song, and Xunliang Cai. Amemgym: Interactive memory benchmarking for assistants in long-horizon conversations. arXiv preprint arXiv:2603.01966 , 2026. - sourceSupported

May combine two claims.

Both LLMLingua and AutoCompressors target inference-time efficiency and assume a static prompt.

“Both target inference-time efficiency and assume a static prompt”

• cites [19] [19] Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing , pages 13358-13376, 2023. - sourcePartly supported

  The Problem Formulation (p. 3) describes LLMLingua as compressing a given original prompt to lower inference cost, supporting the inference-efficiency claim and a fixed prompt for each compression. Our copy does not establish that AutoCompressors has either property, so it does not support the claim about both methods. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [7] [7] Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 3829-3846, 2023. -Partly supported

  The Abstract and Introduction state that AutoCompressors reduce inference costs. But the Introduction describes summaries produced segment by segment, and Related Work says their soft prompts are predicted as a function of the context; the paper does not support the claim that AutoCompressors assume a static prompt. This source also does not establish the claim about LLMLingua. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Process reward models extend this to step-level supervision over reasoning traces.

“process reward models [26] extend this to step-level supervision over reasoning traces.”

• cites [26] [26] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The twelfth international conference on learning representations , 2023. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMLingua compresses prompts at the token level using a small language model to score perplexity, removing low-information tokens before the prompt is sent to the target model.

“LLMLingua [19] compresses prompts at the token level using a small language model to score perplexity, removing low-information tokens before the prompt is sent to the target model.”

• cites [19] [19] Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing , pages 13358-13376, 2023. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

τ-bench and τ2-bench evaluate tool-agent-user workflows in domains such as retail, airline, and telecom.

“τ -bench and τ 2 -bench [55, 5] evaluate tool-agent-user workflows in domains such as retail, airline, and telecom”

• cites [55] [55] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045 , 2024. - sourceSupported
• cites [5] [5] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2 -Bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982 , 2025. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

MemoryArena evaluates agent memory in interdependent multi-session agentic tasks, where later sessions depend on facts established earlier.

“MemoryArena [15] evaluates agent memory in interdependent multi-session agentic tasks, where later sessions depend on facts established earlier.”

• cites [15] [15] Zexue He, Yu Wang, Churan Zhi, et al. Memoryarena: Benchmarking agent memory in interdependent multi-session agentic tasks. arXiv preprint arXiv:2602.16313 , 2026. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

SWE-bench evaluates agents on real GitHub issue resolution, with the original test split spanning thousands of bug reports across mature Python repositories.

“SWE-bench [20] evaluates agents on real GitHub issue resolution, with the original test split spanning thousands of bug reports across mature Python repositories.”

• cites [20] [20] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world GitHub issues? 2023. - sourcePartly supported

  The Introduction (p. 1) says SWE-bench evaluates models on real GitHub issues, and Table 10 (p. 20) lists 2,294 final tasks from 12 Python repositories. But the Introduction describes issues as bug reports or feature requests, while Appendix A.5 (p. 21) notes code changes beyond bug fixes; the source does not establish that the test split contains thousands of bug reports specifically. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

MemGym reports memory-isolated scores that disentangle memory performance from the underlying task, so memory strategies can be ranked without being confounded by reasoning, retrieval, or tool-use ability.

“On top of this interface, MEMGYM reports memory-isolated scores that disentangle memory performance from the underlying task, so that memory strategies can be ranked without being confounded by reasoning, retrieval, or tool-use ability.”

• Also stated in: § Abstract · p. 1 · plain

  “MemGym reports memory-isolated scores that decouple memory performance from reasoning, retrieval, and tool-use ability, so memory strategies can be ranked without those confounders.”

May combine two claims.

MEMRM ranks held-out compression events near-perfectly with calibrated probabilities and remains deployable on a characterized OOD subset.

“MEMRM ranks held-out compression events near-perfectly with calibrated probabilities and remains deployable on a characterized OOD subset (Section 4.4).”

May combine two claims.

Without fictionalization, frontier models score 0.70-0.85 on MEMGYM-DR by answering from pretraining alone; with fictionalization, no-memory performance drops to near-zero and the memory gap rises to 0.85-0.95.

“Without fictionalization, frontier models score 0 . 70 -0 . 85 on MEMGYM-DR by answering from pretraining alone; with it, no-memory drops to near-zero and the memory gap rises to 0 . 85 -0 . 95 (Section C.6).”

May combine two claims.

On the SWE-Gym held-out IID split, MEMRM reaches AUROC 0.985 with near-zero ECE.

“On the SWE-Gym held-out IID split, MEMRM reaches AUROC 0 . 985 with near-zero ECE: the gate ranks compression events near-perfectly and its predicted probabilities are calibrated.”

May combine two claims.

On SWE-Gym, memory buys context compression with per-episode ratios of 1.32-1.47× across the three reasoners, keeping long trajectories within the policy's effective context without changing the resolve rate.

“What memory does buy on SWE-Gym is context compression (per-episode ratios of 1 . 32 -1 . 47 × across the three reasoners), which keeps long trajectories within the policy's effective context without changing the resolve rate.”

May combine two claims.

The augmented MEMRM training corpus contains 18.6K (context, action, label) triples at a median context length of 22K tokens.

“The augmented corpus contains 18 . 6 K (context, action, label) triples at a median context length of 22 K tokens; the train/eval split is grouped by repository (not by instance) to eliminate same-repo leakage.”

May combine two claims.

A 100-episode sweep across the four interactive environments costs $420, and a 5-strategy × 3-seed sweep scales to $6,300.

“A 100-episode sweep across the four interactive environments therefore costs $420 and a 5-strategy × 3-seed sweep scales to $6,300, making per-episode rollouts infeasible as the inner-loop signal for strategy iteration, ablation sweeps, or rollout filtering during training.”

May combine two claims.

MEMGYM-CODEQA and MEMGYM-DR generate length-tunable instances at scale and verify via per-stage ablations that the intended memory channel, not parametric leakage or distractor recall, is the one being tested.

“MEMGYM-CODEQA and MEMGYM-DR generate length-tunable instances at scale, and verify via per-stage ablations that the intended memory channel (not parametric leakage or distractor recall) is the one being tested (§3.4).”

• Also stated in: § Abstract · p. 1 · plain

  “Our synthetic pipelines for MEMGYM-CODEQA and MEMGYM-DR are length-controllable, ablation-verified at every stage, and tightly aligned with downstream scenarios.”

May combine two claims.

A single-check verifier had a 62% false-positive rate, which is why the three checks are jointly necessary.

“the three checks are jointly necessary because a single-check verifier had a 62% false-positive rate”

The quote only partly states this claim.

The same strategy ordering reproduces on MEMGYM-CODEQA and MEMGYM-DR at the largest pressure point on each.

“The same ordering reproduces on two unrelated domains with different bottlenecks; this consistency is the contribution rather than any single number.”

May combine two claims.

Under controlled pressure on two unrelated synthetic axes, the same strategy ranking reproduces and A-Mem leads at the maximum-pressure point on both.

“Under controlled pressure on two unrelated synthetic axes, the same strategy ranking reproduces and A-Mem leads at the maximum-pressure point on both (Section 4.3).”

May combine two claims.

On the synthetic axes, the same strategy ranking reproduces at the largest pressure point on each, with A-Mem the most robust under maximum pressure where flat retrievers fail.

“On the synthetic axes, the same strategy ranking reproduces on two unrelated domains at the largest pressure point on each, with A-Mem the most robust under maximum pressure where flat retrievers fail.”

May combine two claims.

On τ2-bench with Haiku 4.5, Structured memory raises success from 57.6% to 60.1% (∆ = +2.5 pp), at a compression ratio of 1.86×.

“| τ 2 -bench | Haiku 4.5 | Structured | 288 | 57.6 | 60.1 | + 2.5 | 1.86 × |”

May combine two claims.

On SWE-Gym with Sonnet 4.5 and Summary memory, resolve rate is 42.8% both with and without memory (∆ = 0.0 pp), at a compression ratio of 1.47×.

“| SWE-Gym | Sonnet 4.5 | Summary | 1041 | 42.8 | 42.8 | 0.0 | 1.47 × |”

• Also stated in: § 4.2 Memory Across Three Wrapped Gyms · p. 7 · plain

  “Sonnet 4.5 absorbs the lossy summary at zero cost ( ∆=0 ), Haiku 4.5 takes a small drop, and GPTOSS-120B drops the most, reflecting weaker comprehension under compressed context.”

May combine two claims.

The no-memory baseline reaches 0.20 on MEMGYM-CODEQA and 0.009 on MEMGYM-DR's 5/6-hop slice.

“Without memory, both benchmarks collapse under pressure: the no-memory baseline reaches only 0 . 20 on MEMGYM-CODEQA and 0 . 009 on MEMGYM-DR's 5 / 6 -hop slice (essentially random).”

May combine two claims.

The current corpora are 670 verified MEMGYM-CODEQA instances (2,131 deduplicated QA pairs from a 1,000-instance candidate pool) and 1,194 verified MEMGYM-DR instances (161 3-hop, 916 4-hop, 117 5/6-hop).

“The current corpora are 670 verified MEMGYM-CODEQA instances (2,131 deduplicated QA pairs from a 1,000-instance candidate pool) and 1,194 verified MEMGYM-DR instances (161 3-hop, 916 4-hop, 117 5/6-hop).”

May combine two claims.

Cross-entropy descends steadily over 600 steps and IID AUROC plateaus from step 200 onward during MEMRM training.

“Figure 4 traces the training-time dynamics: cross-entropy descends steadily over 600 steps and IID AUROC plateaus from step 200 onward, so the headline metric is not the product of a lucky final checkpoint.”

May combine two claims.

Across the three wrapped gyms, memory is roughly information-neutral on coding and clearly beneficial on dialogue and web.

“Across the three wrapped gyms, memory is roughly information-neutral on coding (where progress lives in the file system and the reasoner can re-read what was summarized away) and clearly beneficial on dialogue and web, where state hidden in past turns is expensive to re-derive (Table 2).”

• Also stated in: § 4 Experiments: When Memory Does and Does Not Matter · p. 6 · boosted

  “Memory's payoff is regime-dependent across three wrapped gyms: roughly neutral on coding, where progress lives in the file system, and clearly positive on dialogue and web (Section 4.2).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The grid contains 17 verified cells and 1 contaminated cell.

“The grid contains 17 verified cells and 1 contaminated cell.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Plugging in the values above yields 446/1041, 431/1003, and 192/1003 as the full-1k +memory resolution counts for Sonnet 4.5, Haiku 4.5, and GPT-OSS-120B respectively.

“Plugging in the values above yields 446 / 1041 , 431 / 1003 , and 192 / 1003 as the full-1k + memory resolution counts for Sonnet 4.5, Haiku 4.5, and GPT-OSS-120B respectively.”

• AssumptionThe authors state this as an assumption: The paired delta observed on the subset can be extrapolated to the full evaluation set.

May combine two claims.

Only a fraction of the full ~1,000-instance evaluation set produces a non-trivial paired comparison: 678/1041 (65.1%) for Sonnet 4.5, 653/1003 (65.1%) for Haiku 4.5, and 556/1003 (55.4%) for GPT-OSS-120B.

“only a fraction of the full ∼ 1 , 000 -instance evaluation set produces a non-trivial paired comparison: 678 / 1041 ( 65 . 1% ) for Sonnet 4.5, 653 / 1003 ( 65 . 1% ) for Haiku 4.5, and 556 / 1003 ( 55 . 4% ) for GPT-OSS-120B.”

Probably not a substantial claim: may not pass S3 (load-bearing).

On the 1,000-instance candidate pool, 1,325 QA pairs (17.2%) failed solvability.

“Broken (fails solvability) | 1,325 (17.2%)”

May combine two claims.

Linear benefits strongly in smart replay, rising from 13% to 40% with summarizing memory.

“Linear also benefits strongly, rising from 13% to 40% with summarizing memory.”

May combine two claims.

These fixes added approximately 37 resolved instances for Sonnet 4.5 (129 → 166) and 15 for GPTOSS-120B (77 → 92) on the 500-instance evaluation.

“These fixes added approximately 37 resolved instances for Sonnet 4.5 (129 → 166) and 15 for GPTOSS-120B (77 → 92) on the 500-instance evaluation”

Probably not a substantial claim: may not pass S3 (load-bearing).

The repo-grouped split yields 15,630 train / 3,007 eval examples.

“the repo-grouped split yields 15 , 630 train / 3 , 007 eval examples.”

May combine two claims.

Paypal and gitlab are mostly flat in smart replay.

“Paypal and gitlab are mostly flat”

May combine two claims.

When fed non-conversational MEMGYM-DR content (paragraphs of scientific prose), the LOCOMO template causes the metadata stage to emit empty or hallucinated entries, which propagate to the vector index and degrade retrieval.

“When fed non-conversational MEMGYM-DR content (paragraphs of scientific prose), this template causes the metadata stage to emit empty or hallucinated entries, which propagate to the vector index and degrade retrieval.”

May combine two claims.

The scenario-OOD covered subset (n=87, two linear WebArena cohorts) is class-balanced, so all metrics agree: AUROC 0.748, AUPRC 0.785, HARMFUL-F1 0.747 at t*, ECE 0.237.

“The scenario-OOD covered subset ( n =87 , two linear WebArena cohorts) is class-balanced, so all metrics agree: AUROC 0 . 748 , AUPRC 0 . 785 , HARMFUL-F1 0 . 747 at t ∗ , ECE 0 . 237”

Probably not a substantial claim: may not pass S3 (load-bearing).

On the 1,000-instance candidate pool, 1,642 QA pairs (21.4%) failed the distractor check.

“Confusable (fails distractor check) | 1,642 (21.4%)”

May combine two claims.

The strategies differ substantially in compression ratio achieved and in how often they trip the limits-exceeded ceiling.

“but they differ substantially in compression ratio achieved and in how often they trip the limits-exceeded ceiling”

Probably not a substantial claim: may not pass S3 (load-bearing).

The per-episode last-ratio on telecom (1.38×) is among the least aggressive.

“but the per-episode last-ratio (1.38 × ) is among the least aggressive”

May combine two claims.

The strategy-OOD covered subset (n=44, two structured-memory cohorts) is class-imbalanced, so we restrict the strong claim to AUROC 0.714 [0.544, 0.872].

“The strategy-OOD covered subset ( n =44 , two structured-memory cohorts) is class-imbalanced, so we restrict the strong claim to AUROC 0 . 714 [ 0 . 544 , 0 . 872 ]”

Probably not a substantial claim: may not pass S3 (load-bearing).

The default 60-second timeout was insufficient for 500MB-2GB images.

“The default 60-second timeout was insufficient for 500MB-2GB images.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Pydantic's output format was incompatible with the default log parser.

“Pydantic's output format was incompatible with the default log parser.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The repo-grouped split is 15,630 train / 3,007 eval.

“The repo-grouped split is 15 , 630 train / 3 , 007 eval”

May combine two claims.

In the Apr. 5 100K-token pilot, no-memory scores remained at 0.70-0.85 even though the task was meant to require retention across turns.

“In the Apr. 5 100K-token pilot, no-memory scores remained at 0.70-0.85 even though the task was meant to require retention across turns.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Of the 18,637 labeled events, 2,280 (12.2%) are SAFE.

“2 , 280 SAFE ( 12 . 2% )”

Probably not a substantial claim: may not pass S3 (load-bearing).

Bash set -u with unset conda variables caused crashes.

“Bash set -u with unset conda variables caused crashes.”

May combine two claims.

Compared to a vanilla-CE baseline (A0), C1 improves ECE by 3× and halves the HARMFUL false-alarm rate.

“Compared to a vanilla-CE baseline (A0), C1 improves ECE by 3 × and halves the HARMFUL false-alarm rate.”

May combine two claims.

The asymmetric thresholds follow the cohort-size distributions (a discrete cliff between n=23 and n=43 on strategy-OOD, with no cohorts in between); both are pre-declared, not tuned.

“The asymmetric thresholds follow the cohort-size distributions (a discrete cliff between n =23 and n =43 on strategy-OOD, with no cohorts in between); both are pre-declared, not tuned.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The final yield after dedup is 670 instances and 2,131 QA pairs.

“After dedup: final instances | 670”

• Also stated in: § D Synthetic Pipeline Details · p. 24 · plain

  “After dedup: final QA pairs | 2,131”

Classified as probably not a substantial claim.

τ2-bench is excluded from this analysis because its runtime persists neither step.context nor the post-compaction summary text for compaction-positive steps, so the (history, task, compressed memory) input triplet that anchors Table 3 cannot be reconstructed from on-disk artefacts.

“We also exclude τ 2 -bench from this analysis: its runtime persists neither step.context nor the post-compaction summary text for compaction-positive steps, so the (history, task, compressed memory) input triplet that anchors Table 3 cannot be reconstructed from on-disk artefacts”

Probably not a substantial claim: may not pass S3 (load-bearing).

The pytest -k flag cannot parse mypy's [case] test markers.

“The pytest -k fl ag cannot parse mypy's [case] test markers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

On the 1,000-instance candidate pool, 648 QA pairs (8.4%) were both confusable and leaky.

“Confusable and leaky | 648 (8.4%)”

May combine two claims.

Tightening to a uniform N ≥ 30 collapses strategy-OOD coverage to 0% but leaves scenario-OOD unchanged, and N ≥ 40 matches N ≥ 30 on both axes.

“Tightening to a uniform N ≥ 30 collapses strategy-OOD coverage to 0% but leaves scenario-OOD unchanged, and N ≥ 40 matches N ≥ 30 on both axes”

May combine two claims.

MEMRM is calibrated on a subset of cohorts and uncalibrated or polarity-inverted on others.

“MEMRM is calibrated on a subset of cohorts and uncalibrated or polarity-inverted on others.”

Probably not a substantial claim: may not pass S4 (based).

A single SWE-Gym rollout requires Docker infrastructure and tens of execution steps, placing systematic memory design iteration beyond most academic budgets.

“Evaluation cost: A single SWE-Gym rollout requires Docker infrastructure and tens of execution steps, placing systematic memory design iteration beyond most academic budgets.”

Probably not a substantial claim: may not pass S4 (based).

Settings that appear memory-intensive often admit strong performance without explicit memory management, because facts remain re-derivable from repositories or recoverable from pretraining.

“Illusory memory pressure: Settings that appear memory-intensive often admit strong performance without explicit memory management, as facts remain re-derivable from repositories or recoverable from pretraining.”

Classified as probably not a substantial claim. May combine two claims.

Trajectory compaction is triggered uniformly at 100 messages or 32K context tokens, whichever fires first, and the same trigger is used for every memory strategy in a comparison.

“Trajectory compaction is triggered uniformly at 100 messages or 32 K context tokens, whichever fires first, and the same trigger is used for every memory strategy in a comparison.”

May combine two claims.

The MEMGYM-CODEQA verifier runs three independent checks per QA pair (solvability, distractor-confusion, question-leakage), each catching a distinct shortcut.

“The verifier in MEMGYM-CODEQA runs three independent checks per QA pair (solvability, distractor-confusion, and question-leakage), each catching a distinct shortcut”

Classified as probably not a substantial claim.

The MEMRM train/eval split is grouped by repository rather than by instance, to eliminate same-repo leakage.

“the train/eval split is grouped by repository (not by instance) to eliminate same-repo leakage”

Classified as probably not a substantial claim.

A MEMGYM-CODEQA instance is kept only if it exposes at least two critical memory-only facts.

“an instance is kept only if it exposes at least two critical memory-only facts (the threshold below which the question reduces to 'search the repo').”

May combine two claims.

The MEMRM gate ranks compression events near-perfectly and its predicted probabilities are calibrated on the SWE-Gym held-out IID split.

“On the SWE-Gym held-out IID split, MEMRM reaches AUROC 0 . 985 with near-zero ECE: the gate ranks compression events near-perfectly and its predicted probabilities are calibrated.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The shared pipeline template (hidden gold artifact, memory-only/discoverable taxonomy, composable hardening, multi-criterion verification) generalizes to any domain with a hidden ground-truth artifact and external documentation.

“The shared template (hidden gold artifact, memoryonly/discoverable taxonomy, composable hardening, and multi-criterion verification) generalizes to any domain with a hidden ground-truth artifact and external documentation (medical guideline adherence, legal precedent application, scientific reproduction).”

May combine two claims.

MEMRM (Qwen3-1.7B fine-tuned with QLoRA) trades a multi-minute Docker rollout for a sub-second compression-quality call, which is what makes the full coding-environment evaluation academically tractable.

“Alongside the wrappers, MEMRM (Qwen3-1.7B fine-tuned with QLoRA) trades a multi-minute Docker rollout for a sub-second compression-quality call, which is what makes the full coding-environment evaluation academically tractable.”

May combine two claims.

In MEMGYM-CODEQA, difficulty is a composable post-hoc dial (prompt fuzzing, distractor scaling, indirection, fact fragmentation), so length and noise are decoupled from the underlying instance.

“difficulty is then a composable post-hoc dial (prompt fuzzing, distractor scaling, indirection, fact fragmentation), so length and noise are decoupled from the underlying instance.”

Classified as probably not a substantial claim. May combine two claims.

MEMRM is trained only on SWE-Gym compression events; OOD probes on memory-strategy and scenario axes are evaluation-only.

“It is trained only on SWE-Gym compression events; OOD probes on memory-strategy and scenario axes (Table 3) are evaluation-only.”

May combine two claims.

Memory gain attributes effects to the memory module under a fixed reasoner but is not a clean ablation of memory as an independent capability, because memory still alters the action distribution downstream of the wrapper.

“Memory still alters the action distribution downstream of the wrapper, so memory gain attributes effects to the memory module under a fixed reasoner; it is not a clean ablation of memory as an independent capability.”

May combine two claims.

MEMGYM fixes the memory interface across five tracks and uses paired memory-isolated scoring so any of the above mechanisms can be plugged in and compared on the same memory event.

“MEMGYM is complementary: it fixes the memory interface across five tracks and uses paired memory-isolated scoring so any of the above mechanisms can be plugged in and compared on the same memory event.”

• AssumptionRests on an assumption: The memory interface can be fixed across five tracks so mechanisms can be plugged in and compared on the same memory event.

May combine two claims.

A-Mem's lead over Naive RAG widens with hop count because note-evolution links bridge passages whose isolated query-relevance is low, which is the failure mode of a flat retriever at 5/6-hop.

“The payoff of memory therefore grows with pressure, and the gap is most visible on retrieval depth: A-Mem's lead over Naive RAG widens with hop count because note-evolution links bridge passages whose isolated query-relevance is low, which is the failure mode of a flat retriever at 5 / 6 -hop.”

Probably not a substantial claim: may not pass S3 (load-bearing).

OOD rows report AUROC on the covered subset selected by a pre-declared per-axis selection rule.

“OOD rows report AUROC on the covered subset selected by a pre-declared per-axis selection rule; the rule, aggregate-OOD numbers, bootstrap protocol, and per-event detail metrics are in Appendix H and Appendix H.”

Classified as probably not a substantial claim. May combine two claims.

τ2-bench wraps both the agent and the user simulator with independent memory managers, enabling a three-condition ablation that isolates whose compression matters.

“τ 2 -bench wraps both the agent and the user simulator with independent memory managers, enabling the three-condition ablation that isolates whose compression matters”

Classified as probably not a substantial claim. May combine two claims.

MEMGYM measures the latter and uses memory-isolated paired rollouts to keep the comparison clean.

“MEMGYM measures the latter and uses memory-isolated paired rollouts to keep the comparison clean.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Cells marked ✓ were verified against the recorded run configuration; the only contaminated cell is MEMGYM-DR × Structured (gpt-4o-mini summarizer used in error before Bedrock Haiku 4.5 became the documented default), reported in Table 8 for completeness and excluded from the main-text figure.

“Cells marked ✓ were verified against the recorded run configuration; the only contaminated cell is MEMGYM-DR × Structured (gpt-4o-mini summarizer used in error before Bedrock Haiku 4.5 became the documented default), reported in Table 8 for completeness and excluded from the main-text figure.”

• AssumptionRests on an assumption: The contaminated cell exclusion does not bias the main results.

Classified as probably not a substantial claim. May combine two claims.

Full evaluations require Dockerized repositories, test execution, and many interaction steps.

“full evaluations require Dockerized repositories, test execution, and many interaction steps”

Probably not a substantial claim: may not pass S3 (load-bearing).

WebArena-Infinity uses a 50% prefix-injection ablation in which a randomly selected prefix of the trajectory is replayed verbatim into the agent before measurement begins.

“WebArena-Infinity uses a 50% prefix-injection ablation in which a randomly selected prefix of the trajectory is replayed verbatim into the agent before measurement begins.”

Probably not a substantial claim: may not pass S3 (load-bearing).

MEMRM is a reward model specialized to memory.

“MEMRM is such a reward model specialized to memory”

Probably not a substantial claim: may not pass S3 (load-bearing).

WebArena-Infinity scales this idea by automatically generating self-contained web applications with verifiable tasks.

“WebArena-Infinity [59] scales this idea by automatically generating self-contained web applications with verifiable tasks.”

• cites [59] [59] Shuyan Zhou. WebArena-Infinity: Generating browser environments with verifiable tasks at scale. shuyanzhou.com , March 2026. URL https://webarena.dev/webarena-infinity/ . -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

All three compression-ratio estimators are conditional on K_e ≥ 1 so episodes that never triggered compaction do not pull the mean toward 1.0.

“All three estimators are conditional on K e ≥ 1 so episodes that never triggered compaction do not pull the mean toward 1 . 0 .”

• AssumptionRests on an assumption: The compression ratio formulas capture memory compression meaningfully.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The two τ2-bench baseline rows (50.0 vs. 57.6 in Table 2) are paired against independently sampled rollouts, so the within-row ∆ is the paired statistic.

“The two τ 2 -bench baseline rows ( 50 . 0 vs. 57 . 6 in Table 2) are paired against independently sampled rollouts, so the within-row ∆ is the paired statistic.”

• AssumptionRests on an assumption: The two τ2-bench baseline rows are comparable despite being independently sampled.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The current web results use two protocols: smart replay on 170 hard tasks across five apps, and prefix injection on 140 hard tasks across three apps.

“The current web results use two protocols: smart replay on 170 hard tasks across five apps, and prefix injection on 140 hard tasks across three apps.”

Classified as probably not a substantial claim. May combine two claims.

Smart replay compares memory strategies after replaying recorded baseline actions; prefix injection fixes the early trajectory prefix and measures whether memory improves the live tail.

“Smart replay compares memory strategies after replaying recorded baseline actions; prefix injection fixes the early trajectory prefix and measures whether memory improves the live tail.”

Classified as probably not a substantial claim.

These benchmarks measure what can be recalled from a static long context.

“These benchmarks measure what can be recalled from a static long context.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All conditional ∆ statistics (R→R, R→U, U→R, U→U transition counts; per-split breakdowns) are reported on the paired subset only.

“All conditional ∆ statistics (R → R, R → U, U → R, U → U transition counts; per-split breakdowns) are reported on the paired subset only.”

May combine two claims.

Across these benchmarks, memory is loadbearing but not separately measured.

“Across these benchmarks, memory is loadbearing but not separately measured.”

Classified as probably not a substantial claim.

The fork-batch protocol only forks instances whose baseline trajectory crosses the compaction-trigger threshold (more than 100 messages).

“The fork-batch protocol only forks instances whose baseline trajectory crosses the compaction-trigger threshold (more than 100 messages).”

Probably not a substantial claim: may not pass S2 (checkable).

Agentic memory must additionally decide what to preserve across an interaction in which the context is generated turn-by-turn under tool-use pressure.

“Agentic memory must additionally decide what to preserve across an interaction in which the context is generated turn-by-turn under tooluse pressure”

Probably not a substantial claim: may not pass S3 (load-bearing).

Evaluation uses the official swebench harness, identical to the public leaderboard.

“Evaluation uses the official swebench harness, identical to the public leaderboard.”

• AssumptionRests on an assumption: The official swebench harness is a valid evaluation for memory effects.

Probably not a substantial claim: may not pass S4 (based).

Coding trajectories expose natural memory events: repeated evidence, stale observations, failed hypotheses, and patch-specific facts that survive multiple debugging steps.

“We use SWE-Gym heavily because coding trajectories expose natural memory events: repeated evidence, stale observations, failed hypotheses, and patch-specific facts that survive multiple debugging steps.”

• AssumptionRests on an assumption: Coding trajectories expose natural memory events that are representative of memory challenges.

Classified as probably not a substantial claim. May combine two claims.

Instances whose baseline trajectory never reaches the compaction trigger (≤100 messages) produce identical baseline and +memory rollouts and are therefore excluded from the paired comparison.

“Instances whose baseline trajectory never reaches the compaction trigger ( ≤ 100 messages) produce identical baseline and + memory rollouts and are therefore excluded from the paired comparison”

Probably not a substantial claim: may not pass S3 (load-bearing).

MEMRM scores compression decisions taken during multi-step interaction.

“in contrast to MEMRM, which scores compression decisions taken during multi-step interaction.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Results are paired baseline-vs-memory runs in mode=both.

“Results are paired baseline-vs-memory runs in mode=both”

Probably not a substantial claim: may not pass S3 (load-bearing).

Reward models trained from human or automated feedback have become a central tool for posttraining LLMs.

“Reward models trained from human or automated feedback have become a central tool for posttraining LLMs [30]”

• cites [30] [30] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022. - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

MEMRM predicts only whether a candidate compression would change behavior given the recorded action.

“it predicts only whether a candidate compression would change behavior given the recorded action.”

• AssumptionRests on an assumption: MEMRM's prediction of whether a candidate compression would change behavior is a valid proxy for memory utility.

May combine two claims.

MEMGYM wraps such environments with an explicit memory boundary, records compression events, and reports memory-isolated scores.

“MEMGYM wraps such environments with an explicit memory boundary, records compression events, and reports memory-isolated scores”

May combine two claims.

End-task resolve rate alone is too expensive and too sparse for systematic memory iteration.

“end-task resolve rate alone is too expensive and too sparse for systematic memory iteration.”

Classified as probably not a substantial claim. May combine two claims.

These web benchmarks require browser/server infrastructure and multi-step exploration.

“These web benchmarks are realistic but require browser/server infrastructure and multi-step exploration”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Across these benchmarks, the memory target is a transcript, a feedback stream, or a personalized-dialogue state.

“Across these benchmarks, the memory target is a transcript, a feedback stream, or a personalized-dialogue state”

Classified as probably not a substantial claim.

Conditions (b)-(d) jointly operationalize the 'not RAG' claim of Section 3.4: an instance survives only if memory genuinely matters above and beyond what dropping all documents into context would achieve.

“Conditions (b)-(d) jointly operationalize the 'not RAG' claim of Section 3.4: an instance survives only if memory genuinely matters above and beyond what dropping all documents into context would achieve.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

MEMGYM explores a new, practical direction for memory evaluation (memory-isolated paired scoring across coding, retrieval, dialogue, and web tracks, with a learned gate (MEMRM) replacing per-event Docker rollouts) and provides a unified playground for studying agentic memory as a foundational capability rather than as a benchmark-specific add-on.

“MEMGYM explores a new, practical direction for memory evaluation (memory-isolated paired scoring across coding, retrieval, dialogue, and web tracks, with a learned gate (MEMRM) replacing per-event Docker rollouts) and provides a unified playground for studying agentic memory as a foundational capability rather than as a benchmark-specific add-on.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The structured row in Table 8 must not be cited as a strategy result.

“it must not be cited as a strategy result.”

Probably not a substantial claim: may not pass S4 (based).

The six perturbation families together cover the dominant failure modes a deployed condenser would induce: information loss, plausible-but-incorrect content, and loss of recent context.

“Together they cover the dominant failure modes a deployed condenser would induce: information loss, plausible-but-incorrect content, and loss of recent context.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This provides a controlled view of the cost side of memory.

“providing a controlled view of the cost side of memory.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This is consistent with kept-floor logic limiting how much can be pruned.

“consistent with kept-floor logic limiting how much can be pruned.”

May combine two claims.

Paypal tasks are short and rarely trigger useful condensation.

“paypal tasks are short and rarely trigger useful condensation”

May combine two claims.

MEMRM is trained on SWE-Gym compression events; out-of-distribution (OOD) behavior is not implied by the IID metrics of Table 3.

“MEMRM is trained on SWE-Gym compression events; out-of-distribution (OOD) behavior is not implied by the IID metrics of Table 3.”

Classified as probably not a substantial claim. May combine two claims.

Distractors are drawn from four sources, each targeting a different shortcut: cross-instance bug reports block topical inference; same-repo docstrings block conventions-based inference; adversarial near-misses block surface-pattern matching; and samefunction contradictions block confidence-via-repetition.

“Distractors are drawn from four sources, each targeting a different shortcut: cross-instance bug reports block topical inference; same-repo docstrings block conventions-based inference; adversarial near-misses block surface-pattern matching; and samefunction contradictions block confidence-via-repetition.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We do not claim deployment outside the covered subset.

“We do not claim deployment outside the covered subset”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Negative impacts are inherited from the underlying agent benchmarks rather than introduced by memory itself: webautomation benchmarks could be repurposed for spam or scraping, and coding agents could be repurposed to generate exploits.

“Negative impacts are inherited from the underlying agent benchmarks rather than introduced by memory itself: webautomation benchmarks could be repurposed for spam or scraping, and coding agents could be repurposed to generate exploits.”

Classified as probably not a substantial claim.

This change preserves apples-to-apples cross-method comparison at the storage layer.

“This change preserves apples-to-apples cross-method comparison at the storage layer.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The decision threshold t*=0.88 was selected on held-out validation under the constraints HARMFUL-F1 ≥ 0.90 and SAFE-precision ≥ 0.80.

“Decision threshold: t ∗ =0 . 88 , selected on held-out validation under the constraints HARMFUL-F1 ≥ 0 . 90 and SAFE-precision ≥ 0 . 80 .”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The mock domain is an LLM-simulated user (rather than the canonical scripted user) and has historically been the highest-variance slice of τ2.

“The mock domain is an LLM-simulated user (rather than the canonical scripted user) and has historically been the highest-variance slice of τ 2”

• AssumptionRests on an assumption: The mock domain's high variance does not invalidate comparisons.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The released pairedtrajectory corpus and pipelines further reduce duplicated infrastructure work.

“the released pairedtrajectory corpus and pipelines further reduce duplicated infrastructure work.”

May combine two claims.

The main limitation of this work is data scale: per-track pilots and the MEMRM training set are sized to support the headline comparisons reported in §4 but are not yet large enough to characterize every cell of the strategy × reasoner × track grid at full statistical power.

“The main limitation of this work is data scale: per-track pilots and the MEMRM training set are sized to support the headline comparisons reported in §4 but are not yet large enough to characterize every cell of the strategy × reasoner × track grid at full statistical power.”

Classified as probably not a substantial claim.

The per-domain ∆ in each row uses that strategy's own independently sampled baseline rollouts.

“The per-domain ∆ in each row uses that strategy's own independently sampled baseline rollouts”

Probably not a substantial claim: may not pass S3 (load-bearing).

Gitlab is near the capability floor for Haiku 4.5.

“gitlab is near the capability floor for Haiku 4.5”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The trajectory fields are sufficient to reconstruct the context seen by the reasoning model, identify rising-edge compression events, and build supervised examples for MEMRM or policy mid-training.

“These fields are sufficient to reconstruct the context seen by the reasoning model, identify rising-edge compression events, and build supervised examples for MEMRM or policy mid-training.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Three additional ported operations (observation masking, sliding-window with pinned anchors, adaptive token budget) are implemented but not used as primary baselines in this paper.

“Three additional ported operations (observation masking, sliding-window with pinned anchors, adaptive token budget) are implemented but not used as primary baselines in this paper.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

MEMRM-gated scoring cuts per-event memory evaluation from minutes of Docker rollout to a sub-second scalar, lowering the compute floor for academic memory research.

“MEMRM-gated scoring cuts per-event memory evaluation from minutes of Docker rollout to a sub-second scalar, lowering the compute floor for academic memory research”

Classified as probably not a substantial claim.

MEMGYM is a benchmark for LLM-agent memory rather than a deployed system.

“MEMGYM is a benchmark for LLM-agent memory rather than a deployed system.”

Classified as probably not a substantial claim. May combine two claims.

Difficulty is a composable post-hoc dial (prompt fuzzing, distractor scaling, indirection, fact fragmentation) with preset combinations mapping to easy/medium/hard, so length and noise are decoupled from the underlying instance.

“Difficulty is then a composable post-hoc dial (prompt fuzzing, distractor scaling, indirection, fact fragmentation) with preset combinations mapping to easy/medium/hard, so length and noise are decoupled from the underlying instance.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The wrapper choice keeps the gym implementation unchanged, makes memory strategies portable across environments, and lets the recorder log both the raw history statistics and the filtered view.

“This choice keeps the gym implementation unchanged, makes memory strategies portable across environments, and lets the recorder log both the raw history statistics and the filtered view.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The structured row in Table 8 was contaminated by an upstream summarizer-model misconfiguration (gpt-4o-mini was used in error before Bedrock Haiku 4.5 became the documented default).

“The structured row was contaminated by an upstream summarizer-model misconfiguration (gpt-4o-mini was used in error before Bedrock Haiku 4.5 became the documented default)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Observation masking is a zero-LLM-cost strategy that reduces token count without rewriting content.

“This is a zero-LLM-cost strategy that reduces token count without rewriting content.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Every other (track, strategy) cell in Section C.2 was verified against its recorded summarizer-model and reasoner-model configuration; this is the only cell affected.

“Every other (track, strategy) cell in Section C.2 was verified against its recorded summarizer-model and reasoner-model configuration; this is the only cell affected.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The replay-and-fork protocol is exposed in the released trajectories so users can separate effects of memory compression from effects of different early search paths.

“This protocol is exposed in the released trajectories so users can separate effects of memory compression from effects of different early search paths.”