Experiential Reflective Learning for Self-Improving LLM Agents
Open accessAbstract
Recent advances in large language models (LLMs) have enabled the development of autonomous agents capable of complex reasoning and multi-step problem solving. However, these agents struggle to adapt to specialized environments and do not leverage past interactions, approaching each new task from scratch regardless of their accumulated experience. We introduce Experiential Reflective Learning (ERL), a simple self-improvement framework that enables rapid environment adaptation through experiential learning. ERL reflects on task trajectories and outcomes to generate heuristics, capturing actionable lessons that transfer across tasks. At test time, relevant heuristics are retrieved based on the current task and injected into the agent's context to guide execution. On the Gaia2 benchmark, ERL improves success rate by 7.8% over a ReAct baseline, with large gains in task completion reliability, and outperforms prior experiential learning methods. Through systematic ablations, we find that selective retrieval is essential and that heuristics provide more transferable abstractions than few-shot trajectory prompting. These results demonstrate that reflecting on single-attempt experiences to extract transferable heuristics enables effective agent self-improvement.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Building procedural memory from past experience demands a curated set of training tasks drawn from the target task distribution, often paired with a verifier that scores each attempt.
“This demands a curated set of training tasks drawn from the target task distribution, often paired with a verifier that scores each attempt.”
From DAEDALUS: Bootstrapping Agent Memory from Self-Generated Tasks (2026) · cites this paper as Allard et al., 2026
The Introduction (p. 1) explicitly says ERL extracts heuristics from single attempts “without curated training sets,” contrary to the claim that building this kind of memory demands them. The paper does use training tasks matching the test distribution and automatic verification in its Gaia2 evaluation (Appendix B, p. 7), but presents those as an experimental setup, not a requirement. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredExisting memory methods retain an insight based on the outcome of the trajectory it came from, without confirming that it helps the agent overcome the failure.
“These methods retain an insight based on the outcome of the trajectory it came from, without confirming that it helps the agent overcome the failure; DAEDALUS instead keeps an insight only once it has turned a failure into repeated success.”
From DAEDALUS: Bootstrapping Agent Memory from Self-Generated Tasks (2026) · cites this paper as Allard et al., 2026
The Introduction (p. 1) says ERL stores heuristics drawn from trajectory outcomes after single attempts, without repeated execution; this supports the description of ERL’s retention process. It does not establish that existing memory methods generally retain insights this way or that none confirms an insight helps overcome the original failure. The same section says ExpeL retries tasks until success, though it does not say whether ExpeL validates each insight. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe other five baseline methods build memory from the benchmark training tasks, and therefore have access to information that DAEDALUS does not.
“The other five build memory from the benchmark training tasks, and therefore have access to information that DAEDALUS does not.”
From DAEDALUS: Bootstrapping Agent Memory from Self-Generated Tasks (2026) · cites this paper as Allard et al., 2026
Appendix B (p. 7) says ExpeL collects experiences from training tasks and AutoGuide generates guidelines from them. The paper does not identify five such baseline methods or discuss DAEDALUS, so it cannot establish that all five have access to information DAEDALUS lacks. (LLM, conf=0.82)
Experimental — accuracy not yet measuredERL and ReasoningBank reflect on a single attempt per task, with the environment reward and an LLM judge as the respective feedback signals.
“ERL (Allard et al., 2026) and ReasoningBank (Ouyang et al., 2026) instead reflect on a single attempt per task, with the environment reward and an LLM judge as the respective feedback signals.”
From DAEDALUS: Bootstrapping Agent Memory from Self-Generated Tasks (2026) · cites this paper as Allard et al., 2026
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 16
- Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ 2 -bench: Evaluating conversational agents in a dual-control environment, 2025. URL https://arxiv.org/abs/ 2506.07982 .
- Zhicheng Cai, Xinyuan Guo, Yu Pei, Jiangtao Feng, Jinsong Su, Jiangjie Chen, Ya-Qin Zhang, WeiYing Ma, Mingxuan Wang, and Hao Zhou. Flex: Continuous agent evolution via forward learning from experience, 2025. URL https://arxiv.org/abs/2511.06449 . · arXiv
- Arthur Chen, Zuxin Liu, Jianguo Zhang, Akshara Prabhakar, Zhiwei Liu, Shelby Heinecke, Silvio Savarese, Victor Zhong, and Caiming Xiong. Grounded test-time adaptation for llm agents, 2026. URL https://arxiv.org/abs/2511.04847 . · arXiv
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374 . · arXiv
- Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. Memp: Exploring agent procedural memory, 2026. URL https://arxiv.org/abs/2508.06433 . · arXiv
- Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurenc ¸on, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre M´ enard, Gerard Moreno-Torres Bertran, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Vladislav Vorotilov, Mengjue Wang, Ian Yu, Amine Benhalloum, Gr´ egoire Mialon, and Thomas Scialom. Are: Scaling up agent environments and evaluations, 2025. URL https://arxiv.org/abs/2509.17158 . · arXiv
- Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of context-aware guidelines for large language model agents. Advances in Neural Information Processing Systems , 37: 119919-119948, 2024.
- Dawei Cheng, Qi Zhang, Tao Gui, Shirui Pan, Yan Zhang, Philip Torr, Zhicheng Dou, Ji-Rong Wen, Xuanjing Huang, Yu-Gang Jiang, and Shuicheng Yan. Memory in the age of ai agents, 2026. URL https://arxiv.org/abs/2512.13564 . · arXiv
- Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. Mcp-universe: Benchmarking large language models with real-world model context protocol servers, 2025. URL https: //arxiv.org/abs/2508.14704 .
- Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. Reasoningbank: Scaling agent self-evolving with reasoning memory, 2025. URL https://arxiv.org/abs/2509.25140 . · arXiv
- Vishnu Sarukkai, Zhiqiang Xie, and Kayvon Fatahalian. Self-generated in-context examples improve llm agents for sequential decision-making tasks, 2025. URL https://arxiv.org/ abs/2505.00234 . · arXiv
- Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems , 36:8634-8652, 2023.
- Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URL https://arxiv.org/abs/2504. 12516 .
- Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id= roNSXZpUDN .
- Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URL https: //arxiv.org/abs/2506.05176 .
- Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pp. 19632-19642, 2024.
Cited by
Sorted by citations
From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms
Luo, J., Tian, Y., Cao, C., Luo, Z., Lin, H., Li, K., Kong, C., Yang, R., & Ma, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.2069 · Cited by 3 · see citing works
“…as errors and hallucinations that manifest across modalities, and will further explore, within the stage of Experience, the extraction of policy priors that are invariant to modality from clusters of trajectories of a multimodal nature (Wang et al., 2025e; Allard et al., 2026; Lei et al., 2025).”
DAEDALUS: Bootstrapping Agent Memory from Self-Generated Tasks
Edy, A., Conti, M., Xing, V., Allard, M.-A., Benhamdane, N., & Viaud, G. (2026) · arXiv · doi:10.48550/arxiv.2610.08048
“In ERL, we use both successful and failed attempts to create heuristics.”
InfluentialSelf-citation
- Authors
- Allard, M.-A., Teinturier, A., Xing, V., & Viaud, G.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2603.24639
- OpenAlex
- W7140993884
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- LLM agents, autonomous agents, human development, environmental adaptation, heuristic generation, selective retrieval, cross-task transfer, few-shot prompting, React
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
