Reflexion: Language Agents with Verbal Reinforcement Learning
Open accessThe DOI matches its DataCite record; 42 of the 42 references it cites were found in a registry or an index; 7 of the 8 sources its claims cite were checked for support.
Abstract
Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.
Overview
CoChat Summary
TL;DR
Reflexion reinforces language agents through verbal self-reflection stored in episodic memory rather than weight updates, improving performance across decision-making, reasoning, and coding tasks. It reports gains including 91% pass@1 on HumanEval, a 22% absolute improvement on AlfWorld, and 20% on HotPotQA.
Key findings
- On HumanEval Python, Reflexion achieves 91.0 pass@1, surpassing the previous state-of-the-art GPT-4 at 80.1; on HumanEval Rust it reaches 68.0 vs. 60.0, MBPP Rust 75.4 vs. 70.9, and Leetcode Hard 15.0 vs. 7.5.
- In AlfWorld, ReAct + Reflexion completes 130 out of 134 tasks using a simple heuristic and improves over strong baselines by an absolute 22% over 12 iterative learning steps; ReAct-only converges at a 22% hallucination rate with no long-term recovery.
- In HotPotQA, Reflexion improves over baselines by 20%; self-reflection gives an 8% absolute boost over episodic memory alone, and although CoT (GT) fails on 39% of questions, Reflexion improves accuracy by 14% without access to ground-truth answers.
- Ablations on the 50 hardest HumanEval Rust problems show full Reflexion at 0.68 vs. baseline 0.60; omitting test generation drops to 0.52, and omitting self-reflection stays at 0.60.
Methods
- Reflexion uses a modular loop with an Actor that generates text/actions, an Evaluator that scores outputs, and a Self-Reflection model that converts feedback into verbal summaries stored in memory; trials repeat until the Evaluator passes or a max trial count is reached, with memory usually bounded to 1–3 experiences.
- Experiments cover AlfWorld (134 environments, six tasks, ReAct + GPT-3), HotPotQA (100 questions, CoT/ReAct, exact-match grading, memory size 3), and programming (HumanEval, MBPP, LeetcodeHardGym; Python/Rust via MultiPL-E; self-generated unit tests up to 6).
- Ablations and analyses vary feedback signals, feedback incorporation methods, and agent types; compromised approaches are tested on HumanEval Rust, and a WebShop experiment is run across 100 environments.
Limitations
- The paper states Reflexion may fall into non-optimal local minima, limits long-term memory to a sliding window with maximum capacity, and notes code-generation test-driven development is limited by non-deterministic, impure, hardware-dependent, or concurrent functions; it also relies on LLM self-evaluation/heuristics and lacks a formal success guarantee.
- The paper reports WebShop failure: after only four trials the agent shows no improvement and does not generate helpful self-reflections, concluding Reflexion cannot solve tasks requiring significant diversity and exploration.
- Evident from the text: self-correction is an emergent quality of stronger, larger models—starchat-beta shows no improvement (baseline 0.26 vs. Reflexion 0.26)—and flaky or incorrect test suites can cause false positives/negatives, with false negatives preferred.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 2 Not valid · 0 Not checked · 59 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Therefore, in the case in which the model generates a flaky test suite, it is possible that all tests pass on an incorrect solution and lead to a false positive label on a code completion [11].”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Analysis
What the sources say
- Partly supportedcites [11] 11. Lam, W., Winter, S., Wei, A., Xie, T., Marinov, D., and Bell, J. (2020). A large-scale longitudinal study of flaky tests. Proc. ACM Program. Lang., 4(OOPSLA). sourceExperimental — accuracy not yet measured
The Introduction (page not provided) says a flaky test can pass and fail on the same code and that changes may be merged when all tests pass. It does not address model-generated test suites, incorrect code completions, or false-positive labels, so those parts of the claim are not established by this paper. (LLM, conf=0.82)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“3 shows an inferior 52% vs 60% (baseline) accuracy, which suggests that the agent is unable to determine if the current implementation is correct without unit tests.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Ablation study
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ReAct + Reflexion significantly outperforms ReAct by completing 130 out of 134 tasks using the simple heuristic to detect hallucinations and inefficient planning.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“4 shows that self-reflection improves learning by an 8% absolute boost over the episodic memory learning advantage.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Analysis
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“on reasoning questions in HotPotQA [28] by 20%”
- Also stated in: § 4 Experiments · plain
“Most notably, Reflexion improves performance over strong baselines by 22% in AlfWorld, 20% in HotPotQA, and 11% on HumanEval.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > 1 Introduction
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4 Experiments · plain
30 more claims
Create a free account to see all 65 claims and every source verdict.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 11
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Memory and reflection systems such as MemAgent (Yu et al., 2025a), Reflexion (Shinn et al., 2023), and Generative Agents (Park et al., 2023) improve retrieval and episodic memory, but do not perform context shaping, state restructuring, or nextinstruction refinement.”
From PAACE: A Plan-Aware Automated Agent Context Engineering Framework (2025) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“However, Reflexion computes verbal reward over a full trajectory, yielding a single end-of-episode signal that conflates credit across reasoning steps: a failure due to a single intermediate error receives the same critique pressure as one due to a flawed strategy.”
From Process Supervision via Verbal Critique Improves Reasoning in Large Language Models (2026) · cites this paper as Shinn et al. [2023]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Unlike Reflexion, which only reflects on unsuccessful trajectories, our approach separately reflects on both successful and unsuccessful ones.”
From An Efficient Context-Dependent Memory Framework for LLM-Centric Agents (2025) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“The dominant approach is to curate memory at write time.”
From Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM Agents (2026) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interaction and code generation are increasingly used together, with code as the modality of choice for actions [48, 59], tool construction [13, 51, 69], and reasoning [39, 66, 67].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [39]
Experimental — accuracy not yet measured
107 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Reflexion: Language Agents with Verbal Reinforcement Learning > 3 Reflexion: reinforcement via verbal reflection
“while M e not pass or t < max trials do”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Self-reflection
“The Actor, Evaluator, and Self-Reflection models work together through trials in a loop until the Evaluator deems τ t to be correct.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Items 2,601–2,700 of 3,537
AutoKB: Automated Creation of Structured Knowledge Bases for Domain-Specific Support
Sahay, R., Jain, A., Aggarwal, P., & Saladi, A. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-industry.58 · Cited by 3
“However, techniques like Chain-of-Thought (Wei et al., 2023) and Reflexion (Shinn et al., 2023) can enhance rule adherence but they may exacerbate input length constraints and increase latency, thereby complicating their practical application.”
Creative-Agent: A Creative Prototype Generation System Driven by Objectives and Key Results
Zheng, Y., Huang, H., Ma, C., Shi, K., Chen, J., Yan, H., & Huang, X. (2025) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd64889.2025.11033384
Exploring humans and LLMs collaboration in the data science pipeline
Nia, S. N., & Roy, S. B. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133719
PromptXpert: The AI for Better AnswersPreprint
de Paula, A. (2025) · doi:10.22541/au.175192065.56937600/v1
Wen, T., Chen, X., He, B., & Sun, L. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.912
“…outputs (Chen et al., 2023; Hu et al., 2024), integrating with external debugging tools (Zhong et al., 2024), and decomposing complex debugging problems into smaller, more manageable steps (Zhang et al., 2023; Madaan et al., 2023; Shinn et al., 2023; Zhou et al., 2023; Wang et al., 2024a).”
Ordinal Folding Index: A Computable Metric for Self-Referential SemanticsPreprint
Alpay, F., & Alakkad, H. (2025) · doi:10.22541/au.175346092.28261233/v1
BOPRO: Towards New Style Bayesian Optimization with Large Language Models
Feng, Q., Huang, Z., Wei, Y., & Xing, W. W. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-0017-8_28 · Cited by 1
Innovative Image Fraud Detection with Cross-Sample Anomaly Analysis: The Power of LLMs
Wang, Q., Yang, J., Lin, Z., Ying, Z., Wang, W., & Chen, L. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.687 · Cited by 1
CoMet: Metaphor-Driven Covert Communication for Multi-Agent Language Games
Xu, S., & Zhong, F. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.389 · Cited by 11
Getting LLM to think and act like a human being: logical path reasoning and replanning
Zhang, L., Li, Q., Wang, Y., & Zhao, J. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00054 · Cited by 2
“Voyager [19] and Reflexion [20] are two methods that can partially address these shortcomings.”
ComplexAgents: Complex Code Generation Framework Based on Multi-agents and Large Language Model
Gao, S., Cai, Z., & Wang, Y. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-93254-0_11
Vallverdú, J. (2025) · SpringerBriefs in intelligent systems · doi:10.1007/978-3-032-03593-6_9
CLIMB: Language-Guided Continual Learning for Task Planning with Iterative Model Building
Byrnes, W., Bogdanović, M., Balakirsky, A., Balakirsky, S. B., & Garg, A. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11127439 · Cited by 5
“SayCan [24] and Reflexion [25] propose action embedding and verbal RL to surprising success on text-based tasks.”
Sensemaking in Novel Environments: How Human Cognition Can Inform Artificial AgentsPreprint
Patterson, R., Buccello-Stout, R. R., Frame, M. E., Maresca, A. M., Nelson, J., Acker-Mills, B. E., Curtis, E., Culbertson, J. L., Schmidt, K., Clouse, H. S., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.07783
Sorstkins, A., Bailey, J. W., & Baron, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.15366
WiP: From Wasted Compute to Quality Gains: LLM Test-Time Scaling on Mobile NPUs
Hao, Z., Ren, J., & Cao, T. (2025) · doi:10.1145/3737902.3768361
PokéLLMon: A Grounding and Reasoning Benchmark for Large Language Models in Pokémon Battles
Hu, S., Huang, T., Liu, G., Kompella, R. R., & Liu, L. (2026) · ACM Transactions on Internet Technology · doi:10.1145/3771095 · Cited by 1
Natural Language Edge Labelling: Decoupling Intent from Execution in Structured LM ReasoningPreprint
Madahar, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.04817
“…Reflexion proposes an LLM agent that improves itself through textual self-feedback: after each trial or attempted solution, the agent generates a natural-language reflection on what went wrong or could be improved, stores this in memory, and uses it to inform the next attempt (Shinn et al., 2023).”
LLM Agents as Programmable Subjects: Assays and Benchmarks for Agentic Behavior and AlignmentPreprint
Koley, G., & Thiruvengadam, A. (2025) · Preprints.org · doi:10.20944/preprints202510.0476.v1 · Cited by 2
Wu, Y., Dong, Q., Liu, J., & Ruan, T. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108203 · Cited by 1
Reflective Thought and Code for Solving Numerical Problems with Large Language Models
Son, L. T. T., Anh, N. V., Linh, N. T. T., & Hà, N. V. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-3358-9_31
Refining Financial Consumer Complaints through Multi-Scale Model InteractionPreprint
Chen, B., Yen, A.-Z., & Chen, C.-C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.09903
“Shinn et al. (2023) takes a step further, leveraging LLMs to convert feedback from the task environment, usually in binary or scalar form, to a verbal text summary, which is then provided as additional context for generation.”
The Reflexive Integrated Information Unit: A Differentiable Primitive for Artificial ConsciousnessPreprint
N'guessan, G. L. R., & Karambal, I. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.13825
Agente Autônomo Guiado por LLM para Extração de Notícias
de Sousa, J. V. C. N., Mingardo, L. M., Freire, C. E. T., Traina, A. J. M., & Traina, C. (2025) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2025.247075
Chen, H., Lin, X., Wang, X., & Tang, Q. (2025) · Abu Dhabi International Petroleum Exhibition & Conference (ADIPEC) · doi:10.2118/229358-ms
Talk2X -- An Open-Source Toolkit Facilitating Deployment of LLM-Powered Chatbots on the WebPreprint
Krupp, L., Geißler, D., Hevesi, P., Hirsch, M., Lukowicz, P., & Karolus, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.03343
“Others use linguistic feedback Shinn et al. [2023], chain-of-abstraction reasoning Gao et al. [2024] or even have multiple agents cooperate to solve a problem Shi et al. [2024].”
AoPSFT: Subplan-Aware Fine-Tuning for Interpretable Multi-turn Reasoning Agents
Zhou, X., Geng, J., Cai, C., Li, L., & Zeng, D. D. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4088-4_12
DynaQuery: A Self-Adapting Framework for Querying Structured and Multimodal DataPreprint
Hassini, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.18029
Unter der Oberfläche: Technische Grundlagen Generativer KI
Gründel, D., & Lämmel, M. (2025) · Springer eBooks · doi:10.1007/978-3-658-47288-7_3
Beyond Elicitation: Provision-Based Prompt Optimization for Knowledge-Intensive Tasks
Xu, Y., Zhang, Z., & Liu, Z. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3682078 · Cited by 2
“The optimization process is detailed in Algorithm 1.”
PromptPilot: Autonomous Prompt Optimization via Genetic Particle Filtering and Dynamic Exploration
Wang, J., & Wang, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3058-8_22
NexaFusion: Integrating Multi-team Collaboration for High-Impact Outcomes
Shen, L., Yu, M., Fan, Y., Ma, J., Wang, H., & Wang, H. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3058-8_33
MFTP: Multi-round Feedback for Dynamic Travel Itinerary Optimization
Lu, Y., Jin, P., Zhang, Y., Xu, L., Wang, B., & Chen, X. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3052-6_8
Amazon Kiro: Transforming Software Development Through AI-Powered Autonomous Coding
Sharma, R. K. (2025) · International Journal of Science and Research (IJSR) · doi:10.21275/sr251115051642
Chain-of-Conceptual-Thought Elicits Daily Conversation in Large Language Models
Gu, Q., Wang, D., Zhao, Y., Wang, X., Jiang, Z., Chen, Y., & Ji, L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7078-2_23
Bhavsar, V., Ereifej, J., & Gurusami, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.02326
“Reflexion [15] adds verbal reinforcement learning, enabling agents to self-criticize and adjust their trajectory.”
Jeong, C. (2025) · Research Square · doi:10.21203/rs.3.rs-8139402/v1
Adaptive Tool Generation with Models as Tools and Reinforcement LearningPreprint
Wang, C., Cheng, X., Wang, C., Yang, L., & Zhang, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.06825
“Chain-of-thought and self-consistency approaches leverage step-by-step reasoning [Wei et al., 2022, Wang et al., 2022], while methods like Reflexion and Tree-of-Thoughts incorporate self-correction and search [Shinn et al., 2023, Yao et al., 2023a].”
CompassLLM: A Multi-Agent Approach toward Geo-Spatial Reasoning for Popular Path Query
Ananto, M. N. I., Fatin, S., Ali, M. E., & Parvez, M. R. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1599
Causal What-Ifs: Rethinking Counterfactuals with LLM Agents
Narendra, Y., & Chatterjee, N. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4367-0_13 · Cited by 3
Two Steps to Precision: Enhancing Reliable API Invocation in Code Generation
Li, Y., Tang, Z., Wang, S., & Qi, G. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3346-6_33
Exploring Agentic Multimodal Large Language Models: A Survey for AIScientistsPreprint
Jian, J., Fung, Y. R., Zhang, D., Liang, Y., Chen, Q., Zhi-yong, L., & Wang, Q. (2025) · doi:10.36227/techrxiv.176344216.60619335/v1
ПРИОРИТЕТНЫЕ НАПРАВЛЕНИЯ ИССЛЕДОВАНИЙ И КЛЮЧЕВЫЕ ТЕНДЕНЦИИ РАЗВИТИЯ ТЕХНОЛОГИЙ ИИ
Visilter, Y. V. (2025) · Высшая школа: научные исследования · doi:10.15622/rcai.2025.001
LLM-Assisted Iterative Evolution with Swarm Intelligence Toward SuperBrainPreprint
Weigang, L., Brom, P. C., & Siefert, L. R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.00510
“Similar iterative refinement paradigms have been explored in Ouyang et al. [2022], Christiano et al. [2017], Shinn et al. [2023] and Madaan et al. [2023], while the swarm-level integration resonates with Burton et al. [2024] and knowledge distillation methods [Hinton et al., 2015; Moslemi et al.,…”
Error Notebook-Guided, Training-Free Part Retrieval in 3D CAD Assemblies via Vision-Language Models
Liu, Y., Zhang, N., & Tan, Z. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2509.01350
“One line of work leverages external critics or verifier models to provide feedback on intermediate reasoning steps, guiding the model away from incorrect trajectories (An et al. (2023); Li et al. (2023); Shinn et al. (2023)).”
Optimizing LLM Chatbot to Human Handover with Multi-Agent AI Approach
Amri, M. F., Fajar, S. P., Ramadhan, M., Ishlah, M. S. N., Fikri, R., & Widiatmansyah, G. (2026) · International Conference on Advancement in Data Science, E-learning and Information System (ICADEIS) · doi:10.1109/icadeis71120.2026.11644567
“Separately, the Reflexion framework [13] enables agents to learn from trial-and-error by generating self-reflections on past failures.”
CodeQUEST: Evaluation and Improvement of Code Quality Using LLMs
Lius, R., Frade, A., Vaidya, A., Labonne, M., Kaiser, M., Chakrabarti, B., Budd, J., Buford, J., Lin, X., & Moran, S. (2025) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew67781.2025.00060
“Self-reflection and correction [42] based on compiler or execution outputs were left as a direction for future work.”
Alshaikh, M., Amin, R., Mehmood, S., & Alsubaei, F. S. (2025) · Contemporary Mathematics · doi:10.37256/cm.7120268211
“The self-evaluation framework of RVP uses a step-by-step procedure adopted from Shinn et al. [29] to guide the LLM through validation processes.”
Generating Verifiable Chain of Thoughts from Exection-TracesPreprint
Thakur, S., Saxena, V., Kulkarni, R., Singh, S., Selvam, P., Patel, H., & Kanayama, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.00127
Exploring Agentic Multimodal Large Language Models: A Survey for AIScientistsPreprint
Jian, J., Fung, Y. R., Zhang, D., Liang, Y., Chen, Q., Zhi-yong, L., & Wang, Q. (2025) · doi:10.36227/techrxiv.176344216.60619335/v2
RoCo: Role-Based LLMs Collaboration for Automatic Heuristic Design
Xu, J.-W., Wei, F.-F., & Chen, W. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3730656
“ReEvo [37] enhances heuristic quality through reflection-based reasoning [28], while HSEvo [7] introduces diversity metrics and harmony search to improve population diversity.”
Li, J., Wang, W., Mao, H., Ni, J., & Xiong, Z. (2025) · International Conference on Human-Engaged Computing (ICHEC) · doi:10.1145/3786995.3787064
Orion: Fuzzing Workflow AutomationPreprint
Bazalii, M., & Fleischer, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.15195 · Cited by 1
Formal Knowledge Augmented Language Models for Explainable and Robust Reasoning
Nguyen, N.-K., Nguyen, V.-H., & Le, A.-C. (2025) · Journal of Intelligent & Fuzzy Systems · doi:10.1177/18758967251394597 · Cited by 1
“• Reflection: LLMs are prompted to review and critique their own generated CoT and final answer, identifying potential errors and iteratively refining their reasoning (Shinn et al., 2023).”
Dong, S. C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.09939
“…of treaty text and contractual logic; • Long-horizon deliberation via reasoning–acting loops [15]; • Self-critique and metacognition via Reflexion [16]; • Typed, structured communication via AutoGen-style protocols [17]; • Tool-mediated inference through catastrophe, capital, and portfolio…”
InfluentialCreating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs
Tzachristas, I., & Τζαχρήστας, Ι. (2025) · DSpace - NTUA (National Technical University of Athens) · doi:10.26240/heal.ntua.28317
Large Language Models for Collective Problem-Solving: Insights into Group Consensus Decision-Making
Cognitive Science Society 2024, Du, Y., Gonzalez, C., & Rajivan, P. (2024) · eScholarship (California Digital Library) · doi:10.48448/mstq-xg46 · Cited by 7
Collaborative AI Agents in the Era of Large Language Models
Josifoski, M. (2025) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.5075/epfl-thesis-10872
Alpay, F., Alakkad, H., Kılıçtaş, B., & Alpay, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.19694
“The number of iterations until stabilisation yields an ordinal-valued metric correlating with model perplexity and chain–of– thought complexity [Shinn et al.(2023), Wang et al.(2022)].”
LLM-agenter som moderna kognitiva arkitekturer
Janson, N. (2025) · Aaltodoc (Aalto University)
Full Stack Approach for Efficient Deep Learning Inference
Kim, S. (2024) · eScholarship (California Digital Library) · Cited by 1
Teaching Large Language Models to Use Tools at Scale
Patil, S. G. (2024) · eScholarship (California Digital Library) · Cited by 1
최지웅 (2023) · Seoul National University Open Repository (Seoul National University)
Improving embodied LLM agents' capabilities through collaboration
Colle, B. (2024) · Research Repository (Delft University of Technology) · Cited by 2
Self-Reflection on Chain-of-Thought Reasoning in Large Language Models
Praas, R. (2023) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 6
Mitigating Generative Agent Social Dilemmas
Yocum, J. R. (2024) · DSpace@MIT (Massachusetts Institute of Technology)
Multi-step Automated Domain Modeling with Large Language Models
Yang, Y. (2024) · eScholarship@McGill (McGill) · doi:10.82308/51342
Aligning LLM Agents to Environment Dynamics
Nottingham, K. (2025) · eScholarship (California Digital Library)
Mendoza Juan, Y. (2024) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · Cited by 7
A Study on Deploying Large Language Models as Agents
Cao, J. (2024) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2
Fernández, E. (2024) · Archivo Digital UPM (Universidad Politécnica de Madrid) · Cited by 4
Från ord till SQL : Optimering av text-till-SQL-generering i stora språkmodeller
Wretblad, N., & Gordh Riseby, F. (2024) · DiVA (Linkoping University)
Leveraging large language models for automated chart summarization
Bazin, R. (2025) · eScholarship@McGill (McGill) · doi:10.82308/30829
Research of approaches to verification and correction of response content of a Large Language Model
Шатилов, М. Р. (2024) · Electronic scientific archive of UrFU (Ural Federal University)
Towards Enhanced Reasoning in Large Language Models
Ling, Z. (2024) · eScholarship (California Digital Library)
The next question after Turing's question: Introducing the Grow-AI testPreprint
Țugui, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.16277
On Modularity: From Neural Circuits to Foundation Models and Agentic Systems
Weiß, M. (2025) · PolyPublie (École Polytechnique de Montréal)
Building a Smart Travel Planner Using Google's Agent Development Kit (ADK)
Vashishtha, P. (2025) · International Journal of Science and Research (IJSR) · doi:10.21275/sr25915100037
A Bilevel Reinforcement Learning Framework with Language Prior Knowledge
Yan, X., Song, Y., Cui, X., Christianos, F., Zhang, H., Wang, J., & Mguni, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06106-5_13
The Gatekeeper Knows EnoughPreprint
Abebayew, F. W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.14881
“Architectures like those in Generative Agents [2] and Reflexion [7] introduced sophisticated memory streams and self-reflection abilities, allowing agents to learn from experience.”
Enhancing LLM Agent Effectiveness via Reflective Multi-Agent System
Mohamed, A. H., Santos, F. A., & dos Reis, J. C. (2025) · Anais do Workshop-Escola de Sistemas de Agentes, seus Ambientes e Aplicações (WESAAC) · doi:10.5753/wesaac.2025.37536 · Cited by 2
“Rather than retraining the LLM to acquire new information, the memory module allows the agent to accumulate knowledge dynamically, increasing adaptability while reducing computational costs [Shinn et al. 2023].”
Bao, Z., Zhou, S., Pi, Q., Chen, J., Xu, M., Tong, M., Zhong, M., Zhu, Y., & Qian, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37657-2_8
MAD-HD: Multi-agent Debate-Driven Ungrounded Hallucination Detection
Gao, Z., Bao, F., & Li, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3343-5_39
Hierarchical Deep Research with Local-Web RAG: Toward Automated System-Level Materials DiscoveryPreprint
Ding, R., Ferreira, R. P., Chen, Y., & Chen, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.18303
Machado, J. D. J. M. D. A. (2025) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
Prompt Engineering in Finance: An LLM-Based Multi-Agent Architecture for Decision Support
Kulpa, A., & Wojarnik, G. (2025) · EUROPEAN RESEARCH STUDIES JOURNAL · doi:10.35808/ersj/4220 · Cited by 3
“Critic/reviewer functions are further strengthened by self-reflection and self-refine techniques, in which an agent critiques and iteratively improves its own outputs without additional supervised training (Shinn et al., 2023; Madaan et al., 2023).”
Autonomous Issue Resolver: Towards Zero-Touch Code Maintenance
Kaliutau, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-32729-1_6
Modular AI Framework for Small Software Companies
Niraula, S. (2025) · Theseus (Ammattikorkeakoulujen)
Towards Trustworthy Multi-Turn LLM Agents via Behavioral GuidancePreprint
Gürsun, G. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.11421
“Although recent advances (Yao et al. 2023; Shinn et al. 2023; Schick et al. 2023; Richards 2023; Packer et al. 2024; Wang et al. 2023; Nakajima 2023) in agentic LLMs have improved their task completion abilities through mechanisms such as memory, tool use, and reflection, these mechanisms remain…”
Sathe, O. (2025) · International journal of research and scientific innovation · doi:10.51244/ijrsi.2025.12110199
Emotion-Inspired Learning Signals (EILS): A Homeostatic Framework for Adaptive Autonomous AgentsPreprint
Tiwari, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.22200
“Systems like Voyager [14] use an iterative prompting mechanism to explore open-ended worlds, while Reflexion [15] employs verbal reinforcement learning to self-correct.”
ASTH agent: Aviation software testing hierarchy agent for test case generation
Zhang, W., Yang, X. I. E., LIANG, X., Chu, X., Qiao, M., SHANG, Z., Zijing, L. U. O., Liguo, L. I., & Cheng, Y. (2026) · Chinese Journal of Aeronautics · doi:10.1016/j.cja.2025.104058 · Cited by 1
Meta-reflexion: A computationally efficient and rigorous approach to LLM assessment
Ippili, Y., Revanth, R., Choolakal, H., & Murthy, G. S. (2026) · AIP conference proceedings · doi:10.1063/5.0309754
Probabilistic Guarantees for Reducing Contextual Hallucinations in LLMsPreprint
Rautenberg, N., & Schippkus, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.00641 · Cited by 1
“…refinement and LLM-as-a-judge systems (Cohen et al. 2023; Du et al. 2023; Jain et al. 2023; Liu et al. 2023; Luo et al. 2023; Madaan et al. 2023; Shinn et al. 2023; Wang et al. 2024) combine several model calls to improve reliability, but empirical studies show that judges can themselves be…”
Talk Less, Verify More: Improving LLM Assistants with Semantic Checks and Execution FeedbackPreprint
Sun, Y., Cai, M., & Kok, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.00224
LLMize: A Framework for Large Language Model-Based Numerical OptimizationPreprint
Oktavian, M. R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.00874
“This formulation is consistent with prior work that frames large language models as feedback-driven proposal generators for search and optimization [Yao et al., 2023, Shinn et al., 2023].”
Brilliant, A. M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6044795
EdgeV-SE: Self-Reflective Fine-Tuning Framework for Edge-Deployable Vision-Language Models
Jeon, Y., Lee, S., & Kim, W. (2026) · Applied Sciences · doi:10.3390/app16020818 · Cited by 1
Assessment of the Dyad Agentic Modeling Language in Process Systems EngineeringPreprint
Lima, F. A. R. D., Abdelrehim, A., Bharambe, A., Micluta-Campeanu, M., Gandhi, D., Bhat, V., Singhvi, A., Piibeleht, M., Secchi, A. R., de Souza, M. B., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-8475139/v1
Metadata
- Authors
- Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 8634-8652
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2303.11366
- OpenAlex
- W7133222210
- Cited by
- 1,087
- References
- 0
- Citations by year
- 2026: 760 · 2025: 167 · 2024: 105 · 2023: 55
- FWCI
- 192.15
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- language agents, verbal behavior, language acquisition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.