Reflexion: Language Agents with Verbal Reinforcement Learning
Open accessThe DOI matches its DataCite record; 42 of the 42 references it cites were found in a registry or an index; 7 of the 8 sources its claims cite were checked for support.
Abstract
Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.
Overview
CoChat Summary
TL;DR
Reflexion reinforces language agents through verbal self-reflection stored in episodic memory rather than weight updates, improving performance across decision-making, reasoning, and coding tasks. It reports gains including 91% pass@1 on HumanEval, a 22% absolute improvement on AlfWorld, and 20% on HotPotQA.
Key findings
- On HumanEval Python, Reflexion achieves 91.0 pass@1, surpassing the previous state-of-the-art GPT-4 at 80.1; on HumanEval Rust it reaches 68.0 vs. 60.0, MBPP Rust 75.4 vs. 70.9, and Leetcode Hard 15.0 vs. 7.5.
- In AlfWorld, ReAct + Reflexion completes 130 out of 134 tasks using a simple heuristic and improves over strong baselines by an absolute 22% over 12 iterative learning steps; ReAct-only converges at a 22% hallucination rate with no long-term recovery.
- In HotPotQA, Reflexion improves over baselines by 20%; self-reflection gives an 8% absolute boost over episodic memory alone, and although CoT (GT) fails on 39% of questions, Reflexion improves accuracy by 14% without access to ground-truth answers.
- Ablations on the 50 hardest HumanEval Rust problems show full Reflexion at 0.68 vs. baseline 0.60; omitting test generation drops to 0.52, and omitting self-reflection stays at 0.60.
Methods
- Reflexion uses a modular loop with an Actor that generates text/actions, an Evaluator that scores outputs, and a Self-Reflection model that converts feedback into verbal summaries stored in memory; trials repeat until the Evaluator passes or a max trial count is reached, with memory usually bounded to 1–3 experiences.
- Experiments cover AlfWorld (134 environments, six tasks, ReAct + GPT-3), HotPotQA (100 questions, CoT/ReAct, exact-match grading, memory size 3), and programming (HumanEval, MBPP, LeetcodeHardGym; Python/Rust via MultiPL-E; self-generated unit tests up to 6).
- Ablations and analyses vary feedback signals, feedback incorporation methods, and agent types; compromised approaches are tested on HumanEval Rust, and a WebShop experiment is run across 100 environments.
Limitations
- The paper states Reflexion may fall into non-optimal local minima, limits long-term memory to a sliding window with maximum capacity, and notes code-generation test-driven development is limited by non-deterministic, impure, hardware-dependent, or concurrent functions; it also relies on LLM self-evaluation/heuristics and lacks a formal success guarantee.
- The paper reports WebShop failure: after only four trials the agent shows no improvement and does not generate helpful self-reflections, concluding Reflexion cannot solve tasks requiring significant diversity and exploration.
- Evident from the text: self-correction is an emergent quality of stronger, larger models—starchat-beta shows no improvement (baseline 0.26 vs. Reflexion 0.26)—and flaky or incorrect test suites can cause false positives/negatives, with false negatives preferred.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 2 Not valid · 0 Not checked · 59 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Therefore, in the case in which the model generates a flaky test suite, it is possible that all tests pass on an incorrect solution and lead to a false positive label on a code completion [11].”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Analysis
What the sources say
- Partly supportedcites [11] 11. Lam, W., Winter, S., Wei, A., Xie, T., Marinov, D., and Bell, J. (2020). A large-scale longitudinal study of flaky tests. Proc. ACM Program. Lang., 4(OOPSLA). sourceExperimental — accuracy not yet measured
The Introduction (page not provided) says a flaky test can pass and fail on the same code and that changes may be merged when all tests pass. It does not address model-generated test suites, incorrect code completions, or false-positive labels, so those parts of the claim are not established by this paper. (LLM, conf=0.82)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“3 shows an inferior 52% vs 60% (baseline) accuracy, which suggests that the agent is unable to determine if the current implementation is correct without unit tests.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Ablation study
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ReAct + Reflexion significantly outperforms ReAct by completing 130 out of 134 tasks using the simple heuristic to detect hallucinations and inefficient planning.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“4 shows that self-reflection improves learning by an 8% absolute boost over the episodic memory learning advantage.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Analysis
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“on reasoning questions in HotPotQA [28] by 20%”
- Also stated in: § 4 Experiments · plain
“Most notably, Reflexion improves performance over strong baselines by 22% in AlfWorld, 20% in HotPotQA, and 11% on HumanEval.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > 1 Introduction
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4 Experiments · plain
30 more claims
Create a free account to see all 65 claims and every source verdict.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 11
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Memory and reflection systems such as MemAgent (Yu et al., 2025a), Reflexion (Shinn et al., 2023), and Generative Agents (Park et al., 2023) improve retrieval and episodic memory, but do not perform context shaping, state restructuring, or nextinstruction refinement.”
From PAACE: A Plan-Aware Automated Agent Context Engineering Framework (2025) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“However, Reflexion computes verbal reward over a full trajectory, yielding a single end-of-episode signal that conflates credit across reasoning steps: a failure due to a single intermediate error receives the same critique pressure as one due to a flawed strategy.”
From Process Supervision via Verbal Critique Improves Reasoning in Large Language Models (2026) · cites this paper as Shinn et al. [2023]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Unlike Reflexion, which only reflects on unsuccessful trajectories, our approach separately reflects on both successful and unsuccessful ones.”
From An Efficient Context-Dependent Memory Framework for LLM-Centric Agents (2025) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“The dominant approach is to curate memory at write time.”
From Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM Agents (2026) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interaction and code generation are increasingly used together, with code as the modality of choice for actions [48, 59], tool construction [13, 51, 69], and reasoning [39, 66, 67].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [39]
Experimental — accuracy not yet measured
107 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Reflexion: Language Agents with Verbal Reinforcement Learning > 3 Reflexion: reinforcement via verbal reflection
“while M e not pass or t < max trials do”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Self-reflection
“The Actor, Evaluator, and Self-Reflection models work together through trials in a loop until the Evaluator deems τ t to be correct.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Items 1,301–1,400 of 3,537
Beyond Fluency: Toward Reliable Trajectories in Agentic IR
Sinha, A., Ranganathan, S., Das, D., & Dharmaratnakar, A. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808511 · Cited by 3
Software Quality Testing Framework based on Machine Learning Analysis
Li, R. (2024) · International Conference on Mobile Computing and Sustainable Informatics (ICMCSI) · doi:10.1109/icmcsi61536.2024.00063 · Cited by 5
“The problem to be solved in test case prioritization is to find an PT T (cid:10) ' such that satisfies: (3) In reinforcement learning [21]-[24] test case ordering, the reward function plays a key role in guiding the effectiveness of test case ordering.”
BackgroundBootstrapping LLM-based Fact-checking via Iterative Rationalization Finetuning
Lyu, X., Cao, C., Sun, M., Liang, B., Yao, L., & Xu, R. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887888 · Cited by 6
“This process imposes a high requirement on the models’ autonomous capabilities [14], [15], rendering it an ideal LLM-based application.”
Lindenbauer, T., Groh, G., & Schuetze, H. (2025) · Workshop for Research on Agent Language Models (REALM) · doi:10.18653/v1/2025.realm-1.30 · Cited by 7
Exploration with Principles for Diverse AI SupervisionPreprint
Liu, H., Zaharia, M., & Abbeel, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.08899 · Cited by 3
Function-constrained Program SynthesisPreprint
Hajali, P., & Budvytis, I. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.15500 · Cited by 3
“For reference, the top performing model, Reflexion [25], achieves 91% accuracy on HumanEval.”
MethodA collection of principles for guiding and evaluating large language modelsPreprint
Hebenstreit, K., Praas, R., & Samwald, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.10059 · Cited by 5
Enhancing Q-Learning with Large Language Model HeuristicsPreprint
Wu, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.03341 · Cited by 8
World Models with Hints of Large Language Models for Goal AchievingPreprint
Liu, Z., Huan, Z., Wang, X., Lyu, J., Tao, J., Li, X., Huang, F., & Xu, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.07381 · Cited by 5
Banerjee, T., Zhu, R., Yang, R., & Narasimhan, K. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.13915 · Cited by 4
Self-citationTextualized Agent-Style Reasoning for Complex Tasks by Multiple Round LLM GenerationPreprint
Chen, L., Feng, Z., Liu, Z. Q., Jiang, W., Xu, J., Chen, Y., & Wang, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.12411 · Cited by 3
“(Qin et al., 2023), Reflexion (Shinn et al., 2023), Toolformer (Schick et al., 2023), HuggingGPT (Shen et al., 2023), We-bGPT (Nakano et al., 2021).”
What Would You Ask When You First Saw $a^2+b^2=c^2$? Evaluating LLM on Curiosity-Driven QuestioningPreprint
Javaji, S. R., & Zhu, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.17172 · Cited by 3
“Current technologies like tree-of-thought (Yao et al. 2024), maieutic prompting (Jung et al. 2022) and Reflexion (Shinn et al. 2023) incorporate functions resembling self-questioning.”
From Reward Shaping to Q-Shaping: Achieving Unbiased Learning with LLM-Guided KnowledgePreprint
Wu, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.01458 · Cited by 5
Towards Agentic Schema RefinementPreprint
Rissaki, A., Fountalis, I., Vasiloglou, N., & Gatterbauer, W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.07786 · Cited by 4
Jiang, H., Ma, Y., Ding, C., Luan, K., & Di, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.17397 · Cited by 4
Sun, J., Yang, R., Han, X., Niu, J., Li, M., Yang, T.-H., Lu, Y., & Peng, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.20729 · Cited by 3
“3) Long-term Memory and Strategy: To address the limitation of Reflexion [23] being only short-term reflection, CoALA, proposed by Sumers et al. (2023) [25], provided a cognitive blueprint that includes long-term memory.”
Lightweight Latent Verifiers for Efficient Meta-Generation StrategiesPreprint
Piotrowski, B., Drzewakowski, W., Staniszewski, K., & Miłoś, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.16760 · Cited by 3
“Another research direction focuses on self-refinement techniques where the LLM responses are iteratively improved / fixed by the model itself, possibly using external feedback (Havrilla et al., 2024; Madaan et al., 2023; Shinn et al., 2023).”
When Your Own Output Becomes Your Training Data: Noise-to-Meaning Loops and a Formal RSI TriggerPreprint
Ando, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.02888 · Cited by 3
Visualizing Thought: Conceptual Diagrams Enable Robust Planning in LMMsPreprint
Borazjanizadeh, N., Herzig, R., Oks, E., Darrell, T., Feris, R. S., & Karlinsky, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.11790 · Cited by 4
“Other works utilize verification and refinement techniques, such as iterative self-reflection and feedback, to enhance reasoning accuracy [36, 43, 48].”
StarBench: A Turn-Based RPG Benchmark for Agentic Multimodal Decision-Making and Information SeekingPreprint
Zhang, H., Zhu, C., Guo, S., Guo, H., Li, H., & Yu, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.18483 · Cited by 3
“In addition to evaluating the VLMs directly, we also assess two lightweight decoding-time prompting strategies: ReAct (Reasoning+Acting) [47] and Re-flexion (Self-critique) [34].”
Debate-Feedback: A Multi-Agent Framework for Efficient Legal Judgment PredictionPreprint
Chen, X., Mao, M., Li, S., & Shangguan, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.05358 · Cited by 3
“Considering that the debate-feedback framework can essentially be seen as a large language model reasoning framework, we also compare it with classic reasoning methods, including Few-shot Learning , Chain of Thought(CoT) (Wei et al., 2023) and Reflexion (Shinn et al., 2023).”
MAPLE: Multi-Agent Adaptive Planning with Long-Term Memory for Table ReasoningPreprint
Bai, Y., Wang, M., & Vu, T.-T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.05813 · Cited by 4
“Natural language representations offer semantic richness and interpretability, as demonstrated in Reflex-ion (Shinn et al., 2023), Voyager (Wang et al., 2023a), and Generative Agents (Park et al., 2023).”
Cognitive Weave: Synthesizing Abstracted Knowledge with a Spatio-Temporal Resonance GraphPreprint
Vishwakarma, A., Lee, H., Suresh, M., Sharma, P. S., Vishwakarma, R., Gupta, S., & Chauhan, Y. A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.08098 · Cited by 3
“A common architectural pattern involves augmenting LLMs with external tools, memory systems, and structured workflows to bolster their innate reasoning, planning, and execution capabilities [7, 8].”
Effective Reinforcement Learning for Reasoning in Language ModelsPreprint
Huang, L., Li, S., Anupam, S., Lee, I., & Bastani, O. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.17218 · Cited by 3
“LM reasoning has also been enhanced through tool use (Yao et al., 2023b; Shinn et al., 2023).”
Plan Verification for LLM-Based Embodied Task Completion AgentsPreprint
Hariharan, A., Dongre, V., Hakkani-Tür, D., & Tur, G. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.02761 · Cited by 4
“Recent work has demonstrated that LLMs can decompose high-level goals into executable sub-goals and, in multi-agent setups, evaluate and revise each other’s outputs through structured dialogue ([1], [2]).”
Explore-Execute Chain: Towards an Efficient Structured Reasoning ParadigmPreprint
Yang, K., Zhang, T., Shah, R., Yang, K., Ma, Q., Liu, D., & Lamb, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.23946 · Cited by 3
“Other approaches focus on iterative refinement through self-correction (Shinn et al., 2023) or problem decomposition (Zhou et al., 2023; Yao et al., 2022).”
A Benchmark for Procedural Memory Retrieval in Language AgentsPreprint
Kohar, I., & Krishnan, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.21730 · Cited by 4
“Existing work on memory-augmented agents, trajectory retrieval, and retrieval-augmented generation [Shinn et al., 2023, Lewis et al., 2020] demonstrates that incorporating prior experience can improve downstream task performance.”
Mistake Notebook Learning: Batch-Clustered Failures for Training-Free Agent AdaptationPreprint
Su, X., Zhang, Y., Luo, H., Liu, X., & Huang, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.11485 · Cited by 3
“Memory-based approaches store instance-level experience (Shinn et al., 2023; Zhao et al., 2024; Zhang et al., 2024) to correct errors locally.”
HB-Eval: A System-Level Reliability Evaluation and Certification Framework for Agentic AIPreprint
Adam, A. M. I. (2025) · Preprints.org · doi:10.20944/preprints202512.2186.v1 · Cited by 4
AWARE-US: Preference-Aware Infeasibility Resolution in Tool-Calling AgentsPreprint
Kurmaz, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.02643 · Cited by 3
“…and arrive at solutions through a series of reasoned actions powered by Large language models (LLMs) (Xi et al., 2023; Deng et al., 2024; Wang et al., 2024) to conduct reasoning (Yao et al., 2022; Shinn et al., 2023), tool use (Qin et al., 2023b) , memory storage and retrival(Wen et al., 2024).”
Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc ReasoningPreprint
Cao, Q., Zhang, S., Zhou, R., Zhang, R., Qin, P., & Xie, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.22323 · Cited by 5
“Reasoning Language Models (RLMs) (OpenAI, 2025b; DeepSeek, 2025) have achieved remarkable performance, but they come with high costs and latency (Wei et al., 2023; Shinn et al., 2023; Wang et al., 2023b; Cao et al., 2025).”
Recoverability Has a Law: The ERR Measure for Tool-Augmented AgentsPreprint
Vuddanti, S. V., & Chittiprolu, S. K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.22352 · Cited by 3
“Self-Correction and Tool-Use Recovery A growing body of work documents that language model agents can recover from execution failures through reflection, critique, or revision (Shinn et al., 2023; Gou et al., 2024; Vuddanti et al., 2025).”
From Perception to Action: Spatial AI Agents and World ModelsPreprint
Felicia, G., Bryant, N., Putra, H., Gazali, A., Lobo, E., & Rojas, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.01644 · Cited by 3
A Dual-Loop Agent Framework for Automated Vulnerability ReproductionPreprint
Liu, B., Zhao, Y., Chen, Z., Gao, X., & Wang, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.05721 · Cited by 4
A Theoretical Analysis of Test-Driven Code GenerationPreprint
Menet, N., Hersche, M., Krause, A., & Rahimi, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.06098 · Cited by 4
“…unique opportunity for boosting LLM performance via post-generation selection [Li et al., 2022, Shi et al., 2022, Chen et al., 2023, Mündler et al., 2024, Li et al., 2025, Liu et al., 2026] or iterative backprompting [Shinn et al., 2023, Olausson et al., 2024, Chen et al., 2024c, Li et al., 2026].”
Architecting AgentOS: From Token-Level Context to Emergent System-Level IntelligencePreprint
Li, C., Liu, X., Meng, X., & Zhao, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.20934 · Cited by 4
Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document CollectionsPreprint
Borchmann, Ł., Landeghem, J. V., Turski, M., Padarha, S., Kearns, R., Mahdi, A., Rogge, N., Fourrier, C., Han, S., Yao, H., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.12180 · Cited by 5
Brady, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.04660 · Cited by 3
“Reflexion [Shinn et al., 2023] uses self-reflection for task performance; our focus is self-observation for safety and operational legibility.”
Structured Distillation of Web Agent Capabilities Enables GeneralizationPreprint
Lù, X. H., & Reddy, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.07776 · Cited by 4
Agentic AI in Healthcare: Bridging the Gap Between Computational Promise and Clinical EvidencePreprint
Yu, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-9374197/v1 · Cited by 5
UniMesh: Unifying 3D Mesh Understanding and GenerationPreprint
Huang, P., chen, Y., Zhang, Z., & Tang, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.17472 · Cited by 3
“To enhance the model’s capability for comprehensive 3D object understanding, we build UniMesh upon the Reflexion [44] framework, which introduces verbal reinforcement learning through self-reflective feedback.”
Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-ThoughtPreprint
Ildiz, M. E., Gozeten, H. A., Taga, E. O., & Oymak, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.17912 · Cited by 4
“Self-feedback and tool-assisted critique loops further enable iterative refinement and reflection without requiring per-instance human corrections (Madaan et al., 2023; Gou et al., 2024; Shinn et al., 2023).”
Can Current Agents Close the Discovery-to-Application Gap? A Case Study in MinecraftPreprint
Ziheng, Z., Tang, H., Zhang, J., Lin, H., Yang, B., Long, Q., Sun, F., Sun, Y., Liang, Y., Wu, Y. N., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.24697 · Cited by 3
The Compliance Gap: Why AI Systems Promise to Follow Process Instructions but Don'tPreprint
Shin, K. S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.01771 · Cited by 5
The Reasoning Trap: An Information-Theoretic Bound on Closed-System Multi-Step LLM ReasoningPreprint
Shin, K. S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.01704 · Cited by 3
Wu, W., Zhu, S., Zhou, K., & Huang, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.02168 · Cited by 3
“Reflective memory approaches (Shinn et al., 2023; Wu et al., 2025a) and decomposition-based replanning (Prasad et al., 2024) further improve recovery in complex tasks.”
MemFlow: Intent-Driven Memory Orchestration for Small Language Model AgentsPreprint
Chen, J., Li, Y., & Wang, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.03312 · Cited by 4
“Reflexion [41] stores episodic failure traces for self-improvement, while MEM1 [53] learns a compact memory state through reinforcement learning.”
The Moltbook Files: A Harmless Slopocalypse or Humanity's Last ExperimentPreprint
Brach, W., Torrielli, F., Beltoft, S. L., Pirchert, A. B., Schneider–Kamp, P., & Poech, L. G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.07462 · Cited by 5
“…literature has enabled agentic reasoning and planning: ReAct interleaves reasoning and action [Yao et al., 2022], Reflexion adds self-reflection [Cassano et al., 2023], Tree of Thoughts supports multi-branch deliberation [Yao et al., 2023], and memory mechanisms allow agents to accumulate…”
PIVOT: Bridging Planning and Execution in LLM Agents via Trajectory RefinementPreprint
Zhang, T., Popa, A.-I., Xu, Y., Song, R., & Dimitriadis, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.11225 · Cited by 4
“Self-Refine [13] and Reflexion [17] improve outputs or behaviors through self-generated feedback, while CRITIC [5] refines reasoning through tool-interactive self-critique.”
DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic WorkflowsPreprint
Gao, Y., Wang, M., Yu, Y. L., Ma, Z. C., & Qu, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.19099 · Cited by 3
“…scaffolds to agent configurations such as Claude Code and Codex CLI; related lines extend single-agent capability through chained reasoning [15, 40, 42, 48], self-feedback [22, 36], open-ended exploration [28, 38], constitution-style behavior shaping [2, 26], and long-context processing [3, 12].”
HTMLCure: Turning Browser Experience into State Guided Repair for Interactive HTMLPreprint
Wu, J., Yang, J., Zheng, T., Zhang, W., Wang, H., Lou, Y., & Liu, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.26807 · Cited by 3
“ReAct introduces reasoning action traces, WebShop evaluates agents in a shopping website environment, and Reflexion uses trial feedback for later attempts [23, 22, 15].”
Yang, T., Schonlau, M., & Sucholutsky, I. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.30087 · Cited by 3
Automating Formal Verification with Reinforcement Learning and Recursive InferencePreprint
Tan, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.30914 · Cited by 4
Eywa: Provenance-Grounded Long-Term Memory for AI AgentsPreprint
Joshi, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.30771 · Cited by 4
“Reflexion stores verbal feedback from failed attempts so that later agent behavior can improve without parameter updates (Shinn et al., 2023).”
MemoGen: Can Past Experience Improve Future Text-to-Image Generation?Preprint
Chen, W., Yu, K., Tian, B., Song, J., Liang, S., Jia, H., Cheng, K., Li, H., Yuan, K., Wang, L., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.03243 · Cited by 5
“This design is also inspired by memory-based agent learning, where previous feedback or experience is stored externally and later reused without parameter updates [11, 12].”
InfluentialAggregating LLM-Based Weak Verifiers for Spatial Layout GenerationPreprint
Zhang, S., Jones, R. K., Wu, J., & Agrawala, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.05268 · Cited by 3
Guan, X., Zhao, Q., & Deng, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.08151 · Cited by 6
“The missing clue may be a failing test, a file-local convention, a stale trace to ignore, or a short rule buried among irrelevant snippets.”
İşcan, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.16999 · Cited by 3
“Repair rewrites a failed attempt: Self-Debug (X. Chen, Lin, Schärli, & Zhou, 2024), Self-Refine (Madaan et al., 2023), and Reflexion (Shinn et al., 2023) feed execution traces or self-critique back into a revision—but intrinsic self-correction does not reliably improve outputs without an external…”
Rethinking Groups in Critic-Free RLVRPreprint
Wu, Y., Ma, L. R., Xiao, L., Li, M., Wang, X., Zhang, Y., & Nie, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.17250 · Cited by 4
Scalable Behaviour Cloning on Browser Using via Skill DistillationPreprint
Yang, K., Jiang, Z., Peng, Y., Qian, H., Zhang, B., Zheng, Y., Hong, S., Liu, Q., Han, R., Lyu, B., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.32014 · Cited by 3
TREK: Distill to Explore, Reinforce to RefinePreprint
Xu, Y., Zhou, Z., Behdin, K., Marković, J., Sang, H., Li, X., Zhu, W., Du, X., Rahmattalabi, A., He, R., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.05339 · Cited by 3
“…additional inference-time context such as execution feedback, reflection, a longer reasoning budget, or failure lessons, can discover verified solutions that the unaided student currently misses [Wang et al., 2022a, Zelikman et al., 2022, Yao et al., 2023a, Shinn et al., 2023, Yao et al., 2023b].”
HarnessMDP: Learning to Control LLM Agent Harnesses with Offline Reinforcement LearningPreprint
Yi, H., & Song, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.05458 · Cited by 3
“Reflex-ion (Shinn et al., 2023) uses verbal feedback for iterative improvement, Voyager (Wang et al., 2024) accumulates reusable skills, LATS (Zhou et al., 2024a) searches over reasoning and action trajectories, and ADaPT (Prasad et al., 2024) adaptively decomposes difficult tasks.”
CodeRescue: Budget-Calibrated Recovery Routing for Coding AgentsPreprint
He, Q., Cheng, J., Le, C., Wang, R., Liu, X., Chen, Y., Mei, J., Wang, Z., Chen, X., Chen, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.19338 · Cited by 3
“Feedback-driven code repair methods such as Self-Debugging (Chen et al., 2024b) and LDB (Zhong et al., 2024) show how execution information can support local debugging, while iterative-improvement methods such as Self-Refine (Madaan et al., 2023) and Reflexion (Shinn et al., 2023) demonstrate the broader value of revi…”
Sterling, S., Berube, L. T., Glenn, A. J., Shaukat, A., Barua, S., Grams, M. E., & Tsirigos, A. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.07.26.740845 · Cited by 3
Dilkes, J., Yazdanpanah, V., & Stein, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.02087 · Cited by 3
“Self-feedback approaches (Madaan et al., 2023; Bai et al., 2022; Shinn et al., 2023; Kim et al., 2023; Kumar et al., 2024; Li et al., 2025; Szot et al., 2026) let the model make multiple attempts, with previous attempts fed back for critique, refinement, or as demonstrations of failure.”
Wang, R., Wang, B.-H., Kourbatski, E., Bai, J., Chen, H., Song, Z., Boire, G., Hudson, M., & Li, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.05375 · Cited by 3
“These limitations lead to unstable optimization, inefficient compute use, repetitive errors, and shallow exploration, echoing broader challenges in agentic systems with limited structured memory and feedback propagation [27, 42].”
SkillGate: Training In-Policy Skill Selection in Long-Horizon AgentsPreprint
Li, Q., Jiao, W., Shao, S., Zhang, K., Yuan, L., Guo, Y., Liu, W., Zhang, W., & Yu, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.18852 · Cited by 3
“…repairing software and controlling computers through multi-turn loops of reasoning, action and observation (Yao et al., 2022; Schick et al., 2023; Shinn et al., 2023; Yang et al., 2024), and are judged on benchmarks that execute their output (Liu et al., 2023; Zhou et al., 2023; Xie et al.,…”
Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle ProblemPreprint
Liang, Y., Gan, C., Ying, R., Wei, H., Cui, Z., & Ni, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.19626 · Cited by 4
“Self-Refine uses a model as generator, critic, and refiner [15]; Reflexion stores verbal feedback from environmental trials [21]; and Self-Debugging conditions code revision on execution and explanation [5].”
A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications
Mokaria, N., Raj, R., Baiju, D., Shen, X., Pramanick, S., Lin, K. Q., Senocak, A., Shou, M. Z., Torr, P. H. S., Elhoseiny, M., et al. (2026) · Transactions on machine learning research · doi:10.48550/arxiv.2608.20379 · Cited by 4
Ning, J., & Li, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.22856 · Cited by 3
“Language-agent architectures store and retrieve experience or feedback [54, 67, 81].”
Learning What to Practice: Diagnosis-Guided Self-Evolution for Language ModelsPreprint
Wei, X., Ding, Y., Xiong, F., Li, Y., Ma, D., Weng, R., Cai, X., Ding, W., & Zhang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.00768 · Cited by 3
“Memory supports task solving (Shinn et al., 2023; Ouyang et al., 2026) and policy training (Xia et al., 2026a).”
PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic TasksPreprint
Zheng, Y., Sun, H., Bao, J., Liu, L., Jiang, H., Song, Y., & Dou, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.02236 · Cited by 3
“We compare against Prompting, ReAct, Reflexion, PPO, RLOO, GRPO, GiGPO, and HGPO (Yao et al., 2023c; Shinn et al., 2023; Schulman et al., 2017; Ahmadian et al., 2024; Shao et al., 2024; Feng et al., 2025; He et al., 2026).”
Tang, H., Ma, Y., Li, P., & Yuan, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.13406 · Cited by 3
“Nowadays, self-improvement is claimed at many scales: agents that refine their own outputs (Madaan et al., 2023; Shinn et al., 2023); systems that rewrite the routine that improves them, from code-level self-modification (Robeyns et al., 2025; Zelikman et al., 2023) to self-referential agent…”
Zhang, Z., Chao, M., Xiao, K., Chen, Z., Zhang, D., Zhang, Y., Ye, Y., & Xu, W. (2026) · arXiv (Cornell University) · Cited by 4
Fork-and-Flush: Escaping Idea Basins in Autoresearch AgentsPreprint
Cai, Z., Ziakas, C., Kontonis, V., Pearce, T., Sen, S., Krishnamurthy, A., Garg, S., & Papailiopoulos, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.07447 · Cited by 5
“Beyond a single run, LLM agents typically improve across tasks either by accumulating experience (Shinn et al., 2023; Wang et al., 2024; Zhou et al., 2025) or by distilling their own successful trajectories back into the model parameters (Zelikman et al., 2022; Singh et al., 2024; Qu et al., 2024;…”
Language Models as AI Research World ModelsPreprint
Wang, Z., Liu, Z., Lin, M., Weng, Z., Shi, Z., He, B., Sang, Y., Wang, D., Dumoulin, B., Jin, W., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.12235 · Cited by 4
“Reflexion turns feedback into verbal reflections for later trials (Shinn et al., 2023), ExpeL extracts and retrieves knowledge across training-task experiences (Zhao et al., 2024), and Voyager accumulates an executable skill library for later tasks (Wang et al., 2023).”
When Non-Adversarial Noise Hurts Learning: Benchmarking Skill Self-Evolution in AgentsPreprint
Yu, T., Luo, Z., Long, Y., Lu, Y., Zhou, Y., Gong, Y., Cao, S., Wu, Y., Jin, H., Zhang, Z., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.19910 · Cited by 3
“…without weight updates by distilling execution experience into persistent artifacts: executable skill libraries [17, 30], natural-language insights [13, 29], workflow memory [15, 19], and evolving playbooks [26]; recent methods add optimizer-style discipline with validation-gated skill edits [7,…”
Takahashi, K. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20476199 · Cited by 3
Zhang, J., Yuan, B., & Zhang, Q. (2026) · Entropy · doi:10.3390/e28090951 · Cited by 2
“Empirically, Self-Refine [7] and Reflexion [6] operate at the prompt/context layer through textual self-critique.”
Accurate Table Question Answering with Accessible LLMs
Jiang, Y., Wei, F., Bao, E., Li, Y., Ding, B., Yang, Y., & Xiao, X. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00020 · Cited by 3
Xu, X., & Wu, J. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2026.3655508 · Cited by 10
“Decision-making thought is the ability guided by the prompt to break down complex tasks into smaller subgoals based on the agents, and make better decisions on complex tasks through each part of the approach [18] and learning from past experiences [19].”
Dang, J., Wan, Q., Liu, D., Wang, Z., Wang, B., Liu, P., Peng, H., Hu, B., & Chua, T. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113622 · Cited by 3
Boutana, Y., Haddad, S., Soukkou, A., Rabhi, A., & Benghanem, M. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123541 · Cited by 2
Neuro-symbolic NLP: taxonomy, assessment, and directions
Chatzikyriakidis, S., & Lappin, S. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1797587 · Cited by 4
“Tool-augmented LLMs, such as Toolformer (Schick et al., 2023), ReAct (Yao et al., 2022), and Reflexion ( Shinn et al., 2023), superficially resemble federative architectures in that a neural LLM invokes external modules and incorporates their outputs.”
Kaplan, A., Kaplan, B. A., Aydilek, A., Ege, T., Şentürk, S., & Solmaz, İ. (2026) · European Spine Journal · doi:10.1007/s00586-026-10094-0 · Cited by 2
Zombie Agents: Detecting Semantic Livelock in Long-Horizon Autonomous Software
Khanna, S. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814895 · Cited by 3
“Reflexion [6] and SWE-agent [7] use action-string matching or retry limits.”
When LLMs Listen to Experts: Accurate Failure Diagnosis in Operating Systems
Zhao, Y., Zhang, S., Sun, Y., Gu, W., Sun, Y., Wang, L., Shi, L., Huang, C., Yang, G., Zhang, L., et al. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786852 · Cited by 4
Operational Hallucination and Safety Drift in AI Agents
Yu, S., Carroll, F., & Bentley, B. L. (2026) · IEEE International Conference on AI and Data Analytics (ICAD) · doi:10.1109/icad69378.2026.11608655 · Cited by 4
Intelligent Agents for Smart Agriculture: Architectures, Applications, and Future Challenges
Tao, W., Sang, Q., Chen, C., & Mao, Q. (2026) · Agriculture · doi:10.3390/agriculture16171808 · Cited by 2
Large Language Model-Based Smart Agents for Attack-Defense Game on Cyber-Physical Power Systems
Wang, B., Zhou, Y., Wang, S., Xu, B., & Deng, R. (2026) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2026.3681079 · Cited by 2
“• The reflection component [41] is activated at the end of each game round.”
Task-Aware LLM Council with Adaptive Decision Pathways for Complex Task Support
Zhu, W., Yu, L., Yao, H.-R., Tang, Z., & Yue, K. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462526 · Cited by 2
A Survey on LLM Agents: Architecture, Applications, and Challenges
Lee, D., & Kim, H. (2026) · International Conference on Information Networking · doi:10.1109/icoin68469.2026.11480623 · Cited by 7
“Iterative Self-Improvement Loop (Figure 2b) This paradigm introduces an explicit Reflection or Critic module to evaluate the outcome of an action or the current state of the plan [8], [16], [17].”
InfluentialSTAR: A Stage-Attributed Triage and Repair Framework for RCA Agents in Microservices
Wang, J., Chen, R., Wang, Y., Liao, X., Fan, Z., Wu, X., Wu, W., Zhao, J., & Liu, X. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00057 · Cited by 4
Intelligent ERP AI Agent with RAG and LLM Integration: A Systematic Literature Review
P, N. K., & Sasikala, E. (2026) · International Conference on Artificial Intelligence and Machine Learning Applications Theme: Healthcare and Internet of Things (AIMLA) · doi:10.1109/aimla67915.2026.11522318 · Cited by 2
“Reflexion [16] build a pattern by allowing agents collect verbal feedback during attempts.”
A Review of Large Language Models for Stock Price Forecasting from a Hedge-Fund Perspective
Zhang, O., & Zhang, Z. (2026) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai68641.2026.11536649 · Cited by 5
“…paradigms that can be plugged into agentic trading loops include ReACT for interleaving reasoning with tool use [52], Chain-of-Thought (CoT) for stepwise deduction [53], Tree-of-Thought (ToT) for exploring alternate reasoning [54], and Reflexion for self-critique and iterative improvement [55].”
FVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verification
Wan, L. J., Ho, C.-T., Bai, Y., Yu, C., Chen, D., & Ren, H. M. (2026) · IEEE VLSI Test Symposium (VTS) · doi:10.1109/vts69484.2026.11563220 · Cited by 2
“…refinement with self-feedback [44], simulators for self-correction in hardware design [45], linguistic feedback for trial-and-error learning [46], Learning-From-Mistakes Prompting for low-resource language translation [47], recursive introspection [48], retrieved in-context principles from…”
From data to theory: Autonomous large language model agents for materials science
Alfred, S. O., & Sundararaghavan, V. (2026) · Computational Materials Science · doi:10.1016/j.commatsci.2026.114862 · Cited by 3
Reflect: Transparent Principle-Guided Reasoning for Constitutional Alignment at Scale
Bell, H., Zhang, C., Haque, M. M., Zaman, S., Potdar, D., & Fain, B. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806454 · Cited by 2
“…iterative self-refinement can be effective at improving task-specific performance including math reasoning, sentiment reversal, and dialogue generation [13, 56, 65, 78], and at reducing hallucinations [31], though other work has questioned the effectiveness of such self-correction [33, 40, 52].”
ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents
Zhang, B., & Shou, M. Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37174-4_7 · Cited by 2
Liu, Z., Ye, H., Zhang, X., & Qi, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37010-5_34 · Cited by 2
Metadata
- Authors
- Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 8634-8652
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2303.11366
- OpenAlex
- W7133222210
- Cited by
- 1,087
- References
- 0
- Citations by year
- 2026: 760 · 2025: 167 · 2024: 105 · 2023: 55
- FWCI
- 192.15
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- language agents, verbal behavior, language acquisition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.