Reflexion: Language Agents with Verbal Reinforcement Learning
Open accessThe DOI matches its DataCite record; 42 of the 42 references it cites were found in a registry or an index; 7 of the 8 sources its claims cite were checked for support.
Abstract
Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.
Overview
CoChat Summary
TL;DR
Reflexion reinforces language agents through verbal self-reflection stored in episodic memory rather than weight updates, improving performance across decision-making, reasoning, and coding tasks. It reports gains including 91% pass@1 on HumanEval, a 22% absolute improvement on AlfWorld, and 20% on HotPotQA.
Key findings
- On HumanEval Python, Reflexion achieves 91.0 pass@1, surpassing the previous state-of-the-art GPT-4 at 80.1; on HumanEval Rust it reaches 68.0 vs. 60.0, MBPP Rust 75.4 vs. 70.9, and Leetcode Hard 15.0 vs. 7.5.
- In AlfWorld, ReAct + Reflexion completes 130 out of 134 tasks using a simple heuristic and improves over strong baselines by an absolute 22% over 12 iterative learning steps; ReAct-only converges at a 22% hallucination rate with no long-term recovery.
- In HotPotQA, Reflexion improves over baselines by 20%; self-reflection gives an 8% absolute boost over episodic memory alone, and although CoT (GT) fails on 39% of questions, Reflexion improves accuracy by 14% without access to ground-truth answers.
- Ablations on the 50 hardest HumanEval Rust problems show full Reflexion at 0.68 vs. baseline 0.60; omitting test generation drops to 0.52, and omitting self-reflection stays at 0.60.
Methods
- Reflexion uses a modular loop with an Actor that generates text/actions, an Evaluator that scores outputs, and a Self-Reflection model that converts feedback into verbal summaries stored in memory; trials repeat until the Evaluator passes or a max trial count is reached, with memory usually bounded to 1–3 experiences.
- Experiments cover AlfWorld (134 environments, six tasks, ReAct + GPT-3), HotPotQA (100 questions, CoT/ReAct, exact-match grading, memory size 3), and programming (HumanEval, MBPP, LeetcodeHardGym; Python/Rust via MultiPL-E; self-generated unit tests up to 6).
- Ablations and analyses vary feedback signals, feedback incorporation methods, and agent types; compromised approaches are tested on HumanEval Rust, and a WebShop experiment is run across 100 environments.
Limitations
- The paper states Reflexion may fall into non-optimal local minima, limits long-term memory to a sliding window with maximum capacity, and notes code-generation test-driven development is limited by non-deterministic, impure, hardware-dependent, or concurrent functions; it also relies on LLM self-evaluation/heuristics and lacks a formal success guarantee.
- The paper reports WebShop failure: after only four trials the agent shows no improvement and does not generate helpful self-reflections, concluding Reflexion cannot solve tasks requiring significant diversity and exploration.
- Evident from the text: self-correction is an emergent quality of stronger, larger models—starchat-beta shows no improvement (baseline 0.26 vs. Reflexion 0.26)—and flaky or incorrect test suites can cause false positives/negatives, with false negatives preferred.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 2 Not valid · 0 Not checked · 59 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Therefore, in the case in which the model generates a flaky test suite, it is possible that all tests pass on an incorrect solution and lead to a false positive label on a code completion [11].”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Analysis
What the sources say
- Partly supportedcites [11] 11. Lam, W., Winter, S., Wei, A., Xie, T., Marinov, D., and Bell, J. (2020). A large-scale longitudinal study of flaky tests. Proc. ACM Program. Lang., 4(OOPSLA). sourceExperimental — accuracy not yet measured
The Introduction (page not provided) says a flaky test can pass and fail on the same code and that changes may be merged when all tests pass. It does not address model-generated test suites, incorrect code completions, or false-positive labels, so those parts of the claim are not established by this paper. (LLM, conf=0.82)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“3 shows an inferior 52% vs 60% (baseline) accuracy, which suggests that the agent is unable to determine if the current implementation is correct without unit tests.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Ablation study
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ReAct + Reflexion significantly outperforms ReAct by completing 130 out of 134 tasks using the simple heuristic to detect hallucinations and inefficient planning.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“4 shows that self-reflection improves learning by an 8% absolute boost over the episodic memory learning advantage.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Analysis
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“on reasoning questions in HotPotQA [28] by 20%”
- Also stated in: § 4 Experiments · plain
“Most notably, Reflexion improves performance over strong baselines by 22% in AlfWorld, 20% in HotPotQA, and 11% on HumanEval.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > 1 Introduction
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4 Experiments · plain
30 more claims
Create a free account to see all 65 claims and every source verdict.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 11
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Memory and reflection systems such as MemAgent (Yu et al., 2025a), Reflexion (Shinn et al., 2023), and Generative Agents (Park et al., 2023) improve retrieval and episodic memory, but do not perform context shaping, state restructuring, or nextinstruction refinement.”
From PAACE: A Plan-Aware Automated Agent Context Engineering Framework (2025) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“However, Reflexion computes verbal reward over a full trajectory, yielding a single end-of-episode signal that conflates credit across reasoning steps: a failure due to a single intermediate error receives the same critique pressure as one due to a flawed strategy.”
From Process Supervision via Verbal Critique Improves Reasoning in Large Language Models (2026) · cites this paper as Shinn et al. [2023]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Unlike Reflexion, which only reflects on unsuccessful trajectories, our approach separately reflects on both successful and unsuccessful ones.”
From An Efficient Context-Dependent Memory Framework for LLM-Centric Agents (2025) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“The dominant approach is to curate memory at write time.”
From Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM Agents (2026) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interaction and code generation are increasingly used together, with code as the modality of choice for actions [48, 59], tool construction [13, 51, 69], and reasoning [39, 66, 67].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [39]
Experimental — accuracy not yet measured
107 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Reflexion: Language Agents with Verbal Reinforcement Learning > 3 Reflexion: reinforcement via verbal reflection
“while M e not pass or t < max trials do”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Self-reflection
“The Actor, Evaluator, and Self-Reflection models work together through trials in a loop until the Evaluator deems τ t to be correct.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Items 1,901–2,000 of 3,537
Argus: A General-Purpose Agentic Reasoning Runtime for Long-Horizon TasksPreprint
Li, B., Wen, Z., Fan, Y., Wen, C., Yang, F., Guo, H., Wu, J., Zhang, J., Lei, J., Li, M., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.05144 · Cited by 1
“Generative Agents combine stored observations with reflection and retrieval (Park et al., 2023); Reflexion retains feedback across attempts (Shinn et al., 2023); Voyager accumulates executable skills (Wang et al., 2023); and MemGPT manages multiple memory tiers (Packer et al., 2023).”
From Trajectories to Evidence: Auditable Experimental Records for Industrial Research AgentsPreprint
Zhuang, Z., Lao, C., Xu, P., Xu, H., Yang, R., He, Y., Zhang, P., Cao, J., Huang, Y., Mu, G., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.05235 · Cited by 1
EMAS: Stabilizing Multi-Agent System Evolution through Evidence-Guided RevisionPreprint
Fei, C., Si, Q., Liang, K., Xiao, Y., Kalnis, P., & Guo, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.07196 · Cited by 1
“Reflexion converts task feedback into verbal reflections, and ExpeL extracts reusable insights from experience across tasks (Shinn et al., 2023; Zhao et al., 2024).”
Xing, K., Wang, Y., Gan, C., Li, Y., Nguyen, T., Chang, Y., & Wang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.06751 · Cited by 1
“General prompt expansion and tool-using agents can add descriptors, retrieve references, and iterate on outputs [40, 41, 49], but they usually do not maintain an inspectable representation of preserve/transform decisions, scene-role evidence bindings, and artist-specific shortcut constraints.”
REIN: Bridging the Gap between Reasoning and Reliability via Reflection and Abstention AlignmentPreprint
Huang, Z., Yu, L., Hong, D., Huang, X., Lin, W., Chu, Z., Qin, Z., & Zheng, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.07931 · Cited by 1
“…methods, including step-level process reward models [Uesato et al., 2022, Lightman et al., 2024], Self-Refine [Madaan et al., 2023], and Reflexion [Shinn et al., 2023], focus on reliability assessment but presume that the errors are self-recoverable—they have no mechanism to distinguish a fixable…”
RLMOpt: Adaptive Prompt Optimization via Recursive Language ModelsPreprint
Satheesha, S. B., Pande, N. A., Duddempudi, D., & Dandala, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.10471 · Cited by 1
“Language-model agents such as Re-Act [18], Reflexion [17], and Voyager [19] use language models as controllers for multi-step reasoning, reflection, and tool interaction.”
The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark PerformancePreprint
Thakur, S., An, S., DeLuca, C., & Patel, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.11694 · Cited by 1
“Prompt optimisation and self-refinement methods (Madaan et al., 2023; Shinn et al., 2023; Pryzant et al., 2023; Khattab et al., 2023; Agrawal et al., 2025) search for a better prompt without isolating which property of the wording mattered.”
Better, Faster, Stronger: Programmatic Skill Learning Best Reduces Agent CostPreprint
Huang, Z., Wang, X., Wang, A., Jurayj, W., Gutiérrez, B. J., Khashabi, D., & Andrews, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.11338 · Cited by 1
“Existing approaches broadly follow two lines: adapting the model’s weights (Qu et al., 2025; Song et al., 2024; Wang et al., 2025b) or augmenting a frozen model with externally stored knowledge (Shinn et al., 2023; Wang et al., 2023; Zhao et al., 2024).”
Ontology-Grounded World Models for Failure Diagnosis and Closed-Loop Repair in Physical AI SystemsPreprint
Wang, K., Jie, H., Yaoyuan, Y., Zhou, J., & Heng, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.13901 · Cited by 1
“Language-agent systems study iterative feedback, self-refinement, critique, search, and tool use (Shinn et al., 2023; Madaan et al., 2023; Gou et al., 2023; Yao et al., 2023; Yang et al., 2024).”
D$^2$ACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent MemoryPreprint
Liu, X., Liu, Y., Li, C., & Kun, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.17756 · Cited by 1
Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL AgentsPreprint
Hounwanou, C. D., Eze, J., & Gaba, Y. U. (2026) · arXiv.org · doi:10.48550/arxiv.2608.18008 · Cited by 1
“…closes the perception–action loop by feeding textual environment feedback back into the LLM. Subsequent frameworks such as ReAct [25], Reflexion [20], Code as Policies [12], ProgPrompt [22], LLM+P [14], Voyager [23], and Silver et al. [21] broaden the design space, exploring verbal reasoning…”
Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM AgentsPreprint
Wu, C., Jia, H., Liu, Y., Yang, Y., Lin, Y., Zhang, C., Zheng, H., Huang, Y., Zhang, J., Qi, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.27141 · Cited by 1
“…trackers, message feeds and build output, plans what to attempt, executes tool calls against real systems, verifies the outcome, persists what it learned, and decides on its own whether to continue (Yao et al., 2023; Shinn et al., 2023; Wang et al., 2023; Yang et al., 2024; Park et al., 2023).”
FocusGen: Expanding Visual Design Exploration with a Simulated Focus Group of Persona AgentsPreprint
Choi, J., Vasconcelos, H., Lee, H., Zou, C., Lee, T. Y., & Bernstein, M. S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.28001 · Cited by 1
“The designer curates this process through a two-stage workflow.”
Audited vibe coding suggests partial fetal-like convergence of tumor proteomesPreprint
Meyer, J. G. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.08.26.745609 · Cited by 1
“Unlike same-model reflection or self-refinement, 20,21 this case assigned the critic role to another model family, while following the broader logic of model-based critique.”
LLM-Guided Reinforcement Learning for Adaptive NPC Behavior in Multi-Agent Combat GamesPreprint
Nair, H. D., & Karim, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.02931 · Cited by 1
VeriPhy: Agentic Physical Reasoning for World Model Evaluation and RefinementPreprint
Xu, W., Zhu, Y., Ge, C., Shen, X., Shi, J., Kuen, J., Chen, Y., Tao, M., McComb, C., Gutiérrez, N. G., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.03153 · Cited by 1
“This form of self-evolution leaves weights unchanged, in the spirit of context-distillation and experience-memory agents [6, 30, 57, 66, 79, 89, 90], and it treats deployment-time adaptation as learning as state optimization .”
Song, C., Qian, B., Feng, Y., & Song, Z. (2026) · arXiv · doi:10.48550/arxiv.2609.06508
Experience Funnel: A State-Policy Alternating Loop for Self-Evolving AgentsPreprint
Gao, W., Song, Z., Ji, Z., Liu, R., Li, X., Yu, X., Li, X., Cheung, J. C., Lin, W., & Wang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.08919 · Cited by 1
“Early approaches externalize feedback and successful behaviors as reflections, episodic experience, or reusable skills that can guide subsequent interactions without modifying the underlying model parameters (Shinn et al., 2023; Zhao et al., 2024; Wang et al., 2023).”
Cheng, Z., Xu, J., Chai, H., Sun, J., Viswanath, P., & Pan, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.05435 · Cited by 1
“Reflexion retains feedback, Voyager builds skills, and MemGPT manages memory [30, 39, 49].”
VST: Verifiable Structured Transport for Auditable Agent-to-Agent Alpha DiscoveryPreprint
Li, Y., Liu, S., & Liu, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.07065 · Cited by 1
“Whereas single-agent LLM policies are typically trained to produce an answer or an action sequence [19, 21] and optimized with policy-gradient methods such as PPO [23] or its group-relative variant [25], our predictor is optimized against a transaction-level outcome from a coupled dual loop.”
Yang, B., Sun, J., Lu, Z., Qiu, Z., Peng, X., & Zheng, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.09115 · Cited by 2
“Early paradigms rely on heuristic retrieval (Park et al. 2023), hierarchical context (Packer et al. 2023), or verbal reflection (Shinn et al. 2023), whereas recent frameworks optimize structured memory operations via graph representations, offline consolidation, or reinforcement learning (Chhikara…”
Studying Without a Syllabus: Task-Agnostic Environment PreprocessingPreprint
Samuel, V., Ursekar, V., Kalmath, V. S., Shanker, A., Chatrath, V., & Yuan (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.10824 · Cited by 1
From Experiments to Decisions: Reusing Evidence in Autonomous Coding ResearchPreprint
Cheng, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.13299 · Cited by 2
“Reflexion already uses linguistic feedback and episodic memory to influence subsequent trials [13]; research–evolution hybrids and budget-aware agent evaluation likewise precede this study [5, 8, 9].”
Grynets, O., Pochernina, O., Seletska, A., Tukalo, D., Kostetskyi, D., Fedorchuk, I. V., & Lyashkevych, V. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.13577 · Cited by 1
“Durable-processing and transactional-workflow principles motivate explicit recovery, replay, and compensation boundaries [47], while tool-using and reflective-agent studies motivate controlled action and bounded repair mechanisms [48], [49].”
InfluentialEchoPath: Execution-Level Replayable Memory for GUI AgentsPreprint
Zhao, Y., Shanmugham, A., Roy, S., & Xu, Y.-X. (2026) · arXiv.org · doi:10.48550/arxiv.2609.16635 · Cited by 2
“Reflexion stores verbal feedback in an episodic buffer [5], and ExpeL extracts natural-language insights and recalls prior experiences at inference time [6].”
VideoResearcher: Self-Improving Tool Design for Long-Video UnderstandingPreprint
Ye, D., Zhao, D., Wang, K., Hu, Q., Sun, J., Liang, Y., Jia, Y., Huang, Y., Tian, Y., Zhang, J., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.19664 · Cited by 2
“Self-Refine revises outputs through self-feedback (Madaan et al., 2023); Reflexion retains verbal lessons across attempts (Shinn et al., 2023).”
EconSkills: Studying Skill Transfer and Retrieval for Web Agents on Live Economic DataPreprint
Quan, Y., & Liu, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.19523 · Cited by 1
“Reflection can help an agent improve from feedback on a task (Shinn et al., 2023), but a one-shot agent normally discards that experience before a related task.”
QEncodeBench: Can Large Language Models Encode Classical Problems into Verified Quantum Oracles?Preprint
Che, X., Wu, H., Yuan, Y., & Yu, C. (2026) · arXiv (Cornell University) · Cited by 1
Zhuang, Y., Yu, B., Yang, J., Sun, R., Li, Z., Tan, C., Zhang, X., Yin, J., & Chen, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.19830 · Cited by 1
How Strongly Should Task State Influence an LLM Agent?Preprint
Zhang, C., Kweon, W., & Han, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.25686 · Cited by 1
“Agents that reflect on or search over their own trajectory, Reflexion (Shinn et al., 2023) and LATS (Zhou et al., 2023a), change how the model reasons rather than how state reaches it; each could sit on any rung, a cross we leave untested.”
Exact Feedback Is Not Control: Evaluating Text-based Closed-Loop Revision in LLMsPreprint
Jiang, H., Liu, C., Wang, Y., & Feng, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.28150 · Cited by 1
“Such systems revise behavior across trials (Shinn et al., 2023), modify code using execution results (Yang et al., 2024), replan from environmental observations (Huang et al., 2023), and refine candidate programs using evaluator scores (Novikov et al., 2025).”
Scope Before You Persist: Preventing Cross-Family Interference in Agent MemoryPreprint
Cheng, Y., Du, R., Liu, Z., Chen, Q., Lyu, H., Zeng, Y., Wei, Y., & Lin, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.29144 · Cited by 1
“Large language models (LLMs) can critique outputs, retain verbal experience, and edit the agent programs that call them (Shinn et al., 2023; Madaan et al., 2023; Hu et al., 2025).”
EverMine: Dissecting the Self-Evolution of Research Capabilities in Long-Horizon Alpha ResearchPreprint
Li, S., Zhang, J., Yao, R., Qiu, W., Xu, M., & Yuan, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.33524 · Cited by 1
“Reflexion converts execution feedback into language-based reflections that can guide future attempts(Shinn et al., 2023).”
Compositional Safety Failures in Harness Evolution: Identification and Runtime MonitoringPreprint
Zhang, Z., Liu, Z., Lai, W. I., chen, H., & She, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.33123 · Cited by 1
“Such evolution may consolidate experience into memory (Ouyang et al., 2026; Shinn et al., 2023; Tang et al., 2026), refine prompts or reusable skills (Agrawal et al., 2026; Wang et al., 2023), or revise tools (Chen et al., 2025; Qin et al., 2024), allowing the agent to adapt over time without…”
Zhang, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.33153 · Cited by 1
“This supports a diagnostic on the triggered subset under that pairing protocol.”
COEVO: Co-Evolving Context and Parameters for Recursive Self-ImprovementPreprint
Chen, S., Bao, X., Cai, X., Cao, Y., Jiang, W., & Chen, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.33398 · Cited by 1
“Reflexion and CURE study feedback-driven refinement and iterative self-improvement (Shinn et al., 2023; Chen et al., 2026a), while OPRO, EvoPrompt, Promptbreeder, GEPA, and TextGrad optimize prompts or system components through search, evolutionary operators, or textual feedback (Yang et al., 2023;…”
Sivia: Learning to Draw, Learning to LearnPreprint
Yuan, F., Gao, Y., 孙自俊, Li, H., Hua, M., Li, X., & Gao, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.33386 · Cited by 1
UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement LearningPreprint
Fu, Z., Li, Z., Ai, Q., Han, X., Zheng, Z., Wu, H., Fu, T., Zhao, C., Wu, M., He, G., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.34810 · Cited by 1
“ReAct couples reasoning with action; Reflexion and Voyager retain feedback and skills (Yao et al., 2023; Shinn et al., 2023; Wang et al., 2023).”
VACE: Validation-Gated Alternating Co-Evolution of Agent Models and HarnessesPreprint
Qi, J., He, Y., Liu, J., Huang, Q., Hu, S., Dang, Z., Chen, G., Yang, R., Jiang, W., Liu, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.37105 · Cited by 2
“Reflexion [Shinn et al., 2023] stores verbal reflections from task feedback, and ExpeL [Zhao et al., 2023] extracts reusable insights from agent experience without parameter updates.”
NavHarness: Adaptive Goals for Agentic Vision-Language NavigationPreprint
Shi, H., Li, Z., Ding, M., Deng, X., Wang, Y., & Nie, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.39915 · Cited by 1
“ReAct interleaves reasoning with actions, and Reflexion retains verbal feedback for subsequent trials (Yao et al., 2023; Shinn et al., 2023).”
From Solo to Social Learning: Characterizing Recursive Social Improvement in LLMsPreprint
Jha, K., Kleiman-Weiner, M., & Jaques, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.38516 · Cited by 1
“Existing methods for improving these instructions, including prompt optimization, reflection, and evolutionary search, typically focus on a single system (Fernando et al., 2023; Khattab et al., 2023; Shinn et al., 2023; Yuksekgonul et al., 2024; Agrawal et al., 2026).”
EngramBench: A Capability-Grounded Benchmark for Skill-Evolution HarnessesPreprint
Tan, Z., Gu, P., Li, Z., Hu, Y., He, X., Li, D., & Hao, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.39284 · Cited by 1
Covert Assistance: Helpful LLM Agents Evade Oversight in Multi-Agent SystemsPreprint
Alnuhait, D., Wang, G., Khalifa, M., & Peng, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.39050 · Cited by 1
Composing Task-specific Agent Harnesses at Test Time with Reusable PrimitivesPreprint
Kuang, P., Jin, H., Wu, D., Deng, F., Yuan, X., Wang, J., & Wang, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.38912 · Cited by 1
“Early systems develop reasoning–action loops and reflection (Yao et al., 2023; Shinn et al., 2023; Kuang et al., 2026b), alongside interfaces for repository editing and environment interaction (Yang et al., 2024; Wang et al., 2025).”
RSIGame: Autonomous Agentic Game Development with Recursive Self-improvementPreprint
Wu, W., Fu, M., You, J., Zhou, K., Liu, S., Salvi, A., Lin, Y., Zhang, C., Lan, X., Zhu, J., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.39045 · Cited by 1
“These efforts follow broader agentic coding paradigms based on repeated reasoning, execution, and feedback (Yang et al., 2024; Hong et al., 2024; Yao et al., 2023; Shinn et al., 2023).”
DeFA: Dependency-Guided Failure Attribution for LLM AgentsPreprint
Deng, B., Zheng, X., Wei, Y., Zhou, K., Tao, C., Liang, R., Chen, X., Guo, L., & Zhang, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.01256 · Cited by 2
“Self-Refine iteratively revises outputs using feed-back (Madaan et al., 2023); Reflexion carries verbal feedback into later trials (Shinn et al., 2023).”
Rules to Tools: Executable Checks for LLM Agents in Scientific ComputingPreprint
Ning, J., Zhao, G., Xu, C., Zhong, S., Li, X., Zeng, J., & Ke, G. (2026) · arXiv (Cornell University) · Cited by 1
TeleTune: Evolving Agent Skills From Offline TelemetryPreprint
Chen, J. C.-Y., Stengel-Eskin, E., Chen, Y., Llado, P., Counts, S., Bansal, M., Durme, B. V., Jhamtani, H., & Verma, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.05437 · Cited by 1
“Learning reusable skills from experience offers a way to provide agents with such procedural knowledge (Ouyang et al., 2026b; Shinn et al., 2023; Wang et al., 2023, 2025; Zhao et al., 2024; Zheng et al., 2025).”
Valid Stopping in Adaptive Generator-Verifier LoopsPreprint
Hegazy, M., Jordan, M. I., & Dieuleveut, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.06432 · Cited by 1
“Related iterative self-feedback and verbal-reflection methods revise candidates using model-generated feedback [Madaan et al., 2023, Shinn et al., 2023].”
Bayes-Sufficient Compression Is Not Enough: How Does Communication Help Multi-Agent Systems?Preprint
Xie, Y., Zhou, Z., Fan, Y., Ge, Y., Han, B., & Liu, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.03769 · Cited by 1
Lu, H., & Gong, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.05303 · Cited by 2
“Relying on the in-context learning capability of LLMs, one line of work stores reflections on completed episodes, memories, and reusable skills as text at the harness level and reuses them in later interactions [32, 59].”
CoTrace: Data Recipes for Training Terminal Agents with Harness-Model Co-EvolutionPreprint
Chen, J., Zhang, J., Ye, Q., Pruksachatkun, Y., Zhang, H., Zhuo, J., Zhang, Y., Dai, Y., Tan, J., Peng, X., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.10426 · Cited by 2
“Agents that improve without touching weights accumulate verbal reflections or skill libraries at test time (Shinn et al., 2023; Wang et al., 2023); agents that rewrite their own code do so under an evaluator they cannot edit (Zhang et al., 2025).”
Chronos Enables Code Agents to Reason over Software EvolutionPreprint
Yin, X., Zhang, Y., Peng, Z., Ni, C., Cui, Z., & Xin, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.11578 · Cited by 1
“…source material through relations [Edge et al. 2024; Hu et al. 2025; Mostafa et al. 2024], while agent-memory systems reuse episodic records, skills, and reflections across interactions [Packer et al. 2023; Park et al. 2023; Shinn et al. 2023; Wang et al. 2023; Zhao et al. 2024; Zhong et al. 2024].”
Ask Your Advisor: Learning When to Consult Stronger Models in Agentic SystemsPreprint
Sowietzki, P., Holter, S., Metz, Y., Kasper, E., Baur, R., & El‐Assady, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.13870 · Cited by 2
“Execution feedback can support correction (Chen et al., 2024b; Shinn et al., 2023), while successful self-repair depends strongly on the feedback available (Olausson et al., 2024; Huang et al., 2024).”
Zhang, Z., Xiong, J., Lin, H., Chung, J., Zhang, Z., Han, S., Liu, S., & Zhu, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.13632 · Cited by 1
“Related methods use self-critique, stored feedback, or tool-grounded critique (Madaan et al., 2023; Shinn et al., 2023; Gou et al., 2024), but self-correction can fail without a reliable external signal (Huang et al., 2024).”
A Self-Defining Operator for Distributed ApplicationsPreprint
Li, S., Anderson, T. E., Mahajan, R., Peter, S., & Zettlemoyer, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14547 · Cited by 1
“Voy-ager [55] grows an executable skill library; Reflexion [49] reflects verbally across repeated attempts at the same task; ExpeL [65] and Agent Workflow Memory [56] distill cross-task lessons as natural-language text; and RLAIF and re-lated techniques fine-tune the model on outcome-derived…”
TileCas: Cascading Kernel Development Across Programming AbstractionsPreprint
Xiao, X.-Y., Song, Z.-H., Ran, D.-Z., Yang, W., & Xie, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14740 · Cited by 1
“Optimization environments and learned compiler policies expose further decision spaces [10, 21, 34, 58], while execution feedback and self-refinement improve program selection and repair [5, 8, 20, 31, 37, 40, 52].”
Agent-Reflex: Teaching Language Agents to Act ReflectivelyPreprint
Dong, G.-T., Yang, X.-Y., Jin, J., Hu, Y., Song, X., Chen, Y.-F., Li, X., Tan, J., Zhu, Y., Chua, T.-S., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.17953 · Cited by 1
From Existing Methods to New Research Ideas: Two Perspectives on Recursive Self-ImprovementPreprint
Zhou, L., Yu, Y., Liu, Z., Xiang, Q., Hu, T., Dong, Y., Wu, Y., Xiang, Y., Wu, X., Chen, X., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14625 · Cited by 1
“Candidates can revise memory, context, skills, or code, or update model parameters using generated learning material [13, 14, 15, 5, 2].”
Degrees of Freedom in Multi-Agent Coordination: Between Centralized and Decentralized RegimesPreprint
Peng, Y., Lin, H., Wang, Z., Zhang, D., Zhu, Z., Han, Y., Zheng, Y., Wang, H., Li, Q., & Qin, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.17751 · Cited by 1
“…memories, shared traces, selectively retained execution states, and transferable orchestration skills can modify later routing and interaction behavior (Shinn et al., 2023; Zhao et al., 2024; Park et al., 2023; Gao & Zhang, 2024; Yang et al., 2025; Rajib et al., 2026; Lin et al., 2026b).”
Self-Evolving Coding Agents via Harness-Policy OptimizationPreprint
Wei, Z., Yao, W., Peng, B., Wang, D., Wu, Q., Cheng, H., Meng, Y., & Gao, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.22791 · Cited by 1
“Agents reuse feedback through refinement and reflection (Madaan et al., 2023; Shinn et al., 2023), skills (Wang et al., 2023), and .”
PAUSE: Search-Free Steering of Coding Agents Before Their First Source-Code EditPreprint
Patel, V., 谭博引, Xue, J., & Sun, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.29870 · Cited by 1
“Reflexion stores verbal feedback from prior attempts in an episodic memory to guide subsequent trials [31].”
Toward Recursive Self-Improvement in Physical AIPreprint
Luo, M., Bai, Z., Xu, S., Li, Y., Li, K., Qian, M., Lu, W., Zhang, R., Zou, Y., Luo, Z., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.29677 · Cited by 2
ProductWebBench: Evaluating Website Continuity Beyond Task CompletionPreprint
Wu, J., Yang, J., Du, Y., Liu, S., Chai, L., Zheng, T., Zhang, W., Yang, H., Bai, Y., Lou, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.31118 · Cited by 1
“…et al., 2025); agent interfaces and execution platforms support editing and adaptation to observations (Yang et al., 2024; Wang et al., 2024; 2025), and complementary work studies refinement, reflection and the limits of self-repair (Madaan et al., 2023; Shinn et al., 2023; Olausson et al., 2024).”
RSIHub: A Composable Framework for Agent-Evolution MethodsPreprint
Wang, Z., Wang, X., Ye, S., Li, J., Chen, S., Shi, Y., Wang, R., Zhang, J., Wang, C., Wu, K., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.32253 · Cited by 2
“Reflexion retains linguistic feedback across trials; Self-Refine revises outputs at test time [17, 28].”
Discernment and Social Learning as a Companion Training LayerPreprint
Grillos, C. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.22967822 · Cited by 1
Beer, R. D. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.22858925 · Cited by 1
Xu, K., Wang, J., Zhang, N., Wu, Y., Wang, Y., & Cao, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.36618 · Cited by 1
“Language agents also adapt through textual feedback, from iterative output refinement to reflections retained across trials (Madaan et al., 2023; Shinn et al., 2023).”
EvoTeam: Multi-Agent Collaboration via Evolving TeamsPreprint
Cao, Q., He, X., Fu, D., Zhang, L., 邢兆卿, Wang, J., Wang, X. E., Zou, N., Chang, Y., & Zhang, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.39813 · Cited by 1
“Early approaches establish feedback-based refinement through iterative output revision and reflection-guided repeated attempts, with different scopes of experience retention (Madaan et al., 2023; Shinn et al., 2023).”
SelfImprove-Bench: Evaluating Generalization in Self-Improving AgentsPreprint
BangRui, X., Yang, J., Chen, S., Zhou, X., Qiu, J., Yang, Q., Wang, Z., Wu, F., & Wen, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.02797 · Cited by 1
Huang, W., Bi, J., Yang, S., Si, R., Ni, R., Wang, Q., Jiang, S., Zhang, Y., Ruan, F., Zhang, K., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.08318 · Cited by 2
Chen, S. (2026) · arXiv · doi:10.48550/arxiv.2606.22528 · Cited by 21
“…assistants to autonomous systems that reason and act over long, multi-step tool-use trajectories [24, 27], their histories outgrow the context window, and the dominant response is context compaction : an LLM-based summarization or eviction step that periodically compresses the history [4, 14,…”
SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering
Pan, Z., Wang, Q., Hu, Z., Chen, C., Guan, Y., Zhou, Y., Swanson, S., & Ding, Y. (2026) · arXiv · doi:10.48550/arxiv.2607.00151
“Large language model (LLM)-based agents [3, 17, 23, 36, 41, 43, 45, 58, 71, 74] are increasingly deployed in real-world applications, where they execute multi-turn workflows with interleaved model inference, tool usage, and user interaction, as shown in Figure 1(a).”
ALOBED, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7291548
Design and Implementation of a Lightweight Neural Controller for LLM Agent Systems
Langay, B. B. (2026) · International Journal of Latest Technology in Engineering Management & Applied Science · doi:10.51583/ijltemas.2026.150500172
Self-enhancing prompt optimization for language style generation
Liu, H., He, H., Yang, W., Zhou, J., & Zhao, N. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116519
Switch, Reason, and Revise: Enhancing Reasoning Capability of Video Game AI by Large Language Models
Li, W., Liu, H., Lv, J., Huang, K., Song, A., & Lei, Z. (2025) · IEEE Transactions on Games · doi:10.1109/tg.2025.3630322
Bao, M., Zhang, K., Liu, X., Liu, Q., Zhao, H., & Chen, E. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3658068 · Cited by 2
Interactive Visual Assessment for Text-to-Image Generation Models
Mi, X., Tang, F., Cao, J., Sheng, Q., Huang, Z., Li, P., Liu, Y., & Lee, T. (2026) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2026.3658714
“SD3 displays minimal structural differences between successful and failed cases, demonstrating superior robustness to test input complexity variations.”
[Withdrawn] AI Agents Need Memory Control Over More ContextPreprint
Bousetouane, F. (2026) · Qeios · doi:10.32388/mzqb3t
Mizuno, T. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401804
Cognitive Consistency Verification: Neuro-Symbolic Approach for Mitigating VLM Hallucination
Jo, H. S., & Kwon, Y. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402484 · Cited by 1
“[17], and iteratively refines the answer through targeted feed-back [18,19].”
CLMC: Code-Level Metacognition for LLM Agents
Jian, R., Shang, Y., Zhu, Z., Liu, K., Bao, C., & Liu, X. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00295 · Cited by 1
“We compare the one-shot success rate, iteration success rate, and iteration failure rate on AgentBoard’s Tool Operation (Hard) tasks across React [24], Reflexion [18], Galaxy, and our proposed method.”
Iterative Chain-of-Thought: Self-Correcting Reasoning via Verifier-Guided Feedback Loop
Thanjavuru, T. (2026) · Annual Reliability and Maintainability Symposium (RAMS) · doi:10.1109/rams50514.2026.11424546
No skin in the game: why agentic AI requires principal-agent governance
Prause, M. (2026) · AI and Ethics · doi:10.1007/s43681-026-01067-6 · Cited by 2
Reflective LLM Prompt Optimisation for Interpreting GNN Predictions in Traffic Forecasting
Zhang, Z., Kim, J., He, D., & Yildirimoḡlu, M. (2025) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc60802.2025.11423238
Park, Y., Yang, N., Han, E., & Oh, C. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2639655
Meta–cognitive Multi–scale Hierarchical Reasoning for Motor Imagery Decoding
Kim, S., Kwak, H.-G., Kwon, B.-H., & Lee, S.-W. (2026) · International Winter Conference on Brain-Computer Interface (BCI) · doi:10.1109/bci69045.2026.11435070
“Agent–level frameworks such as Reflexion steer multi–step reasoning using internal confidence and reflective feedback [23], and recent large language model frameworks autonomously construct and adapt multi–stage reasoning procedures such as chain–of– though, highlighting meta–cognitive mechanisms…”
CORTEX: A Capability-Driven Reasoning Framework for Zero-Shot Large Language Model Selection
Zhao, K., Liu, Z., Wang, Z., Wang, X., Yang, H., Wang, N., Wang, K., & Lian, S. (2025) · International Conference on Computer Science and Artificial Intelligence · doi:10.1145/3788149.3788171
Teng, Y., & Wang, C. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00643-4
An AI Education Framework Based on LLM-Knowledge Graph for Personalized Learning
Liang, Z., Wang, Y., Zhao, M., Feng, S., Zhang, Y., & Zhang, Y. (2025) · International Conference on Computational Intelligence and Security (CIS) · doi:10.1109/cis69366.2025.11433876
“Although self-reflection mechanisms [14] improve reliability, they typically incur substantial computational costs and adapt poorly to dynamic knowledge updates.”
Zhou, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-9171815/v1
NLP Rover for Disaster Area Inspection
Nitharshana, S., Nivetha, B., S, S. K. N., & Krishnammal, N. (2025) · International Conference on Signal Processing, Computation, Electronics, Power and Telecommunication (IConSCEPT) · doi:10.1109/iconscept66142.2025.11437189
ModSolAgent: Automated Finite Element Code Generation for Abaqus via LLM-Based Agent
Li, Z., Ge, H., Tang, G., & Liu, Y. (2026) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2026.3669495 · Cited by 1
An Agentic AI Framework for Strengthening Preventive Healthcare
Arul Kumaran, G., Balasastha, E., Jayakumar, N., & Jayakumar, R. (2026) · International Conference on Intelligent Systems for Communication, IoT and Security (ICISCoIS) · doi:10.1109/iciscois62701.2026.11447737
“Techniques such as SELF-REFINE and Reflexion help agents optimize decisions, correct errors, and refine preventive care recommendations [7], [8].”
Automatic Diagram Generation with LLMs for Visual Reasoning in Education
Liu, S., & Lin, T. (2025) · International Conference on Modern Educational Technology (ICMET) · doi:10.1109/icmet67594.2025.11451935
“Meanwhile, Shinn et al. (Reflexion) [9] introduce a verbal-reinforcement learning loop that allows LLM agents to reflect on mistakes using text-based feedback, outperforming GPT-4 on programming benchmarks without model fine-tuning.”
Architectural Design Patterns for Agentic Systems
Nagasubramanian, D. (2026) · Apress eBooks · doi:10.1007/979-8-8688-2361-9_5
PC-MAL: Enhancing Log Labeling Consistency via Multi-Agent Collaboration and Reflective Feedback
Liu, J., & Ji, Y. (2026) · International Conference on Communication Networks and Machine Learning (CNML) · doi:10.1109/cnml68938.2026.11452255
Nagasubramanian, D. (2026) · Apress eBooks · doi:10.1007/979-8-8688-2361-9_6
Agentic AI Meets RAG: A Framework for Domain-Specific Knowledge Evaluation
Vijay Kumar, K., M, A. A., Uma, K. V., & S, V. V. (2025) · IEEE International Conference on High Performance Computing, Data and Analytics Workshop (HiPCW) · doi:10.1109/hipcw66559.2025.00044
“Several emerging works highlight the promise of this integration [20], [21], but most implementations are still experimental and face challenges related to scalability, hallucination control, and domain adaptation.”
Metadata
- Authors
- Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 8634-8652
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2303.11366
- OpenAlex
- W7133222210
- Cited by
- 1,087
- References
- 0
- Citations by year
- 2026: 760 · 2025: 167 · 2024: 105 · 2023: 55
- FWCI
- 192.15
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- language agents, verbal behavior, language acquisition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.