CoChat
Sign Up
PreprintCited by 0AI-based Problem Solving and PlanningEvolutionary Algorithms and ApplicationsNatural Language Processing Techniques

Recursive Self-Evolving Agents via Held-Out Selection

Open access
Nguyen, Michael, Nguyen, Quoc, Vuong, Paul·2026·arXiv
DOI 10.48550/arxiv.2606.28374arXiv 2606.28374
Open this paper in CoChat — free accountOpen PDF

Abstract

LLM agents are increasingly improved without weight updates by evolving a natural-language artifact, such as reflections, workflows, playbooks, cheatsheets, or optimized prompts, that conditions a frozen policy. Such methods are typically reported as wins on the single benchmark where they help. We study them apples-to-apples and surface a sharper picture. We introduce RSEA, a Recursive Self-Evolving Agent that carries a compact three-layer natural-language state: an imperative strategy, reusable skills, and a procedural playbook. Across generations, RSEA rewrites all three layers from its own trajectories and commits a candidate only if it does not regress on a disjoint held-out split, using a strict keep-better gate. Across four diverse benchmarks, ALFWorld, GAIA, (τ)-bench, and WebShop, and six faithful baselines, ReAct, Reflexion, GEPA, AWM, ACE, and Dynamic Cheatsheet, all evaluated on one shared local backbone, we find three main results. First, no artifact universally wins. RSEA is the strongest single-pass method on ALFWorld, reaching 69.3% compared with 64.6% for ReAct (McNemar (p=0.015)), and reaches 79.4% with retry, the best overall result. However, concrete-workflow induction, represented by AWM, is best on the strong-backbone tool-use tasks. Second, unguarded context evolution is high-variance and unsafe. Dynamic Cheatsheet, which curates context online without a held-out gate, is near-best on ALFWorld at 70.7%, yet collapses on WebShop, with a score of 0.14 compared with 0.43 for ReAct. Third, RSEA's strict held-out selection is what makes recursive self-evolution monotone-safe: it never significantly underperforms the base agent on any benchmark and falls back to vanilla ReAct when evolved context would hurt.

AI snapshot

Written from the paper’s full text.

TL;DR

RSEA evolves a three-layer natural-language state for frozen LLM agents and commits a candidate only if it does not regress on a disjoint held-out split. Across four benchmarks and six baselines on shared backbones, no artifact universally wins; RSEA is strongest single-pass on ALFWorld and never significantly regresses, while unguarded context evolution can collapse.

Key findings

  • On ALFWorld (134 tasks × 5 seeds, Qwen2.5-7B), RSEA reaches 69.3% vs. ReAct 64.6% (McNemar p=0.015), GEPA 63.9% (p=0.004), and AWM 65.4% (p=0.037); RSEA R is best overall at 79.4%.
  • No method dominates across benchmarks: AWM is marginally best on the 30B tool-use tasks (τ-bench 51.7%, WebShop 0.460), while RSEA is statistically tied with ReAct on τ-bench, WebShop, and GAIA (p>0.1) and never significantly worse; on WebShop RSEA ≈ ReAct (0.437 vs. 0.429) because the strict gate rejected all evolved candidates.
  • Unguarded context evolution is high-variance: Dynamic Cheatsheet is near-best on ALFWorld (70.7%) but collapses on WebShop (0.136 vs. ReAct 0.429) and is worst on τ-bench (36.7%).
  • Ablations show every layer helps (each layer alone 67–70% vs. ReAct 64.2%; skills and playbook 69.8% each; full state 68.5%), and removing held-out selection yields 100% in-sample but a 33-point drop to test (66.7 vs. strict gate 67.3%).

Methods

  • RSEA carries a three-layer natural-language state—strategy, skills, and playbook—and across generations rolls out on an evolve pool, rewrites all layers from trajectories, scores candidates on a disjoint held-out validation split, accepts laterally if no regression, and updates the frozen best state only on strict improvement; RSEA R adds a Reflexion-style per-task retry loop.
  • Baselines are faithful reimplementations of ReAct, Reflexion, GEPA, AWM, ACE, and Dynamic Cheatsheet on one shared harness with identical backbone and decoding budget: Qwen2.5-7B-Instruct for ALFWorld and Qwen3-30B-A3B-Instruct for GAIA, τ-bench, and WebShop, served via vLLM on 4×A100.
  • Benchmarks: ALFWorld (134-task test, 5 seeds, 30-task dev pool), GAIA v1 Level-1 (30 tasks), τ-bench retail (60 held-out eval), and WebShop (1,000-product subset, 100 held-out eval); metrics are success rate or mean dense score, with multi-seed mean±95% CI and paired McNemar tests.

Limitations

  • The paper states that transfer benchmarks use a single seed of the shuffled split and modest eval sizes; GAIA uses live web retrieval, adding run-to-run noise (the ≤1-task gap flips sign across runs).
  • The paper states its scope is the weight-frozen, NL-state regime and does not compare to code- or weight-updating self-improvement; the strict gate guarantees safety on held-out val, not on every test draw when val is small.
  • Evident from the text: ALFWorld single-pass RSEA has a wide 95% CI (±9.3), several retry comparisons only trend toward significance (Reflexion p=0.09, ACE R p=0.13, GEPA R p=0.052), and the full three-layer state (68.5%) does not strictly outperform individual layers (69.8%), so the necessity of all three layers is not established.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2606.28374

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

  • Findingown data

    RSEA with retry (RSEA R) attains the best overall success on ALFWorld at 79.4%.

    “RSEA R attains the best overall success (79.4%), significantly above ReAct (p<10 -4 ), single-pass RSEA (p<10 -4 ), and every no-evolution baseline”
    • Also stated in: § Contributions. · plain
      “RSEA R is best overall (79.4%)”
    Quote in the paperSubstantial claim
  • Findingown data

    Concrete-workflow induction (AWM) is best on the strong-backbone tool-use tasks.

    “concrete-workflow induction (AWM) is best on the strong-backbone tool-use tasks”
    • Also stated in: § 5 Experimental Setup · plain
      “on the 30B tool-use benchmarks the concrete-workflow method AWM is marginally best (τ -bench 51.7%, WebShop 0.460)”
    Quote in the paperSubstantial claim
  • Findingown data

    No artifact universally wins across benchmarks.

    “no artifact universally wins”
    • Also stated in: § 5 Experimental Setup · plain
      “No single method dominates.”
    Quote in the paperSubstantial claim
  • Findingown data

    RSEA is statistically tied with ReAct on τ-bench, WebShop, and GAIA (McNemar p>0.1).

    “RSEA is statistically tied with ReAct on τ -bench, WebShop, and GAIA (McNemar p>0.1)”
    Quote in the paperSubstantial claim
  • Findingown data

    Removing held-out selection causes severe overfitting (a perfect in-sample score but a 33-point drop to test).

    “removing held-out selection causes severe overfitting (a perfect in-sample score but a 33-point drop to test)”
    • Also stated in: § Contributions. · plain
      “removing held-out selection overfits the evolve pool (100% in-sample, a 33-point drop to test)”
    Quote in the paperSubstantial claim
  • May combine two claims. 36
  • Classified as probably not a substantial claim. 14

  • May combine two claims.

    Findingown data

    Each layer of the evolved state on its own lifts ALFWorld success to 67-70%, with skills and playbook layers carrying most benefit (69.8% each).

    “each layer on its own lifts success to 67-70%, with the skills and playbook layers (the concrete sub-routines and procedures) carrying most of the benefit (69.8% each)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Combining layers does not strictly improve further at this scale (full state 68.5%, within multi-seed band), so the three layers encode overlapping procedural knowledge rather than three orthogonal signals.

    “Combining layers does not strictly improve further at this scale (full state 68.5%, within the multi-seed band), so the three layers encode overlapping procedural knowledge rather than three orthogonal signals”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Dynamic Cheatsheet, an unguarded online context curation method, is near-best on ALFWorld (70.7%) yet collapses on WebShop (score 0.14 vs. ReAct 0.43).

    “Dynamic Cheatsheet, which curates context online with no held-out gate, is near-best on ALFWorld (70.7%) yet collapses on WebShop (score 0.14 vs. ReAct 0.43)”
    • Also stated in: § 5 Experimental Setup · plain
      “Dynamic Cheatsheet, which curates context online with no held-out gate, is near-best on ALFWorld (70.7%) yet collapses on WebShop (score 0.136 vs. ReAct 0.429) and is worst on τ -bench (36.7%)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Ablations show every layer of the evolved state helps and that removing held-out selection overfits the evolve pool (100% in-sample, a 33-point drop to test).

    “Ablations show every layer of the evolved state helps (the layers overlap rather than being strictly complementary) and that removing held-out selection overfits the evolve pool (100% in-sample, a 33-point drop to test)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    RSEA never regresses on the tool-use benchmarks; where injected NL context does not help, its strict gate falls back toward vanilla ReAct (e.g., on WebShop every evolved candidate hurt held-out val, so frozen state is empty and RSEA ≈ ReAct 0.437 vs. 0.429).

    “RSEA never regresses.Where injected NL context does not help, RSEA's strict gate falls back toward vanilla ReAct -e.g. on WebShop every evolved candidate hurt held-out val, so the frozen state is empty and RSEA ≈ ReAct (0.437 vs. 0.429)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The single regression on ALFWorld (clean, -6.5 points) is mild distraction and is recovered by retry.

    “The single regression (clean, -6.5 points) is mild distraction, and is recovered by retry”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Methods without a held-out gate are high-variance and can catastrophically regress.

    “Methods without a held-out gate are high-variance and can catastrophically regress”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On the strong-backbone tool-use benchmarks, no NL artifact dominates; RSEA is statistically tied with ReAct (and never significantly worse), while AWM's concrete workflows give the only consistent small lift.

    “On the strong-backbone tool-use benchmarks no NL artifact dominates; RSEA is statistically tied with ReAct (and never significantly worse), while AWM's concrete workflows give the only consistent small lift”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    RSEA is the strongest single-pass method on ALFWorld, achieving 69.3% success versus ReAct's 64.6% (McNemar p=0.015).

    “RSEA is the strongest single-pass method on ALFWorld (69.3% vs. ReAct 64.6%, McNemar p=0.015; 79.4% with retry, best overall)”
    • Also stated in: § 5 Experimental Setup · plain
      “The single-pass evolved prior (RSEA, 69.3%) significantly improves over vanilla ReAct (64.6%, McNemar p=0.015)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GAIA results are within run-to-run web noise: two independent evaluations of the same frozen states disagree on the sign of the ≤1-task gap.

    “GAIA in particular is within run-to-run web noise: two independent evaluations of the same frozen states disagree on the sign of the ≤ 1-task gap”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Evolving context is not a free lunch; an injected artifact can degrade an otherwise-capable base policy (context distraction).

    “evolving context is not a free lunch.We name the failure mode context distraction: an injected artifact that degrades an otherwise-capable base policy”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    At inference RSEA adds only the rendered preamble (≤ a few hundred tokens, bounded by construction since the rewrite may drop layers), versus online methods whose context grows unboundedly with the test stream.

    “At inference RSEA adds only the rendered preamble (≤ a few hundred tokens, bounded by construction since the rewrite may drop layers), versus online methods whose context grows unboundedly with the test stream”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    RSEA is significantly best on ALFWorld and never regresses elsewhere.

    “RSEA is significantly best on ALFWorld and never regresses elsewhere.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    No held-out gate leads to 100% in-sample performance but a 33-point drop to test (overfitting).

    “No gate ⇒ 100% in-sample but a 33-pt drop to test (overfitting).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The strict variant returns the simpler (often empty) state when a candidate merely ties, which is the fall-back-to-ReAct behavior observed on WebShop and τ-bench.

    “the strict variant returns the simpler (often empty) state in that case, which is exactly the fall-back-to-ReAct behavior we observe on WebShop and τ -bench.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    DC has no held-out gate and is high-variance (best on ALFWorld, worst on WebShop/τ-bench).

    “DC has no held-out gate and is high-variance (best on ALFWorld, worst on WebShop/τ -bench)”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    For the tool-use benchmarks, the official dev split is saturated for our backbone, so we carve disjoint evolve/val/eval slices from a seed-shuffled test pool (no leakage).

    “For the tool-use benchmarks the official dev split is saturated for our backbone, so we carve disjoint evolve/val/eval slices from a seed-shuffled test pool (no leakage)”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The official dev split is saturated for the backbone on the tool-use benchmarks.

  • May combine two claims.

    Premisereasoning

    Unguarded commitment is the more damaging: methods that append or rewrite without a held-out check commit context that can distract the base policy, and are high-variance across benchmarks.

    “Unguarded commitment is the more damaging: methods that append or rewrite without a held-out check (notably online curation) commit context that can distract the base policy, and -because the same mechanism that helps when context is useful is unconstrained when it is not -they are high-variance across benchmarks”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Only the frozen best state is run on a held-out test split, so there is no leakage.

    “Only the frozen best state is run on a held-out test split, so there is no leakage”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Premisenone given

    Single-form artifacts conflate complementary kinds of knowledge - a high-level strategy, a reusable sub-routine, and a concrete procedure are not interchangeable.

    “Single-form artifacts conflate complementary kinds of knowledge -a high-level strategy, a reusable sub-routine, and a concrete procedure are not interchangeable, yet most methods fold them into one string”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The only difference from vanilla ReAct is the injected, evolved state, which keeps the comparison clean.

    “the only difference from vanilla ReAct is this injected, evolved state, which keeps the comparison clean”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    An evolution loop that commits whatever it produces inherits the variance of its own rewrites; one that commits a change only when it improves held-out performance cannot do worse than not evolving at all.

    “An evolution loop that commits whatever it produces inherits the variance of its own rewrites; one that commits a change only when it improves held-out performance cannot do worse than not evolving at all”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Every method (RSEA and all baselines) uses the same locally served model and decoding budget; the task LLM and the reflection/rewrite LLM are identical.

    “Every method -ours and all baselines -uses the same locally served model and decoding budget; the task LLM and the reflection/rewrite LLM are identical”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    RSEA's strict held-out selection makes recursive self-evolution monotone-safe: it never significantly underperforms the base agent on any benchmark, falling back to vanilla ReAct when evolved context would hurt.

    “RSEA's strict held-out selection is what makes recursive self-evolution monotone-safe: it never significantly underperforms the base agent on any benchmark, falling back to vanilla ReAct when evolved context would hurt”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    The three-layer structure is best read as an interpretable organization of that knowledge, not as a claim that all three are individually necessary.

    “The three-layer structure is best read as an interpretable organization of that knowledge -and the substrate the selection gate operates on -not as a claim that all three are individually necessary”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Baselines are faithful reimplementations on our harness, injected at the same point as RSEA and held to the same per-method rollout budget.

    “ReAct, Reflexion, GEPA, AWM, ACE, and Dynamic Cheatsheet, each a faithful reimplementation on our harness (we open the original repositories and port the core loops), injected at the same point as RSEA and held to the same per-method rollout budget”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Both ablations use the ALFWorld test set, the identical ReAct loop, and temp-0.6 multi-seed decoding; only the injected preamble (resp. the selection rule) changes.

    “Both ablations use the ALFWorld test set (where RSEA helps), the identical ReAct loop, and temp-0.6 multi-seed decoding; only the injected preamble (resp.the selection rule) changes”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    The results are limited to the weight-frozen, NL-state regime; they do not extend to code- or weight-updating self-improvement.

    “Our scope is the weight-frozen, NL-state regime; we do not compare to code-or weight-updating self-improvement.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    ALFWorld numbers are 5 seeds with paired McNemar over matched (task, seed) cells.

    “are 5 seeds with paired McNemar over matched (task, seed) cells”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Ablations use the ALFWorld test set with temp-0.6 multi-seed decoding.

    “ablations use the ALFWorld test set with temp-0.6 multi-seed decoding.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    An evolved NL state helps least when a strong backbone already follows a detailed tool API (τ-bench) or when the bottleneck is retrieval/grounding (WebShop, GAIA).

    “it helps least when a strong backbone already follows a detailed tool API (τ -bench) or when the bottleneck is retrieval/grounding (WebShop, GAIA).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The τ-bench retail frozen best state is a faithful summary of the retail policy; the 30B agent already follows most of it from the wiki, so the held-out gain is neutral-consistent with our scope condition.

    “This is a faithful summary of the retail policy; the 30B agent already follows most of it from the wiki, so the held-out gain is neutral -consistent with our scope condition.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 30B agent already follows most of the retail policy from the wiki.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    All methods share one locally served backbone and decoding budget; task and reflection LLMs are identical.

    “All methods share one locally served backbone and decoding budget; task and reflection LLMs are identical.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The transfer benchmarks use a single seed of the shuffled split and modest eval sizes.

    “The transfer benchmarks use a single seed of the shuffled split and modest eval sizes”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    GAIA uses live web retrieval, which adds run-to-run noise (the ≤1-task gap flips sign across runs).

    “GAIA uses live web retrieval, which adds run-to-run noise (the ≤ 1-task gap flips sign across runs).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The strict gate guarantees safety on held-out val, not on every test draw when val is small.

    “The strict gate guarantees safety on held-out val, not on every test draw when val is small.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Transfer benchmarks use a seed-shuffled disjoint split with paired McNemar over tasks.

    “transfer benchmarks use a seedshuffled disjoint split with paired McNemar over tasks”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The ALFWorld frozen best state's strategy, skills, and playbook are precisely the failure modes of the 7B ReAct agent, and injecting them yields the significant ALFWorld gain.

    “These are precisely the failure modes of the 7B ReAct agent, and injecting them yields the significant ALFWorld gain.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The evolved state helps on ALFWorld and is neutral on the strong tool-use backbone.

    “which lets us read why the method helps on ALFWorld and is neutral on the strong tool-use backbone.”
    Quote in the paperSubstantial claim
39 of 39

  • The paper relies on this without support.

    The base policy Qwen2.5-7B-Instruct is below ceiling on ALFWorld.

    “We serve Qwen2.5-7B-Instruct for ALFWorld (to keep the base agent below ceiling)”

  • The paper relies on this without support.

    The official dev split is saturated for the backbone on the tool-use benchmarks.

    “For the tool-use benchmarks the official dev split is saturated for our backbone”

  • The paper relies on this without support.

    The 30B agent already follows most of the retail policy from the wiki.

    “the 30B agent already follows most of it from the wiki”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • RSEA is called the strongest single-pass method despite a higher-scoring baseline

    Major

    Dynamic Cheatsheet scores 70.7% on ALFWorld, above RSEA's 69.3%. The paper also says their results are statistically tied, so RSEA is not the strongest single-pass method by reported success rate.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 5

  1. Fanxiao Li, Jiaying Wu, Canyuan He, and Wei Zhou. CMIE: Combining mllm insights with external evidence for explainable out-of-context misinformation detection. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9342-9354, 2025a.
  2. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):198343, 2025a.
  3. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji- Rong Wen. From exploration to mastery: Enabling llms to master tools via self-driven interactions. In The Thirteenth International Conference on Learning Representations (ICLR), 2025b. URL https://openreview.net/forum?id=QKBu1BOAwd.
  4. Changle Qu, Sunhao Dai, Hengyi Cai, Jun Xu, Shuaiqiang Wang, and Dawei Yin. MatchTIR: Fine-grained supervision for tool-integrated reasoning via bipartite matching. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), 2026.
  5. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023b.

Authors
Nguyen, M., Nguyen, Q., & Vuong, P.
Affiliations
Monash University Malaysia (MY)
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2606.28374
arXiv
2606.28374
OpenAlex
W7166692021
Cited by
0
References
0
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
LLM agents, React, ALFWorld, WebShop, tool use

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-24.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-24.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review