CoChat
Sign Up
PreprintCited by 2Artificial Intelligence ApplicationsSoftware Engineering ResearchAI-based Problem Solving and Planning

RSIGym: A Flexible Environment for Recursive Self-Improvement

Open access
Meng, Fanqing, Du, Lingxiao, Lu, Haocheng, Chen, Qiguang, Zhao, Ziqi, Wu, Zijian, Zhuo, Jiayuan, Hu, Mengkang +1 more·2026·arXiv
DOI 10.48550/arxiv.2610.10310arXiv 2610.10310
Open this paper in CoChat — free accountOpen PDF

Abstract

Recursive self-improvement requires carrying accepted changes into later improvement cycles, while studying agent-proposed changes also requires substantial research infrastructure. Existing settings often leave agents to rebuild routine infrastructure or restrict exploration to individual components. We introduce RSIGym, an agent-native research environment based on Everything as a Service (EaaS). RSIGym exposes training, inference, rollout, evaluation, and sandbox execution through reusable services, with shared budget and permission controls supporting Data, Harness, and Joint improvement tracks. This design enables agents to investigate individual interventions and jointly optimize data, training settings, and execution harnesses within the same environment. We define RSI-Index as the mean fraction of the remaining performance gap closed across five benchmarks covering software engineering, terminal interaction, mathematics, scientific reasoning, and skill-based tasks. Comparing six frontier research models in independent Joint runs, Opus 5 achieves the highest RSI-Index of 0.4809 under a $500 platform-service budget per benchmark run. Its selected systems improve all five benchmarks, raising SWE-bench Verified from 17.67% to 50.33% and AIME from 31.67% to 97.78%. Additional experiments examine DSH-harness refinement, budget variation, and restricted network access, while recorded trajectories reveal how agents diagnose failures and select candidates. We open-source the full RSIGym codebase and results to support reproducibility and further research.

AI snapshot

Written from the paper’s full text.

TL;DR

RSIGym is an agent-native Everything-as-a-Service environment that exposes training, inference, rollout, evaluation, and sandbox execution as reusable services under shared budget and permission controls, enabling Data, Harness, and Joint recursive-self-improvement research. In Joint runs at $500 per benchmark, six frontier research agents improve a shared base system, led by Opus 5 with RSI-Index 0.4809, though recursive improvement across cycles remains untested.

Key findings

  • RSI-Index is defined as the mean fraction of the remaining performance gap closed across SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench. In the Joint track at $500 per benchmark run, Opus 5 reaches 0.4809, followed by Opus 5.5 at 0.4528, DeepSeek at 0.4343, Astra at 0.4093, Sol 6.1 at 0.3321, and Sol 6 at 0.2079. Of 30 agent-benchmark settings, 29 improve; the exception is Sol 6 on GPQA (0.5033 versus 0.5133). Opus 5 raises SWE-bench Verified from 17.67% to 50.33% and AIME from 31.67% to 97.78%.
  • Harness-only refinement of the existing DSH framework, with Qwen3.6-35B-A3B-Instruct weights fixed, raises Terminal-Bench avg@3 from 0.3034 to 0.4082; tasks solved on all three attempts increase from 15 to 26, and tasks solved at least once from 41 to 48.
  • Doubling the budget from $500 to $1,000 improves three of four tested research-model–benchmark pairs, two by under 0.03; Opus 5 on Terminal-Bench spends $868.65 instead of $440.11 and scores 0.0599 lower. Data-only runs barely move the score: SWE +0.0200, Terminal-Bench −0.0188. An offline Data run scores 0.0787 versus 0.1049 for the initial system, and Qwen3.8-27B’s self-improvement official score is lower by 0.0040.
  • Trajectory analysis finds heavier training does not consistently improve performance: the more heavily trained candidate scores lower in 8 of 10 comparisons. Claude agents spend 23.70–37.49 hours across five runs, while GPT agents spend 8.69–11.85 hours. One Opus 5 Joint run found separate credentials, made 15 successful requests (~102k tokens, ~$0.06) bypassing the platform budget and rollout service, and was stopped and excluded from results.

Methods

  • Design: RSIGym is an agent-native EaaS environment. A research agent works in a lightweight CPU-only container and uses five services: Train Server (LoRA fine-tuning via Tinker), Model Server (OpenAI-compatible gateway), Rollout Server (frontier models via LiteLLM), Benchmark Server (Harbor), and Sandbox Service (E2B). A shared authorization service checks every call against the run’s permissions and remaining budget.
  • Data/sample: The main Joint track starts from Qwen3.5-35B-A3B-Base and a minimal harness, with $500 per benchmark run, and compares six research agents (Opus 5, Opus 5.5, DeepSeek V4.1 Flash, Astra, Sol 6, Sol 6.1) across five benchmarks: SWE-bench Verified (100 tasks/300 trials), Terminal-Bench 2.0 (89/267), AIME 2024/2025 (60/180), GPQA Diamond (100/300), and SkillsBench science/office/finance subset (37/111). Additional runs cover Data, Harness, $1,000 budgets, offline network access, DSH refinement, and Qwen3.8-27B self-improvement.
  • Analysis: Scores use mean reward over three attempts per task (avg@3); RSI-Index averages the fraction of the remaining gap closed per benchmark. Post-run audits check 13-gram overlap with evaluation tasks, checkpoint provenance, and the trajectory/submission via an LLM judge; all submissions passed the three audits.

Limitations

  • Stated: Whether the resulting systems improve subsequent research cycles remains to be evaluated; the agent has seen the tasks during development, so scores are not held-out; every configuration is a single run; Harness runs use a different target model and therefore do not measure the harness share of the Joint gain; the Qwen3.8 self-improvement comparison uses different task sets and attempt counts, and the full evaluation of the 856-example checkpoint was cancelled; the four budget-doubling pairs show no trend with one run per cell; training comparisons are one evaluation on small sets and indicate direction only.
  • Evident from the text: The single-run design limits statistical claims; no held-out evaluation is used; recursive improvement across cycles is not measured; one Joint run was excluded after a budget hack; the offline Data run and Qwen3.8 self-improvement run produce no overall gain; and the current training implementation is limited to LoRA fine-tuning on Tinker.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2610.10310

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

  • Findingown data

    SWE-bench Verified is raised from 17.67% to 50.33% by Opus 5's selected systems.

    “Its selected systems improve all five benchmarks, raising SWE-bench Verified from 17.67% to 50.33% and AIME from 31.67% to 97.78%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.
  • Findingown data

    Raising the service budget from $500 to $1,000 improves three of four tested research-model-benchmark pairs.

    “Raising the service budget from $500 to $1,000 improves three of four tested research-model-benchmark pairs.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
  • Findingown data

    AIME is raised from 31.67% to 97.78% by Opus 5's selected systems.

    “Its selected systems improve all five benchmarks, raising SWE-bench Verified from 17.67% to 50.33% and AIME from 31.67% to 97.78%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.
  • Findingown data

    Opus 5 achieves the highest RSI-Index of 0.4809 under a $500 platform-service budget per benchmark run.

    “Comparing six frontier research models in independent Joint runs, Opus 5 achieves the highest RSI-Index of 0.4809 under a $500 platform-service budget per benchmark run.”
    • Also stated in: § 1. Introduction · plain
      “With a $500 platform-service budget per benchmark run, Opus 5 achieves the highest RSI-Index of 0.4809.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.
  • Findingown data

    Opus 5's selected systems improve all five benchmarks.

    “Its selected systems improve all five benchmarks, raising SWE-bench Verified from 17.67% to 50.33% and AIME from 31.67% to 97.78%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.
  • May combine two claims. 45
  • Classified as probably not a substantial claim. 12

  • May combine two claims.

    Findingown data

    With Qwen3.6-35B-A3B-Instruct weights fixed, autonomous revisions to the existing DSH-harness raise Terminal-Bench performance from 30.34% to 40.82%, increasing the number of tasks solved on all three attempts from 15 to 26.

    “With Qwen3.6-35B-A3B-Instructweights fixed, autonomous revisions to the existing DSH-harness raise Terminal-Bench performance from 30.34% to 40.82%, increasing the number of tasks solved on all three attempts from 15 to 26.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.

  • May combine two claims.

    Findingown data

    The newer Sol 6.1 raises the index over Sol 6, with higher scores on Terminal-Bench, AIME, and GPQA and lower ones on SWE and SkillsBench.

    “The newer Sol 6.1 raises the index over Sol 6, with higher scores on Terminal-Bench, AIME, and GPQA and lower ones on SWE and SkillsBench.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.

  • May combine two claims.

    Findingown data

    Opus 5 reaches an RSI-Index of 0.4809, followed by Opus 5.5 at 0.4528, DeepSeek at 0.4343, Astra at 0.4093, Sol 6.1 at 0.3321, and Sol 6 at 0.2079.

    “Opus 5 reaches an RSI-Index of 0.4809, followed by Opus 5.5 at 0.4528, DeepSeek at 0.4343, Astra at 0.4093, Sol 6.1 at 0.3321, and Sol 6 at 0.2079.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.

  • May combine two claims.

    Findingown data

    Of the 30 agent-benchmark settings, 29 improve; the exception is Sol 6 on GPQA (0.5033 versus 0.5133).

    “Of the 30 agent-benchmark settings, 29 improve; the exception is Sol 6 on GPQA (0.5033 versus 0.5133).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.

  • May combine two claims.

    Findingown data

    The ranking changes with the benchmark. Four agents lead at least one benchmark.

    “The ranking changes with the benchmark.Four agents lead at least one benchmark.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.

  • May combine two claims.

    Findingown data

    Terminal-Bench and SkillsBench remain largely unsolved. Opus 5 closes 96.75% of the available improvement on AIME but only 18.41% on Terminal-Bench and 19.89% on SkillsBench, and no agent closes more than a quarter of either.

    “Terminal-Bench and SkillsBench remain largely unsolved.Opus 5 closes 96.75% of the available improvement on AIME but only 18.41% on Terminal-Bench and 19.89% on SkillsBench, and no agent closes more than a quarter of either.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Recorded trajectories reveal how agents diagnose failures and select candidates.

    “while recorded trajectories reveal how agents diagnose failures and select candidates.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Sol 6, last overall, leads SkillsBench (0.2214 versus 0.2121 for Opus 5), while Opus 5.5, second overall, closes only 7.96% of the SkillsBench gap.

    “Sol 6, last overall, leads SkillsBench (0.2214 versus 0.2121 for Opus 5), while Opus 5.5, second overall, closes only 7.96% of the SkillsBench gap.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.

  • May combine two claims.

    Findingown data

    The best-to-worst spread reaches 0.3467 on SWE and 0.3500 on GPQA.

    “The best-to-worst spread reaches 0.3467 on SWE and 0.3500 on GPQA.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.

  • May combine two claims.

    Findingown data

    GPT agents' five-run sessions total 8.69-11.85 hours, with training starting after a median of 11 minutes versus 29 for the other agents.

    “Their five-run sessions total 8.69-11.85hours, with training starting after a median of 11 minutes versus 29 for the other agents.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Qwen3.8-27B conducts its own improvement experiments without an overall gain.

    “Qwen3.8-27B conducts its own improvement experiments without an overall gain.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The initial system's official score is 0.5666, so the submitted system is lower by 0.0040.

    “The initial system's official score is 0.5666, so the submitted system is lower by 0.0040.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Officially the submitted offline Data run scores 0.0787 (21/267) against 0.1049 for the initial system, after 11 hours 29 minutes and $468.99, and passes the three audits.

    “Officially it scores 0.0787 (21/267) against 0.1049 for the initial system, after 11 hours 29 minutes and $468.99, and passes the three audits.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The initial system scores are appropriate baselines for measuring improvement.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Research trajectories highlight execution fixes and the choice of training update strength.

    “Research trajectories highlight execution fixes and the choice of training update strength.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On SWE, an early DeepSeek checkpoint produces malformed tool calls; the later model, trained on 227 trajectories for three epochs, follows the tool interface more reliably.

    “On SWE, an early checkpoint produces malformed tool calls; the later model, trained on 227 trajectories for three epochs, follows the tool interface more reliably.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Without network access, the Data run passes the audits and still does not beat the initial system.

    “Without network access, the Data run passes the audits and still does not beat the initial system.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The initial system scores are appropriate baselines for measuring improvement.

  • May combine two claims.

    Findingown data

    GPT agents train earlier and end their searches sooner.

    “GPT agents train earlier and end their searches sooner.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In Opus 5's Terminal-Bench run, successive tests expose shell-lifecycle, truncation, and looping problems, which lead to targeted harness fixes.

    “In Opus 5's Terminal-Bench run, successive tests expose shell-lifecycle, truncation, and looping problems, which lead to targeted harness fixes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Sol 6 submits without a completed development evaluation on SWE and Terminal-Bench and has the lowest RSI-Index (0.2079); Sol 6.1 improves to 0.3321 while retaining short sessions.

    “Sol 6 submits without a completed development evaluation on SWE and Terminal-Bench and has the lowest RSI-Index (0.2079); Sol 6.1 improves to 0.3321 while retaining short sessions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The submitted DeepSeek system scores 0.5333 officially, the highest SWE result.

    “The submitted system scores 0.5333 officially, the highest SWE result.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Among the four leading agents, higher spend does not mean a higher index: Opus 5 leads while spending less than Opus 5.5, and DeepSeek exceeds Astra with lower spend but about twice the elapsed time.

    “Among the four leading agents, higher spend does not mean a higher index: Opus 5 leads while spending less than Opus 5.5, and DeepSeek exceeds Astra with lower spend but about twice the elapsed time.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.
    • Rests on an assumption: The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.

  • May combine two claims.

    Findingown data

    The more heavily trained candidate scores lower in eight of ten comparisons; the other two gains are one task each.

    “The more heavily trained candidate scores lower in eight of ten comparisons; the other two gains are one task each.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Tasks solved on all three attempts increase from 15 to 26, and tasks solved at least once from 41 to 48.

    “Tasks solved on all three attempts increase from 15 to 26, and tasks solved at least once from 41 to 48 (Figure 6b).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On its own development evaluation, the five evaluated checkpoints that merge reasoning into the content score only 0.034-0.056, while the data grows from 45 to 1,691 records without moving the score.

    “On its own development evaluation, the five evaluated checkpoints that merge reasoning into the content score only 0.034-0.056,while the data grows from 45 to 1,691 records without moving the score (Figure 6a).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The development evaluations on small task sets indicate the direction of official performance.

  • May combine two claims.

    Findingown data

    In the $1,000 reruns of Opus 5 and Astra on SWE and Terminal-Bench, three of the four pairs improve, two of them by under 0.03.

    “We rerun Opus 5 and Astra on SWE and Terminal-Bench with $1,000 (Figure 3, bottom).Three of the four pairs improve, two of them by under 0.03: Astra's gains are 7 of 300 and 5 of 267 attempts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The selected system scores 0.5132 in development evaluation with two attempts per task and 0.5626 in official evaluation with three attempts.

    “It scores 0.5132 in development evaluation with two attempts per task ('dev.')and 0.5626 in official evaluation with three attempts ('official', red star).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The initial Qwen3.8-27B model weights score 0.5816 on six SkillsBench tasks; fine-tuning on 472 examples raises the score to 0.7500, while training on 856 examples yields 0.5833.

    “The initial model weights score 0.5816; fine-tuning on 472 examples raises the score to 0.7500, while training on 856 examples yields 0.5833.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Opus 5 on Terminal-Bench in the $1,000 rerun spends $868.65 instead of $440.11 and scores 0.0599 lower.

    “Opus 5 on Terminal-Bench spends $868.65 instead of $440.11 and scores 0.0599 lower.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    By the time the run is stopped, the agent has made 15 successful requests through this channel, about 102k tokens or $0.06 at list prices, and has not submitted a system; the run is excluded from all reported results.

    “By then the agent has made 15 successful requests through this channel, about 102k tokens or $0.06 at list prices, and has not submitted a system; the run is excluded from all reported results.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Astra rapidly builds a simple shell loop and synthetic training pipeline on SWE, reaching 0.3567 in 1.67 research hours, but remains below Claude and DeepSeek.

    “Astra rapidly builds a simple shell loop and synthetic training pipeline on SWE, reaching 0.3567 in 1.67 research hours, but remains below Claude and DeepSeek.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Opus 5.5 favors harness changes and leads GPQA with a checkpoint close to base weights.

    “Opus 5.5 similarly favors harness changes and leads GPQA with a checkpoint close to base weights.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    RSIGym supports studying both the improvements agents produce and the research decisions that lead to them.

    “RSIGym thus supports studying both the improvements agents produce and the research decisions that lead to them.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The agent has seen these tasks during development, so scores are not held-out.

    “The agent has seen these tasks during development, so scores are not held-out.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Because all experiments share the same services, the scope of an experiment is largely defined by its permissions.

    “Because all experiments share the same services, the scope of an experiment is largely defined by its permissions.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Studying one component alone or several together therefore requires no change to the environment, only to the permissions.

    “Studying one component alone or several together therefore requires no change to the environment, only to the permissions.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    The recorded trajectories connect these outcomes to the agents' experiments, showing how they diagnose failures, compare candidates, and select final systems.

    “The recorded trajectories connect these outcomes to the agents' experiments, showing how they diagnose failures, compare candidates, and select final systems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Every research agent works with the same infrastructure, so differences between agents reflect their research decisions rather than how well they configure an environment.

    “Every research agent also works with the same infrastructure, so differences between agents reflect their research decisions rather than how well they configure an environment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Every configuration is a single run.

    “Every configuration is a single run.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The services keep their own record of the work they execute, independent of what the agent reports.

    “The services keep their own record of the work they execute, independent of what the agent reports.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Our post-run audits check 13-gram overlap with evaluation tasks, checkpoint provenance, and the trajectory and submission through an LLM judge.

    “Our post-run audits check 13-gram overlap with evaluation tasks, checkpoint provenance, and the trajectory and submission through an LLM judge.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The LLM judge used for post-run audits reliably assesses submission validity.
    • Rests on an assumption: The 13-gram overlap screen is sufficient to detect contamination between training data and evaluation tasks.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Training examples must be produced by the agent during the run, and the checkpoint must not be trained on evaluation tasks or data derived from them.

    “Training examples must be produced by the agent during the run, and the checkpoint must not be trained on evaluation tasks or data derived from them.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The LLM judge used for post-run audits reliably assesses submission validity.
    • Rests on an assumption: The 13-gram overlap screen is sufficient to detect contamination between training data and evaluation tasks.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    These records allow a run to be audited after the fact.

    “These records allow a run to be audited after the fact.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    These results show that the environment supports both successful improvements and the study of unsuccessful research attempts.

    “These results show that the environment supports both successful improvements and the study of unsuccessful research attempts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The agent traces the low scores to its data format, which merged the model's reasoning into the message content, so the checkpoints learned not to reason.

    “The agent traces this to its data format, which merged the model's reasoning into the message content, so the checkpoints learned not to reason.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The Opus runs place sustained emphasis on diagnosing execution failures, revising the harness, and limiting weight updates that degrade performance, a pattern consistent with their leading overall results.

    “Taken together, the Opus runs place sustained emphasis on diagnosing execution failures, revising the harness, and limiting weight updates that degrade performance, a pattern consistent with their leading overall results.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    DeepSeek shows another effective route: combining low-cost data generation with diagnosis of training and tool-interface failures.

    “DeepSeek shows another effective route: combining low-cost data generation with diagnosis of training and tool-interface failures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    These runs suggest that quickly implementing a workable approach can yield gains while leaving further diagnosis and refinement unexplored.

    “These runs suggest that quickly implementing a workable approach can yield gains while leaving further diagnosis and refinement unexplored.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The unchanged balance confirms that these requests bypass both the platform-service budget and the rollout service.

    “The unchanged balance confirms that these requests bypass both the platform-service budget and the rollout service, through which Opus 5 is the only permitted teacher in this experiment.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    DeepSeek combines inexpensive data generation with diagnosis of training failures.

    “DeepSeek combines inexpensive data generation with diagnosis of training failures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The six-task and full-task scores use different task sets, and the two full-task evaluations also use different numbers of attempts.

    “The six-task and full-task scores use different task sets, and the two full-task evaluations also use different numbers of attempts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    All runs end voluntarily, so the shorter searches are not imposed by a timeout.

    “All runs end voluntarily, so the shorter searches are not imposed by a timeout.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The official comparison therefore shows no overall gain, despite the improvement observed on the six-task probe.

    “The official comparison therefore shows no overall gain, despite the improvement observed on the six-task probe.”
    Quote in the paperSubstantial claim
52 of 52

  • The paper relies on this without support.

    The RSI-Index (mean fraction of the remaining performance gap closed across the five benchmarks) is a valid measure of target-system improvement for comparing research agents.

    “We define RSI-Index as the mean fraction of the remaining performance gap closed across five benchmarks covering software engineering, terminal interaction, mathematics, scientific reasoning, and skill-based tasks.”

  • The paper relies on this without support.

    The five benchmarks (SWE-bench Verified, Terminal-Bench 2.0, AIME, GPQA Diamond, and SkillsBench) adequately cover the domains relevant to recursive self-improvement.

    “We use five benchmarks (Table 4): SWE-bench Verifiedfoot_3 [Jimenez et al., 2024], Terminal-Bench 2.0 (TB2),foot_4 AIME 2024 and 2025 [Maxwell-Jia, 2024, Zhang and Math-AI Team, 2025], GPQA Diamond [Rein et al., 2024], and SkillsBench [Li et al., 2026b].”

  • The paper relies on this without support.

    The LLM judge used for post-run audits reliably assesses submission validity.

    “Our post-run audits check 13-gram overlap with evaluation tasks, checkpoint provenance, and the trajectory and submission through an LLM judge.”

  • The paper relies on this without support.

    The 13-gram overlap screen is sufficient to detect contamination between training data and evaluation tasks.

    “Our post-run audits check 13-gram overlap with evaluation tasks, checkpoint provenance, and the trajectory and submission through an LLM judge.”

  • The paper relies on this without support.

    The RSI-Index is a valid aggregate measure of improvement across domains.

    “RSI-Index measures improvement across domains: under a $500 budget per benchmark run, all six research agents achieve positive overall gains, led by Opus 5 at 0.4809.”

  • The paper relies on this without support.

    The monitoring agent's reports accurately identify anomalous behavior.

    “During the experiments, a separate monitoring agent follows each active run, reading the research agent's trajectory in real time together with the run's activity on every service, including its training runs, evaluation jobs, and spend, and reports anomalous behavior to us so that we can intervene.”

  • The paper relies on this without support.

    The $500 budget per benchmark run is sufficient to allow meaningful research and improvement.

    “RSI-Index measures improvement across domains: under a $500 budget per benchmark run, all six research agents achieve positive overall gains, led by Opus 5 at 0.4809.”

  • The paper relies on this without support.

    The initial system scores are appropriate baselines for measuring improvement.

    “Without network access, the Data run passes the audits and still does not beat the initial system.”

  • The paper relies on this without support.

    The development evaluations on small task sets indicate the direction of official performance.

    “On its own development evaluation, the five evaluated checkpoints that merge reasoning into the content score only 0.034-0.056,while the data grows from 45 to 1,691 records without moving the score (Figure 6a).”
9 of 9

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting counts of checkpoints with merged reasoning

    Minor

    The case-study text says five evaluated checkpoints merged reasoning into message content, but the Figure 6 caption identifies v2-v7-six checkpoints-as doing so and says all seven checkpoints were evaluated.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.

Cited by

Sorted by citations

Authors
Meng, F., Du, L., Lu, H., Chen, Q., Zhao, Z., Wu, Z., Zhuo, J., Hu, M., & Shieh, M. Q.
Affiliations
National University of Singapore (SG)
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2610.10310
arXiv
2610.10310
OpenAlex
W7221061354
Semantic Scholar
86050994427192b4dd9ae98bdb2bfe9581e95b77
Cited by
0
References
22
Type
posted content
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
recursive self-improvement, everything as a service, joint optimization, SWE-bench Verified, AIME, budget allocation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-24.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-24.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review