CoChat
Sign Up
PreprintCited by 16Software Engineering ResearchAI-based Problem Solving and PlanningEvolutionary Algorithms and Applications

A Self-Improving Coding Agent

Open access
Robeyns, Maxime, Szummer, Martin, Aitchison, Laurence·2025·arXiv
DOI 10.48550/arxiv.2504.15228arXiv 2504.15228
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advancements in Large Language Models (LLMs) have spurred interest in deploying LLM agents to undertake tasks in the world. LLMs are often deployed in agent systems: code that orchestrates LLM calls and provides them with tools. We demonstrate that an agent system, equipped with basic coding tools, can autonomously edit itself, and thereby improve its performance on benchmark tasks. We find performance gains from 17% to 53% on a random subset of SWE Bench Verified, with additional performance gains on LiveCodeBench, as well as synthetically generated agent benchmarks. Our work represents an advancement in the automated and open-ended design of agentic systems, and demonstrates a data-efficient, non gradient-based learning mechanism driven by LLM reflection and code updates.

AI snapshot

Written from the paper’s full text.

TL;DR

SICA is a self-improving coding agent that can edit its own codebase without separating a meta-agent from a target agent. It improves benchmark performance, notably from 17% to 53% on a random subset of SWE Bench Verified, with smaller gains on LiveCodeBench and reasoning benchmarks.

Key findings

  • SICA improved from 17% to 53% on a random subset of SWE Bench Verified (Table 1: iteration 0 = 0.17, iteration 14 = 0.53). It also reports gains on LiveCodeBench (0.65 to 0.71) and synthetic benchmarks: file editing reached 0.96 at iteration 6, and symbol navigation reached 0.43 at iteration 11.
  • The agent autonomously implemented improvements such as self-termed “smart” editing tools based on diffs and ranges, and an AST-based symbol locator at iteration 9, which yielded improvements also reflected in other tasks.
  • On reasoning-heavy AIME 2024 and GPQA Diamond, improvements were less pronounced: o3-mini alone scored 87% and 79% with high reasoning effort, while the agent system averaged 76% across the two benchmarks.
  • Safety analysis states that SICA does not update model weights; if weights were updated, risks of recursively self-improving systems would be higher, and observability mechanisms alone would be inadequate.

Methods

  • SICA runs a loop with an archive of previous agents and benchmark results; the best-performing agent is selected as the meta-agent, instructed to inspect the archive, identify an improvement, and implement it. New agents are evaluated, and “best” is defined by utility \(U = 0.5 p_{\text{score}} + 0.25(1-\min(1, p_{\text{cost}}/\$10)) + 0.25(1-\min(1, p_{\text{time}}/300\text{s}))\), with a timeout penalty \(\tau = 0.5\).
  • The initial agent is a Python coding agent with tools to open/close/overwrite files, execute shell commands, use a calculator, submit answers, return/exit from sub-agents, and analyze the archive. It includes coding, problem-solver, and reasoning sub-agents, plus an asynchronous LLM overseer.
  • Benchmarks include a fixed random subset of 50 SWE Bench Verified questions, 50 random LiveCodeBench questions, and two synthetic benchmarks (file editing and codebase navigation). Most agents use Sonnet 3.5 (v2), with o3 mini for the reasoning agent; a separate experiment uses AIME 2024 and GPQA Diamond.

Limitations

  • The paper states that self-improvement is purely scaffolding-based and does not modify model weights; limits are seen in Figure 4, and if weights were updated, risks would be higher and observability alone would be inadequate.
  • The paper states that reasoning tasks show less improvement; the main agent often merely delegated to the o3-mini reasoning agent and did not leverage the full agent system, and crude reasoning components may interrupt a reasoning model’s chain of thought.
  • The paper attributes the low symbol locator score to poor data quality (some target symbols point to uninstalled external libraries). Evident from the text: main results come from fixed random subsets of 50 questions and a single self-improvement run per benchmark set, with no repeated seeds or variance reported for those main results.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.15228

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

3 of 17 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisereasoning

    Gödel Agent was evaluated on language understanding and mathematical benchmarks rather than coding benchmarks.

    “And as such, as with ADAS, it was evaluated on language understanding and mathematical benchmarks (DROP Dua et al., 2019, MGSM Shi et al., 2022, MMLU Hendrycks et al., 2020and GPQA Rein et al., 2023), rather than coding benchmarks.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Dua et al., 2019 source
    • cites Shi et al., 2022 source
    • cites Hendrycks et al., 2020 source
  • Findingnone given

    SICA eliminates the distinction between meta-agent and target agent.

    “A self-improving coding agent (SICA) that eliminates the distinction between meta-agent and target agent, and is capable of editing its own codebase to improve itself with respect to its cost, speed and benchmark performance.”
    Quote in the paperSubstantial claim
  • Findingown data

    Performance improves from 17% to 53% on a random subset of SWE Bench Verified even with consideration given to safety constraints and resource efficiency.

    “Empirical evidence that self-referential agents can effectively improve their own implementations; we find performance improves from 17% to 53% performance on a random subset of SWE Bench Verified, even with consideration given to safety constraints and resource efficiency.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The fixed random subset of 50 SWE Bench Verified questions is representative of the full SWE Bench Verified benchmark.
  • Findingown data

    Additional performance gains were found on LiveCodeBench.

    “We find performance gains from 17% to 53% on a random subset of SWE Bench Verified, with additional performance gains on LiveCodeBench, as well as synthetically generated agent benchmarks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 50 random questions from LiveCodeBench are representative of the full LiveCodeBench benchmark.
  • Findingown data

    The LLM coding agent improves its performance on benchmark tasks by autonomously editing itself.

    “We demonstrate that an LLM coding agent, equipped with basic coding tools, can autonomously edit itself, and thereby improve its performance on benchmark tasks.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 19
  • May combine two claims. 17
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Agent systems often show dramatic improvements in benchmark performance over plain LLMs through combinations of prompting strategies and methods for combining different LLM outputs.

    “These agent systems often show dramatic improvements in benchmark performance over”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Yao et al., 2023b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zelikman et al., 2023 source

      The Abstract (p. 1) describes Parsel combining LLM-generated function implementations using tests and reports substantially higher benchmark pass rates than direct LLM sampling. The Introduction (p. 1) also describes few-shot prompting and improvement over GPT-4 alone. These passages support Parsel as an example, but this paper does not establish that agent systems *often* achieve such improvements as a class. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Chen et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingnone given

    SICA is fully self-improving, with no distinction between the meta and target agent.

    “In contrast, our self-improving coding agent is fully self-improving (i.e.there is no distinction between the meta and target agent), and it operates over the agent's full codebase.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The agent implemented a SmartEditor tool to intelligently select and execute the optimal file editing strategy based on edit characteristics.

    “1. Smart Editor: Implemented a new SmartEditor tool to intelligently select and execute the optimal file editing strategy based on edit characteristics.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The LLM function calling interface allows the LLM to invoke tools and sub-agents at run-time without pre-defined control flow.

    “The LLM function calling interface allows the LLM to invoke tools and sub-agents at run-time without pre-defined control flow.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    The meta-improvement task is challenging, and the main agent may take on the order of one hour to finish one meta-improvement iteration.

    “The meta-improvement task is challenging, and the main agent may take on the order of one hour to finish one metaimprovement iteration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingown data

    In a characteristic meta-improvement run, the agent starts by invoking the 'archive explorer' sub-agent, which compares iterations and identifies best and worst performing benchmark problems, then invokes reasoning agents, and returns a recommendation to the main agent.

    “The above shows a characteristic meta-improvement run: the agent starts by invoking the 'archive explorer' sub-agent, which has access to a number of tools to facilitate archive analysis.”
    • Also stated in: § Additional information · plain
      “The archive explorer compares different iterations, looks at the best performing benchmark problems in a given iterations to understand success cases, as well as the worst performing benchmark problems at a given iteration to understand the failure modes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The benchmark accuracy numbers in Table 1 correspond to a synthetic file editing benchmark, a symbol location (codebase navigation) benchmark, SWE Bench Verified, and LiveCodeBench, respectively.

    “Table 1 : Performance per iteration: the benchmark accuracy numbers correspond to a synthetic file editing benchmark and symbol location (codebase navigation) benchmark, SWE Bench Verified, and LiveCodeBench, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The agent implemented a Quick Overwrite Tool to address performance issues with file editing operations to improve on full file overwrites, reducing token usage.

    “2. Quick Overwrite Tool: to address performance issues with file editing operations to improve on full file overwrites; reducing token usage.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the file editing benchmark, the agent initially only had the ability to overwrite files, which is slow and token inefficient, and it quickly implemented 'smart' editing tools based on diffs and ranges.

    “For the file editing benchmark, the agent is initially only equipped with the ability to overwrite files which is slow and token inefficient, and it quickly implemented self-termed 'smart' editing tools based on diffs and ranges.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    For the symbol navigation task, the agent implemented a simple AST based symbol locator at iteration 9.

    “For the symbol navigation task, the agent implemented a simple AST based symbol locator at iteration 9 (perhaps exploiting the fact that our benchmarks were predominantly Python based), which yielded good improvements also reflected in other tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The main agent then invokes a software developer sub-agent to implement the feature, which after debugging gets tests to pass and documents the change, and then a second software developer sub-agent independently verifies the implementation.

    “The main agent then invokes the 'software developer' sub-agent with the instruction to implement the feature.The coding sub-agent first looks at relevant files and documentation, implement the feature, and then enters into a debugging loop as it fixes the bugs in its implementation.About 3 minutes later, it successfully gets all the tests (that it wrote) to pass, documents the change in the agent”
    • Also stated in: § Additional information · plain
      “The main agent then invokes a second software developer sub-agent to independently verify the implementation of the feature, which passes after an initial issue, and the agent returns a description of the newly functioning tool to the main agent which then exits.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The LiveCodeBench scores saw a subtle improvement, although not as pronounced as some of the other benchmarks.

    “The LiveCodeBench scores saw a subtle improvement, although not as pronounced as some of the other benchmarks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 50 random questions from LiveCodeBench are representative of the full LiveCodeBench benchmark.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The metrics in Table 1 are averaged across all problem instances across all benchmarks.

    “The metrics are averaged across all problem instances across all benchmarks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The agent implemented a Diff-Enhanced Smart Editor that adds intelligent diff-based strategy selection and improved pattern-based editing.

    “3. Diff-Enhanced Smart Editor: adds intelligent diff-based strategy selection and improved pattern-based editing.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The results on reasoning-heavy domains show less improvement and highlight an important interplay between the base models and the scaffolding system.

    “The results here show less improvement, and highlight an important interplay between the base models and the scaffolding system.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    ADAS uses a meta-agent to optimise agent implementations.

    “Recent work in the Automated Design of Agentic Systems (ADAS) (Hu et al., 2024) uses a meta-agent to optimise agent implementations.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    ADAS used a target-agent which performs the actual task, and a meta-agent which improves the target agent.

    “ADAS used a target-agent which performs the actual task, and a meta-agent which improves the targetagent.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The overseer's broad representation of the system's behaviour allows it to pick out patterns and detect repeated work completed by previous agent calls.

    “This broad representation of the system's behaviour, allows the overseer to to pick out patterns, as well as detect repeated work completed by previous agent calls.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The LLM context is laid out to allow new content to be appended, preserving the KV cache to the benefit of latency between tool calls and cost.

    “The LLM context is laid out in this way to allow new content to be appended, hence preserving the KV cache to the benefit of latency between tool calls and cost.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    ADAS has a fixed meta-agent, so it does not select the meta-agent from the archive.

    “Note that this is different from ADAS, in that ADAS has a fixed meta-agent, so does not select the meta-agent from the archive (the archive in ADAS is an archive of target-agents, not meta-agents).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hu et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductreasoning

    SICA has sophisticated yet user-friendly observability mechanisms, including an interactive web interface and an asynchronous LLM-based overseer.

    “We have thus architected SICA with sophisticated yet user-friendly observability mechanisms, which provide rich insight into the chain-of-thought, actions and sub-agent invocations at each step.These include an interactive web interface for easy oversight of all events occurring in the agent system, as well as the asynchronous LLM-based overseer, which may be prompted to treat various behaviors as”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    The asynchronous overseer enables extended, autonomous operation while remaining on-task, avoiding loops or other behaviours that frustrate progress.

    “Finally, one important component that enables extended, autonomous operation while remaining on-task, avoiding loops or other behaviours that frustrate progress is the asynchronous overseer.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The authors use these mechanisms to oversee the system at every step.

    “We use these mechanisms to oversee the system at every step.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Zelikman et al.'s system self-improves and evaluates only on algorithmic tasks such as learning parity with noise, String Grid Distance, and 3-SAT.

    “Again, this is not a general-purpose coding agent with the ability to e.g.work on arbitrary software engineering tasks, and as such, it self-improves and evaluates and only on algorithmic tasks such as learning parity with noise (Blum et al., 2003), String Grid Distance, and 3-SAT.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Blum et al., 2003 source

  • The quote only partly states this claim.

    Interpretationcitation

    Given the lack of model weight updates, the observability mechanisms provide strong safety mitigation.

    “Given the lack of model weight updates, which make it difficult but not impossible for the model to encode e.g.stenographic messages (Motwani et al., 2023;2024) in the chain-of-thought, we believe this provides strong safety mitigation.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Motwani et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Motwani et al., 2024 source

  • Classified as probably not a substantial claim.

    Premisenone given

    XML as a tool calling format does not require escaping string literals, unlike JSON.

    “We choose XML over JSON as the tool calling format since it does not require escaping string literals, which can become burdensome when including file content verbatim as an argument.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The second opportunity for safety mitigation is inherent to the iterative, evaluation-based update procedure.

    “The second opportunity for safety mitigation is inherent to the iterative, evaluation-based update procedure.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The low score in the symbol locator benchmark is attributed to poor data quality, with some target symbols pointing to uninstalled external libraries.

    “Note that we attribute the low score in the symbol locator benchmark to poor data quality-some of the target symbols point to uninstalled external libraries for instance.”
    Quote in the paperNot a substantial claim
28 of 28

  • The paper relies on this without support.

    The fixed random subset of 50 SWE Bench Verified questions is representative of the full SWE Bench Verified benchmark.

    “we select a fixed random subset of 50 questions”
    “performance improves from 17% to 53% performance on a random subset of SWE Bench Verified”

  • The paper relies on this without support.

    The 50 random questions from LiveCodeBench are representative of the full LiveCodeBench benchmark.

    “We also include 50 random questions from LiveCodeBench (Jain et al., 2024), the questions from which resemble competitive programming and often include more theoretical reasoning.”

  • The paper relies on this without support.

    The synthetic benchmark tasks are valid measures of agentic coding performance.

    “We also include two example 'synthetic' benchmark tasks, defined and curated entirely within the agent's codebase.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior work on self-evolving agents and tool creation does not specifically target real-world software engineering problems that require long-horizon evolution across multiple commits and versions.

    “While these works demonstrate the potential of self-evolving agents and tool creation, they do not specifically target real-world software engineering problems that require long-horizon evolution across multiple commits and versions.”

    From SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios (2025) · cites this paper as Robeyns et al., 2025

    The Introduction describes SICA as a self-improving coding agent evaluated on SWE Bench Verified, and the Experiments and Results section explicitly calls its benchmark tasks “relatively long-horizon.” That contradicts the claim that this cited work does not target long-horizon software-engineering problems. The paper does not establish that its tasks require evolution across multiple commits and versions. (LLM, conf=0.82)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Both DGM and SICA, explicitly or implicitly, assume that higher software benchmark scores correspond to greater self-improvement capacity.

    “Both DGM and SICA, explicitly or implicitly, assume that higher software benchmark scores correspond to greater self-improvement capacity.”

    From Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine (2025) · cites this paper as Robeyns et al., 2025

    In Methods (§3; page not provided), SICA selects its next meta-agent from the best-performing agents, using a utility function that includes benchmark scores. This supports an implicit link between measured performance and the agent chosen to pursue further improvements, but the utility also includes cost and time. Our copy does not establish the stated assumption for DGM or directly show that higher benchmark scores mean greater self-improvement capacity. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Self-evolving agents rely heavily on offline evolution, where improvements are learned on certain benchmarks and baked into a static agent, and may not generalize well beyond that.

    “they rely heavily on offline evolution, where improvements are typically learned on certain benchmarks and then baked into a static agent. In this way, the learned agents may become specialized to the given benchmarks and underlying LLMs, and may not generalize well beyond that.”

    From Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly? (2025) · cites this paper as [30]

    The Methods section describes SICA and ADAS evaluating successive agents on a set of benchmarks, which supports the benchmark-driven improvement part of the claim. The Experiments and Results section notes that one improvement may have exploited predominantly Python-based benchmarks. Neither passage establishes that self-evolving agents generally rely heavily on offline evolution, bake improvements into static agents, or fail to generalize across benchmarks or underlying LLMs. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Prior self-evolving techniques produce a static agent used for all problems.

    “prior self-evolving techniques produce a static agent used for all problems.”

    From Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly? (2025) · cites this paper as [30]

    The Methods section says ADAS uses a fixed meta-agent and evaluates new target agents on a set of benchmarks. That supports a narrower point about one prior technique, but it does not establish that prior self-evolving techniques generally produce a static agent used for all problems. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Such self-improving agents require costly offline training on known benchmarks, and may not generalize well across different LLMs, benchmarks, and issue types.

    “However, such self-improving agents require costly offline training on known benchmarks, and may not be generalize well across different LLMs, benchmarks, and issue types.”

    From Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly? (2025) · cites this paper as [30]

    The Methods section describes iterative evaluation on a set of benchmarks, and §4.1 reports smaller gains on reasoning tasks and an interplay between the base models and the agent framework. These passages support a qualified concern about performance across tasks and models, but not the claim that such agents require costly offline training: Methods explicitly says this agent performs no model-weight updates. The paper does not establish poor generalization across issue types. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • LLM context diagram cited as the wrong figure

    Minor

    The text points to Figure 3 for the initial agent's context structure, but the figure captions identify Figure 2 as the LLM context-window diagram. Figure 3 is captioned as performance across iterations.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Sumeet Ramesh Motwani, Mikhail Baranchuk, Lewis Hammond, and Christian Schroeder de Witt. A perfect collusion benchmark: How can ai agents be prevented from colluding with information- theoretic undetectability? In Multi-Agent Security Workshop@ NeurIPS'23, 2023.
  2. Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive Introspection: Teaching Language Model Agents How to Self-Improve. In The Thirty-eighth An- nual Conference on Neural Information Processing Systems, November 2024. URL https://openreview.net/forum?id=DRC9pZwBwR&referrer=%5Bthe+ profile+of+Yuxiao+Qu%5D%28%2Fprofile%3Fid%3D~Yuxiao_Qu1%29.
  3. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations, February 2023b. URL https://openreview. net/forum?id=WE_vluYUL-X.
  4. Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation, August 2024. URL http://arxiv. org/abs/2310.02304. 1 main [agent_b8845adf] (134.4s | 95752 tokens (cached 0.57%) | $0.185 | Success) [Stats] Events: 1 tool calls, 6 messages [Assistant] t+0.1s | "I'll orchestrate the solution to this problem foll..." 1.3 reasoning_agent [agent_20fccbc1] (12.0s | 5673 tokens (cached 0.00%) | $0.013 | Success) [Stats] Events: 1 messages [Assistant] t+9.8s | "Below is an explanation that addresses the three p..."

Cited by

Sorted by citations

Authors
Robeyns, M., Szummer, M., & Aitchison, L.
Affiliations
University of Bristol (GB)
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2504.15228
arXiv
2504.15228
OpenAlex
W6948264221
Cited by
1
References
34
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
Keywords
LLM agents, self-improving agents, self-editing, code generation, SWE-bench, LiveCodeBench, data-efficient learning, agentic systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review