Does Failure Feedback Direct Agent Self-Modification?

TL;DR

The paper tests whether failure feedback directs self-modification by randomizing the failure shown to an editor and measuring target repair, category spread, and net score change. Across Darwin Gödel Machine logs it finds no detectable target-specific advantage, while controlled instruction-revision studies find directed repair for some models but held-out gains mainly where failures have identifiable, repairable mechanisms.

Key findings

• In 145 eligible DGM modifications, 12 repaired their target versus 16.4 expected under the null; target repair was 8.3% versus 11.6% for the parent’s other failures (difference −3.3 percentage points, 95% CI −8.2 to +2.0), pooled odds ratio 0.69 (0.31–1.18), e-value 0.15, and martingale z = −1.23. No run showed a significant directed effect, and five of six pointed the other way.
• In controlled study 1, clean-playbook instance children repaired the shown question 51.9% of the time versus 27.4% for other failures (+24.5 points, 95% CI +6.5 to +42.6, p = 0.007), but their net score changed by −19.0 points; planted-rule children repaired the target 60.7% versus 24.4% (+36.4 points, p < 10⁻⁴) and gained +5.4 points.
• In study 2, holding the target fixed, Qwen3-4B repaired the target 20.0 points more often when its own failure was shown than a matched same-category failure (95% CI +8.5 to +31.8; Holm-adjusted p = 0.003); Phi-4-mini showed +6.7 points, not significant. Held-out gains were concentrated in settings with identifiable, repairable failure mechanisms, including planted errors and natural tool-interface failures.
• Eight-step revision loops and GEPA ended at similar mean held-out accuracy on the planted-rule parent: +14.5 points for the whole-rewrite loop, +13.3 for the localized loop, and +14.4 for GEPA. Self-editing yielded working agents in 49% of SWE-bench attempts versus 33% for the fixed editor, and 75% versus 23% on Polyglot.

Methods

• DGM analysis: six public runs, per-modification randomized target selection, directedness \(d\), category-level spread \(c\), and net score change \(\Delta\); inference uses an e-value/test martingale and auxiliary statistics, with 145 eligible modifications.
• Study 1: Qwen3-4B-Instruct-2507 (4-bit, MLX) as agent and editor, 180 BIG-Bench Hard questions from six categories, clean versus planted-rule playbooks, random evidence conditions, and all viable children plus both parents scored on all 180 questions.
• Study 2: pre-specified analysis plan; Qwen3-4B and Phi-4-mini (bf16, vLLM); Setting A uses a seven-module instruction playbook with D/V/T splits, Setting B uses a tool-using agent with an editable helper library; factorial feedback × edit scope, plus localization, regression screening, eight-step loops, and GEPA.

Limitations

• Stated: the directedness test assumes the randomized draw and a null under which the target affects neither viability nor the evaluation set; only the e-value is valid without asymptotic regularity conditions. Human-named goal comparisons condition on survival, and the all-attempt analysis rests on 25 and 34 goal attempts. A diagnosis could identify the right cause while the edit remains necessary but not sufficient. The collateral-damage comparison between studies 1 and 2 is not randomized.
• Stated: each DGM agent is evaluated once, so the noise bound rests on the no-archive run and comparable noise across runs, and most Polyglot children were evaluated on 10 tasks. In study 1, scores depend on strict answer extraction and a 512-token limit; sensitivity analyses are approximations, and diagnosis coding has a single coder. In study 2, generation is not bitwise reproducible, each child is scored once, and prompt-perturbation sensitivity reaches up to 4 points for Phi.
• Stated: external validity is limited by one DGM system with one pair of foundation models and 80 iterations; study 1 uses one 4-billion-parameter model, six categories, and artificial planted rules; study 2 adds a second similar-size model, a single planted cause in its instruction setting, a synthetic tool environment, and oracle localization. Evident from the text: clean and planted parents also differ in accuracy, error composition, and headroom; the presence of a shared cause was not manipulated independently; and the studies differ in model, tasks, editable interface, and path from failure to edit.