Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

Classified as probably not a substantial claim. May combine two claims.

Building a software system from scratch requires agents to translate high-level requirements into executable plans, coordinate interdependent tasks, design and integrate components, and continuously test and debug the evolving system.

“Building a software system from scratch requires agents not only to generate code snippets, but also to translate high-level requirements into executable plans, coordinate interdependent tasks, design and integrate components, and continuously test and debug the evolving system [8, 32, 36].”

• cites [8] [8] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations , pages 23247-23275, 2024. https://proceedings.iclr.cc/ paper\_files/paper/2024/file/6507b115562bb0a305f1958ccc87355a-Paper-Conference.pdf . - sourceSupported
• cites [32] [32] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 15174-15186. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.810. https://aclanthology.org/2024.acl-long.810/ . - sourceSupported
• cites [36] [36] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Daniel Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations , pages 65882-65919, 2025. https://proceedings.iclr. cc/paper\_files/paper/2025/file/a4b6ad6b48850c0c331d1259fc66a69c-Paper-Conference.pdf . - sourcePartly supported

  The Introduction (p. 1) describes the challenges of having agents create and modify complex software and gather information to debug problems. Section 2.4 (p. 5) describes delegation of subtasks between agents. But the paper does not establish that building a system from scratch requires agents to turn requirements into plans, design and integrate components, or continuously test the evolving system; its integration-testing discussion (Appendix E, p. 19) concerns testing agents during OpenHands development. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

May combine two claims.

Failed attempts and suboptimal decisions may accumulate, while new evidence from testing can invalidate earlier assumptions.

“Failed attempts and suboptimal decisions may accumulate, while new evidence from testing can invalidate earlier assumptions [5, 24, 33].”

• cites [5] [5] Mengzhuo Chen, Junjie Wang, Zhe Liu, Yawen Wang, Haiming Zheng, and Qing Wang. From failed trajectories to reliable LLM agents: Diagnosing and repairing harness flaws, 2026. https://arxiv.org/ abs/2606.06324 . - sourcePartly supported

  The Pre-validation Checks table describes repeated actions, bad retries, and stale or polluted prior results. The End-to-end Example shows an evaluator finding that no payment requests were created despite an earlier report of successful execution. These passages support the possibility of recurring failures and testing overturning an apparent success, but they do not establish that failed attempts and suboptimal decisions accumulate generally. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [24] [24] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems , volume36, pages 46534-46594, 2023. https://proceedings.neurips.cc/paper\_files/paper/2023/file/ 91edff07232fb1b55a505a9e9f6c0ff3-Paper-Conference.pdf . - sourcePartly supported

  The Analysis section (page unavailable) reports that output quality can decline during an iteration, and the response-generation error analysis says refinements sometimes introduce new problems. These findings support the possibility of suboptimal iterative decisions, but the paper does not say failed attempts accumulate or that testing evidence invalidates earlier assumptions. (LLM, conf=0.69)

  Experimental — accuracy not yet measured
• cites [33] [33] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems , volume 36, pages 8634-8652, 2023. https://proceedings.neurips.cc/paper\_files/paper/ 2023/file/1b44b878bb782e6954cd888628510e90-Paper-Conference.pdf . - sourcePartly supported

  The Related Work section describes a persistent memory of mistakes, and the Programming section (p. 8) says tests can reveal syntax and logic errors after failed attempts. These passages support learning from accumulated failures and test feedback, but do not specifically say that new testing evidence invalidates earlier assumptions. (LLM, conf=0.77)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Modern coding agents operate within a harness that provides tools, manages execution and mediates the LLM's interaction with the development environment.

“Modern coding agents operate within a harness-the surrounding system that provides tools, manages execution and mediates the LLM's interaction with the development environment [36, 39, 45].”

• cites [36] [36] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Daniel Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations , pages 65882-65919, 2025. https://proceedings.iclr. cc/paper\_files/paper/2025/file/a4b6ad6b48850c0c331d1259fc66a69c-Paper-Conference.pdf . - sourceSupported
• cites [39] [39] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-Agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems , volume 37, pages 50528-50652, 2024. doi: 10.52202/079017-1601. https://proceedings.neurips.cc/paper\_files/paper/2024/file/ 5a7c947568c1b1328ccc5230172e1e7c-Paper-Conference.pdf . - sourceSupported
• cites [45] [45] Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. Self-Harness: Harnesses that improve themselves, 2026. https://arxiv.org/abs/2606.09498 . - sourceSupported

May combine two claims.

Long trajectories can lead to repetitive cycles of inspection and repair, redundant verification of completed components, or premature declaration of completion despite missing or incorrect functionality.

“Long trajectories can also lead to repetitive cycles of inspection and repair, redundant verification of completed components, or premature declaration of completion despite missing or incorrect functionality [3, 11].”

• cites [3] [3] Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei A. Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? In Advances in Neural Information Processing Systems , volume 38. Curran Associates, Inc., 2025. https://proceedings.neurips.cc/paper\_files/ paper/2025/file/b1041e52d3be19f0a9bc491657488e4a-Paper-Datasets\_and\_Benchmarks\_Track.pdf . - sourceNot checked — no full text of the source
• cites [11] [11] Jek Huang, Jeffery Hsia, Jiayi Sun, Freddie Shi, Wei Huang, and Ian H. White. Proof-or-stop: Don't trust the agent, trust the evidence-loop engineering for verifiable evidence-gated lifecycle control, 2026. https://arxiv.org/abs/2607.14890 . - sourcePartly supported

  The Model section describes an agent shipping an artifact that passes visible tests but fails hidden ground truth, supporting the risk of declaring incorrect work complete. The clean-task pilot also finds that review adds overhead without improving completion on easy tasks. Neither passage establishes that long trajectories cause repetitive inspection-and-repair cycles or redundant verification of completed components. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

May combine two claims.

The preregistered validation with 1,518 players found better fit for the ten-factor model-or the eleven-factor model when Enjoyment is included-than for models with a general player-experience factor or higher-order consequence factors.

“the preregistered validation with 1,518 players found better fit for the ten-factor model-or the eleven-factor model when Enjoyment is included-than for models with a general player-experience factor or higher-order consequence factors [31].”

• cites [31] [31] Sebastian A. C. Perrig, Nicolas Scharowski, Florian Brühlmann, Nick von Felten, Klaus Opwis, and Lena Fanya Aeschbach. Independent validation of the player experience inventory: Findings from a large set of video game players. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , CHI '24, New York, NY, USA, 2024. Association for Computing Machinery. doi: 10.1145/3613904.3642270. https://doi.org/10.1145/3613904.3642270 . - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The validated core of the Player Experience Inventory comprises ten constructs, each measured by three items on the official seven-point scale from -3 to +3.

“The validated core comprises ten constructs, each measured by three items on the official seven-point scale from -3 to + 3.”

• cites [34] [34] Vero Vanden Abeele, Katta Spiel, Lennart E. Nacke, Daniel Johnson, and Kathrin Gerling. Development and validation of the player experience inventory: A scale to measure player experiences at the level of functional and psychosocial consequences. International Journal of Human-Computer Studies , 135: 102370, 2020. doi: 10.1016/j.ijhcs.2019.102370. https://doi.org/10.1016/j.ijhcs.2019.102370 . - sourcePartly supported

  The Method section states that the validated model has ten constructs measured by three items each, and the Confirmatory factor analysis section describes the resulting 30-item model. The passages in our copy do not establish that its official response scale is seven points ranging from -3 to +3. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims. The quote only partly states this claim.

HoH achieves an average relative gain of 52.25% and a maximum gain of 82.86% after three iterations across the three benchmarks.

“achieving an average relative gain of 52.25% and a maximum gain of 82.86% after three iterations.”

• AssumptionRests on an assumption: The benchmark metrics used (Overall score, official rewards, Avg. Test Pass Rate) are valid measures of software artifact quality.

May combine two claims.

In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game with a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio.

“In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio.”

• Also stated in: § 1 Introduction · p. 3 · plain

  “Over multiple days of autonomous development, HoH transforms the initial requirements into a complete, human-playable game with a coherent storyline, fully implemented core mechanics, polished visuals and integrated audio.”

May combine two claims.

On ProgramBench, Avg. Test Pass Rate increases from 60.41 to 66.50 for Codex, from 45.27 to 57.56 for OpenCode, and from 35.83 to 52.68 for Pi.

“On ProgramBench, Avg. Test Pass Rate increases from 60.41 to 66.50 for Codex, from 45.27 to 57.56 for OpenCode, and from 35.83 to 52.68 for Pi.”

• AssumptionRests on an assumption: The benchmark metrics used (Overall score, official rewards, Avg. Test Pass Rate) are valid measures of software artifact quality.

May combine two claims.

The advantage of HoH over Vanilla is not explained by greater token use alone: HoH@2 achieves 64.84 with 5.67M tokens, exceeding the 58.24 obtained by three-pass Vanilla Continuation with 6.33M tokens.

“The advantage is not explained by greater token use alone: HoH@2 achieves 64.84 with 5.67M tokens, exceeding the 58.24 obtained by three-pass Vanilla Continuation with 6.33M tokens.”

May combine two claims.

On GameCraft-Bench, mean Overall scores increase from 49.58 to 71.52 for Codex, from 26.90 to 48.98 for OpenCode, and from 42.16 to 58.78 for Pi.

“On GameCraft-Bench, mean Overall scores increase from 49.58 to 71.52 for Codex, from 26.90 to 48.98 for OpenCode, and from 42.16 to 58.78 for Pi.”

• AssumptionRests on an assumption: The benchmark metrics used (Overall score, official rewards, Avg. Test Pass Rate) are valid measures of software artifact quality.

May combine two claims.

Removing plan updates, excluding execution evidence from replanning, and removing warm-start lowers the score by 8.13, 6.28, and 7.85 points, respectively.

“Removing plan updates, excluding execution evidence from replanning, and removing warm-start lowers the score by 8.13, 6.28, and 7.85 points, respectively.”

May combine two claims.

Codex with GPT-5.5 (high) reaches the highest final GameCraft-Bench score of 71.52 after improving by 21.93 points.

“Codex with GPT-5.5 (high), the strongest Vanilla configuration, reaches the highest final GameCraft-Bench score of 71.52 after improving by 21.93 points.”

May combine two claims.

On GameCraft-Bench, HoH@3 improves all four quality components under every harness-model configuration, with gains of 20.00-25.56 points for Codex, 19.25-34.63 points for OpenCode, and 11.32-25.38 points for Pi.

“HoH@3 improves all four components under every harness-model configuration, with gains of 20.00-25.56 points for Codex, 19.25-34.63 points for OpenCode, and 11.32-25.38 points for Pi.”

May combine two claims.

On ProgramBench, Codex and OpenCode attain their highest Pass Rates at HoH@3, while Pi peaks at HoH@2.

“ProgramBench exhibits a similar overall pattern: Codex and OpenCode attain their highest Pass Rates at HoH@3, while Pi peaks at HoH@2.”

May combine two claims.

Pi with MiniMax-M3 records the largest gain on FrontierSWE, increasing by 0.29 from 0.26 to 0.55, and the largest ProgramBench gain, increasing the average test pass rate by 16.85 points.

“Pi with MiniMax-M3 records the largest gain on FrontierSWE, increasing by 0.29 from 0.26 to 0.55, and the largest ProgramBench gain, increasing the average test pass rate by 16.85 points.”

May combine two claims.

OpenCode with HoH@3 exceeds Codex Vanilla in Action and Simulation on GameCraft-Bench and in Performance on FrontierSWE.

“OpenCode with HoH@3 further exceeds Codex Vanilla in Action and Simulation on GameCraft-Bench and in Performance on FrontierSWE.”

May combine two claims.

In Momentum Lab, Kitchen Rush, and Ant Empire, the Overall scores increase from 34.05 to 70.61, from 42.63 to 73.38, and from 65.52 to 87.88, respectively.

“The corresponding Overall scores increase from 34.05 to 70.61, from 42.63 to 73.38, and from 65.52 to 87.88, respectively.”

Probably not a substantial claim: may not pass S3 (load-bearing).

At the analysis cutoff, the system had completed 70 development loops.

“At the analysis cutoff, the system had completed 70 development loops.”

May combine two claims.

On FrontierSWE, rewards increase from 0.31 to 0.54, from 0.23 to 0.31, and from 0.26 to 0.55, respectively.

“On FrontierSWE, rewards increase from 0.31 to 0.54, from 0.23 to 0.31, and from 0.26 to 0.55, respectively.”

• AssumptionRests on an assumption: The benchmark metrics used (Overall score, official rewards, Avg. Test Pass Rate) are valid measures of software artifact quality.

May combine two claims.

For Codex, Functional Visuals shows the largest increase, from 48.67 to 74.23, while Art and Presentation rises from 45.28 to 65.28.

“For Codex, Functional Visuals shows the largest increase, from 48.67 to 74.23, while Art and Presentation rises from 45.28 to 65.28.”

May combine two claims.

OpenCode with DeepSeek-V4-Pro starts from the lowest Vanilla score on GameCraft-Bench and FrontierSWE, yet HoH@3 raises its scores to 48.98 and 0.31, respectively, while increasing its ProgramBench average test pass rate from 45.27 to 57.56.

“OpenCode with DeepSeek-V4-Pro starts from the lowest Vanilla score on GameCraft-Bench and FrontierSWE, yet HoH@3 raises its scores to 48.98 and 0.31, respectively, while increasing its ProgramBench average test pass rate from 45.27 to 57.56.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

For Codex + GPT-5.5 (high) on GameCraft-Bench, the mean Overall score was 49.58 under Vanilla.

“| Mean | | 49.58 | 59.71 | 64.84 | 71.52 | +21.93 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

May combine two claims.

In the multi-day Fusepoint case, HoH developed a game project over 70 loops, with the versioned workspace, issue history, and evidence packets recording the development trajectory.

“In the multi-day Fusepoint case, HoH developed a game project over 70 loops, with the versioned workspace, issue history, and evidence packets recording the development trajectory.”

May combine two claims. The quote only partly states this claim.

On the 45 GameCraft-Bench tasks in the budget-controlled comparison using Codex with GPT-5.5 (high), the mean cumulative coding-harness tokens per task are 2.59M for Vanilla, 6.33M for three-pass Vanilla Continuation, and 8.41M for HoH.

“| Mean | 2.59 | 4.56 | 6.33 | 8.41 |”

The quote only partly states this claim.

For Pi + MiniMax-M3 on GameCraft-Bench, the mean Overall score was 55.04 under HoH@2.

“| Mean | | 42.16 | 49.06 | 55.04 | 58.78 | +16.62 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

The quote only partly states this claim.

For OpenCode + DeepSeek-V4-Pro on GameCraft-Bench, the mean Overall score was 40.32 under HoH@2.

“| Mean | | 26.90 | 28.61 | 40.32 | 48.98 | +22.08 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

The quote only partly states this claim.

For Codex + GPT-5.5 (high) on FrontierSWE, the mean official reward was 0.54 under HoH@3.

“| Mean | 0.31 | 0.50 | 0.50 | 0.54 |”

• AssumptionRests on an assumption: Official FrontierSWE rewards validly measure task performance.

The quote only partly states this claim.

For Codex + GPT-5.5 (high) on FrontierSWE, the mean official reward was 0.50 under HoH@2.

“| Mean | 0.31 | 0.50 | 0.50 | 0.54 |”

• AssumptionRests on an assumption: Official FrontierSWE rewards validly measure task performance.

The quote only partly states this claim.

For Codex + GPT-5.5 (high) on GameCraft-Bench, the mean difference in Overall score between HoH@3 and Vanilla was +21.93.

“| Mean | | 49.58 | 59.71 | 64.84 | 71.52 | +21.93 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

The quote only partly states this claim.

For Codex + GPT-5.5 (high) on GameCraft-Bench, the mean Overall score was 71.52 under HoH@3.

“| Mean | | 49.58 | 59.71 | 64.84 | 71.52 | +21.93 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

May combine two claims.

As planned capabilities approached completion, feature additions slowed and issue resolution became more prevalent, producing a stabilization phase in which the active backlog began to decline.

“As the planned capabilities approached completion, feature additions slowed and issue resolution became more prevalent, producing a stabilization phase in which the active backlog began to decline.”

May combine two claims.

On the 45 GameCraft-Bench tasks in the budget-controlled comparison using Codex with GPT-5.5 (high), the mean scores are 49.58 for Vanilla, 58.24 for three-pass Vanilla Continuation, and 71.52 for HoH.

“Averaged over the 45 tasks, Vanilla, three-pass Vanilla Continuation, and HoH obtain scores of 49.58, 58.24, and 71.52 using 2.59M, 6.33M, and 8.41M tokens per task, respectively.”

May combine two claims.

In the GameCraft-Bench ablation study with Codex+GPT-5.5 (high) at T=3, the mean score is 71.52 for full HoH, 63.39 without plan update, 65.23 without evidence feedback, and 63.67 without artifact warm-start.

“The mean score decreases from 71.52 for full HoH to 63.39 without plan update, 65.23 without evidence feedback, and 63.67 without artifact warm-start.”

The quote only partly states this claim.

For Codex + GPT-5.5 (high) on FrontierSWE, the mean official reward was 0.50 under HoH@1.

“| Mean | 0.31 | 0.50 | 0.50 | 0.54 |”

• AssumptionRests on an assumption: Official FrontierSWE rewards validly measure task performance.

The quote only partly states this claim.

For Codex + GPT-5.5 (high) on GameCraft-Bench, the mean Overall score was 64.84 under HoH@2.

“| Mean | | 49.58 | 59.71 | 64.84 | 71.52 | +21.93 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

May combine two claims.

During initial construction (Loops 1-27), HoH established the executable project and its core interaction paths.

“During initial construction (Loops 1-27), HoH established the executable project and its core interaction paths.”

The quote only partly states this claim.

For Pi + MiniMax-M3 on GameCraft-Bench, the mean Overall score was 49.06 under HoH@1.

“| Mean | | 42.16 | 49.06 | 55.04 | 58.78 | +16.62 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

The quote only partly states this claim.

For OpenCode + DeepSeek-V4-Pro on GameCraft-Bench, the mean Overall score was 28.61 under HoH@1.

“| Mean | | 26.90 | 28.61 | 40.32 | 48.98 | +22.08 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

The quote only partly states this claim.

For OpenCode + DeepSeek-V4-Pro on GameCraft-Bench, the mean Overall score was 48.98 under HoH@3.

“| Mean | | 26.90 | 28.61 | 40.32 | 48.98 | +22.08 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

For OpenCode + DeepSeek-V4-Pro on GameCraft-Bench, the mean Overall score was 26.90 under Vanilla.

“| Mean | | 26.90 | 28.61 | 40.32 | 48.98 | +22.08 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

May combine two claims.

In the GameCraft-Bench ablation variants, the mean cumulative coding-harness tokens per task are 7.56M without plan update, 7.46M without evidence feedback, 11.12M without warm-start, and 8.41M for full HoH.

“| w/o Plan Update | 7.56 |”

• Also stated in: § C.4 Ablation Study · p. 48 · plain

  “| w/o Evidence Feedback | 7.46 |”

• Also stated in: § C.4 Ablation Study · p. 48 · plain

  “| w/o Warm-Start | 11.12 |”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

For Codex + GPT-5.5 (high) on FrontierSWE, the mean official reward was 0.31 under Vanilla.

“| Mean | 0.31 | 0.50 | 0.50 | 0.54 |”

• AssumptionRests on an assumption: Official FrontierSWE rewards validly measure task performance.

May combine two claims.

Adding initial capabilities exposed missing requirements and defects, so the active issue backlog increased as the artifact became more testable.

“Adding these initial capabilities also exposed missing requirements and defects, so the active issue backlog increased as the artifact became more testable.”

The quote only partly states this claim.

For Pi + MiniMax-M3 on GameCraft-Bench, the mean Overall score was 58.78 under HoH@3.

“| Mean | | 42.16 | 49.06 | 55.04 | 58.78 | +16.62 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

The quote only partly states this claim.

For Codex + GPT-5.5 (high) on GameCraft-Bench, the mean Overall score was 59.71 under HoH@1.

“| Mean | | 49.58 | 59.71 | 64.84 | 71.52 | +21.93 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

May combine two claims.

By Loop 70, 65 of the 81 recorded issues had been closed, leaving 16 unresolved.

“By Loop 70, 65 of the 81 recorded issues had been closed, leaving 16 unresolved.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

For Pi + MiniMax-M3 on GameCraft-Bench, the mean Overall score was 42.16 under Vanilla.

“| Mean | | 42.16 | 49.06 | 55.04 | 58.78 | +16.62 |”

• AssumptionRests on an assumption: GameCraft-Bench Overall scores validly measure the quality of game artifacts.

May combine two claims. The quote only partly states this claim.

The design of HoH (bounded increments, independent testing, etc.) helps prevent development from collapsing into repetitive local repairs and reduces the risk of uncontrolled changes.

“This helps prevent development from collapsing into repetitive local repairs, while the limited scope makes progress easier to verify and reduces the risk of uncontrolled changes.”

Classified as probably not a substantial claim.

HoH's constraint on verifiable outcomes preserves agent autonomy over reasoning, tool use and implementation strategy.

“This constrains verifiable outcomes while preserving agent autonomy over reasoning, tool use and implementation strategy.”

Classified as probably not a substantial claim.

These records allowed the three roles to modify, execute, and inspect the same project while retaining the artifact and observed test outcomes across loops.

“These records allowed the three roles to modify, execute, and inspect the same project while retaining the artifact and observed test outcomes across loops.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Code changes and role-specific artifacts are committed to a public GitHub repository after each agent stage, making the complete development trajectory traceable.

“Code changes and role-specific artifacts are committed to a public GitHub repository after each agent stage, making the complete development trajectory traceable.”

May combine two claims.

As trajectories grow, agents may lose track of earlier requirements and design decisions, or introduce local fixes that violate constraints elsewhere.

“As trajectories grow, agents may lose track of earlier requirements and design decisions, or introduce local fixes that violate constraints elsewhere [3, 26].”

• cites [3] [3] Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei A. Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? In Advances in Neural Information Processing Systems , volume 38. Curran Associates, Inc., 2025. https://proceedings.neurips.cc/paper\_files/ paper/2025/file/b1041e52d3be19f0a9bc491657488e4a-Paper-Datasets\_and\_Benchmarks\_Track.pdf . - sourceNot checked — no full text of the source
• cites [26] [26] Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev. SWT-Bench: Testing and validating real-world bug-fixes with code agents. In Amir Globerson, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems , volume 37, pages 81857-81887. Curran Associates, Inc., 2024. doi: 10.52202/079017-2601. https://proceedings.neurips.cc/paper\_files/paper/2024/ file/94f093b41fc2666376fb1f667fe282f3-Paper-Conference.pdf . - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based).

A criterion is verified only when candidate-bound records support the required behavior.

“A criterion is verified only when candidate-bound records support the required behavior.”

Classified as probably not a substantial claim.

The two forms of testing (black-box and white-box) provide complementary views of the same frozen candidate.

“The two forms of testing provide complementary views of the same frozen candidate.”

May combine two claims.

Human involvement was limited to restoring network or API availability and did not extend to planning, implementation, debugging, testing, or acceptance.

“Human involvement was limited to restoring network or API availability and did not extend to planning, implementation, debugging, testing, or acceptance.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Encouraging agents to draw on existing resources rather than recreate standard capabilities reduces redundant effort on routine engineering tasks.

“Agents are encouraged to draw on existing resources rather than recreate standard capabilities, reducing redundant effort on routine engineering tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study used GitHub for version control and issue tracking, retaining the commit and issue histories of the evolving project.

“We used GitHub for version control and issue tracking, retaining the commit and issue histories of the evolving project.”

May combine two claims.

By preserving the software state together with concise accounts of how it changes, HoH can return to previously verified states after major regressions and draw on evidence from earlier attempts when similar failures recur.

“By preserving the software state together with concise accounts of how it changes, HoH can return to previously verified states after major regressions and draw on evidence from earlier attempts when similar failures recur to inform the diagnosis and resolution.”

May combine two claims.

On GameCraft-Bench, HoH produces more complete and refined game artifacts, with richer gameplay mechanics, clearer visual presentation, and deeper progression.

“On GameCraft-Bench, HoH produces more complete and refined game artifacts, with richer gameplay mechanics, clearer visual presentation, and deeper progression.”

Classified as probably not a substantial claim. May combine two claims.

Freezing the artifact for QA separates artifact production from artifact assessment and gives every observation a single candidate identity.

“Freezing separates artifact production from artifact assessment: the implementation cannot change while its evidence is being collected. It also gives every observation a single candidate identity, so that an assessment cannot combine behavior from different artifact versions.”

Classified as probably not a substantial claim.

The separation of roles (Planner, Developer, QA Tester) prevents implementation and acceptance from collapsing into the same decision.

“The separation makes the target of an increment explicit, preserves implementation autonomy within that target, and prevents implementation and acceptance from collapsing into the same decision.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the Fusepoint case, the Tester additionally examined live keyboard input and the resulting game responses, audio behavior, and the integration of 3D assets.

“For this case, the Tester additionally examined live keyboard input and the resulting game responses, audio behavior, and the integration of 3D assets.”

Classified as probably not a substantial claim.

Each plan must both address outstanding problems and deliver a small yet concrete new capability, following the principle of iterative and incremental development.

“Each plan must both address outstanding problems and deliver a small yet concrete new capability, following the principle of iterative and incremental development [13].”

• cites [13] [13] Craig Larman and Victor R Basili. Iterative and incremental developments. a brief history. Computer , 36 (6):47-56, 2003. - sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

The separation ensures that acceptance follows observable evidence rather than the Developer's knowledge of its implementation or its completion claim.

“This separation ensures that acceptance follows observable evidence rather than the Developer's knowledge of its implementation or its completion claim.”

Classified as probably not a substantial claim. May combine two claims.

HoH specifies the artifacts and evidence that agents must deliver, but does not prescribe a rigid workflow for producing them.

“Throughout this process, HoH specifies the artifacts and evidence that agents must deliver, but does not prescribe a rigid workflow for producing them.”

May combine two claims.

HoH autonomously builds a complex game from scratch, given only high-level product requirements, which exposes challenges that are largely absent from conventional benchmarks.

“In our second setting, HoH autonomously builds a complex game from scratch, given only high-level product requirements, which exposes challenges that are largely absent from conventional benchmarks.”

Classified as probably not a substantial claim.

At each loop, the working state was materialized in a development document, the versioned software workspace, and testing records comprising an issue table and evidence-packet files.

“At each loop, the working state was materialized in a development document, the versioned software workspace, and testing records comprising an issue table and evidence-packet files.”

May combine two claims.

Later HoH iterations benefit from both revising the development document with execution evidence and continuing from the preceding implementation.

“These results show that later HoH iterations benefit from both revising the development document with execution evidence and continuing from the preceding implementation.”

Classified as probably not a substantial claim. May combine two claims.

Within a harness-model configuration, Vanilla and HoH use the same model, native harness settings, public task materials, and benchmark tools.

“Within a harness-model configuration, Vanilla and HoH use the same model, native harness settings, public task materials, and benchmark tools.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Benchmark-specific adapters change how an artifact is launched and observed but do not change the planning, coding, or testing roles.

“Benchmark-specific adapters change how an artifact is launched and observed; they do not change the planning, coding, or testing roles.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Benchmark repositories, task data, raw run artifacts, analysis records, environment files, private credentials, provider secrets, and benchmark-hidden evaluator contents are not included in the anonymous code package.

“Benchmark repositories, task data, raw run artifacts, analysis records, environment files, private credentials, provider secrets, and benchmark-hidden evaluator contents are not included.”

Classified as probably not a substantial claim. May combine two claims.

The Docker-in-Docker execution design preserves the task software stack and prevents dependencies from one task from affecting another.

“This preserves the task software stack and prevents dependencies from one task from affecting another.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The GameCraft-Bench subset was constructed incrementally from a fixed earlier subset and completed to three tasks per family by seeded stratified sampling (seed 20260707), without reference to model scores.

“The subset was constructed incrementally from a fixed earlier subset and completed to three tasks per family by seeded stratified sampling (seed 20260707), without reference to model scores.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Aggregate scores average over tasks, not over multiple generations of the same task-condition pair.

“Aggregate scores therefore average over tasks, not over multiple generations of the same task-condition pair.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The exact commands depend on the selected harness and benchmark, but the role of each capability is fixed across iterations.

“The exact commands depend on the selected harness and benchmark, but the role of each capability is fixed across iterations.”

Classified as probably not a substantial claim. May combine two claims.

The implementation retains the development document, harness command, process logs, public media manifest, structured QA report, and resulting project workspace for each iteration, making the transition from A_{t-1} to A_t and the construction of E_t auditable without exposing evaluator-only information.

“The implementation retains the development document, harness command, process logs, public media manifest, structured QA report, and resulting project workspace for each iteration. These records make the transition from 𝐴𝑡 -1 to 𝐴𝑡 and the construction of E 𝑡 auditable without exposing evaluator-only information.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The selected clients do not expose a common reproducible generation seed, and temperature and top-p are not overridden; client and provider defaults are used throughout.

“The selected clients do not expose a common reproducible generation seed, and we do not override temperature or top𝑝 ; the corresponding client and provider defaults are used throughout.”

Classified as probably not a substantial claim. May combine two claims.

The ablations retain the three-iteration budget and the same harness-model configuration as full HoH.

“The ablations retain the three-iteration budget and the same harness-model configuration as full HoH.”

Classified as probably not a substantial claim.

Task-condition runs start from separate copies of the benchmark-provided workspace.

“Task-condition runs start from separate copies of the benchmark-provided workspace.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A record is placed in E_ver only when the evidence visibly supports the corresponding claim.

“A record is placed in E ver 𝑡 only when the evidence visibly supports the corresponding claim.”

Classified as probably not a substantial claim.

Each ablation variant changes one cross-iteration input while leaving the remaining interfaces unchanged.

“each variant changes one cross-iteration input while leaving the remaining interfaces unchanged”

Classified as probably not a substantial claim. May combine two claims.

Benchmark evaluation is separated from development; hidden tests, benchmark scores, private rubrics, evaluation formulas, and evaluator rationales are not included in the role prompts or evidence bundle and are never returned to a subsequent iteration.

“Benchmark evaluation is separated from development. Hidden tests, benchmark scores, private rubrics, evaluation formulas, and evaluator rationales are not included in the role prompts or evidence bundle and are never returned to a subsequent iteration.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When an attempt fails because of an infrastructure or modelprovider transport error, the failed attempt is replaced rather than included as an additional replicate.

“When an attempt fails because of an infrastructure or modelprovider transport error, the failed attempt is replaced rather than included as an additional replicate.”

Classified as probably not a substantial claim.

Harness versions, models, and exposed reasoning settings are held fixed between Vanilla and HoH within each configuration.

“Harness versions, models, and exposed reasoning settings are held fixed between Vanilla and HoH within each configuration.”

Probably not a substantial claim: may not pass S3 (load-bearing).

frogsgame-rl was excluded because it requires authenticated access to the external Tinker API, which was unavailable in the evaluation environment.

“frogsgame-rl | Requires authenticated access to the external Tinker API, which was unavailable in the evaluation environment.”

Classified as probably not a substantial claim.

Evaluator outputs are not returned to the planning, coding, or testing stages.

“Evaluator outputs are not returned to the planning, coding, or testing stages.”

Classified as probably not a substantial claim.

Every reported task-condition score is obtained from one valid run.

“Every reported task-condition score is obtained from one valid run.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 95% confidence intervals for the GameCraft-Bench component analysis in the main paper are percentile intervals from 20,000 task-bootstrap resamples.

“The 95% confidence intervals for the GameCraft-Bench component analysis in the main paper are percentile intervals from 20,000 task-bootstrap resamples.”

Classified as probably not a substantial claim. May combine two claims.

HoH reports the artifact produced after the prescribed iteration budget; intermediate artifacts are evaluated only for analysis and are not selected using benchmark scores.

“HoH reports the artifact produced after the prescribed iteration budget; intermediate artifacts are evaluated only for analysis and are not selected using benchmark scores.”

Classified as probably not a substantial claim.

Warm-starting preserves not only source files but also the project structure and resources required to continue development from A_{t-1}.

“Warm-starting therefore preserves not only source files but also the project structure and resources required to continue development from 𝐴𝑡 -1.”

Classified as probably not a substantial claim.

Source-code presence alone is not treated as behavioral verification.

“Source-code presence alone is not treated as behavioral verification.”

Classified as probably not a substantial claim.

Within one experimental condition, the Project Planner, Developer, and QA Tester are three independent invocations of the same harness-model configuration H, with model and native harness capabilities fixed.

“Within one experimental condition, the Project Planner , Developer , and Quality Assurance (QA) Tester are three independent invocations of the same harness-model configuration 𝐻 . Their model and native harness capabilities remain fixed, while role-specific instructions determine what each invocation may read, modify, and return.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The anonymous code package accompanying the submission contains the core HoH implementation, role prompt templates, the GameCraft-Bench adapter, and the necessary wrappers for the evaluated harness-model configurations.

“The anonymous code package accompanying the submission contains the core HoH implementation, role prompt templates, the GameCraft-Bench adapter, and the necessary wrappers for the evaluated harness-model configurations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The fixed seeds control task sampling and statistical resampling rather than model generation.

“The fixed seeds reported below control task sampling and statistical resampling rather than model generation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

modular-stack-wan21 was excluded because it requires a Modular MAX software stack needing an NVIDIA driver of at least 580 (CUDA 13), whereas the available H200 worker used driver 570.133.20.

“modular-stack-wan21 | Requires a Modular MAX software stack needing an NVIDIA driver of at least 580 (CUDA 13), whereas the available H200 worker used driver 570.133.20.”

Classified as probably not a substantial claim. May combine two claims.

HoH adapts to observable execution and QA findings while the public task specification remains the authoritative requirement source.

“HoH therefore adapts to observable execution and QA findings while the public task specification remains the authoritative requirement source.”

Probably not a substantial claim: may not pass S3 (load-bearing).

HoH does not replace the tools exposed by the underlying coding harness.

“HoH does not replace the tools exposed by the underlying coding harness.”

Classified as probably not a substantial claim.

The budget-controlled comparison holds the initial task set and harness-model configuration fixed while separating repeated coding passes from the planning-coding-testing structure of HoH.

“The comparison therefore holds the initial task set and harness-model configuration fixed while separating repeated coding passes from the planning-coding-testing structure of HoH.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The five reporting groups are introduced only for aggregate analysis; all task scores continue to use the benchmark's original family definitions.

“These five groups are introduced only for aggregate analysis; all task scores continue to use the benchmark's original family definitions.”

May combine two claims.

The versioned workspace allowed implementation work to accumulate, while its GitHub commit history made individual changes traceable.

“The versioned workspace allowed implementation work to accumulate, while its GitHub commit history made individual changes traceable.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Token usage is compared within each harness-model configuration because cache accounting differs across providers.

“We compare token usage within each harness-model configuration because cache accounting differs across providers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The two benchmarks use different execution environments because they exercise different software artifacts.

“The two benchmarks use different execution environments because they exercise different software artifacts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Two official FrontierSWE tasks are not included in the evaluated subset, and their omission is determined by execution requirements rather than model outcomes.

“Two official FrontierSWE tasks are not included in the evaluated subset. Their omission is determined by execution requirements rather than model outcomes.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The public task specification is inserted without modification.

“The public task specification is inserted without modification.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Private benchmark evaluators and their outputs are excluded from the development and inspection interfaces.

“Private benchmark evaluators and their outputs are excluded from these interfaces.”