CoChat
Sign Up
Cited by 1AI-based Problem Solving and PlanningArtificial Intelligence in GamesArtificial Intelligence Applications

Executable World Models for ARC-AGI-3 in the Era of Coding Agents

Open access
Rodionov, Sergey·2026·Lecture Notes in Computer Science
DOI 10.1007/978-3-032-33195-3_15

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

We evaluate an initial coding-agent system for ARC-AGI-3 in which the agent maintains an executable Python world model, verifies it against previous observations, refactors it toward simpler abstractions as a practical proxy for an MDL-like simplicity bias, and plans through the model before acting. The system is intentionally direct: it uses a scripted controller, predefined world-model interfaces, verifier programs, and a plan executor, but no hand-coded game-specific logic. The agent-facing prompts, workspace, and controller contain no game-specific code, game-specific prompts, hand-coded heuristics, hidden solutions, or other game-specific information; the same agent and prompts are used across games. Because the coding agent has broad system access, we audit unintended information channels, describe earlier vulnerable harnesses, and explain how the current harness closes observed leakage channels while reducing benchmark-specific information exposure. We report results on the 25 public ARC-AGI-3 games. Each playthrough starts from a fresh agent instance and clean workspace, with no access to files or conversation state from earlier playthroughs. With GPT-5.5 high reasoning effort, the agent fully solved 15 games and achieved a mean per-game RHAE of 58.12%. With GPT-5.4 high reasoning effort, it fully solved 8 games and achieved a mean per-game RHAE of 41.29%. Performance on the private validation set, which is not yet available to us, remains to be tested. Overall, the results provide preliminary evidence that verifier-driven executable world models are a promising approach for ARC-AGI-3 agents. Full run artifacts are released with the code at https://github.com/astroseger/arc-3-agents-baseline1.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper evaluates an initial coding-agent system for ARC-AGI-3 that maintains, verifies, refactors, and plans through an executable Python world model. On the 25 public games, it fully solved 7 games and achieved a mean per-game RHAE of 32.58%, with highly uneven performance.

Key findings

  • The agent fully solved 7 of 25 public ARC-AGI-3 games and achieved mean game RHAE greater than 75% on 6 of 25 games.
  • Mean per-game RHAE was 32.58%, median per-game RHAE was 14.65%, and mean game RHAE was below 5% on 9 of 25 games.
  • Across all 29 recorded runs, the agent solved 106 of 209 attempted levels.
  • Performance varied substantially between fresh playthroughs of the same game, e.g. cn04 produced RHAE values of 62.15% and 0.01%, while g50t produced 21.43% and 34.03%.

Methods

  • The system uses a scripted external controller and a coding agent that fills in and maintains an executable Python world model, verifier programs, and a plan executor, with no hand-coded game-specific logic.
  • Evaluation covered all 25 public ARC-AGI-3 games; each recorded playthrough used a fresh agent instance with no access to previous playthrough-specific files or conversation state, and RESET counted as an ordinary environment action.
  • Runs used Codex CLI version 0.122.0 and GPT-5.4; RHAE was the official efficiency-based score, aggregated by first averaging RHAE across runs of the same game and then averaging per-game means across the 25 public games.

Limitations

  • The paper states that performance on the private validation set remains to be tested, and that the system should be viewed as a baseline rather than a finished architecture.
  • The paper states that this first implementation does not support robust continuation after technical failures, so some runs ended early and are reported as-is; it also notes that the coding agent can still bypass the plan executor.
  • The paper reports failure modes including early tunnel vision on initial hypotheses and failure to build effective planners over partially correct models; it also notes that most development used the public game ls20, which the text makes evident as a possible generalization concern.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/978-3-032-33195-3_15

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 12 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Large language models are most reliable when they are not used as final authorities, but as proposal mechanisms inside systems that can check their outputs.

    “Large language models are most reliable when they are not used as final authorities, but as proposal mechanisms inside systems that can check their outputs.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [8] Li, Y., et al.: Competition-level code generation with AlphaCode. Science 378(6624), 1092–1097 (2022). https://doi.org/10.1126/science.abq1158 source

      The Approach (p. 9) describes AlphaCode generating many candidate programs, then filtering them with example tests and clustering before submission. Filtering & clustering (p. 19) reports that these checks improve its solve rate. This supports the proposal-and-check pattern for competitive-programming code, but not the broader claim that large language models are most reliable this way across tasks. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [10] Romera-Paredes, B., et al.: Mathematical discoveries from program search with large language models. Nature 625, 468–475 (2024). https://doi.org/10.1038/s41586-023-06924-6 source

      The Main section says LLMs can confabulate when asked to find verifiably correct solutions, and the FunSearch > Evaluation section says generated programs are checked and invalid ones discarded. This supports using LLMs to propose outputs within a checking system. The paper demonstrates that approach for program search, but does not establish that it is the most reliable way to use LLMs generally. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [12] Trinh, T.H., Wu, Y., Le, Q.V., He, H., Luong, T.: Solving olympiad geometry without human demonstrations. Nature 625, 476–482 (2024). https://doi.org/10.1038/s41586-023-06747-5 source

      The Main and “Combining language modelling and symbolic engines” sections describe a language model proposing auxiliary constructions while a symbolic engine performs deduction and searches for a proof. This supports the proposal-and-checking approach in geometry theorem proving, but the paper does not establish that large language models are generally *most reliable* in such systems rather than as final authorities. The indexed passages provide no page numbers. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    The generate-and-verify pattern appears in recent successes such as AlphaCode, FunSearch, and AlphaGeometry.

    “This pattern appears in recent successes such as AlphaCode, where generated programs are filtered by execution [8], FunSearch, where proposed programs are selected by automated evaluators [10], and AlphaGeometry, where a neural model proposes auxiliary geometric constructions and an exact symbolic engine derives and verifies their consequences [12].”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [8] Li, Y., et al.: Competition-level code generation with AlphaCode. Science 378(6624), 1092–1097 (2022). https://doi.org/10.1126/science.abq1158 source
    • cites [10] Romera-Paredes, B., et al.: Mathematical discoveries from program search with large language models. Nature 625, 468–475 (2024). https://doi.org/10.1038/s41586-023-06924-6 source
    • cites [12] Trinh, T.H., Wu, Y., Le, Q.V., He, H., Luong, T.: Solving olympiad geometry without human demonstrations. Nature 625, 476–482 (2024). https://doi.org/10.1038/s41586-023-06747-5 source
  • Findingown data

    Performance varied substantially between playthroughs of the same game.

    “The repeated runs illustrate that performance can vary substantially between playthroughs.”
    • Also stated in: § 3.3 Public-game results · plain
      “performance can vary substantially between playthroughs.”
    Quote in the paperNot a substantial claim
  • Findingown data

    In g50t, the two recorded playthroughs produced RHAE values of 21.43% and 34.03%.

    “g50t produced 21.43% and 34.03%.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Across all 29 recorded runs, the agent solved 106 of 209 attempted levels.

    “Across all 29 recorded runs, including repeated games, the agent solved 106 of 209 attempted levels.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 31
  • May combine two claims. 11

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    World models are central to model-based reinforcement learning and agentic intelligence more generally.

    “World models are central to model-based reinforcement learning and agentic intelligence more generally [6].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [6] Ha, D., Schmidhuber, J.: World models (2018). https://doi.org/10.48550/arXiv.1803.10122 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    WorldCoder builds a Python program representing an agent's knowledge of the world from interaction data.

    “WorldCoder builds a Python program representing an agent's knowledge of the world from interaction data [11].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [11] Tang, H., Key, D., Ellis, K.: WorldCoder, a model-based LLM agent: building world models by writing code and interacting with the environment (2024). https://doi.org/10.48550/arXiv.2402.12275 source

  • May combine two claims.

    Findingown data

    The agent solved ar25, lp85, and tr87 with 100% RHAE.

    “The agent solved several games completely, including ar25, lp85, and tr87 with 100% RHAE.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Small differences in early hypotheses can lead to very different later trajectories.

    “small differences in early hypotheses can lead to very different later trajectories.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The agent may not restart the whole game to obtain a better trajectory.

    “may not restart the whole game to obtain a better trajectory”
    • Also stated in: § 2.1 ARC-AGI-3 interaction protocol · plain
      “The agent cannot restart the whole game to obtain a better run”
    • Also stated in: § 3.1 Evaluation setup · plain
      “Restarting the entire game run to obtain a better trajectory is not allowed.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    When an agent observes only a small number of transitions, many world models may be consistent with the data.

    “When an agent observes only a small number of transitions, many world models may be consistent with the data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    In each case, the language model supplies approximate search, while reliability comes from an external verification process.

    “In each case, the language model supplies approximate search, while reliability comes from an external verification process.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Each recorded playthrough uses a fresh agent instance with no access to previous playthrough-specific files or conversation state.

    “Each recorded playthrough uses a fresh agent instance with no access to previous playthrough-specific files or conversation state.”
    • Also stated in: § 3 Results on 25 Public Games · plain
      “Each repeated playthrough used a fresh agent launch with no access to previous playthrough-specific files, logs, or conversation state.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    The private validation set is the decisive test of whether the current ARC-AGI-3-specific tools are genuinely game-general within the benchmark or whether they have indirectly adapted to the public games.

    “The private validation set is the decisive test of whether the current ARC-AGI-3-specific tools are genuinely game-general within the benchmark or whether they have indirectly adapted to the public games.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The planner verifier checks that the main planner can produce plans that reach LEVEL COMPLETED inside the learned model for solved levels.

    “The planner verifier checks that the main planner can produce plans that reach LEVEL COMPLETED inside the learned model for solved levels.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    ARC-AGI-3 places agents in novel, abstract, turn-based environments where they must explore, infer goals, build models of environment dynamics, and plan action sequences without explicit natural language instructions.

    “ARC-AGI-3 places agents in novel, abstract, turn-based environments where they must explore, infer goals, build models of environment dynamics, and plan action sequences without explicit naturallanguage instructions [1].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [1] ARC Prize Foundation: ARC-AGI-3: A new challenge for frontier agentic intelligence (2026). https://doi.org/10.48550/arXiv.2603.24621 source

      Queued — the background check reaches it, usually within a day.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    In the current implementation, the coding agent still has direct access to the game client and can therefore bypass the plan executor.

    “In the current implementation, the coding agent still has direct access to the game client and can therefore bypass the plan executor.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The agent plays each game without cross-game learning.

    “The agent plays each game without cross-game learning.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    ARC-AGI-3 is a controlled version of a broader AGI problem.

    “This makes ARC-AGI-3 a controlled version of a broader AGI problem.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    The refactoring loop is a practical proxy for an MDL-like bias.

    “This refactoring loop is our practical proxy for an MDL-like bias: the model should not merely fit the observed transitions, but should explain them through a simpler executable structure that remains valid as new levels are encountered.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Code World Models study world models generated by large language models in the form of Python code for model-based reinforcement learning.

    “Code World Models study world models generated by large language models in the form of Python code for model-based reinforcement learning [2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2] Dainese, N., Merler, M., Alakuijala, M., Marttinen, P.: Generating code world models with large language models guided by monte carlo tree search (2024). https://doi.org/10.48550/arXiv.2405.15383 source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    The environments are simple enough that explicit computational models are plausible, but direct experimentation is still costly enough that blind trial and error is inadequate.

    “The environments are simple enough that explicit computational models are plausible, but direct experimentation is still costly enough that blind trial and error is inadequate.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisereasoning

    The useful model is not merely one that fits previous observations, but one that captures the underlying regularities compactly enough to support future planning.

    “The useful model is not merely one that fits previous observations, but one that captures the underlying regularities compactly enough to support future planning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Humans can solve the ARC-AGI-3 environments, while frontier AI systems scored below 1% as of March 2026.

    “humans can solve the environments, while frontier AI systems scored below 1% as of March 2026 [1].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [1] ARC Prize Foundation: ARC-AGI-3: A new challenge for frontier agentic intelligence (2026). https://doi.org/10.48550/arXiv.2603.24621 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The agent may not use information from previous playthroughs of the same game.

    “may not use information from previous playthroughs of the same game”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The agent receives only one exposure to the target game per playthrough.

    “the agent receives only one exposure to the target game.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    ARC-AGI-3 is far simpler than the real world: its environments are discrete, deterministic, and visually abstract.

    “ARC-AGI-3 is far simpler than the real world: its environments are discrete, deterministic, and visually abstract.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Premisenone given

    Direct interaction is costly, and the agent must decide which hypotheses and plans are worth testing.

    “Direct interaction is costly, and the agent must decide which hypotheses and plans are worth testing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The variance in performance is expected for the current implementation.

    “This variance is expected for the current implementation”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    World models play a role analogous to verifiers in programming or formal reasoning: they provide an internal space in which candidate actions can be evaluated before acting.

    “In such settings, world models play a role analogous to verifiers in programming or formal reasoning: they provide an internal space in which candidate actions can be evaluated before acting.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    At the start of a game, the agent receives only the initialized workspace, the general ARC-AGI-3 interaction tools, and the current game observations.

    “At the start of a game, it receives only the initialized workspace, the general ARC-AGI-3 interaction tools, and the current game observations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    To perform well, an agent must learn from limited interaction, compress observations into a useful model, and use that model to decide which actions are worth spending in the real environment.

    “To perform well, an agent must learn from limited interaction, compress observations into a useful model, and use that model to decide which actions are worth spending in the real environment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Most of the development of this first version was performed using the public game ls20.

    “most of the development of this first version was performed using the public game ls20.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The world-model verifier checks that the executable model reproduces the recorded observations from previous attempts.

    “The world-model verifier checks that the executable model reproduces the recorded observations from previous attempts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    In our evaluation, RESET is treated as an ordinary environment action and counts against the action budget.

    “In our evaluation, RESET is treated as an ordinary environment action and counts against the action budget.”
    • Also stated in: § 3.1 Evaluation setup · plain
      “this is counted as an ordinary environment action.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Minimum Description Length perspective treats good explanations as those that compress the observations while accounting for the complexity of the model itself.

    “This is closely related to the Minimum Description Length perspective, which treats good explanations as those that compress the observations while accounting for the complexity of the model itself [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] Grünwald, P.: A tutorial introduction to the minimum description length principle (2004). https://doi.org/10.48550/arXiv.math/0406077 source

      Checked before — its earlier verdict is held back until a new check runs.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The agent may not return to previously completed levels.

    “may not return to previously completed levels”
    • Also stated in: § 2.1 ARC-AGI-3 interaction protocol · plain
      “the agent cannot return to previously completed levels”
    • Also stated in: § 3.1 Evaluation setup · plain
      “It cannot return to previously completed levels.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Successful plan execution is an online test of the current world model.

    “Thus, successful plan execution is not merely action replay: it is an online test of the current world model.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Runs were performed in fresh isolated agent environments to avoid access to previous conversations or previous run-specific files.

    “Runs were also performed in fresh isolated agent environments to avoid access to previous conversations or previous run-specific files.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The ARC-AGI-3 library is isolated behind a server-client interface that does not expose game source code or hidden mechanics.

    “The ARC-AGI-3 library itself is isolated behind a server-client interface: the coding agent interacts with the game through our client, which does not expose game source code or hidden mechanics.”
    Quote in the paperSubstantial claim
35 of 35

  • The paper relies on this without support.

    The underlying language model (GPT-5.4) has not been trained on the public ARC-AGI-3 games or their solutions in a way that would allow it to solve them from memory.

    “The ARC-AGI-3 library itself is isolated behind a server-client interface: the coding agent interacts with the game through our client, which does not expose game source code or hidden mechanics.”

  • The paper relies on this without support.

    The agent's performance is attributable to the verifier-driven executable world model rather than solely to the underlying language model's capabilities.

    “Overall, the results provide preliminary evidence that verifier-driven executable world models are a promising approach for ARC-AGI-3 agents.”

  • The paper relies on this without support.

    The world-model verifier's consistency check against previous observations is sufficient to ensure the world model is correct enough for planning.

    “The world-model verifier checks that the executable model reproduces the recorded observations from previous attempts.”
3 of 3

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Rodionov, S.
Corresponding author
Rodionov, Sergey
ORCID
Rodionov, Sergey
Venue
Lecture Notes in Computer Science
Publisher
Springer Nature Switzerland
Published
2026
Pages
198-210
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
DOI
10.1007/978-3-032-33195-3_15
OpenAlex
W7170085180
Semantic Scholar
71a1a1a1377f5cdd7e115e1fa53b0e91366473fb
Cited by
1
References
11
Citations by year
2026: 1
FWCI
4.23
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
coding agents, refactoring, model complexity, plan execution, information leakage, GPT-5

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-06.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review