Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?

May combine two claims.

In the other three model-effort settings, the same verification variant scores 47.62, 100.00, and 85.93 on r11l and leaves at most two levels unsolved; at gpt-5.5-xhigh, the textual and simplification variants score 90.56 and 100.00.

“In the other three model-effort settings, the same variant scores 47.62, 100.00, and 85.93 on r11l and leaves at most two levels unsolved; at gpt-5.5-xhigh, the textual and simplification variants score 90.56 and 100.00.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

May combine two claims.

The most conspicuous case is r11l under the verification variant with gpt-5.5-xhigh: the playthrough scores only 4.76 and leaves five levels unsolved.

“The most conspicuous case is r11l under the verification variant with gpt-5.5-xhigh: the playthrough scores only 4.76 and leaves five levels unsolved.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

May combine two claims.

The verification variant solves every level in both blocks and scores 100 on 23 of 25 games in each; the two sub-100 games are sc25 and tn36 at xhigh, but bp35 and s5i5 at max.

“It solves every level in both blocks and scores 100 on 23 of 25 games in each; the two sub-100 games are sc25 and tn36 at xhigh, but bp35 and s5i5 at max.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

May combine two claims.

At gpt-5.5-xhigh, verification scores 74.78, while the textual and simplification variants reach 72.51 and 73.09.

“At gpt-5.5-xhigh, for example, verification scores 74.78, while the textual and simplification variants already reach 72.51 and 73.09.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

May combine two claims.

Simplification improves on the executable variant in three settings, by 3.39-8.38 points, but is 2.94 points worse with gpt-5.4-high.

“Simplification improves on the executable variant in three settings, by 3.39-8.38points, but is 2.94 points worse with gpt-5.4-high.”

May combine two claims.

The within-setting range between the best and worst variant is 5.08-14.48 RHAE points, whereas changing model or effort often produces a larger shift.

“The within-setting range between the best and worst variant is 5.08-14.48RHAE points, whereas changing model or effort often produces a larger shift.”

May combine two claims.

The textual variant improves by 3.63 RHAE points from xhigh to max and solves the five remaining levels, consistent with the model-and-effort pattern in the main study.

“The textual variant improves by 3.63 RHAE points from xhigh to max and solves the five remaining levels, consistent with the model-and-effort pattern in the main study.”

May combine two claims.

The textual and executable variants are substantially cheaper than simplification and verification, although their ordering depends on the model and effort.

“The textual and executable variants are substantially cheaper, although their ordering depends on the model and effort.”

May combine two claims.

The two verification blocks use less than half the human-baseline total despite scoring slightly below 100 RHAE.

“The two verification blocks use less than half the human-baseline total despite scoring slightly below 100 RHAE.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

May combine two claims.

At xhigh, the verification variant completes all 183 public levels, whereas the textual variant leaves five levels across two games unsolved.

“It completes all 183 public levels, whereas the textual variant leaves five levels across two games unsolved.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

Probably not a substantial claim: may not pass S3 (load-bearing).

The excluded block described in section 4.4 contained a second qualitative warning.

“The excluded block described in section 4.4 contained a second qualitative warning.”

May combine two claims.

At max, the verification and textual variants score 98.77 and 95.97, a 2.80-point difference, and both variants complete all levels.

“At max, the corresponding scores are 98.77 and 95.97, a 2.80-point difference, and both variants complete all levels.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

May combine two claims.

All three v1.6 blocks that complete all 183 levels use substantially fewer actions than the summed human baseline of 17,135; their totals range from 7,758 to 10,111, a reduction of 41-55%.

“All three v1.6 blocks that complete all 183 levels use substantially fewer actions than the summed human baseline of 17,135; their totals range from 7,758 to 10,111, a reduction of 41-55%.”

May combine two claims.

At max, the verification variant uses 7,758 actions, 589 fewer than at xhigh, even though its RHAE is 0.20 points lower.

“At max, the verification variant uses 7,758 actions, 589 fewer than at xhigh, even though its RHAE is 0.20 points lower.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

May combine two claims.

For gpt-5.5, the textual variant outperforms the executable variant at both efforts: 58.85 versus 51.16 at high, and 72.51 versus 69.70 at xhigh.

“Indeed, for gpt-5.5 the textual variant outperforms the executable variant at both efforts: 58.85 versus 51.16 at high, and 72.51 versus 69.70 at xhigh.”

May combine two claims.

In every model-effort setting, the verification variant uses the most cost tokens and the simplification variant uses the second most.

“Simplification and verification are resource-intensive: in every model-effort setting, the verification variant uses the most cost tokens and the simplification variant uses the second most.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

May combine two claims.

For every agent variant, xhigh reasoning effort outperforms high within each model, and gpt-5.5 outperforms gpt-5.4 at each matched effort.

“For every agent variant, xhigh outperforms high within each model, and gpt-5.5 outperforms gpt-5.4 at each matched effort.”

• Also stated in: § 6.1 Capability and reasoning effort dominate the main comparison · plain

  “Every one of the 16 matched capability comparisons has the same direction: increasing reasoning effort helps each variant under each model, and moving from gpt-5.4 to gpt-5.5 helps each variant at each effort.”

May combine two claims.

Every agent variant improves with a stronger model and with greater reasoning effort.

“The most robust result is that every agent variant improves with a stronger model and with greater reasoning effort.”

May combine two claims.

With gpt-5.4, the executable variant is essentially tied with the textual variant at high and is 4.05 points better at xhigh.

“With gpt-5.4,the executable variant is essentially tied with the textual variant at high and is 4.05 points better at xhigh.”

May combine two claims.

For gpt-5.5, the textual variant is comparable to the simplification variant, which scores 58.35 and 73.09.

“It is instead comparable to the simplification variant, which scores 58.35 and 73.09.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the main study, the textual variant with gpt-5.5 at xhigh effort achieved a mean RHAE of 72.51 with 6 unsolved games and 17 unsolved levels.

“gpt-5.5 xhigh | 72.51 (6/17) | 69.70 (9/17) | 73.09 (6/14) | 74.78 (5/12)”

May combine two claims.

The gpt-5.6-sol verification variant solved all 183 levels across the 25 public games at both efforts, achieved approximately 99 RHAE, and at each effort used less than half as many total actions as the summed human first-contact baseline.

“It solved all 183 levels across the 25 public games at both efforts, achieved approximately 99 RHAE, and at each effort used less than half as many total actions as the summed human first-contact baseline.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The twma_v1.6 variant with gpt-5.6-sol at xhigh effort achieved a mean RHAE of 92.34 with 2 unsolved games and 5 unsolved levels.

“twma_v1.6 | gpt-5.6-sol xhigh | 92.34 | 2/5 | - | 32.32”

May combine two claims.

The ewma_sv_v1.6 variant with gpt-5.6-sol at max effort achieved a mean RHAE of 98.77 with 0 unsolved games and 0 unsolved levels.

“ewma_sv_v1.6 gpt-5.6-sol max | 98.77 | 0/0 | 7,758 103.49”

Classified as probably not a substantial claim.

In the main study, the verification variant with gpt-5.5 at xhigh effort achieved a mean RHAE of 74.78 with 5 unsolved games and 12 unsolved levels.

“gpt-5.5 xhigh | 72.51 (6/17) | 69.70 (9/17) | 73.09 (6/14) | 74.78 (5/12)”

May combine two claims.

The twma_v1.6 variant with gpt-5.6-sol at max effort achieved a mean RHAE of 95.97 with 0 unsolved games and 0 unsolved levels.

“twma_v1.6 | gpt-5.6-sol max | 95.97 | 0/0 | 10,111 | 30.60”

May combine two claims.

The ewma_sv_v1.6 variant with gpt-5.6-sol at xhigh effort achieved a mean RHAE of 98.97 with 0 unsolved games and 0 unsolved levels.

“ewma_sv_v1.6 gpt-5.6-sol xhigh | 98.97 | 0/0 | 8,347 | 90.75”

Probably not a substantial claim: may not pass S3 (load-bearing).

The ewma_sv_v1.5 variant with gpt-5.5 at xhigh effort achieved a mean RHAE of 82.01 with 5 unsolved games and 9 unsolved levels.

“ewma_sv_v1.5 gpt-5.5 | xhigh | 82.01 | 5/9 | -207.41”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

The sum of level_baseline_actions across the 183 levels is 17,135.

“Actions is reported only for blocks that completed all 183 levels; the corresponding sum of level_baseline_actions is 17,135.”

May combine two claims.

Increasing the reasoning budget improved the textual variant, whereas the verification variant was already near the measurement ceiling.

“Increasing the reasoning budget improved the textual variant, whereas the verification variant was already near the measurement ceiling.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

Probably not a substantial claim: may not pass S3 (load-bearing).

In the main study, the simplification variant with gpt-5.5 at xhigh effort achieved a mean RHAE of 73.09 with 6 unsolved games and 14 unsolved levels.

“gpt-5.5 xhigh | 72.51 (6/17) | 69.70 (9/17) | 73.09 (6/14) | 74.78 (5/12)”

Classified as probably not a substantial claim.

In the main study, the executable variant with gpt-5.5 at xhigh effort achieved a mean RHAE of 69.70 with 9 unsolved games and 17 unsolved levels.

“gpt-5.5 xhigh | 72.51 (6/17) | 69.70 (9/17) | 73.09 (6/14) | 74.78 (5/12)”

Classified as probably not a substantial claim.

Within v1.6, the two variants share those updates, so each effort provides a matched textual-verification comparison.

“Within v1.6, however, the two variants share those updates, so each effort provides a matched textual-verification comparison.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In model-based reinforcement learning, an agent predicts the consequences of actions and uses those predictions to plan.

“In model-based reinforcement learning, an agent predicts the consequences of actions and uses those predictions to plan [8].”

• cites [8]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

This does not eliminate the possibility of contamination, since a model could still recognize a game from its observations alone.

“This does not eliminate the possibility of contamination, since a model could still recognize a game from its observations alone.”

Classified as probably not a substantial claim.

The agent-visible system contains no reference to ARC or ARC-AGI.

“The agent-visible system also contains no reference to ARC or ARC-AGI.”

Classified as probably not a substantial claim. May combine two claims.

Verification is prompted and tool-supported, but it is not silently enforced by the external controller.

“Verification is prompted and tool-supported, but it is not silently enforced by the external controller: the coding agent is responsible for invoking the verifiers and responding to their diagnostics.”

May combine two claims.

Early development harnesses exposed enough information for an agent to recover the true game identifier, use web search, or start a second local client as an unscored simulator.

“As described in our first article, early development harnesses exposed enough information for an agent to recover the true game identifier, use web search, or start a second local client as an unscored simulator.”

• cites [19]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Under those prices, one million cost tokens would correspond to USD 15 for gpt-5.4 and USD 30 for gpt-5.5.

“Under those prices, one million cost tokens would correspond to USD 15 for gpt-5.4 and USD 30 for gpt-5.5.”

Classified as probably not a substantial claim.

We use an all-or-nothing policy at the evaluation-block level.

“We use an all-or-nothing policy at the evaluation-block level.”

Classified as probably not a substantial claim. May combine two claims.

If any of the 25 playthroughs in a block suffers an irrecoverable technical failure, none of the playthroughs from that block enters the analysis; the complete 25-game block is run again from scratch.

“If any of the 25 playthroughs in a block suffers an irrecoverable technical failure, none of the playthroughs from that block enters the analysis; the complete 25-game block is run again from scratch.”

May combine two claims.

An ARC Prize analysis of 160 GPT-5.5 and Claude Opus 4.7 traces found that agents often identified local action effects without integrating them into a correct global model, adopted an inappropriate abstraction from a familiar game, or completed an early level without learning a rule that transferred to later levels.

“An ARC Prize analysis of 160 GPT-5.5 and Claude Opus 4.7 traces found that agents often identified local action effects without integrating them into a correct global model, adopted an inappropriate abstraction from a familiar game, or completed an early level without learning a rule that transferred to later levels [10].”

• cites [10]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

The study follows a nested ablation ladder and does not compare all possible combinations of executable modeling, simplification, and verification.

“The study follows a nested ablation ladder and does not compare all possible combinations of executable modeling, simplification, and verification.”

Classified as probably not a substantial claim.

The coding agent performs the substantive work: it inspects observations, edits its workspace, runs programs, forms hypotheses, plans, and submits actions through the game client.

“The coding agent performs the substantive work: it inspects observations, edits its workspace, runs programs, forms hypotheses, plans, and submits actions through the game client.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The renderer's override hook keeps a partially understood model usable while preserving a visible signal that some mechanic, latent variable, object identity, or rendering rule remains missing.

“This escape hatch keeps a partially understood model usable while preserving a visible signal that some mechanic, latent variable, object identity, or rendering rule remains missing.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The requested boundaries are advisory rather than perfectly enforced by the language model.

“The requested boundaries are advisory rather than perfectly enforced by the language model.”

Classified as probably not a substantial claim. May combine two claims.

The isolation controls remove the benchmark name and true game identifier, which reduces direct retrieval cues, but a model could still recognize a game from its visual observations.

“The isolation controls in section 4.1 remove the benchmark name and true game identifier, which reduces direct retrieval cues, but a model could still recognize a game from its visual observations.”

Classified as probably not a substantial claim.

Because playthroughs may last many hours, treating every such interruption as terminal would discard substantial progress.

“Because playthroughs may last many hours, treating every such interruption as terminal would discard substantial progress.”

Classified as probably not a substantial claim. May combine two claims.

This follow-up changes more than one implementation detail and is therefore not treated as a controlled estimate of the prompt's effect.

“This follow-up changes more than one implementation detail and is therefore not treated as a controlled estimate of the prompt's effect.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All main-study playthroughs use Codex CLI version 0.128.0.

“All main-study playthroughs use Codex CLI version 0.128.0[17].”

• cites [17]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

The resulting 4×2×2 design tests not only whether a component helps on average, but also whether its value changes with model capability or inference-time reasoning effort.

“The resulting 4×2×2 design tests not only whether a component helps on average, but also whether its value changes with model capability or inference-time reasoning effort.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For the GPT-5.6 follow-up, we retain the same formula only to keep the resource proxy comparable; it is not a GPT-5.6 price model.

“For the GPT-5.6 follow-up, we retain the same formula only to keep the resource proxy comparable; it is not a GPT-5.6 price model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We did not isolate the effect of the v1.5 changes experimentally.

“We did not isolate this change experimentally.”

Classified as probably not a substantial claim.

Recoverable coding-agent interruptions are handled within the original trajectory and do not trigger this policy.

“Recoverable coding-agent interruptions are handled within the original trajectory and do not trigger this policy.”

May combine two claims.

GPT-5.6 may have encountered game-specific information during training or evaluation because it was released after the public ARC-AGI-3 games, creating a substantial contamination concern.

“There is also a substantial contamination concern.GPT-5.6 was released after the public ARC-AGI-3 games, so it may have encountered game-specific information during training or evaluation.”

May combine two claims.

Averaging over 25 games reduces the influence of any one transient failure on the aggregate score, but a single playthrough per game cannot estimate run-to-run variability for that game.

“Averaging over 25 games reduces the influence of any one transient failure on the aggregate score, but a single playthrough per game cannot estimate run-to-run variability for that game.”

Classified as probably not a substantial claim.

The same game-general client, prompts, and controller are used for every game.

“The same game-general client, prompts, and controller are used for every game.”

Classified as probably not a substantial claim.

The verification variant begins with general templates and utilities but no game-specific rules, level layouts, or solutions.

“The verification variant begins with general templates and utilities but no game-specific rules, level layouts, or solutions.”

Classified as probably not a substantial claim.

All four systems remain coding agents.

“All four systems remain coding agents.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We do not report converted dollar estimates because the experiments used ChatGPT Pro-backed Codex accounts and were not billed through the API.

“We do not report converted dollar estimates because the experiments used ChatGPT Pro-backed Codex accounts and were not billed through the API.”

Classified as probably not a substantial claim.

The original baseline1 agent accumulated several tightly coupled mechanisms, especially around verification.

“The original baseline1 agent was developed as a proof of concept and accumulated several tightly coupled mechanisms, especially around verification.”

Classified as probably not a substantial claim.

Our first article combined executable representation, scheduled simplification, exact replay verification, and model-based plan execution in a single proof-of-concept agent.

“Our first article combined executable representation, scheduled simplification, exact replay verification, and model-based plan execution in a single proof-of-concept agent [19].”

• cites [19]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The controller does not contain game-solving logic.

“The controller does not contain game-solving logic.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Changes to the runtime's built-in prompting, context management, and tool-use behavior can change performance even when our controller and prompts are held fixed.

“We treat the coding-agent runtime and its version as part of the experimental system: changes to the runtime's built-in prompting, context management, and tool-use behavior can change performance even when our controller and prompts are held fixed.”

Classified as probably not a substantial claim.

At each effort, the v1.6 evaluation provides a matched comparison of the textual and verification variants under the same v1.6 harness.

“At each effort, this provides a matched comparison of the textual and verification variants under the same v1.6 harness.”

May combine two claims.

The verification workflow gives the best aggregate score but is also the most resource-intensive in every setting.

“The cost-token results reinforce the practical trade-off: the verification workflow gives the best aggregate score but is also the most resource-intensive in every setting.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

Classified as probably not a substantial claim.

Real actions are not required to pass through an online plan executor in the main ablation.

“Likewise, real actions are not required to pass through an online plan executor in the main ablation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

These measures are engineering controls, not a formal security proof against an arbitrarily adversarial agent.

“These measures are engineering controls, not a formal security proof against an arbitrarily adversarial agent.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each public game is played once per evaluation block.

“Each public game is played once per evaluation block.”

Classified as probably not a substantial claim. May combine two claims.

General internet access and Codex web search are disabled; a separate restricted proxy permits only the service endpoints needed by the coding-agent runtime.

“General internet access and Codex web search are disabled; a separate restricted proxy permits only the service endpoints needed by the coding-agent runtime.”

Probably not a substantial claim: may not pass S3 (load-bearing).

These controls may also reduce the risk that the agent benefits from benchmark-specific information memorized during training.

“These controls may also reduce the risk that the agent benefits from benchmark-specific information memorized during training.”

Classified as probably not a substantial claim.

Every playthrough starts from a fresh agent process and a clean workspace, with no files, conversation state, or learned artifacts carried over from another game.

“Every playthrough starts from a fresh agent process and a clean workspace, with no files, conversation state, or learned artifacts carried over from another game.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The prompts do not define a numerical complexity objective.

“The prompts do not define a numerical complexity objective.”

May combine two claims.

The action totals provide further evidence that the small reversal does not show that max is worse; with one block per cell and scores pressed against 100, the experiment cannot resolve such a small difference or estimate run-to-run variability.

“The action totals therefore provide further evidence that the small reversal does not show that max is worse; with one block per cell and scores pressed against 100, the experiment cannot resolve such a small difference or estimate run-to-run variability.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

Classified as probably not a substantial claim.

The experiment asks whether a coding agent benefits from making its environment hypothesis executable, not whether the agent should be prohibited from using code at all.

“This distinction is important: the experiment asks whether a coding agent benefits from making its environment hypothesis executable, not whether the agent should be prohibited from using code at all.”

May combine two claims.

Language models can be particularly effective as proposal mechanisms when their outputs can be checked externally; this generate-and-verify pattern underlies systems such as AlphaCode, FunSearch, and AlphaGeometry.

“Language models can be particularly effective as proposal mechanisms when their outputs can be checked externally; this generate-andverify pattern underlies systems such as AlphaCode, FunSearch, and AlphaGeometry [12,20,26].”

• cites [12]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [26]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

The final comparison changes both the verification objective and the supporting workspace: the verification variant receives fixed interfaces, templates, and verifier utilities.

“Moreover, the final comparison changes both the verification objective and the supporting workspace: the verification variant receives fixed interfaces, templates, and verifier utilities.”

Classified as probably not a substantial claim.

The client-server protocol rejects attempts to start a second game client.

“Finally, the client-server protocol rejects attempts to start a second game client.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The current experiment cannot infer whether simplification would help or harm the fixed-interface verification variant under gpt-5.4-high, because it does not include a verification-without-simplification variant.

“We cannot infer whether simplification would help or harm the fixed-interface verification variant under gpt-5.4-high,because the current experiment does not include a verificationwithout-simplification variant.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 1,500-action limit may be restrictive for some games.

“The 1,500-action limit may be restrictive for some games.”

May combine two claims.

The current harness runs the agent in an isolated Docker environment and closes the observed leakage channels.

“The current harness runs the agent in an isolated Docker environment and closes the observed leakage channels.”

Classified as probably not a substantial claim.

The effect of the verification variant should be interpreted as the effect of the complete verification treatment rather than as the isolated effect of a single instruction.

“Its effect should therefore be interpreted as the effect of the complete verification treatment rather than as the isolated effect of a single instruction.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The CLI update enables GPT-5.6 support in Codex.

“The CLI update enables GPT-5.6 support in Codex [16].”

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The present experiment does not test the hypothesis that agents without supplied verifiers sometimes notice decisive visual cues directly or create their own checks when uncertainty makes them useful.

“The present experiment does not test this hypothesis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Cost tokens should be read as a consistent proxy for resources consumed by the recorded runs, not as a billing forecast.

“Cost tokens should be read as a consistent proxy for resources consumed by the recorded runs, not as a billing forecast.”

May combine two claims.

Although the two v1.5 changes were introduced together and their effects cannot be separated, the result demonstrates that the v1.2 verification implementation was not a performance ceiling in this setting.

“Although the two changes were introduced together and their effects cannot be separated, the result demonstrates that the v1.2 verification implementation was not a performance ceiling in this setting.”

• AssumptionThe authors state this as an assumption: The ARC-AGI-3 games are deterministic: each attempt starts from exactly the same state, so the same actions lead to exactly the same results.

May combine two claims.

This concern is particularly relevant to gpt-5.5 in the main study and gpt-5.6-sol in the follow-up, both of which were released after the public ARC-AGI-3 games became available.

“This concern is particularly relevant to gpt-5.5 in the main study and gpt-5.6-sol in the follow-up, both of which were released after the public ARC-AGI-3 games became available [14,15].”

• cites [14]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [15]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Because the model postdates these games and held-out performance remains untested, the gpt-5.6-sol result should be interpreted as saturation of the public set only.

“Because the model postdates these games and held-out performance remains untested, this result should be interpreted as saturation of the public set only.”

May combine two claims.

A coding agent can inspect files and processes, execute arbitrary local code, and search for unintended interfaces.

“A coding agent can inspect files and processes, execute arbitrary local code, and search for unintended interfaces.”

Classified as probably not a substantial claim.

The primary metric is Relative Human Action Efficiency (RHAE), the official ARC-AGI-3 score.

“The primary metric is Relative Human Action Efficiency (RHAE), the official ARC-AGI-3 score [1].”

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Comparisons with v1.5 or the main study are not controlled across versions because both the model and several harness components changed.

“Comparisons with v1.5 or the main study are not controlled across versions because both the model and several harness components changed.”

Classified as probably not a substantial claim.

In the textual variant, only the requirement to maintain an executable world model as a persistent deliverable is removed.

“In the textual variant, only the requirement to maintain an executable world model as a persistent deliverable is removed.”

Classified as probably not a substantial claim.

The evaluation harness must therefore do more than hide the game name in the natural-language prompt.

“The evaluation harness must therefore do more than hide the game name in the natural-language prompt.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The v1.5 and v1.6 configurations differ from v1.2 in several components and are not additional cells of the controlled main ablation.

“The v1.5 and v1.6 configurations differ from v1.2 in several components and are not additional cells of the controlled main ablation.”

Classified as probably not a substantial claim.

ARC-AGI-3 evaluates exploration, modeling, goal discovery, planning, and execution through Relative Human Action Efficiency (RHAE), which rewards level completion while penalizing larger numbers of environment actions relative to a human first-contact baseline.

“It evaluates exploration, modeling, goal discovery, planning, and execution through Relative Human Action Efficiency (RHAE), which rewards level completion while penalizing larger numbers of environment actions relative to a human first-contact baseline [1].”

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

We do not evaluate verification without simplification.

“In particular, we do not evaluate verification without simplification.”

Classified as probably not a substantial claim.

The comparison between ewma_s_v1.2 and ewma_sv_v1.2 tests whether adding the explicit verification treatment improves the agent's problem-solving process.

“Accordingly, the comparison between ewma_s_v1.2and ewma_sv_v1.2tests whether adding the explicit verification treatment improves the agent's problem-solving process.”

Classified as probably not a substantial claim.

No results produced under the vulnerable configurations are used in this study.

“No results produced under the vulnerable configurations are used in this study.”

Classified as probably not a substantial claim.

All four agent variants use the same coding-agent runtime, game client, controller logic, action limits, and game-general prompts, differing only in the aspects shown in Table 1.

“Nested agent variants.All four use the same coding-agent runtime, game client, controller logic, action limits, and game-general prompts except for the differences shown.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The result suggests that different variants may benefit from different targeted improvements.

“More broadly, however, the result suggests that different variants may benefit from different targeted improvements.”

May combine two claims.

We lack direct access to ARC-AGI-3's semi-private and fully private sets and have not yet obtained a verified evaluation on them.

“We lack direct access to ARC-AGI-3's semi-private and fully private sets and have not yet obtained a verified evaluation on them [1].”

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The matched v1.6 comparison supports the main qualitative findings under a shared updated harness.

“The matched v1.6 comparison supports the main qualitative findings under a shared updated harness.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The v1.6 rows form matched textual-verification pairs within each effort, and the v1.5 row is not part of those comparisons.

“The v1.6 rows form matched textual-verification pairs within each effort; the v1.5 row is not part of those comparisons.”