CoChat
Sign Up
PreprintCited by 15AI-based Problem Solving and PlanningPsychiatry, Mental Health, NeuroscienceArtificial Intelligence in Games

ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence

Open access
Foundation, ARC Prize·2026·arXiv
DOI 10.48550/arxiv.2603.24621arXiv 2603.24621
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce ARC-AGI-3, an interactive benchmark for studying agentic intelligence through novel, abstract, turn-based environments in which agents must explore, infer goals, build internal models of environment dynamics, and plan effective action sequences without explicit instructions. Like its predecessors ARC-AGI-1 and 2, ARC-AGI-3 focuses entirely on evaluating fluid adaptive efficiency on novel tasks, while avoiding language and external knowledge. ARC-AGI-3 environments only leverage Core Knowledge priors and are difficulty-calibrated via extensive testing with human test-takers. Our testing shows humans can solve 100% of the environments, in contrast to frontier AI systems which, as of March 2026, score below 1%. In this paper, we present the benchmark design, its efficiency-based scoring framework grounded in human action baselines, and the methodology used to construct, validate, and calibrate the environments.

AI snapshot

Written from the paper’s full text.

TL;DR

ARC-AGI-3 is an interactive, turn-based benchmark for agentic intelligence in novel environments where agents must explore, infer goals, model dynamics, and plan without instructions. Humans solve 100% of environments, while frontier AI systems scored below 1% as of March 2026.

Key findings

  • ARC-AGI-3 evaluates four components of agentic intelligence: exploration, modeling, goal-setting, and planning/execution; environments use only Core Knowledge priors and no language or cultural symbols.
  • Human testing found 100% solvability: environments were included only if fully solved by at least two of 10 participants independently; across 486 unique participants, 414 candidate environments, and 2,893 attempts, total play time was 427.9 hours, with median attempt duration 7.4 minutes.
  • At release, official semi-private leaderboard scores were Opus 4.6 (Max) 0.50%, Gemini 3.1 Pro Preview 0.40%, GPT 5.4 (High) 0.20%, and Grok-4.20 (Beta 0309 Reasoning) 0.10%.
  • Dataset composition: 25 public demo environments, 55 semi-private, and 55 fully private; scoring uses RHAE (Relative Human Action Efficiency), comparing per-level action counts to an upper-median human baseline, squaring efficiency, capping per-level score at 1.15x human baseline, and applying level weights and per-environment caps.

Methods

  • Benchmark design: each environment is a series of levels in a turn-based 64x64 grid with 16 colors; action space is a subset of five key actions plus Undo and a cell-selection action; agents receive frames or frame sequences and must act without explicit instructions.
  • Construction and validation: an in-house game studio used a four-stage pipeline (specification, internal, external, done); automated validation included random regimes up to 50,000 and 1,000,000 steps, graph-based state-space exploration, and an acceptance threshold that a random policy should not solve a level more than 1 in 10,000 times.
  • Human calibration and AI evaluation: continuous testing sessions at a San Francisco testing center, 90-minute sessions with 20-minute soft and 30-minute hard per-environment limits, 10 participants per environment; official leaderboard uses no harness, a shared system prompt, and an action budget of five times the human median per level.

Limitations

  • The paper states that the public set is intentionally easier and does not comprehensively represent private-set mechanics, so public scores are never reported on the official leaderboard; the private set is designed to be out-of-distribution and more difficult.
  • The paper states that the official leaderboard excludes domain-specific harnesses and reports only systems behind general-purpose APIs, while the community leaderboard is self-reported and unverified; it also notes a small risk of data leakage for the semi-private set.
  • The paper states that the five-times-human action budget may yield marginally lower scores than the environment’s intrinsic limit, though it calls the differential negligible; evident from the text, human baselines come from 10 participants per environment, and some low-effort participant sessions were excluded from analysis.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2603.24621

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 8 cited sources checked · 0 supported · 0 flagged

  • Findingcitation
    § 1.3.3 Overfitting and memorization shortcuts

    In Gemini 3 verification, the model used the correct ARC-AGI color mapping in its reasoning chain even though the verification prompt did not mention ARC-AGI or the integer-to-color mapping.

    “Our verification model prompt does not mention”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [7] ARC Prize Foundation. Gemini 3 Deep Think Preview Verification on ARC-AGI-2. https:// huggingface.co/datasets/arcprize/arc_agi_v2_public_eval, 2026. https://arcprize.org/competitions/2024 https://arcprize.org/competitions/2025 https://arcprize.org/ https://github.com/arcprize/ARC-AGI-Community-Leaderboard https://github.com/arcprize/ARC-AGI-Community-Leaderboard https://github.com/arcprize/ARC-AGI https://huggingface.co/datasets/arcprize/arc_agi_v2_public_eval https://huggingface.co/datasets/arcprize/arc_agi_v2_public_eval

      Not checked in the free run.

  • Findingcitation
    § 1.3.1 Predictive power

    OpenAI's o1 and o3 systems first demonstrated test-time reasoning on ARC-AGI-1.

    “This was first demonstrated by OpenAI's breakthrough o1 and o3 systems (9) on ARC-AGI-1.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] François Chollet. OpenAI o3 Breakthrough High Score on ARC-AGI-Pub. https://arcprize.org/ blog/oai-o3-pub-breakthrough, December 2024.

      Not checked in the free run.

  • Findingreasoning
    § 1.3.1 Predictive power

    The latest generation of base LLMs (as of March 2026) still perform poorly on ARC-AGI-1.

    “The latest generation of base LLMs (as of March 2026) still perform poorly on ARC-AGI-1.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § Abstract

    Frontier AI systems, as of March 2026, score below 1% on ARC-AGI-3.

    “in contrast to frontier AI systems which, as of March 2026, score below 1%.”
    Quote in the paperSubstantial claim
  • Findingreasoning
    § 3.2 Production pipeline

    Throughput was highest when three to four environments were in development simultaneously by a single developer, at different stages of the pipeline.

    “Because ideation, implementation, playtesting, and revision progressed at different rates, throughput was highest when three to four environments were in development simultaneously by a single developer, at different stages of the pipeline.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 35
  • Classified as probably not a substantial claim. 49

  • May combine two claims.

    Findingown data
    § 1.1 ARC-AGI-1 and 2

    On average, a task from ARC-AGI-1 takes humans about 30 seconds to solve, while a task from ARC-AGI-2 takes humans about 300 seconds to solve.

    “On average, a task from ARC-AGI-1 takes humans about 30 seconds to solve, while a task from ARC-AGI-2 takes humans about 300 seconds to solve.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation
    § Lvl/8 Score

    StochasticGoose, Tufa Labs achieved 12.58% and completed 18 levels.

    “StochasticGoose, Tufa Labs (12.58%, first place) (18): A CNN with reinforcement learning to predict which actions would cause frame changes, encoding 64x64 frames through a four-layer convolutional network. It achieved 12.58% and completed 18 levels.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] Dries Smit. ARC3 Solution. https://github.com/DriesSmit/ARC3-solution, 2025.

      Not checked in the free run.

  • May combine two claims.

    Findingcitation
    § Lvl/8 Score

    In a variant of environment TR87, Opus 4.6 scores 0.0% with no harness and 97.1% with the Duke harness, yet in environment BP35, Opus 4.6 scores 0.0% under both configurations.

    “For example, in a variant of environment TR87, Opus 4.6 scores 0.0% with no harness and 97.1% with the Duke harness (11), yet in environment BP35, Opus 4.6 scores 0.0% under both configurations.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [11] Alexis Fox, Junlin Wang, Paul Rosu, and Bhuwan Dhingra. Hill-climbing arc-agi-3, 2026.

      Not checked in the free run.

  • May combine two claims.

    Findingown data
    § Lvl/8 Score

    Humans are able to reliably solve the environments within bounded time and action budgets, while current AI systems struggle to achieve consistent performance without significant external scaffolding.

    “Humans are able to reliably solve the environments within bounded time and action budgets, while current AI systems struggle to achieve consistent performance without significant external scaffolding.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § Lvl/8 Score

    The public set is materially easier than the private set.

    “because the public set is materially easier than the private set”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Lvl/8 Score

    At release, frontier models score on the official ARC-AGI-3 leaderboard as follows: Anthropic Opus 4.6 (Max) 0.50%, Google Gemini 3.1 Pro Preview 0.40%, OpenAI GPT 5.4 (High) 0.20%, xAI Grok-4.20 (Beta 0309 Reasoning) 0.10%.

    “At release, frontier models score on the official ARC-AGI-3 leaderboard as follows: Provider Model Score Anthropic Opus 4.6 (Max) 0.50% Google Gemini 3.1 Pro Preview 0.40% OpenAI GPT 5.4 (High) 0.20% xAI Grok-4.20 (Beta 0309 Reasoning) 0.10%”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Lvl/8 Score

    In evaluation, the Duke approach was able to solve all three public environments with action counts comparable to human performance.

    “In evaluation, this approach was able to solve all three public environments with action counts comparable to human performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § Lvl/8 Score

    Both winning approaches used an informed search approach, exploring as much of the action space of the environment as possible in the hope of encountering a winning combination by chance.

    “Both winning approaches used an informed search approach, exploring as much of the action space of the environment as possible in the hope of encountering a winning combination by chance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation
    § Lvl/8 Score

    The ARC-AGI-3 Preview Agent Competition ran for 30 days from July 18 to August 19, 2025.

    “The ARC-AGI-3 Preview Agent Competition (13) ran for 30 days from July 18 to August 19, 2025.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] Greg Kamradt. ARC-AGI-3 Preview: 30-Day Learnings. https://arcprize.org/blog/ arc-agi-3-preview-30-day-learnings, August 2025.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 3.5.2 Exploring graph-based state space construction and win probability estimation

    Pwin for the first level of ls20 is exactly 1 in 355.

    “Pwin for this level is exactly 1 in 355.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Lvl/8 Score

    The total recorded play time for all attempts was 427.9 hours.

    “The total recorded play time for all attempts was 427.9 hours.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Lvl/8 Score

    The median duration of an attempt was 7.4 minutes.

    “The median duration of an attempt was 7.4 minutes”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Lvl/8 Score

    Participants were members of the general public and not selected for any special training, abilities, or skill sets.

    “Participants were members of the general public and not selected for any special training, abilities, or skill sets”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Lvl/8 Score

    Symbolica AI's Arcgentica was able to solve all three public environments.

    “This approach was also able to solve all three public environments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Lvl/8 Score

    Successful attempts had a median duration of 8.1 minutes, whereas unsuccessful attempts had a median of 5.9 minutes.

    “Successful attempts had a median duration of 8.1 minutes, whereas unsuccessful attempts had a median of 5.9 minutes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation
    § Lvl/8 Score

    Blind Squirrel achieved 6.71%.

    “Blind Squirrel (6.71%, second place) (22): A directed state graph from observed frames.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Lvl/8 Score

    In total, we recorded 486 unique participants across 414 candidate environments, resulting in 2,893 total environment attempts.

    “In total, we recorded 486 unique participants across 414 candidate environments. This resulted in 2,893 total environment attempts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given
    § 1.3.3 Overfitting and memorization shortcuts

    Frontier LRMs demonstrate non-zero fluid intelligence and can adapt to tasks further away from their training distribution while still requiring extensive domain knowledge.

    “However, frontier LRMs demonstrate non-zero fluid intelligence and can thus adapt to tasks further away from their training distribution (while still requiring extensive domain knowledge).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 2.2 Intelligence as efficiency

    Action efficiency penalizes brute-forcing.

    “It penalizes”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 2.2 Intelligence as efficiency

    Action efficiency accounts for data efficiency and risk efficiency.

    “It accounts for data efficiency and risk efficiency: Fewer actions naturally translate to lower exposure to environmental hazards.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given
    § 2.1 ARC-AGI-3 goals

    In ARC-AGI-3, the agent is never told the objective nor provided instructions; it must autonomously infer mechanics including win conditions.

    “In particular, one of the most significant hurdles in ARC-AGI-3 is that the agent is never told the objective nor provided instructions. It must autonomously infer the mechanics of each new environment, including the win conditions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § 1.3.2 Known limits of LRM fluid intelligence

    Even the latest LRMs remain bottlenecked by human intelligence and show limited ability to cover novel domains, which is a key argument why they fall short of AGI.

    “Even the latest LRMs remain bottlenecked by human intelligence and show limited ability to cover novel domains, which is a key argument why they fall short of AGI.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Human reasoning capability is not bound by domain knowledge.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data
    § 1.3.1 Predictive power

    Modern-day models excel at reasoning, corroborated by ARC-AGI-2 progress.

    “Modern-day models excel at reasoning, precisely corroborated by ARC-AGI-2 progress.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 1.1 ARC-AGI-1 and 2

    Every ARC-AGI-2 task was human-calibrated with over 400 untrained participants to ensure 100% solvability.

    “Every task was human-calibrated with over 400 untrained participants to ensure 100% solvability.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 2.3 The ARC-AGI-3 environment format

    ARC-AGI-3 uses a turn-based interface designed to prioritize offline reasoning over real-time sensorimotor reflexes.

    “The benchmark utilizes a turn-based interface designed to prioritize offline reasoning over real-time sensorimotor”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given
    § 1.1 ARC-AGI-1 and 2

    ARC-AGI-2 was introduced in March 2025 to measure complexity scaling of AI reasoning on static tasks.

    “ARC-AGI-2 was introduced in March 2025 to measure complexity scaling of AI reasoning on static tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 2.1 ARC-AGI-3 goals

    ARC-AGI-3 uses interactive turn-based environments to evaluate exploration, modeling, goal-setting, and planning/execution.

    “Specifically, ARC-AGI-3 uses a set of interactive turn-based environments to evaluate a test-taker across four core functional components of agentic intelligence:”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning
    § 1.1 ARC-AGI-1 and 2

    Each ARC-AGI-1 task is unique, which rules out memorization.

    “Each task is unique, which rules out memorization.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 1.3.3 Overfitting and memorization shortcuts

    ARC-AGI-1 and ARC-AGI-2 were designed to be resistant to memorization shortcuts by using a private dataset and ensuring tasks were unique.

    “ARC-AGI-1 and ARC-AGI-2 were designed to be resistant to this style of memorization shortcuts, by using a private dataset (inaccessible to model developers) for official scoring and verification and making sure all tasks were reasonably unique – both distinct from each other and distinct from tasks found on the web.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § Lvl/8 Score

    This gap reflects not only differences in reasoning capability, but also limitations in exploration strategies, hypothesis revision, and efficient planning under uncertainty.

    “This gap reflects not only differences in reasoning capability, but also limitations in exploration strategies, hypothesis revision, and efficient planning under uncertainty.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § Lvl/8 Score

    The community leaderboard will be public, anybody can submit, scores self-reported, and ARC Prize foundation will not verify anything by default.

    “This leaderboard will be public. Anybody can submit to it, and scores will be self-reported. By default, the ARC Prize foundation will not verify anything on the community leaderboard”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 3.6 ARC-AGI-3 environment selection

    The public demonstration set is intentionally easier for both humans and AI, with a stronger emphasis on clarity and fun.

    “These environments are intentionally easier for both humans and AI, with a stronger emphasis on clarity and fun.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § Lvl/8 Score

    If we disregard level efficiency and simply look at actions across an entire environment, the longest levels dominate the score and reduce signal from short levels.

    “If we disregard level efficiency and simply look at actions across an entire environment, the longest levels dominate the score and reduce signal from short levels.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3.4 Environment design constraints

    ARC-AGI-3 environments are designed to be solvable by humans within a bounded play session of approximately 20 minutes.

    “Environments are designed to be solvable by humans within a bounded play session of approximately 20 minutes (but most environments can be solved in only a few minutes).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § Lvl/8 Score

    The 1.15 cap allows AI systems that are moderately more efficient than the human baseline to receive credit for that efficiency, while still preventing extreme outliers from distorting environment scores.

    “The 1.15 cap allows AI systems that are moderately more efficient than the human baseline to receive credit for that efficiency, while still preventing extreme outliers from distorting environment scores.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 3.4 Environment design constraints

    Each ARC-AGI-3 environment is required to be novel both with respect to preexisting video games and with respect to the previously created set of environments.

    “Each environment is required to be novel both with respect to preexisting video games, and with respect to the previously created set of environments.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § Lvl/8 Score

    We introduce a per-environment score cap to prevent the score from being inflated by high efficiency on a subset of levels.

    “Because the per-level score can exceed 1.0 (up to 1.15x), we introduce a per-environment score cap to prevent the score from being inflated by high efficiency on a subset of levels.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.4 Environment design constraints

    Each ARC-AGI-3 environment contains multiple mechanics.

    “Each environment contains multiple mechanics.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given
    § Lvl/8 Score

    ARC-AGI-3 introduces an interactive reasoning benchmark for evaluating agentic intelligence, focusing on a system's efficiency at acquiring new skills through exploration, model formation, goal inference, and planning in unfamiliar environments.

    “ARC-AGI-3 introduces an interactive reasoning benchmark for evaluating agentic intelligence, focusing on a system's efficiency at acquiring new skills through exploration, model formation, goal inference, and planning in unfamiliar environments.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Action efficiency relative to a human baseline is a valid measure of general intelligence.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § 3.5.1 Environment qualification

    The second regime extends to 1,000,000 steps and strengthened the constraint that non-tutorial levels must remain unbeaten under uninformed random play.

    “The second regime extends to 1,000,000 steps and strengthened the constraint that non-tutorial levels must remain unbeaten under uninformed random play.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given
    § 4 Measuring performance on ARC-AGI-3

    To complete ARC-AGI-3, test takers must use actions in two ways: exploration and execution.

    “To complete ARC-AGI-3, test takers must use actions in two ways: exploration (learning the environment mechanics and goals) and execution (carry out a strategy to reach a goal) to complete an environment.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 3.5.2 Exploring graph-based state space construction and win probability estimation

    The graph-based state-space analysis reveals structural properties of each environment, exposes cycles, estimates reachability, and quantifies stochastic solvability in a reproducible manner.

    “The graph reveals structural properties of each environment, exposes cycles, estimates reachability, and quantifies stochastic solvability in a reproducible manner.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § Lvl/8 Score

    Final scoring only measured the ability for AI systems to generalize to this hidden evaluation set.

    “Final scoring only measured the ability for AI systems to generalize to this hidden evaluation set.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given
    § 3.6 ARC-AGI-3 environment selection

    The private set is significantly more difficult for both humans and AI and is intentionally out-of-distribution relative to the public set.

    “These environments are significantly more difficult for both humans and AI, and are intentionally out-of-distribution relative to the public set.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § Lvl/8 Score

    Exactly 10 members of the public are tested on each environment.

    “Exactly 10 members of the public are tested on each environment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The human baseline derived from 10 public participants is representative of human performance.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 3.5.1 Environment qualification

    The first random regime runs for up to 50,000 steps and asserts that no level can be beaten by accident.

    “The first random regime runs for up to 50,000 steps and asserts that no level can be beaten by accident.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § Lvl/8 Score

    We cap the per-level efficiency an AI can receive at 1.15x human baseline.

    “To stop a single technical anomaly or outlier level from distorting an entire environment score, we cap the per-level efficiency an AI can receive at 1.15x human baseline.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3.5 Automated environment validation

    The validation pipeline is divided into deterministic system qualification and exploratory state-space analysis.

    “The validation pipeline is divided into two complementary layers: deterministic system qualification and exploratory state-space analysis.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given
    § 3.6 ARC-AGI-3 environment selection

    The public set does not comprehensively represent the mechanics found in the private set, reducing the risk of overfitting or targeted optimization.

    “To preserve evaluation integrity, the public set does not comprehensively represent the mechanics found in the private set, reducing the risk of overfitting or targeted optimization.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given
    § Lvl/8 Score

    AI performance is compared to human performance per-level, then aggregated per environment.

    “AI performance is compared to human performance per-level, then aggregated per environment.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § Lvl/8 Score

    AI performance gets normalized to human level action efficiency.

    “A defining characteristic of ARC-AGI-3 scoring is that AI performance gets normalized to human level action efficiency.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Action efficiency relative to a human baseline is a valid measure of general intelligence.

  • May combine two claims.

    Study design and conductcitation
    § 3.4 Environment design constraints

    All ARC-AGI-3 environments are strictly limited to Core Knowledge priors and seek to avoid similarities with existing games.

    “To ensure the benchmark remains a test of innate reasoning rather than acquired knowledge, all environments are strictly limited to Core Knowledge priors (20), and seek to avoid similarities with existing games.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [20] Elizabeth S. Spelke and Katherine D. Kinzler. Core knowledge. Developmental science, pages 89–96, 2007. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 3.5.1 Environment qualification

    A third 1,000,000-step sweep is run across all levels as a combined performance and fuzzing harness.

    “A third 1,000,000-step sweep is run across all levels. This combined performance and fuzzing harness testing.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 2.3 The ARC-AGI-3 environment format

    The environment's state does not change asynchronously from the agent's actions.

    “The environment's state does not change asynchronously from the agent's actions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1 Scoring methodology

    The formulation is inspired by the robotics navigation Success weighted by Path Length (SPL) metric.

    “This formulation is inspired by the robotics navigation Success weighted by Path Length (SPL) metric (4), which evaluates not only task completion but also path efficiency.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir R. Zamir. On evaluation of embodied navigation agents, 2018. source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3.4 Environment design constraints

    The first level in an ARC-AGI-3 environment functions as a tutorial level and is intentionally easy to both humans and AI.

    “The first level in an environment functions as a tutorial level and is intentionally easy (to both humans and AI).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.6 ARC-AGI-3 environment selection

    The public set shifts from a training resource to a demonstration interface, while the private set becomes the primary basis for evaluation.

    “The public set shifts from a training resource to a demonstration interface, while the private set becomes the primary basis for evaluation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § Lvl/8 Score

    This constraint may yield marginally lower reported scores than would be obtained under the environment's intrinsic action limit, but the resulting differential is negligible.

    “This constraint may yield marginally lower reported scores than would be obtained under the environment's intrinsic action limit. However, given the power-law decay of the scoring function, the resulting differential is negligible.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 2.3.2 The Action Space

    The small action space ensures that the complexity of the benchmark lies in the logic of the environment, not the difficulty of the controls.

    “This small action space ensures that the complexity of the benchmark lies in the logic of the environment, not the difficulty of the controls.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § Lvl/8 Score

    The upper-median human baseline provides a robust human capability baseline that is resistant to outlier performance while reflecting strong but representative human efficiency.

    “This provides a robust human capability baseline that is resistant to outlier performance while reflecting strong but representative human efficiency.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § Lvl/8 Score

    To our knowledge, ARC-AGI-3 is the only unsaturated general agentic intelligence benchmark as of March 2026.

    “To our knowledge, ARC-AGI-3 is the only unsaturated general agentic intelligence benchmark as of March 2026.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ARC-AGI-3 environments are novel and out-of-distribution.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 3.4 Environment design constraints

    ARC-AGI-3 environments are developed with a level-based structure, with at least six levels per environment.

    “Environments are developed with a level-based structure, with at least six levels per environment.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 4.1 Scoring methodology

    The ARC-AGI-3 scoring method scores the test taker by per-level action efficiency compared to a human baseline, normalized per environment, across all environments.

    “The ARC-AGI-3 scoring method seeks to score the test taker by its per-level action efficiency (as compared to a human baseline), normalized per environment, across all environments.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 3.1 The ARC-AGI-3 game studio

    The ARC-AGI-3 game studio model allowed creative environment design to be coupled with standardized interfaces, evaluation procedures, and validation criteria, enabling environment production to scale without sacrificing consistency.

    “This studio model allowed creative environment design to be coupled with standardized interfaces, evaluation procedures, and validation criteria which enabled environment production to scale without sacrificing consistency.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § Lvl/8 Score

    This ensures that introductory or tutorial-like levels have the smallest influence on the final score, while later levels contribute the most.

    “This ensures that introductory or tutorial-like levels have the smallest influence on the final score, while later levels, which typically require a more complete understanding of the environment's mechanics, contribute the most.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 4 Measuring performance on ARC-AGI-3

    Counting the total number of actions taken on first exposure to beat an environment accounts for both exploration and execution.

    “Counting the total number of actions taken on first exposure to beat an environment accounts for both of these.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 3.6 ARC-AGI-3 environment selection

    The private set covers a broader and more diverse set of mechanics with limited overlap with public environments, and probes greater adaptation capabilities involving deeper compositional reasoning while remaining entirely solvable by humans.

    “They cover a broader and more diverse set of mechanics with limited overlap with the mechanics found in the public environments, and they probe greater adaptation capabilities, involving deeper compositional reasoning (while remaining entirely solvable by humans, which is ensured by our human calibration process).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § Lvl/8 Score

    Participants were limited to a single attempt per environment and could not revisit previously completed levels.

    “Participants were limited to a single attempt per environment and could not revisit previously completed levels.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3.2 Production pipeline

    Automated validation was used to detect development bugs, trivial environments, and regressions before an environment advanced.

    “Throughout the process, automated validation was used to detect development bugs, trivial environments, and regressions before an environment advanced.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § Lvl/8 Score

    The per-environment cap limits the maximum environment score to the weighted fraction of levels completed.

    “The per-environment cap limits the maximum environment score to the weighted fraction of levels completed.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.5.2 Exploring graph-based state space construction and win probability estimation

    The acceptance threshold was that a random policy should not successfully solve a level more often than 1 in 10,000 times.

    “Our acceptance threshold was that a random policy should not successfully solve a level more often than 1 in 10,000 times.”
    Quote in the paperSubstantial claim
71 of 71

  • The paper relies on this without support.

    Human reasoning capability is not bound by domain knowledge.

    “Human reasoning capability is not bound by domain knowledge.”

  • The paper relies on this without support.

    Action efficiency relative to a human baseline is a valid measure of general intelligence.

    “A defining characteristic of ARC-AGI-3 scoring is that AI performance gets normalized to human level action efficiency.”

  • The paper relies on this without support.

    The human baseline derived from 10 public participants is representative of human performance.

    “Exactly 10 members of the public are tested on each environment.”

  • The paper relies on this without support.

    Frontier models' performance on ARC-AGI-3 is indicative of general intelligence.

    “Our intent with the official leaderboard is to accurately help the public sense how close frontier models are to human-level general intelligence.”

  • The paper relies on this without support.

    The ARC-AGI-3 environments are novel and out-of-distribution.

    “ARC-AGI-3 emphasizes novelty, compositional generalization, and out-of-distribution design”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • When a benchmark's accuracy saturates, the prevailing response is to retire it and replace it with a more challenging version.

    “When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version.”

    From Life After Benchmark Saturation: A Case Study of CORE-Bench (2026) · cites this paper as [21]

    Section 2.1 (p. 4) says the ARC-AGI series needs new versions as frontier AI capabilities advance, and the Conclusions (p. 21) describe ARC-AGI-3 as a response to limitations of earlier benchmarks. The paper does not say that saturated benchmarks are often retired or that replacing them with harder versions is the prevailing practice. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Humans can solve the ARC-AGI-3 environments, while frontier AI systems scored below 1% as of March 2026.

    “humans can solve the environments, while frontier AI systems scored below 1% as of March 2026 [1].”

    From Executable World Models for ARC-AGI-3 in the Era of Coding Agents (2026) · cites this paper as [1]

  • ARC-AGI-3 places agents in novel, abstract, turn-based environments where they must explore, infer goals, build models of environment dynamics, and plan action sequences without explicit natural language instructions.

    “ARC-AGI-3 places agents in novel, abstract, turn-based environments where they must explore, infer goals, build models of environment dynamics, and plan action sequences without explicit naturallanguage instructions [1].”

    From Executable World Models for ARC-AGI-3 in the Era of Coding Agents (2026) · cites this paper as [1]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Preview competition dates span more than 30 days

    Minor

    July 18 to August 19, 2025 is 32 elapsed days, rather than the stated 30 days.

  • Scoring assumes five levels, but environments require at least six

    Minor

    The design constraints require at least six levels per environment, while the scoring formula fixes the number of levels at five. The stated weights and denominator therefore do not cover every required level.

2 of 2

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 1

  1. [23]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https: //arxiv.org/abs/2201.11903, 2022. https://arxiv.org/abs/1911.01547 https://arcprize.org/blog/oai-o3-pub-breakthrough https://arcprize.org/blog/oai-o3-pub-breakthrough https://kaggle.com/competitions/abstraction-and-reasoning-challenge https://x.com/hsu_steve/status/1996034522308026435 https://x.com/hsu_steve/status/1996034522308026435 https://arcprize.org/blog/arc-agi-3-preview-30-day-learnings https://arcprize.org/blog/arc-agi-3-preview-30-day-learnings https://github.com/symbolica-ai/ARC-AGI-3-Agents https://lab42.global/past-challenges/2022-arcathon/ https://lab42.global/past-challenges/2023-arcathon/ https://github.com/DriesSmit/ARC3-solution https://drive.google.com/file/d/1vkEluaaJTzaZiJL69TkZovJUkPSDH5Xc/view https://drive.google.com/file/d/1vkEluaaJTzaZiJL69TkZovJUkPSDH5Xc/view https://arxiv.org/abs/1706.03762 https://github.com/wd13ca/ARC-AGI-3-Agents https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903 · arXiv

Cited by

Sorted by citations

Related

Authors
Foundation, A. P.
Corresponding author
Foundation, ARC Prize
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2603.24621
arXiv
2603.24621
OpenAlex
W7141572184
Cited by
10
References
0
Citations by year
2026: 10
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
Keywords
benchmark design, goal inference, action planning, AI evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-05.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-05.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review