CoChat
Sign Up
PreprintCited by 0AI-based Problem Solving and PlanningArtificial Intelligence ApplicationsReinforcement Learning in Robotics

Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

Open access
Li, Zongxia, Li, Zhongzhi, Shi, Yucheng, Wang, Ruhan, Yang, Junyao, Liu, Zhichao, Wu, Xiyang, Li, Anhao +7 more·2026·arXiv
DOI 10.48550/arxiv.2607.08964arXiv 2607.08964
Open this paper in CoChat — free accountOpen PDF

Abstract

AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.

AI snapshot

Written from the paper’s full text.

TL;DR

Long-Horizon-Terminal-Bench introduces 46 containerized terminal tasks across nine categories with dense subtask-based grading to evaluate long-horizon agent performance. Across 17 frontier models, even the best tested model, Grok 4.5, reaches only 28.3% pass@1 at R≥0.95, with average runs consuming 239 episodes, 9.8M tokens, and 88.9 minutes.

Key findings

  • The benchmark contains 46 tasks spanning nine categories; across 17 frontier models, runs average 239 episodes, 9.8M tokens, 88.9 minutes, and $10.8 per task.
  • Grok 4.5 is strongest, resolving 28.3% (13/46) at R≥0.95 and 19.6% at R≥1.0; the abstract reports mean pass rate across models of 6.4% at R≥0.95 and 3.2% at R≥1.0.
  • Dense rewards expose partial progress: 62.8% of runs achieve 0.05≤R<0.95, 28.5% reach R≥0.5, and near-misses (0.75≤R<0.95) occur 90 times versus 50 passes.
  • Failure analysis shows 79% of unresolved runs (518/660) time out, 19% are early exits, and 3% are harness errors; 14 early-exit runs with R≥0.75 are classified as false finishes.

Methods

  • Design: Terminal-Bench-style containerized tasks, each with a natural-language instruction, Docker image, task configuration, and oracle/simulator; tasks are decomposed into weighted subtasks graded deterministically via binary, continuous/thresholded, or episode-aggregating checks.
  • Data/sample: 46 tasks across nine categories, selected from 120 candidates with quality filtering and calibrated using Deepseek-V4-Pro under 1.5-hour budgets; domains include experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing.
  • Evaluation: 17 frontier models run under the Harbor framework with the Terminus-2 agent harness, except GPT-5.3 which uses Codex; metrics include pass@1 at R≥0.9, R≥0.95, and R≥1.0, mean normalized reward, episodes, time, tokens, and estimated cost, with a 90-minute timeout.

Limitations

  • The paper does not state a limitations section.
  • Evident from the text: the 90-minute timeout may truncate long-horizon tasks; GPT-5.3 uses a different agent harness (Codex) than the shared Terminus-2 setup; cost estimates rely on public list prices as of June 2026 and assume no prompt-cache discounts; the benchmark has 46 tasks, and some tasks are adapted from or inspired by prior benchmarks such as APEX-Agents.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2607.08964

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 48 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    GPT-5.6-sol and GPT-5.5 each achieve 15.2% (7/46) at R≥0.95.

    “GPT-5.6-sol and GPT-5.5 follow at 15.2% (7/46) each, with GPT-5.6-solposting the slightly higher mean reward.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The deterministic grader and hidden verifier correctly measure task completion.
  • Findingown data

    Grok 4.5 leads mean normalized reward at R = 0.51.

    “Mean normalized rewards follow a similar ordering: Grok 4.5 also leads this metric at R = 0.51, indicating that many models make partial progress on hard tasks without fully resolving them.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The deterministic grader and hidden verifier correctly measure task completion.
  • Findingown data

    14 false-finish runs with R ≥ 0.75 were identified.

    “We identify 14 such runs with R ≥ 0.75.”
    Quote in the paperSubstantial claim
  • Findingown data

    Kimi K2.6 exits at an average of only R = 0.11.

    “By contrast, Kimi K2.6 exits at an average of only R = 0.11, abandoning tasks while most of the work remains.”
    Quote in the paperSubstantial claim
  • Findingown data

    Early exits account for only 19% of unresolved runs.

    “By contrast, early exits that account for only 19% of unresolved runs often indicate overconfidence: the agent stops on its own despite not yet satisfying the hidden verifier.”
    Quote in the paperSubstantial claim
  • May combine two claims. 39
  • Classified as probably not a substantial claim. 20
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    Timed-out runs are not close to completion, with mean reward ranging only from 0.10 to 0.35 across models.

    “However, these timed-out runs are not close to completion, with mean reward ranging only from 0.10 to 0.35 across models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The 46 tasks in Long-Horizon-Terminal-Bench span 21 high-level categories.

    “The 46 tasks in Long-Horizon-Terminal-Bench span 21 high-level categories, including interactive games, multimodal audits, software engineering, systems and performance, experiment reproduction, earth and environment, scientific computing, robotics, materials science, health and medicine, climate science, and chip design (EDA).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Doubao Seed 2.1 Pro and MiniMax M3 achieve pass@R≥0.95 rates of 4.3% and 6.5% at roughly $5 and $6 per task.

    “while Doubao Seed 2.1 Pro and MiniMax M3 also lie on the frontier, achieving pass@R ≥ 0.95 rates of 4.3% and 6.5% at roughly $5 and $6 per task.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Kimi K2.7 Code's early exits average R = 0.51 and MiniMax M3's average R = 0.42.

    “Kimi K2.7 Code's early exits average R = 0.51 and MiniMax M3's average R = 0.42, indicating that these models tend to stop later than other models (R = 0.22 to 0.39), after completing a substantial fraction of the task.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    DeepSeek V4 Pro achieves the same 6.5% pass@R≥0.95 rate as MiniMax M3 at a similar cost.

    “DeepSeek V4 Pro sits just off the frontier, achieving the same 6.5% pass@R ≥ 0.95 rate as MiniMax M3 at a similar cost.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Kimi K2.6's strongest near-miss reaches R = 0.94 on grammar-fuzz-coverage-hunt, and it also comes close on spot-scheduler-traces (R = 0.90), poc-exploit-craft (R = 0.89), and nbody-accel-iterative (R = 0.89).

    “Its strongest near-miss reaches R = 0.94 on grammar-fuzz-coverage-hunt, and it also comes close on spot-scheduler-traces (R = 0.90), poc-exploit-craft (R = 0.89), and nbody-accel-iterative (R = 0.89).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The deterministic grader and hidden verifier correctly measure task completion.

  • May combine two claims.

    Findingown data

    Agents consume on average 9.8M tokens per task, with roughly 239 episodes and 88.9 minutes of execution time per run.

    “We evaluate 17 frontier models and find that agents consume on average 9.8M tokens per task, with roughly 239 episodes and 88.9 minutes of execution time per run, making Long-Horizon-Terminal-Bench substantially more demanding”
    • Also stated in: § 1 Introduction · plain
      “Across 17 frontier agents evaluated, rollouts on Long-Horizon-Terminal-Bench average 239 episodes, 9.8M tokens, and 88.9 minutes of wall-clock time before exit under a 90-minute timeout.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The mean pass rate across models is 6.4% at R≥0.95 and 3.2% at R≥1.0.

    “while the mean pass rate across models is just 6.4% and 3.2% under the two thresholds, respectively.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The deterministic grader and hidden verifier correctly measure task completion.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    GLM 5.2, Qwen3.6 Plus, GPT-5.4, and Hy3 each resolve only a single task at R≥0.95.

    “while GLM 5.2, Qwen3.6 Plus, GPT-5.4,and Hy3 each resolve only a single task.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The deterministic grader and hidden verifier correctly measure task completion.

  • May combine two claims.

    Findingreasoning

    GPT-5.4 is the most expensive overall (about $26 per task) with a much lower pass rate.

    “and GPT-5.4 is the most expensive overall (about $26 per task) with a much lower pass rate.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Grok 4.5 achieves the highest pass rate at about $11 per task.

    “Although Grok 4.5 achieves the highest pass rate, it does so at only about $11 per task, well below the most expensive models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    At R≥0.95, only 6.4% of runs pass while 30.8% score below 0.05.

    “even at R ≥ 0.95, only 6.4% of runs pass while 30.8% score below 0.05 (Figure 4).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The deterministic grader and hidden verifier correctly measure task completion.

  • May combine two claims.

    Findingown data

    Kimi K2.7 Code stops on duckdb-optimizer-closure at R = 0.92, GLM 5.2 stops on apex-ib244-matter at R = 0.90, and seven different models stop on apex-law433-matter between R = 0.80 and 0.87, each with roughly 20 minutes still remaining.

    “For example, Kimi K2.7 Code stops on duckdb-optimizer-closure at R = 0.92, GLM 5.2 stops on apex-ib244-matter at R = 0.90, and seven different models stop on apex-law433-matter between R = 0.80 and 0.87, each with roughly 20 minutes still remaining.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Kimi K2.6 solves zero tasks at R≥0.95, yet has five near-misses and a mean reward of 0.25.

    “For example, Kimi K2.6 solves zero tasks at the R ≥ 0.95 threshold, yet has five near-misses and a mean reward of 0.25.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The deterministic grader and hidden verifier correctly measure task completion.

  • May combine two claims.

    Findingown data

    Timeouts dominate for every model (79% of all unresolved runs), showing that many failures reflect incomplete progress rather than immediate breakdowns, while the smaller early-exit segments highlight cases where agents stop too soon despite not having fully solved the task.

    “Timeouts dominate for every model (79% of all unresolved runs), showing that many failures reflect incomplete progress rather than immediate breakdowns, while the smaller early-exit segments highlight cases where agents stop too soon despite not having fully solved the task.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    No single category dominates the benchmark: the largest groups are interactive games and multimodal audits.

    “No single category dominates the benchmark: the largest groups are interactive games and multimodal audits, with the remaining tasks spread across a broad range of scientific and engineering workflows.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In false-finish cases, the agent has completed most of the task, judges itself finished, and exits with substantial time remaining.

    “In these cases, the agent has completed most of the task, judges itself finished, and exits with substantial time remaining.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    This partial progress is often substantial rather than trivial: 223 runs (28.5%) reach R ≥ 0.5, and the mass in [0.85, 0.95) (68 runs) is still larger than the number of full passes.

    “This partial progress is often substantial rather than trivial: 223 runs (28.5%) reach R ≥ 0.5, and the mass in [0.85, 0.95) (68 runs) is still larger than the number of full passes.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The deterministic grader and hidden verifier correctly measure task completion.

  • May combine two claims.

    Findingown data

    120 candidate tasks were generated with quality filtering, and the final benchmark contains 46 tasks implemented in the Harbor format.

    “We generated 120 candidate tasks with quality filtering, where the final benchmark contains 46 tasks implemented in the Harbor format [22], all with containerized environments and a shared agent harness.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Grok 4.5 is the strongest model, achieving a 28.3% pass rate at R ≥ 0.95.

    “Yet the benchmark remains far from saturated: the strongest model, Grok 4.5, achieves only a 28.3% pass rate at R ≥ 0.95, while the mean pass rate across models is just 6.4%.”
    • Also stated in: § 5 Conclusion · plain
      “Grok 4.5 achieves the strongest overall performance, resolving 28.3% of tasks at R ≥ 0.95.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Grok 4.5 achieves the highest pass rate among the reported models while remaining on the Pareto frontier of cost against pass rate at R ≥ 0.95.

    “Grok 4.5 achieves the highest pass rate among the reported models while remaining on the Pareto frontier, and Hy3 stays at the low-cost end of that frontier.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    On Long-Horizon-Terminal-Bench, agents can often string together many locally correct actions, but they cannot reliably turn that progress into a finished artifact before the horizon expires.

    “On Long-Horizon-Terminal-Bench, the more common failure is different: agents can often string together many locally correct actions, but they cannot reliably turn that progress into a finished artifact before the horizon expires.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Short-horizon benchmarks primarily measure whether agents can act correctly; Long-Horizon-Terminal-Bench additionally measures whether they can budget a long horizon, and the latter appears to be the binding constraint.

    “Short-horizon benchmarks primarily measure whether agents can act correctly; Long-Horizon-Terminal-Bench additionally measures whether they can budget a long horizon, and the latter appears to be the binding constraint.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The subtask-based grading provides partial credit and a much denser signal about long-horizon performance than binary solved/unsolved grading.

    “This design provides partial credit and a much denser signal about long-horizon performance, revealing how far agents get through complex workflows rather than only whether they reach the final goal.”
    • Also stated in: § 3.2 Dense Reward Is Necessary to Rank Models · plain
      “A key contribution of Long-Horizon-Terminal-Bench is dense, subtask-level grading rather than binary pass/fail, which is essential at this difficulty level.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent work has demonstrated impressive performance on short, well-scoped tasks such as fixing a code repository issue, completing a coding ticket, or issuing a handful of shell commands.

    “Recent work has demonstrated impressive performance on short, well-scoped tasks such as fixing a code repository issue, completing a coding ticket, or issuing a handful of shell commands [22].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Existing terminal benchmarks largely focus on relatively simple problems that finish within a few minutes and are typically evaluated only by their final outcome.

    “However, existing terminal benchmarks largely focus on relatively simple problems that finish within a few minutes and are typically evaluated only by their final outcome.”
    • Also stated in: § 1 Introduction · plain
      “Dominant terminal and software-engineering benchmarks typically evaluate within a few minutes to at most an hour, and grade agents solely on whether the final state satisfies a test suite [3,5].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [3]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    The false-finish pattern suggests that current agents systematically overestimate completion and under-invest in final verification.

    “The falsefinish pattern suggests that current agents systematically overestimate completion and under-invest in final verification.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Early exits reveal two distinct abilities: whether a model chooses to stop prematurely, and how accurately it can judge how much work remains before the task is truly complete.

    “Early exits reveal two distinct abilities: whether a model chooses to stop prematurely, and how accurately it can judge how much work remains before the task is truly complete.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    AI agents built on large language models are rapidly improving at autonomous decision making and tool use.

    “AI agents built on large language models (LLMs) are rapidly improving at autonomous decision making and tool use [2,6].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [2]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [6]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    SWT-Bench studies test generation and validation for real-world bug fixes, and Terminal-Bench places agents in containerized terminals to complete realistic technical tasks such as system configuration, model training, and research-code reproduction.

    “Subsequent benchmarks further broaden the scope of software-agent evaluation: SWT-Bench [26] studies test generation and validation for real-world bug fixes, and Terminal-Bench [22] places agents in containerized terminals to complete realistic technical tasks such as system configuration, model training, and research-code reproduction.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [26]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [22]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent agent benchmarks have shifted from static question answering and isolated programming exercises toward interactive, execution-grounded tasks in realistic work environments.

    “Recent agent benchmarks have shifted from static question answering and isolated programming exercises toward interactive, execution-grounded tasks in realistic work environments [47,52].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [47]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [52]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Subtasks are evaluated using objective evidence from the final container state, such as files, outputs, test results, or simulator responses.

    “Subtasks are chosen to correspond to meaningful intermediate goals in the workflow, and each is evaluated using objective evidence from the final container state, such as files, outputs, test results, or simulator responses.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Long-Horizon-Terminal-Bench combines extended, many-step autonomy tasks with subtask-level grading, thereby enabling a finer-grained characterization of partial progress and failure throughout long-horizon execution.

    “Long-Horizon-Terminal-Bench is designed to complement this line of work by combining extended, many-step autonomy tasks with subtask-level grading, thereby enabling a finer-grained characterization of partial progress and failure throughout long-horizon execution.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Long-Horizon-Terminal-Bench decomposes each task into semantically meaningful subtasks, each checked by deterministic, environment-grounded graders, providing dense partial-credit signals that capture how far an agent progresses and not only whether it finishes.

    “Long-Horizon-Terminal-Bench applies the same core intuition at evaluation time: each task is decomposed into semantically meaningful subtasks, and each subtask is checked by deterministic, environment-grounded graders.”
    • Also stated in: § 5 Conclusion · plain
      “Unlike outcome-only benchmarks, Long-Horizon-Terminal-Bench uses deterministic, environment-grounded subtasks to provide dense partial-credit signals, allowing evaluation to capture not only whether an agent finishes a task, but also how far it progresses.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The deterministic, environment-grounded subtask graders validly measure semantically meaningful progress on each task.

  • Classified as probably not a substantial claim.

    Interpretationcitation

    The false-finish phenomenon is the long-horizon analogue of the verification failures identified in Terminal-Bench 2.0, such as premature termination, weak verification, and missed final checks.

    “This phenomenon is the long-horizon analogue of the verification failures identified in Terminal-Bench 2.0, such as premature termination, weak verification, and missed final checks [22].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [22]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    METR's time-horizon analysis estimates the human-time duration of tasks that frontier agents can complete at a fixed success rate.

    “METR's time-horizon analysis [14] makes this perspective explicit by estimating the human-time duration of tasks that frontier agents can complete at a fixed success rate.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Closing this gap for long-horizon tasks will require not only stronger local reasoning, but also better self-verification and more calibrated stopping decisions.

    “Closing this gap for long-horizon tasks will require not only stronger local reasoning, but also better self-verification and more calibrated stopping decisions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    SWE-Bench evaluates repository-level issue resolution on real GitHub projects, while LiveCodeBench provides contamination-aware evaluation of code generation and related coding abilities.

    “In software engineering, SWE-Bench [12] evaluates repository-level issue resolution on real GitHub projects, while LiveCodeBench [11] provides contamination-aware evaluation of code generation and related coding abilities such as execution, self-repair, and test-output prediction.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [12]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [11]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Horizon length is treated as a first-class capability variable and is grounded in human-calibrated evaluations such as RE-Bench and HCAST, which compare agents against human performance on realistic research-engineering, software-engineering, cybersecurity, and reasoning tasks.

    “This framing treats horizon length as a first-class capability variable and is grounded in human-calibrated evaluations such as RE-Bench [43] and HCAST [34], which compare agents against human performance on realistic research-engineering, software-engineering, cybersecurity, and reasoning tasks.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [43]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [34]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductreasoning

    Public checks only validate command-line behavior, file formats, and a small number of simple examples, and carry relatively low reward weight.

    “To discourage overfitting to superficial public tests, the public checks only validate command-line behavior, file formats, and a small number of simple examples, and they carry relatively low reward weight.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    More recent efforts push toward substantially harder and longer workflows, including FrontierSWE, SWE-EVO, SWE-Marathon, and Edge-Bench, which target expert-level, multi-stage, or ultra-long-horizon tasks.

    “More recent efforts push toward substantially harder and longer workflows, including FrontierSWE [3], SWE-EVO [15], SWE-Marathon [5], and Edge-Bench, which target expert-level, multi-stage, or ultra-long-horizon tasks.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [3]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [15]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Binary grading collapses a wide range of partially successful runs into a single failure bucket, whereas dense subtask-level rewards preserve the signal needed to distinguish model progress at this difficulty level.

    “Binary grading collapses a wide range of partially successful runs into a single failure bucket, whereas dense subtask-level rewards preserve the signal needed to distinguish model progress at this difficulty level.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    A binary pass/fail metric would collapse all false-finish runs into the same failure bucket, whereas dense rewards show how far each agent gets before stopping.

    “A binary pass/fail metric would collapse all of these runs into the same failure bucket, whereas dense rewards show how far each agent gets before stopping.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    On Terminal-Bench 2.0, most trials complete in under 20 minutes, and many failures are execution-related, including disobeying the specification, repeating steps, and missing termination conditions.

    “On Terminal-Bench 2.0, most trials complete in under 20 minutes, and trajectory-level error analyses find that many failures are execution-related, including disobeying the specification, repeating steps, and missing termination conditions [22].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    Process reward models provide supervision over intermediate reasoning steps rather than only final answers, yielding denser signals than outcome-only supervision.

    “Process reward models provide supervision over intermediate reasoning steps rather than only final answers, yielding denser signals than outcome-only supervision [13,18].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [13]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [18]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    These benchmarks measure progress in coding and terminal agents, but many still primarily measure end-to-end success or failure, providing limited visibility into how much useful progress an agent made before failing.

    “These benchmarks measure progress in coding and terminal agents, but many still primarily measure end-to-end success or failure, providing limited visibility into how much useful progress an agent made before failing.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Each Long-Horizon-Terminal-Bench task is graded by a deterministic grader that runs inside the container at the end of a rollout.

    “Each Long-Horizon-Terminal-Bench task is graded by a deterministic grader that runs inside the container at the end of a rollout.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Rubric-based approaches decompose evaluation into individually assessable criteria or weighted dimensions of quality, making it possible to score complex outputs beyond a single holistic judgment.

    “In parallel, rubric-based approaches decompose evaluation into individually assessable criteria or weighted dimensions of quality, making it possible to score complex outputs beyond a single holistic judgment [30,33,38].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [30]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [33]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [38]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Tightening the threshold to R ≥ 1.0 collapses most models to zero pass rate, showing that partial-credit evaluation captures progress that a binary threshold misses.

    “Tightening the threshold to R ≥ 1.0 collapses most models to zero pass rate, showing that partial-credit evaluation captures progress that a binary threshold misses.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The deterministic grader and hidden verifier correctly measure task completion.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The official gold solution is required to achieve a verifier score of 1.0 on the full hidden evaluation suite.

    “The official gold solution is required to achieve a verifier score of 1.0 on the full hidden evaluation suite.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Recent work characterizes long-horizon autonomy in terms of the duration of tasks that agents can complete reliably, rather than aggregate performance on a fixed benchmark.

    “Recent work has increasingly characterized long-horizon autonomy in terms of the duration of tasks that agents can complete reliably, rather than aggregate performance on a fixed benchmark [14,43].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [14]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [43]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Open-source agents such as Codex and OpenClaw package nontrivial scaffolding around the base model, including prompting, tool orchestration, context management, and action-selection policies.

    “Opensource agents such as Codex [27] and OpenClaw [29] package nontrivial scaffolding around the base model, including prompting, tool orchestration, context management, and action-selection policies.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [27]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [29]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Dense rewards reveal that many runs make meaningful but incomplete progress, exposing failure patterns that binary pass/fail would miss.

    “Dense rewards further reveal that many runs make meaningful but incomplete progress, exposing failure patterns that binary pass/fail would miss.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The deterministic, environment-grounded subtask graders validly measure semantically meaningful progress on each task.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Open frameworks such as SWE-agent, OpenHands, and the Terminus family provide reusable interfaces for autonomous repository- and terminal-based work.

    “Open frameworks such as SWEagent [48], OpenHands [41], and the Terminus family introduced alongside Terminal-Bench [22] provide reusable interfaces for autonomous repository-and terminal-based work.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [48]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [41]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [22]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

54 of 54

  • The paper relies on this without support.

    The deterministic grader and hidden verifier correctly measure task completion.

    “Each Long-Horizon-Terminal-Bench task is graded by a deterministic grader that runs inside the container at the end of a rollout.”
    “The official gold solution is required to achieve a verifier score of 1.0 on the full hidden evaluation suite.”

  • The paper relies on this without support.

    The benchmark tasks are not contaminated in the training data of the evaluated models.

    “We benchmark GPT-5.6-sol[28], GPT-5.5 [28], GPT-5.4,GPT-5.3Codex, DeepSeek V4 Pro [4], Gemini 3.1 Pro [9], GLM 5.1 [8], GLM 5.2, Kimi K2.6 [24], Kimi K2.7 Code [25], MiniMax M3 [23], Qwen3.7 Max [32], Qwen3.6 Plus [31], Doubao Seed 2.1 Pro [1], Hy3 [37], Grok 4.20 [45], and Grok 4.5 [46].”

  • The paper relies on this without support.

    The deterministic, environment-grounded subtask graders validly measure semantically meaningful progress on each task.

    “each task is decomposed into semantically meaningful subtasks, and each subtask is checked by deterministic, environment-grounded graders.”

  • The paper relies on this without support.

    The 46 tasks and the shared terminus-2 agent setup are representative enough that the observed failure patterns generalize to long-horizon terminal tasks and current agents generally.

    “Our analysis suggests that the main bottleneck is not only local reasoning, but reliable long-horizon completion.”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Unresolved-run denominator excludes 72 runs

    Major

    The paper reports 17 × 46 = 782 runs, of which 50 passed, leaving 732 unresolved-not the 660 used in the termination-cause analysis. This changes the reported share of unresolved runs that timed out.

  • Category count changes from nine to 21

    Minor

    The task-composition section says the 46 tasks span 21 high-level categories and directs readers to Figure 2, but the paper describes Figure 2's taxonomy as nine categories.

  • Interactive games are incorrectly named among the largest categories

    Minor

    Figure 2 assigns interactive games four tasks, versus seven for software and reverse engineering and six for several other categories. Interactive games therefore are not one of the largest groups.

  • Conclusion calls the agent harness shared despite an exception

    Minor

    The conclusion says all 17 models were evaluated under a shared Terminus-2 agent, but the experiments section explicitly says GPT-5.3 used Codex instead.

  • Reported run time exceeds the stated timeout

    Minor

    The paper describes a 90-minute timeout, but Table 1 reports 176.1 minutes per task for Hy3. The text does not explain how that average execution time fits within the stated timeout.

  • Near-completion reward bin has two different counts

    Minor

    The text assigns 68 runs to [0.85, 0.95), whereas Figure 4 shows 59 runs in that bin.

6 of 6

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 2

  1. DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026. URL https: //arxiv.org/abs/2606.19348.
  2. Shangheng Du, Jiabao Zhao, Jinxin Shi, Zhentao Xie, Xin Jiang, Yanhong Bai, and Liang He. A survey on the optimization of large language model-based agents. ACM Computing Surveys, 58(9):1-37, 2026.

Authors
Li, Z., Li, Z., Shi, Y., Wang, R., Yang, J., Liu, Z., Wu, X., Li, A., Huang, C., Shang, E., Yu, Y., Liu, N., Sun, L., Mi, H., & Liang, L.
Affiliations
  • Tencent (China) (CN)
  • National University of Singapore (SG)
  • University of Maryland, College Park (US)
  • Indiana University (US)
  • Hong Kong Polytechnic University (HK)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2607.08964
arXiv
2607.08964
OpenAlex
W7168172066
Cited by
0
References
0
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
partial credit, long-horizon planning, failure mode analysis, software engineering tasks, multimodal analysis, interactive games, scientific computing, LLM agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-16.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-16.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review