StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling

Probably not a substantial claim: may not pass S3 (load-bearing).

The BusinessBench dataset contains 477 eligible instances across seven task families.

“The dataset contains 477 eligible instances across seven task families.”

• cites Yang et al., 2026 sourcePartly supported

  The Experiment Setups and Table I in Methodology describe seven business-relevant tasks. Table I lists 100 instances for each of six tasks and 50 for the seventh—a total of 650, not 477. The cited passages do not identify this suite as BusinessBench or describe 477 eligible instances. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Agentic Harness Engineering uses execution observability to evolve coding-agent tools, middleware, prompts, and memory from trajectory feedback.

“Agentic Harness Engineering uses execution observability to evolve coding-agent tools, middleware, prompts, and memory from trajectory feedback (Lin et al., 2026;Gu, 2026).”

• cites Lin et al., 2026 sourceSupported
• cites Gu, 2026Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Self-Harness lets an agent mine its weaknesses, propose harness modifications, and promote them through regression-aware validation.

“Self-Harness lets an agent mine its weaknesses, propose harness modifications, and promote them through regression-aware validation (Zhang et al., 2026).”

• cites Zhang et al., 2026 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Terminal-Bench 2.1 contains 89 tasks.

“Terminal-Bench 2.1 (Merrill et al., 2026) contains 89 tasks.”

• cites Merrill et al., 2026 sourcePartly supported

  The Introduction (p. 2) and Dataset Construction (p. 3) say that Terminal-Bench 2.0 contains 89 tasks. Our copy does not establish that Terminal-Bench 2.1 has the same task count, so the cited paper supports the number but not the version in the claim. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Better Harnesses, Smaller Models studies automated harness adaptation as a way for lower-cost models to recover much of the performance of larger models.

“Better Harnesses, Smaller Models studies automated harness adaptation as a way for lower-cost models to recover much of the performance of larger models (Yang et al., 2026).”

• cites Yang et al., 2026 sourceSupported

The quote only partly states this claim.

Even the entire $52.22 adaptation-and-evaluation campaign uses about 1/11 of the public GPT-5.6 Sol max submission's recorded cost.

“Even the entire $52.22 adaptation-and-evaluation campaign uses about 1/11 of that amount.”

• AssumptionRests on an assumption: The public submission records and OpenAI-reported scores are accurate and comparable to the authors' evaluation conditions.

May combine two claims.

Budget Approval improves from 62.91% to 75.12% (+12.21), and Machine Operating improves from 90.79% to 100.00% (+9.21).

“Budget Approval improves from 62.91% to 75.12% (+12.21), and Machine Operating improves from 90.79% to 100.00% (+9.21).”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

gpt2-codegolf is 0/5 under the standard timeout, but DeepSeek-V4-Flash with StateM solves it in 3 of 5 trials under an extended per-task timeout.

“gpt2-codegolf is 0/5 under the standard timeout, but DeepSeek-V4-Flash with StateM solves it in 3 of 5 trials under an extended per-task timeout.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

Sol xhigh with StateM records 95.28% raw accuracy on Terminal-Bench 2.1, compared with the 84.9% GPT-5.6 Sol xhigh reference.

“Sol xhigh with StateM records 424 445 = 95.28%raw accuracy, reported as 95.3% at one decimal, compared with the 84.9% GPT-5.6 Sol xhigh reference.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

RefactorBench changes from 80.56% to 77.78% (-2.78), and WooCommerce Stock changes from 92.59% to 88.89% (-3.70).

“RefactorBench changes from 80.56% to 77.78% (-2.78), and WooCommerce Stock changes from 92.59% to 88.89% (-3.70).”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

The DeepSeek evidence uses 2.65% of the public GPT-5.6 Sol max submission's recorded cost, or 1/37.8 as much; its standard-timeout full-suite score is 88.09%, compared with 83.37% raw for the matched public GPT artifact.

“The DeepSeek evidence uses 2.65% of that submission's recorded cost, or 1/37.8 as much; its standard-timeout full-suite score is 88.09%, compared with 83.37% raw for the matched public GPT artifact.”

• AssumptionRests on an assumption: The public submission records and OpenAI-reported scores are accurate and comparable to the authors' evaluation conditions.

May combine two claims.

With GPT-5.5 xhigh, the StateM system records 92.1% on Terminal-Bench 2.1, compared with the 83.1% GPT-5.5 Codex reference.

“With GPT-5.5 xhigh, our StateM system records 92.1% on Terminal-Bench 2.1, compared with the 83.1% GPT-5.5 Codex reference.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

GPT-5.5 xhigh with StateM reaches 92.1% accuracy on Terminal-Bench 2.1, compared with an 83.1% GPT-5.5 reference and numerically close to the 91.9% next-generation reference.

“On Terminal-Bench 2.1, GPT-5.5 xhigh with StateM reaches 92.1% accuracy, compared with an 83.1% reference and surpassing GPT-5.6 Sol Ultra (91.9%).”

• Also stated in: § Results and scope. · plain

  “At the quality frontier, GPT-5.5 xhigh with StateM reaches 92.1% accuracy on Terminal-Bench 2.1, compared with the 83.1% GPT-5.5 reference.”

• Also stated in: § Results and scope. · plain

  “Its 92.1% mean is also numerically close to the 91.9% next-generation reference.”

May combine two claims.

The complete DeepSeek final-score evidence costs $15.20 in realized API charges; provider-specific adaptation costs $37.02, bringing all recorded DeepSeek adaptation and final-evaluation expenditure to $52.22.

“The complete DeepSeek final-score evidence costs $15.20 in realized API charges.Provider-specific adaptation costs $37.02, bringing all recorded DeepSeek adaptation and final-evaluation expenditure to $52.22.”

• AssumptionRests on an assumption: The model cost figures reflect the true economic cost of the runs.

May combine two claims.

With no GPT-5.6-specific change to its control logic, the frozen profile raises GPT-5.6 Luna from 76.7% to 85.4%, a gain of 8.7 points.

“With no GPT-5.6-specificchange to its control logic, the profile accompanies the 84.9% to 95.28% Sol xhigh difference reported above and raises GPT-5.6 Luna from 76.7% to 85.4%, a gain of 8.7 points.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

Provider-specific adaptation raises the full 89-task standard-timeout result to 392/445 = 88.09%, a gain of 5.39 points over the 82.7% full-suite baseline.

“Starting from the same runtime, runbook structure, applicable controls, and golden rules, provider-specific adaptation raises the full 89-task standard-timeout result to 392 445 = 88.09%, a gain of 5.39 points over the 82.7% full-suite baseline.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

On BusinessBench, development performance improves from 86.07% to 91.71%, and performance across all treated instances improves from 84.76% to 88.72%.

“Development performance improves from 86.07% to 91.71%, while performance across all treated instances improves from 84.76% to 88.72%.”

May combine two claims.

The DeepSeek cost amounts are approximately 38.9× and 11.0× lower than the GPT reference amounts.

“These amounts are approximately 38.9× and 11.0× lower, respectively.”

May combine two claims.

Combining the disclosed extended-timeout trials with the 88-task common-core result yields a descriptive full-suite aggregate of 395/445 = 88.76%, which rounds to 88.8% and matches the separately reported GPT-5.6 Sol max score at one-decimal precision.

“Combining those disclosed trials with the 88-task common-core result yields a descriptive full-suite aggregate of 392 + 3 445 = 395 445 = 88.76%, which rounds to 88.8% and matches the separately reported GPT-5.6 Sol max score at one-decimal precision.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

The frozen runbook raises GPT-5.6 Luna from 76.7% to 85.4%, numerically above the 84.9% Sol xhigh reference.

“The frozen profile also raises GPT-5.6 Luna from 76.7% to 85.4%, numerically above the 84.9% Sol xhigh reference.”

• Also stated in: § Results and scope. · plain

  “The frozen runbook also raises GPT-5.6 Luna from 76.7% to 85.4%, a gain of 8.7%.”

• Also stated in: § Results and scope. · plain

  “This result is numerically above the 84.9% Sol xhigh reference.”

May combine two claims.

Across five trials, GPT-5.5 with StateM solves 88 of the 89 tasks at least once, corresponding to 98.9% five-trial task coverage.

“Across five trials, GPT-5.5 with StateM solves 88 of the 89 tasks at least once, corresponding to 98.9% five-trial task coverage.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

The complete DeepSeek final-score evidence costs approximately $15 in API usage, compared with $574.68 reported for the GPT reference.

“The complete DeepSeek final-score evidence costs approximately $15 in API usage, compared with the $574.68 reported for the GPT reference.”

• Also stated in: § Results and scope. · plain

  “The complete DeepSeek final-score evidence costs approximately $15 in realized API charges, compared with $574.68 reported for the GPT-5.6Sol max reference.”

May combine two claims.

Development performance improves from 86.07% to 91.71% (+5.64), and performance over all treated instances improves from 84.76% to 88.72% (+3.96).

“Development performance improves from 86.07% to 91.71% (+5.64), and performance over all treated instances improves from 84.76% to 88.72% (+3.96).”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

The StateM reference differences are +9.0 points on GPT-5.5 and +10.4 points on GPT-5.6 Sol.

“The StateM reference differences are +9.0 points on GPT-5.5 and +10.4 points on GPT-5.6 Sol.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

Concrete practices transfer unchanged across the GPT-5.5 and GPT-5.6 generations, but do not transfer unchanged to DeepSeek-V4 Flash.

“Concrete practices transfer unchanged across the GPT-5.5 and GPT-5.6 generations, but do not transfer unchanged to DeepSeek-V4 Flash.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Early WooCommerce control misses the invariants that connect its systems.

“RefactorBench receives excessive compatibility procedure, retrieval-oriented WebArena receives a workflow that is too heavy, and early WooCommerce control misses the invariants that connect its systems.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Attendance shows that the correct policy can be not to introduce a StateM workflow at all.

“Attendance shows that the correct policy can be not to introduce a StateM workflow at all.”

May combine two claims.

GPT-5.6 Sol xhigh + StateM records 95.28% raw and $1,062.95 submission-reported model cost, and DeepSeek-V4-Flash + StateM records 88.76% descriptive accuracy and $15.20 realized final-evaluation expenditure.

“GPT-5.6 Sol xhigh + StateM at 95.28% raw and $1,062.95submission-reported model cost, and DeepSeek-V4-Flash + StateM at 88.76% descriptive accuracy and $15.20 realized final-evaluation expenditure.”

May combine two claims.

The treated set contains 405 instances across six families, or 810 genuine treatment/control executions.

“The treated set therefore contains 405 instances across six families, or 810 genuine treatment/control executions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the reported formal run submission, the authors use AWS m7i.4xlarge because Daytona would encounter sandbox/verifier/network/pytorch-install timeout more frequently.

“For the reported formal run submission, we use AWS m7i.4xlarge, because daytona would encounter sandbox/verifier/network/pytorch-install timeout more frequently, and personal Macbook Pro 2025 (M4 chip) is not compatible with tune-mjcf kernel requirement.”

May combine two claims.

After replacing generic control with WooCommerce invariants, a post-evaluation matched rerun improves overall performance from 86.42% to 90.12%, including an improvement from 85.37% to 90.24% on the reused held-out split.

“After replacing generic control with these invariants, a post-evaluation matched rerun improves overall performance from 86.42% to 90.12%, including an improvement from 85.37% to 90.24% on the reused held-out split.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

At the quality frontier, GPT-5.6 Sol xhigh with a runbook frozen from GPT-5.5 records 95.3% raw accuracy on Terminal-Bench 2.1, or 424 successful trials out of 445, with at least one success on every task.

“At the quality frontier, GPT-5.6 Sol xhigh with a runbook frozen from GPT-5.5 records 95.3% raw accuracy on Terminal-Bench 2.1, or 424 successful trials out of 445, with at least one success on every task.”

May combine two claims.

WebArena's frozen evaluation contains only three non-tied A/B outcomes across 25 instances: development has 2 wins, 10 ties, and 0 losses; held-out has 0 wins, 12 ties, and 1 loss.

“Its frozen evaluation contains only three non-tied A/B outcomes across 25 instances: development has 2 wins, 10 ties, and 0 losses; held-out has 0 wins, 12 ties, and 1 loss.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

Probably not a substantial claim: may not pass S4 (based).

A personal MacBook Pro 2025 (M4 chip) is not compatible with the tune-mjcf kernel requirement.

“For the reported formal run submission, we use AWS m7i.4xlarge, because daytona would encounter sandbox/verifier/network/pytorch-install timeout more frequently, and personal Macbook Pro 2025 (M4 chip) is not compatible with tune-mjcf kernel requirement.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The public $574.68 GPT-5.6 Sol max submission is plotted at its matched 83.37% raw score.

“The public $574.68GPT-5.6 Sol max submission is plotted at its matched 83.37% raw score.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The actual usage was less than $125 of the $200 budget.

“Thanks again to Codex's pro plan so we can have chance to do such an individual research work within 200$ budget, and the actual usage is less than 125$ of it.”

May combine two claims.

WebTest later matched reruns reach 100.0% on development and 99.5% on the reused held-out split for both arms, or 99.75% overall.

“WebTest is nearly saturated: later matched reruns reach 100.0% on development and 99.5% on the reused held-out split for both arms, or 99.75% overall.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

Probably not a substantial claim: may not pass S3 (load-bearing).

The horizontal 88.8% reference denotes the separately reported GPT-5.6 Sol max score.

“The horizontal 88.8% reference denotes the separately reported GPT-5.6 Sol max score.”

Probably not a substantial claim: may not pass S3 (load-bearing).

RefactorBench receives excessive compatibility procedure.

“RefactorBench receives excessive compatibility procedure, retrieval-oriented WebArena receives a workflow that is too heavy, and early WooCommerce control misses the invariants that connect its systems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Retrieval-oriented WebArena receives a workflow that is too heavy.

“RefactorBench receives excessive compatibility procedure, retrieval-oriented WebArena receives a workflow that is too heavy, and early WooCommerce control misses the invariants that connect its systems.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The main computation resource for the work was a personal Codex pro plan and a MacBook Pro 2025 (M4 chip).

“The main computation resouce, is personal Codex pro plan and Macbook Pro 2025 (M4 chip).”

May combine two claims.

The profile developed with GPT-5.5 transfers unchanged to GPT-5.6 Sol xhigh, where the reference difference grows from +9.0 to +10.4 points.

“The profile developed with GPT-5.5 transfers unchanged to GPT-5.6 Sol xhigh, where the reference difference grows from +9.0 to +10.4 points.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

WooCommerce improves when generic procedure is replaced by the actual cross-system invariants.

“WooCommerce improves when generic procedure is replaced by the actual cross-system invariants.”

May combine two claims.

In a post-evaluation matched rerun, RefactorBench overall score changes from 76.39% to 79.17%; on the development split, the revised StateM arm improves from 75.00% to 80.56%; on the reused held-out split, it remains 77.78%.

“In a post-evaluation matched rerun, the overall score changes from 76.39% to 79.17%.On the development split, the revised StateM arm improves from 75.00% to 80.56%; on the reused held-out split, it remains 77.78%.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

On a personal MacBook Pro 2025, GPT-5.5 xhigh reaches around 91% pass rate on Terminal Bench 2.1.

“On a personal Macbook Pro 2025, GPT-5.5 xhigh can reaches around 91% pass rate on Terminal Bench 2.1.”

May combine two claims.

The baseline agent scores 0/5 on configure-git-webserver because it does not reliably preserve and validate the required end-to-end live state; with StateM, the result moves from 0/5 to 5/5.

“The baseline agent can configure Git, SSH, hooks, and an HTTP server, yet scores 0/5 because it does not reliably preserve and validate the required end-to-end live state.”

• Also stated in: § 4.6 Failure-Boundary Evidence: Where StateM Intervenes · plain

  “The result moves from 0/5 to 5/5.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

May combine two claims.

At the cost frontier, an adapted DeepSeek-V4 Flash system reaches 88.09% on the full benchmark under standard timeouts and 89.09% on the disclosed 88-task common core.

“At the cost frontier, an adapted DeepSeek-V4 Flash system reaches 88.09% on the full benchmark under standard timeouts and 89.09% on the disclosed 88-task common core.”

May combine two claims.

StateM improves task-level success on several Terminal-Bench 2.1 tasks for GPT-5.5, with gains ranging from +1 to +5 successful trials out of five.

“Table 3 Representative Terminal-Bench 2.1 task-level improvements for GPT-5.5Each result is the number of successful trials out of five for one task.”

• AssumptionRests on an assumption: The reference and StateM runs differ only in the harness, so the observed score differences are attributable to the harness.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

RefactorBench improves when repeated compatibility review is replaced by a smaller obligation-and-closure practice.

“RefactorBench improves when repeated compatibility review is replaced by a smaller obligation-and-closure practice.”

May combine two claims.

The adapted DeepSeek-V4 Flash system's final evidence costs approximately $15 in DeepSeek API usage, while the complete recorded DeepSeek API expenditure for adaptation and evaluation is $52.22.

“Its final evidence costs approximately $15 in DeepSeek API usage, while the complete recorded DeepSeek API expenditure for adaptation and evaluation is $52.22.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The descriptive aggregate is 395/445 = 88.76% after evaluating only gpt2-codegolf with the extended timeout.

“The standard-timeout full-suite result is 392/445 = 88.09%; the common-core result is 392/440 = 89.09%;and the descriptive aggregate is 395/445 = 88.76% after evaluating only gpt2-codegolf with the extended timeout.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The 91.9% Sol ultra value is a numeric frontier reference from a different high-compute configuration.

“The 91.9% Sol ultra value is likewise a numeric frontier reference from a different high-compute configuration”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The public GPT-5.5 reference is 83.15% over trials with Codex agent version 0.125.0 and $2,059.19 reported model cost; the authors round its score to 83.1%.

“The public GPT-5.5 reference is 83.15% over trials with Codex agent version 0.125.0 and $2,059.19 reported model cost; we round its score to 83.1%.”

May combine two claims.

The $574.68 submission reports 83.37% raw before adjudication and 76.18% after adjudication.

“The $574.68 submission reports 83.37% raw before adjudication and 76.18% after adjudication.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The StateM runtime is designed to be reusable across agents and workflows.

“The StateM runtime is designed to be reusable across agents and workflows”

May combine two claims.

Runtime semantics and principles for harness development can transfer broadly, while concrete practices transfer locally or require inexpensive adaptation.

“We instead advocate that runtime semantics and principles for harness development can transfer broadly, while concrete practices transfer locally or require inexpensive adaptation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

StateM's transition protocol is checked, logged, and recoverable, rather than fully transactional.

“We describe this protocol as checked, logged, and recoverable, rather than fully transactional.”

Probably not a substantial claim: may not pass S3 (load-bearing).

StateM cannot generally roll back arbitrary external side effects produced by hooks.

“StateM delays the runtime-state commit until required pre-commit operations succeed, but it cannot generally roll back arbitrary external side effects produced by hooks.”

Classified as probably not a substantial claim. May combine two claims.

Reliable recovery depends on placing persistence hooks at appropriate phase boundaries and designing repeated hooks to tolerate retry.

“Reliable recovery consequently depends on placing persistence hooks at appropriate phase boundaries and designing repeated hooks to tolerate retry.”

Probably not a substantial claim: may not pass S4 (based).

The specific contribution of StateM is a lightweight agent-facing control representation in which state boundaries are both refreshed control contexts and auditable transition contracts.

“The specific contribution of StateM is a lightweight agent-facing control representation in which state boundaries are both refreshed control contexts and auditable transition contracts.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The results do not suggest that model scaling has ceased to matter or that a static runbook can dominate every model, task, and benchmark.

“The results do not suggest that model scaling has ceased to matter or that a static runbook can dominate every model, task, and benchmark.”

May combine two claims.

Current agents frequently depart from their plans, omit necessary steps, or terminate without satisfying the original completion criteria.

“Nevertheless, current agents frequently depart from their plans, omit necessary steps, or terminate without satisfying the original completion criteria (Liu et al., 2026).”

• cites Liu et al., 2026Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

A state serves as a context boundary that refreshes phase-local control signal and makes durable progress available.

“A state serves first as a context boundary.”

• Also stated in: § 3.2 States as Context-and-Contract Boundaries · plain

  “Entry hooks refresh the phase-local control signal and make durable progress available.”

Classified as probably not a substantial claim. May combine two claims.

The Terminal-Bench results evaluate the combined runtime and an evolved, benchmark-adapted control profile, and should not be interpreted as isolating the effect of the state-machine abstraction alone.

“In particular, the Terminal-Bench results in Section 4 evaluate the combined runtime and an evolved, benchmark-adapted control profile.They should not be interpreted as isolating the effect of the state-machine abstraction alone.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A state serves as a contract boundary with exit hooks and checks specifying conditions for leaving it.

“A state also serves as a contract boundary.”

• Also stated in: § 3.2 States as Context-and-Contract Boundaries · plain

  “The state prompt describes the work expected within the phase, while exit hooks and checks specify the conditions for leaving it.”

May combine two claims.

StateM addresses the tension between runtime enforceability and agent autonomy by preserving broad agent discretion within phases while making progress between phases explicit, persistent, and checkable.

“StateM addresses this tension by preserving broad agent discretion within phases while making progress between phases explicit, persistent, and checkable.”

Probably not a substantial claim: may not pass S2 (checkable).

The content of a runbook may encode workflow-specific or experience-derived procedural knowledge.

“the content of a runbook may encode workflow-specific or experience-derived procedural knowledge.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

StateFlow represents problem solving through explicit states and transitions, demonstrating that state-driven execution can improve task grounding and failure recovery.

“StateFlow represents problem solving through explicit states and transitions, demonstrating that state-driven execution can improve task grounding and failure recovery (Wu et al., 2024).”

• cites Wu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LangGraph and related durable workflow systems provide persistent state, conditional edges, checkpointing, interruption, and human-in-the-loop recovery.

“LangGraph and related durable workflow systems provide persistent state, conditional edges, checkpointing, interruption, and human-in-the-loop recovery (Wang and Duan, 2024).”

• cites Wang and Duan, 2024 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Life-Harness converts recurring failures into reusable interventions over environment contracts, procedural skills, action realization, and trajectory regulation, then freezes the resulting harness for held-out evaluation.

“Life-Harness converts recurring failures into reusable interventions over environment contracts, procedural skills, action realization, and trajectory regulation, then freezes the resulting harness for held-out evaluation (Xu et al., 2026).”

• cites Xu et al., 2026Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

AgingBench frames deployed agents as time-evolving systems whose effective behavior can change even when model weights remain frozen.

“AgingBench frames deployed agents as time-evolving systems whose effective behavior can change even when model weights remain frozen (Zhu et al., 2026).”

• cites Zhu et al., 2026Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Concrete runbook rules generalize when tasks share the relevant execution structure, while the methodology for identifying and enforcing such controls remains applicable across more heterogeneous workflows.

“These results suggest that concrete runbook rules generalize when tasks share the relevant execution structure, while the methodology for identifying and enforcing such controls remains applicable across more heterogeneous workflows.”

• Also stated in: § Results and scope. · hedged

  “This pattern suggests that the largest gains occur when the control profile matches a specific execution boundary.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Development principles transfer across providers.

“Together with the benchmark results, this completes the transfer hierarchy: exact runbooks transfer across nearby models, development principles transfer across providers, and control-boundary abstractions transfer across tasks.”

May combine two claims.

In StateM, the agent continues to reason and act in its native interaction loop, while states, hooks, transition conditions, and practices provide a control surface that both agents can modify and humans can inspect and enforce.

“The agent continues to reason and act in its native interaction loop, while states, hooks, transition conditions, and practices provide a control surface that both agents can modify and humans can inspect and enforce.”

May combine two claims.

This makes the control layer itself amenable to rapid iteration, a prerequisite for harness scaling, where repeated execution experience is used to continuously improve the runtime rather than only the model or prompt.

“This makes the control layer itself amenable to rapid iteration-a prerequisite for harness scaling, where repeated execution experience is used to continuously improve the runtime rather than only the model or prompt.”

May combine two claims.

The resulting behavior agrees with the evaluator, but its semantics come from the evaluator rather than the stated task contract.

“The resulting behavior agrees with the evaluator, but its semantics come from the evaluator rather than the stated task contract.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Under an append-only context with no in-place editing, each update is appended rather than written back.

“Under an append-only context with no in-place editing, each update is appended rather than written back.”

Probably not a substantial claim: may not pass S2 (checkable). The quote only partly states this claim.

Stronger-agent or human review matters at the level of deciding which abstraction deserves to become durable control.

“Stronger-agent or human review matters at this level.”

Classified as probably not a substantial claim. May combine two claims.

Agent-native systems such as Codex and Claude Code allow the model to plan and revise its behavior fluidly inside the agent loop, but much of this control remains model-mediated: plans, checklists, and self-review can guide behavior without providing the same persistent, auditable, and externally enforceable transition semantics.

“At the other extreme, agent-native systems such as Codex and Claude Code allow the model to plan and revise its behavior fluidly inside the agent loop, but much of this control remains model-mediated: plans, checklists, and self-review can guide behavior without providing the same persistent, auditable, and externally enforceable transition semantics.”

Classified as probably not a substantial claim. May combine two claims.

The table associates each gain with the StateM control active in that task; the association is not a component ablation.

“The table associates each gain with the StateM control active in that task; the association is not a component ablation.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Harness scaling helps the system remember what experience has taught it to do without training, and ensures that it actually does so before proceeding.

“Model scaling expands what an agent can do; harness scaling helps the system remember what experience has taught it to do without training, and ensures that it actually does so before proceeding.”

Probably not a substantial claim: may not pass S2 (checkable).

More remembered procedure is not better memory.

“More remembered procedure is not better memory.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Separating agents by role can provide different context views and independent decision boundaries, while role-specific read/write permissions can restrict which artifacts, state, or actions each agent may modify.

“Separating agents by role can provide different context views and independent decision boundaries, while role-specific read/write permissions can restrict which artifacts, state, or actions each agent may modify.”

Classified as probably not a substantial claim. May combine two claims.

The results measure the combined StateM system, including the profile's evolved prompts, routing, checks, and practices; they do not isolate the state-machine runtime from the procedural content encoded in the runbook.

“The results measure this combined system, including the profile's evolved prompts, routing, checks, and practices; they do not isolate the state-machine runtime from the procedural content encoded in the runbook.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Such isolation may be useful for review, verification, delegation, and recovery from decisions that would otherwise become self-reinforcing within one context.

“Such isolation may be useful for review, verification, delegation, and recovery from decisions that would otherwise become self-reinforcing within one context.”

Probably not a substantial claim: may not pass S2 (checkable).

One runbook does not fit every workflow.

“StateM does not make every model universally stronger, and one runbook does not fit every workflow.”

Classified as probably not a substantial claim.

A stop hook does not guarantee eventual success or termination.

“A stop hook does not guarantee eventual success or termination.”

Probably not a substantial claim: may not pass S2 (checkable).

The objective is to remember the consequential boundary, not every failure trace.

“The objective is to remember the consequential boundary, not every failure trace.”

Probably not a substantial claim: may not pass S2 (checkable).

Deciding which abstraction deserves to become durable control is itself a difficult reasoning task.

“Failure traces provide experience; deciding which abstraction deserves to become durable control is itself a difficult reasoning task.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 95.3% raw accuracy result remains a public, pre-adjudication submission rather than a finalized leaderboard result.

“This remains a public, pre-adjudication submission rather than a finalized leaderboard result.”

Probably not a substantial claim: may not pass S4 (based).

In every setting, profile construction may use the visible task specification, workspace artifacts, public documentation, and observable execution feedback.

“In every setting, profile construction may use the visible task specification, workspace artifacts, public documentation, and observable execution feedback.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The same separation of state, context, and permissions also motivates multi-agent runtimes.

“This work focuses on controlling the execution of a single autonomous agent, but the same separation of state, context, and permissions also motivates multi-agent runtimes.”

Probably not a substantial claim: may not pass S2 (checkable).

Effective procedural memory requires addition, revision, and removal.

“Effective procedural memory requires addition, revision, and removal.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

As the control policy evolves, modifying the structure of conventional runtimes becomes an engineering task of its own, making rapid failure-driven iteration increasingly heavy.

“As the control policy evolves, modifying this structure becomes an engineering task of its own, making rapid failure-driven iteration increasingly heavy.”

Probably not a substantial claim: may not pass S2 (checkable).

Harness scaling is an abstraction problem, not rule accumulation.

“Harness scaling is an abstraction problem, not rule accumulation.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Commands that modify external services or non-versioned artifacts should be idempotent where possible and should record sufficient information for explicit compensation or retry.

“Commands that modify external services or non-versioned artifacts should therefore be idempotent where possible and should record sufficient information for explicit compensation or retry.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The central scaling problem is not how many rules a harness remembers, but which lessons should persist, where they should intervene, and whether they survive a change of model or task.

“The central scaling problem is therefore not how many rules a harness remembers, but which lessons should persist, where they should intervene, and whether they survive a change of model or task.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

StateM's value is to make execution failures inspectable and to convert selected lessons into reusable control.

“Its value is to make execution failures inspectable and to convert selected lessons into reusable control.”

May combine two claims.

The runtime sustains a day-scale development loop while keeping procedural state external to the changing model context.

“The runtime sustains a day-scale development loop while keeping procedural state external to the changing model context; the result does not imply unbounded execution.”

Classified as probably not a substantial claim.

The default precision values show how an ambiguous specification can be resolved post hoc and then stored as if it were a universal practice.

“These defaults may be operationally useful, but they also show how an ambiguous specification can be resolved post hoc and then stored as if it were a universal practice.”

Probably not a substantial claim: may not pass S2 (checkable).

Experience must be filtered before it becomes memory.

“The design rule is direct: experience must be filtered before it becomes memory.”

May combine two claims.

StateM adds no new component-level capability in the configure-git-webserver case; it composes, checks, and closes capabilities already available to the model before the consequential handoff boundary.

“StateM adds no new component-level capability in this case.It composes, checks, and closes capabilities already available to the model before the consequential handoff boundary.”

Classified as probably not a substantial claim.

For the GPT-5.6 Sol and Luna evaluations, the versioned profile and its activation policy are frozen: no prompt, state, routing, check catalog, default, or repair policy is revised using target-model evaluation outcomes.

“For the GPT-5.6Sol and Luna evaluations, the versioned profile and its activation policy are frozen: no prompt, state, routing, check catalog, default, or repair policy is revised using target-model evaluation outcomes.”

Classified as probably not a substantial claim.

StateM does not make every model universally stronger.

“StateM does not make every model universally stronger, and one runbook does not fit every workflow.”

Probably not a substantial claim: may not pass S2 (checkable).

The BusinessBench iterations show why procedural memory must be selective.

“The BusinessBench iterations show why this memory must be selective.”

Probably not a substantial claim: may not pass S3 (load-bearing).

StateM does not make untrusted hook code safe.

“StateM does not make untrusted hook code safe; sandboxing and permission control remain host responsibilities.”

May combine two claims.

BusinessBench yields the task-side lesson that frozen aggregate generalization is positive but modest, and the largest gains occur where control is sparse and matched to an explicit execution boundary.

“BusinessBench yields the corresponding task-side lesson: frozen aggregate generalization is positive but modest, and the largest gains occur where control is sparse and matched to an explicit execution boundary.”

Probably not a substantial claim: may not pass S2 (checkable).

Failure traces provide experience.

“Failure traces provide experience; deciding which abstraction deserves to become durable control is itself a difficult reasoning task.”

Classified as probably not a substantial claim.

An active procedure may not be completed.

“Relevant knowledge may be unavailable at the decision point; a lesson from prior experience may not be retained or activated; or an active procedure may not be completed.”

Classified as probably not a substantial claim. May combine two claims.

StateM is an agent-native control layer that externalizes mutable execution state, refreshes phase-local context, and checks consequential transitions while preserving the unified reasoning loop of a general-purpose CLI agent.

“We introduce StateM, an agent-native control layer that externalizes mutable execution state, refreshes phase-local context, and checks consequential transitions while preserving the unified reasoning loop of a general-purpose CLI agent.”

May combine two claims.

StateM can prevent a configured requirement from being silently skipped, but it cannot detect a missing requirement that neither the runbook nor the agent introduces.

“Similarly, StateM can prevent a configured requirement from being silently skipped, but it cannot detect a missing requirement that neither the runbook nor the agent introduces.”

May combine two claims.

StateM moves both the quality and cost frontiers on Terminal-Bench 2.1.

“Figure 5 StateM moves both the quality and cost frontiers on Terminal-Bench 2.1”

May combine two claims.

Across providers, the runtime and failure-driven development methodology remain reusable, while model-dependent practices must be adapted.

“Across providers, the runtime and failure-driven development methodology remain reusable, while model-dependent practices must be adapted.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Exact runbooks transfer across nearby models.

“Together with the benchmark results, this completes the transfer hierarchy: exact runbooks transfer across nearby models, development principles transfer across providers, and control-boundary abstractions transfer across tasks.”

Probably not a substantial claim: may not pass S2 (checkable).

The runtime itself must be cheap to modify.

“The experiments above highlight a practical requirement that is easy to miss when viewing a harness only as a fixed workflow: the runtime itself must be cheap to modify.”

Probably not a substantial claim: may not pass S2 (checkable).

Harness scaling requires a runtime that is both agent-native and enforceable.

“Harness scaling requires a runtime that is both agent-native and enforceable.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

WebArena appears to require a still lighter query-state boundary.

“WebArena appears to require a still lighter query-state boundary.”

Probably not a substantial claim: may not pass S4 (based).

The agent-native and enforceable features of StateM make harness scaling feasible in efficiency and cost.

“The agent-native and enforceable features of StateM make harness scaling feasible in efficiency and cost.”

Classified as probably not a substantial claim. May combine two claims.

StateM provides control points, not a correctness oracle.

“StateM provides control points, not a correctness oracle.”

May combine two claims.

StateM addresses these operational gaps through state-local context, versioned procedural practices, and checked transitions, respectively.

“StateM addresses these operational gaps through state-local context, versioned procedural practices, and checked transitions, respectively.”

Classified as probably not a substantial claim. May combine two claims.

The protocol abstains from applying a StateM workflow to attendance-payroll; all 72 treatment-arm runs in that family receive no StateM intervention and are excluded from efficacy aggregates.

“The protocol abstains from applying a StateM workflow to attendance-payroll; all 72 treatment-arm runs in that family receive no StateM intervention and are excluded from efficacy aggregates.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In long-running work, a single agent can become overconfident in an early task interpretation and continue reasoning inside a context shaped by its own previous decisions.

“In long-running work, a single agent can become overconfident in an early task interpretation and continue reasoning inside a context shaped by its own previous decisions.”

Classified as probably not a substantial claim. May combine two claims.

Profile construction does not use hidden tests, verifier implementations, public solutions, answer artifacts, task hashes, or manually enumerated task identifiers as routing keys.

“It does not use hidden tests, verifier implementations, public solutions, answer artifacts, task hashes, or manually enumerated task identifiers as routing keys.”

May combine two claims.

StateM was built for an operational requirement beyond benchmark accuracy: a long-running agent must preserve its place and obligations as the interaction grows.

“StateM was built for an operational requirement beyond benchmark accuracy: a long-running agent must preserve its place and obligations as the interaction grows.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Hooks execute with the permissions of the surrounding host environment.

“Hooks execute with the permissions of the surrounding host environment.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Conventional state-machine and graph-based agent runtimes provide explicit orchestration and strong control, but typically externalize substantial reasoning into manually constructed nodes, edges, and handlers.

“Conventional state-machine and graph-based agent runtimes provide explicit orchestration and strong control, but typically externalize substantial reasoning into manually constructed nodes, edges, and handlers.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Model scaling expands what an agent can do.

“Model scaling expands what an agent can do; harness scaling helps the system remember what experience has taught it to do without training, and ensures that it actually does so before proceeding.”

Classified as probably not a substantial claim.

The resulting BusinessBench profile is frozen before the first held-out evaluation.

“The resulting profile is frozen before the first held-out evaluation.”

Probably not a substantial claim: may not pass S4 (based).

Control-boundary abstractions transfer across tasks.

“Together with the benchmark results, this completes the transfer hierarchy: exact runbooks transfer across nearby models, development principles transfer across providers, and control-boundary abstractions transfer across tasks.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Plan tokens are at a higher abstraction level, which is much fewer but critical.

“Plan tokens are at higher abstraction level, which is much fewer but critical.”

May combine two claims.

The stop-hook mechanism reduces premature termination and supports longer unattended runs.

“This mechanism reduces premature termination and supports longer unattended runs.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The authors use 88.8% as a score-only reference and plot $574.68 only at its matched submission score.

“We therefore use 88.8% as a score-only reference and plot $574.68only at its matched submission score”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sandboxing and permission control remain host responsibilities.

“StateM does not make untrusted hook code safe; sandboxing and permission control remain host responsibilities.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Only the first frozen held-out evaluation is untouched; later results are explicitly post-evaluation validation.

“Only the first frozen held-out evaluation is untouched; later results are explicitly post-evaluation validation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The public GPT-5.5 reference is a model-matched public reference rather than a newly rerun, agent-version-matched A/B control.

“It is a model-matched public reference rather than a newly rerun, agent-version-matched A/B control.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

BusinessBench family sizes are unequal and each instance has one stochastic trajectory per arm.

“BusinessBench family sizes are unequal and each instance has one stochastic trajectory per arm.”