The DOI matches its DataCite record; 8 of the 8 references it cites were found in a registry or an index; 1 of the 7 sources its claims cite were checked for support.
Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents
Open accessAbstract
Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to this divergence. We introduce a reliability science framework for long-horizon LLM agents with four metrics: Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP). We evaluate 10 models across 23,392 episodes on a 396-task benchmark spanning four duration buckets and three domains. Key findings: (1) reliability decay is domain-stratified -- SE GDS drops from 0.90 to 0.44 while document processing is nearly flat (0.74 to 0.71); (2) VAF bifurcates by capability tier -- high VAF is a capability signature, not an instability signal; (3) capability and reliability rankings diverge substantially, with multi-rank inversions at long horizons; (4) frontier models have the highest meltdown rates (up to 19%) because they attempt ambitious multi-step strategies that sometimes spiral; and (5) memory scaffolds universally hurt long-horizon performance across all 10 models. These results motivate reliability as a first-class evaluation dimension alongside capability.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes a reliability science framework for long-horizon LLM agents with four metrics—RDC, VAF, GDS, and MOP—and evaluates 10 open-source models on a 396-task benchmark across four duration buckets and three domains over 23,392 episodes. It finds that reliability decays super-linearly with task duration, diverges from capability rankings, and that naive memory scaffolds do not improve long-horizon reliability.
Key findings
- Reliability decay is universal: aggregate pass@1 drops from 76.3% at short horizon to 52.1% at very-long horizon, a 24.3 percentage-point decline; SE GDS falls from 0.90 to 0.44 while DP is nearly flat (0.74 to 0.71).
- VAF bifurcates by capability tier: frontier models have VAF ≥ 2.37 (up to 2.60), while mid/small models have VAF ≤ 1.26; high variance amplification is interpreted as a capability signature, not an instability signature.
- Capability and reliability rankings diverge with multi-rank inversions; e.g., Llama 3.3 70B recovers from 5th–6th at short horizon to 3rd–4th at very-long, while GLM-4.5 Air falls from 1st at short to 4th at very-long.
- The “MOP paradox” shows frontier models have the highest meltdown rates (DeepSeek V3 19%, MiniMax M2.5 13% at very-long) while also achieving the best very-long GDS; memory scaffolds never improve long-horizon GDS (neutral for 4 models, hurts 6, with largest penalties Kimi K2.5 −0.14 and Mistral 24B −0.13).
Methods
- Formalizes a task model and four reliability metrics: Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP); RDS summarizes RDC slope.
- Constructs a 396-task benchmark with 33 tasks per duration × domain cell across four duration buckets (short ≤5 min, medium 5–30 min, long 30–120 min, very long ≥120 min) and three domains (software engineering, web research, document processing), using programmatic evaluation wherever possible.
- Evaluates 10 open-source models via OpenRouter under two scaffolds (ReAct and memory-augmented), with k = 3 repeats, temperature 0.7, and 23,392 completed episodes (23,760 planned); analyses include pass@1, GDS, VAF, MOP, domain stratification, and cost.
Limitations
- Stated: human-estimated duration is an imperfect proxy for agent difficulty (evidenced by DP non-monotonicity); infrastructure reliability is a validity threat (free-tier quota exhaustion inflated apparent pass@1 by up to 15%, and two models returned HTTP 404 throughout); only 10 open-source models are evaluated, excluding proprietary frontier models; task coverage is limited to three domains; programmatic evaluation may have blind spots; live web content threatens temporal reproducibility.
- Stated: MOP thresholds are auto-calibrated from 1,590 short-horizon baseline episodes, and full manual labeling of meltdown episodes with formal precision/recall is deferred to future work.
- Evident from the provided text: some tables and values are missing or formatting-corrupted (e.g., the MOP window-size sensitivity table is blank), which limits independent verification of those details.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
68 Own result · 53 Own interpretation · 17 Own method · 7 No basis given · 5 Sources not found · 1 Partly supported · 1 Not checked yet · accuracy not yet measured
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We test this i.i.d.assumption empirically and find systematic violations”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 3.1 Task Model
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our finding that reliability decays super-linearly with task duration (Section 6) implies that the i.i.d.Bernoulli model of step-level errors is systematically wrong -errors are positively correlated across steps.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 7.2 Theoretical Implications
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Kimi K2.5 and Mistral 24B show the largest penalties (-0.14 and -0.13 respectively).”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.5 Memory Scaffolds Universally Hurt at Long Horizons
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Llama 3.1 8B achieves only 4.9% long+very-long pass@1 (Table 10)”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This result is a reversal of the preliminary finding in the short+medium study, where Llama 3.1 8B and Llama 3.3 70B showed small positive memory effects.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.5 Memory Scaffolds Universally Hurt at Long Horizons
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For GLM-4.5 Air, the fraction of episodes that terminate before completing even the first subtask rises from 1% (short) to 25% (very long)”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The aggregate medium-horizon drop (-16.5 pp) already exceeds the geometric prediction for most models”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“memory scaffolds universally hurt long-horizon GDS (negative or neutral for all 10 models)”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All other models have meltdown rates of 0-4% across all buckets.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.4 The MOP Paradox: Frontier Models Melt Down More
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“quota was exhausted after completing short-horizon tasks but before reaching long-horizon tasks, inflating apparent pass@1 by up to 15% relative to full-horizon coverage”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Model selection rationale.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 74
- Probably not a substantial claim. 13
- The quote only partly states this claim. 1
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“τ -bench [Yao et al., 2024] showed that GPT-4o achieves 61% pass@1 on retail agent tasks but only 25% pass@8”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 1 Introduction
What the sources say
- Partly supportedcites Yao et al., 2024 sourceExperimental — accuracy not yet measured
Table 2 in §5.1 reports 61.2% pass^1 for GPT-4o on τ-retail; §3 states that pass^1 equals pass@1. But §5.1 attributes the approximately 25% eight-trial result to pass^8, not pass@8. Section 3 defines these as different metrics: pass^8 requires success on all eight trials, while pass@8 requires success on at least one. (LLM, conf=0.95)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The bottom six models all have VAF < 1.3, with the weakest models (Llama 3.1 8B, Mistral Nemo) having VAF = 0.26 and 0.42 respectively”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Llama 3.3 70B ranks 5th-6th by short-horizon pass@1 (74.7%) but recovers to 3rd-4th place at very-long (54.5%).”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.3 Capability Rank ̸ = Reliability Rank: Evidence from Rank Inversions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GLM-4.5 Air is a borderline case: strong short-horizon (94.9%) but a steeper decay curve (very-long 66.7%), making it a”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Two-tier reliability structure.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Rank inversions: Capability rank (short-horizon pass@1) and reliability rank (very-long pass@1) diverge substantially, with Llama 3.3 70B recovering from 5th at short to 3rd-4th at very-long, and GLM-4.5 Air falling from 1st at short to 4th at very-long.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 8 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For Qwen3 30B (p short = 75.8%), the geometric prediction for long horizon (T ≈ 4×) is 0.758 4 ≈ 33.0%; observed is 22.2% -1.5× below baseline”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For Mistral Nemo (p short = 53.5%), the geometric prediction for medium is 0.535 2 ≈ 28.6%; observed is 12.1% -2.4× below baseline”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“document-processing tasks that take humans 45-60 minutes may require only 4-8 agent tool calls, while software engineering tasks of equivalent human duration require 15-25”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Duration calibration and agent complexity.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At very-long horizon, Llama 3.3 70B (74.7% short) outperforms Qwen3 30B (75.8% short) by 20 percentage points (54.5% vs. 34.3%)despite nearly identical short-task performance.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 7.1 Implications for Production Deployment
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Everything below GLM-4.5 Air shows ≤ 55.6% very-long pass@1 and VAF ≤ 1.26, indicating that the capability infrastructure required for reliable long-horizon completion requires more than the medium-tier models examined here provide.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Two-tier reliability structure.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Llama 3.3 70B at $0.10/M output achieves only 44.6% long+very-long pass@1reasonable reliability-per-dollar for cost-constrained scenarios but not competitive with frontier models for consequential long-horizon tasks.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.7 Cost-Reliability Trade-offs
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For DeepSeek V3, this represents a 92.9% -79.8% = 13.1 pp reliability gain; for Qwen3 30B, a 75.8% -34.3% = 41.5 pp gain.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Memory scaffolds are counterproductive at long horizons.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Subtask independence holds when decomposing a long task into shorter segments.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“DeepSeek V3's early failure rate rises from 2% (short) to 11% (very long), with the sharpest jump at the long-to-very-long transition”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“DeepSeek V3 achieves the highest mean very-long GDS (0.87) at a moderate cost ($0.14/Minput, $0.28/M output via DeepInfra routing) and represents the Pareto-dominant choice: high reliability, moderate cost, wide provider coverage.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.7 Cost-Reliability Trade-offs
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The top four models by long-horizon pass@1 (DeepSeek V3, Kimi K2.5, MiniMax M2.5, GLM-4.5 Air) have VAF = 2.37-2.60 with non-overlapping confidence intervals above the bottom six”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Memory scaffolds universally hurt: Across all 10 models, the memory scaffold never improves long-horizon GDS relative to ReAct (neutral for 4 models, negative for 6).”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 8 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the Memory scaffold reduces long+very-long GDS relative to ReAct for 5 of 9 models, with the largest penalties on the strongest models”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Key observations.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Mistral 24B ranks below several models at medium (52.5%) but recovers to 55.6%-62.6% at long and very-long.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.3 Capability Rank ̸ = Reliability Rank: Evidence from Rank Inversions
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“DeepSeek V3 and MiniMax M2.5 achieve the best very-long GDS (0.87, 0.89) yet also have the highest very-long meltdown rates (19%, 13%).”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.4 The MOP Paradox: Frontier Models Melt Down More
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across 10 models and four duration buckets, aggregate pass@1 drops from 76.3% at short horizon to 52.1% at very-long -a 24.3 percentage-point decline that is super-linear relative to the i.i.d.Bernoulli baseline for most models.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 8 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The full 23,392-episode study cost approximately $40.83 for short+medium and is estimated at $80-120 total including”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.7 Cost-Reliability Trade-offs
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At long horizon, Llama 3.3 70B has GDS= 0.55 and pass@1= 0.39 -a gap of 0.16.At very-long, GDS= 0.62 and pass@1= 0.50 -a gap of 0.12.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.3 Capability Rank ̸ = Reliability Rank: Evidence from Rank Inversions
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“capability and reliability rankings diverge substantially, with multi-rank inversions between medium and very-long horizons”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MiniMax M2.5 achieves competitive very-long GDS (0.89) at higher cost but with 16 fallback providers, making it the most infrastructure-reliable frontier option.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.7 Cost-Reliability Trade-offs
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The 1,322-episode validation run cost $4.26, or approximately $0.0032 per episode”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Key observations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The full-study result is unambiguous: the memory scaffold never improves long-horizon reliability, and hurts 6 of 10 models (Section 6.5).”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Memory scaffolds are counterproductive at long horizons.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The full 23,392-episode study cost approximately $40.83 for short+medium buckets; total study cost including all four buckets is estimated at $80-120”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Key observations.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Kimi K2.5 is the most domain-balanced frontier model, with 79-83% across all three domains.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Domain is the strongest single predictor of reliability decay, dominating model-size effects for most of the performance range. Table 11 (Section 5.3) reports aggregate GDS by domain and bucket.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all 10 models, the memory scaffold either hurts or is neutral at long horizons; no model gains GDS from memory augmentation.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.5 Memory Scaffolds Universally Hurt at Long Horizons
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Software engineering tasks collapse fastest (aggregate GDS 0.90 → 0.44 over the full duration range) while document processing tasks remain nearly flat (0.74 → 0.71), demonstrating that human-estimated task duration and agent-relevant complexity are orthogonal dimensions.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 8 Conclusion
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GLM-4.5 Air starts as the highest-performing model at short horizon (94.9%) but drops to fourth place at very-long (66.7%).”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 7.1 Implications for Production Deployment
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The GDS-pass@1 gap widens at long horizons, capturing partial progress that binary evaluation discards entirely”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“reliability decay is domain-stratified -SE GDS drops from 0.90 to 0.44 over the full duration range, while DP is nearly flat (0.74 to 0.71)”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“MOP paradox: The two best-performing frontier models (DeepSeek V3, MiniMax M2.5) also have the highest meltdown rates (19% and 13% at very-long), because frontier models pursue ambitious multi-step strategies that generate tool-call entropy spikes when they spiral -while weaker models emit rote, low-entropy sequences that never complete long tasks.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 8 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Llama 4 Maverick (400B MoE) ranks 8th on short tasks (pass@1 = 0.40) and last among paid models on long+very-long tasks (pass@1 = 0.00)”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Key observations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Seven of 9 models have negative RDS, confirming the reliability decay hypothesis”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Key observations.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The memory scaffold never helps.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.5 Memory Scaffolds Universally Hurt at Long Horizons
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“SE remains the most discriminating domain: spread ranges from 0.5% (Llama 3.1 8B) to 82.3% (Kimi K2.5), an 82-point gap.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Domain is the strongest single predictor of reliability decay, dominating model-size effects for most of the performance range. Table 11 (Section 5.3) reports aggregate GDS by domain and bucket.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Llama 3.3 70B shows the opposite pattern: strong WR (68.2%) but weak SE (12.1%).”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Domain is the strongest single predictor of reliability decay, dominating model-size effects for most of the performance range. Table 11 (Section 5.3) reports aggregate GDS by domain and bucket.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“VAF bifurcates cleanly by capability tier (frontier VAF ≥ 2.37; mid-tier VAF ≤ 1.26)”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Abstract
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GLM-4.5 Air achieves strong SE (77.3%) and DP (90.9%) at long horizons but collapses on WR (50.0%) -28 percentage points below its next-closest domain.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Domain is the strongest single predictor of reliability decay, dominating model-size effects for most of the performance range. Table 11 (Section 5.3) reports aggregate GDS by domain and bucket.
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its long-horizon GDS also rises from 0.55 (long) to 0.62 (very long) -the only non-frontier model that improves from long to very-long.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.3 Capability Rank ̸ = Reliability Rank: Evidence from Rank Inversions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“VAF bifurcation: VAF splits cleanly into two regimes -frontier models (VAF = 2.37-2.60)and mid/small models (VAF ≤ 1.26) -because high variance amplification requires high long-horizon pass@1 to amplify.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 8 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MOP is not a binary failure detector; it identifies ambition mismatches -episodes where a capable model's exploratory strategy has exceeded its reliability envelope.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.4 The MOP Paradox: Frontier Models Melt Down More
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Web research tasks use live web search; all other tasks use only local file operations, eliminating network non-determinism”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 4.3 Task Construction Principles
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“frontier models exhibit the highest meltdown rates (up to 19%) because they pursue ambitious multi-step strategies -the”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Each episode runs in a fresh copy of the task workspace (via shutil.copytree to a temporary directory), ensuring that outputs from one episode do not contaminate another”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 4.3 Task Construction Principles
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We calibrate θ H and δ on the pilot dataset to maximize the F1 score of detecting manually labeled meltdown episodes”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 3.5 Metric 4: Meltdown Onset Point (MOP)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“capable models sometimes succeed and sometimes fail at long horizons, creating high variance.Weak models reliably fail at long horizons -near-zero pass@1 means near-zero variance”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our central finding is that reliability is a two-dimensional property -shaped jointly by task duration and domain structure -that current benchmarks are structurally blind to because they report only pass@1 on short, atomic tasks.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 8 Conclusion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Web research tasks evaluate against live web content, which changes over time.We use tolerant matching (range checks, substring matching) to reduce sensitivity, but WR results may not be perfectly reproducible at different time points.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Temporal validity.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MOP-based early stopping targets ambition mismatches.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Memory scaffolds are counterproductive at long horizons.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The baseline ReAct loop is the better long-horizon scaffold, and naive episodic memory augmentation should not be adopted as a default reliability intervention.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 8 Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These findings collectively demonstrate that optimizing for short-task pass@1 is an insufficient and sometimes counterproductive strategy for production reliability.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 8 Conclusion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our finding that reliability decays super-linearly with task duration (Section 6) implies that the i.i.d.Bernoulli model of step-level errors is systematically wrong -errors are positively correlated across steps.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 7.2 Theoretical Implications
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We mitigate this by designing edge-case-covering test suites, but cannot fully eliminate the concern.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 7.3 Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our 396-task benchmark, while the most comprehensive reliability-focused agent study we are aware of, cannot cover all possible task types.The three domains (SE, WR, DP) are representative but not exhaustive; tasks involving embodied action, tool fabrication, or multi-agent coordination are not covered.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 7.3 Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The VAF bifurcation (Section 6.2) further implies that there is a capability threshold below which models enter a uniform failure regime: short-and long-horizon failures are equally certain, so duration adds no new variance.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 7.2 Theoretical Implications
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We evaluate 10 open-source models only, for cost and reproducibility reasons.We do not evaluate GPT-4o, Claude 3.7, Gemini 2.0 Ultra, or other frontier proprietary models, which are likely more reliable than the models studied here.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 7.3 Limitations
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This U-shaped reliability curve is invisible to short-task benchmarks and would mislead any practitioner relying on medium-horizon evaluation alone.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.3 Capability Rank ̸ = Reliability Rank: Evidence from Rank Inversions
hedgeddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“reliability is defined as the probability that a system performs its required function under stated conditions for a specified period of time [Modarres et al., 2016]”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Reliability in safety-critical engineering.
What the sources say
- Not matchedcites Modarres et al., 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This overhead is tolerable at short horizons (few scratchpad entries) but becomes load-bearing at long horizons where scratchpad entries accumulate.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.5 Memory Scaffolds Universally Hurt at Long Horizons
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The DP non-monotonicity (Section 6.6) is a direct demonstration of this proxy's imperfection: DP-L tasks classified as”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 7.3 Limitations
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“ReliabilityBench [rel, 2026b] is the closest prior work to ours.It evaluates LLMs across three reliability dimensions -consistency, robustness, and fault tolerance -and constructs a three-dimensional reliability surface”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 2.2 Reliability and Robustness Evaluation
What the sources say
- Not matchedcites rel, 2026b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“τ -bench introduced pass k but did not study duration as a variable”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 1 Introduction
What the sources say
- Not checked yetcites Yao et al., 2024 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Instead, a meltdown signal should trigger context resetting: saving the current subtask state, starting a fresh context window, and continuing from the last verified checkpoint.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Memory scaffolds are counterproductive at long horizons.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Programmatic evaluation eliminates judge variance from the reliability measurements and makes results fully reproducible”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 4.3 Task Construction Principles
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“ReliabilityBench [rel, 2026b] is the closest prior work, introducing a three-dimensional reliability surface, but covers only 2 models and short-horizon tasks”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 1 Introduction
What the sources say
- Not matchedcites rel, 2026b
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This preserves partial progress while breaking the entropy-spiraling trajectory.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Memory scaffolds are counterproductive at long horizons.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Above this threshold, duration introduces genuine uncertainty -the model sometimes succeeds and sometimes failscreating the variance amplification we observe.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 7.2 Theoretical Implications
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Duration buckets are assigned based on estimated human completion time, validated against author completion times on a sample of tasks”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Duration calibration and agent complexity.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The METR long-horizon study [met, 2025] measures task-completion horizons but does not analyze variance or reliability across repeated attempts”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 1 Introduction
What the sources say
- Not matchedcites met, 2025
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because high meltdown rates co-occur with high GDS for frontier models, MOP should not trigger immediate task abandonment.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Memory scaffolds are counterproductive at long horizons.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its planning style (broad web search and synthesis) suits agentic research tasks but fails at SE's requirement for precise, verifiable multi-step execution.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Domain is the strongest single predictor of reliability decay, dominating model-size effects for most of the performance range. Table 11 (Section 5.3) reports aggregate GDS by domain and bucket.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“With 33 tasks per (bucket × domain) cell, 95% confidence intervals on pass@1 are ±4-10% depending on model performance, providing reliable per-cell estimates across all four duration buckets”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In practice, errors are positively correlated across steps (a confused agent tends to stay confused), which pushes variance even higher and faster”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 3.3 Metric 2: Variance Amplification Factor (VAF)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The two near-zero slopes (DeepSeek V3, Llama 3.3 70B, Llama 3.1 8B) reflect ceiling effects at the top and a floor effect at the bottom, not immunity to decay”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Key observations.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“VAF is not a diagnostic for instability; it is a capability filter”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Weaker models, by contrast, emit stable low-entropy tool-call sequences because they follow rote, shallow strategies that never generate entropy spikes -but also never complete the task.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.4 The MOP Paradox: Frontier Models Melt Down More
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Meltdown is particularly common in long-horizon tasks where the context window fills and the model loses track of its earlier state”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 3.5 Metric 4: Meltdown Onset Point (MOP)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
The 396-task benchmark is representative of the distribution of real-world long-horizon agent tasks.
“We construct a 396-task benchmark across four duration buckets and three domains”
The paper relies on this without support.
The 10 open-source models evaluated are representative of the broader class of LLM agents used in production.
“We evaluate 10 open-source models via two scaffolds (ReAct and memory-augmented) with k = 3 repeats”
The authors state this as an assumption.
Subtask independence holds when decomposing a long task into shorter segments.
“(assuming subtask independence)”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.3 Full Study Results (23,392 Episodes)
“The top four models by long-horizon pass@1 (DeepSeek V3, Kimi K2.5, MiniMax M2.5, GLM-4.5 Air) have VAF = 2.37-2.60 with non-overlapping confidence intervals above the bottom six.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Additional information
“MiniMax M2.5 2.60 [1.53, 5.24] 93.9% 83.2% DeepSeek V3 2.49 [1.44, 4.44] 93.3% 83.2% Kimi K2.5 2.48 [1.46, 5.26] 94.4% 81.8% GLM-4.5 Air 2.37 [1.59, 4.16] 90.7% 72.7% Qwen3 32B 1.26 [1.07, 1.54]”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.7 Cost-Reliability Trade-offs
“DeepSeek V3 achieves the highest mean very-long GDS (0.87)”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 6.7 Cost-Reliability Trade-offs
“MiniMax M2.5 achieves competitive very-long GDS (0.89)”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 4.4 Evaluation Protocol
“Each episode runs until the agent calls finish() or reaches a 50-step limit.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Model selection rationale.
“Hyperparameters.Temperature: 0.7; maximum steps: 70 (headroom above the 60-step estimate for very-long tasks)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Reliability Decay Slope (RDS).
“We summarize the RDC with a single scalar: the slope of a linear regression of GDS on bucket index”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Reliability Decay Curve.
“RDS = Reliability Decay Slope (linear regression slope of pass@1 vs. integer bucket index: short= 1, medium= 2, long= 3, very_long= 4).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“We began with a 19-task validation suite (2 tasks per bucket per domain, covering all four duration buckets including long and very long)”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > Task construction trajectory.
Reported 19 · recomputed 24
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“the 1,322 remaining episodes over 9 paid models form the basis of this section.”
§ Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents > 5.2 19-Task Validation Results
Reported 1322 · recomputed 1026
Citations
Cites
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74
OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application WorkflowsPreprint
Wang, W., Han, D., Diaz, D. M., Xu, J., Rühle, V., & Rajmohan, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.09124 · Cited by 67
ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress ConditionsPreprint
Gupta, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.06112 · Cited by 50
Measuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14499 · Cited by 14
$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024) · arXiv · doi:10.48550/arxiv.2406.12045 · Cited by 9
Towards a Science of AI Agent Reliability
Rabanser, S., Kapoor, S., Kirgis, P., Liu, K., Utpala, S., & Narayanan, A. (2026) · arXiv · doi:10.48550/arxiv.2602.16666 · Cited by 1
OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety
Vijayvargiya, S., Soni, A. B., Zhou, X., Wang, Z. Z., Dziri, N., Neubig, G., & Sap, M. (2025) · arXiv · doi:10.48550/arxiv.2507.06134
Research works printed but not linked: 5
- In-Context Learning through the Bayesian Prism. 2024. A Survey of LLM Agent Evaluation. arXiv preprint arXiv:2507.21504, 2025. · arXiv
- Carlos E Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In ICLR, 2024.
- Mohammad Modarres, Mark Kaminskiy, and Vasiliy Krivtsov. Reliability Engineering and Risk Analysis: A Practical Guide. CRC Press, 2016.
- Shunyu Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR, 2023.
- Shuyan Zhou et al. WebArena: A Realistic Web Environment for Building Autonomous Agents. In ICLR, 2024.
Cited by
Sorted by citations
Why Retrying Fails: Context Contamination in LLM Agent PipelinesPreprint
Yang, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.08563 · Cited by 6
Measuring the Unmeasurable: Markov Chain Reliability for LLM AgentsPreprint
Tran-Truong, P. T., & Le, X.-B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.24579 · Cited by 5
What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion MetricsPreprint
Liu, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.16370 · Cited by 3
Zhang, L., Sun, Y., & Shi, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.23700 · Cited by 1
Chen, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.16067 · Cited by 1
Metadata
- Authors
- Khanal, A., Tao, Y., & Zhou, J.
- Affiliations
- Northern Kentucky University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2603.29231
- OpenAlex
- W7147088641
- Cited by
- 10
- References
- 7
- Citations by year
- 2026: 10
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- LLM agents, reliability, graceful degradation, task duration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
