The DOI matches its DataCite record; 18 of the 18 references it cites were found in a registry or an index; none of the 5 sources its claims cite is checked for support yet.
How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making
Open accessAbstract
Production deployments of large language model (LLM) agents remain unreliable on long, multi-step workflows even as benchmark success rates climb steadily. We argue this gap is largely an artifact of task horizon: benchmarks are dominated by short-to-medium horizons where success remains high, while production workloads demand an order of magnitude more dependent steps. We measure the effect directly, characterizing the shape of agent degradation and disentangling its cause across a large controlled study spanning nine models, six open models from 1.2B to 671B parameters, and three deployed proprietary systems; four task families, including a genuinely agentic tool-use loop; five horizons; and three context regimes. Task success follows a geometric law governed by a single per-step reliability parameter, which rises with model scale but saturates well below 1 even for the strongest models, guaranteeing eventual collapse at sufficiently long horizons. The effect is sharpest on the agentic task, where every model tested, including widely deployed systems, falls from near-perfect success to near zero within sixteen steps of (n=10,664 analyzed trajectories. Degradation is driven by step count rather than context length: bounding the context window steepens decay rather than easing it (logit slope -0.69 vs. -0.44), p=3x10-6), contradicting a lost-in-the-middle explanation and warning against a common production shortcut. Projecting measured reliability onto representative benchmark horizons quantifies a substantial gap between benchmark and production conditions, from 0.42 at GAIA-length horizons to 0.24 at hundred-step production horizons. For teams responsible for agent orchestration and reliability at scale, these results argue for horizon-aware evaluation and reliability budgeting in place of aggregate pass-rate metrics. Code, prompts, seeds, and raw trajectories are released.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Across 10,664 analyzed trajectories from nine models (six open, 1.2B–671B, plus three proprietary systems), task success on controlled agent tasks follows a geometric decay law governed by a single per-step reliability parameter that never reaches 1. The driver is step count rather than context length, and on a genuinely agentic tool-use task every model tested collapses from near-ceiling success to near zero within sixteen steps.
Key findings
- Success follows a geometric law in 28 of 36 model-task cells tested (AIC selection among geometric, threshold, and linear forms); the remainder follow a threshold/cliff shape.
- On the agentic ToolQA task (natural regime), eight of nine models solve at or near ceiling at H=2, but by H=16 every model has collapsed — e.g., Qwen2.5-72B goes 1.00 (H=2) → 1.00 (H=4) → 0.87 (H=8) → 0.13 (H=16); DeepSeek-V3 goes 1.00 → 1.00 → 0.67 → 0.27.
- Degradation is driven by step count, not context length: bounding context steepens decay (logit slope −0.69 compressed vs. −0.44 natural, p=3×10⁻⁶), while collapsing all operations into one prompt (padded, −0.40) is statistically indistinguishable from natural (p=0.51).
- Per-step reliability rises with scale (Pearson r=+0.36 vs. log10(params)) but saturates below 1; hazard also accelerates within trajectories (mean per-step accuracy 0.58 in the first third vs. 0.44 in the last third), and format/tool-call drift affects 21% of trajectories.
- Projecting mean measured reliability (r=0.61) onto benchmark horizons gives 0.42 at GAIA-length (8 steps), 0.36 at WebArena-length (15), 0.33 at tau-bench-length (20), 0.30 at SWE-bench/OSWorld-length (30), and 0.24 at hundred-step production horizons.
Methods
- Four synthetic, oracle-verifiable task families (Ledger, Refchain, Cipher, and the agentic ToolQA ReAct-style tool-use loop) paired with ground-truth simulators to avoid LLM-judge noise; five horizons and three context regimes (natural, compressed, padded) applied to the same underlying task instances.
- Nine instruction-tuned models: six open models from 1.2B to 671B parameters and three deployed proprietary systems via hosted interfaces; 10,664 total analyzed trajectories; fixed temperature and recorded seeds.
- Paired-difference comparisons across regimes, Wilson score intervals for per-cell success, and AIC for selecting among geometric, threshold, and linear decay shapes.
Limitations
- Stated: task families are synthetic (chosen for oracle verifiability), so ecological validity against fully open-ended real-world workflows remains open; the agentic family's API budget ran out before the longest horizon, making the H=16 collapse a lower bound; the nine-model sample cannot establish that all current or future models follow the identical geometric form; three additional small models were excluded for failing the structured response protocol; results are at a fixed moderate decoding temperature.
- Evident from the text: the agentic-task table shows non-monotonic cells (e.g., Qwen2.5-7B 0.07 at H=4 vs. 0.33 at H=16; Gemini-2.5-Flash-Lite 0.20 at H=2 vs. 0.53 at H=4), suggesting small per-cell sample sizes and noisy estimates at the agentic horizons.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
21 Own interpretation · 17 Own result · 11 Own method · 5 Sources not found · 1 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Success declines monotonically with horizon wherever the task is non-trivial for the model in question.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 3.5 Data and code availability
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The first error tends to land roughly a third of the way through a trajectory, and once an agent errs it rarely recovers, consistent with prior reports that models get lost after a wrong turn and do not find their way back [11].”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.4 Degradation accelerates within a trajectory
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“no model in this study reaches perfect per-step reliability on any non-trivial task.”
- Also stated in: § 3.5 Data and code availability · boosted
“because per-step reliability never reaches 1 on any non-trivial task in this data, collapse at a sufficiently long horizon is not a possibility.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.3 Per-step reliability rises with scale but never reaches one
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.5 Data and code availability · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and -0.40 for padded (p=0.51,(statistically indistinguishable from natural).”
- Also stated in: § 4.4 Degradation accelerates within a trajectory · plain
“Collapsing all operations into a single prompt decays at essentially the same rate as the natural regime, just from a lower starting point, which rules out multi-turn conversational structure itself as the driver.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 6. Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.4 Degradation accelerates within a trajectory · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across the open-model ladder, per-step reliability rises with parameter count (Pearson r=+0.36 vs. log10(params); Author's primary research data), and the deployed proprietary models cluster alongside the strongest open models rather than exceeding them by a wide margin.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.3 Per-step reliability rises with scale but never reaches one
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the shortest horizon tested, eight of the nine models solve the task at or near ceiling, confirming this is not a task any model finds intrinsically hard in isolation.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.2 The agentic task: deployed models collapse within sixteen steps
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The natural regime performs best at every horizon tested.Both interventions perform worse:”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 6. Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The magnitude of decay varies with both model and task difficulty.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 3.5 Data and code availability
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By horizon 4, the models are already separate into two rough bands.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.2 The agentic task: deployed models collapse within sixteen steps
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The first error tends to land roughly a third of the way through a trajectory, and once an agent errs it rarely recovers, consistent with prior reports that models get lost after a wrong turn and do not find their way back [11].”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.4 Degradation accelerates within a trajectory
What the sources say
- Not matchedcites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
- It may state two separate claims as one. 14
- Probably not a substantial claim. 13
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“format and tool-call drift, a turn whose output cannot be parsed as a valid action, affects 21% of trajectories and rises with horizon , pointing to an interfacelevel failure mode compounding alongside the reasoning failures.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.6 Mechanism
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“bounding the context window steepens decay rather than easing it (logit slope -0.69 vs. -0.44),p=3x10-6)”
- Also stated in: § 6. Conclusion · plain
“the logit slope per horizon doubling is -0.44 for natural, -0.69 for compressed (p=3x10-6 (vs.natural), and -0.40 for padded (p=0.51,(statistically indistinguishable from natural).”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > Abstract
plaindirectn = 10,664 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 6. Conclusion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This data shows the hazard rising instead: pooling across long-horizon trajectories, mean per-step accuracy falls from 0.58 in the first third of a trajectory to 0.44 in the last third .”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.4 Degradation accelerates within a trajectory
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Refchain shows the flattest decay of the three for the strongest models.Ledger shows a visible but moderate decay across most models.Cipher shows the steepest decay of the three streaming families for every model tested, including the largest ones, because a single misapplied edit early in the sequence invalidates every downstream step regardless of how well the model reasons afterward.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 3.5 Data and code availability
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Qwen2.5-72Bsolves the task perfectly at both horizon 2 and horizon 4, which would read as an unambiguous success on any benchmark that stops sampling at those horizons.By horizon 16, the same model has fallen to roughly one in eight trials succeeding.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.2 The agentic task: deployed models collapse within sixteen steps
plaincorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our disentangling design tests that prediction directly and finds the opposite: bounding context worsens performance rather than improving it, which implicates step count instead.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 2. Related Work
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The agentic toolqa family directly addresses the not-really-agentic objection this design choice invites, but broader ecological validity against fully open-ended realworld agentic workflows remains an open question beyond what this study can settle.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 5. Limitations and Threats to Validity
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, this study measures reliability under a fixed, moderate decoding temperature; sensitivity to temperature and other decoding parameters is a natural direction for follow-up work.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 5. Limitations and Threats to Validity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our compressed regime instead removes information the agent itself generated during the trajectory, which is a different failure mode entirely: it is not that the model cannot locate a fact in a long document, but that it loses the scaffold of its own prior reasoning once that scaffold is pruned away.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 2. Related Work
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Because the first error concentrates early in a trajectory and per-step accuracy never recovers once an error occurs, errors behave as absorbing states that propagate silently, much as hallucinated content can survive undetected without an external verifier [15].”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.6 Mechanism
What the sources say
- Not matchedcites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Scale and frontier engineering buy per-step reliability, but with diminishing returns, and no model in this study reaches perfect per-step reliability on any non-trivial task.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.3 Per-step reliability rises with scale but never reaches one
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our design instead fixes the models under test and resolves success directly against horizon inside a controlled environment, isolating the shape of the decay curve rather than only its trend over calendar time.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 2. Related Work
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Temperature is fixed for agentic determinism, and seeds are recorded for reproducibility.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 3.3 Models, design, and metrics
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Three additional small models were excluded during data collection for failing to adhere to the structured response protocol via their hosted routes.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 5. Limitations and Threats to Validity
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The same underlying task instance repeats across all three regimes so comparisons stay paired.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 3.2 Context regimes: disentangling step count from context length
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Model selection among geometric, threshold, and linear decay shapes uses AIC rather than a fixed significance threshold.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 3.4 Statistical procedure
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The nine-model sample spans a wide parameter range and six vendor families, but it cannot establish that every current or future model follows the identical geometric form.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 5. Limitations and Threats to Validity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Confidence intervals on per-cell success use the Wilson score interval rather than a normal approximation, since several cells involve success rates close to 0 or 1.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 3.4 Statistical procedure
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Our agentic task family uses an unmodified ReAct loop specifically so the degradation we measure reflects the mechanism practitioners actually deploy rather than an artifact of a bespoke control structure.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 2. Related Work
What the sources say
- Not matchedcites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All comparisons across regimes and models use the paired structure of the design: because the same underlying task instance is reused across the natural, compressed, and padded regimes, regime comparisons are computed as paired differences rather than independent-sample comparisons, which increases statistical power.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 3.4 Statistical procedure
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This ordering, refchain flattest, ledger intermediate, and cipher steepest, holds consistently enough across the model roster that task structure appears to matter almost as much as raw model capability in determining where a given model's decay curve sits.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 3.5 Data and code availability
hedgedcorrelationaln = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This design produces 10,664 total analyzed trajectories across the streaming and agentic families combined .”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 3.3 Models, design, and metrics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The agentic toolqa family directly addresses the not-really-agentic objection this design choice invites, but broader ecological validity against fully open-ended realworld agentic workflows remains an open question beyond what this study can settle.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 5. Limitations and Threats to Validity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using mean measured per-step reliability (r=0.61) and the geometric law, we project task success onto horizons representative of widely used benchmarks: 0.42 at GAIA-length horizons (8 steps), 0.36 at WebArena-length horizons (15 steps), 0.33 at tau-bench-length horizons (20 steps), 0.30 at SWE-bench/OSWorld-length horizons (30 steps), and 0.24 at hundred-step production horizons.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.7 The benchmark gap, quantified
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
Benchmarks predominantly sample short-to-medium horizons, while production agents typically operate an order of magnitude longer.
“benchmarks are dominated by short-to-medium horizons where success remains high, while production workloads demand an order of magnitude more dependent steps.”
“Benchmarks predominantly sample short-to-medium horizons; production agents typically operate an order of magnitude longer.”
The authors state this as an assumption.
Each step succeeds independently with probability r, so a task of horizon H succeeds with probability approximately r to the power H.
“If each step succeeds independently with probability r, a task of horizon H succeeds with probability approximately r to the power H, close to 1 for short tasks, collapsing geometrically as H grows.”
The authors state this as an assumption.
A pure geometric decay law assumes a constant per-step hazard rate.
“A pure geometric decay law assumes a constant per-step hazard rate.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“Using mean measured per-step reliability (r=0.61) and the geometric law, we project task success onto horizons representative of widely used benchmarks: 0.42 at GAIA-length horizons (8 steps)”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.7 The benchmark gap, quantified
Reported 0.42 · recomputed 0.019171
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.2 The agentic task: deployed models collapse within sixteen steps
“At the shortest horizon tested, eight of the nine models solve the task at or near ceiling”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.2 The agentic task: deployed models collapse within sixteen steps
“Llama-3.2-1B0.00 0.00 0.00 0.00 Qwen2.5-7B1.00 0.07 0.13 0.33 Llama-3.1-8B1.00 0.73 0.07 0.13 GPT-4o-mini 1.00 1.00 0.20 0.07 Gemini-2.5-Flash-Lite0.20 0.53 0.00 0.00”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.2 The agentic task: deployed models collapse within sixteen steps
“Llama-3.2-1Bnever exceeds roughly half of its already modest starting success rate even at the shortest horizon tested.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.2 The agentic task: deployed models collapse within sixteen steps
“Llama-3.2-1B0.00 0.00 0.00 0.00”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 3.5 Data and code availability
“Success declines monotonically with horizon wherever the task is non-trivial for the model in question.”
§ How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making > 4.2 The agentic task: deployed models collapse within sixteen steps
“Qwen2.5-7B1.00 0.07 0.13 0.33”
Citations
Cites
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · Cited by 1,068 · cited 1×
“Synthetic long-context probes such as RULER [14] and NoLiMa [18] test retrieval and reasoning at extreme context lengths independent of any agentic loop, generally finding that effective context usage falls well short of a model’s advertised context window.”
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608
LLMs Get Lost In Multi-Turn Conversation
Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025) · arXiv · Cited by 419 · cited 2×
“Dziri et al. [4] show transformer accuracy decays sharply with compositional complexity, and Laban et al. [11] report large single-to-multi-turn performance drops in conversational settings.”
Faith and Fate: Limits of Transformers on Compositionality
Dziri, N., Lu, X., Sclar, M., Li, X. (., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Bras, R. L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3081 · Cited by 335
(2023) · doi:10.52202/075280-1220 · Cited by 134
Levy, M., Jacoby, A., & Goldberg, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.818 · Cited by 119
AgentBench: Evaluating LLMs as Agents
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.03688 · Cited by 79
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.07972 · Cited by 64
WebArena: A Realistic Web Environment for Building Autonomous Agents
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.13854 · Cited by 31
TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
Xu, F. F., Song, Y., Li, B., Tang, Y., Jain, K., Bao, M., Wang, Z. Z., Zhou, X., Guo, Z., Cao, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.14161 · Cited by 17
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
Trivedi, H., Khot, T., Hartmann, M., Manku, R., Dong, V., Li, E., Gupta, S., Sabharwal, A., & Balasubramanian, N. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.850 · Cited by 14
Measuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14499 · Cited by 14
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2407.01502 · Cited by 13
GAIA: a benchmark for General AI Assistants
Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2311.12983 · Cited by 12
$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024) · arXiv · doi:10.48550/arxiv.2406.12045 · Cited by 9
Research works printed but not linked: 4
- N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang, "Lost in the Middle: How Language Models Use Long Contexts," TACL, 12, 157-173, 2024. https://aclanthology.org/2024.tacl-1.9/
- M. Cemri, M. Z. Pan, S. Yang, et al., "Why Do Multi-Agent LLM Systems Fail?," NeurIPS, 2025.
- A. Modarressi, H. Deilamsalehy, F. Dernoncourt, et al., "NoLiMa: Long-Context Evaluation Beyond Literal Matching," arXiv preprint, 2025.
- S. G. Patil, H. Mao, et al., "The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation," ICML, 2025.
Cited by
Sorted by citations
When Is Smaller Greener? Right-Sizing Language Models for Agentic AIPreprint
Siraj, S., & Barbati, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.19345 · Cited by 1
Metadata
- Authors
- Mittal, S.
- Corresponding author
- Mittal, Shubhra
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2609.01660
- OpenAlex
- W7207803844
- Semantic Scholar
- cffd95e0b09cb52aafdf865991a5a7f1d205f875
- Cited by
- 1
- References
- 18
- Citations by year
- 2026: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- LLM agents, multi-step workflows, agent reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
