On Randomness in Agentic Evals

Recent work has identified multiple sources of non-determinism in large language models.

“Recent work has identified multiple sources of non-determinism in large language models.”

• cites Yuan et al. (2025) sourceNot checked — no full text of the source
• cites He & Lab (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ouyang et al. (2025) sourceSupported

For code generation, repeated queries yield different implementations even with greedy sampling.

“For code generation specifically, Ouyang et al. (2025) find that repeated queries yield different implementations even with greedy sampling.”

• cites Ouyang et al. (2025) sourceSupported

Meaning-preserving changes (spacing, punctuation, example ordering, case) cause substantial performance shifts.

“Zhuo et al. (2024); Sclar et al. (2024); Andersson et al. (2025) show that meaning-preserving changes (spacing, punctuation, example ordering, case) cause substantial performance shifts.”

• cites Zhuo et al. (2024) sourcePartly supported

  Section 4.1.3 says the rewritten prompts were checked for semantic similarity, and Section 4.2 reports sensitivity to those variants on challenging questions, while finding relative robustness on simpler questions. The paper does not establish that the specific changes listed—spacing, punctuation, example ordering, and case—caused substantial performance shifts. Page numbers are unavailable in our copy. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Sclar et al. (2024)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Andersson et al. (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The observed variance in single-run pass@1 estimates persists across all configurations, including temperature 0.

“This variance persists across all configurations, including theoretically deterministic settings (temperature 0), where non-determinism from inference engines and environments still clearly produces measurable variance.”

• AssumptionRests on an assumption: In SWE-Bench-Verified, automated unit tests correctly validate issue resolution.

Across all twelve configurations, the ranges of single-run resolution rates span 2.2 to 6.0 percentage points.

“Across all twelve configurations in the table, the ranges span 2.2 to 6.0 percentage points, representing substantial variability where a single run could report performance anywhere within this window.”

• AssumptionRests on an assumption: In SWE-Bench-Verified, automated unit tests correctly validate issue resolution.

Relative to the size of the trace, the first divergence always happens in the 1% of the total trajectory.

“Relative to the size of the trace, the first divergence always happen in the 1% of the total trajectory (bottom of the figure).”

Deterministic sampling (temperature 0.0) shifts divergence substantially later.

“Deterministic sampling (temperature 0.0) shifts divergence substantially later, as expected.”

For DeepSWE-preview on nano-agent, the median first token divergence position increases from 5 at temperature 1.0 to 56 at temperature 0.0.

“For DeepSWE-preview on nano-agent , the median first token divergence position increases from 5 (default temperature 1.0) to 56 when using temperature 0.0.”

Gaps up to 24.9 percentage points exist between best-case (pass@k) and worst-case (pass^k) performance.

“Using pass@k (the probability that at least one of k attempts succeeds) (Chen, 2021) and passˆk (the probability that all k attempts succeed) (Yao et al., 2025), we find gaps up to 24.9 percentage points between bestcase and worst-case performance”

• cites Chen, 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Yao et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

On task django django-9296, two independent runs of Qwen3-32B on nano-agent at temperature 0 generated identical reasoning for the first 93 tokens.

“On task django django-9296 , two independent runs of Qwen3-32B on nano-agent at temperature 0 generated identical reasoning for the first 93 tokens.”

At token 94, within the model's internal reasoning trace, a subtle difference emerged: run 1 reasoned 'Let me search for the Paginator class' while run 2 reasoned 'Let me check the Django source code.'

“At token 94, within the model's internal reasoning trace, a subtle difference emerged: run 1 reasoned 'Let me search for the Paginator class' while run 2 reasoned 'Let me check the Django source code.'”

For DeepSWE-preview on nano-agent, the median first token divergence occurs at token position 5 with default temperature (1.0), that is at 0.5% of the total trajectory length.

“For example, for DeepSWE-preview on nano-agent , the median first token divergence occurs at token position 5 with default temperature (1.0), that is at 0.5% of the total trajectory length.”

Agent trajectories diverge early, often within the first few percent of tokens.

“Through token-level analysis, we show that trajectories diverge early, often within the first few percent of tokens”

The impact of temperature on performance varies significantly across models.

“The impact of temperature on performance varies significantly across models, highlighting the importance of statistical testing rather than relying solely on single-runs or mean differences.”

• AssumptionRests on an assumption: In SWE-Bench-Verified, automated unit tests correctly validate issue resolution.

Temperature 0 does not result in determinism.

“To sum up, temperature 0 does not result in determinism.”

Across all twelve configurations, the maximum improvement from pass@1 to pass@5 is 24.9 percentage points (Devstral-2 on r2e-gym at temperature 0).

“Across all twelve configurations, the maximum improvement from pass@1 to pass@5 is 24.9 percentage points ( Devstral-2 on r2e-gym at temperature 0), demonstrating that scaffold choice and modelscaffold interaction significantly impact the degree of stochastic dependence.”

• AssumptionRests on an assumption: In SWE-Bench-Verified, automated unit tests correctly validate issue resolution.

For DeepSWE-preview on nano-agent at temperature 1.0, mean single-run resolution rate is 31.4% ± 1.0%, with individual runs ranging from 28.8% to 32.4% (3.6 percentage point spread).

“For example, DeepSWE-preview on nano-agent with temperature 1.0 achieves r = 31 . 4 ± 1 . 0% ( pass@1 ), with individual runs ranging from 28.8% to 32.4% (a 3.6 percentage point spread.”

• AssumptionRests on an assumption: In SWE-Bench-Verified, automated unit tests correctly validate issue resolution.

Trajectories typically diverge very early, within the first tokens immediately after the common system and user prompts.

“Figure 2 shows that the distribution of first token divergence reveals that trajectories typically diverge very early, within the first tokens (top figure) immediately after the common system and user prompts.”

For Qwen3-32B on R2E-Gym at temperature 0.6, mean single-run resolution rate is 23.9% ± 1.4%, ranging from 21.4% to 26.4% (5.0 percentage point spread).

“Similarly, Qwen3-32B on R2E-Gym with temperature 0.6 shows a mean of 23 . 9 ± 1 . 4% , ranging from 21.4% to 26.4% (a 5.0 percentage point spread).”

• AssumptionRests on an assumption: In SWE-Bench-Verified, automated unit tests correctly validate issue resolution.

Through 60,000 trajectories and 25.58B tokens across six agentic systems (three models and two scaffolds), the paper quantified substantial variance in single-run pass@1 estimates (2.2-6.0 pp ranges, persisting at temperature 0).

“Through 60,000 trajectories and 25.58B tokens across six agentic systems (three models and two scaffolds), we quantified substantial variance in single-run pass@1 estimates (2.2-6.0 pp ranges, persisting at temperature 0).”

Detecting a 1% improvement at median variance levels requires 36 runs.

“Detecting a 1% improvement at median variance levels requires 36 runs, while the same detection at the lowest observed variance ( σ = 0 . 7% ) would require only 8 runs.”

• AssumptionThe authors state this as an assumption: Pass@1 scores are approximately normally distributed across runs.

Detecting a 2% improvement with 80% power requires 9 runs per agent.

“For example, detecting a 2% improvement with 80% power requires 9 runs per agent, while achieving 95% power for the same effect size requires 15 runs.”

• AssumptionThe authors state this as an assumption: Pass@1 scores are approximately normally distributed across runs.

Detecting a 1% improvement at the lowest observed variance (σ = 0.7%) requires only 8 runs.

“Detecting a 1% improvement at median variance levels requires 36 runs, while the same detection at the lowest observed variance ( σ = 0 . 7% ) would require only 8 runs.”

• AssumptionThe authors state this as an assumption: Pass@1 scores are approximately normally distributed across runs.

The problem was traced to early trajectory divergence (median within first 1% of tokens) that cascades through autoregressive conditioning.

“We traced the problem to early trajectory divergence (median within first 1% of tokens) that cascades through autoregressive conditioning.”

Detecting a 2% improvement with 95% power requires 15 runs per agent.

“For example, detecting a 2% improvement with 80% power requires 9 runs per agent, while achieving 95% power for the same effect size requires 15 runs.”

• AssumptionThe authors state this as an assumption: Pass@1 scores are approximately normally distributed across runs.

Our trajectory analysis demonstrates that the cascading effect might amplify variance.

“Our trajectory analysis (Section 2.2) demonstrates that this cascading effect might amplify variance.”

Detecting a 1% improvement at p < 0.05 with 80% power requires approximately 36 runs per agent under test.

“Detecting a 2% improvement at p &lt; 0 . 05 with 80% power requires approximately 9 runs per agent under test, while detecting a 1% improvement requires 36 runs.”

• AssumptionThe authors state this as an assumption: Pass@1 scores are approximately normally distributed across runs.

Eleven of the 12 configuration-scaffold groups pass the Shapiro-Wilk test at p ≥ 0.05.

“Eleven of the 12 groups pass the test at p ≥ 0 . 05 .”

• Also stated in: § 3.2 RECOMMENDATIONS · p. 9 · plain

  “11 of 12 passed at p ≥ 0 . 05 .”

Performance envelopes show gaps up to 24.9 pp between optimistic and pessimistic bounds.

“We characterized performance envelopes showing gaps up to 24.9 pp between optimistic and pessimistic bounds.”

Detecting a 2% improvement at p < 0.05 with 80% power requires approximately 9 runs per agent under test.

“Detecting a 2% improvement at p &lt; 0 . 05 with 80% power requires approximately 9 runs per agent under test, while detecting a 1% improvement requires 36 runs.”

• AssumptionThe authors state this as an assumption: Pass@1 scores are approximately normally distributed across runs.

The seemingly minor phrasing difference at token 94 propagated through autoregressive conditioning to lead to different first tool calls.

“This seemingly minor phrasing difference propagated through the autoregressive conditioning to lead to different first tool calls, where the first searches for the Paginator class in a specific file while the other searches in an entire directory, leading to different tool call outputs.”

Single-run scores in technical reports and articles can be misleading without statistical bounds.

“The practical implications are significant: single-run scores in technical reports and articles can be misleading without statistical bounds.”

The gap between pass@k and pass^k quantifies how much agent performance depends on favorable stochastic exploration.

“In summary, the gap between pass@k (optimistic bound) and passˆk (pessimistic bound) quantifies how much agent performance depends on favorable stochastic exploration.”

Stochastic exploration genuinely improves the problem-solving capability of DeepSWE-preview.

“indicating that stochastic exploration genuinely improves the problem-solving capability of this model.”

A single early difference can propagate through the trajectory, affecting ever more tokens, tool calls, observations, and ultimately the final outcome.

“This creates a butterfly effect: a single early difference can propagate through the trajectory, affecting ever more tokens, tool calls, observations, and ultimately the final outcome.”

Once a single token differs between two runs, the probability distribution for subsequent tokens changes because the model conditions on the full context including the divergent token.

“Once a single token differs between two runs, the probability distribution (logits) computed by the LLM for subsequent tokens also changes, since the model conditions on the full context including the divergent token.”

Statistical testing is essential to distinguish genuine effects from random variation.

“These examples demonstrate that statistical testing is essential to distinguish genuine effects from random variation, as we further develop in Section 3.2 and Section A.”

The selection of models and scaffolds is designed to mitigate potential implementation-specific artifacts that could confound variance measurements.

“Our selection of models and scaffolds is designed to mitigate potential implementation-specific artifacts that could confound our variance measurements.”

Estimating pass@1 from a single binary outcome per task provides a high-variance estimate of the true success probability.

“First, estimating pass@1 from a single binary outcome per task provides a high-variance estimate of the true success probability.”

The observed variance in evaluation outcomes is substantial enough that improvements measured with a single run might be purely due to randomness.

“This variability is significant enough that improvements measured with a single run might be purely due to randomness in the evaluation process rather than a genuine improvement.”

Reported improvements of 2-3 percentage points may reflect evaluation noise rather than genuine algorithmic progress.

“This variance has critical implications: reported improvements of 2-3 percentage points may reflect evaluation noise rather than genuine algorithmic progress.”

Randomness in agentic trajectories can be leveraged to increase performance via retrying.

“Randomness in agentic trajectories has a good aspect: it can be leveraged to increase performance via retrying.”

Randomness is not a minor perturbation but a fundamental component of agent performance.

“This dependence is substantial across all configurations and demonstrate that randomness is not a minor perturbation but a fundamental component of agent performance, with implications for evaluation methodologies and interpretation of results.”

A newly released model claiming a 2-3 point improvement over its predecessor might simply be reporting a favorable outcome from the same underlying distribution.

“This affects how we interpret progress in the field: a newly released model claiming a 2-3 point improvement over its predecessor might simply be reporting a favorable outcome from the same underlying distribution.”

Small differences in early tokens cascade into different solution strategies through the autoregressive conditioning mechanism of agentic loops.

“and these initial differences cascade into fundamentally different solution strategies through the autoregressive conditioning mechanism of agentic loops.”

The paper verified the normality assumption empirically using the Shapiro-Wilk test.

“We verified this assumption via Shapiro-Wilk tests (Shapiro &amp; Wilk, 1965) across all 12 configurationscaffold groups ( n = 10 runs each); 11 of 12 passed at p ≥ 0 . 05 .”

• cites Shapiro & Wilk, 1965 sourceNot checked — no full text of the source

• AssumptionThe authors state this as an assumption: Pass@1 scores are approximately normally distributed across runs.

All sources of non-determinism impact agentic evaluation scores.

“All sources of non-determinism impact agentic evaluation scores.”

The exponential growth in required sample size for smaller effect sizes demonstrates why single-run evals are insufficient for reliably detecting small improvements.

“The exponential growth in required sample size for smaller effect sizes demonstrates why single-run evals are insufficient for reliably detecting small improvements.”

• AssumptionThe authors state this as an assumption: Pass@1 scores are approximately normally distributed across runs.

Leaderboards that rank systems based on single-run scores may reflect evaluation noise rather than true capability ordering.

“Leaderboards that rank systems based on single-run scores may reflect evaluation noise rather than true capability ordering.”

Even at temperature 0, randomness causes divergent reasoning that compounds through autoregressive conditioning into fundamentally different problem-solving strategies with opposite outcomes.

“Even at temperature 0, randomness causes divergent reasoning that compounds through autoregressive conditioning into fundamentally different problem-solving strategies with opposite outcomes.”

• Also stated in: § 2.4.3 UNDERSTANDING THE MECHANISM: WHEN DO TRAJECTORIES DIVERGE? · p. 8 · plain

  “Even at temperature 0, non-determinism causes trajectory divergence that compounds into fundamentally different problem-solving strategies.”

Since divergence occurs early and propagates through the remainder of the trajectory via the autoregressive conditioning mechanism, longer trajectories exhibit amplified variance.

“Since divergence occurs early and propagates through the remainder of the trajectory via the autoregressive conditioning mechanism, longer trajectories exhibit amplified variance.”

In agentic evals, more than zero-shot prompting, small initial perturbations lead to increasingly divergent outcomes as the trajectory lengthens.

“In agentic evals, more than zero-shot prompting, small initial perturbations lead to increasingly divergent outcomes as the trajectory lengthens.”

Single-run evaluations can lead to researchers not being able to determine whether observed differences represent genuine capability gaps or merely different samples from overlapping performance distributions.

“Single-run evaluations can lead to researchers not being able to determine whether observed differences represent genuine capability gaps or merely different samples from overlapping performance distributions.”

The early divergence cascaded into fundamentally different problem-solving strategies with opposite outcomes.

“At the end, this cascades into fundamentally different problem-solving strategies, and with opposite outcomes.”

The exponential increase in required runs for smaller improvements, particularly at higher variance levels, demonstrates that single-run evals cannot reliably distinguish small performance differences from random variations.

“The exponential increase in required runs for smaller improvements, particularly at higher variance levels, demonstrates that single-run evals cannot reliably distinguish small performance differences from random variations.”

• AssumptionThe authors state this as an assumption: Pass@1 scores are approximately normally distributed across runs.

Research directions may be chosen based on apparent improvements that are not statistically distinguishable from noise.

“Research directions may be chosen based on apparent improvements that are not statistically distinguishable from noise.”

Randomness fundamentally affects the reliability of agentic evals.

“We have demonstrated that randomness fundamentally affects the reliability of agentic evals.”

Machine learning research is no exception to reproducibility challenges.

“Machine learning research is no exception.”

At the infrastructure level, non-associative floating point operations, rounding errors, hardware configuration, and batch size variations impact reproducibility at temperature 0.

“At the infrastructure level, Yuan et al. (2025) and He &amp; Lab (2025) demonstrate that non-associative floating point operations, rounding errors, hardware configuration, and batch size variations impact reproducibility at temperature 0.”

• cites Yuan et al. (2025) sourceNot checked — no full text of the source
• cites He & Lab (2025)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.