CoChat
Sign Up
PreprintCited by 11Reliability and Agreement in MeasurementInformation Retrieval and Search BehaviorTopic Modeling

Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation

Open access
Mustahsan, Zairah, Lim, Abel, Anand, Megna, Jain, Saahil, McCann, Bryan·2025·arXiv
DOI 10.48550/arxiv.2512.06710arXiv 2512.06710
Open this paper in CoChat — free accountOpen PDF

Abstract

As large language models become components of larger agentic systems, evaluation reliability becomes critical: unreliable sub-agents introduce brittleness into downstream system behavior. Yet current evaluation practice, reporting a single accuracy number from a single run, obscures the variance underlying these results, making it impossible to distinguish genuine capability improvements from lucky sampling. We propose adopting Intraclass Correlation Coefficient (ICC), a metric from measurement science, to characterize this variance. ICC decomposes observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), highlighting whether reported results reflect true capability or measurement noise. We evaluated on GAIA (Levels 1-3, measuring agentic capabilities across varying reasoning complexity) and FRAMES (measuring retrieval and factuality across multiple documents). We found that ICC varies dramatically with task structure, with reasoning and retrieval tasks (FRAMES) exhibit ICC=0.4955-0.7118 across models, and agentic tasks (GAIA) exhibiting ICC=0.304-0.774 across models. For sub-agent replacement decisions in agentic systems, accuracy improvements are only trustworthy if ICC also improves. We demonstrate that ICC converges by n=8-16 trials for structured tasks and n>=32 for complex reasoning, enabling practitioners to set evidence-based resampling budgets. We recommend reporting accuracy alongside ICC and within-query variance as standard practice, and propose updated Evaluation Cards capturing these metrics. By making evaluation stability visible, we aim to transform agentic benchmarking from opaque leaderboard competition to trustworthy experimental science. Our code is open-sourced at https://github.com/youdotcom-oss/stochastic-agent-evals.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes using the Intraclass Correlation Coefficient (ICC) to quantify trial-to-trial inconsistency in agentic evaluations, decomposing variance into between-query task difficulty and within-query agent stochasticity. Across GAIA and FRAMES, ICC varies substantially by task structure and model, so the authors recommend reporting accuracy alongside ICC and within-query variance.

Key findings

  • ICC varied dramatically with task structure: FRAMES showed ICC=0.4955–0.7118 across models, while GAIA showed ICC=0.304–0.774 across models. On GAIA Level 3, GPT-4o had 6.6% accuracy and ICC=0.304, while GPT-5 had 44.2% accuracy and ICC=0.629.
  • Higher accuracy did not always mean higher consistency: on FRAMES, GPT-5 search had the highest accuracy at 77.31% but a lower ICC of 0.496, compared with GPT-4o search at 63.54% accuracy and ICC=0.735.
  • ICC converged by n≈8–16 trials for structured tasks and by n≈32 for complex reasoning; GAIA Level 3 converged more slowly than Levels 1–2.
  • For a fixed budget B=nT, variance was minimized by maximizing the number of tasks n; with B=400 and σ²b/σ²w≈5, allocating n=100, T=4 yielded 68% lower standard error than n=10, T=40.

Methods

  • Evaluated GAIA validation set across Levels 1–3 (53, 86, and 26 questions respectively) and FRAMES using a random sample of 50 of 824 questions (random state=42), with 64 trials per question; o4-mini deep research used 8 trials per question. Models included GPT-4o search, GPT-5 search, Claude 4.5 Haiku/Sonnet, Gemini 2.5 Pro, Qwen3-235b-a22b, and Deepseek-v3p1, with o4-mini as LLM judge; timeouts were 120 seconds and failed runs were recorded as incorrect.
  • Used a one-way random effects model, ICC(1,1), to decompose observed variance into between-query variance (task difficulty) and within-query variance (agent inconsistency), with confidence intervals, ICC convergence analysis across trial counts, and budget allocation analysis.

Limitations

  • The paper states that ICC analysis assumes binary correct/incorrect scoring; extension to partial-credit metrics requires reformulation. It also states that o4-mini was used as an LLM judge without formal validation of inter-trial consistency, that ICC(1,1) treats questions as random effects, and that generalization is limited because the analysis focuses on reasoning and search tasks with OpenAI models. FRAMES used only 50/824 questions, and deep research evaluation used one agent with 8 trials.
  • Evident from the text: model comparisons vary in tool availability (some with web search, some without), which may confound ICC comparisons; the FRAMES subsample may not represent the full benchmark; and evaluation was conducted in October 2025, so API/model versions may change.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2512.06710

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 18 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Premisenone given
    § Background and Motivation · p. 2

    ICC has not been systematically applied to agentic AI evaluation, where variance arises from both task difficulty and agent stochasticity.

    “While ICC is standard practice for evaluating inter-rater reliability and test-retest consistency in clinical settings, it has not been systematically applied to agentic AI evaluation, where variance arises from both task difficulty and agent stochasticity.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Koo and Li 2016 source

      The Introduction and Conclusion say ICC is widely used in conservative care medicine to evaluate interrater and test-retest reliability, supporting the clinical-practice part of the sentence. The cited passages do not address agentic AI evaluation or variance from task difficulty and agent stochasticity, so they do not support the claim about that application. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation
    § Background and Motivation · p. 1

    Some studies attempt to reduce bias from agent inconsistency by reporting average@k values (k=3), but without confidence intervals or statistical significance measures.

    “Some studies attempt to reduce bias from agent inconsistency by reporting average@k values (where k=3), but still do so without providing confidence intervals or statistical significance measures (Yao et al. (2024)Team (2025)).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Yao et al. (2024) source

      Section 5 (Experiments; page unavailable) says the main results use at least three trials per task, and Section 3 says the paper reports average reward across tasks. But Section 3 calls that main metric pass^1, not average@k with k=3; the cited passages also do not establish whether the paper provides confidence intervals or statistical significance measures. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Team (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation
    § Stochasticity in Agentic Evaluations · p. 2

    Agentic evaluations are inherently stochastic processes.

    “Agentic evaluations are inherently stochastic processes.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Brown et al. 2020 source

      Section 6.1.3 (p. 35) says language-model outputs are stochastic and discusses a bot that may occasionally produce incoherent outputs. It does not establish that agentic evaluations themselves are inherently stochastic processes; the claim extends the point from model outputs to evaluations. (LLM, conf=0.80)

      Experimental — accuracy not yet measured
    • cites Holtzman et al. 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites He and Lab 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data
    § Introduction · p. 1

    Replicating GAIA and FRAMES benchmark runs across multiple trials and all three difficulty levels shows evaluation stability varies significantly with task complexity and model capability.

    “Replicating these benchmark runs across multiple trials and all three difficulty levels shows that evaluation stability varies significantly with task complexity and model capability, a finding obscured by current single-run reporting practices.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Tasks are random effects sampled from a larger population of possible evaluation items.
    • The authors state this as an assumption: Trials are random effects representing stochasticity in the agent's behavior.
    • The authors state this as an assumption: ICC(1,1) measures the reliability of an individual trial, with within-task variance estimated by pooling variances across all tasks.
    • Rests on an assumption: The evaluation tasks represent a sample from a broader domain of interest.
  • Findingreasoning
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 4

    With B=400 and typical variance ratio σ2b/σ2w≈5, allocating n=100, T=4 yields 68% lower standard error than n=10, T=40.

    “For example, with B = 400 and typical variance ratios σ 2 b /σ 2 w ≈ 5 , allocating n = 100 , T = 4 yields 68% lower standard error than n = 10 , T = 40 (see Figure 3 in Appendix A).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 22
  • May combine two claims. 38
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 4

    ICC ≥0.75 indicates good reliability; ICC 0.50-0.75 indicates moderate reliability; ICC <0.50 indicates poor reliability.

    “ICC ≥ 0.75 : Good reliability.”
    • Also stated in: § Recommended Statistical Protocols for Agentic Benchmarks · p. 4 · plain
      “ICC = 0.50-0.75 : Moderate reliability.”
    • Also stated in: § Recommended Statistical Protocols for Agentic Benchmarks · p. 4 · plain
      “ICC &lt; 0.50 : Poor reliability.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Koo and Li 2016 source

      The ICC Interpretation section calls values between 0.75 and 0.9 good reliability, but values greater than 0.90 excellent. Thus the cited source does not support the citing sentence’s unbounded “ICC ≥ 0.75” category. The source also presents these thresholds as a rule of thumb, not a universal standard. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 4

    The contribution of within-task variance to total variance scales as σ2w/T, so more trials reduce its impact.

    “The contribution of within-task variance to total variance scales as σ 2 w /T (Miller 2024), so more trials reduce its impact.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Miller 2024 source
    • The authors state this as an assumption: Trials are random effects representing stochasticity in the agent's behavior.

  • Classified as probably not a substantial claim.

    Premisecitation
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 4

    ICC values provide standardized measures of evaluation stability.

    “ICC values provide standardized measures of evaluation stability (Koo and Li 2016):”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Koo and Li 2016 source

  • May combine two claims.

    Premisecitation
    § Stochasticity in Agentic Evaluations · p. 2

    Underlying LLM models sample from probability distributions rather than deterministically computing outputs, introducing trial-to-trial variance.

    “The underlying LLM models sample from probability distributions rather than deterministically computing outputs (Brown et al. 2020; Holtzman et al. 2020; He and Lab 2025), introducing trial-to-trial variance..”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Brown et al. 2020 source

      Appendix F (p. 48) reports four GPT-3 completions from the same prompt, generated with temperature-1 nucleus sampling. This supports the point that sampled outputs can vary between trials, but the paper does not establish that LLM outputs are always sampled rather than produced with deterministic decoding. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Holtzman et al. 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites He and Lab 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Datasets and Evaluation Scope · p. 4

    FRAMES comprises challenging multihop questions requiring integration of information from multiple sources, with baseline LLM performance around 0.408 accuracy.

    “FRAMES comprises challenging multihop questions requiring integration of information from multiple sources, with baseline LLM performance around 0.408 accuracy.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Krishna et al. 2024 source

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    ICC converges by n=8-16 trials for structured tasks and n ≥ 32 for complex reasoning.

    “We demonstrate that ICC converges by n=8-16 trials for structured tasks and n ≥ 32 for complex reasoning, enabling practitioners to set evidencebased resampling budgets.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Tasks are random effects sampled from a larger population of possible evaluation items.
    • The authors state this as an assumption: Trials are random effects representing stochasticity in the agent's behavior.
    • The authors state this as an assumption: ICC(1,1) measures the reliability of an individual trial, with within-task variance estimated by pooling variances across all tasks.
    • Rests on an assumption: The evaluation tasks represent a sample from a broader domain of interest.

  • May combine two claims.

    Findingown data
    § Abstract · p. 1

    ICC varies dramatically with task structure: FRAMES reasoning and retrieval tasks exhibit ICC=0.4955-0.7118 across models, and GAIA agentic tasks exhibit ICC=0.304-0.774 across models.

    “We found that ICC varies dramatically with task structure, with reasoning and retrieval tasks (FRAMES) exhibit ICC=0.4955-0.7118 across models, and agentic tasks (GAIA) exhibiting ICC= 0.304-0.774 across models.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Tasks are random effects sampled from a larger population of possible evaluation items.
    • The authors state this as an assumption: Trials are random effects representing stochasticity in the agent's behavior.
    • The authors state this as an assumption: ICC(1,1) measures the reliability of an individual trial, with within-task variance estimated by pooling variances across all tasks.
    • Rests on an assumption: The evaluation tasks represent a sample from a broader domain of interest.

  • May combine two claims.

    Findingown data
    § Interpretation by Level · p. 5

    GPT-5 outperforms GPT-4o on GAIA Level 2 by +31.0 pp accuracy and +0.083 ICC.

    “and Level 2 (+31.0 pp accuracy, +0.083 ICC)”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • The quote only partly states this claim.

    Findingown data
    § Stochasticity in Agentic Evaluations · p. 2

    Per-question accuracy estimates with 95% confidence intervals show substantial trial-to-trial inconsistency that single-run reporting obscures.

    “Wide confidence intervals show trial-to-trial variance in agent behavior.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: o4-mini as an LLM judge accurately evaluates whether agent outputs match ground-truth answers.

  • May combine two claims.

    Findingown data
    § Interpretation by Level · p. 5

    For GAIA Levels 1-2, GPT-4o ICC is 0.561-0.662 and GPT-5 ICC is 0.745-0.774.

    “Levels 1-2 (Easier-Medium Reasoning): Both levels exhibit moderate-to-high ICC for GPT-4o (0.561-0.662) and high ICC for GPT-5 (0.745-0.774)”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • May combine two claims.

    Findingown data
    § Interpretation by Level · p. 5

    GPT-5 outperforms GPT-4o on GAIA Level 1 by +39.6 pp accuracy and +0.213 ICC.

    “GPT-5 outperforms GPT-4o substantially: Level 1 (+39.6 pp accuracy, +0.213 ICC)”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • May combine two claims.

    Findingown data
    § FRAMES: ICC in Retrieval and Factuality Tasks · p. 5

    On FRAMES, GPT-5 with web search exhibits lower ICC (0.4955) than GPT-4o with web search (0.7355) despite having higher accuracy.

    “However, GPT-5 with web search exhibits lower ICC (0.4955) than GPT-4o with web search (0.7355) on FRAMES despite having higher accuracy”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • May combine two claims.

    Findingown data
    § Deep Research Agents · p. 6

    o4-mini deep research on FRAMES (n=8, 100 questions) achieves 74.6% accuracy and ICC=0.664.

    “1. Frames (n=8, 100 questions): 74.6% accuracy, ICC=0.664”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • May combine two claims.

    Findingown data
    § Interpretation by Level · p. 5

    GPT-5 improves GAIA Level 3 accuracy to 44.2% (+37.6 pp) and ICC to 0.629 (+0.325).

    “GPT-5 dramatically improves both accuracy (44.2%, +37.6 pp) and consistency (ICC=0.629, +0.325).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • May combine two claims.

    Findingown data
    § FRAMES: ICC in Retrieval and Factuality Tasks · p. 5

    GPT-5 search has the highest accuracy but the lowest stability and determinism in response compared to GPT-4o and GPT-4o search on FRAMES.

    “Though GPT-5 search has the highest accuracy, it also has the lowest stability and determinism in response when compared to GPT-4o and GPT-4o search.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • May combine two claims.

    Findingown data
    § Deep Research Agents · p. 6

    o4-mini deep research on GAIA (n=8, full validation set, 165 questions) achieves 40.0% accuracy and ICC=0.621.

    “2. GAIA (n=8, full validation set, 165 questions): 40.0% accuracy, ICC=0.621”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • May combine two claims.

    Findingown data
    § Appendix D · p. 10

    On GAIA (n=50, seed 42), GPT-5 search achieves 59.44% accuracy and ICC=0.745, while Claude 4.5 Sonnet achieves 39.71% accuracy and ICC=0.756.

    “GPT-5 search achieves the highest accuracy (59.44%) but exhibits lower ICC (0.745) than Claude 4.5 Sonnet (ICC 0.756 at 39.71% accuracy)”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • May combine two claims.

    Findingown data
    § Interpretation by Level · p. 5

    GPT-4o achieves 6.6% accuracy with ICC=0.304 on GAIA Level 3.

    “GPT-4o achieves only 6.6% accuracy with ICC=0.304”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • May combine two claims.

    Findingown data
    § Interpretation by Level · p. 5

    ICC convergence occurs by n ≈ 8-16 for GAIA Levels 1-2 and by n ≈ 32 for Level 3.

    “Convergence occurs by n ≈ 8 -16 for Levels 1-2 and by n ≈ 32 for Level 3”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • May combine two claims.

    Findingown data
    § Appendix D · p. 10

    Other models (Deepseek, Qwen, Gemini) degrade gracefully on harder tasks, maintaining modest ICC (0.70-0.77), while Claude models cluster consistently around ICC 0.73-0.77 regardless of accuracy variation.

    “Other (Deepseek, Qwen, Gemini) degrade gracefully on harder tasks, maintaining modest ICC (0.70-0.77), while Claude models cluster consistently around ICC 0.73-0.77 regardless of accuracy variation.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.
    • The authors state this as an assumption: Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.
    • The authors state this as an assumption: The ICC(1,1) variant treats questions as random effects.
    • The authors state this as an assumption: Performance scores follow a one-way random effects model with a random task effect and within-task error.
    • The authors state this as an assumption: Task effects and within-task error terms are normally distributed with zero mean and constant variance.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 3

    Performance varies due to two distinct sources: between-task variance (some tasks inherently harder) and within-task variance (agent behaves inconsistently on same task across trials).

    “When evaluating agents across multiple trials, performance varies due to two distinct sources: (1) some tasks are inherently harder than others (between-task variance), and (2) the agent behaves inconsistently on the same task across trials (within-task variance).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning
    § Background and Motivation · p. 1

    These differences make agentic evaluations richer, but also more sensitive to randomness.

    “These differences make agentic evaluations richer, but also more sensitive to randomness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisereasoning
    § Stochasticity in Agentic Evaluations · p. 2

    Task specifications, API configuration, retry policies, timeout behavior, and external environments all introduce trial-to-trial variance that ICC directly measures.

    “This inherent stochasticity is compounded by the evaluation framework: task specifications, API configuration, retry policies, timeout behavior, and external environments all introduce trial-to-trial variance that ICC directly measures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § Interpretation by Level · p. 5

    The combined accuracy and ICC improvements indicate GPT-5 is not just more capable but also more reliable, and its performance gains reflect genuine capability gains, not fortunate sampling.

    “The combined accuracy and ICC improvements indicate GPT-5 is not just more capable but also more reliable-performance gains reflect genuine capability gains, not fortunate sampling.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 4

    The precision of ICC can be quantified by its standard error, which for ICC(1,1) is approximately given by Bonett's formula and decreases with both n and T.

    “The precision of ̂ ICC can be quantified by its standard error, which for ICC(1,1) is approximately (Bonett 2002):”
    • Also stated in: § Recommended Statistical Protocols for Agentic Benchmarks · p. 4 · plain
      “This decreases with both n and T , with the relative benefit of additional trials depending on the true ICC value.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Bonett 2002 source
    • The authors state this as an assumption: Tasks are random effects sampled from a larger population of possible evaluation items.
    • The authors state this as an assumption: Trials are random effects representing stochasticity in the agent's behavior.
    • The authors state this as an assumption: ICC(1,1) measures the reliability of an individual trial, with within-task variance estimated by pooling variances across all tasks.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning
    § Datasets and Evaluation Scope · p. 5

    GAIA and FRAMES represent diversity in task structure and complexity.

    “GAIA and FRAMES represent diversity in task structure and complexity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning
    § Datasets and Evaluation Scope · p. 5

    Task design, not just model capability, predicts evaluation stability.

    “Together, they illustrate how task design, not just model capability, predicts evaluation stability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 4

    Lower ICC does not necessarily indicate a bad benchmark; some tasks may inherently require exploration or have legitimate stochasticity in their solutions.

    “Lower ICC does not necessarily indicate a 'bad' benchmark. Some tasks may inherently require exploration or have legitimate stochasticity in their solutions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 4

    When agents have similar accuracy, ICC serves as a tiebreaker-higher ICC indicates more predictable behavior.

    “When agents have similar accuracy, ICC serves as a tiebreaker-higher ICC indicates more predictable behavior.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given
    § Abstract · p. 1

    Current evaluation practice of reporting a single accuracy number from a single run obscures the variance underlying results, making it impossible to distinguish genuine capability improvements from lucky sampling.

    “Yet current evaluation practice, reporting a single accuracy number from a single run, obscures the variance underlying these results, making it impossible to distinguish genuine capability improvements from lucky sampling.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 3

    The variance of the accuracy estimator declines with T; even a modest replication (e.g., T ≥ 5) substantially stabilizes estimates.

    “The variance of this estimator declines with T ; thus, even a modest replication (e.g., T ≥ 5 ) substantially stabilizes estimates.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning
    § Datasets and Evaluation Scope · p. 5

    GAIA's open-ended reasoning with multi-modality naturally introduces higher trial-to-trial variance, while FRAMES' focused retrieval-and-reasoning scope may exhibit different variance patterns.

    “GAIA's open-ended reasoning with multi-modality naturally introduces higher trial-to-trial variance, while FRAMES' focused retrievaland-reasoning scope may exhibit different variance patterns.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 4

    With low T, ICC estimates are less reliable; increasing T provides more confidence in the measured ICC value.

    “With low T , ICC estimates are less reliable. Increasing T provides more confidence in the measured ICC value”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Trials are random effects representing stochasticity in the agent's behavior.

  • May combine two claims.

    Premisereasoning
    § Background and Motivation · p. 1

    Comparisons between agents are frequently made without statistical testing, and scoring functions are not always defined in detail.

    “Comparisons between agents are frequently made without statistical testing, and scoring functions are not always defined in detail.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 4

    The estimated ICC converges to the true population ICC as sample sizes increase.

    “The estimated ICC converges to the true population ICC as sample sizes increase:”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Tasks are random effects sampled from a larger population of possible evaluation items.
    • The authors state this as an assumption: Trials are random effects representing stochasticity in the agent's behavior.
    • The authors state this as an assumption: ICC(1,1) measures the reliability of an individual trial, with within-task variance estimated by pooling variances across all tasks.

  • May combine two claims.

    Interpretationreasoning
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 4

    The strategy of maximizing n is optimal until all available items are exhausted (n=nmax), at which point increasing T is the only option for further variance reduction.

    “This strategy is optimal until all available items are exhausted ( n = n max ), at which point increasing T is the only option for further variance reduction.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § Datasets and Evaluation Scope · p. 4

    GAIA is representative of current evaluation practice: ambitious in scope, widely cited, but methodologically opaque regarding reliability.

    “This makes GAIA representative of current evaluation practice: ambitious in scope, widely cited, but methodologically opaque regarding reliability.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 3

    ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of dataset difficulty (variance between tasks) and agent consistency (variance within tasks).

    “The ICC quantifies the proportion of total variance attributable to differences between tasks and can be understood as a function of both the difficulty of the dataset (variance between tasks) and the consistency of the agent (variance within tasks).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Shrout and Fleiss 1979 source
    • The authors state this as an assumption: Tasks are random effects sampled from a larger population of possible evaluation items.
    • The authors state this as an assumption: Trials are random effects representing stochasticity in the agent's behavior.
    • The authors state this as an assumption: ICC(1,1) measures the reliability of an individual trial, with within-task variance estimated by pooling variances across all tasks.

  • May combine two claims.

    Interpretationown data
    § Interpretation by Level · p. 5

    GAIA Levels 1-2 questions clearly separate by difficulty and agent behavior is relatively consistent.

    “indicating questions clearly separate by difficulty and agent behavior is relatively consistent”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § Recommended Statistical Protocols for Agentic Benchmarks · p. 4

    ICC helps distinguish genuine capability limitations from measurement noise, informing resampling strategies and benchmark design.

    “ICC also helps distinguish genuine capability limitations from measurement noise, informing resampling strategies and benchmark design.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § Implications for Agentic System Design · p. 6

    Without ICC visibility, practitioners risk discovering inconsistency problems in production.

    “Without ICC visibility, practitioners risk discovering inconsistency problems in production.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning
    § Ethical Statement · p. 7

    High accuracy with low ICC indicates unreliable evaluation, not a suitable system for deployment, and signals the need for further investigation before deployment in safety-critical contexts.

    “High accuracy with low ICC indicates unreliable evaluation, not a suitable system for deployment-it signals the need for further investigation before deployment in safety-critical contexts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § Implications for Agentic System Design · p. 6

    It is possible to reduce within-question variance (improving ICC) through better prompting or tool design without changing accuracy.

    “Yet it's possible to reduce withinquestion variance (improving ICC) through better prompting or tool design without changing accuracy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § Implications for Agentic System Design · p. 6

    GAIA Level 3 demonstrates that GPT-5's 7× accuracy improvement paired with doubled ICC signals a genuine, deployable improvement.

    “GAIA Level 3 demonstrates this: GPT-5's 7× accuracy improvement paired with doubled ICC signals a genuine, deployable improvement.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning
    § Implications for Agentic System Design · p. 6

    Consistency is a tunable property independent of accuracy.

    “Consistency is a tunable property independent of accuracy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § Implications for Agentic System Design · p. 6

    By monitoring ICC, practitioners directly optimize for predictability and stability that enable robust agentic systems.

    “By monitoring ICC, practitioners directly optimize for the emergent properties-predictability and stability-that enable robust agentic systems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § Interpretation by Level · p. 5

    GPT-5's ICC improvement on GAIA Level 3 nearly doubles GPT-4o's value, suggesting GPT-5's superior reasoning makes hard questions succeed/fail more deterministically based on their actual difficulty.

    “The improvement in ICC is particularly striking: it nearly doubles GPT-4o's value, suggesting GPT-5's superior reasoning makes hard questions succeed/fail more deterministically based on their actual difficulty.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning
    § Implications for Agentic System Design · p. 6

    Improvements that reduce within-question variance are invisible to standard metrics but critical for system reliability.

    “Such improvements are invisible to standard metrics but critical for system reliability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § Implications for Agentic System Design · p. 6

    When LLM-based agents become sub-components of larger systems, system-level reliability depends on both capability and consistency.

    “When LLM-based agents become sub-components of larger systems, system-level reliability depends on both capability and consistency.”
    Quote in the paperNot a substantial claim
49 of 49

  • The authors state this as an assumption.

    Tasks are random effects sampled from a larger population of possible evaluation items.

    “Tasks are random effects sampled from a larger population of possible evaluation items”

  • The authors state this as an assumption.

    Trials are random effects representing stochasticity in the agent's behavior.

    “Trials are random effects representing stochasticity in the agent's behavior”

  • The authors state this as an assumption.

    ICC(1,1) measures the reliability of an individual trial, with within-task variance estimated by pooling variances across all tasks.

    “We report single-trial reliability: ICC(1,1) measures the reliability of an individual trial, with within-task variance estimated by pooling variances across all tasks”

  • The paper relies on this without support.

    The evaluation tasks represent a sample from a broader domain of interest.

    “making it appropriate when evaluation tasks represent a sample from a broader domain of interest”

  • The paper relies on this without support.

    o4-mini as an LLM judge accurately evaluates whether agent outputs match ground-truth answers.

    “We used o4-mini as a LLM judge to evaluate whether agent outputs matched ground-truth answers.”

  • The paper relies on this without support.

    Failed runs (timeouts, unrecoverable errors) recorded as incorrect answers reflect real-world deployment scenarios.

    “Failed runs (timeouts, unrecoverable errors) were recorded as incorrect answers, reflecting real-world deployment scenarios.”

  • The paper relies on this without support.

    The random sample of 50 FRAMES questions is representative of the full FRAMES dataset.

    “For FRAMES, a random sample of 50 questions ( random state = 42) was used due to computational cost concerns.”

  • The authors state this as an assumption.

    ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.

    “Scoring assumption: ICC analysis assumes binary (correct/incorrect) scoring; extension to partial-credit metrics requires reformulation.”

  • The authors state this as an assumption.

    Scorer variance from using o4-mini as an LLM judge without formal validation of inter-trial consistency contributes minimally to measured ICC.

    “We use o4-mini as LLM judge without formal validation of inter-trial consistency; scorer variance may contribute to measured ICC but is expected to be minimal.”

  • The authors state this as an assumption.

    The ICC(1,1) variant treats questions as random effects.

    “Our ICC(1,1) variant treats questions as random effects; alternative formulations may suit other contexts.”

  • The authors state this as an assumption.

    Performance scores follow a one-way random effects model with a random task effect and within-task error.

    “Under a one-way random effects model:”
    “where α i ∼ N (0 , σ 2 α ) represents the random task effect and ε ij ∼ N (0 , σ 2 ε ) represents the within-task error term.”

  • The authors state this as an assumption.

    Task effects and within-task error terms are normally distributed with zero mean and constant variance.

    “where α i ∼ N (0 , σ 2 α ) represents the random task effect and ε ij ∼ N (0 , σ 2 ε ) represents the within-task error term.”
12 of 12

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Stability in agentic systems varies with task complexity and model capability.

    “Most directly related to our work, Mustahsan et al. (2025) proposes using intraclass correlation to quantify evaluation stability in agentic systems, showing that stability varies with task complexity and model capability.”

    From On Randomness in Agentic Evals (2026) · cites this paper as Mustahsan et al. (2025)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • FRAMES ICC range excludes reported models

    Minor

    The abstract gives 0.7118 as the highest FRAMES ICC across models, but the paper reports 0.7355 for GPT-4o with web search. Table 3 also reports 0.713 for Gemini 2.5 Pro.

  • FRAMES sample size differs for deep research

    Minor

    The evaluation setup says FRAMES used a sample of 50 questions, while the deep-research results report 100 FRAMES questions. The setup notes the deep-research exception for trial count but not for question count.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. 2025. When Not to Rely on the Central Limit Theorem in LLM Evaluation. In Proceedings of the 42nd International Conference on Machine Learning (ICML) . Position Paper Track.

Cited by

Sorted by citations

Authors
Mustahsan, Z., Lim, A., Anand, M., Jain, S., & McCann, B.
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2512.06710
arXiv
2512.06710
OpenAlex
W7111220031
Semantic Scholar
704dc54bf1b3ba908fd75bfc8604d09f6c61b470
Cited by
0
References
24
Type
posted content
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
LLM agents, intraclass correlation coefficient, measurement noise

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review