Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations

Error bars remain noticeably absent from traditional question-and-answer evals.

“Chatbot Arena[4] has popularized the use of confidence intervals in its Elo scores, but error bars remain noticeably absent from traditional question-and-answer evals.”

• cites [4] [4] Wei-Lin Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference . 2024. arXiv: 2403.04132 [cs.AI] . - sourcePartly supported

  The citation is attached to the statement about Chatbot Arena’s confidence intervals, not the citing authors’ subsequent observation about question-and-answer evaluations. Sections 5 and 7.1 report confidence intervals for Arena’s Bradley–Terry scores, but Section 4 says Arena had replaced its earlier Elo scores with Bradley–Terry coefficients. The cited paper therefore supports the use of intervals, but not their description as intervals for its Elo scores. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [7] [7] Abhimanyu Dubey et al. The Llama 3 Herd of Models . 2024. arXiv: 2407.21783 [cs.AI] . url : https://arxiv.org/abs/2407.21783 . - sourceDisputed

  Section 5.1.1, “Standard Benchmarks” (page unavailable), explicitly says the paper reports 95% confidence intervals for benchmark scores. Its table gives scores with ± intervals for question-answering benchmarks including SQuAD and QuAC, contrary to the claim that error bars are absent from traditional question-and-answer evaluations. (LLM, conf=0.90)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

[7] incorrectly estimates all of its standard errors using the Bernoulli standard error even when scores take fractional values such as F1 scores.

“We also note that [7] incorrectly estimates all of its standard errors using SE Bernoulli , even when s i takes fractional values, such as with an F1 score.”

• cites [7] [7] Abhimanyu Dubey et al. The Llama 3 Herd of Models . 2024. arXiv: 2407.21783 [cs.AI] . url : https://arxiv.org/abs/2407.21783 . - sourcePartly supported

  Section 5.1.1 (page unavailable) describes confidence intervals based on a benchmark score and sample size, and reports intervals for reading-comprehension benchmarks. But the formula is undecoded in our copy, so it does not establish that the calculation uses a Bernoulli standard error for fractional F1 scores. The section also says it omits intervals for scores that are not simple averages; it does not support the claim about all standard errors. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

The number of independent questions n required to achieve a Type I error rate alpha and Type II error rate beta with a given Minimum Detectable Effect delta is given by the paper's sample-size formula.

“Then the number of independent questions n required to achieve a Type I error rate α and Type II error rate β with a given Minimum Detectable Effect δ is”

• cites [14] [14] John A List, Sally Sadoff, and Mathis Wagner. So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design . Working Paper 15701. National Bureau of Economic Research, Jan. 2010. doi : 10.3386/w15701 . url : http: //www.nber.org/papers/w15701 . - sourcePartly supported

  Section 3.1 derives a sample-size formula using the Type I error rate, Type II error rate, and minimum detectable effect, assuming independently drawn treatment and control observations. It gives n as observations per treatment cell, not as a number of independent questions; applying the formula to questions requires assumptions the cited passage does not establish. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

• AssumptionThe authors state this as an assumption: The power analysis assumes a paired analysis and that answers will be sampled K_A times from model A and K_B times from model B.

The confidence intervals for reading-comprehension evals reported in [7] are likely anti-conservative (too narrow).

“We therefore advise that the confidence intervals for reading-comprehension evals reported in [7] are likely anti-conservative (too narrow).”

• cites [7] [7] Abhimanyu Dubey et al. The Llama 3 Herd of Models . 2024. arXiv: 2407.21783 [cs.AI] . url : https://arxiv.org/abs/2407.21783 . - sourceSupported

• AssumptionThe authors state this as an assumption: Draws of clusters are independent.

In a single-token true/false eval with conditional score means uniformly distributed at T=1, reducing temperature to T=0 triples the minimum variance in the score data from 1/12 to 1/4.

“In this case, reducing the sampling temperature, and thereby eliminating the conditional variance, has inadvertently tripled the minimum variance in the score data from 1/12 to 1/4.”

In the paper's real-world example using Anthropic models, clustered standard errors can be over 3 times larger than naive standard errors.

“Analyzing the same data, clustered standard errors can be over 3X larger than naive standard errors.”

• AssumptionThe authors state this as an assumption: Draws of clusters are independent.

In the uniform-difficulty example, next-token probabilities reduce the variance of the estimator by 2/3 compared with grading a single sample from each question.

“Using the uniform-difficulty example from the previous section, next-token probabilities will reduce the variance of the estimator by 2/3 (the upper limit achievable via resampling) compared to grading a single sample from each question.”

In an example with next-token probabilities, uniformly distributed scores, and correlation 0.5, paired differences reduce the variance of the estimator by 1/3 in relative terms, from 1/6 to 1/9 in absolute terms.

“In this case, using paired differences will reduce the variance of the estimator by 1/3 in relative terms (that is, from 1/6 to 1/9 in absolute terms).”

• AssumptionRests on an assumption: The cross-model residuals are uncorrelated.

Under the stated conditions, increasing K_A and K_B from 1 to 10 reduces the Minimum Detectable Effect from 13.2% to 7.5%.

“It follows from Equation 10 that increasing K A = K B from 1 to 10 reduces the Minimum Detectable Effect from 13.2% to 7.5%.”

• AssumptionThe authors state this as an assumption: The power analysis assumes a paired analysis and that answers will be sampled K_A times from model A and K_B times from model B.

Bootstrapping is unnecessary for evals with finite-variance scores and a large number of questions unless a complicated sampling scheme or estimator is used.

“But the Central Limit Theorem is applicable to any evals having scores with finite variance and a large number of questions, and so we regard bootstrapping as unnecessary unless a complicated sampling scheme or estimator is being used.”

• AssumptionThe authors state this as an assumption: The questions in an eval do not represent all possible questions but are drawn at random from a hypothetical, infinite, unseen super-population of questions.
• AssumptionThe authors state this as an assumption: An eval consists of n independently drawn questions.

When scores take fractional values, the Bernoulli standard error tends to be conservative (too wide) compared with the Central Limit Theorem standard error.

“In these cases, SE Bernoulli will tend to be conservative (too wide) compared to SE C . L . T . .”

The standard error of the mean eval score can be estimated using the Central Limit Theorem.

“It follows from the Law of Large Numbers that µ may be estimated using ˆ µ = ¯ s , and from the Central Limit Theorem (C.L.T.) that the standard error of ˆ µ can be estimated as”

• AssumptionThe authors state this as an assumption: The questions in an eval do not represent all possible questions but are drawn at random from a hypothetical, infinite, unseen super-population of questions.
• AssumptionThe authors state this as an assumption: An eval consists of n independently drawn questions.

The Inspect framework correctly computes the Central Limit Theorem standard error with its built-in stderr() metric.

“The Inspect framework[12] correctly computes SE C . L . T . with its built-in stderr() metric.”

• cites [12] [12] Inspect: An open-source framework for large language model evaluations . https://inspect. ai-safety-institute.org.uk/ . Accessed: 2024-09-03. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Once the expected conditional variance divided by K is much smaller than the variance of the conditional mean, increasing K further has little effect on the standard error of the estimated mean eval score.

“Once E [ σ 2 i ] /K ≪ Var( x ), increasing K further will have little effect on the standard error of ˆ µ .”

• AssumptionRests on an assumption: The conditional variances are equal for all K answer scores.

The variance of the estimated mean eval score can be decomposed additively into the variance of the conditional mean and the mean conditional variance, following the law of total variance.

“The variance associated with ˆ µ may be decomposed into two components: the variance of the conditional mean , that is, the variance associated with choosing questions from the superpopulation, and the mean conditional variance , which is the mean variance of scores associated with the questions that were chosen. This decomposition is additive, and follows from the law of total variance.”

• AssumptionThe authors state this as an assumption: The questions in an eval do not represent all possible questions but are drawn at random from a hypothetical, infinite, unseen super-population of questions.

For language model evals that do not use chain-of-thought reasoning, the conditional variance can be removed by analyzing next-token probabilities rather than sampled output.

“For language model evals that do not utilize chain-of-thought reasoning, the conditional variance can be removed by analyzing next-token probabilities, rather than evaluating the model's sampled (or resampled) output.”

• AssumptionThe authors state this as an assumption: The questions in an eval do not represent all possible questions but are drawn at random from a hypothetical, infinite, unseen super-population of questions.

The simplest way to reduce the variance of the estimated mean eval score is to increase the number of sampled questions.

“This equation has a few implications: the simplest way to reduce the variance of ˆ µ is to increase n , the number of sampled questions.”

• AssumptionThe authors state this as an assumption: The questions in an eval do not represent all possible questions but are drawn at random from a hypothetical, infinite, unseen super-population of questions.

If each question is sampled K times and the score is the mean of the K answers, the overall conditional variance becomes the expected conditional variance divided by K.

“Since the conditional variances are equal for all K answer scores, the overall conditional variance becomes”

• AssumptionRests on an assumption: The conditional variances are equal for all K answer scores.

Because eval question scores are likely positively correlated even across unrelated models, paired differences represent a free reduction in estimator variance when comparing two models.

“Because eval question scores are likely to be positively correlated, even across unrelated models, paired differences represent a 'free' reduction in estimator variance when comparing two models.”

• AssumptionRests on an assumption: Eval question scores are likely positively correlated, even across unrelated models.

Computing a pooled standard error across all K times N answers is inconsistent because multiple answers to the same question violate the assumption of independent draws.

“Note that computing a pooled standard error across all KN answers will be inconsistent, as multiple answers to the same question would violate the assumption of independent draws.”

• AssumptionThe authors state this as an assumption: An eval consists of n independently drawn questions.

Paired differences reduce the variance of the estimator as long as the conditional means of the two model scores are correlated.

“We can thus reduce the variance with paired differences as long as the conditional means of the model scores are correlated; that is to say, if the two models have some amount of agreement on which questions are 'easy' and which questions are 'hard'.”

• AssumptionRests on an assumption: The cross-model residuals are uncorrelated.

Failure to adjust standard errors for clustered sampling may lead an analyst to suppose that the overall eval score is measured much more precisely than it actually is.

“Failure to adjust standard errors for clustered sampling may lead an unsuspecting analyst to suppose that the measurement of the overall eval score is much more precise than it actually is.”

• AssumptionThe authors state this as an assumption: Draws of clusters are independent.

The clustered standard error acts as a sliding scale between perfect within-cluster correlation, where each cluster is a single independent observation, and perfect within-cluster uncorrelation, where it equals the unclustered standard error.

“The clustered standard error acts as a kind of 'sliding scale' between cases where scores within a cluster are perfectly correlated (in which case each cluster acts as a single independent observation) and perfectly uncorrelated (in which case the clustered standard error is equivalent to the unclustered case).”

• AssumptionThe authors state this as an assumption: Draws of clusters are independent.