Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data

Chaganty et al. (2018) found their method, essentially equivalent to PPI, only improved data efficiency by around 10% using 2018's automated metrics.

“The authors find that their method, which is essentially equivalent to PPI, only improved data efficiency by around 10% using 2018's automated metrics.”

• cites Chaganty et al. (2018) sourceSupported

For binary classification with judge errors independent of the performance of different models, a sufficiently large sample size guarantees correct rankings as long as the error rate is below 50%.

“For binary classification with judge errors that are independent of the performance of different models, a sufficiently large sample size guarantees correct rankings as long as the error rate is below 50% (Dorner & Hardt, 2024).”

• cites Dorner & Hardt, 2024 sourcePartly supported

  Section 3 shows that when label accuracy exceeds 50% and the better classifier has a positive accuracy margin, noisy labels favor it in expectation. Section 3.3 bounds the probability of ranking the best classifier incorrectly as sample size grows, but guarantees a high probability of a correct ranking—not certainty for any finite sample. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

The Prediction Powered Inference framework and an equivalent method by Chaganty et al. can construct an unbiased estimator using a large IID sample of model judgments and a small subset of corresponding ground truth labels.

“Specifically, we follow the Prediction Powered Inference (PPI) framework (Angelopoulos et al., 2023a) and an for our case equivalent method by Chaganty et al. ( 2018): Alongside a large IID sample of model judgments si (m) = s(x i , m) for integer i ∈ [1, N + n], we assume access to the corresponding ground truth label s j (m) = s(x j , m) for a small subset of integer j ∈ [1, n].”

• cites Angelopoulos et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Chaganty et al., 2018 sourcePartly supported

  Section 3.2 describes an unbiased control-variates estimator using independent samples, automatic scores, and human judgments. Section 3.3 says automatic scores must be available for substantially more examples than have human judgments, matching the claimed sampling setup. But the implemented estimator estimates a coefficient from the data and has O(1/n) bias, so the paper does not establish that its practical method is exactly unbiased. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

The squared correlation between the true score and the proxy score is small whenever the judge model performs worse than the evaluated model.

“Our main result (Theorem 6) then shows that this correlation is small whenever the judge model performs worse than the evaluated model.”

• Also stated in: § 3.5 Limited gains for evaluating state-of-the-art models · plain

  “Assume 0.5 ≤ AG(m) ≤ b(m).Then, ρ(s(m), s(m)) 2 ≤ 0.5.”

• AssumptionThe authors state this as an assumption: At the evaluation frontier, the evaluated model's score is larger than the judge's agreement rate.

Using LLM labels heavily perturbs model rankings despite high judge accuracy.

“Using LLM labels heavily perturbs the ranking despite high judge accuracy.”

• Also stated in: § 2.1 Judge bias can strongly perturb model rankings · boosted

  “While the ranking is not fully reversed, it is strongly perturbed.”

For any agreement rate between 0.5 and 1, there exist values of the model score, true positive rate, and true negative rate such that judge bias and squared correlation are zero and the maximum sample efficiency factor is 1.

“Proposition 8.For any agreement rate 0.5 ≤ AG(m) < 1, there exist values of b(m), p(m), q(m) ∈ (0, 1) such that JB(m) = ρ(s(m), s(m)) 2 = 0 and thus τ max = 1.”

The sample efficiency factor for the PPI estimator is upper bounded by one over one minus the squared Pearson correlation between the true and proxy scores.

“Proposition 4. The sample efficiency factor for the PPI estimator is upper bounded by τ ( θP P λ * ) ≤ 1 1 -ρ(s(m), s(m)) 2 , where ρ(s(m), s(m)) 2 = b(m) (1 -b(m)) (p(m) -E s(m)) 2 E s(m)(1 -E s(m)) .”

The main result remains true in a slightly weaker sense when the proxy score is allowed to be continuous rather than binary.

“This result remains true in a slightly weaker sense, when we allow for the proxy s to be continuous rather than binary (Theorem 10).”

In a realistic setting with Llama-3 70B judging the top-10 MMLU models, the ranking is strongly perturbed, though not fully reversed.

“Figure 1b shows a more realistic setting, with LLama-3 70B judging the top-10 MMLU models according to HELM (Liang et al., 2023).While the ranking is not fully reversed, it is strongly perturbed.”

• cites Liang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Proxy scores can cause highly misleading rankings of model performance.

“To motivate the issue, Figure 1 shows how proxy scores can cause highly misleading rankings of model performance.”

When the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half.

“Our main result shows that when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half.”

• Also stated in: § 1 Introduction · plain

  “Whenever the judge model performs worse at its task than the evaluated model, the optimal debiasing method is no better than using twice the ground truth data.”

• AssumptionThe authors state this as an assumption: At the evaluation frontier, the evaluated model's score is larger than the judge's agreement rate.

When the judge model performs worse than the evaluated model, the best possible outcome is an effective doubling of the number of ground truth samples, no matter how many proxy samples are available.

“In that case the best possible outcome is an effective doubling of the number of samples s, no matter how many samples of the proxy score s are available (Corollary 7).”

• Also stated in: § 3.5 Limited gains for evaluating state-of-the-art models · plain

  “Assume 0.5 ≤ AG(m) ≤ b(m).Then, τ max = max θ∈Θ τ ( θ) ≤ 2.”

• AssumptionThe authors state this as an assumption: At the evaluation frontier, the evaluated model's score is larger than the judge's agreement rate.

At the optimal interpolation weight, the interpolated PPI estimator never increases variance compared to the classical ground truth estimator.

“At this λ, the estimator θP P λ * never increases variance compared to the classical ground truth estimator: Var θP P λ * = 1 n Var s(m) - 1 n + n 2 N Cov(s(m), s(m)) 2 Var(s(m)) ≤ 1 n Var s(m) = Var θGT .”

For a binary classifier and a set of strictly better binary classifiers such that the classifier's correct predictions imply the better classifiers' correct predictions, if one better classifier has higher true accuracy than another, then its proxy accuracy is lower.

“Proposition 1.Consider a binary classifier m and a set of strictly better binary classifiers M such that m(x) = y(x) implies m i (x) = y(x) for all m i ∈ M. Let E s(m i ) represent the accuracy of model m i evaluated on the correct labels, and E s(m i ) its accuracy evaluated on predictions of model m.Then for m i , m j ∈ M, E s(m i ) > E s(m j ) implies E s(m i ) < E s(m j ).”

The sample size savings achievable in practice are even more modest than the theoretical limit suggests.

“Through an empirical evaluation, we demonstrate that the sample size savings achievable in practice are even more modest than what our theoretical limit suggests.”

Assume 0.5 is at most the judge agreement rate and the judge agreement rate is at most the evaluated model's score. Then the squared correlation between true and proxy scores is at most 0.5.

“Theorem 6. Assume 0.5 ≤ AG(m) ≤ b(m).Then, ρ(s(m), s(m)) 2 ≤ 0.5.”

• AssumptionThe authors state this as an assumption: At the evaluation frontier, the evaluated model's score is larger than the judge's agreement rate.

Assume 0.5 is at most the judge agreement rate and the judge agreement rate is at most the evaluated model's score. Then the maximum sample efficiency factor over unbiased estimators is at most 2.

“Corollary 7. Assume 0.5 ≤ AG(m) ≤ b(m).Then, τ max = max θ∈Θ τ ( θ) ≤ 2.”

• AssumptionThe authors state this as an assumption: At the evaluation frontier, the evaluated model's score is larger than the judge's agreement rate.

The best possible sample size savings are determined by the Pearson correlation between the true score and the proxy score.

“We first show that the best possible savings are determined by the Pearson correlation between s and s (Theorem 5).”

Even when using the stronger LLaMa3.1 405B and Claude to judge GPT-4, sample efficiency gains above two remain rare: observed in 6 out of 57 (Claude) and 1 out of 57 (LLaMa) cases.

“Even when using the stronger LLaMa3.1 405B (Figure 8d) and Claude (Figure 8c) to judge GPT-4, sample efficiency gains above two remain rare: When only observe them in 6 out of 57 (Claude) and 1 out of 57 (LLaMa) cases.”

The agreement rate does not provide any lower bound on the sample efficiency factor, even when the judge bias is zero.

“Our next proposition goes a step further and shows that the agreement AG(m) does not provide any lower bound on the sample efficiency factor τ , even when the judge bias JB(m) is zero.”

For any proxy score s with ε(1-b) ≤ (1-SO(R(s))), ρ^2(s, s) ≤ 1 - ε/2.

“Theorem 15.For any proxy score s with ϵ(1 -b) ≤ (1 -SO(R(s))), we have ρ 2 (s, s) ≤ 1 -ϵ 2 .”

On TruthfulQA, the sample efficiency factor τmax is consistently below two when weaker models are used to evaluate stronger ones.

“As predicted by our theory, the sample efficiency factor τ max is consistently below two when weaker models are used to evaluate stronger ones.”

In an experiment where the judge reproduces the correct label 60% of the time but picks a random wrong label otherwise, model rankings are preserved near-perfectly despite low agreement.

“In that experiment, the judge reproduces the correct label 60% of the time, but picks a random wrong label otherwise.Despite this judge's low agreement with the correct labels, model rankings are preserved near-perfectly.”

The sample efficiency factor τmax on MMLU and MT-Bench consistently stays below two suggested by Corollary 7, except when current flagship models judge the significantly worse LLama2-7b on MMLU.

“The sample efficiency factor τ max consistently stays below the value of two suggested by Corollary 7, except when current flagship models are used to judge the significantly worse LLama2-7b on MMLU.”

In the local maximizing case, the maximum value of ρ(s(m), s(m))^2 is at x = 2/3, where it equals 1/4.

“This means that the maximum value is at x = 2 3 , where ρ(s(m), s(m)) 2 = 1 4 .”

Given black-box estimators that do not model the prompt-conditional distribution and access to N IID proxy samples and n IID joint samples, no unbiased estimator has lower variance than the optimal PPI estimator.

“Given black-box estimators that do not model the prompt-conditional distribution P[(s(m), s(m))|x] and access to N IID samples of proxy scores s(m) and n IID parallel samples (s(m), s(m)) we show that the answer is no:”

Using a non-binary proxy consistently improves sample efficiency in the MMLU experiment.

“As expected, using the non-binary proxy consistently improves sample efficiency.”

As suggested by Theorem 10, the sample efficiency factor τ(θPPλ*) remains below two when using LLama3.1-405B to evaluate the stronger Claude 3.5.

“However, as suggested by Theorem 10, the sample efficiency factor τ ( θP P λ * ) remains below two when we use LLama3.1-405B to evaluate the stronger Claude 3.5.”

τmax is often substantially smaller than two, especially on MMLU.

“We also note that τ max is often substantially smaller than two, especially on MMLU.”

On TruthfulQA, τmax even stays below two when the strongest model considered (Palmyra-x) judges the weakest (LLaMa2 7B).

“Beyond that, the sample efficiency factor τ max even stays below two when we use the strongest model we considered (Palmyra-x) to judge the weakest (LLaMa2 7B).”

The maximum attainable value for ρ^2 is 0.5.

“With this, the maximum attainable value for ρ 2 equals 0.5.”

Unlike in the binary case, the factor two is already slightly exceeded for LLaMa3-70B (which is 5% less accurate than LLama3.1-405B), rather than just for the much weaker LLaMa2-7B.

“That said, unlike in the binary case, the factor two is already (slightly) exceeded for LLaMa3-70B (which is 5% less accurate than LLama3.1-405B),rather than just for the much weaker LLaMa2-7B.”

The optimal interpolation weight is the covariance between the true and proxy scores divided by one plus n over N times the variance of the proxy score.

“We obtain the optimum value λ * = Cov(s(m), s(m))/ 1 + n N Var s (m) .”

τmax stays below two in all other cases, even when stronger models like GPT-4 evaluate weaker models like LLama3-70B.

“Interestingly, τ max stays below two in all other cases, even when stronger models like GPT-4 are used to evaluate weaker models like LLama3-70B.”

Assume b ≥ 0.5. For any 0 < ε ≤ 1, and any binary proxy s such that AG ≥ 0.5 and ε(1-b) ≤ (1-AG), ρ^2 ≤ 1 - ε/2.

“Theorem 16.Assume that b ≥ 0.5.Then for any 0 < ϵ ≤ 1, and any binary proxy s such that AG ≥ 0.5 and ϵ(1 -b) ≤ (1 -AG), we have ρ 2 ≤ 1 -ϵ 2 .”

• AssumptionThe authors state this as an assumption: b ≥ 0.5
• AssumptionThe authors state this as an assumption: AG ≥ 0.5
• AssumptionThe authors state this as an assumption: ε(1-b) ≤ (1-AG)

Assume b ≥ 0.5. For any 0 < ε ≤ 1, take any binary proxy s such that AG ≥ 0.5 and ε(1-b) ≤ (1-AG). Then the recalibrated proxy R(s) fulfills ε(1-b) ≤ (1-SO).

“Proposition 17.Assume that b ≥ 0.5.Then for any 0 < ϵ ≤ 1, take any binary proxy s such that AG ≥ 0.5 and ϵ(1 -b) ≤ (1 -AG).Then, the recalibrated proxy R(s) fulfills ϵ(1 -b) ≤ (1 -SO).”

• AssumptionThe authors state this as an assumption: b ≥ 0.5
• AssumptionThe authors state this as an assumption: AG ≥ 0.5
• AssumptionThe authors state this as an assumption: ε(1-b) ≤ (1-AG)

Model rankings on MMLU using a 60% accurate judge with random errors are preserved despite agreement substantially below LLM judges.

“Model rankings on MMLU using a 60% accurate judge with random errors.Ranks are preserved despite agreement substantially below LLM judges.”

The sample efficiency of PPI is bounded: τ(θ_PPI^λ*) ≤ 2/ε.

“Correspondingly, the sample effficiency of PPI is bounded: τ ( θP P λ * ) ≤ 2 ϵ .”

For any balanced accuracy BA, ρ(s(m), s(m))^2 ≤ |2 BA - 1|.

“Joining both cases, we thus obtain ρ(s(m), s(m)) 2 ≤ |2 BA -1|.”

• AssumptionThe authors state this as an assumption: 0 < b < 1 and BA ≠ 0.5

In some cases, despite access to proxy scores, no unbiased estimator has less variance than the ground-truth sample average estimator.

“It implies that in some cases despite access to s, no unbiased estimator has less variance than θGT .”

If judge bias is positive for low-performing models and negative for better-performing models, using the judge score can easily perturb model rankings.

“If JB(m) is positive for models with low performance, and negative for better performing models, using the score of the judge s can easily perturb model rankings.”

If the correlation between the true and proxy scores is small, the variance inequality is reversed.

“However if the correlation between s(m) and s(m) is small, the inequality is reversed.”

Even if agreement is constant across models, one could only reliably rank models whose true scores differ by more than twice one minus the agreement rate.

“This means that even if AG(m) = r was constant across models m, we could only reliably rank models for which the true score E s(m) differs by more than 2(1 -r).”

Without further assumptions, an agreement rate above 99 percent for each evaluated model would be required to ensure asymptotically correct rankings.

“As it is common for state-of-the-art models to differ in performance by only low single digit percentages, without further assumptions an agreement rate above 99% (for each evaluated model!) would be required to ensure (asymptotically) correct rankings.”

For agreement equal to r, there can be a judge bias of 1 minus r in either direction.

“Proposition 2 shows that for agreement AG(m) = r, there can be a judge bias of 1 -r in either direction.”

For MT-Bench, the ranking of the best models is not consistently preserved when using LLM judges.

“This effect can also be observed in prior work: For MT-Bench, the ranking of the best models is not consistently preserved when using LLM judges (Zheng et al., 2024).”

• cites Zheng et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

In light of the worst-case results from Proposition 2, one cannot rely on small bias for all models when solely basing estimates on proxy scores.

“In light of the worst-case results from Proposition 2, we cannot rely on this to be true when solely basing our estimates on the proxy scores s.”

If sufficiently small estimator bias could be guaranteed, a large amount of proxy samples would yield correct rankings with high probability.

“Proposition 3 implies that if we could guarantee sufficiently small estimator bias, a large amount of proxy samples s would yield correct rankings with high probability.”

When N is much larger than n and the true and proxy scores are strongly correlated, the PPI estimator has lower variance than the ground-truth sample average estimator.

“Whenever N ≫ n, the first term on the right hand side is small.Moreover, when s(m) and s(m) are strongly correlated, the second term is small.Assuming both, we have Var θP P < Var θGT , where θGT is the ground-truth sample average estimator θGT = 1 n n j=1 s j (m).”

Judge bias can strongly depend on the evaluated model.

“However, JB(m) can strongly depend on the evaluated model m.”

If judge bias were constant across all models, model rankings would be unaffected.

“If the judge bias JB(m) was constant across all models, rankings would be unaffected.”

Whenever judge bias has large magnitude, the proxy score misrepresents the performance of the evaluated model even with large sample sizes.

“Whenever JB(m) has large magnitude, the proxy score s misrepresents the performance b(m) of model m, even with large sample sizes.”

If the crowdworker majority vote is less accurate than the evaluated model, Theorem 6 implies that crowdworker labels are of low value compared to ground truth labels.

“Similarly, if the crowdworker majority vote is less accurate than the evaluated model, Theorem 6 implies that crowdworker labels are of low value compared to ground truth labels.”

For any binary proxy s such that 0.5 ≤ AG ≤ b, 0.5 ≤ SO(R(s)) ≤ b.

“Proposition 9.For any binary proxy s such that 0.5 ≤ AG ≤ b, we have 0.5 ≤ SO(R(s)) ≤ b.”

The main claim of no sample efficiency gains of more than a factor two at the frontier is robust.

“This shows that our main claim of”

When using a classifier m with accuracy x > 0.5 to evaluate a strictly better classifier m with accuracy x + δ, with q(m)=0, b(m)=x+δ, p(m)=x/(x+δ), x>0.5, δ>0, x+δ≤1, then ρ2(s(m),s(m)) ≤ 1/9 and τmax ≤ 1.125.

“Proposition 11.Fix q(m) = 0 and assume b(m) = x + δ as well as p(m) = x x+δ for x > 0.5, δ > 0 and x + δ ≤ 1.Then ρ(s(m), s(m)) 2 ≤ 1 9 . Correspondingly, the sample efficiency factor is bounded by τ max ≤ 1.125.”

Whenever BA(m) ≥ 0.5, ρ2(s(m),s(m)) ≤ min{p(m), q(m)}.

“Corollary 13.Whenever BA(m) ≥ 0.5, we have ρ(s(m), s(m)) 2 ≤ min{p(m), q(m)}.”

The exception where τmax exceeds two when flagship models judge LLama2-7b on MMLU is not surprising because AG ≫ b, heavily violating the assumptions for Corollary 7.

“This exception is not surprising, as in that case AG ≫ b such that the assumptions for Corollary 7 are heavily violated.”

For binary classification, the agreement rate equals the judge model's accuracy and is constant across evaluated models.

“For binary classification, however, the agreement rate equals the judge model's accuracy and is thus constant across evaluated models.”

In the configuration from Proposition 8, estimating the judge bias is as hard as directly estimating the real score.

“In the configuration from the proposition, estimating the judge bias JB(m) turns out as hard as directly estimating the real score E s(m).”

Theorem 6 holds for arbitrarily high levels of agreement.

“Theorem 6 holds for arbitrarily high levels of agreement.”

Corollary 7 provides a best-case upper bound rather than a guarantee that τmax will be close to two in practice.

“This highlights that Corollary 7 provides a best-case upper bound rather than a guarantee that τ max will be close to two in practice.”

The upper bound on the sample efficiency factor is large whenever the squared correlation is large.

“This upper bound is large whenever ρ 2 is large.”

The upper bound on τmax from Corollary 7 is fairly robust.

“This suggests that the upper bound on τ max from Corollary 7 is fairly robust.”

For tasks in which evaluation is substantially easier than the task itself, or tasks for which data to train a specialized evaluator is abundant, gains in sample efficiency of more than a factor two are possible even when evaluating models at the frontier.

“Thus, for tasks in which evaluation is substantially easier than the task itself, or tasks for which data to train a specialized evaluator is abundant, gains in sample efficiency of more than a factor two are possible even when evaluating models at the frontier.”

Manually enumerating and correcting for all judge biases appears infeasible.

“While correcting for these known biases (Dubois et al., 2024a;Zheng et al., 2024) is an important first step, manually enumerating and correcting for all judge biases appears infeasible.”

For any value of BA(m), 4b(m)(1-b(m))(2BA(m)-1)^2 ≤ ρ2(s(m),s(m)) ≤ |2BA(m)-1|.

“Theorem 12.For any value of BA(m), we have: 4b(m)(1 -b(m))(2 BA(m) -1) 2 ≤ ρ(s(m), s(m)) 2 ≤ |2 BA(m) -1| .”

The PPI estimator θPPλ* is always optimal in the binary evaluation setting (Theorem 5).

“In comparison to their worst-case result, our Theorem 5 shows that θP P λ * is always optimal in our binary evaluation setting.”

If abstention correlates with the evaluated model's performance, having judges abstain based on confidence can introduce its own biases to model evaluations.

“However, if abstention correlates with the evaluated model's performance, this approach can introduce its own biases to model evaluations.”

Our results do not only apply to LLM judges, but any form of biased evaluators including (poorly instructed) crowdworkers.

“Finally, we would like to reiterate that our results do not only apply to LLM judges, but any form of biased evaluators including (poorly instructed) crowdworkers.”

Theorem 6 only holds if the proxy s is less capable at predicting s than the evaluated model m is at its task.

“Second, Theorem 6 only holds if the proxy s is less capable at predicting s than the evaluated model m is at its task.”

The agreement rate can depend on the evaluated model, not just on the judge.

“In particular, this highlights that the agreement AG(m) can indeed depend on the evaluated model m, rather than just the judge m.”

The sample efficiency factor of the PPI estimator is finite unless the true and proxy scores have perfect correlation.

“However, the sample efficiency factor τ ( θP P λ * ) is finite unless we have perfect correlation ρ 2 = 1.”

In the classification setting, the judge model's accuracy lower bounds the agreement.

“In the classification setting, the judge model's accuracy lower bounds the agreement AG(m): Clearly s(x, m) = s(x, m) whenever the judge model m is correct.”

The squared correlation ρ^2(s, s) falls monotonously in MSE(s, f(s)), where f is the affine transformation with the smallest value of MSE(s, f(s)).

“the squared correlation ρ 2 (s, s) = R 2 (s, f (s)) = 1 -MSE(s,f (s)) Var(s) falls monotonously in MSE(s, f (s)), where f is the affine transformation with the smallest value of MSE(s, f (s)) (Larsen & Marx, 2005).”

• cites Larsen & Marx, 2005Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The Bayes-optimal predictor R(s) minimizes the mean squared error MSE(g(s), s) for all post-processing functions g.

“the Bayes-optimal predictor R(s) minimizes the mean squared error MSE(g(s), s) for all post-processing functions g.”

• cites Gneiting & Raftery, 2007 sourceNot checked — no full text of the source

The mean squared error/Brier score is a proper scoring rule.

“because the Mean squared error/Brier score is a proper scoring rule (Gneiting & Raftery, 2007)”

• cites Gneiting & Raftery, 2007 sourceNot checked — no full text of the source