JudgeMoE: Distributional Aggregation for LLM-as-a-Judge

May combine two claims.

Across all 16 cells, JudgeMoE improves over the strongest local single judge by +0.0393 mean Spearman, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank p = 0.0091 (p = 0.0182 two-sided).

“Applying the same configuration to six additional cells yields a +0.0393 mean gain over the strongest local single judge across 16 cells, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank p = 0.0091.”

• Also stated in: § 5.2 Conditional Aggregation · plain

  “Across all 16 cells, JudgeMoE improves over the strongest single judge in the local pool by +0.0393 mean Spearman, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank p = 0.0091 (p = 0.0182 two-sided).”

May combine two claims.

On the original 10-cell benchmark, JudgeMoE improves mean Spearman over uniform log pooling by +0.079, with one-sided Wilcoxon p = 0.005 and two-sided p = 0.010.

“On the original 10-cell benchmark, JudgeMoE improves mean Spearman over uniform log pooling by +0.079.”

• Also stated in: § 5.2 Conditional Aggregation · plain

  “Judge-MoE gains +0.079 mean Spearman, with Wilcoxon signed-rank tests giving p = 0.005 for the onesided improvement test and p = 0.010 for the two-sided difference test.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Its selected fixed subsets use 3.38 judges on average.

“Its selected fixed subsets use 3.38 judges on average (Appendix Table 23).”

May combine two claims.

BERTScore helps on SummEval@1-10, and a Skywork reward model helps on held-out HelpSteer2 after train-split subset repair.

“BERTScore helps on SummEval@1-10, and a Skywork reward model helps on held-out HelpSteer2 after train-split subset repair.”

May combine two claims.

Hard-score-only features obtain 0.2634 mean Spearman, soft-score-only features obtain 0.3100, hard and soft scores obtain 0.3671, and probability-derived scalar summaries obtain 0.3776.

“Hard-score-only features obtain 0.2634 mean Spearman, soft-score-only features obtain 0.3100, hard and soft scores obtain 0.3671, and probabilityderived scalar summaries obtain 0.3776.”

May combine two claims.

At WMT 1-5, the full-6 log pool degrades sharply and the learned mainline only partially recovers.

“At WMT 1-5, the full-6 log pool degrades sharply and the learned mainline only partially recovers (Table 11).”

May combine two claims.

The additional gain from probability-derived summaries over hard and soft scores is +0.0106 with p = 0.0801.

“The difference between hard and soft scores together and hard scores alone is +0.1036 with a one-sided Wilcoxon signed-rank p = 0.0068, while the additional gain from probability-derived summaries over hard and soft scores is +0.0106 with p = 0.0801.”

May combine two claims.

Across all 16 cells, the mean difference from covariance-aware log pooling is +0.0250 with 9/16 positive cells (p = 0.0964 one-sided).

“The mean difference from uniform log pooling is +0.0601 with 12/16 positive cells (p = 0.0017 one-sided), while the difference from covariance-aware log pooling is +0.0250 with 9/16 positive cells (p = 0.0964 onesided).”

May combine two claims.

A stronger validation-only selector reaches 0.3732 average test Spearman across the eight held-out direct-score cells, close to the 0.3789 obtained by selecting the best family using test performance.

“A stronger validation-only selector reaches 0.3732 average test Spearman across the eight heldout direct-score cells, close to the 0.3789 obtained by selecting the best family using test performance.”

May combine two claims.

Keeping only the validation-selected top-2 subset and retraining sparse JudgeMoE raises WMT 1-5 to 0.1648, above both the best single judge at 0.1413 and the fixed sparse pool at 0.1473.

“Keeping only the validation-selected top-2 subset and retraining sparse JudgeMoE raises WMT 1-5 to 0.1648, above both the best single judge at 0.1413 and the fixed sparse pool at 0.1473 (Table 53).”

May combine two claims.

On the original 10 cells, the sample-weighted mean differences are +0.0413 relative to the best single judge and +0.0549 relative to covariance-aware log pooling; excluding ROSCOE gives +0.0413 and +0.0548, respectively.

“On the original 10 cells, the sample-weighted mean differences are +0.0413relative to the best single judge and +0.0549 relative to covariance-aware log pooling; excluding ROSCOE gives +0.0413 and +0.0548, respectively (Appendix Table 26).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The family-diverse 3-judge control is weaker.

“A family-diverse 3-judge control enforces three different model families across the selected judges; it is weaker and appears in the appendix.”

May combine two claims.

Both new HelpSteer2-validation cells are positive against the best single judge and covariance-aware log pooling.

“Unlike the FLASK support check, both new HelpSteer2-validation cells are positive against the best single judge and covariance-aware log pooling.”

May combine two claims.

Against covariance-aware log pooling and the best single judge, JudgeMoE has positive mean gains, but the confidence intervals cross zero.

“Against covariance-aware log pooling and the best single judge, JudgeMoE has positive mean gains, but the confidence intervals cross zero (Table 2).”

Classified as probably not a substantial claim.

The distributional approach depends on access to informative judge score probabilities.

“The distributional approach also depends on access to informative judge score probabilities.”

May combine two claims.

COMET becomes the strongest single expert at 0.4326 and uniform log pooling rises to 0.3926, while mixed-pool JudgeMoE falls to 0.1620.

“COMET is more instructive: it becomes the strongest single expert at 0.4326 and uniform log pooling rises to 0.3926, while mixedpool JudgeMoE still falls to 0.1620.”

May combine two claims.

WMT MQM@1-5 is different because a weak judge contaminates the pool, so removing that judge and retraining the aggregator improves the result.

“WMT MQM@1-5 is different because a weak judge contaminates the pool, so removing that judge and retraining the aggregator improves the result.”

May combine two claims.

JudgeMoE improves over the best single judge in most cells, but fails on both TopicalChat score ranges and WMT MQM@1-5.

“Appendix Table 27 shows that JudgeMoE improves over the best single judge in most cells, but fails on both TopicalChat score ranges and WMT MQM@1-5.”

Probably not a substantial claim: may not pass S4 (based).

Each judge carries complementary information through its full score distribution, confidence, and entropy.

“while each judge carries complementary information through its full score distribution, confidence, and entropy.”

May combine two claims.

COMET is the strongest standalone evaluator at 0.4326, while mixed-pool JudgeMoE reaches 0.1620.

“COMET is the strongest standalone evaluator at 0.4326, while mixed-pool JudgeMoE reaches 0.1620.”

May combine two claims.

The one-sided paired Wilcoxon tests are p=0.0212 against the best single judge and p=0.0461 against covariance-aware log pooling.

“the corresponding one-sided paired Wilcoxon tests are p=0.0212against the best single judge and p=0.0461 against covariance-aware log pooling.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Several small judges can still lose to a larger judge, as the WMT failure case shows.

“Several small judges can still lose to a larger judge, as the WMT failure case shows.”

May combine two claims.

Across the 10 direct-score cells, the mean judge-error correlation is 0.7775 for the full 6-judge pool and 0.7071 for the strong 3-judge pool.

“Across the 10 direct-score cells, the mean judge-error correlation is 0.7775 for the full 6-judge pool and 0.7071 for the strong 3-judge pool (Appendix Table 20).”

May combine two claims.

The difference between hard and soft scores together and hard scores alone is +0.1036 with a one-sided Wilcoxon signed-rank p = 0.0068.

“The difference between hard and soft scores together and hard scores alone is +0.1036 with a one-sided Wilcoxon signed-rank p = 0.0068, while the additional gain from probability-derived summaries over hard and soft scores is +0.0106 with p = 0.0801.”

May combine two claims.

Across all 16 cells, the mean difference from uniform log pooling is +0.0601 with 12/16 positive cells (p = 0.0017 one-sided).

“The mean difference from uniform log pooling is +0.0601 with 12/16 positive cells (p = 0.0017 one-sided), while the difference from covariance-aware log pooling is +0.0250 with 9/16 positive cells (p = 0.0964 onesided).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The sequential sum of single-judge jobs was 134.4 minutes.

“The sequential sum was 134.4 minutes.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Under the 1K-example SummEval 1-5 workload (roughly 474K processed tokens), the compute proxy corresponds to about 26.5 PFLOPs for the local 3-judge pool, 50.2 PFLOPs for the local 6-judge pool, and 35.1 PFLOPs for DeepSeek-V3.2-Exp.

“Under the 1K-example SummEval 1-5 workload used in our deployment note (roughly 474K processed tokens), this proxy corresponds to about 26.5 PFLOPs for the local 3-judge pool, 50.2 PFLOPs for the local 6-judge pool, and 35.1 PFLOPs for DeepSeek-V3.2-Exp.”

May combine two claims.

At 1-5, sparse composition and dropping one weak judge both improve over naive full-pool aggregation, and the sparse retraining follow-up shows that a cleaned judge pool can repair the failure.

“At 1-5, sparse composition and dropping one weak judge both improve over naive full-pool aggregation, and the sparse retraining follow-up shows that a cleaned judge pool can repair the failure.”

The quote only partly states this claim.

DeepSets + log, T =0.7 achieves the highest average Spearman (0.3832) among learned aggregation configurations.

“DeepSets + log, T =0.7 | 0.3832”

May combine two claims.

Comparisons to covariance-aware fixed pooling and the best single-judge baseline have positive mean gaps, but their confidence intervals cross zero and are not statistically significant.

“Comparisons to covariance-aware fixed pooling and the best single-judge baseline have positive mean gaps, but their confidence intervals cross zero and are not statistically significant.”

May combine two claims.

Mean residual correlation increases, and average Judge-MoE performance drops slightly, indicating that added protocol diversity can fail to yield useful complementarity.

“Mean residual correlation increases, and average Judge-MoE performance drops slightly, indicating that added protocol diversity can fail to yield useful complementarity.”

May combine two claims.

At 1-10, better score resolution lifts both sparse pooling and the learned mainline model, with sparse retraining retaining most of the mainline gain.

“At 1-10, better score resolution lifts both sparse pooling and the learned mainline model, with sparse retraining retaining most of the mainline gain.”

May combine two claims.

Positive ∆ in 7/10 cells; significant positive ∆ in 5/10 cells.

“Positive ∆ in 7/10 cells; significant positive ∆ in 5/10 cells.”

Probably not a substantial claim: may not pass S3 (load-bearing).

WildBench remains difficult for all methods.

“JudgeMoE remains stronger on the two positive follow-up cells, while WildBench remains difficult for all methods.”

May combine two claims.

The best fixed subset uses 3.38 judges on average and is smaller than the full six-judge pool in all eight cells.

“The best fixed subset uses 3.38 judges on average and is smaller than the full sixjudge pool in all eight cells.”

May combine two claims.

The gains vary across dataset and scale cells: the main gains come from SummEval at 1-5, HelpSteer, and WMT MQM at 1-10, while TopicalChat and SummEval at 1-10 remain challenging against strong fixed baselines.

“The gains vary across dataset and scale cells: the main gains come from SummEval at 1-5, HelpSteer, and WMT MQM at 1-10, while TopicalChat and SummEval at 1-10 remain challenging against strong fixed baselines.”

Probably not a substantial claim: may not pass S3 (load-bearing).

FLASK@1-5 ties covariance-aware log pooling up to rounding.

“FLASK@1-5 ties covariance-aware log pooling up to rounding, consistent with the mixed fixed-pool picture.”

May combine two claims.

Rank correlation between per-judge Spearman and mean gate is 0.71 at 1-5 and 1.00 at 1-10.

“Rank correlation between per-judge Spearman and mean gate is 1-5: 0.71, 1-10: 1.00.”

May combine two claims.

The joint model improves on both pairwise datasets at both scales.

“The joint model improves on both pairwise datasets at both scales, which is why the paper treats pairwise supervision as a useful auxiliary signal for direct-score training.”

May combine two claims.

All eight folds recover the same selector: choose sparse when the leave-one-out gain is at least 0.14; otherwise choose fixed when ρ ≥ 0.5 and the fixed pool is already at least as strong as full JudgeMoE on validation; otherwise choose full.

“All eight folds recover the same selector: choose sparse when the leave-one-out gain is at least 0.14; otherwise choose fixed when ρ ≥ 0.5 and the fixed pool is already at least as strong as full JudgeMoE on validation; otherwise choose full.”

May combine two claims.

Both the SummEval 1-10 + BERTScore and HelpSteer2 1-10 + reward settings beat the best single judge with only 5% of training data.

“Both settings beat the best single judge with only 5% of training data, but the reward-model append saturates more slowly than the BERTScore expert.”

May combine two claims.

Log-opinion pooling rewards agreement across judges but is sensitive to any judge that assigns near-zero mass to the correct bucket.

“We use log-opinion pooling because multiplicative fusion rewards agreement across judges.The trade-off is increased sensitivity to any judge that assigns near-zero mass to the correct bucket: a single near-zero probability can push a bucket down unless that judge receives very small weight.”

Classified as probably not a substantial claim.

This is the setting in which entropy, confidence, and near-tie structure become useful.

“This is the setting in which entropy, confidence, and near-tie structure become useful.”

May combine two claims.

At T*, the expected-score predictor equals the Bayes-optimal point prediction E[y | x], and for any T ≠ T*, the conditional MSE increases by the squared bias.

“At this temperature the expected-score predictor ŷT * (x) = k s k q T * k (x) equals the Bayes-optimal point prediction E[y | x], and for any T ̸ = T * the conditional MSE increases by the squared bias: E (ŷ T -y) 2 | x = Var(y | x)+ ŷT (x)-ŷ T * (x) 2 .(22)”

• AssumptionThe authors state this as an assumption: Each judge produces a temperature-scaled version of the ground-truth posterior over scores, with judge-specific calibration temperature τ_j.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Judge aggregation adds risks: protocol choices can privilege some score distributions over others, and weak-judge contamination can distort pooled rankings.

“Judge aggregation adds its own risks: protocol choices can privilege some score distributions over others, and weak-judge contamination can distort pooled rankings.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Results depend on candidate quality, task match, and score resolution.

“This paper studies learned aggregation over a fixed pool of small local judges, so results depend on candidate quality, task match, and score resolution.”

May combine two claims.

Original split labels were preserved for each dataset; model, judge-pool, and aggregation choices were made using validation labels only, and test labels were used only for final reporting.

“To prevent leakage across train, validation, and test splits, we preserve the original split labels for each dataset and use exact example-ID intersections only when constructing aligned subset comparisons.Model, judge-pool, and aggregation choices are made using validation labels only; test labels are used only for final reporting.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Several cells still share datasets, score scales, and judge families.

“The resulting 16-cell analysis covers more evaluation conditions, while several cells still share datasets, score scales, and judge families.”

May combine two claims.

Log pooling rewards agreement, but it also gives every judge multiplicative influence; a weak judge that places too little mass on the correct bucket can veto the remaining judges.

“Log pooling rewards agreement, but it also gives every judge multiplicative influence.A weak judge that places too little mass on the correct bucket can therefore veto the remaining judges.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The compact 3-judge pool is selected once from the protocol-study screening and then held fixed.

“The compact 3judge pool is selected once from the protocol-study screening and then held fixed.”

May combine two claims.

Task-aligned experts can help when they add new evidence to the homogeneous pool and pass train-split diagnostics, while prompt-diverse expansions of the same base models hurt, and both translation-BERTScore and COMET degrade the WMT mixed pool.

“The focused heterogeneous follow-ups support the same conclusion.Task-aligned experts can help when they add new evidence to the homogeneous pool and pass train-split diagnostics, while promptdiverse expansions of the same base models hurt, and both translation-BERTScore and COMET degrade the WMT mixed pool.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The present benchmarks provide scalar human targets; annotator disagreement and multimodal human preferences remain outside the current scope.

“In addition, the present benchmarks provide scalar human targets; annotator disagreement and multimodal human preferences remain outside the current scope.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Pairwise preference learning provides relative supervision that can complement direct scores.

“Pairwise preference learning provides relative supervision that can complement direct scores (Bradley and Terry, 1952;Jeong et al., 2025).”

• cites Bradley and Terry, 1952 sourceNot checked — no full text of the source
• cites Jeong et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

None of the propositions imply that several small judges must beat a larger single judge.

“None of them imply that several small judges must beat a larger single judge.”

May combine two claims.

Sparse repair is a calibration-aware intervention, with no universal preference for smaller pools.

“Sparse repair is therefore a calibration-aware intervention, with no universal preference for smaller pools.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Propositions 1, 3, and 4 are standard background facts; Propositions 6-8 are specific to the JudgeMoE formulation and serve as structured diagnostic results with limited scope.

“Propositions 1, 3, and 4 are standard background facts stated in compact form; Propositions 6-8 are specific to the JudgeMoE formulation and serve as structured diagnostic results with limited scope.”

May combine two claims.

For scores {1, 2, 3}, the distributions (0.5, 0, 0.5) and (0, 1, 0) have the same expected score, 2, while encoding very different uncertainty; a score-only aggregator treats them the same, while a distributional aggregator can distinguish them.

“For scores {1, 2, 3}, the distributions (0.5, 0, 0.5) and (0, 1, 0) have the same expected score, 2, while encoding very different uncertainty: the first is a polarized near-tie between low and high scores, while the second is concentrated on the middle score.A score-only aggregator treats them the same; a distributional aggregator can distinguish them.”

Classified as probably not a substantial claim. May combine two claims.

The exact source and candidate fields differ across summarization, dialogue, reasoning, and translation, but the output contract is deliberately uniform so that scale and extraction effects can be isolated cleanly.

“The exact source and candidate fields differ across summarization, dialogue, reasoning, and translation, but the output contract is deliberately uniform so that scale and extraction effects can be isolated cleanly.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The six additional cells from HelpSteer2 validation, FLASK, and WildBench were evaluated after the judge pool, architecture, feature construction, and training procedure had been fixed.

“The six additional cells from HelpSteer2 validation, FLASK, and WildBench were evaluated after the judge pool, architecture, feature construction, and training procedure had been fixed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the WMT subset experiment, judge subsets are selected on WMT validation data and then retrained and evaluated on the corresponding test split.

“For the WMT subset experiment, judge subsets are selected on WMT validation data and then retrained and evaluated on the corresponding test split.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The reported ECE measures the calibration of the final scalar predictions against human scores and does not establish calibration of the underlying score-token probabilities.

“The reported ECE measures the calibration of the final scalar predictions against human scores and does not establish calibration of the underlying scoretoken probabilities.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Under the same calibration condition, the hard score h(P) = v arg max k P k is a Bayes action under exact-class 0-1 loss.

“Under the same calibration condition, the hard score h(P ) = v arg max k P k is a Bayes action under exact-class 0-1 loss.”

• AssumptionThe authors state this as an assumption: The judge distribution is calibrated with respect to the human score distribution, so that m(P) = µ(P).

May combine two claims.

TopicalChat favors fixed pooling, while WMT MQM@1-5 motivates sparse judge selection and retraining.

“Section 6 uses these failures diagnostically: TopicalChat favors fixed pooling, while WMT MQM@1-5 motivates sparse judge selection and retraining.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A held-out protocol study over 24 dataset-model cells selected the protocol on validation and evaluated it once on test.

“We also run a held-out protocol study over 24 datasetmodel cells in which the protocol is selected on validation and evaluated once on test.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All aggregation experiments are script-driven and use fixed split manifests, cached judge score distributions, and reproducible checkpoint configurations.

“All aggregation experiments are script-driven and use fixed split manifests, cached judge score distributions, and reproducible checkpoint configurations.”

May combine two claims.

The comparisons against covariance-aware log pooling and the best single judge are inconclusive, so validation data should choose among learned weighting, fixed pooling, and single-judge deployment.

“We therefore treat these comparisons as inconclusive and use validation data to choose among learned weighting, fixed pooling, and single-judge deployment.”

Classified as probably not a substantial claim.

Distributional information from a single LLM evaluator's judgment distribution can improve over greedy decoded judgments.

“Wang et al. (2025a) study inference from the judgment distribution of a single LLM evaluator and show that distributional information can improve over greedy decoded judgments.”

• cites Wang et al. (2025a)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

Coarse scores leave too little coherent ranking signal in the contaminated full pool.

“coarse scores leave too little coherent ranking signal in the contaminated full pool.”

Probably not a substantial claim: may not pass S3 (load-bearing).

After the judge pool, architecture, feature construction, and training procedure had been fixed, the same JudgeMoE configuration was evaluated on six additional cells.

“After the judge pool, architecture, feature construction, and training procedure had been fixed, we evaluated the same configuration on six additional cells from HelpSteer2 validation, FLASK, and WildBench.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Together, they support the validation-guided deployment heuristic: study calibration first, keep distributional evidence whenever the score range is fine enough to support it, and use sparse repair only when the pool diagnostics indicate contamination.

“Together, they support the validation-guided deployment heuristic in Table 4: study calibration first, keep distributional evidence whenever the score range is fine enough to support it, and use sparse repair only when the pool diagnostics indicate contamination.”

Probably not a substantial claim: may not pass S2 (checkable).

The scoring protocol is part of the measurement and belongs with the judge output.

“Individual judge quality remains important, yet the scoring protocol is part of the measurement and belongs with the judge output.”

May combine two claims.

JudgeMoE pools the full score distributions before scoring, retaining confidence, entropy, near-tie mass, and bucket-level agreement during aggregation.

“JudgeMoE pools the full score distributions before scoring, retaining confidence, entropy, near-tie mass, and bucket-level agreement during aggregation.”

May combine two claims.

When K = 2, σ(μ) = σ(P) and the gap is exactly zero; when K ≥ 3 and judge distributions are generically non-degenerate, σ(μ) ⊊ σ(P) and the gap is generically strict.

“(iii) When K = 2, σ(µ) = σ(P) and the gap is exactly zero.When K ≥ 3 and judge distributions are generically non-degenerate, σ(µ) ⊊ σ(P) and the gap is generically strict.”

May combine two claims.

Lower temperatures sharpen the fused distribution toward buckets supported by the weighted pool, while higher temperatures flatten it.

“Lower temperatures sharpen the fused distribution toward buckets supported by the weighted pool, while higher temperatures flatten it.”

May combine two claims.

Under equal-weight linear pooling in the same homogeneous setting, the ranking flips only when δ' - δ > m(p - p'), so single-judge pollution is less explosive than in log pooling.

“Under equal-weight linear pooling in the same homogeneous setting, the ranking flips only when δ ′ -δ > m(p -p ′ ). The weak judge enters additively in probability space, so single-judge pollution is less explosive than in log pooling.”

Classified as probably not a substantial claim. May combine two claims.

Prompts are task-specific but structurally aligned so the comparison stays focused on scale and aggregation.

“Prompts are task-specific but structurally aligned so the comparison stays focused on scale and aggregation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The compact 3-judge configuration was selected once from the protocol study and then held fixed for the aggregation benchmark.

“The compact 3-judge configuration retains Gemma-2-9B-IT, Gemma-3-12B-IT, and Qwen2.5-7B-Instruct;this pool is selected once from the protocol study and then held fixed for the aggregation benchmark.”

Classified as probably not a substantial claim. May combine two claims.

The temperature-scaled model is a first-order approximation; real judges exhibit instance-dependent miscalibration, so Proposition 6 characterizes the pool-average calibration condition without predicting one exact optimal T for every example.

“The temperature-scaled model above is a first-order approximation.Real judges exhibit instance-dependent miscalibration, so Proposition 6 characterizes the pool-average calibration condition without predicting one exact optimal T for every example.”

May combine two claims.

For uniform averaging of judge errors with zero mean, variance σ2, and pairwise correlation ρ, the error variance is Var(ε) = σ2(ρ + (1-ρ)/m), and as m→∞, Var(ε) → σ2ρ.

“For uniform averaging, Var(ε j ) = σ 2 , Corr(ε i , ε j ) = ρ for i ̸ = j, ε = 1 m m j=1 ε j , the error variance is Var(ε) = σ 2 ρ + 1 -ρ m .(13) Therefore, lim m→∞ Var(ε) = σ 2 ρ.(14)”

• AssumptionThe authors state this as an assumption: Judge score errors are zero mean with equal variance σ² and pairwise correlation ρ for i≠j.

Probably not a substantial claim: may not pass S3 (load-bearing).

This supports validating candidate experts before they enter the pool.

“This supports validating candidate experts before they enter the pool.”

Probably not a substantial claim: may not pass S3 (load-bearing).

ROSCOE is reported as a small-sample diagnostic cell and is interpreted cautiously because it contains only 151 examples.

“ROSCOE is reported as a small-sample diagnostic cell and is interpreted cautiously because it contains only 151 examples.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Under the same evaluation workload, relative compute is well approximated by active-parameter counts per token.

“For autoregressive inference, weight-matrix FLOPs scale approximately linearly with active parameters and processed tokens, so under the same evaluation workload the relative compute is well approximated by active-parameter counts per token.”

May combine two claims.

The 10-cell benchmark is sufficient for the uniform-log control but underpowered for the smaller gaps against covariance-aware pooling and the best single judge.

“The table explains why the 10-cell benchmark is sufficient for the uniform-log control but underpowered for the smaller gaps against covariance-aware pooling and the best single judge.”

Probably not a substantial claim: may not pass S4 (based).

For autoregressive inference, weight-matrix FLOPs scale approximately linearly with active parameters and processed tokens.

“For autoregressive inference, weight-matrix FLOPs scale approximately linearly with active parameters and processed tokens, so under the same evaluation workload the relative compute is well approximated by active-parameter counts per token.”

Classified as probably not a substantial claim. May combine two claims.

The protocol study checks whether scoring choices change the measurement; the aggregation experiments then freeze a shared protocol to isolate pooling from repeated scale search.

“The protocol study checks whether scoring choices change the measurement; the aggregation experiments then freeze a shared protocol to isolate pooling from repeated scale search.”

May combine two claims.

The large correlations on TopicalChat imply that most remaining error is shared across judges, which is consistent with Proposition 3 and helps explain why fixed covariance-aware pooling is already strong there.

“The large correlations on TopicalChat imply that most remaining error is shared across judges, which is consistent with Proposition 3 and helps explain why fixed covariance-aware pooling is already strong there.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The selected configuration is unchanged for the six additional evaluation cells.

“The selected configuration is unchanged for the six additional evaluation cells.”

May combine two claims.

JudgeMoE combines full score distributions with example-specific judge weighting.

“JudgeMoE combines full score distributions with example-specific judge weighting.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 6-judge architecture selected on the original 10 cells is reused for the HelpSteer2 validation, FLASK, WildBench, and heterogeneous evaluations.

“The 6-judge architecture selected on the original 10 cells is reused for the HelpSteer2 validation, FLASK, WildBench, and heterogeneous evaluations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All four settings in the 2×2 gate and pooling comparison use the same judge manifests, feature construction, pooling temperature, 30-epoch training schedule, and three random seeds.

“All four settings use the same judge manifests, feature construction, pooling temperature, 30-epoch training schedule, and three random seeds.”