CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

May combine two claims.

LLM judges are vulnerable to simple token manipulations and reasoning prompts, often interpreting them as signs of correctness.

“LLM judges have been shown to be vulnerable to simple token manipulations (e.g.,”

• cites Zhao et al., 2025 sourceSupported

May combine two claims.

Both WS and UWS are probabilistic graphical models that do not account for confounding factors (WS operates on discrete labels, while UWS handles continuous scores).

“Both WS and UWS are probabilistic graphical models that do not account for confounding factors (WS operates on discrete labels, while UWS handles continuous scores).”

• cites Bach et al., 2019 sourcePartly supported

  The Background section describes Snorkel DryBell’s weak-supervision generative model using discrete label votes, and the Sampling-Free Generative Model section specifies a conditionally independent model. Our copy does not discuss UWS or establish that both WS and UWS fail to account for confounding factors; the conditional-independence assumption alone does not support that broader claim. The indexed passages provide no page numbers. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites Shin et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

LLM-as-a-judge systems offer a scalable alternative to expert annotation by substantially reducing cost and latency.

“LLM-as-a-judge systems offer a scalable alternative to expert annotation by substantially reducing cost and latency (Zheng et al., 2023).”

• cites Zheng et al., 2023 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

We follow the suggested criteria from Huang et al. (2025), which are studied that can be translated into executable code.

“We follow the suggested criteria from Huang et al. (2025), which are studied that can be translated into executable code.”

• cites Huang et al., 2025 sourceSupported

May combine two claims.

On classification and preference benchmarks, CARE achieves the best accuracy on 5 of 6 datasets, with CARE-Tensor leading on three (PKU-BETTER, SHP, and Summarize), including a 13.4% relative improvement in accuracy on Summarize over the strongest baseline.

“On classification and preference benchmarks, CARE achieves the best accuracy on 5 of 6 datasets, with CARE-Tensor leading on three (PKU-BETTER, SHP, and Summarize), including a 13.4% relative improvement in accuracy on Summarize over the strongest baseline.”

May combine two claims.

Confounder 1 is positively correlated with verbosity (length, ρ ≈ 0.49) and negatively correlated with sentence length and SMOG complexity (ρ ≈ -0.39 and -0.38, respectively).

“Confounder 1 is positively correlated with verbosity (length, ρ ≈ 0.49) and negatively correlated with sentence length and SMOG complexity (ρ ≈ -0.39 and -0.38, respectively).”

May combine two claims.

Figure 6a shows that CARE-SVD and CARE-Tensor maintain high accuracy across the entire sweep, whereas baselines (AVG, MV, WS, UWS) degrade markedly as confounding strengthens.

“Figure 6a shows that CARE-SVD and CARE-Tensor maintain high accuracy across the entire sweep, whereas baselines (AVG, MV, WS, UWS) degrade markedly as confounding strengthens.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The marginal precision of J satisfies Θ := Σ^{-1}_JJ = K_JJ - K_JH K^{-1}_HH K_HJ = S - L.

“The marginal precision of J satisfies Θ := Σ -1 JJ = K JJ -K JH K -1 HH K HJ = S -L,”

May combine two claims.

Table 3 shows that CARE achieves strong robustness against beauty and authority biases, with the lowest MAE compared to baseline methods.

“Table 3 shows that CARE achieves strong robustness against beauty and authority biases, with the lowest MAE compared to baseline methods.”

May combine two claims.

Confounder 2 exhibits the opposite pattern: shorter but more complex responses, with slightly higher numerical density (ρ ≈ -0.46 for length, 0.28-0.29 for complexity, and 0.17 for numeracy).

“In contrast, Confounder 2 exhibits the opposite pattern: shorter but more complex responses, with slightly higher numerical density (ρ ≈ -0.46 for length, 0.28-0.29 for complexity, and 0.17 for numeracy).”

May combine two claims.

Averaged across scoring datasets, CARE-SVD yields a 17.37% relative improvement over AVG and a 12.75% improvement over MV.

“Averaged across scoring datasets, CARE-SVD yields a 17.37% relative improvement over AVG and a 12.75% improvement over MV.”

May combine two claims.

A CP decomposition of T recovers {μ^{(k)}_qc}, concatenating the groups yields {μ_qc}, and the weights {π_qc} follow from component norms.

“A CP decomposition of T recovers {µ (k) qc }, concatenating the groups yields {µ qc }, and the weights {π qc } follow from component norms.”

The quote only partly states this claim.

Proposition 4.1: Assume K_HH = diag(d_1, ..., d_h) with d_1 > ... > d_h > 0 and the columns of K_JH are orthogonal. Then the columns of K_JH are identifiable from L up to sign and permutation. Moreover, if K_JH is perturbed to K_JH = K_JH + E, letting δ_i denote the eigengap of L at eigenvector u_i, and û_i the eigenvector of the estimated low rank matrix L, then ||û_i - u_i||_2 ≲ ||K^{-1}_HH||_2

“Proposition 4.1 (Identifiability and stability of latentjudge directions).Assume K HH = diag(d 1 , . . ., d h ) with d 1 > • • • > d h > 0 and the columns of K JH are orthogonal.Then the columns of K JH (equivalently, the latent directions encoded by L) are identifiable from L up to sign and permutation.Moreover, if K JH is perturbed to KJH = K JH + E, letting δ i denote the eigengap of L at eigen”

• AssumptionThe authors state this as an assumption: K_HH = diag(d_1, ..., d_h) with d_1 > ... > d_h > 0 and the columns of K_JH are orthogonal.

May combine two claims.

Table 7 shows that the leading factor consistently yields the best MAE among all recovered factors, supporting our default choice of the top eigenvector as the quality direction in CARE-SVD.

“Table 7 shows that the leading factor consistently yields the best MAE among all recovered factors, supporting our default choice of the top eigenvector as the quality direction in CARE-SVD.”

May combine two claims.

Table 1 and Table 2 show that CARE consistently outperforms baseline methods across both scoring and classification/preference tasks.

“Table 1 and Table 2 show that CARE consistently outperforms baseline methods across both scoring and classification/preference tasks.”

May combine two claims.

In a second-order-insufficient regime where C ⊥ Q and second-order structure is confounder-dominated, baselines and CARE-SVD approach chance as confounding increases, while CARE-Tensor remains accurate by exploiting identifiable three-view third-order cross-moments.

“In a second-order-insufficient regime where C ⊥ Q and second-order structure is confounder-dominated, baselines and CARE-SVD approach chance as confounding increases, while CARE-Tensor remains accurate by exploiting identifiable three-view third-order cross-moments.”

• Also stated in: § E.9.2. SECOND-ORDER-INSUFFICIENT REGIME. · plain

  “In a second-order-insufficient regime where C ⊥ Q and second-order structure is confounder-dominated, baselines and CARE-SVD approach chance as confounding increases, while CARE-Tensor remains accurate by exploiting identifiable three-view third-order cross-moments.”

May combine two claims.

With the early-stopping criterion, CARE achieves the lowest error by integrating eight additional programmatic judges, reducing MAE by 15.2% compared to MV and by 11.1% compared to UWS.

“With the early-stopping criterion, CARE achieves the lowest error by integrating eight additional programmatic judges, reducing MAE by 15.2% compared to MV and by 11.1% compared to UWS.”

May combine two claims.

On datasets where higher-order factors are effectively negligible (e.g., FeedbackQA and Summarize), alternative factors are not meaningfully recoverable, and the benefit of CARE-SVD is correspondingly limited.

“We also observe that on datasets where higher-order factors are effectively negligible (e.g., FeedbackQA and Summarize), alternative factors are not meaningfully recoverable, and the benefit of CARE-SVD is correspondingly limited.”

May combine two claims.

Under the conditions of latent independence and orthogonality, the columns of K_JH (and hence the eigen-directions of L) are uniquely identified up to sign and permutation; moreover, the recovery is stable to small violations of orthogonality.

“Under these conditions, the columns of K JH (and hence the eigen-directions of L) are uniquely identified up to sign and permutation; moreover, the recovery is stable to small violations of orthogonality.”

• AssumptionThe authors state this as an assumption: Latent independence and orthogonality in the latent-observable link hold.

Classified as probably not a substantial claim. May combine two claims.

With (μ_qc, Σ, π_qc), Bayes' rule gives the posteriors α_qc(J) := Pr(Q=q, C=c | J) ∝ π_qc φ(J; μ_qc, Σ), E[Q | J] = Σ_{c∈{0,1}} α_{1c}(J).

“With (µ qc , Σ, π qc ), Bayes' rule gives the posteriors α qc (J) := Pr(Q = q, C = c | J) ∝ π qc ϕ(J; µ qc , Σ), E[Q | J] = c∈{0,1} α 1c (J).”

May combine two claims.

The sparse-plus-low-rank decomposition and the induced three-view partition are unchanged, and the third-order moment factorization holds with μ^{(k)}_qc := E[X_k | Q=q, C=c] ∈ [0,1]^{d_k} interpreted as per-judge success probabilities.

“The sparse-plus-low-rank decomposition and the induced three-view partition are unchanged, and the third-order moment factorization holds with µ (k) qc := E[X k | Q=q, C=c] ∈ [0, 1] d k interpreted as per-judge success probabilities.”

May combine two claims.

Figure 6b shows that as confounding strengthens, baselines and CARE-SVD degrade toward chance accuracy, while CARE-Tensor remains robust across the sweep.

“Figure 6b shows that as confounding strengthens, baselines and CARE-SVD degrade toward chance accuracy, while CARE-Tensor remains robust across the sweep.”

May combine two claims.

In a second-order-sufficient regime, both CARE-SVD and CARE-Tensor remain accurate across the confounder-strength sweep, whereas standard baselines (AVG, MV, WS, UWS) deteriorate sharply at high confounding.

“In a second-order-sufficient regime, both CARE-SVD and CARE-Tensor remain accurate across the confounder-strength sweep, whereas standard baselines (AVG, MV, WS, UWS) deteriorate sharply at high confounding.”

May combine two claims.

CARE-SVD achieved MAE of 27.629 on ASSET, 0.730 on FeedbackQA, 1.957 on Review-5K, 1.325 on Summarize, 0.623 on UltraFeedback, and 0.694 on Yelp.

“CARE-SVD 27.629 ± 0.156 0.730 ± 0.002 1.957 ± 0.018 1.325 ± 0.004 0.623 ± 0.006 0.694 ± 0.004”

May combine two claims.

In the second-order-sufficient regime, the dominant shared second-order structure remains aligned with the true-quality factor Q, so both CARE-SVD and CARE-Tensor remain robust as confounding increases.

“The first is a second-order-sufficient regime, where the dominant shared second-order structure remains aligned with the true-quality factor Q, so both CARE-SVD and CARE-Tensor remain robust as confounding increases.”

May combine two claims.

CARE-SVD achieved MAE of 0.375 under Beauty Bias and 0.233 under Authority Bias.

“CARE-SVD | 0.375 | 0.233 |”

May combine two claims.

The proposed heuristic improves estimation in the non-orthogonal setting but slightly degrades performance in the orthogonal setting where true and confounding components are disjoint.

“The heuristic improves estimation in the non-orthogonal setting, but slightly degrades performance in the orthogonal setting where true and confounding components are disjoint.”

May combine two claims.

In the second-order-insufficient regime, pairwise structure is dominated by a confounder C (with C ⊥ Q), causing spectral recovery to degrade toward chance, while multi-view third-order moments remain identifiable.

“The second is a second-order-insufficient regime, where pairwise structure is dominated by a confounder C (with C ⊥ Q), causing spectral recovery to degrade toward chance, while multi-view third-order moments remain identifiable.”

Classified as probably not a substantial claim.

LLM-as-a-judge ensembles are a common/standard paradigm for scalable evaluation.

“LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation”

• Also stated in: § 1. Introduction · plain

  “Consequently, a common evaluation paradigm ensembles multiple LLM judges to produce consensus scores (Hu et al., 2024).”

• cites Hu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Standard aggregation rules like majority vote or averaging provide little gain or even amplify systematic mistakes.

“causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes.”

• Also stated in: § 1. Introduction · hedged

  “As a result, heuristic aggregation strategies-such as majority vote, averaging, or ad-hoc reweighting-can yield diminishing returns and may even amplify shared mistakes.”

Classified as probably not a substantial claim.

Recent work deploys multiple judges and aggregates their opinions-typically via majority vote or average pooling-to improve robustness.

“To address this, recent work (Verga et al., 2024) deploys multiple judges and aggregates their opinions-typically via majority vote or average poolingto improve robustness.”

• cites Verga et al., 2024 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

When key influences are unobserved-most notably the true underlying quality of a model output-augmenting an MRF with latent nodes enables recovery of this hidden structure from noisy observations.

“When key influences are unobserved-most notably the true underlying quality of a model output-augmenting an MRF with latent nodes enables recovery of this hidden structure from noisy observations.”

May combine two claims.

Similarly, the recovered mixing weights satisfy the stability bound. By equation 11 and equation 8, with probability at least 1-ε/2, max_{r∈[4]} |ŵ_{τ(r)} - w_r| ≤ C_π ∥E∥_op ≤ C_π C σ_max^3 sqrt(p log(p/ε)/n). Under the normalization implicit in Assumption D.8(A4), the above translates to the stated O(sqrt(p log(p/ε)/n)) rate for max_{q,c} |π_qc - π̂_qc| (after absorbing model-independent fixed 1

“Similarly, the recovered mixing weights satisfy the stability bound.By equation 11 and equation 8, with probability at least 1 -ε/2, max r∈[4] | ŵτ(r) -w r | ≤ C π ∥E∥ op ≤ C π C σ 3 max p log(p/ε) n . Under the normalization implicit in Assumption D.8(A4) (which fixes the scaling ambiguity in CP factors), the above translates to the stated O p log(p/ε)/n rate for max q,c |π qc -π qc | (after”

• AssumptionThe authors state this as an assumption: The normalization implicit in Assumption D.8(A4) fixes the scaling ambiguity in CP factors.

Probably not a substantial claim: may not pass S4 (based).

C3: even if we recover the model parameters, we must be able to distinguish between Q and the confounders C to identify the model.

“Third, C3: even if we recover the model parameters, we must be able to distinguish between Q and the confounders C to identify the model.”

May combine two claims.

CARE effectively mitigates superficial stylistic changes, such as cosmetic formatting or fake citations, while maintaining consistent aggregated evaluations.

“This indicates that CARE effectively mitigates superficial stylistic changes, such as cosmetic formatting or fake citations, while maintaining consistent aggregated evaluations.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Together, these results demonstrate CARE's broader applicability for analyzing and interpreting latent confounding factors across modeling assumptions.

“Together, these results demonstrate CARE's broader applicability for analyzing and interpreting latent confounding factors across modeling assumptions.”

May combine two claims.

This choice (leading eigenvector as quality axis) is objective and works well when the dominant shared variation reflects quality.

“This choice is objective and works well when the dominant shared variation reflects quality.”

• AssumptionThe authors state this as an assumption: Quality induces the strongest shared variation across judges, whereas confounders are typically weaker or concentrated on a subset.

Probably not a substantial claim: may not pass S4 (based).

In practice, even the weak assumption on judge quality can be removed by employing simple heuristics to identify the true-quality factor among the latent factors.

“In practice, even this assumption can be removed by employing simple heuristics to identify the true-quality factor among the latent factors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

These associations are used for qualitative interpretation rather than as uniquely identifiable causal explanations.

“These associations are used for qualitative interpretation rather than as uniquely identifiable causal explanations.”

May combine two claims.

Setting the right-hand side to a target accuracy α ∈ (0,1) yields n ≥ 8 C_1^2 ξ(T)^2 / δ^2 * η/α^2. Equivalently, to ensure failure probability at most ζ ∈ (0,1), set η = log(2/ζ).

“Set the right-hand side to a target accuracy α ∈ (0, 1), i.e., 2 3/2 C 1 ξ(T ) δ η n ≤ α. Solving for n yields n ≥ 8C 2 1 ξ(T ) 2 δ 2 • η α 2 . Equivalently, to ensure failure probability at most ζ ∈ (0, 1), set η = log(2/ζ).”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

While cost-effective, programmatic judges' deterministic nature can introduce systematic biases, resulting in noisy evaluation signals.

“While cost-effective, their deterministic nature can introduce systematic biases, resulting in noisy evaluation signals.”

May combine two claims.

Aggregating the recovered factors into full mean vectors gives ∥μ_{τ(r)} - μ_r∥_2^2 = ∥â_{τ(r)} - a_r∥_2^2 + ∥b̂_{τ(r)} - b_r∥_2^2 + ∥ĉ_{τ(r)} - c_r∥_2^2 ≤ 3 C_dec^2 ∥E∥_op^2 / δ^2. Taking square roots and maximizing over r gives max_{r∈[4]} ∥μ_{τ(r)} - μ_r∥_2 ≤ √3 C_dec ∥E∥_op / δ.

“We then aggregate these factors into the full mean vectors µ r := concat(a r , b r , c r ) ∈ R p and μτ(r) := concat(â τ (r) , bτ(r) , ĉτ(r) ).After aligning signs as in equation 10, ∥μ τ (r) -µ r ∥ 2 2 = ∥â τ (r) -a r ∥ 2 2 + ∥ bτ(r) -b r ∥ 2 2 + ∥ĉ τ (r) -c r ∥ 2 2 ≤ 3C 2 dec ∥E∥ 2 op δ 2 . Taking square roots and maximizing over r gives max r∈[4] ∥μ τ (r) -µ r ∥ 2 ≤ √ 3 C dec ∥E∥ op δ .(12)”

May combine two claims.

When all variables are jointly Gaussian, odd central moments vanish, so third-order tensors carry no information.

“When all variables are jointly Gaussian, odd central moments vanish, so third-order tensors carry no information.”

May combine two claims.

By explicitly modeling shared confounders during aggregation, CARE offers a principled alternative to heuristic ensembling and substantially improves the reliability of LLM-as-a-judge evaluation.

“By explicitly modeling shared confounders during aggregation, CARE offers a principled alternative to heuristic ensembling and substantially improves the reliability of LLM-as-a-judge evaluation.”

May combine two claims.

Existing approaches remain largely heuristic: they target specific failure modes or rely on assumptions that may be violated when judges share latent confounders.

“Additionally, existing approaches remain largely heuristic: they target specific failure modes or rely on assumptions that may be violated when judges share latent confounders.”

Probably not a substantial claim: may not pass S4 (based).

C2: we cannot assume any particular interaction in the graph.

“Second, C2: we cannot assume any particular interaction in the graph.”

May combine two claims.

Beyond aggregation, CARE's latent-variable structure enables diagnostic analysis: identifying which non-quality latent factors are present and which response attributes they capture.

“Beyond aggregation, CARE's latent-variable structure enables diagnostic analysis: identifying which non-quality latent factors are present and which response attributes they capture.”

Probably not a substantial claim: may not pass S4 (based).

C1: we never observe the latents in H-neither ground truth nor confounders.

“First, C1: we never observe the latents in H-neither ground truth nor confounders.”

Probably not a substantial claim: may not pass S4 (based).

In the binary-Gaussian mixture scenario, an additional step resolves ambiguity between latent states (Q = 0 vs. Q = 1).

“In the binary-Gaussian mixture scenario, an additional step resolves ambiguity between latent states (Q = 0 vs. Q = 1).”

Probably not a substantial claim: may not pass S4 (based).

Doing so will require additional information that can come from modeling assumptions, the use of ground-truth samples, or heuristics.

“Doing so will require additional information that can come from modeling assumptions, the use of ground-truth samples, or heuristics.”

May combine two claims.

For the final posterior, adopting a Gaussian prior α_qc(J) ∝ π_qc φ(J; μ_qc, Σ) preserves residual within-view dependence among judges; this leaves the identifiable tensor backbone intact and performs well in practice.

“For the final posterior we adopt a Gaussian prior α qc (J) ∝ π qc ϕ(J; µ qc , Σ) to preserve residual within-view dependence among judges; this leaves the identifiable tensor backbone intact and performs well in practice.”

May combine two claims.

The theoretical results characterize regimes in which heuristic aggregation methods are fundamentally misspecified and demonstrate when confounder-aware aggregation can reliably recover latent quality without access to ground-truth labels.

“These results characterize regimes in which heuristic aggregation methods are fundamentally misspecified and demonstrate when confounder-aware aggregation can reliably recover latent quality without access to ground-truth labels.”

May combine two claims.

Finally, ensuring the decomposition operates in the perturbative regime equation 9, the sufficient condition ∥E∥_op ≤ c_0 δ holds whenever C σ_max^3 sqrt(p log(p/ε)/n) ≤ c_0 δ, i.e., n ≥ C_2 c_0^2 σ_max^6 δ^2 p log(p/ε). Incorporating the additional detectability requirement for the smallest component (Assumption D.8(A2) with π_min > 0) yields the stated sufficient sample size condition n ≥ C_1 σ

“Finally, it remains to ensure the decomposition operates in the perturbative regime equation 9.By equation 8, the sufficient condition ∥E∥ op ≤ c 0 δ holds whenever C σ 3 max p log(p/ε) n ≤ c 0 δ, i.e., n ≥ C 2 c 2 0 σ 6 max δ 2 p log p/ε . Incorporating the additional detectability requirement for the smallest component (Assumption D.8(A2) with π min > 0) yields the stated sufficient sample size”

• AssumptionThe authors state this as an assumption: Assumption D.8(A2) with π_min > 0 (detectability requirement for the smallest component).

May combine two claims.

Overall, these results highlight CARE's ability to reduce compounding biases and deliver more reliable aggregation across diverse evaluation settings.

“Overall, these results highlight CARE's ability to reduce compounding biases and deliver more reliable aggregation across diverse evaluation settings.”

May combine two claims.

Combining equation 3 into equation 4 yields ∥û_i - u_i∥_2 ≤ 2^{3/2} C_1 δ ξ(T) ε/n ∀ i ∈ [h], and taking the maximum over i proves max_{i≤h} ∥û_i - u_i∥_2 = O( sqrt(ε/n) ξ(T)/δ ).

“Insert equation 3 into equation 4: ∥ ûi -u i ∥ 2 ≤ 2 3/2 C 1 δ ξ(T ) ϵ n ∀ i ∈ [h], and take the maximum over i.This proves the advertised high-probability bound max i≤h ∥ ûi -u i ∥ 2 = O √ ϵ/n ξ(T ) δ .”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The 'best factor' uses the test set for selection and is therefore optimistic; the goal is diagnostic rather than a deployable selection rule.

“The”

Probably not a substantial claim: may not pass S3 (load-bearing).

Programmatic judges, synthesized by LLMs, distill the evaluation logic of LLMs into interpretable and cost-effective program code.

“Programmatic judges, synthesized by LLMs, distill the evaluation logic of LLMs into interpretable and cost-effective program code (Huang et al., 2025;2024).”

• cites Huang et al., 2025;2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Respecting judge dependencies during view formation is critical for accurate tensor recovery, validating graph-aware view formation as a key component of CARE-Tensor.

“This shows that respecting judge dependencies during view formation is critical for accurate tensor recovery, validating graph-aware view formation as a key component of CARE-Tensor.”