SkillAggregation: Reference-free LLM-Dependent Aggregation

May combine two claims.

LLM-as-a-judge has limitations such as self-preference bias, verbosity bias, and sensitivity to prompt phrasing.

“Despite its advantages, LLM-as-a-judge has limitations such as self-preference bias, where an LLM tends to favour its own responses; verbosity bias, where some LLMs may prefer longer, more detailed responses (Zheng et al., 2023;Stureborg et al., 2024); or sensitivity to prompt phrasing (Verga et al., 2024).”

• cites Zheng et al., 2023 sourcePartly supported

  Section 3.3 identifies verbosity bias and reports that LLM judges may favor longer responses. It also observes that GPT-4 and Claude-v1 favored their own answers, but says the study could not determine whether models exhibit a self-enhancement bias; the claim presents that tendency more firmly than the paper does. The citing sentence attributes sensitivity to prompt phrasing to Verga et al., not Zheng et al. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites Stureborg et al., 2024 sourceSupported
• cites Verga et al., 2024 sourceSupported

Classified as probably not a substantial claim.

Human evaluation has long been considered the gold standard for evaluating the quality of natural language generation systems.

“Human evaluation has long been considered the gold standard for evaluating the quality of natural language generation (NLG) systems (Belz and Reiter, 2006;Lai and Tetreault, 2018;Fabbri et al., 2021).”

• cites Belz and Reiter, 2006Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lai and Tetreault, 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Fabbri et al., 2021 sourcePartly supported

  In Related Work (p. 2), SummEval says it assesses automatic summarization metrics against expert human annotators; Section 5.2 (p. 9) likewise compares metric scores with human judgments. This supports treating human evaluation as a benchmark for summarization, but the paper does not establish the broader claim about NLG systems generally or that human evaluation has long been considered the gold standard. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

May combine two claims.

Existing aggregation methods in NLP either assign equal weight to all LLM judgments or are designed for specific tasks such as hallucination detection.

“Existing methods in NLP either assign equal weight to all LLM judgments or are designed for specific tasks such as hallucination detection.”

• Also stated in: § 1 Introduction · plain

  “Furthermore, existing aggregation approaches in NLP weigh judges equally (Verga et al., 2024;Badshah and Sajjad, 2024) or are tailored to specific tasks (Sun et al., 2024;Li et al., 2024a), resulting in the following limitations.”

• cites Verga et al., 2024 sourcePartly supported

  Section 3.1 describes the paper’s own panel aggregating judges’ scores by max voting or average pooling, consistent with treating their judgments equally. Section 2.2 notes that prior multi-judge work was limited to pair-wise evaluations, but the paper does not establish that existing NLP methods generally fall into the two stated categories or identify hallucination detection as one of them. Our copy provides no page numbers for these passages. (LLM, conf=0.74)

  Experimental — accuracy not yet measured
• cites Sun et al., 2024 sourcePartly supported

  The Introduction’s Related Work discussion says SelfCheckGPT was designed for hallucination detection, supporting the task-specific part of the claim. Section 3.3 proposes confidence-based weighting because evidence models vary in quality, but the passages do not establish that existing NLP methods assign equal weight to all LLM judgments. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Li et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Verga et al. (2024) introduced PoLL, a multi-judge framework where each judge is assigned equal weight.

“Building on the LLM judge, Verga et al. (2024) introduced PoLL, a multi-judge framework where each judge is assigned equal weight.”

• cites Verga et al., 2024 sourceSupported

May combine two claims.

Commonly adopted approaches such as averaging and majority voting treat judges equally and neglect differences in quality and reliability of individual judges.

“However, both methods treat judges equally and neglect the differences in the quality and reliability of individual judges.”

• cites Sun et al., 2024 sourcePartly supported

  Section 3.2, “Confidence-based Weighting for Evidence Models,” says evidence quality varies with models’ propensity to hallucinate and proposes weighting scores by model uncertainty. This supports the concern about treating models equally, but our copy does not specifically discuss averaging and majority voting as the two methods in the claim. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Verga et al., 2024 sourcePartly supported

  Section 3.1 describes average pooling across judges, and Section 2.2 describes pooling their independent scores without specifying judge-specific weights. The paper also shows differences in judge performance (Section 4.1), but it uses max voting—not majority voting—for its binary evaluations and does not explicitly make the claimed criticism of both methods. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

The Crowdlayer method performs inference by directly selecting the class with the highest probability from the bottleneck output s_n without utilizing worker predictions.

“The Crowdlayer method performs inference by directly selecting the class with the highest probability from the bottleneck output s n without utilizing worker predictions.”

• cites Rodrigues and Pereira, 2018 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Crowdlayer predicts a context-dependent distribution P(c|X) using a neural context encoder.

“Crowdlayer predicts a context-dependent distribution P (c n |X n ) using a neural context encoder.”

• cites Rodrigues and Pereira, 2018 sourcePartly supported

  The Crowd layer section (p. 3) says a neural network’s softmax output gives the probability that an input belongs to class c. It does not describe c as a context variable or identify a neural context encoder; the crowd layer instead maps those class probabilities to annotator-specific label outputs. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

CrossCheckGPT designed a hallucination evaluation method based on cross-model consistency using information from a group of LLMs.

“Recent efforts, such as CrossCheckGPT (Sun et al., 2024), designed a hallucination evaluation method based on a crossmodel consistency idea that uses information generated from a group of LLMs.”

• cites Sun et al., 2024 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

PoLL's simple aggregation led to improved performance.

“This simple aggregation led to improved performance.”

• cites Verga et al., 2024 sourceSupported

Classified as probably not a substantial claim.

LLM-as-a-judge has shown strong correlation with human judgments across tasks.

“This method has shown strong correla-tion with human judgments across tasks.”

• cites Zheng et al., 2023 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

FEWL uses answers generated from a group of LLMs along with an answer weighing mechanism for hallucination evaluation.

“Similarly, FEWL (Wei et al., 2024) uses answers generated from a group of LLMs along with an answer weighing mechanism for hallucination evaluation.”

• cites Wei et al., 2024 sourceSupported

May combine two claims.

Without regularization, over-confident skill-estimate vectors are learned, dominating the posterior distribution and yielding a sub-optimal aggregation effect.

“Without regularization, over-confident skill-estimate vectors are learned, dominating the posterior distribution and yielding a sub-optimal aggregation effect.”

May combine two claims.

While SkillAggregation still outperforms Crowdlayer, the performance of both context-dependent aggregation methods with only 1,000 samples is much noisier and does not necessarily outperform majority voting.

“As shown in Fig. 6, while SkillAggregation still outperforms Crowdlayer, the performance of both context-dependent aggregation methods with only 1,000 samples is much noisier and does not necessarily outperform majority voting.”

May combine two claims.

SkillAggregation shows larger improvements when most judges perform above average and when there are a few weak models that deteriorate the majority voting performance, such as when majority voting accuracy is around 65%-68%.

“On the contrary, SkillAggregation shows larger improvements when most judges perform above average and when there are a few weak models that deteriorate the majority voting performance, such as when majority voting accuracy is around 65%-68% in Fig. 4.”

May combine two claims.

While differential weighting aggregation methods consistently outperform majority voting and SkillAggregation, achieving the best performance, the gains are smaller.

“While differential weighting aggregation methods consistently outperform majority voting and SkillAggregation, achieving the best performance, the gains are smaller.”

Classified as probably not a substantial claim. May combine two claims.

Gemma-2B achieves an accuracy of 85.59% on TruthfulQA in contrast to 84.57% with GPT-2.

“This is also supported by the fact that Gemma-2B achieves an accuracy of 85.59% on TruthfulQA in contrast to 84.57% with GPT-2.”

May combine two claims.

By simply training on majority voting, a 1.5% gain is already achieved on HaluEval compared to majority voting, however, on other datasets training on majority voting leads to worse performance.

“By simply training on majority voting, a 1.5% gain is already achieved on HaluEval compared to majority voting, however, on other datasets training on majority voting leads to worse performance.”

May combine two claims.

The best performance across all neural methods is obtained using SkillAggregation-X, which achieved 4.9%, 1.3%, and 0.5% absolute accuracy improvements on HaluEval-Dialogue, TruthfulQA, and Chatbot Arena, respectively.

“The best performance across all neural methods is obtained using SkillAggregation-X, which achieved 4.9%, 1.3%, and 0.5% absolute accuracy improvements on HaluEval-Dialogue, TruthfulQA, and Chatbot Arena, respectively.”

May combine two claims.

Similar plots for TruthfulQA and Chatbot Arena have PCCs of 89.9% and 69.5%, respectively.

“Similar plots for TruthfulQA and Chatbot Arena are shown in Appendix E with PCCs of 89.9% and 69.5%, respectively.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The LLM judges considered can sometimes make correlated errors when predicting the ground truth.

“Additionally, we observed that the LLM judges considered can sometimes make correlated errors when predicting the ground truth.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLM-as-a-judge offers a more cost-effective alternative to traditional human evaluation.

“This zero-shot LLM-based method, also termed LLM-as-ajudge, offers a more cost-effective alternative to traditional human evaluation (Li et al., 2024b).”

• cites Li et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

SkillAggregation does not prompt the judges for any information besides the estimates, nor does it need each judge to assess all others.

“Unlike prior work, we do not prompt the judges for any information besides the estimates, nor does our algorithm need each judge to assess all others.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Most existing aggregation methods focus on weighting worker contributions without accounting for contextual information, though other methods introduce context-aware mechanisms.

“Most existing aggregation methods focus on weighting worker contributions without accounting for contextual information (Zhang et al., 2016;Zheng et al., 2017;Zhang, 2022)), though other methods introduce context-aware mechanisms (Jin et al., 2020;Zhang, 2022).”

• cites Zhang et al., 2016 sourceNot checked — no full text of the source
• cites Zheng et al., 2017 sourceNot checked — no full text of the source
• cites Jin et al., 2020 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

SkillAggregation adopts a similar route to Crowdlayer with critical modifications that improve performance for aggregating LLM judgments.

“SkillAggregation adopts a similar route with critical modifications that improve performance for aggregating LLM judgments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Aggregation methods have been widely studied in crowdsourcing and ensemble learning.

“Aggregation methods have been widely studied in fields such as crowdsourcing and ensemble learning, where strategies are developed to combine predictions (also referred to as worker predictions) to enhance decision quality during training and inference.”

• cites Zhang et al., 2016 sourceNot checked — no full text of the source
• cites Zheng et al., 2017 sourceNot checked — no full text of the source
• cites Zhang, 2022 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based).

The context encoder (GPT-2) already has some inherent ability to perform the HaluEval task.

“The improvements on HaluEval are considerably larger compared to the other two tasks, likely because the context encoder (GPT-2) already has some inherent ability to perform this task.”

May combine two claims.

Gemma-2B does not necessarily give better context representations than GPT-2 for human preference prediction, hence yielding a similar performance.

“While Gemma-2B is larger than GPT-2, it does not necessarily give better context representations than GPT-2 for human preference prediction, hence yielding a similar performance.”

The quote only partly states this claim.

The smaller gains are particularly noticeable in Chatbot Arena, where the impact of noisy labels is more pronounced.

“This is particularly noticeable in Chatbot Arena, where the impact of noisy labels is more pronounced.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Dawid-Skene estimates P(b|c) under the assumption that contexts are not informative and P(c=1)=0.5.

“This matrix is learned using the EM algorithm as shown in Algorithm 3 in Appendix C, and it estimates P (b n,k |c n ) under the assumption that contexts are not informative and P (c n = 1) = 0.5.”

• cites Dawid and Skene, 1979Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

SkillAggregation can be applied to any problem where the LLM estimates are binary or probabilistic.

“This method is more general as it can be applied to any problem where the LLM estimates are binary or probabilistic.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Dawid-Skene estimates the confusion matrix associated with each judge and weights workers based on the confusion matrices.

“Instead of treating each judge equally, it estimates the confusion matrix associated with each judge and weights the workers based on the confusion matrices.”

• cites Dawid and Skene, 1979Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The context is more informative in HaluEval, resulting in higher improvements with context-dependent methods.

“This suggests that the context is more informative, resulting in higher improvements with context-dependent methods.”

May combine two claims.

Using a single large LLM judge amplifies biases and can require high computational resources that make local evaluations impractical.

“Using a single large LLM judge not only amplifies biases but can also require high computational resources that make local evaluations impractical.”

Classified as probably not a substantial claim.

LLM judgments are available for each context during inference.

“Specific to our problem setup, the LLM judgments are available for each context during inference.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Skill-estimate vectors can be task-specific or context-specific.

“As a design choice, skill-estimate vectors can be task-specific, using the same set of p (n,k) for all data samples n in one task, or context-specific where a trainable mapping from the context to p (n,k) is done with a linear layer followed by a Sigmoid activation function to ensure the range of values.”

May combine two claims.

Compared to TruthfulQA, the context in Chatbot Arena is more challenging to encode, and human preference evaluation is inherently noisier, leading to the smallest gains across the three datasets.

“Compared to TruthfulQA, the context in Chatbot Arena is more challenging to encode, and human preference evaluation is inherently noisier, leading to the smallest gains across the three datasets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For LLMs where probabilities cannot be obtained, binary predictions can be used, which is a special case covered by the problem setup.

“Note that for LLMs where probabilities cannot be obtained, binary predictions where y n,k ∈ {0, 1} can be used, which is a special case covered by this problem setup.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Training on the development set does not yield good performance across the test set.

“This set is small enough to have negligible influence on the overall dataset, and training on it does not yield good performance across the test set.”

Classified as probably not a substantial claim.

The development set of 250 samples with reference labels per task is small enough to have negligible influence on the overall dataset.

“This set is small enough to have negligible influence on the overall dataset, and training on it does not yield good performance across the test set.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

All aggregation methods are reference-free, and we learn directly from LLM judgments using the entire test set with no reference labels.

“All aggregation methods are reference-free, and we learn directly from LLM judgments using the entire test set with no reference labels.”

Classified as probably not a substantial claim.

Li et al. (2024a) impose the constraint that each judge must evaluate all others.

“Li et al. (2024a) impose the constraint that each judge must evaluate all others.”

• cites Li et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The noisier performance with 1,000 samples is likely because the context encoder cannot provide reasonable estimates of the ground truth, and the skill-estimate vectors are not sufficiently accurate.

“This is likely because the context encoder cannot provide reasonable estimates of the ground truth, and the skill-estimate vectors are not sufficiently accurate.”

Probably not a substantial claim: may not pass S3 (load-bearing).

PRD adopts multiple LLMs for pairwise comparison ranking.

“PRD (Li et al., 2024a) adopts multiple LLMs for pairwise comparison ranking.”

• cites Li et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Traditional evaluation methods for NLP tasks, such as BLEU and ROUGE, have long been task-specific and reliant on manually designed metrics.

“Traditional evaluation methods for NLP tasks, such as BLEU for machine translation (Papineni et al., 2002) and ROUGE for summarization (Lin, 2004), have long been taskspecific and reliant on manually designed metrics.”

• cites Papineni et al., 2002 sourceNot checked — no full text of the source
• cites Lin, 2004Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

DawidSkene and Crowdlayer are two representative common aggregation algorithms that are first applied to combine LLM judgments in this paper.

“DawidSkene and Crowdlayer are two representative common aggregation algorithms that are first applied to combine LLM judgments in this paper.”

May combine two claims.

The estimation of posterior relies on the quality of the skill-estimate vectors; hence, it benefits from the proposed regularization term that prevents the impact of over-confident LLMs.

“This estimation relies on the quality of the skillestimate vectors; hence, it also benefits from the proposed regularization term that prevents the impact of over-confident LLMs.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Choosing the best class based on the estimation of the posterior distribution P(c|X,y) is useful when the context encoder is a relatively small LM and is not powerful enough to make an accurate prediction for the ground truth, and the LLM judge predictions are, in general, better than the bottleneck outputs.

“This is useful when the context encoder is a relatively small LM and is not powerful enough to make an accurate prediction for the ground truth, and the LLM judge predictions are, in general, better than the bottleneck outputs.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

SkillAggregation-X can be useful when the performance of LLM judges changes with subtasks or topics inside a task, allowing more flexibility.

“This is referred to as SkillAggregation-X which can be useful when the performance of LLM judges changes with subtasks or topics inside a task, allowing more flexibility.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study's evaluation focused primarily on classification tasks.

“Although our results show that SkillAggregation outperforms the baselines, our focus has primarily been on classification tasks, leaving room for further work on general LLM evaluation methods, such as those involving regression tasks.”