Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge

Classified as probably not a substantial claim.

Self-enhancement bias arises when LLMs favor their outputs as both generators and judges.

“▷ Self-enhancement bias: This bias arises when LLMs favor their outputs as both generators and judges.”

• cites Zheng et al. (2024)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Meng et al. (2024) sourceCould not verify

  Table 2 lists GPT-4 Turbo as both the baseline model and judge for AlpacaEval 2, but it does not say the judge favors its own outputs. Searches for “self-enhancement bias,” “self-preference bias,” and judges favoring their own responses found no passage addressing that claim in our copy. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

May combine two claims.

GPT-4-Turbo exhibits inconsistency when judging emotional responses, whereas ChatGPT demonstrates more stable performance.

“For example, despite its advanced capabilities (Zheng et al., 2023), GPT-4-Turbo exhibits inconsistency when judging emotional responses, whereas ChatGPT demonstrates more stable performance.”

• cites Zheng et al., 2023 sourcePartly supported

  The citation is attached to “advanced capabilities,” not to the subsequent comparison of emotional-response judgments. Section 4.3 reports that GPT-4 performs significantly better than other models on MT-bench, supporting the general point, but the cited passage does not establish that result for GPT-4-Turbo specifically. The paper does not support the sentence’s GPT-4-Turbo-versus-ChatGPT comparison. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

LLMs with stronger capabilities are preferred as judges because weaker LLMs may exhibit greater randomness in their judgments, undermining reliability.

“Based on the recent study (Gao et al., 2024;Liu et al., 2023a;Li et al., 2024b), LLMs with stronger capabilities are prefered to be used as judges, because weaker LLMs may exhibit greater randomness in their judgments, which can undermine the reliability of judging results.”

• cites Gao et al., 2024 sourcePartly supported

  Section E.2 reports that the GPT-4 judge agreed with human annotators on 91.43% of the evaluated pairs, supporting the use of a capable LLM as a judge in this study. But the paper does not compare the randomness of stronger and weaker judges or say that weaker judges undermine reliability; Section 5.1 instead says it averaged three judgments per response. (LLM, conf=0.80)

  Experimental — accuracy not yet measured
• cites Liu et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Li et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The imbalance in pre-training data can lead to imbalances during model training, resulting in biases against certain demographic groups, such as different genders, ages, and various languages.

“The imbalance in pre-training data can lead to imbalances during model training (Liu et al., 2024), resulting in biases against certain demographic groups, such as different genders (Wan et al., 2023), ages (Macnicol, 2006), and various languages (Jiao et al., 2023;Bang et al., 2023).”

• cites Liu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wan et al., 2023 sourceSupported
• cites Macnicol, 2006 sourceNot checked — no full text of the source

May combine two claims.

Significant biases persist in certain specific tasks in LLM-as-a-Judge, despite advanced models achieving commendable overall performance.

“Our experiments cover multiple popular language models, and the results indicate that while advanced models have achieved commendable overall performance, significant biases persist in certain specific tasks.”

May combine two claims.

The other three emotions show greater effects than cheerful, with fear having the most significant impact.

“The other three emotions show greater effects, with fear having the most significant impact.”

May combine two claims.

The LLM consistently scored refined answers higher, despite providing similar justifications for different instances and never explicitly mentioning refinement as a factor in its decision-making process.

“Here, the LLM consistently scored refined answers higher, despite providing similar justifications for different instances and never explicitly mentioning refinement as a factor in its decision-making process.”

May combine two claims.

GPT-4o remained consistent, while Claude-3.5 was more influenced by popular opinion.

“GPT-4o remained consistent, while Claude-3.5 was more influenced by popular opinion.”

May combine two claims.

LLMs tend to prefer content without emotional elements and generally resist emotionally colored responses.

“LLMs tend to prefer content without emotional elements.”

• Also stated in: § 4.2 ANALYSIS OF EXPLORATORY EXPERIMENTS · hedged

  “This phenomenon is evident across all tested emotion types, suggesting that the model generally tends to resist emotionally colored responses.”

May combine two claims.

Position bias becomes more pronounced as the number of answers increases, particularly when evaluating three or four options, resulting in a decreased robustness rate, with most models scoring below 0.5.

“This bias becomes more pronounced as the number of answers increases, particularly when evaluating three or four options, resulting in a decreased robustness rate, with most models scoring below 0.5.”

May combine two claims.

Some models exhibited an aversion to excessively verbose answers, while others demonstrated a positive correlation between model preference and response length.

“Some models exhibited an aversion to excessively verbose answers, while others demonstrated a positive correlation between model preference and response length.”

Probably not a substantial claim: may not pass S2 (checkable).

LLMs are influenced by various factors.

“The findings indicate that LLMs are influenced by various factors.”

Classified as probably not a substantial claim.

The paper identified two distinct types of biases: explicit and implicit.

“We identified two distinct types of biases: explicit and implicit.”

May combine two claims. The quote only partly states this claim.

Different types of fake authorities interfere with LLMs to varying degrees, with the impact differing by the format used.

“Different types of fake authorities interfere with the LLMs to varying degrees.”

• Also stated in: § 4.2 ANALYSIS OF EXPLORATORY EXPERIMENTS · plain

  “As illustrated in Figure 6, the impact of fake authorities on judge models differs based on the format used.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Refinement-aware bias represents an implicit bias.

“Conversely, the refinement-aware bias represents an implicit bias.”

May combine two claims.

The effects of biases differ across models, and advanced models may not always exhibit better performance when dealing with these biases.

“Notably, the effects of these biases differ across models, and advanced models may not always exhibit better performance when dealing with these biases.”

May combine two claims.

An extensive evaluation of six popular LLMs using CALM reveals that while some LLMs demonstrate notable fairness in judgment, there remains significant room for improvement in achieving more robust decision-making across various types of bias.

“An extensive evaluation of six popular LLMs using the CALM framework, as shown in Figure 1, reveals that while some LLMs demonstrate notable fairness in judgment, there remains significant room for improvement in achieving more robust decision-making across various types of bias.”

May combine two claims.

The bias detection mechanism shows promise in uncovering a majority of biases, though effectiveness varies across different bias types.

“As presented in Table 6, our results demonstrate that while the effectiveness varies across different bias types, this method shows promise in uncovering a majority of biases.”

May combine two claims.

In the authority bias case study, after automatically generated fake references were added to Assistant B's answer, GPT-3.5-Turbo considered Assistant B's quality superior and cited the fake references as justification.

“However, when we added automatically generated fake references to Assistant B's answer, the judge model considered Assistant B's quality to be superior and cited our provided fake references as justification.”

May combine two claims.

The influence of varying percentages of public opinion on judge models is not uniform and does not demonstrate a statistical pattern.

“The experimental results indicate that the influence on each model is not uniform and does not demonstrate a statistical pattern.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

In the refinement-aware bias case study, GPT-4o scored the answer 6 before refinement, 7 after refinement, and 8 after refinement with dialogue history.

“In the example above, we asked the model to score the answer in three stages: before refinement, after refinement, and after refinement with the dialogue history.The respective scores were 6, 7, and 8.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Different percentages of popular opinion relate to robustness rate in bandwagon-effect bias.

“(c) shows the relationship between different percentages of popular opinion and robustness rate in bandwagon-effect bias.”

May combine two claims.

Different models and different fake citation formats relate to robustness rate in authority bias.

“(d) shows the relationship between different models and robustness rate in authority bias with different fake citation formats.”

Classified as probably not a substantial claim.

Twelve distinct types of bias can undermine the reliability and trustworthiness of LLM-as-a-Judge.

“A systematic definition and categorization of 12 distinct types of bias that can undermine the reliability and trustworthiness of LLM-as-a-Judge.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Another study highlights order bias, compassion-fade bias, and egocentric bias, along with salience bias, bandwagon-effect bias, and attentional bias.

“Another study (Koo et al., 2023) highlights order bias, compassion-fade bias, and egocentric bias, along with salience bias, bandwagon-effect bias, and attentional bias.”

• cites Koo et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

CALM enhances the integrity of the assessment process without relying on human resources.

“The introduction of CALM, a framework for evaluating biases in LLM-as-a-Judge systems, which enhances the integrity of the assessment process without relying on human resources.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some studies discuss biases such as position bias, verbosity bias, and self-enhancement bias.

“Some studies (Zheng et al., 2024;Shi et al., 2024b;Wang et al., 2023b) discuss biases such as position bias, verbosity bias, and self-enhancement bias.”

• cites Zheng et al., 2024;Shi et al., 2024b;Wang et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Setting temperature to 0.7 for all judge and generating models ensures stable output quality and strong reproducibility.

“We followed the experimental setup of Chen et al. (2024a) by setting the temperature to 0.7 and applied it to all judge models and generating models to ensure stable output quality and strong reproducibility.”

• cites Chen et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In the fact-related dataset, quality differences between answers are more evident, so bias is insufficient to completely offset this quality gap; in contrast, the alignment dataset typically has smaller quality differences, making judge choices more vulnerable to bias.

“One possible explanation for this is that, in the fact-related dataset, the quality differences between answers are more evident, which means that the influence of bias is insufficient to completely offset this quality gap.In contrast, the alignment dataset typically has smaller quality differences between answers, making the choices of the judge model more vulnerable to bias.”

• AssumptionRests on an assumption: The alignment dataset typically has smaller quality differences between answers.

May combine two claims.

Identifying the best model is not straightforward; it depends on the specific bias involved, and even top-tier models may display unexpected weaknesses.

“This complexity suggests that identifying the best model is not straightforward; it depends on the specific bias involved, and even top-tier models may display unexpected weaknesses.”

May combine two claims.

Models may reliably serve as judges for specific tasks, but there remains significant room for improvement in the broader use of LLMs as judges.

“Our experiments show that while models may reliably serve as judges for specific tasks, there remains significant room for improvement in the broader use of LLMs as judges.”

Classified as probably not a substantial claim.

LLM judgments may be influenced by the anonymity of model names (compassion-fade bias).

“▷ Compassion-fade bias: LLM judgments may be influenced by the anonymity of model names.”

• cites Koo et al. (2023)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable).

LLMs cannot truly achieve absolute fairness in a meaningful sense.

“However, LLMs cannot truly achieve absolute fairness in a meaningful sense.”

Classified as probably not a substantial claim.

Introducing distractions could affect the judgments of both high-quality and low-quality model outputs (distraction bias).

“▷ Distraction bias: Introducing distractions could affect the judgments of both high-quality and low-quality model outputs.”

• cites Koo et al. (2023)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Fallacy-oversight bias relates to the LLM's ability to recognize and avoid logical fallacies.

“▷ Fallacy-oversight bias: This bias relates to the LLM's ability to recognize and avoid logical fallacies.”

• cites Chen et al. (2024c)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Further biases noted in additional research include fallacy-oversight bias, authority bias, and beauty bias.

“Further biases noted in additional research (Chen et al., 2024c;Stureborg et al., 2024) include fallacy-oversight bias, authority bias, and beauty bias.”

• cites Chen et al., 2024c;Stureborg et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Pairwise comparison mitigates the issue of scoring without a reference answer and has been widely utilized for alignment data based on human annotations.

“In contrast, pairwise comparison mitigates this issue and has been widely utilized for alignment data based on human annotations (Ouyang et al., 2022).”

• cites Ouyang et al., 2022 sourceNot checked — no full text of the source

• AssumptionRests on an assumption: Without a reference answer, it can be challenging for LLM judges to provide an objective score, as their judgments can be easily influenced by contextual factors.

May combine two claims.

Many previous studies have relied on human evaluators to assess answer quality and compare them against LLM judgments, which incurs substantial costs and introduces human subjectivity, complicating reliable ground truth and reproducibility.

“Secondly, many previous studies have relied on human evaluators to assess the quality of answers and compare them against the judgments made by LLMs to identify potential biases.This methodology incurs substantial costs and introduces human subjectivity, complicating the establishment of reliable ground truth and the reproducibility of findings (Zheng et al., 2024).”

• cites Zheng et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The low interference of URL citations is likely due to their concise nature and models' familiarity with web-based references.

“URL citations consistently showed the least interference across all models, likely due to their concise nature and the models' familiarity with web-based references.”

Classified as probably not a substantial claim.

Recent research has identified various cognitive biases that influence the evaluation of LLMs.

“Recent research has identified various cognitive biases that influence the evaluation of LLMs.”

• cites Zheng et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Koo et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Chen et al., 2024cNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The variation in CoT effectiveness likely depends on models' inherent reasoning capabilities.

“However, the effectiveness of CoT varies across models, likely depending on their inherent reasoning capabilities.”

May combine two claims.

LLMs perform inconsistently when dealing with sentiment bias, potentially reflecting the phenomenon that humans are often influenced by emotions when making judgments.

“For instance, LLMs perform inconsistently when dealing with sentiment bias, potentially reflecting the phenomenon that humans are often influenced by emotions when making judgments.”

• cites Slovic et al., 2002 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The bias in LLMs may originate from the inherent limitations of human cognition.

“The bias in LLMs may originate from the inherent limitations of human cognition.”

• cites Slovic et al., 2002 sourceNot checked — no full text of the source
• cites Li et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Li et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

CALM eliminates subjective human assessments and reduces testing costs, resulting in a more objective and scalable evaluation approach.

“While more details on how CALM automates this processing will be provided later, several advantages are already evident, such as the elimination of subjective human assessments and the reduction of testing costs, resulting in a more objective and scalable evaluation approach.”