Explaining Length Bias in LLM-Based Preference Evaluations

Previous empirical studies found a strong correlation between response length and perceived quality (win rate).

“Previous empirical studies have explored a strong correlation between the length of response and its perceived quality represented by win rate (Zhao et al., 2024;Dubois et al., 2024a;Park et al., 2024).”

• cites Zhao et al., 2024 sourceDisputed

  In “Comparison on generations of similar length” (page unavailable), the paper says response length and win rate were anticorrelated during fine-tuning. Section 4.2 (page unavailable) also reports that increasing response length slightly reduced win rate and concludes that length alone did not significantly affect benchmark results. These findings run contrary to the attributed strong correlation. (LLM, conf=0.87)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured
• cites Dubois et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Park et al., 2024 sourceSupported

DPO tends to make model responses longer.

“Previous work (Gu et al., 2024;Ivison et al., 2023) has shown that DPO (Rafailov et al., 2023) tends to make model responses longer”

• cites Gu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ivison et al., 2023 sourceSupported
• cites Rafailov et al., 2023 sourceNot checked — no full text of the source

LLM-based preference evaluations exhibit a notable bias towards longer responses.

“The use of large language models (LLMs) as judges, particularly in preference comparisons has become widespread, but this reveals a notable bias towards longer responses, undermining the reliability of such evaluations.”

• Also stated in: § 1 Introduction · plain

  “One of the important biases observed in LLM-based evaluations is the preference for longer textual responses.”

• cites Zhao et al., 2024 sourcePartly supported

  The Discussion (section 6; page unavailable) treats a preference for longer replies as a possible bias of LLM judges. It does not establish a notable bias that undermines preference evaluations; instead, the authors conclude that their own results are not due to length bias. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Dubois et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Park et al., 2024 sourceSupported

GPT-4 used as an annotator to represent human preference prefers longer sentences over shorter ones.

“Specifically, when using a GPT-4 to represent human preference, it will prefer to choose a long sentence rather than a short sentence (Dubois et al., 2024a;Ivison et al., 2023;Gu et al., 2024;Hu et al., 2024b;Koo et al., 2023;Wang et al., 2023a;Wu and Aji, 2023;Dubois et al., 2024b;Chen et al., 2023;Hu et al., 2024c).”

• cites Dubois et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ivison et al., 2023 sourcePartly supported

  The Evaluation Suite section says the paper uses GPT-4 to compare model responses in AlpacaEval. Table 4 and the ‘DPO training increases model verbosity’ section report longer DPO-model outputs alongside higher AlpacaEval win rates, but they do not establish that GPT-4 prefers a longer response over a shorter one because of its length. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites Gu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Length itself is not inherently problematic.

“While many prior works have treated response length as a confounding factor in evaluation (Dubois et al., 2024a;Koo et al., 2023), we posit that length itself is not inherently problematic.”

• cites Dubois et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Koo et al., 2023 sourceSupported

Responses with negative desirability significantly decrease the win rate.

“Our experiments demonstrate that responses with negative desirability significantly decrease the win rate, whereas information mass, when not negatively influenced by desirability, is positively correlated with the win rate, thus confirming the effectiveness of our metric.”

• Also stated in: § 3.3 Desirability Influences Quality · plain

  “Second, responses generated from negative prompts (i.e., Biased, Inconsistent, Illogical, Verbose, and Toxic) exhibit low desirability, resulting in win rates substantially lower than Origin (50%), despite having similar information mass.”

Information mass, when not negatively influenced by desirability, is positively correlated with the win rate.

“Our experiments demonstrate that responses with negative desirability significantly decrease the win rate, whereas information mass, when not negatively influenced by desirability, is positively correlated with the win rate, thus confirming the effectiveness of our metric.”

• Also stated in: § 3.4 Information Mass Influences Quality · boosted

  “Specifically, responses with higher information mass, measured by conditional entropy, consistently achieved higher win rates.”

• AssumptionThe authors state this as an assumption: Information mass is represented by conditional entropy and is directly related to response length.

Length bias in LLM-based preference evaluations undermines the reliability of win rate metrics.

“In this paper, we identify and address the critical issue of length bias in LLM-based preference evaluations, which undermines the reliability of win rate metrics.”

• Also stated in: § E Addressing Confounding Factors in Evaluation Metrics · plain

  “Evaluating the quality of responses solely based on win rates can be misleading when external factors, such as response length, unduly influence outcomes.”

The difference between human evaluation and WR decreases as the quality of responses improves (from concise to detailed to Quality Enhancement).

“Additionally, we find that the difference between human evaluation and WR decreases as the quality of responses improves (from concise to detailed to Quality Enhancement).”

Responses in the AlpacaEval dataset predominantly fall within the 0-1000 word range.

“we analyzed the word count distribution within the AlpacaEval dataset, observing that responses predominantly fall within the 0-1000 word range.”

Information mass significantly affects the win rate without a negative desirability prompt.

“Our findings indicate that information mass significantly affects the win rate without a negative desirability prompt.”

• AssumptionThe authors state this as an assumption: Information mass is represented by conditional entropy and is directly related to response length.

All models have higher win rates on AdapAlpaca compared to AlpacaEval.

“Additionally, we found that all models have higher win rates on AdapAlpaca compared to AlpacaEval.”

When evaluated using current AlpacaEval, GPT-4 selected a redundant answer that is significantly unaligned from human preference.

“When evaluated using the current AlpacaEval response (shown in the red box), the annotator (i.e., GPT-4) selected this redundant answer, which is significantly unaligned from human preference, as the simplicity of the question does not warrant such extensive verbosity.”

Responses with higher information mass, measured by conditional entropy, consistently achieved higher win rates.

“Specifically, responses with higher information mass, measured by conditional entropy, consistently achieved higher win rates.”

• AssumptionThe authors state this as an assumption: Information mass is represented by conditional entropy and is directly related to response length.

The relationship between information mass and win rate is Detailed > Origin > Concise.

“Thus, we observe the following relationship: Detailed > Origin > Concise.”

As the length of a response increases, the information mass also grows.

“As shown in Figure 4, our analysis demonstrates a clear trend: as the length of a response increases, the information mass also grows.”

• AssumptionThe authors state this as an assumption: Information mass is represented by conditional entropy and is directly related to response length.

DPO's gains in human preference are influenced by response length.

“Additionally, our analysis of DPO demonstrates that its gains in human preference are influenced by response length, underscoring the importance of unbiased evaluation benchmarks.”

AdapAlpaca effectively mitigates length bias by dynamically adjusting reference answer lengths to match test model responses, ensuring fairer evaluation metrics.

“Our proposed framework, AdapAlpaca, effectively mitigates length bias by dynamically adjusting reference answer lengths to match test model responses, ensuring fairer evaluation metrics.”

• Also stated in: § C Discussion on Length Bias · plain

  “Our proposed AdapAlpaca further adopts a proactive stance by mitigating length bias early in the evaluation pipeline, thus ensuring that win rates reflect genuine improvements in response quality rather than inflated text length.”

• AssumptionThe authors state this as an assumption: Content quality is preserved across varying lengths when AdapAlpaca adjusts the length of reference answers to match test model responses.

Longer responses generally exhibit higher vocabulary sizes and word counts.

“1) Longer responses generally exhibit higher vocabulary sizes and word counts, suggesting a richer linguistic structure.”

The human evaluation process validated the effectiveness of the proposed methodologies.

“This comprehensive human evaluation process not only validated the effectiveness of our proposed methodologies but also provided critical insights that significantly enriched our understanding of automated metric evaluations.”

Changes in correctness and logical consistency (Desirability) significantly influence win rates even when response length remains constant.

“Our results show that changes in correctness and logical consistency (Desirability) significantly influence win rates even when response length remains constant.”

In 14 commonly used preference datasets, the lengths of chosen responses are generally longer than those of rejected responses.

“We found that the lengths of chosen responses are generally longer than those of rejected responses.”

Scores given by 10 commonly used reward models to detailed, original, and concise responses from GPT-4 consistently decrease across all reward models.

“We find that the scores consistently decrease across all reward models.”

Win rate calculated using AdapAlpaca closely aligns with human results, showing an average difference of 0.99%.

“In contrast, the win rate calculated using AdapAlpaca closely aligns with the human results, showing an average difference of 0.99% (1.12% + 1.07% + 0.78% / 3).”

Prompts enhancing desirability (Consistent and Logical) yield increased win rates compared to Origin.

“Conversely, prompts enhancing desirability (i.e., Consistent and Logical) yield increased win rates compared to Origin.”

Information mass influences quality for GPT-4 when using Llama3-8B as the annotator model.

“Figure 18: Validation of information mass's influence on quality for GPT-4 (using Llama3-8B as the annotator model).”

The absolute magnitude of change in artifact scores as length increases is small.

“While there is a mild upward trend as length increases (particularly in redundancy), the absolute magnitude of change is small and does not suggest significant degradation in response quality.”

Desirability impacts quality for GPT4-o.

“Figure 13: Validation of desirability's impact on quality for GPT4-o.”

When using Llama3-8B as the annotator model, there is a more pronounced length bias.

“However, when using Llama3-8B (AI@Meta, 2024) as the annotator model, we observe a more pronounced length bias.”

• cites AI@Meta, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Information mass influences quality for GPT-4 when using Llama3-70B as the annotator model.

“Figure 20 : Validation of information mass's influence on quality for GPT-4 (using Llama3-70B as the annotator model).”

There is a mild upward trend in artifact scores as length increases, particularly in redundancy.

“While there is a mild upward trend as length increases (particularly in redundancy), the absolute magnitude of change is small and does not suggest significant degradation in response quality.”

All three artifact metrics (confusion, redundancy, dilution of meaning) remain below 2.0 across all length intervals in AdapAlpaca.

“As shown in Table 14, all three metrics remain below 2.0 across all length intervals, indicating a low level of confusion, redundancy, and dilution of meaning-even for longer responses.”

Under Win Rate (WR) metrics, longer responses disproportionately receive higher preference scores.

“3) Under Win Rate (WR) metrics, longer responses disproportionately receive higher preference scores due to their higher information mass.”

In 14 commonly used preference datasets on Hugging Face, the lengths of chosen responses are generally longer than those of rejected responses.

“The analysis shows that the lengths of chosen responses are generally longer than those of rejected responses, indicating a length bias in human preference labeling.”

The quality of information, as measured by win rate, does not necessarily increase proportionally with response length.

“Our results demonstrate that although longer responses generally possess higher information mass, the quality of information, as measured by win rate, does not necessarily increase proportionally.”

The win rate of AdapAlpaca-1000 is lower than that of AdapAlpaca-800.

“For instance, in Table 10, the win rate of AdapAlpaca-1000 is lower than that of AdapAlpaca-800.”

For GPT-4 responses, as word count increases, information mass also increases.

“Correlation between information mass and word count for responses of GPT-4.As the word count increases, the information mass also increases.”

LCWR leads to a more balanced distribution of scores across different response lengths.

“However, applying the length-controlled win rate (LCWR) significantly mitigates this bias, leading to a more balanced distribution of scores across different response lengths.”

At larger annotator model scales, such as Llama3-70B, the results are consistent with those obtained using GPT-4 (1106) in the main text.

“Our findings show that at larger model scales, such as Llama3-70B (AI@Meta, 2024), the results are consistent with those obtained using GPT-4 (1106) (Achiam et al., 2023) in the main text.”

• cites AI@Meta, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Achiam et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Information mass impacts quality for Qwen1.5-72b.

“Figure 12: Validation of information mass's impact on quality for Qwen1.5-72b.”

Information mass impacts quality for GPT4-o.

“Figure 14: Validation of information mass's impact on quality for GPT4-o.”

Desirability impacts quality for Qwen1.5-72b.

“Figure 11: Validation of desirability's impact on quality for Qwen1.5-72b.”

Desirability influences quality for GPT-4 when using Llama3-70B as the annotator model.

“Figure 19: Validation of desirability's influence on quality for GPT-4 (using Llama3-70B as the annotator model).”

Win rate is influenced by desirability and information mass in separable and predictable ways.

“Our experiments empirically validate that win rate is influenced by these two components in separable and predictable ways.”

Longer responses generally possess higher information mass.

“Our results demonstrate that although longer responses generally possess higher information mass, the quality of information, as measured by win rate, does not necessarily increase proportionally.”

With Llama3-8B as annotator, longer copy-pasted responses receive a significantly higher win rate.

“This is evidenced by a significantly higher win rate for longer, copy-pasted responses, indicating that weaker models are more affected by length bias.”

Reward model scores consistently decrease from detailed to concise responses.

“The analysis shows that the scores consistently decrease from detailed to concise responses, highlighting the length bias within the reward model.”

Applying the length-controlled win rate (LCWR) significantly mitigates the length bias in preference scores.

“However, applying the length-controlled win rate (LCWR) significantly mitigates this bias, leading to a more balanced distribution of scores across different response lengths.”

In AdapAlpaca, GPT-4 and humans vote for the same answer.

“In AlpacaEval, GPT-4 votes for the verbose answer, but humans vote for the concise reference answer, while in AdapAlpaca, GPT-4 and humans vote for the same answer, demonstrating a better LLM-human alignment on AdapAlpaca.”

Win rate can be decomposed into desirability (length-independent, related to trustworthiness) and information mass (length-dependent, amount of information).

“we propose to decompose the preference evaluation metric, specifically the win rate, into two key components: desirability and information mass, where the former is length-independent and related to trustworthiness such as correctness, toxicity, and consistency, and the latter is length-dependent and represents the amount of information in the response.”

• AssumptionThe authors state this as an assumption: For a given answer and instruction, the quality recognized by an annotator can be viewed as a function of two key components: desirability and information mass.

Response length impacts evaluations primarily by influencing information mass.

“We empirically demonstrated the decomposition through controlled experiments and found that response length impacts evaluations by influencing information mass.”

• Also stated in: § 1 Introduction · plain

  “Through our decomposition of the quality of a response, we observe that response length impacts evaluations primarily by influencing information mass.”

• Also stated in: § 4 Adaptive AlpacaEval · boosted

  “By integrating this observation with the findings from Section 3.2, we conclude that the primary mechanism through which length affects win rate is its contribution to the overall information mass.”

• AssumptionThe authors state this as an assumption: For a given answer and instruction, the quality recognized by an annotator can be viewed as a function of two key components: desirability and information mass.
• AssumptionThe authors state this as an assumption: Desirability is independent of response length.
• AssumptionThe authors state this as an assumption: Information mass is represented by conditional entropy and is directly related to response length.

Responses with good desirability and higher information mass are generally more favored.

“Our decomposition reveals that responses with good desirability and higher information mass are generally more favored.”

Longer responses may appear more informative or thorough, increasing the likelihood of higher win rates, even when additional content is redundant or irrelevant.

“Longer responses may appear more informative or thorough, thereby increasing the likelihood of higher win rates, even when the additional content consists of redundant or irrelevant information.”

RLHF ranking data generally ranks longer responses higher than shorter ones.

“This leads to ranking data where longer responses are generally ranked higher than shorter ones”

The consistent improvement in win rates across all tested models underscores the critical role of response quality in LLM evaluation.

“The consistent improvement in win rates across all tested models underscores the critical role of response quality in LLM evaluation.”

Length bias observed in LLMs essentially originates from the RLHF process.

“We believe that the length bias observed in LLMs essentially originates from the RLHF (Ouyang et al., 2022) process.”

• cites Ouyang et al., 2022 sourceNot checked — no full text of the source

During RLHF, humans may generally prefer more detailed responses when labeling preference data.

“during the RLHF process, humans may generally prefer more detailed responses when labeling preference data.”

Raters assigned to each segment generally reached a consistent judgment.

“These results demonstrate that, despite different segments being allocated to different groups of participants, the raters assigned to each segment generally reached a consistent judgment.”

Information mass is a crucial factor influencing the quality of responses.

“These results confirm that information mass is a crucial factor influencing the quality of responses.”

AdapAlpaca removes length bias from the outset rather than correcting for it post hoc.

“Our work adopts a similar principle through the design of AdapAlpaca, which seeks to remove length bias from the outset rather than correcting for it post hoc.”

• AssumptionThe authors state this as an assumption: Content quality is preserved across varying lengths when AdapAlpaca adjusts the length of reference answers to match test model responses.

Desirability and Information Mass were validated as independent through controlled experiments in Section 3.3 and Section 3.4.

“Empirically, we validated the independence of Desirability and Information Mass through controlled experiments in Section 3.3 and Section 3.4.”

Desirability plays a significant role in determining quality.

“In summary, desirability plays a significant role in determining quality.”

As response quality increases, the preferences of annotators and human evaluators converge.

“This suggests that as response quality increases, the preferences of annotators and human evaluators converge.”

This length bias is propagated to the aligned model during training using the reward model.

“This bias is further propagated to the aligned model during the training process using the reward model.”

Longer responses have a richer linguistic structure.

“1) Longer responses generally exhibit higher vocabulary sizes and word counts, suggesting a richer linguistic structure.”

AdapAlpaca allows control for content while varying length, thereby isolating the effect of length from content quality.

“In contrast, when using AdapAlpaca, it allows us to control for content while varying the length, thereby isolating the effect of length from that of content quality.”

The controlled generation setup preserves semantic coherence and content fidelity across varying information mass levels.

“These results validate the design of our controlled generation setup: by guiding GPT-4 to produce responses that naturally fall into target length ranges, we preserve semantic coherence and content fidelity across varying information mass levels.”

Weaker models are more affected by length bias.

“This is evidenced by a significantly higher win rate for longer, copy-pasted responses, indicating that weaker models are more affected by length bias.”

The win rate gain from DPO is further amplified by response length, leading to higher performance in AlpacaEval compared to AdapAlpaca.

“The results indicate that while DPO increases response length and improves win rate, the win rate gain is further amplified by the response length, leading to higher performance in AlpacaEval compared to AdapAlpaca.”

AdapAlpaca dynamically selects a reference answer that matches the length of the test model's answer.

“In contrast, AdapAlpaca dynamically selects a reference answer that matches the length of the test model's answer.”

AdapAlpaca demonstrates better LLM-human alignment than AlpacaEval.

“In AlpacaEval, GPT-4 votes for the verbose answer, but humans vote for the concise reference answer, while in AdapAlpaca, GPT-4 and humans vote for the same answer, demonstrating a better LLM-human alignment on AdapAlpaca.”

The influence of length on evaluation emerges through two intertwined but distinguishable dimensions: information mass and desirability.

“Rather, its influence emerges through two intertwined but distinguishable dimensions: how much content is conveyed (i.e., information mass) and whether that content is desirable in terms of trustworthiness and utility.”

In conditional settings, responses with more unique, relevant, and elaborated content exhibit higher conditional entropy.

“In conditional settings-i.e., given an instruction-responses with more unique, relevant, and elaborated content exhibit higher conditional entropy.”

In AlpacaEval, the reference answer has a fixed length regardless of the test model's answer length.

“In AlpacaEval, the reference answer has a fixed length, regardless of the length of the test model's answer.”

The increase in artifact scores with length does not suggest significant degradation in response quality.

“While there is a mild upward trend as length increases (particularly in redundancy), the absolute magnitude of change is small and does not suggest significant degradation in response quality.”

There is length bias within the reward model.

“The analysis shows that the scores consistently decrease from detailed to concise responses, highlighting the length bias within the reward model.”

Longer responses often receive higher win rates because they accumulate more information mass, which is rewarded by LLM-based annotators even when desirability remains constant.

“By modeling response quality as a function of these two components, we obtain a framework that explains why longer responses often receive higher win rates: they accumulate more information mass, which is rewarded by LLM-based annotators-even when desirability remains constant.”

Increasing length without regard to desirability may artificially inflate quality judgments without improving actual usefulness.

“Conversely, increasing length without regard to desirability may artificially inflate quality judg-ments without improving actual usefulness.”

Excessively lengthy responses can result in a decline in desirability, such as reduced consistency.

“Excessively lengthy responses can result in a decline in desirability, such as reduced consistency.”

There is a low level of confusion, redundancy, and dilution of meaning even for longer responses.

“As shown in Table 14, all three metrics remain below 2.0 across all length intervals, indicating a low level of confusion, redundancy, and dilution of meaning-even for longer responses.”

Human preference labeling in the 14 datasets exhibits a length bias.

“The analysis shows that the lengths of chosen responses are generally longer than those of rejected responses, indicating a length bias in human preference labeling.”

The reward model learns and propagates a spurious correlation from ranking data where longer responses are ranked higher to the aligned model.

“This creates a spurious correlation that the reward model learns and propagates to the aligned model.”