CoChat
Sign Up
PreprintCited by 23Topic ModelingNatural Language Processing TechniquesEthics and Social Impacts of AI

Verbosity Bias in Preference Labeling by Large Language Models

Open access
Saito, Keita, Wachi, Akifumi, Wataoka, Koki, Akimoto, Youhei·2023·arXiv
DOI 10.48550/arxiv.2310.10076arXiv 2310.10076
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent years, Large Language Models (LLMs) have witnessed a remarkable surge in prevalence, altering the landscape of natural language processing and machine learning. One key factor in improving the performance of LLMs is alignment with humans achieved with Reinforcement Learning from Human Feedback (RLHF), as for many LLMs such as GPT-4, Bard, etc. In addition, recent studies are investigating the replacement of human feedback with feedback from other LLMs named Reinforcement Learning from AI Feedback (RLAIF). We examine the biases that come along with evaluating LLMs with other LLMs and take a closer look into verbosity bias -- a bias where LLMs sometimes prefer more verbose answers even if they have similar qualities. We see that in our problem setting, GPT-4 prefers longer answers more than humans. We also propose a metric to measure this bias.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper studies verbosity bias in LLM preference labeling, finding that GPT-4 tends to prefer longer answers in creative writing tasks and that its verbosity preference diverges from human feedback when humans prefer shorter answers. It proposes an accuracy-parity-based metric to quantify verbosity bias, reporting values of 0.328 for GPT-4 and 0.428 for GPT-3.5.

Key findings

  • In creative writing tasks, GPT-4 generally preferred longer answers generated by Vicuna-7b-v1.5; when the word count difference was large enough, GPT-4 almost always preferred the longer answer.
  • Using the HH-RLHF dataset, human alignment was high when human feedback preferred the longer answer, but low when humans chose the answer with fewer words, because the LLMs still chose the longer answers.
  • The proposed verbosity bias metric, based on accuracy parity, gave GPT-4 a value of 0.328 and GPT-3.5 a value of 0.428, where positive values indicate preference for verbose answers.
  • RLAIF evaluation cost around $0.05 each in the authors’ case, described as 1/15th compared to human feedback in Wang et al. (2023), which cost around 3 minutes ($0.75 if $15.00 per hour) per evaluation.

Methods

  • Experiment 1 generated 100 sample answers each to 3 prompts from the “creative” category using Vicuna-7b-v1.5 with temperature 0.7, then asked GPT-4 to choose between pairs of answers using a judge template; each pair was evaluated twice with positions swapped, and counted as a draw unless GPT-4 gave the same result on both permutations.
  • Experiment 2 used the HH-RLHF dataset of human feedback comparing pairs of answers; GPT-4 evaluated whole conversations with the same prompt template and was asked which assistant was more helpful, and the authors measured human alignment by word count difference between chosen and rejected options.
  • The paper formulates verbosity bias using a sensitive attribute based on which text is longer, equal opportunity/accuracy parity, and a directional inaccuracy difference equation (6), then computes values for GPT-4 and GPT-3.5 from the Section 5 data.

Limitations

  • The paper states that Section 4 uses the same questions, while Section 5 mixes results from various questions, so Section 5 gives only results across many kinds of questions rather than for any specific question; it says it is debatable which result is preferable.
  • The paper states that its verbosity bias metric only accounts for bias between two groups divided by whether y0 is longer than y1, cannot detect bias within each group, and is agnostic to cases where y0 is barely longer versus significantly longer; it recommends showing the human alignment plot alongside the metric.
  • The text makes evident that the experiments are limited to GPT-4 and GPT-3.5 as judges and Vicuna-7b-v1.5-generated samples, use only “creative” category prompts because other categories did not vary in word count enough and GPT-4 was not good at judging them, and the paper states that a further experiment on other LLMs for comparison is required.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.10076

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

13 of 20 cited sources checked · 7 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    Verbosity bias still exists for GPT-4.

    “we see that the verbosity bias still exists for GPT-4.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Wang et al. (2023) source

      Section 2.2, “Revealing the Positional Bias,” reports that GPT-4 favors responses in the first position, and the Conclusion likewise discusses positional bias. Searches for verbosity, response length, and preference for longer responses found no passage addressing whether GPT-4 has a verbosity bias. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Findingown data

    Humans seem to prefer longer answers in the HH-RLHF dataset.

    “As seen in Figure 4 which plots verbosity preference of humans in the HH-RLHF dataset described later, humans seem to prefer longer answers too.”
    Quote in the paperSubstantial claim
  • Findingown data

    When the word count difference is large enough, GPT-4 almost always prefers the longer answer.

    “When the word count difference is large enough, GPT-4 almost always prefers the longer answer.”
    Quote in the paperSubstantial claim
  • Findingown data

    GPT-4 has improved in verbosity bias compared to GPT-3.5.

    “From these numbers, we can conclude that GPT-4 has improved in verbosity bias.”
    Quote in the paperSubstantial claim
  • Findingown data

    There is a discrepancy between the preferences of LLMs and those of humans in verbosity preference.

    “there is a discrepancy between of LLMs and those of humans in verbosity preference”
    • Also stated in: § 8 Conclusion · plain
      “alignment with humans varies on verbosity with lower human alignment in cases where humans preferred shorter answers”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 18
  • May combine two claims. 13

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    RLHF significantly enhances alignment with human preferences.

    “This approach not only mitigates data scarcity but also significantly enhances alignment with human preferences, a critical factor in applications such as question answering.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Stiennon et al., 2020 source
    • cites Ouyang et al., 2022 source

  • May combine two claims.

    Premisecitation

    Previous studies on verbosity bias have focused on specific cases: Zheng et al. limited their problem setting to questions answered with lists, and Huang et al. conducted experiments on summarization tasks.

    “While previous studies have explored the concept of verbosity bias, they have tended to focus on specific cases.Zheng et al. (2023) limits their problem setting to questions answered with lists in their experiment on verbosity bias, and Huang et al. (2023) conducted experiments on summarization tasks.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Zheng et al. (2023) source
    • cites Huang et al. (2023) source

  • May combine two claims.

    Findingcitation

    In the paper's case, evaluation cost around $0.05 each, which is 1/15th compared to human feedback in Wang et al. (2023).

    “This brings down the cost significantly; in our case, evaluation cost around $0.05 each, which is 1/15th compared to human feedback in the previously cited paper (Wang et al., 2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Wang et al., 2023 source

      Section 4.1 describes human evaluation of 80 examples, and Table 4 in Section 4.3 lists a $30 cost per human annotator. That is $0.375 per annotation, not the $0.75 implied by calling $0.05 one-fifteenth of the human-feedback cost. The $0.05 figure is the citing authors’ own cost, not a figure attributed to Wang et al.; the paper supports the existence of a human-feedback cost comparison, but not the stated ratio. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    GPT-4 has high human alignment and agrees with humans on over 80% of evaluations.

    “Zheng et al. (2023) evaluate LLMs with other LLMs as a judge and show that GPT-4 has a high human alignment and agrees with humans on over 80% of evaluations.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Zheng et al. (2023) source

  • Classified as probably not a substantial claim.

    Premisecitation

    Human feedback in RLHF can become costly.

    “However, issues arise with RLHF where human feedback can become costly.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Wang et al., 2023 source

      Section 6.1 (page unavailable) says human evaluation is costly and time-consuming at scale, and Section 4.3 reports annotation costs. These passages concern evaluating model responses, not human feedback specifically within RLHF, so the RLHF qualification is unsupported. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    RLAIF replaces human feedback with feedback from other LLMs.

    “Reinforcement Learning from AI Feedback (RLAIF) was proposed (Bai et al., 2022b;Lee et al., 2023), which replaces human feedback with inexpensive feedback from other LLMs.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Bai et al., 2022b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Lee et al., 2023 source

  • May combine two claims.

    Premisecitation

    GPT-4 tends to prefer the first option given to it, while ChatGPT prefers the second option.

    “For example, GPT-4 tends to prefer the first option given to it, while ChatGPT prefers the second option (Wang et al., 2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Wang et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Zheng et al. found that GPT-4 is significantly less prone to the repetitive list attack with below 10% success rate, while GPT-3.5 and Claude-v1 both suffer over 90% success rate.

    “Their results show GPT-4 is significantly less prone to this attack with below 10% success rate, while GPT-3.5 and Claude-v1 both suffer over 90% success rate.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Zheng et al. (2023) source

      Section 3.3 (page unavailable) describes the repetitive-list attack and says GPT-4 defended significantly better than the other LLM judges. The Figure 12 caption (page unavailable) says GPT-3.5 and Claude-v1 favored the repetitive answer in the illustrated example. The passages retrieved from our copy do not give the claimed below-10% and over-90% success rates, so those numerical parts remain unsupported. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The LLMs trained by Lee et al. (2023) achieved near-human performance in summarization tasks with RLAIF without any human feedback.

    “The LLMs trained in the work by Lee et al. (2023) achieved near-human performance in summarization tasks with RLAIF without any human feedback.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Lee et al. (2023) source

      Section 4.1 reports that RLAIF and RLHF summarization models were preferred over the supervised baseline 71% and 73% of the time, respectively, and says RLAIF does not depend on human annotation for preference labels. This supports performance comparable to a human-feedback-trained model, not performance comparable to human summarizers. The summarization models also began with supervised fine-tuning on Reddit TL;DR data (Appendix F), so “without any human feedback” is too broad if it refers to the entire training process. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In Wang et al. (2023), it cost around 3 minutes ($0.75 if $15.00 per hour) per evaluation.

    “For example, in Wang et al. (2023), it cost around 3 minutes ($0.75 if $15.00 per hour) per evaluation.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Wang et al. (2023) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Zheng et al. (2023) experimented on verbosity bias by attempting a repetitive list attack on several LLMs.

    “Alongside their experiment on position bias, they experimented on verbosity bias by attempting a”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Zheng et al. (2023) source

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The prompts used in the first experiment were taken from Zheng et al. (2023)'s library, all from the creative category, because answers generated to other categories did not vary in word count enough to see verbosity bias and GPT-4 was not good at judging answers in those categories.

    “The prompts are taken from the library introduced by Zheng et al. (2023), all from the”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zheng et al. (2023) source

      Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    For question 1 and 2, GPT-4's preference for longer answers is smooth and clear; for question 3, when the word count difference is small, there is high variance in evaluation.

    “For question 1 and 2, the preference is smooth and clear, while for question 3, when the word count difference is small, there is high variance in evaluation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Table 1 shows the verbosity bias values of GPT-3.5 and GPT-4 calculated with data from Section 5.

    “Table 1 shows the verbosity bias values of GPT-3.5 and GPT-4 calculated with data from Section 5.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Vicuna achieved performance comparable to ChatGPT by utilizing user-shared conversations with ChatGPT as expert data.

    “Vicuna achieved performance comparable to ChatGPT by utilizing user-shared conversations with ChatGPT as expert data.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Chiang et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductreasoning

    The paper's formulation of verbosity bias only accounts for bias between two groups divided by whether y0 is longer than y1, and cannot detect bias within each of these groups.

    “Our formulation of verbosity bias only accounts for bias between two groups divided by whether y 0 is longer than y 1 .What it cannot detect is the bias within each of these groups; it is agnostic to the bias between cases where y 0 is barely longer than y 1 and cases where y 0 is significantly longer than y 0 .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Verbosity does not rely entirely on word count and is different for each question.

    “As we can see the shape varies between questions, and therefore we can deduce that verbosity does not rely entirely on word count and is different for each question.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Huang et al. found that GPT-4 prefers short responses in faithfulness and coverage for summarization, although this is seen strongly only in single-answer grading, and not in comparison grading.

    “They found that GPT-4 actually prefers short responses in faithfulness and coverage when it comes to summarization, although this is seen strongly only in single-answer grading, and not in comparison grading.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Huang et al. (2023) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    In Section 5, mixing results from various questions led to only attaining the result across many kinds of questions, not the result on any specific question.

    “However, in our experiment in Section 5, we mix together results from various questions.This has led us to only attain the result across many kinds of questions, not the result on any specific question like in the experiment in Section 4.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Training with RLAIF with verbosity bias present can lead to LLMs generating excessively long responses.

    “Training with RLAIF with verbosity bias present can lead to LLMs generating excessively long responses, when in reality a much more concise response would suffice.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    The experiments in prior papers measure the difference in evaluations when the texts are artificially made verbose while maintaining the same content.

    “The experiments conducted in these papers measure the difference in evaluations when the texts are artificially made verbose while maintaining the same content.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The study used position swapping to account for position bias: GPT-4 evaluated each pair twice with the position swapped, and a draw was recorded unless the same result was given on both permutations.

    “In order to account for position bias, GPT-4 evaluated the pair twice with the position swapped the second time.It was considered a draw unless it gave the same result on both permutations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Without accounting for verbosity bias, this may result in failures in downstream tasks such as lengthy summarizations or chatbots that return verbose responses to simple questions.

    “This may result in failures in downstream tasks such as lengthy summarizations or chatbots that return verbose responses to simple questions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The HH-RLHF dataset only has one feedback data per prompt, so the paper cannot plot the verbosity preference of humans like in the previous experiment.

    “It only has one feedback data per prompt, so we cannot plot the verbosity preference of humans like in the experiment in the previous chapter.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The first experiment by itself does not indicate that GPT-4 suffers from verbosity bias; the longer answers could be higher in quality and helpfulness.

    “However, this experiment by itself does not indicate that GPT-4 suffers from verbosity bias; it could be that the longer answers generated by vicuna are actually higher in quality and helpfulness.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Bai et al. (2022b) trained an LLM via RLAIF with limited human feedback.

    “Bai et al. (2022b) trained an LLM via RLAIF with limited human feedback.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Bai et al. (2022b)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

26 of 26

  • The paper relies on this without support.

    The aim of LLM judgment in RLAIF is human alignment and not the eradication of biased preference in verbosity.

    “We can view this as verbosity bias since the aim of LLM judgment in RLAIF is human alignment and not the eradication of biased preference in verbosity.”

  • The paper relies on this without support.

    Word count difference is a valid measure of verbosity.

    “We define the sensitive attribute S ∈ {0, 1} which equals 0 when y 0 has more words than y 1 , and 1 when y 1 has more words than y 0 .”

  • The paper relies on this without support.

    Human feedback in the HH-RLHF dataset is a valid oracle for human preference or helpfulness.

    “In our attempt to broaden the problem setting, we make use of human feedback as the oracle instead of making this assumption.”

  • The paper relies on this without support.

    In many cases, questions lack a clear-cut correct answer and require creativity and imagination.

    “In many cases, the question lack a clear-cut”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Systematic biases of single judges (position, verbosity, self-enhancement, sycophancy, and refusal artefacts) are well documented.

    “The systematic biases its backbone exhibits, e.g., position, verbosity, self-enhancement, sycophancy, and refusal artefacts, are by now well documented (Wang et al., 2023;Panickssery et al., 2024;Saito et al., 2023;Stureborg et al., 2024); they propagate uncorrected to every score, and the cost-quality profile of the resulting evaluation is fixed to that of the single model.”

    From RoPoLL: Robust Panel of LLM Judges (2026) · cites this paper as Saito et al., 2023

    Section 2.3, “Biases in Automated LLM Evaluation” (page not provided), describes position, self-enhancement, and verbosity biases; the paper also reports its own findings on verbosity bias. The passages do not document sycophancy or refusal artefacts, so this source supports only part of the cited list. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Longer lengths are associated with higher ratings, agreeing with previous studies.

    “This means that longer lengths are associated with higher ratings, which agrees with findings from previous studies [Saito et al., 2023].”

    From Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge (2025) · cites this paper as Saito et al., 2023

  • LLMs prefer longer responses, even if they are of similar qualities.

    “Prior studies have shown that LLMs prefer longer responses, even if they are of similar qualities (Hu et al., 2025; Saito et al., 2023).”

    From Challenging the Evaluator: LLM Sycophancy Under User Rebuttal (2025) · cites this paper as Saito et al., 2023

Cites

The publisher hasn’t deposited this paper’s reference list.

Cited by

Sorted by citations

Related

Authors
Saito, K., Wachi, A., Wataoka, K., & Akimoto, Y.
Affiliations
University of Tsukuba (JP)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.10076
arXiv
2310.10076
OpenAlex
W4387724705
Cited by
15
References
0
Citations by year
2026: 3 · 2025: 7 · 2024: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
large language models, reinforcement learning from AI feedback, reinforcement learning from human feedback, GPT-4, bias metrics, answer length

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review