PreprintCited by 6Topic ModelingEthics and Social Impacts of AINatural Language Processing Techniques

Preference Leakage: A Contamination Problem in LLM-as-a-judge

Open access
Li, Dawei, Sun, Renliang, Huang, Yue, Zhong, Ming, Jiang, Bohan, Han, Jiawei, Zhang, Xiangliang, Wang, Wei +1 more·2025·arXiv

Abstract

Large Language Models (LLMs) as judges and LLM-based data synthesis have emerged as two fundamental LLM-driven data annotation methods in model development. While their combination significantly enhances the efficiency of model training and evaluation, little attention has been given to the potential contamination brought by this new model development paradigm. In this work, we expose preference leakage, a contamination problem in LLM-as-a-judge caused by the relatedness between the synthetic data generators and LLM-based evaluators. To study this issue, we first define three common relatednesses between the data generator LLM and the judge LLM: being the same model, having an inheritance relationship, and belonging to the same model family. Through extensive experiments, we empirically confirm the bias of judges towards their related student models caused by preference leakage across multiple LLM baselines and benchmarks. Further analysis suggests that preference leakage is a pervasive and real-world problem that is harder to detect compared to previously identified biases in LLM-as-a-judge scenarios. All of these findings imply that preference leakage is a widespread and challenging problem in the area of LLM-as-a-judge. We release all codes and data at: https://github.com/David-Li0406/Preference-Leakage.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper identifies “preference leakage,” a contamination problem in LLM-as-a-judge where a judge LLM favors student models trained on synthetic data from a related generator LLM. Across multiple benchmarks and model pairs, the authors find systematic bias that is pervasive, scales with synthetic-data proportion and relatedness, and is hard for judges to recognize.

Key findings

  • Preference leakage exists in most model pairs: with the same data generator and judge, the average preference leakage score (PLS) was 23.6% (Arena-Hard 28.7%, AlpacaEval 2.0 18.4%).
  • Relatedness severity varies: inheritance settings averaged 19.3% with same instructions and 22.3% with different instructions; same-family same-series averaged 8.9%, while different-series averaged 2.8%.
  • Learning method matters: SFT showed the highest average leakage at 23.6%, DPO was 5.2%, and ICL was -2.7%. Smaller student models exhibited higher PLS than larger counterparts.
  • In real-world leaderboards, preference leakage produced a ranking difference of 1.33, higher than egocentric bias’s 1.00. Judge LLMs performed poorly at recognizing their related student models’ generations (e.g., GPT-4o 41.0% pointwise / 52.0% pairwise; Gemini-1.5 53.2% / 44.2%; LLaMA-3.3 41.8% / 29.8%), while a BERT classifier reached 82.4% accuracy.

Methods

  • The authors define three generator–judge relatednesses: being the same model, having an inheritance relationship, and belonging to the same model family. They formalize preference leakage and propose the preference leakage score (PLS) to quantify judge bias toward related student models.
  • Main experiments used GPT-4o-2024-11-20, Gemini-1.5-flash, and LLaMA-3.3-70B-Instruct-turbo as data generators/judges; Mistral-7B-v0.1 and Qwen-2.5-14B pretrained models as students; 30,000 prompts sampled from Ultrafeedback for synthetic data; SFT training; and pairwise evaluation on Arena-Hard (500 questions) and AlpacaEval 2.0 (805 questions).
  • Further analyses covered data mixing (10%, 30%, 50%, 70% synthetic data), relatedness settings, learning methods (SFT, DPO, ICL), real-world leaderboard impact, judge recognition of student generations, and question-type/judgment-dimension categories. Manual annotation used 100 randomly selected AlpacaEval 2.0 questions with three annotators and majority voting.

Limitations

  • The paper states that detection, mitigation, and calibration methods for preference leakage are left for future work.
  • Due to cost, several further analyses were conducted only on the Mistral-GPT-4o vs Mistral-Gemini-1.5 pair, and manual annotation covered only 100 randomly selected AlpacaEval 2.0 questions.
  • Evident from the text: the study focuses on specific LLM baselines, benchmarks, and training setups, so broader generalization across other judges, data sources, and evaluation protocols is not established.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.01534

Cites

101 of 101 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 21 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Judges show systematic bias towards their related student models due to preference leakage across multiple LLM baselines and benchmarks.

    “Through extensive experiments, we empirically confirm the bias of judges towards their related student models caused by preference leakage across multiple LLM baselines and benchmarks.”
    • Also stated in: § 1 Introduction · hedged
      “The analysis results suggest an obvious bias of judging LLMs toward their related student models due to preference leakage.”
    • Also stated in: § 4.2 Main Results · plain
      “Preference leakage exists in most model pairs.”
    Quote in the paperSubstantial claim
  • Findingown data

    In inheritance settings, preference leakage causes obvious bias of judges towards their related students.

    “Preference leakage under inheritance settings causes obvious bias of judges towards their related students.”
    Quote in the paperSubstantial claim
  • Findingown data

    Models within the same series tend to cause more significant bias.

    “Models within the same series tend to cause more significant bias.”
    Quote in the paperSubstantial claim
  • Findingown data

    Preference leakage exists in most model pairs in the main SFT experiment.

    “Preference leakage exists in most model pairs.”
    • Also stated in: § 4.2 Main Results · hedged
      “This finding suggests that preference leakage, along with the resulting bias, is widespread in SFT when the data generator and evaluator are the same.”
    Quote in the paperSubstantial claim
  • Findingown data

    Smaller student models cause even more bias from judge LLMs.

    “Smaller student models cause even more bias from judge LLMs.”
    • Also stated in: § 4.2 Main Results · hedged
      “a notable finding is that the smallest models (Qwen-2.5-3Band Qwen-3-1.7B)exhibit the highest PL scores than their larger counterparts, indicating greater bias from preference leakage.”
    Quote in the paperSubstantial claim
  • May combine two claims. 27
  • The quote only partly states this claim. 3
  • Classified as probably not a substantial claim. 11
  • May combine two claims.

    Findingown data

    Gemini-1.5 shows a strong bias toward its students, followed by GPT-4o, with LLaMA-3.3 displaying the least bias.

    “we observe that Gemini-1.5 shows a strong bias toward its students, followed by GPT-4o, with LLaMA-3.3 displaying the least bias.”
    Quote in the paperSubstantial claim
  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In real-world LLM-as-a-Judge leaderboards, preference leakage causes a ranking difference of 1.33, higher than egocentric bias's 1.00.

    “Preference Leakage Vicuna 7B/ 13B/ 33B 1.33”
    • Also stated in: § GPT-4 Preview GPT-4 Preview 1.00 Preference Leakage · plain
      “the ranking difference associated with preference leakage is even higher than that of egocentric bias, highlighting the substantial impact of preference leakage on real-world LLM-as-a-Judge leaderboards.”
    Quote in the paperSubstantial claim
  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Preference leakage consistently affects judge LLMs under various data mixing strategies, relatedness settings, tuning techniques and real-world applications.

    “finding that preference leakage consistently affects judge LLMs.”
    • Also stated in: § 6 Conclusion · plain
      “preference leakage can cause significant bias across diverse scenarios.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The severity of preference leakage correlates with the degree of relatedness between the data generator and LLM judges, as well as the proportion of synthetic data.

    “the severity of preference leakage correlates with the degree of relatedness between the data generator and LLM judges, as well as the proportion of synthetic data.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    For same family with different series, the preference leakage score is 3.3% on Arena-Hard and 2.2% on AlpacaEval 2.0, with an average of 2.8%.

    “| Same Family -w/ different series | 3.3% | 2.2% | 2.8% |”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    For inheritance with the same instruction, the preference leakage score is 17.8% on Arena-Hard and 20.7% on AlpacaEval 2.0, with an average of 19.3%.

    “| Inheritance -w/ same ins. | 17.8% | 20.7% | 19.3% |”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    GPT-4o student recognition accuracy is 41.0% pointwise and 52.0% pairwise.

    “| | GPT-4o | 41.0% | 52.0% |”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    When the data generator and judge are the same model, the preference leakage score is 28.7% on Arena-Hard and 18.4% on AlpacaEval 2.0, with an average of 23.6%.

    “| Same Model | 28.7% | 18.4% | 23.6% |”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    For Mistral-7B with LLaMA-3.3 as data generator and Gemini-1.5 as judge, the preference leakage score is 13.1% on Arena-Hard and 19.8% on AlpacaEval 2.0, with an average of 16.4%.

    “| | LLaMA-3.3 & Gemini-1.5 | 13.1% | 19.8% | 16.4% |”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    For Qwen-2.5-14B with GPT-4o as data generator and Gemini-1.5 as judge, the preference leakage score is 37.1% on Arena-Hard and 18.6% on AlpacaEval 2.0, with an average of 27.9%.

    “| | GPT-4o & Gemini-1.5 | 37.1% | 18.6% | 27.9% |”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Findingown data

    For Mistral-GPT-4o vs Mistral-Gemini-1.5 on AlpacaEval 2.0, the GPT-4o judge gives Mistral-GPT-4o wins 55.1% and Mistral-Gemini-1.5 wins 44.9%.

    “| GPT-4o | 55.1% | | | 44.9% |”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    For Qwen-2.5-14B with LLaMA-3.3 as data generator and Gemini-1.5 as judge, the preference leakage score is 25.4% on Arena-Hard and 18.4% on AlpacaEval 2.0, with an average of 21.9%.

    “| | LLaMA-3.3 & Gemini-1.5 | 25.4% | 18.4% | 21.9% |”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    LLaMA-3.3 student recognition accuracy is 41.8% pointwise and 29.8% pairwise.

    “| | LLaMA-3.3 | 41.8% | 29.8% |”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    With ICL as the learning method, the preference leakage score is -4.2% on Arena-Hard and -1.1% on AlpacaEval 2.0, with an average of -2.7%.

    “| ICL | -4.2% | -1.1% | -2.7% |”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    With SFT as the learning method, the preference leakage score is 28.7% on Arena-Hard and 18.4% on AlpacaEval 2.0, with an average of 23.6%.

    “| SFT | 28.7% | 18.4% | 23.6% |”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    For same family with the same series, the preference leakage score is 10.1% on Arena-Hard and 7.6% on AlpacaEval 2.0, with an average of 8.9%.

    “| Same Family -w/ same series | 10.1% | 7.6% | 8.9% |”
    Quote in the paperSubstantial claim
  • May combine two claims. The quote only partly states this claim.

    Findingown data

    For Mistral-7B with GPT-4o as data generator and Gemini-1.5 as judge, the preference leakage score is 28.7% on Arena-Hard and 18.4% on AlpacaEval 2.0, with an average of 23.6%.

    “| | GPT-4o & Gemini-1.5 | 28.7% | 18.4% | 23.6% |”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    With DPO as the learning method, the preference leakage score is 7.7% on Arena-Hard and 2.7% on AlpacaEval 2.0, with an average of 5.2%.

    “| DPO | 7.7% | 2.7% | 5.2% |”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Gemini-1.5 student recognition accuracy is 53.2% pointwise and 44.2% pairwise.

    “| SR | Gemini-1.5 | 53.2% | 44.2% |”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Different from Arena-Hard, there is no tie in AlpacaEval 2.0.

    “Different from Arena-Hard, there is no tie in AlpacaEval 2.0.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The preference leakage score for Mistral-GPT-4o vs Mistral-Gemini-1.5 on AlpacaEval 2.0 is 18.4%.

    “| Preference Leakage Score | | | 18.4% | |”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    For inheritance with different instructions, the preference leakage score is 18.3% on Arena-Hard and 26.3% on AlpacaEval 2.0, with an average of 22.3%.

    “| Inheritance -w/ different ins. | 18.3% | 26.3% | 22.3% |”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    For Mistral-7B with GPT-4o as data generator and LLaMA-3.3 as judge, the preference leakage score is -1.5% on Arena-Hard and 1.4% on AlpacaEval 2.0, with an average of -0.1%.

    “| Mistral-7B | GPT-4o & LLaMA-3.3 | -1.5% | 1.4% | -0.1% |”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    For Qwen-2.5-14B with GPT-4o as data generator and LLaMA-3.3 as judge, the preference leakage score is 1.0% on Arena-Hard and 2.3% on AlpacaEval 2.0, with an average of 1.7%.

    “| Qwen-2.5-14B | GPT-4o & LLaMA-3.3 | 1.0% | 2.3% | 1.7% |”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    AlpacaEval 2.0 significantly reduces the effect of text length on evaluation results compared to version 1.0.

    “Compared to version 1.0, AlpacaEval 2.0 significantly reduces the effect of text length on the evaluation results.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationcitation

    Preference leakage can arise from various types of relatedness between data-generating and evaluating LLMs, as well as the intricate flow of synthetic data among modern LLMs.

    “It can arise from various types of relatedness between data-generating and evaluating LLMs, as well as the intricate flow of synthetic data among modern LLMs [57].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [57]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductreasoning

    Using pre-trained rather than instruct versions of student models avoids potential preference leakage due to distilling data from other LLMs during instruction-tuning.

    “To avoid potential preference leakage due to distilling data from other LLMs during the instruction-tuning process, we choose to use the -PRE-TRAINED version rather than the -INSTRUCT version of these student models.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisereasoning

    Models within the same family share a common architectural blueprint and are often developed from foundational models pre-trained on substantially overlapping datasets, leading to correlated preferences and systemic biases.

    “Models within such a family typically share a common architectural blueprint (A X ) and are often developed from foundational models pre-trained on substantially overlapping datasets (D X ).This shared foundation (A X , D X ) would lead to correlated preferences and systemic biases characteristic of the common origin:”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Models within the same family share a common architectural blueprint and are often developed from foundational models pre-trained on substantially overlapping datasets.
  • May combine two claims.

    Interpretationcitation

    Detecting preference leakage is more challenging because LLMs often do not disclose their training data and distillation quantification is difficult.

    “Moreover, detecting preference leakage is also more challenging, given LLMs often do not disclose their training data and the difficulty in distillation quantification [61,32].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [61]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [32]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: Most LLMs do not disclose their training data.
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Egocentric bias is most closely related to preference leakage.

    “Among these, egocentric bias, where LLM evaluators tend to favor their generations [29,47,63,68,52,49,5], is most closely related to the preference leakage proposed in this work.”
    Quote in the paperNot a substantial claim0 of 7 sources checked
    • cites [29]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [47]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [63]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [68]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [52]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [49]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Preference leakage is a pervasive and real-world problem that is harder to detect compared to previously identified biases in LLM-as-a-judge scenarios.

    “Further analysis suggests that preference leakage is a pervasive and real-world problem that is harder to detect compared to previously identified biases in LLM-as-a-judge scenarios.”
    • Also stated in: § 6 Conclusion · boosted
      “demonstrating that it is a challenging and hard-to-detect issue, especially in subjective questions and judgment dimensions.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Most LLMs do not disclose their training data.
  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Preference leakage presents a more complex and dynamic challenge than egocentric bias.

    “in contrast to the relatively straightforward setting of egocentric bias, preference leakage presents a more complex and dynamic challenge.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    In an inheritance relationship, the descendant model is likely to internalize and thus favor the preferences, styles, or biases of its progenitor.

    “In such cases, the descendant model is likely to internalize and thus favor the preferences, styles, or biases of its progenitor.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: In an inheritance relationship, the descendant model is likely to internalize and favor the preferences, styles, or biases of its progenitor.
  • May combine two claims.

    Interpretationreasoning

    Preference leakage is a widespread and challenging problem in the area of LLM-as-a-judge.

    “All of these findings imply that preference leakage is a widespread and challenging problem in the area of LLM-as-a-judge.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The variation in preference leakage across judges may stem from differences in the level of leaked preference in synthetic responses generated by data generator LLMs.

    “This variation in preference leakage may stem from differences in the level of leaked preference in the synthetic responses generated by the data generator LLMs.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationown data

    Preference leakage is a widespread problem in both LLM-as-a-judge datasets and LLM-relevant research.

    “All of these suggest preference leakage to be a widespread problem in both LLM-as-a-judge datasets and LLM-relevant research.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    An LLM with a distinctive style or format offers more opportunities for student models to learn these characteristics, potentially leading to more pronounced preference leakage.

    “For instance, an LLM with a distinctive style or format in its responses offers more opportunities for student models to learn these characteristics, potentially leading to more pronounced preference leakage during evaluation.”
    Quote in the paperNot a substantial claim
37 of 37
  • The paper relies on this without support.

    The three relatedness types (same model, inheritance, same family) are commonly encountered in real-world applications.

    “Each of these scenarios is commonly encountered in real-world applications.”
  • The paper relies on this without support.

    Little attention has been given to potential contamination when generator and evaluator LLMs share a close relationship.

    “However, limited attention has been given to the potential contamination that occurs when the generator and evaluator LLMs share a close relationship.”
  • The paper relies on this without support.

    Most LLMs do not disclose their training data.

    “especially given that most LLMs do not disclose their training data.”
  • The paper relies on this without support.

    Models within the same family share a common architectural blueprint and are often developed from foundational models pre-trained on substantially overlapping datasets.

    “Models within such a family typically share a common architectural blueprint (A X ) and are often developed from foundational models pre-trained on substantially overlapping datasets (D X ).”
  • The paper relies on this without support.

    In an inheritance relationship, the descendant model is likely to internalize and favor the preferences, styles, or biases of its progenitor.

    “In such cases, the descendant model is likely to internalize and thus favor the preferences, styles, or biases of its progenitor.”
  • The paper relies on this without support.

    In real-world applications, synthetic data from a single LLM is often mixed with manually-written data or other multi-source synthetic data to train student models.

    “In real-world applications, synthetic data from a single LLM is often mixed with manually-written data or other multi-source synthetic data to train student models.”
6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Question-type conclusion reverses the reported finding

    Minor

    The section says objective questions and dimensions show the least preference leakage and subjective ones show more. Its concluding sentence instead says leakage is more significant in objective questions and dimensions.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.

Cited by

Sorted by citations

Authors
Li, D., Sun, R., Huang, Y., Zhong, M., Jiang, B., Han, J., Zhang, X., Wang, W., & Liu, H.
Affiliations
  • Arizona State University (US)
  • University of California, Los Angeles (US)
  • University of Notre Dame (US)
  • University of Illinois Urbana-Champaign (US)
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4407176798
Cited by
2
References
98
Citations by year
2025: 2
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
LLM-as-a-judge, synthetic data generation, contamination, LLM-based evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review