Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

To mitigate stochastic variance and 'self-preference' bias, the study implemented a 3-vote consensus (Majority Vote) system for the LLM-as-a-judge evaluation.

“To mitigate stochastic variance and”

Existing efforts employing the LLM-as-a-judge methodology lack both a comprehensive framework design and verification of human consistency.

“Others employ the LLM-as-a-judge methodology [36], yet these efforts lack both a comprehensive framework design and verification of human consistency.”

The choice of LLM judge can shift accuracy by 5-10 percentage points on identical retrieval output.

“The choice of judge can shift accuracy by 5-10 percentage points on identical retrieval output (Zheng et al., 2023).”

Alpaca is known to perform worse than Vicuna and Llama-2 on relevant tasks.

“considering that Alpaca is known to perform worse than our other open-source autoregressive models, Vicuna and Llama-2 (refs. 21,35)”

Length bias, position bias, and style bias can distort evaluation results regardless of model capability.

“Additional biases further complicate assessment due to inherent limitations in LLMs: length bias (favoring unnecessarily long outputs), position bias (order effects in candidates), and style bias (favoring certain linguistic styles over content) [337-338,343], potentially distorting results regardless of model capability.”

Fine-tuning open-source LLMs as evaluators improves scalability and efficiency but may inherit biases present in training data, such as position and verbosity bias.

“While this method improves scalability and efficiency, it may also inherit biases present in the training data, such as position and verbosity bias.”

Automated evaluation results can be affected by prompt sensitivity, demonstration setup, and data contamination.

“While automated and scalable, evaluation results can be affected by several factors [337-338], including prompt sensitivity, demonstration setup, and data contamination.”

As a result, generation-based metrics systematically bias comparisons across models, conflating memory fidelity with surface realization.

“As a result, generation-based metrics systematically bias comparisons across models, conflating memory fidelity with surface realization.”

Modern LLMs exhibit diverse generation behaviors in terms of verbosity, reasoning style, and response length.

“modern LLMs exhibit diverse generation behaviors in terms of verbosity, reasoning style, and response length.”

Although the 3-vote majority consensus protocol reduces stochastic fluctuation, the judge model may exhibit systematic biases toward responses generated by models in the same model family, and may be more lenient on verbose near-misses than a human annotator would be.

“Although the 3-vote majority consensus protocol reduces stochastic fluctuation, the judge model may exhibit systematic biases toward responses generated by models in the same model family (Zheng et al., 2023;Ye et al., 2024), and may be more lenient on verbose near-misses than a human annotator would be (Zheng et al., 2023;Ye et al., 2024;Chen et al., 2024).”

The LLM-as-a-judge framework using GPT-5-mini as evaluator introduces its own sources of variance.

“The LLM-as-a-judge framework using GPT-5-mini as evaluator (Zheng et al., 2023) introduces its own sources of variance.”

Most systems lean on LLM judges without validation on their credibility in reviewing scientific research.

“However, despite their widespread usage as 'generators' in the forward pass of scientific research, their utility in the backward pass of academic verification or as verifiers remains underexplored, a blind spot in which most systems lean on LLM judges [16] without validation on their credibility in reviewing scientific research.”

Concurrent works often omit a rigorous backward pass or 'verification' and instead rely on LLM judges.

“However, concurrent works often omit a rigorous backward pass or 'verification' and instead rely on LLM judges [16].”

The Chatbot Arena can be viewed as an effort to scale human preference judgments into a systematic ranking methodology, producing orderings that frequently diverge from those derived from standard benchmarks.

“The Chatbot Arena (Zheng et al., 2023) can be viewed in a similar light-as an effort to scale human preference judgments into a systematic ranking methodology, producing orderings that frequently diverge from those derived from standard benchmarks.”

Large-scale systems experience shows that production failure modes are subtle, and validation must include representative workloads and fault patterns.

“Large-scale systems experience shows that production failure modes are subtle, and validation must include representative workloads and fault patterns [164], [165].”

GPT-4-Turbo holds the zero-shot state-of-the-art on numerous NLP tasks as of this writing, and achieves first place in human evaluations of chat-based models in Chatbot Arena.

“GPT-4-Turbo holds the zero-shot state-of-the-art on numerous NLP tasks as of this writing, and achieves first place in human evaluations of chat-based models in Chatbot Arena [77].”

LLM-as-a-judge systems can exhibit self-bias, systematically assigning higher scores to their own outputs than to equally good outputs from other models.

“models may systematically assign overly favorable ratings to their own outputs-a phenomenon known as self-bias-which can distort evaluations of true model performance.”

Unlike other biases, self-bias cannot be artificially introduced via perturbations of a completion, making its study particularly challenging.

“While other biases can be artificially introduced via perturbations of a completion, this is not the case for self-bias, making its study particularly challenging [Zheng et al., 2023].”

A more diverse set of answers is beneficial because selecting the correct solution from multiple options is often easier than relying on a single agent's solution.

“A more diverse set of answers is beneficial because selecting the correct solution from multiple options is often easier than relying on a single agent's solution (Zheng et al., 2023).”

Published LoCoMo numbers are not directly comparable because evaluation frameworks differ along multiple axes: answer LLM, judge model and prompt, answer prompt, and dataset split.

“Published LoCoMo numbers are not directly comparable because evaluation frameworks differ along multiple axes (Zheng et al., 2023; Wang et al., 2024): (i) answer LLM (gpt-4o-mini vs. gpt-4.1-mini), (ii) judge model and prompt (binary J-score vs. 7-step chainof-thought rubric), (iii) answer prompt (direct vs. CoT), and (iv) dataset split (LoCoMo-10 vs. LoCoMo-5).”

Open-ended correctness is determined by majority vote from a three-way cross-vendor judge ensemble; judge-produced auxiliary scores are used for preference alignment, bias identification, and memory fidelity.

“Open-ended correctness is determined by majority vote from a three-way cross-vendor judge ensemble (Zheng et al. 2023); judge-produced auxiliary scores are used for preference alignment, bias identification, and memory fidelity.”

Generation-based metrics systematically bias comparisons across models, conflating memory fidelity with surface realization.

“As a result, generation-based metrics systematically bias comparisons across models, conflating memory fidelity with surface realization.”

LLMs are increasingly used as evaluators for text generation tasks and in leaderboards for benchmarking AI models.

“Large Language Models (LLMs) are gaining widespread acceptance as the gold standard for evaluation in numerous applications, thanks to their efficiency and significant reductions in cost & time compared to human evaluators (Kim et al., 2023”

GPT-4o was used as an external judge to evaluate Factual Consistency against ground truth, penalizing contradictions or 'I don't know' responses if the answer exists.

“We use GPT-4o as an external judge. The judge evaluates Factual Consistency against the ground truth, penalizing contradictions or 'I don't know' responses if the answer exists [31].”

LLM-as-a-Judge models can approximate the depth and consistency of manual human evaluation while providing better consistency and stability.

“These models have shown they can approximate the depth and consistency of manual human evaluation, but also provide better consistency and stability.”

LLM-based evaluation is more effective than exact matches or F1 scores for numeric evaluation and long-form responses.

“as LLM-based evaluation is more effective than exact matches or F1 scores for numeric evaluation and long-form responses [14, 21, 68].”

LLM-as-a-judge typically has high success at capturing human preferences across a range of tasks.

“previous work has shown that ‘LLM-as-a-judge’ typically has high success at capturing human preferences across a range of tasks40.”

LLM-as-a-judge achieves roughly 80% judge-human agreement, and exhibits position, verbosity, and self-enhancement biases.

“MT-Bench and Chatbot Arena made LLM-as-a-judge a measurable methodology, quantifying both the ~80% judge-human agreement that justifies the paradigm and the position, verbosity, and self-enhancement biases that qualify it [131]”

The LLM-as-a-judge approach for Relevant Content and Fact Check evaluations may retain biases inherent to the judge model, including position bias and self-enhancement effects, despite calibration through human review.

“the LLM-as-a-judge approach for Relevant Content and Fact Check evaluations, despite calibration through human review, may retain biases inherent to the judge model, including position bias and self-enhancement effects (Wang et al., 2024; Zheng et al., 2023).”

LLM judges provide superior accuracy in the evaluation of open-ended tasks.

“Our further experiments suggest that LLM judges (Li et al., 2023c;Zheng et al., 2023) provide superior accuracy in the evaluation of open-ended tasks.”

Arena Elo ratings of LLMs are indicative of their overall capabilities.

“the Arena Elo ratings of LLMs (Zheng et al., 2023), which are indicative of their overall capabilities.”

LLM-as-judge uses a second, ideally different-vendor model to evaluate the first against a rubric; it scales where manual checking cannot but imports its own failure modes.

“LLM-as-judge30 uses a second, ideally different-vendor model to evaluate the first against a rubric; it scales where manual checking cannot but imports its own failure modes.”

A fixed LLM Judge can evaluate relevance, specificity, actionability, and overall quality of suggestions, and chain consistency.

“A fixed LLM Judge (Liu et al. 2023;Zheng et al. 2023) additionally evaluates the paper, predicted critique, and generated suggestion for relevance, specificity, actionability, and overall quality on a 1-5 scale, where higher scores are better.”

Accurately judging response quality in high-stakes domains hinges on identifying whether a response is correct, which can require complex reasoning and domain knowledge.

“In these settings, accurately judging response quality hinges on identifying whether a response is correct, which itself can require complex reasoning and domain knowledge (Tan et al., 2025;Zheng et al., 2023).”

Zheng et al. [2023] identify systematic biases in LLM judges (position preference, verbosity bias, self-enhancement) but study response quality evaluation, not citation-specific rubric tasks.

“Zheng et al. [2023] identify systematic biases in LLM judges (position preference, verbosity bias, self-enhancement) but study response quality evaluation, not citation-specific rubric tasks.”

LLM-as-judges may be falsely lenient on open-ended tasks like slightly incorrect product descriptions.

“LLM-asjudges-while correlating well with human judgment for simple question-answer tasks [154]-may be falsely lenient on more openended tasks, like slightly incorrect product descriptions (e.g., Coke Zero versus Diet Coke) [122].”

LLM-as-a-judge has limitations such as self-preference bias, verbosity bias, and sensitivity to prompt phrasing.

“Despite its advantages, LLM-as-a-judge has limitations such as self-preference bias, where an LLM tends to favour its own responses; verbosity bias, where some LLMs may prefer longer, more detailed responses (Zheng et al., 2023;Stureborg et al., 2024); or sensitivity to prompt phrasing (Verga et al., 2024).”

Recent advancements have demonstrated high alignment between strong LLMs and human judgments across various NLP tasks.

“Recent advancements have demonstrated high alignment between”

GPT-4-Turbo exhibits inconsistency when judging emotional responses, whereas ChatGPT demonstrates more stable performance.

“For example, despite its advanced capabilities (Zheng et al., 2023), GPT-4-Turbo exhibits inconsistency when judging emotional responses, whereas ChatGPT demonstrates more stable performance.”

LLM evaluators may be biased toward longer and more fluent answers, even when those answers are not necessarily more accurate or pedagogically better.

“However, other studies show that LLM evaluators may be biased toward longer and more fluent answers, even when those answers are not necessarily more accurate or pedagogically better [35].”

Zheng et al. [2023] showed that careful rubric prompting can match human agreement rates above 80%.

“Zheng et al. [2023] showed that careful rubric prompting can match human agreement rates above 80%.”

Accuracy, agreement, win rate, F1, or κ are used for binary or pairwise verdicts.

“accuracy, agreement, win rate, F 1 , or κ are used for binary or pairwise verdicts (e.g., Zheng et al., 2023;Verga et al., 2024;Lee et al., 2026)”

Zheng et al. found that GPT-4 is significantly less prone to the repetitive list attack with below 10% success rate, while GPT-3.5 and Claude-v1 both suffer over 90% success rate.

“Their results show GPT-4 is significantly less prone to this attack with below 10% success rate, while GPT-3.5 and Claude-v1 both suffer over 90% success rate.”

The prompts used in the first experiment were taken from Zheng et al. (2023)'s library, all from the creative category, because answers generated to other categories did not vary in word count enough to see verbosity bias and GPT-4 was not good at judging answers in those categories.

“The prompts are taken from the library introduced by Zheng et al. (2023), all from the”

Providing rationales along with examples is more effective than incorporating Chain-of-Thought reasoning in the prompt.

“We also tried to incorporate Chain-of-Thought reasoning (Wei et al., 2022) in our prompt, but found that providing rationales along with the examples is more effective.”

Vicuna-7/13/33B is a collection of open-source chatbot aligned from LLaMA with user-shared conversations.

“Vicuna-7/13/33B [8, 59] is a collection of open-source chatbot aligned from LLaMA with user-shared conversations.”

We use blind rubric-based adjudication, inspired by LLM-as-a-judge work, to validate model-based outputs against human judgments.

“We use blind rubric-based adjudication, which is inspired by LLM-as-a-judge work that validates model-based outputs against human judgments [56], [57].”

Existing LLM benchmarks for general NLP tasks do not evaluate performance on relevant clinical tasks.

“Although LLM benchmarks for general natural language processing (NLP) tasks exist20,21, they do not evaluate performance on relevant clinical tasks.”

Studies have adopted powerful LLMs as surrogate judges, an approach known as LLM-as-judge.

“To reduce the cost of human evaluation, studies have adopted powerful LLMs as surrogate judges [338-339], an approach known as LLM-as-judge .”

Recent work has explored fine-tuning open-source LLMs as evaluators using human-annotated scoring data.

“Furthermore, recent work [338] has explored fine-tuning open-source LLMs as evaluators using human-annotated scoring data.”

When tasked with evaluating options presented simultaneously, LLMs can more reliably identify the superior response.

“Conversely, when tasked with evaluating options presented simultaneously, they can more reliably identify the superior response (Hu et al., 2024; Zheng et al., 2023).”

MT-Bench and related work demonstrate that structured LLM-as-judge protocols can provide scalable model comparison when prompt templates and score aggregation are controlled.

“MT-Bench and related work demonstrate that structured LLM-as-judge protocols can provide scalable model comparison when prompt templates and score aggregation are controlled [13].”

The utility of LLMs in the backward pass of academic verification or as verifiers remains underexplored.

“However, despite their widespread usage as 'generators' in the forward pass of scientific research, their utility in the backward pass of academic verification or as verifiers remains underexplored, a blind spot in which most systems lean on LLM judges [16] without validation on their credibility in reviewing scientific research.”

The dataset size aligns with recent trends toward compact, high-quality benchmarks.

“Although modest in size, our dataset aligns with recent trends toward compact, high-quality benchmarks: MT-Bench (80 items) [16], GPQA-D (198 items) [4], AIME 2024/2025 (30 items each) [35], USAMO 2025 (6 items) [36] and PaperBench (20 items) [37].”

Automatic judging LLMs have previously been proposed to act as judges to evaluate submissions for tasks.

“Automatic judging LLMs have previously been proposed to act as judges to evaluate submissions for tasks (Zheng et al., 2023; Chen et al., 2024; Fu et al., 2023).”

Studies on LLM-as-a-judge indicate that LLMs can imitate human behavior and give evaluations matching human judgments, providing a scalable and transparent alternative to human evaluation.

“Findings from these researches indicate that LLM can mimic human behavior and provide evaluations that correspond with human judgments, revealing a potentially scalable and transparent alternative to costly and laborious human evaluations.”

LLM-as-a-judge achieves over 80% agreement with expert human ratings.

“Zheng et al. (2023) demonstrated that strong LLMs achieve over 80% agreement with expert human ratings using MT-Bench and ChatBot Arena, positioning this method as a scalable and reliable alternative to manual annotation.”

LLM judges exhibit systematic biases that favor completions with certain superficial characteristics rather than reflecting genuine quality differences.

“LLM judges are shown to exhibit systematic biases that favor completions with certain superficial characteristics rather than reflecting genuine quality differences.”

Prior work that interprets higher self-scores as evidence of self-bias does not analyze the statistical assumptions required to infer the presence and magnitude of self-bias.

“Although this method may suggest self-bias, the authors do not dive into the statistical assumptions required in order to make a statistical inference on the presence and magnitude of the self-bias, e.g., how the two sets of scores are related.”

Strong LLM judges like GPT-4 can match both controlled and crowd-sourced human preferences well, achieving over 80% agreement, the same level of agreement as between humans.

“Recent research25 has shown that strong LLM judges like GPT-4 can match both controlled and crowd-sourced human preferences well, achieving over 80% agreement, which is the same level of agreement between humans”

Prior research has demonstrated that the input order of candidate responses can influence the evaluation outcome.

“Prior research on LLM-as-a-Judge systems [50] has demonstrated that the input order of candidate responses can influence the evaluation outcome.”

ChatGPT and GPT-4 have demonstrated remarkable performance across various tasks, leading to their widespread use as both annotators and evaluators.

“ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023) have recently demonstrated remarkable performance across various tasks, leading to their widespread use as both the annotators (Peng et al., 2023;Xu et al., 2023) and evaluators (Zheng et al., 2023;Peng et al., 2023;Sun et al., 2023;Zhou et al., 2023;Gao et al., 2023;Wang et al., 2023b;Dubois et al., 2023;Wang et al., 2023a).”

Pair-wise evaluation can give more accurate evaluation results than point-wise evaluation.

“Pair-wise evaluation can give more accurate evaluation results than point-wise evaluation”

Using an LLM as a judge can partly solve the difficulty of evaluating open-ended tasks, since an LLM-as-judge prompt can incorporate human preferences and adaptively score such tasks.

“By introducing LLM as a judge (Zheng et al., 2023), this challenge can be addressed to some extent, as the LLM-as-judge evaluation prompt can effectively incorporate human preferences to adaptively evaluate open-ended tasks.”

Our verifier audit and trajectory analysis use a model to adjudicate agent trajectories, following the LLM-as-a-judge paradigm.

“Our verifier audit and trajectory analysis use a model to adjudicate agent trajectories, following the LLM-as-a-judge paradigm introduced by Zheng et al. [2023].”

Recent studies have reported strong correlations between LLM evaluators and human evaluations.

“Recent studies have explored the effectiveness of LLMs as evaluators and have reported strong correlations with human evaluations (Dubois et al., 2023;Zheng et al., 2023).”

For all the strategies, evaluation prompts P eval are adapted from Zheng et al. (2023); Zeng et al. (2023); Hada et al. (2023).

“For all the strategies evaluation prompts P eval are adapted from Zheng et al. (2023); Zeng et al. (2023); Hada et al. (2023).”

Many prior studies have explored LLM-as-a-Judge in text-only settings, analyzing agreement with human preferences and identifying sources of bias in LLM-based evaluation.

“Many prior studies have explored this paradigm in text-only settings, analyzing agreement with human preferences and identifying sources of bias in LLM-based evaluation (Zheng et al., 2023; Chiang et al., 2024).”

Minimum viable alternatives (BERTScore, LLM-as-judge, entailment-based verification) are available but not yet standard practice in benchmarks and leaderboards.

“Minimum viable alternatives , BERTScore, LLM-as-judge, entailment-based verification , are available [16,17] , but are not yet standard practice in the benchmarks and leaderboards that drive architectural decisions.”

Zheng et al. (2023) identify position bias, where models favor responses that appear earlier in the prompt, and verbosity bias, where longer responses are rated higher regardless of quality.

“Zheng et al. (2023) identify position bias, where models favor responses that appear earlier in the prompt, and verbosity bias, where longer responses are rated higher regardless of quality.”

Recent work shows that LLMs, particularly GPT-4, are effective as evaluators.

“Drawing from recent work showing the effectiveness of LLMs as evaluators, particularly GPT-4 (Zheng et al., 2023; Zeng et al., 2024), we adopt an LLM-as-a-Judge evaluation protocol for REALINSTRUCT.”

GPT-4's relatively high constraint-level accuracy could indicate scoring bias.

“However, GPT-4's relatively high constraint-level accuracy could indicate scoring bias, as previous studies suggest GPT-4 tends to favor its own outputs (Zheng et al., 2023; Panickssery et al., 2024; Verga et al., 2024).”

GPT-4 tends to favor its own outputs.

“as previous studies suggest GPT-4 tends to favor its own outputs (Zheng et al., 2023; Panickssery et al., 2024; Verga et al., 2024).”

The authors adapted the prompt proposed by Zheng et al. (2023) for the OQA.

“We adapt the prompt proposed by Zheng et al. (2023), converting it to the Prometheus format, and introduce the option for tied responses, a feature not originally supported by Prometheus' pairwise ranking system.”

These systems hold their evaluation criterion fixed outside the loop, exposing them to benchmark saturation, judge biases, and reward hacking.

“These systems hold their evaluation criterion fixed outside the loop, exposing them to benchmark saturation [12], judge biases [15, 16], and reward hacking [17].”

GPT-4 is used as a proxy of the user to judge whether the system successfully handles the query.

“For model evaluation, we leverage GPT-4 as a proxy of the user to judge whether the system successfully handles the query (Zheng et al., 2023; Fu et al., 2023; Wang et al., 2023b).”

The LLM-as-a-Judge approach leverages the reasoning capabilities of LLMs to evaluate agent responses based on qualitative criteria provided through instructions.

“The LLM-as-a-Judge approach [125] leverages the reasoning capabilities of LLMs to evaluate agent responses based on qualitative criteria, assessing responses according to the criteria provided through instructions.”

The use of LLMs as automated evaluators has gained significant attention as an alternative to costly human annotation.

“The use of LLMs as automated evaluators has gained significant attention as an alternative to costly human annotation (Zheng et al., 2023).”

LLM evaluators favor more detailed and lengthy answers.

“LLM evaluators have been reported to favor more detailed and lengthy answers (Zheng et al., 2023).”

LLM judges exhibit strong biases.

“Similar to Zheng et al. (2023), we found strong biases with LLM judges.”

Language models can be influenced by the ordering of the responses in their evaluations.

“Order bias has been extensively studied (Jung et al., 2019;Wang et al., 2023a;Zheng et al., 2023), and it is well-known that language models can be influenced by the ordering of the responses in their evaluations.”

LLM judgments can align substantially with human preferences on open-ended generation tasks.

“Rubric-conditioned scoring in G-Eval and pairwise comparison in MT-Bench demonstrated that LLM judgments can align substantially with human preferences on open-ended generation tasks(Liu et al., 2023;Zheng et al., 2023).”

Prometheus-2 is an LLM-as-a-judge model fine-tuned on feedback and judgment data generated by GPT-4.

“The Prometheus-2 model is an LLM-as-a-judge model (Zheng et al., 2023) fine-tuned on feedback and judgment data generated by GPT-4 on a large set of custom score rubrics.”

LLM-as-a-Judge establishes strong correlations with human judgement on general tasks while documenting systematic biases such as position, verbosity, and self-preference effects.

“establishing strong correlations with human judgement on general tasks [Zheng et al., 2023, Kim et al., 2023, 2024] while documenting systematic biases such as position, verbosity, and self-preference effects [Ye et al., 2024, Wataoka et al., 2024, Dubois et al., 2024, Kim et al., 2025, Ye et al., 2025].”

We use the LLM-as-Judge evaluation protocol to assess answer correctness.

“We use the LLM-asJudge evaluation protocol [Zheng et al., 2023] to assess answer correctness.”

LLM ratings of text quality track human ratings.

“a growing literature finds that LLM ratings of text quality track human ratings (Chiang and Lee, 2023; Liu et al., 2023; Zheng et al., 2023)”

The complexity of agentic search answers is far beyond what conventional LLM-as-a-Judge methods are used for.

“The complexity is far beyond what conventional LLM-as-a-Judge [45] methods are used for.”

Good-performing LLMs tend to perform better in LLM-as-a-judge tasks.

“As good-performing LLMs tend to perform better in LLM-as-a-judge tasks (Zheng et al., 2023)”

We run each pair twice, reversing the ordering of the references to account for positional bias.

“Following Zheng et al. (2023), we run each pair twice, reversing the ordering of the references so as to account for any positional bias the judge model may have (e.g., always favoring the first model's response).”

GPT-4 accurately simulates human preferences in both summarisation and instruction following.

“Previous work has shown that GPT-4 accurately simulates human preferences in both summarisation (Rafailov et al., 2023) and instruction following (Dubois et al., 2023;Zheng et al., 2023), so we use it as the main performance metric in both tasks.”

LLM-as-a-judge has shown strong correlation with human judgments across tasks.

“This method has shown strong correla-tion with human judgments across tasks.”

Some weak LLM judges on Chatbot Arena exhibited positional bias, consistently preferring the first response.

“We observed positional bias in some weak LLM judges on Chatbot Arena where the first response was consistently preferred, similar to (Zheng et al., 2023).”

MT-Bench and Chatbot Arena assess open-domain, multi-turn dialogue abilities, but their focus remains distinct from our objective.

“These benchmarks evaluate capabilities in writing, reasoning, extraction, and domain knowledge (e.g., STEM and humanities/social sciences), but their focus remains distinct from our objective.”

LLM-as-a-judge systems offer a scalable alternative to expert annotation by substantially reducing cost and latency.

“LLM-as-a-judge systems offer a scalable alternative to expert annotation by substantially reducing cost and latency (Zheng et al., 2023).”

Zheng et al. [2023] showed that GPT-4 judgments align with human preferences at scale.

“Zheng et al. [2023] introduced MT-Bench and Chatbot Arena, showing that GPT-4 judgments align with human preferences at scale.”

GPT-4 has high human alignment and agrees with humans on over 80% of evaluations.

“Zheng et al. (2023) evaluate LLMs with other LLMs as a judge and show that GPT-4 has a high human alignment and agrees with humans on over 80% of evaluations.”

Previous studies on verbosity bias have focused on specific cases: Zheng et al. limited their problem setting to questions answered with lists, and Huang et al. conducted experiments on summarization tasks.

“While previous studies have explored the concept of verbosity bias, they have tended to focus on specific cases.Zheng et al. (2023) limits their problem setting to questions answered with lists in their experiment on verbosity bias, and Huang et al. (2023) conducted experiments on summarization tasks.”

Zheng et al. (2023) experimented on verbosity bias by attempting a repetitive list attack on several LLMs.

“Alongside their experiment on position bias, they experimented on verbosity bias by attempting a”

There is work demonstrating bias in using LLMs as evaluators.

“there is also work demonstrating bias in using LLMs as evaluators (Wang et al., 2024;Zheng et al., 2023).”

Static benchmarks with clear ground-truth do not capture flexible and interactive tasks needed to navigate scenarios in the real-world.

“Static benchmarks with clear ground-truth MMLU [29], GSM8k [11], and others like it do not capture flexible and interactive tasks needed to navigate scenarios in the real-world”

Many existing datasets contain completions generated by relatively weak LLMs.

“Since many of the existing datasets contain completions generated by relatively weak LLMs [Zhang et al., 2024, Zheng et al., 2023], we collect new model completions and corresponding LLM-as-a-judge judgments, which we will publicly release along with the publication.”

There is a positive correlation between the valid rate and the Arena rating of different LLM agents.

“Generally, we observe a positive correlation between the valid rate and the Arena rating of different LLM agents.”

Opportunities to practice consultation skills and receive constructive assessment remain limited in clinical settings.

“However, opportunities to practice these skills and receive constructive assessment remain limited in clinical settings.”

Judges were drawn from three vendors so that judge-family effects were measurable, and all judges ran as non-reasoning models.

“Judges were drawn from three vendors so that judge-family effects were measurable (Zheng et al., 2023;Panickssery et al., 2024), and all judges ran as nonreasoning models (Appendix E).”

The LLM-as-Judge paradigm was established by Zheng et al. (2023), demonstrating that strong models such as GPT-4 can serve as reliable proxies for human annotators.

“The LLM-as-Judge paradigm was established by Zheng et al. (2023) (MT-Bench, Chatbot Arena), demonstrating that strong models such as GPT-4 can serve as reliable proxies for human annotators.”

Per-judge calibration on a labeled validation slice removes judge bias at the source.

“A separate line of work removes judge bias at the source via per-judge calibration on a labeled validation slice (Zheng et al., 2023).”