PreprintCited by 2,066Artificial Intelligence in LawTopic ModelingNatural Language Processing Techniques

The DOI matches its DataCite record; 58 of the 58 references it cites were found in a registry or an index; 4 of the 29 sources its claims cite were checked for support.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Open access

Zheng, Lianmin, Chiang, Wei-Lin, Sheng, Ying, Zhuang, Siyuan, Wu, Zhanghao, Zhuang, Yonghao, Lin, Zi, Li, Zhuohan, Li, Dacheng, Xing, Eric P., Zhang, Hao, Gonzalez, Joseph E., Stoica, Ion·2023·arXiv

DOI 10.48550/arxiv.2306.05685arXiv 2306.05685

Open this paper in CoChat — free account Open PDF

Abstract

Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper examines whether strong LLMs can serve as judges for chatbot responses by comparing their judgments with human preferences on two new benchmarks, MT-bench and Chatbot Arena. It finds that GPT-4 can reach over 80% agreement with human preferences, comparable to human-human agreement, while also documenting several judge biases and mitigation strategies.

Key findings

GPT-4’s judgments aligned closely with human experts on MT-bench: 85% agreement without ties, versus 81% agreement among humans; on Chatbot Arena, GPT-4 also showed high agreement with crowd votes, above 80%.
Position bias was substantial: with the default prompt, consistency was 65.0% for GPT-4, 46.2% for GPT-3.5, and 23.8% for Claude-v1; swapping positions and few-shot examples raised GPT-4 consistency to 77.5%.
Verbosity bias appeared in a “repetitive list” attack: Claude-v1 and GPT-3.5 failed 91.3% of cases, while GPT-4 failed 8.7%.
Reference-guided grading reduced math-judging failures from 14/20 under the default prompt to 3/20, with chain-of-thought at 6/20; self-enhancement was suggested by GPT-4 favoring itself with a 10% higher win rate and Claude-v1 with a 25% higher win rate, but the study could not confirm the bias.

Methods

The authors created MT-bench, 80 multi-turn questions across 8 categories with 10 questions each, and Chatbot Arena, a crowdsourced anonymous battle platform; they collected about 3K expert votes from 58 labelers on MT-bench and about 30K Arena votes, sampling 3K for analysis.
They compared LLM judges—mainly GPT-4, GPT-3.5, and Claude-v1—using pairwise comparison, single-answer grading, and reference-guided grading against human expert and crowd votes, measuring agreement and win rates.
They also evaluated LLaMA and Vicuna variants on MMLU, TruthfulQA, and MT-bench with GPT-4 single-answer grading to test how preference-based and capability-based benchmarks complement each other.

Limitations

The paper states that it emphasizes helpfulness and largely neglects safety, honesty, and harmlessness; it also notes that helpfulness dimensions such as accuracy, relevance, and creativity are combined into one metric, and its bias mitigations are preliminary.
The paper reports that self-enhancement bias could not be determined because of limited data and small differences, and that its position-bias test is difficult because the compared answers are very similar and sometimes indistinguishable even to humans.
The text makes evident that few-shot judging may introduce new biases and makes API calls about 4× more expensive, while zero-shot Vicuna-13B as a judge had low consistency (11.2–16.2%) and high error rates (22.5–78.8%), limiting its use as a scalable open-source judge.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.05685

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

56 Own result · 35 Own interpretation · 7 Sources not found · 7 No basis given · 3 Partly supported · 2 Own method · 1 Not checked yet · accuracy not yet measured

The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Once aligned with humans, these chat models are strongly preferred by human users over the original, unaligned models on which they are built.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites [31] source
- Partly supported
  Section 4.5 (p. 21) says crowdworkers clearly preferred the online RLHF models in comparisons with context-distilled and earlier RLHF models. But those preference comparisons do not establish that users strongly preferred them over the original, unaligned base models; the paper discusses base-model improvements on other evaluations (Introduction, p. 4). (LLM, conf=0.86)
  cites [2] sourceExperimental — accuracy not yet measured
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [30]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [8]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [52]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [48]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [14]
boosted
direct
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We show that some of the biases are minor or can be mitigated.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that in math/coding category, GPT-3.5 and GPT-4 have similar overall win-rate because they both failed to answer some hard questions, but GPT-4 is still significantly better than GPT-3 in the direct pairwise comparison or single-answer grading.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.3 Win rates under different judges
plain
correlational
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, a small high-quality conversation dataset can quickly teach the model a style preferred by GPT-4 (or approximately human) but cannot improve MMLU significantly, as shown by the Vicuna-7B (selected) which is trained with only 4.8M tokens or 3K conversations.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
direct
Single case
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most LLM judges favor the first position.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only GPT-4 successfully detected this attack.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plain
Single case
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As in Table 2, we found all of them exhibit strong position bias.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, it has a high error rate (from 22.5% to 78.8%) because of its weaker instruction-following capability.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plain
direct
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Claude-v1 favors itself with a 25% higher win rate.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
direct
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 12 (Appendix), the few-shot judge can significantly increase the consistency of GPT-4 from 65.0% to 77.5%.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
direct
n = 3

Show all 65 claims

Probably not a substantial claim. 22
It may state two separate claims as one. 29
The quote only partly states this claim. 2

The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As LLMs continue to improve, they show potential in replacing human annotators in many tasks [17,20].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3 LLM as a Judge
What the sources say
- Supported
  The Discussion (p. 4) says LLMs could transform text annotation across a variety of tasks and that ChatGPT may already outperform crowd annotators. The Introduction (p. 2) also describes the possibility of annotating entire samples with ChatGPT. These passages support the hedged claim about potential replacement in annotation tasks, particularly those performed by crowd-workers. (LLM, conf=0.88)
  cites [17] source
- Partly supported
  The Discussion and Conclusion (Section 5; page unavailable) says ChatGPT showed great potential as a data annotation tool for implicit hateful tweets. It does not establish that LLMs could replace human annotators across many tasks; it also urges caution about incorrect decisions. (LLM, conf=0.86)
  cites [20] sourceExperimental — accuracy not yet measured
hedged
conditional
Probably not a substantial claim.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“DynaBench addresses the challenges posed by static standardized benchmarks, such as saturation and overfitting, by emphasizing dynamic data with human-in-the-loop.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
What the sources say
- Partly supported
  The Introduction (p. 1) presents Dynabench as dynamic data collection with humans and models in the loop, in contrast to static benchmarking. Caveats and Objections (p. 8) explicitly says it anticipates dataset saturation. The copy does not identify overfitting as a challenge of static benchmarks; its discussion of annotators ‘overfitting’ on models (p. 7) describes a potential risk of the dynamic approach instead. (LLM, conf=0.84)
  cites [21] sourceExperimental — accuracy not yet measured
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite the base LLaMA models showing competitive performance on conventional benchmarks (Table 8), its answers to open-ended questions are often not preferred by humans.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.”
- Also stated in: § GPT-4 Judgment: · plain
  “Once addressed, our results from 3K controlled expert votes and 3K crowdsourced human votes in the wild verify that GPT-4 judge match human evaluations at an agreement rate exceeding 80%, achieving the same level of human-human agreement ( §4.2, Table 4).”
- Also stated in: § 4.2 High agreement between GPT-4 and humans · plain
  “The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Abstract
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The consistency improves significantly from 16.2% to 65.0%.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plain
direct
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves 56.8% when including all three labels, and 85.5% when excluding tie predictions and labels, significantly outperforming random guesses of 33% and 50% respectively, and show positive signals to match GPT-4 (66% and 87% respectively).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plain
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Despite GPT-4 knows how to solve the question (if asked separately), it made a wrong judgement saying both assistants' wrong answers are correct.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 14:
plain
Single case
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to the classification formulation, every output is recognizable (error rate 0%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plain
direct
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 15, Vicuna-13B has a consistency rate from 11.2% to 16.2% across different prompt templates, much lower than all the closed-sourced models.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in Figure 13 (Appendix), we present an example of an elementary math question in which GPT-4 makes an incorrect judgment.It's worth noting that although GPT-4 can solve the problem (when asked separately), it was misled by the provided answers, ultimately resulting in incorrect judgment.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
direct
Single case
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 3 shows the failure rate of LLM judges under this attack, demonstrating that all LLMs may be prone to verbosity bias though GPT-4 defends significantly better than others.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
hedged
conditional
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 11, the position bias is more noticeable for models with close performance and can almost disappear when the performance of the two models differs a lot.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > D.1 Position bias
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results reveal that strong LLMs can achieve an agreement rate of over 80%, on par with the level of agreement among human experts, establishing a foundation for an LLM-based evaluation framework.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 7 Conclusion
plain
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Comparing GPT-4 and other LLM judges, we find they reach a similar non-tie agreement ratio between humans but the number of non-tied votes from GPT-4 is much larger.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plain
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When Assistant A is placed in the first position, GPT-4 thinks A is better, but its verdict changes when we swap the position of A and B.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 11: An example of position bias.
plain
direct
Single case
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that fine-tuning on high-quality dialog datasets (i.e., ShareGPT) can consistently improve the model performance on MMLU and the improvement scales with fine-tuning data size.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
direct
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both GPT-3.5 and Claude-v1 show a verbosity bias towards the longer and repetitive answer.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plain
direct
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both tables, GPT-4 with single-answer grading matches both pairwise GPT-4 and human preferences very well.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
boosted
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On math and coding questions, LLM judges are more confident even though their judgments can often be wrong, as we show in Section 3.3.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > D.1 Position bias
plain
correlational
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Besides, the longer prompts make API calls 4× more expensive.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
direct
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, even with the CoT prompt, we find that in many cases LLM makes exactly the same mistake as the given answers in its problem-solving process (See example in Figure 15 (Appendix), suggesting that LLM judge may still be misled by the context.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Chain-of-thought and reference-guided judge.
hedged
conditional
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe similar pattern from other LLM judges such as Claude/GPT-3.5.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 11: An example of position bias.
plain
correlational
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, the pairwise comparison may lack scalability when the number of players increases, given that the number of possible pairs grows quadratically;”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.1 Types of LLM-as-a-Judge
hedged
conditional
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“However, the heightened user preference does not always correspond to improved scores on traditional LLM benchmarks -benchmarks like MMLU [19] and HELM [24] cannot effectively tell the difference between these aligned models and the base models.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 1 Introduction
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [19]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [24]
plain
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This misalignment of conventional benchmarks underscores the core problem driving this paper: the need for a robust and scalable automated method to evaluate LLM alignment with human preferences.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plain
direct
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“single answer grading may be unable to discern subtle differences between specific pairs, and its results may become unstable, as absolute scores are likely to fluctuate more than relative pairwise results if the judge model changes.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.1 Types of LLM-as-a-Judge
hedged
conditional
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Conversational benchmarks, like CoQA [35], MMDialog [15] and OpenAssistant [23], are closest to our intended use cases.However, the diversity and complexity of their questions often fall short in challenging the capabilities of the latest chatbots.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 2.1 Motivation
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [35]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [15]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [23]
plain
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, high consistency may not imply high accuracy and we are not sure whether the few-shot examples will introduce new biases.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
hedged
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means that GPT-4 is more affirmative and less suffered from position bias but other models also perform well when they give an affirmative answer.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plain
correlational
It may state two separate claims as one.
In the paper
“LLMs are known to have limited math and reasoning capability [10], which results in its failure of grading such questions because they do not know the correct answers.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [10]
boosted
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They focus on different aspects of a model and the recommended way is to comprehensively evaluate models with both kinds of benchmarks.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
correlational
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To invite participants, we obtained their consent by letting them sign an application form.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > C.1 MT-bench human evaluation
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another more aggressive approach is to assign positions randomly, which can be effective at a large scale with the correct expectations.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
conditional
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Table 8, no single benchmark can determine model quality, meaning that a comprehensive evaluation is needed.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This bias is not unique to our context and has been seen in human decision-making [3,34] and other ML domains [22,41].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [3]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [34]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [22]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [41]
plain
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Chain-of-thought is a widely used technique to improve LLM's reasoning capability [47].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Chain-of-thought and reference-guided judge.
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [47]
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our LLM-as-a-judge approach can automate and scale platforms of this nature.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
direct
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results indicate that using LLM-as-a-judge to approximate human preferences is highly feasible and could become a new standard in future benchmarks.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
hedged
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Users are required to accept the terms of use, which obtain their consent and give us the right to release the conversation data.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > C.2 Chatbot Arena
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A conservative approach is to call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
direct
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These responses are rendered as natural languages and are difficult to be parsed automatically, making the model less useful in a scalable and automatic evaluation pipeline.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plain
direct
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLM-as-a-judge offers two key benefits: scalability and explainability.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.2 Advantages of LLM-as-a-Judge
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that this test is challenging because the answers are very similar and occasionally indistinguishable even to humans.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The single-answer grading can be converted into pairwise comparison results for calculating the agreement.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plain
direct
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Honesty and harmlessness are crucial for a chat assistant as well [2].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 6 Discussion
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [2]
plain
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.

46 of 46

The paper relies on this without support.
LLMs trained with RLHF already exhibit strong human alignment.
“Because these models are often trained with RLHF, they already exhibit strong human alignment.”
The paper relies on this without support.
There has not been a systematic study of the LLM-as-a-judge approach.
“However, there has not been a systematic study of this approach.”
The paper relies on this without support.
Human preferences serve as a direct measure of a chatbot's utility in open-ended, multi-turn human-AI interactions.
“While largely overlooked by existing LLM benchmarks, human preferences serve as a direct measure of a chatbot's utility in open-ended, multi-turn human-AI interactions.”
The paper relies on this without support.
The human labelers are mostly graduate students and are considered experts and more skilled than average crowd workers.
“The labelers are mostly graduate students so they are considered experts and more skilled than average crowd workers.”
The paper relies on this without support.
Human evaluation is the gold standard for assessing human preferences.
“While human evaluation is the gold standard for assessing human preferences, it is exceptionally slow and costly.”
The paper relies on this without support.
Model services such as GPT-4 can become expensive with a growing number of evaluations.
“However, model services such as GPT-4 can also become expensive with a growing number of evaluations.”
The paper relies on this without support.
Popular open-sourced LLMs, e.g. Vicuna-13B, show strong language understanding capability and are much cheaper than close-sourced LLMs.
“On the other hand, popular open-sourced LLMs, e.g.Vicuna-13B shows strong language understanding capability, and are much cheaper than close-sourced LLMs.”

7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Traditional evaluation metrics based on the similarity between outputs and reference answers (e.g., ROUGE [25], BLEU [32]) are also ineffective for these questions.”
Method
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 6×
“5, Claude-V1, Vicuna-13B, Alpaca-13B [38], and LLaMA-13B [39].”
BackgroundInfluential
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1×
“Chain-of-thought is a widely used technique to improve LLM’s reasoning capability [39].”
Method
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Chain-of-thought is a widely used technique to improve LLM’s reasoning capability [39].”
Method
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832 · cited 1×
“This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
Background
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“There has been a proliferation of LLM-based chat assistants (chatbots) that leverage supervised instruction fine-tuning and reinforcement learning with human feedback (RLHF) to unlock new instruction following and conversational abilities [27, 2, 26, 7, 43].”
Background
HellaSwag: Can a Machine Really Finish Your Sentence?Preprint
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07830 · Cited by 1,693 · cited 2×
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
Method
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 2×
“This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
Background
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 2×
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
Method
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243

Show all 2 references

Research works printed but not linked: 2

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and Ion Stoica. SkyPilot: An intercloud broker for sky computing. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 437-455, Boston, MA, April 2023. USENIX Association.

Sorted by citations

Items 1,201–1,300 of 2,066

Token-Anchored Semantic Drift: Untargeted Transferable Attacks Against Commercial MLLMs
Shen, T. (2026) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519348
EVA: Editing for Versatile Alignment Against Jailbreaks
Wang, Y., Qiu, H., Xu, Y., Yang, S., Qin, Z., Huang, M., & Wang, W. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3694189 · Cited by 1
“5 [59], Mistral-7B-Instruct-v0.”
The Crimson Hexagonal Archive Hugging Face Dataset: Work Plan v3 (Classifier-Centric Methodology)
Sharks, L. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20313253
Towards Data-Driven VR Environment Recommendation for Residents with Major Neurocognitive Conditions: A Feasibility and Proof-of-Concept Study (Preprint)Preprint
Reyes-Consuelo, M. A., Kiss, J., & Voisin, J. I. A. (2026) · doi:10.2196/preprints.101192
Hybrid Retrieval Augmented Generation: Reliable and Accurate Question Answering
Dalali, Y., Zeguendry, A., & Aaroui, A. (2026) · International Conference on Global Aeronautical Engineering and Satellite Technology (GAST) · doi:10.1109/gast67799.2026.11523296
SigmaForge: Retrieval-Enhanced Sigma Rule Generation with DAG-Based Quality Evaluation
Diaz-Santos, A., & Socha, K. (2026) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-032-23335-6_4
Small Language Model Adaptation to the Italian Agricultural Legal Domain
Pelosi, M., Ghosh, R. C., Cirillo, E., et al. (2026) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-032-23335-6_19
The Crimson Hexagonal Archive Hugging Face Dataset: Work Plan v3 (Classifier-Centric Methodology)
Sharks, L. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.20313252
Does Safety Molt? Evaluating LLM Safety in Multi-Agent Social Environments
Priyanshu, A., Vijay, S., & Pahwa, E. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813173 · Cited by 1
Workflow Topology–Guided Skill Loading for Agentic AI in Clinical Trial Data Pipelines: Sequential Exploratory Evaluation
Yan, J. (2026) · JMIR Formative Research · doi:10.2196/preprints.100769
A Multi-Agent Cognitive Isolation Framework for Defect-Driven Power Grid Document Assessment
Yang, J., Ao, Z., Ren, H., Zhu, H., Ding, L., Wang, M., Jiang, X., & Yang, Q. (2026) · Asia Conference on Power and Electrical Engineering (ACPEE) · doi:10.1109/acpee69242.2026.11524710
“With the advent of Large Language Models (LLMs), utilizing models for few-shot scoring has become an emerging research hotspot [1]–[3].”
What Makes a Good Response? Learning Personal Preferences from Interpretable Features
Afsharrad, A., Soroka, E., O’Neill, D., & Lall, S. (2026) · ICON · doi:10.1109/icnlp69856.2026.11527497
The action-gating test: a behavioral diagnostic for performative versus genuine ethical reasoning in large language models
Baxi, P., & Baxi, R. (2026) · AI and Ethics · doi:10.1007/s43681-026-01175-3
A large language model-driven scientific literature surveys generation framework based on multi-agents and retrieval-augmented generation
Qi, R., Lyu, H., Li, W., Chen, S., & Guo, X. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3882
A Policy-Driven Runtime Layer for Agentic LLM Serving
Zhang, R., Kim, C.-E., Ching, C.-W., Zhong, Y., & Hu, L. (2026) · ACM SIGOPS Asia-Pacific Workshop on Systems (APSys) · doi:10.1145/3838177.3841725 · Cited by 3
Legal Zero-Days: a novel risk vector for advanced AI systems
Sadler, G., & Sherburn, N. (2026) · AI and Ethics · doi:10.1007/s43681-026-01189-x
Mining Subscenario Refactoring Opportunities in Behaviour-Driven Software Test Suites: ML Classifiers and LLM-Judge BaselinesPreprint
Mughal, A. H., Fatima, N., & Bilal, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6793726
Encoding Domain Expertise in Agents: Lessons from NFL Fantasy AI
Butler, M., Wang, H., Lee, J., Blacutt, K., Volk, D., Band, M., & Socolinsky, D. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-27272-0_31
From Labels to Context: Context-Sensitive User Modeling of Affect with Large Language Models
Nalis, I., Klüwer, N., & Neidhardt, J. (2026) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3774935.3812706
DIVA: Diverse Video Generation with Agents
Kazimi, T., Yu, H., & Lao, Z. (2026) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai68641.2026.11536189
DUPLet: Dynamic User Modeling for Personalized Cover Letter Generation
Bovie, T., Saini, B. K., Kumar, C., et al. (2026) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3774935.3812726
Lucid: A high-performance local LLM platform for operational efficiency
Kaplan, T., & Kalkan, E. N. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.04.103
Mitigating Hallucination in Multimodal Information Systems: A Comparative Analysis of Modular LLM Architectures
Chawla, R., Dahiya, S., Singh, G., & Fortino, G. (2026) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2026.3691181
“Our primary scoring mechanism uses GPT-4 as the LLM-as-a-Judge, a framework that has demonstrated strong alignment with human evaluations in prior research [25], [28].”
Making Sense of Sensors: Improving LLM Interpretation of Time-Series Data
Rico, A., & Larson, K. (2026) · AI Sensors · doi:10.3390/aisens2020007
Machine Learning Evaluation of Generative AI Debugging Feedback Plausibility for Introductory Python Programming
Toscano, B., Bouzar-Benlabiod, L., & Benoit, D. G. (2026) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri69576.2026.00028
“LLM-as-judge systems such as G-Eval and MT-Bench/Chatbot Arena show why this approach is attractive [11], [12], but Chehbouni et al. [5] argue that the validity and reliability of such protocols have not been scrutinized in proportion to their adoption.”
Repowise: A Hybrid RAG-Based Conversational System for Software Repository Analysis
Kocadere, M. O., Cömert, Y. E., Yazıcı, S., Karaçay, Ş. G., Yıldız, İ. B., & Erdoğan, T. G. (2026) · International Congress on Human-Computer Interaction, Optimization and Robotic Applications (ICHORA) · doi:10.1109/ichora69329.2026.11537220
Extending QuantumBench: A Selective Symbolic Enhancement and Multi-Perspective Evaluation Framework for Solving Quantum ProblemsPreprint
Luo, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-9663460/v1
Beyond patent citations: a benchmark for detecting implicit science–technology linkages using open large language modelsPreprint
Darold, D. L. (2026) · Research Square · doi:10.21203/rs.3.rs-9589637/v1
GE-LLM: Graph-Enhanced Large Language Models for Efficient Transistor-Level Circuit Simulation
Wang, C., Niu, D., Dong, Y., et al. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539692
Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systems
Hou, Q., Park, S., Zecchin, M., Cai, Y., Yu, G., Simeone, O., & Melodia, T. (2026) · IEEE Transactions on Communications · doi:10.1109/tcomm.2026.3700473
“Q: [question] [19], [34], [35].”
Diverse and Task-Specific Data Selection for Instruction Tuning
Diao, J., Guan, S., Zhu, G., Guo, J., & Cheng, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1462-4_33
Quantifying Evidence for Competing Biomedical Hypotheses using Large Language Models and Bayesian AnalysisPreprint
Moore, B. M., Freeman, J., Millikin, R. J., Mohanty, C., George, K. W., Bal, A., Lock, C., Sauer, J., Spurgeon, M. E., Moore, D. L., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.05.730173
DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following
Basta, N., & Kaafar, D. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1468-6_13
Multi-Agent LLM Framework for PRECEDE-PROCEED Aligned Behavior Change Intervention Design
Lee, J. S., & Oh, H. (2026) · ACM Symposium on Applied Computing · doi:10.1145/3748522.3779982
Large Language Models: Education and Learning Tool for Young Generation
Chavan, V. V., & Redekar, S. S. (2026) · International Conference on Signal, Systems, and Computing for Next-Gen Automation (ICSSCNA) · doi:10.1109/icsscna68616.2026.11546931
“Zheng et al. demonstrated that LLMs can serve as reliable judges for evaluating output quality, validating the LLM-as-Judge paradigm adopted in this work [21].”
GenAI-Supported strategy development in VUCA scenarios and Multi-GenAI evaluation
Yurdasever, E., & Yıldıran, C. (2026) · Management · doi:10.58691/man/221173
Evaluating and Enhancing an AI Mental Health Guide in High-Risk Mental Health ContextsPreprint
Swirbul, M. S., Peake, E. E., McDonald, P., Rodio, M. J., Padmanabhan, A., & Ben‐Zion, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-9588991/v1
A Controlled Study of Decoding-Time Truthfulness Methods on Instruction-Tuned LLMs
Sun, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38410-2_15
University Websites Going West: Algorithmic Gentrification and the Identity Tax on Interface Design Evaluation
Chen, W. (2026) · Designing Interactive Systems Conference · doi:10.1145/3802974.3809439
An Empirical Comparison of Dense, Hybrid, and Reranked Retrieval under Monolingual and Multilingual Embeddings for Indic Languages
Chedambath, A., Shinde, N., Bohra, R. P., & Babu, S. (2026) · International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV) · doi:10.1109/icicv68925.2026.11554809
“The system used the Gemini 2.0 Flash foundation model [5] as an LLM-as-a-judge [6].”
Influential
An Empirical Study of Version-Control and Validation Workflows for Enterprise AI Systems
Choudhary, A., Vatsa, A., Puppala, R., & Rao, A. (2026) · Wireless and Optical Communications Conference · doi:10.1109/wocc69802.2026.11556187
“LLM-as-judge evaluation, in which a more capable model scores a target model’s responses against structured criteria, has emerged as a standard technique for this purpose [8], [9].”
Influential
AssembleIt: Generating Adaptive On-Demand 3D Animations for Context-Aware Mechanical Assembly Guidance
Patel, M., Jain, R., Unmesh, A., Hsia, S.-K., & Karthik Ramani, A. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3812912
Identity-cued scoring asymmetry in LLM verification of realist CMO extraction: a within-rater noise-floor calibration study (Preprint)Preprint
Søfteland, J. M., Kvernebo, T. S., & Bhatti-Søfteland, M. (2026) · doi:10.2196/preprints.103608
EC-Distill-ZeroDiag: a cloud-edge collaborative framework for zero-shot industrial fault diagnosis via large language model distillation
Yao, Z., Xu, Y., Xia, T., Chen, Z., Pan, E., & Xi, L. (2026) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-026-02897-1
JARVIS for HVAC: LLM-based Question-Answer Framework for Sensor-driven HVAC System Interaction
Lee, S., Lee, J., Kang, M., Lee, S., Kim, D., Hong, J., Shin, J., Zhang, P., & Ko, J. (2026) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3810210 · Cited by 8
Ivy-Fake: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection
Jiang, C., Dong, W., Zhang, Z., Yu, F., Peng, W., Yuan, X., Bi, Y., Zhao, M., Zhou, Z., Si, C.-W., et al. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810615 · Cited by 4
A Review of a Hybrid Evaluation Framework for Clinical AI: Integrating Interrater Reliability with the "LLM as a Judge" MethodologyPreprint
Limaye, D., Shah, R., & Jonelgadda, C. (2026) · Research Square · doi:10.21203/rs.3.rs-10047888/v1
Prompt Perturbation for reliable LLM evaluation over comparison graphs
Huang, D., Sun, J., & Yang, P. (2026) · Statistical Learning and Data Science · doi:10.1016/j.slads.2026.100022
Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA
Shen, Y., Zhang, H., & Unberath, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38233-7_60
Self-citation
Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition
Choi, J. Y., Kim, S. G., Yoon, H., Lee, T., Lee, D., Chung, J., Kil, J., Rossi, R., Lee, S.-J., & Lee, T. Y. (2026) · IEEE Pacific Visualization Conference (PacificVis) · doi:10.1109/pacificvis68791.2026.00023
“To mitigate ordering bias in language models [40], we randomized the order of activity descriptions and example activities for every query.”
Potential and limitations of LLMs for analyzing narratives on corporate profits
Bonhoure, É., Bordea, G., Doucet, A., Fabre, A., Hamdi, A., Hendieh, J., Labardin, P., Moreno, J. G., & Riva, A. (2026) · Revue de l’organisation responsable · doi:10.54695/ror.212.0098
FVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verification
Wan, L. J., Ho, C.-T., Bai, Y., Yu, C., Chen, D., & Ren, H. M. (2026) · IEEE VLSI Test Symposium (VTS) · doi:10.1109/vts69484.2026.11563220
“We employ an LLM-as-a-judge [58] to assign a confidence score S llm ∈ [0 , 1] for each reasoning trace τ ∗ and natural language specification x .”
LATEE: An Intelligent Tutoring System with AI Assistant and Virtual Patients for Clinical Diagnostic Reasoning Training
Nguyen, T., Dang, T., Nguyen, D., & Vo, C. (2026) · International Conference on Information and Education Technology (ICIET) · doi:10.1109/iciet69664.2026.11561634
“To mitigate the inherent biases of single-model evaluation [22], we implemented a Multi-Judge Triangulation framework employing three distinct LLMs as independent evaluators: Gemini-2.”
An Information Platform for English Language Acquisition in the IT Domain Utilizing Large Language Models
Abdullaev, N., Soboleva, E. V., & Юрьевич, Е. Д. (2026) · International Conference on Industrial Engineering, Applications and Manufacturing (ICIEAM) · doi:10.1109/icieam69213.2026.11549843
“This creates a bottleneck in the learning process and limits the frequency of practice [11].”
Benchmarking generative AI on fermentation knowledge
Caccavale, F., Krühne, U., Gernaey, K. V., & Gargalo, C. L. (2026) · Systems and Control Transactions · doi:10.69997/sct.137474
Evol-Preference: Automatic evolution of preference data for safety alignment
Liu, Y., Huang, Y., Liu, S., & Pang, X. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133366 · Cited by 1
Scalable Training of Spatially Grounded 2D Vision–Language Models for Radiology
Salcan, Y., Ging, S., Schirrmeister, R. T., Arnold, P., Kotter, E., Bozorgtabar, B., & Brox, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38059-3_53
Geometric insights into the relationship between weight landscape and generalization
Lu, Z., Xu, B., Sun, Y., Zhang, Y., Tulajiang, P., & Lin, H. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114262
Benchmarking Serverless AI Architectures: Modular RAG, Serverless RAG, and Long Context Inference
Petrusel, F. A. S., & Garcı́a-López, P. (2026) · ACM International Conference on Distributed and Event-Based Systems (DEBS) · doi:10.1145/3809481.3816479
An LLM-Based Agentic AI System for Automated Construction of Knowledge Taxonomies in Data Science Problem Solving
Sourove, M. S. U. R., Pan, S., & Chen, L. (2026) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3774398.3811567
GCMembrane-LLM: An Evidence-Grounded Domain-Specific Large Language Model for Structure–Performance Reasoning in Graphene and Carbon Nanotube Separation Membranes
Liu, Y., Liu, S., He, Y., et al. (2026) · Membranes · doi:10.3390/membranes16060214
“5-plus was used as the automatic judge model for comparative evaluation, following the general use of LLM-as-a-judge protocols while recognizing their potential evaluation biases [52].”
Self-citation
Who Defines “Best”? Towards Interactive, User-Defined Evaluation of LLM Leaderboards
Jung, M., Lee, M., Kim, Y., Choi, S., & Kahng, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812353 · Cited by 2
Scenic Test Scenario Generation and Repair for Autonomous Driving via Semantic Feedback and Knowledge Augmentation
Li, X., Su, Q., Li, X., & Yin, Y. (2026) · International Conference on Intelligent Systems, Automation and Control (ISAC) · doi:10.1109/isac69835.2026.11567478
“SFC employs a single large language model (DeepSeek-V3) as the Judge [11] to ensure inference efficiency.”
The Stability Trap: Evaluating the Reliability of LLM-Based Instruction Adherence Auditing
Shergadwala, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806536 · Cited by 1
“…often masks underlying reasoning failures, particularly in quantitative syntactic tasks; and (3) We discuss a Hybrid Auditing Protocol for enterprise governance, recommending that all deterministically verifiable logic be delegated to code, while LLMs are reserved for complex semantic evaluation.”
An Intelligent Fault Diagnosis Agent based on Knowledge Graph For High-speed Train
Yao, Y., Wang, H., Sun, P., et al. (2026) · International Symposium on Autonomous Systems (ISAS) · doi:10.1109/icaisisas68969.2026.11567864
“We introduced LLM-as-a-judge[24] technique as the evaluation method to various answers.”
CLARA: An AI-Augmented Analytics Dashboard for Collaboration Literacy
Xie, D., Anderson, K., Eze, T. D., Lin, C., Shin, B., & Worsley, M. B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29755-6_23
Exploiting Inductive Biases for Imperceptible Adversarial Examples on Multimodal Large Language Models
Jie, X., Chen, Z., & Chen, L. (2026) · International Conference on Sensor-Cloud and Edge Computing System (SCECS) · doi:10.1109/scecs69812.2026.11565848
“Open-source models consist of three components: an Image Encoder (e.g., NFNet-F6 [5], ViT, CLIP), an Input Projector (e.g., linear projector or Q-Former), and an LLM Backbone (e.g., LLaMA [14], Qwen, Vicuna [19]).”
From BOPPPS Stages to Cognitive-Adaptive Prompts: Controlling Instructional Drift in LLM-Based Tutoring Dialogues
Du, L., Zhong, R., 朱鎭晩, Dong, S., & ShuYuan, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29755-6_42
Trustworthy Image Authentication Using Forensic Knowledge Graphs
Nguyen, T. D., & Stamm, M. C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37020-4_3
When to Stop? An Experimental Study on AI Teachable Agent Stopping Mechanisms and Their Learning Affordance in Mathematics
Zhang, A. Y., Li, C., Gülfidan, G., Li, H., Castaneda-Rios, M., Guo, R., Song, Y., & Xing, W. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29755-6_17
DCR-Code: A Difficulty-Aware Multi-Agent Framework for Code Generation Via Decompose-Compose-Refine
Liu, H. (2026) · International Conference on Artificial Intelligence and Industrial Technology Applications (AIITA) · doi:10.1109/aiita69518.2026.11567118
“For code redundancy, we employ an LLM-as-a-Judge [24]: a Deepseek-based model scores each solution from 0 (most concise) to 100 (most verbose) based on whether it contains superfluous statements or over-engineering beyond the problem requirements.”
Adaptive orthogonal role‐aware transformer for robust instruction alignment
Yoon, Y. C., Kim, S., & Kim, G. (2026) · ETRI Journal · doi:10.4218/etrij.2025-0449
“Constitutional AI [7] reduces human-feedback dependence through self-regulating rules, and open instruction-tuning efforts [8-10] and mul-titurn instruction data [11] have further advanced general-purpose instruction following.”
ResVaRe: Parameter-efficient fine-tuning for large language models via cross-layer residual vector adaptation and representation editing
Wang, Y., Li, Y., & Wang, Y. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109273
C3-Bench: A Context-Aware Change Captioning Benchmark
Kim, J.-W., Kim, H., & Kim, U.-H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37435-6_14
Ellipsoid Control: A White-List Jailbreak Defense via Benign Latent Modeling
Chen, L., Wang, W., Tian, Z., et al. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3707452
“We provide the pseudo-code explaining the projection step in Algorithm 2.”
Enhancing Brain MRI Anomaly Detection and Reasoning with ROI Rethink and Synthetic Data
Li, S., Xu, J., Guo, Y., Li, Z., & Wang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38091-3_24
Self-citation
QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI
Veysi, M., Shamsinejadbabaki, P., Zare, M., & Sabouri, M. (2026) · International Biennial Conference on Advances in Artificial Intelligence and Data Science (IBAAIDS) · doi:10.1109/ibaaids66771.2026.11567442
“This approach is inspired by recent work on LLM-as-a-judge paradigms, which demonstrate that large language models can approximate human evaluation when provided with sufficient structure and context [10], [11].”
Influential
HeRoN: a mediated RL-LLM framework for adaptive NPC behavior in interactive environments
Cimino, G., Deufemia, V., & Selice, A. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12275-w
Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation
Jeon, H., Medler, S., Voyles, J., & Wood, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38410-2_43 · Cited by 1
The roles of 6G, blockchain, artificial intelligence, cloud computing, digital twin, and quantum computing in future hospitals
Kumar, P., Maan, S., Singh, V. K., Sharma, A. K., Yadav, J., & Nagpal, V. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-45875-0.00029-6
Decoding Student Dialogue: A Multi-dimensional Comparison and Bias Analysis of Large Language Models as Annotation Tools
Cao, J., Hao, Z., & Yu, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29773-0_29
Closing the Feedback Loop: Automated Quality Assurance for LLM-Generated Feedback
Bulut, O., & Walsh, C. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-29788-4_19
A Multimodal RAG Platform for Feedback-Adaptive Assessment and Cognitive Scaffolding
Kunuku, M. T., & Dehbozorgi, N. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-29788-4_64
Personalized AI Practice Replicates Learning Rate Regularity at Scale
Beauchesne, J., Maroti, C., Bratman, J., Pesenti, J., Holt, L., Tambellini, A., McGrath, A., Guo, M., & Peterson, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29744-0_5
Retriever or reasoner? Decomposing retrieval-augmented generation performance in external audit supervision
Alonso-Robisco, A., Carbó, J. M., García, C. J., Quintana, J., & Tarancón, J. (2026) · Documento ocasional/Documento ocasional - Banco de España · doi:10.53479/43986
“3 This approach follows recent work that uses language models to evaluate model outputs at scale when direct human evaluation is costly or impractical, while remaining anchored in a reference defined by experts (Gu et al., 2024; Zheng et al., 2023; Liu et al., 2023).”
DEERO-prompter: Dual perspective encoding and optimized prompting framework for enhancing mathematical reasoning
Xue, J., Qin, S., Hao, F., et al. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133432
Learning from Giants: Attribution-Aligned Knowledge Distillation for Interpretable ELL Essay Assessment
Bhat, S., & Varma, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29763-1_18
A Multi-dimensional Framework for Automatic Assessment Question Evaluation
Nejjar, O. A., & Berrada, I. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-29788-4_38
Genie Training the Wisher: Six-Dimension Task-Agnostic AI Coaching for Learning Transferable LLM Prompting Skills
Martinenghi, A., Guidotti, S., Donabauer, G., Koyutürk, C., Ortiz-Beltrán, A., Theophilou, E., Chimisso, R., Bink, M., Garzotto, F., Taibi, D., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29763-1_29
The Hidden Cost of Contextual Sycophancy: An AI Literacy Intervention in Human–AI Collaboration
Koyutürk, C., Guidotti, S., & Ognibene, D. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-29788-4_44
“…post-intervention), and (b) advice quality as the NDCG@6 of the assistant’s top-6 recommendations, extracted using an LLM-as-judge pipeline [21,10], in which a separate model (GPT 5.2) analyzed each full conversation transcript and extracted the assistant’s final recommended ranking when…”
Influential
ANVIL: Analogies and Videos for Lecturers
Noviello, Y., Birillo, A., & Migut, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29744-0_23
On Generating and Validating Erroneous Examples in CS1 Using LLMs
Chen, Y., Zhao, C., Levine, J., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29760-0_14
Multi-Agent LLMs for Occupational Profiling: Psychometric Validation on 1636 Chinese Occupations
Han, Y., Luo, X., Ji, F., & Kong, X. (2026) · Behavioral Sciences · doi:10.3390/bs16071064
“…the “LLM‑as‑a‑judge” paradigm has gained traction as a scalable approach to eval‑ uation, with strong LLMs achieving over 80% agreement with human preferences when judging the quality of chatbot responses, on par with the level of agreement among hu‑ man evaluators themselves (Zheng et al., 2023).”
MMVUF: Improving video-based human value understanding in multimodal large language models
Wang, Y., Chang, H., Qiu, H., Jin, L., & Tao, Z. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133457
Mathematical Modeling and Generalization Inference Mechanisms of Large Language Models Under Transformer Architecture
Guo, M., Wu, H., & 郭庆琳 (2026) · Mathematics · doi:10.3390/math14132301
MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
Chen, H., Zhou, K., Hua, H., Zhang, K., Qian, J., Ma, W., Chen, H., Liu, C., Zhao, Y., Wang, X., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37174-4_38 · Cited by 15
Design and Evaluation of a Fully on-Premises RAG-Based Enterprise Chatbot for Technical Support
Li, S., Hung, K., Ho, R., et al. (2026) · IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE) · doi:10.1109/iscaie68866.2026.11576406
“This method is commonly referred to as LLM-as-a-Judge and has been widely applied in modern chatbot benchmark frameworks such as MT Bench and Chatbot Arena [14].”
Density-Guided Response Optimization: Community-Grounded Alignment via Implicit Acceptance Signals
Gerard, P., & Volkova, S. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806449 · Cited by 1
“Following established practices in alignment research [17, 48], we then use LLM-as-judge comparisons along two criteria— relevance (contextual appropriateness to the prompt and community norms) and authenticity (consistency with the community’s characteristic tone, framing, and interactional…”
Fed-RAFT: Enabling Robust Dynamic Clustering in Non-Stationary Federated LLM Fine-Tuning via Asymmetric Fusion
Ma, J., Dang, S., Fang, Z., Zhang, Z., Ge, Z., & Li, X. J. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571275
“5 presents the out-of-distribution generalization performance on vicuna bench [21], with GPT-4 serving as the judge.”
Revisiting Multi-Level Network Modeling and Simulation With Domain-Adapted LLMs
Wu, D., Liu, J., Lin, X., Xu, D., Gao, P., & Liu, Y. (2026) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2026.3708012

Authors: Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I.
ORCID: Zheng, Lianmin
Chiang, Wei-Lin
Sheng, Ying
Zhuang, Siyuan
Li, Dacheng
Gonzalez, Joseph E.
Venue: arXiv
Publisher: arXiv
Published: 2023
Pages: 46595-46623
ISSN: 1049-5258 (ISSN-L)
DOI: 10.48550/arxiv.2306.05685
arXiv: 2306.05685
OpenAlex: W7133239970
Cited by: 1,888
References: 2
Citations by year: 2026: 1115 · 2025: 413 · 2024: 308 · 2023: 51
FWCI: 2548.93
Type: conference paper
Language: en
Open access: green
arxiv.org
Licence: arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations: neural information processing systems (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Funders: VMware
Samsung
Topics: Social Sciences › Law › Artificial Intelligence in Law
Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords: large language models, LLM-as-a-judge, automated assessment, MT-Bench, Chatbot Arena, benchmarking, chatbots

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Abstract

Overview

AI snapshot

TL;DR

Key findings

Methods

Limitations

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks