PreprintCited by 2,065Artificial Intelligence in LawTopic ModelingNatural Language Processing Techniques

The DOI matches its DataCite record; 58 of the 58 references it cites were found in a registry or an index; 4 of the 29 sources its claims cite were checked for support.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Open access

Zheng, Lianmin, Chiang, Wei-Lin, Sheng, Ying, Zhuang, Siyuan, Wu, Zhanghao, Zhuang, Yonghao, Lin, Zi, Li, Zhuohan, Li, Dacheng, Xing, Eric P., Zhang, Hao, Gonzalez, Joseph E., Stoica, Ion·2023·arXiv

DOI 10.48550/arxiv.2306.05685arXiv 2306.05685

Open this paper in CoChat — free account Open PDF

Abstract

Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper examines whether strong LLMs can serve as judges for chatbot responses by comparing their judgments with human preferences on two new benchmarks, MT-bench and Chatbot Arena. It finds that GPT-4 can reach over 80% agreement with human preferences, comparable to human-human agreement, while also documenting several judge biases and mitigation strategies.

Key findings

GPT-4’s judgments aligned closely with human experts on MT-bench: 85% agreement without ties, versus 81% agreement among humans; on Chatbot Arena, GPT-4 also showed high agreement with crowd votes, above 80%.
Position bias was substantial: with the default prompt, consistency was 65.0% for GPT-4, 46.2% for GPT-3.5, and 23.8% for Claude-v1; swapping positions and few-shot examples raised GPT-4 consistency to 77.5%.
Verbosity bias appeared in a “repetitive list” attack: Claude-v1 and GPT-3.5 failed 91.3% of cases, while GPT-4 failed 8.7%.
Reference-guided grading reduced math-judging failures from 14/20 under the default prompt to 3/20, with chain-of-thought at 6/20; self-enhancement was suggested by GPT-4 favoring itself with a 10% higher win rate and Claude-v1 with a 25% higher win rate, but the study could not confirm the bias.

Methods

The authors created MT-bench, 80 multi-turn questions across 8 categories with 10 questions each, and Chatbot Arena, a crowdsourced anonymous battle platform; they collected about 3K expert votes from 58 labelers on MT-bench and about 30K Arena votes, sampling 3K for analysis.
They compared LLM judges—mainly GPT-4, GPT-3.5, and Claude-v1—using pairwise comparison, single-answer grading, and reference-guided grading against human expert and crowd votes, measuring agreement and win rates.
They also evaluated LLaMA and Vicuna variants on MMLU, TruthfulQA, and MT-bench with GPT-4 single-answer grading to test how preference-based and capability-based benchmarks complement each other.

Limitations

The paper states that it emphasizes helpfulness and largely neglects safety, honesty, and harmlessness; it also notes that helpfulness dimensions such as accuracy, relevance, and creativity are combined into one metric, and its bias mitigations are preliminary.
The paper reports that self-enhancement bias could not be determined because of limited data and small differences, and that its position-bias test is difficult because the compared answers are very similar and sometimes indistinguishable even to humans.
The text makes evident that few-shot judging may introduce new biases and makes API calls about 4× more expensive, while zero-shot Vicuna-13B as a judge had low consistency (11.2–16.2%) and high error rates (22.5–78.8%), limiting its use as a scalable open-source judge.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.05685

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

56 Own result · 35 Own interpretation · 7 Sources not found · 7 No basis given · 3 Partly supported · 2 Own method · 1 Not checked yet · accuracy not yet measured

The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Once aligned with humans, these chat models are strongly preferred by human users over the original, unaligned models on which they are built.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites [31] source
- Partly supported
  Section 4.5 (p. 21) says crowdworkers clearly preferred the online RLHF models in comparisons with context-distilled and earlier RLHF models. But those preference comparisons do not establish that users strongly preferred them over the original, unaligned base models; the paper discusses base-model improvements on other evaluations (Introduction, p. 4). (LLM, conf=0.86)
  cites [2] sourceExperimental — accuracy not yet measured
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [30]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [8]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [52]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [48]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [14]
boosted
direct
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We show that some of the biases are minor or can be mitigated.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that in math/coding category, GPT-3.5 and GPT-4 have similar overall win-rate because they both failed to answer some hard questions, but GPT-4 is still significantly better than GPT-3 in the direct pairwise comparison or single-answer grading.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.3 Win rates under different judges
plain
correlational
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, a small high-quality conversation dataset can quickly teach the model a style preferred by GPT-4 (or approximately human) but cannot improve MMLU significantly, as shown by the Vicuna-7B (selected) which is trained with only 4.8M tokens or 3K conversations.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
direct
Single case
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most LLM judges favor the first position.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only GPT-4 successfully detected this attack.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plain
Single case
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As in Table 2, we found all of them exhibit strong position bias.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, it has a high error rate (from 22.5% to 78.8%) because of its weaker instruction-following capability.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plain
direct
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Claude-v1 favors itself with a 25% higher win rate.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
direct
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 12 (Appendix), the few-shot judge can significantly increase the consistency of GPT-4 from 65.0% to 77.5%.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
direct
n = 3

Show all 65 claims

Probably not a substantial claim. 22
It may state two separate claims as one. 29
The quote only partly states this claim. 2

The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As LLMs continue to improve, they show potential in replacing human annotators in many tasks [17,20].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3 LLM as a Judge
What the sources say
- Supported
  The Discussion (p. 4) says LLMs could transform text annotation across a variety of tasks and that ChatGPT may already outperform crowd annotators. The Introduction (p. 2) also describes the possibility of annotating entire samples with ChatGPT. These passages support the hedged claim about potential replacement in annotation tasks, particularly those performed by crowd-workers. (LLM, conf=0.88)
  cites [17] source
- Partly supported
  The Discussion and Conclusion (Section 5; page unavailable) says ChatGPT showed great potential as a data annotation tool for implicit hateful tweets. It does not establish that LLMs could replace human annotators across many tasks; it also urges caution about incorrect decisions. (LLM, conf=0.86)
  cites [20] sourceExperimental — accuracy not yet measured
hedged
conditional
Probably not a substantial claim.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“DynaBench addresses the challenges posed by static standardized benchmarks, such as saturation and overfitting, by emphasizing dynamic data with human-in-the-loop.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
What the sources say
- Partly supported
  The Introduction (p. 1) presents Dynabench as dynamic data collection with humans and models in the loop, in contrast to static benchmarking. Caveats and Objections (p. 8) explicitly says it anticipates dataset saturation. The copy does not identify overfitting as a challenge of static benchmarks; its discussion of annotators ‘overfitting’ on models (p. 7) describes a potential risk of the dynamic approach instead. (LLM, conf=0.84)
  cites [21] sourceExperimental — accuracy not yet measured
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite the base LLaMA models showing competitive performance on conventional benchmarks (Table 8), its answers to open-ended questions are often not preferred by humans.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.”
- Also stated in: § GPT-4 Judgment: · plain
  “Once addressed, our results from 3K controlled expert votes and 3K crowdsourced human votes in the wild verify that GPT-4 judge match human evaluations at an agreement rate exceeding 80%, achieving the same level of human-human agreement ( §4.2, Table 4).”
- Also stated in: § 4.2 High agreement between GPT-4 and humans · plain
  “The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Abstract
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The consistency improves significantly from 16.2% to 65.0%.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plain
direct
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves 56.8% when including all three labels, and 85.5% when excluding tie predictions and labels, significantly outperforming random guesses of 33% and 50% respectively, and show positive signals to match GPT-4 (66% and 87% respectively).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plain
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Despite GPT-4 knows how to solve the question (if asked separately), it made a wrong judgement saying both assistants' wrong answers are correct.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 14:
plain
Single case
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to the classification formulation, every output is recognizable (error rate 0%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plain
direct
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 15, Vicuna-13B has a consistency rate from 11.2% to 16.2% across different prompt templates, much lower than all the closed-sourced models.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in Figure 13 (Appendix), we present an example of an elementary math question in which GPT-4 makes an incorrect judgment.It's worth noting that although GPT-4 can solve the problem (when asked separately), it was misled by the provided answers, ultimately resulting in incorrect judgment.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
direct
Single case
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 3 shows the failure rate of LLM judges under this attack, demonstrating that all LLMs may be prone to verbosity bias though GPT-4 defends significantly better than others.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
hedged
conditional
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 11, the position bias is more noticeable for models with close performance and can almost disappear when the performance of the two models differs a lot.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > D.1 Position bias
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results reveal that strong LLMs can achieve an agreement rate of over 80%, on par with the level of agreement among human experts, establishing a foundation for an LLM-based evaluation framework.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 7 Conclusion
plain
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Comparing GPT-4 and other LLM judges, we find they reach a similar non-tie agreement ratio between humans but the number of non-tied votes from GPT-4 is much larger.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plain
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When Assistant A is placed in the first position, GPT-4 thinks A is better, but its verdict changes when we swap the position of A and B.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 11: An example of position bias.
plain
direct
Single case
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that fine-tuning on high-quality dialog datasets (i.e., ShareGPT) can consistently improve the model performance on MMLU and the improvement scales with fine-tuning data size.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
direct
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both GPT-3.5 and Claude-v1 show a verbosity bias towards the longer and repetitive answer.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plain
direct
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both tables, GPT-4 with single-answer grading matches both pairwise GPT-4 and human preferences very well.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
boosted
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On math and coding questions, LLM judges are more confident even though their judgments can often be wrong, as we show in Section 3.3.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > D.1 Position bias
plain
correlational
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Besides, the longer prompts make API calls 4× more expensive.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
direct
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, even with the CoT prompt, we find that in many cases LLM makes exactly the same mistake as the given answers in its problem-solving process (See example in Figure 15 (Appendix), suggesting that LLM judge may still be misled by the context.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Chain-of-thought and reference-guided judge.
hedged
conditional
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe similar pattern from other LLM judges such as Claude/GPT-3.5.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 11: An example of position bias.
plain
correlational
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, the pairwise comparison may lack scalability when the number of players increases, given that the number of possible pairs grows quadratically;”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.1 Types of LLM-as-a-Judge
hedged
conditional
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“However, the heightened user preference does not always correspond to improved scores on traditional LLM benchmarks -benchmarks like MMLU [19] and HELM [24] cannot effectively tell the difference between these aligned models and the base models.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 1 Introduction
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [19]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [24]
plain
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This misalignment of conventional benchmarks underscores the core problem driving this paper: the need for a robust and scalable automated method to evaluate LLM alignment with human preferences.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plain
direct
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“single answer grading may be unable to discern subtle differences between specific pairs, and its results may become unstable, as absolute scores are likely to fluctuate more than relative pairwise results if the judge model changes.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.1 Types of LLM-as-a-Judge
hedged
conditional
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Conversational benchmarks, like CoQA [35], MMDialog [15] and OpenAssistant [23], are closest to our intended use cases.However, the diversity and complexity of their questions often fall short in challenging the capabilities of the latest chatbots.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 2.1 Motivation
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [35]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [15]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [23]
plain
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, high consistency may not imply high accuracy and we are not sure whether the few-shot examples will introduce new biases.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
hedged
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means that GPT-4 is more affirmative and less suffered from position bias but other models also perform well when they give an affirmative answer.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plain
correlational
It may state two separate claims as one.
In the paper
“LLMs are known to have limited math and reasoning capability [10], which results in its failure of grading such questions because they do not know the correct answers.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [10]
boosted
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They focus on different aspects of a model and the recommended way is to comprehensively evaluate models with both kinds of benchmarks.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
correlational
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To invite participants, we obtained their consent by letting them sign an application form.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > C.1 MT-bench human evaluation
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another more aggressive approach is to assign positions randomly, which can be effective at a large scale with the correct expectations.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
conditional
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Table 8, no single benchmark can determine model quality, meaning that a comprehensive evaluation is needed.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This bias is not unique to our context and has been seen in human decision-making [3,34] and other ML domains [22,41].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [3]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [34]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [22]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [41]
plain
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Chain-of-thought is a widely used technique to improve LLM's reasoning capability [47].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Chain-of-thought and reference-guided judge.
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [47]
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our LLM-as-a-judge approach can automate and scale platforms of this nature.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
direct
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results indicate that using LLM-as-a-judge to approximate human preferences is highly feasible and could become a new standard in future benchmarks.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
hedged
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Users are required to accept the terms of use, which obtain their consent and give us the right to release the conversation data.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > C.2 Chatbot Arena
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A conservative approach is to call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
direct
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These responses are rendered as natural languages and are difficult to be parsed automatically, making the model less useful in a scalable and automatic evaluation pipeline.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plain
direct
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLM-as-a-judge offers two key benefits: scalability and explainability.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.2 Advantages of LLM-as-a-Judge
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that this test is challenging because the answers are very similar and occasionally indistinguishable even to humans.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The single-answer grading can be converted into pairwise comparison results for calculating the agreement.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plain
direct
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Honesty and harmlessness are crucial for a chat assistant as well [2].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 6 Discussion
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [2]
plain
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.

46 of 46

The paper relies on this without support.
LLMs trained with RLHF already exhibit strong human alignment.
“Because these models are often trained with RLHF, they already exhibit strong human alignment.”
The paper relies on this without support.
There has not been a systematic study of the LLM-as-a-judge approach.
“However, there has not been a systematic study of this approach.”
The paper relies on this without support.
Human preferences serve as a direct measure of a chatbot's utility in open-ended, multi-turn human-AI interactions.
“While largely overlooked by existing LLM benchmarks, human preferences serve as a direct measure of a chatbot's utility in open-ended, multi-turn human-AI interactions.”
The paper relies on this without support.
The human labelers are mostly graduate students and are considered experts and more skilled than average crowd workers.
“The labelers are mostly graduate students so they are considered experts and more skilled than average crowd workers.”
The paper relies on this without support.
Human evaluation is the gold standard for assessing human preferences.
“While human evaluation is the gold standard for assessing human preferences, it is exceptionally slow and costly.”
The paper relies on this without support.
Model services such as GPT-4 can become expensive with a growing number of evaluations.
“However, model services such as GPT-4 can also become expensive with a growing number of evaluations.”
The paper relies on this without support.
Popular open-sourced LLMs, e.g. Vicuna-13B, show strong language understanding capability and are much cheaper than close-sourced LLMs.
“On the other hand, popular open-sourced LLMs, e.g.Vicuna-13B shows strong language understanding capability, and are much cheaper than close-sourced LLMs.”

7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Traditional evaluation metrics based on the similarity between outputs and reference answers (e.g., ROUGE [25], BLEU [32]) are also ineffective for these questions.”
Method
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 6×
“5, Claude-V1, Vicuna-13B, Alpaca-13B [38], and LLaMA-13B [39].”
BackgroundInfluential
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1×
“Chain-of-thought is a widely used technique to improve LLM’s reasoning capability [39].”
Method
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Chain-of-thought is a widely used technique to improve LLM’s reasoning capability [39].”
Method
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832 · cited 1×
“This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
Background
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“There has been a proliferation of LLM-based chat assistants (chatbots) that leverage supervised instruction fine-tuning and reinforcement learning with human feedback (RLHF) to unlock new instruction following and conversational abilities [27, 2, 26, 7, 43].”
Background
HellaSwag: Can a Machine Really Finish Your Sentence?Preprint
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07830 · Cited by 1,693 · cited 2×
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
Method
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 2×
“This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
Background
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 2×
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
Method
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243

Show all 2 references

Research works printed but not linked: 2

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and Ion Stoica. SkyPilot: An intercloud broker for sky computing. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 437-455, Boston, MA, April 2023. USENIX Association.

Sorted by citations

Items 1,301–1,400 of 2,065

Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Risks
Gur-Arieh, S., Wang, A., & Fazelpour, S. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806469 · Cited by 2
Comparative analysis of evaluation methods for generative artificial intelligence systems and development of selection algorithm
Meshkov, A. (2026) · Journal of Electrical Systems and Information Technology · doi:10.1186/s43067-026-00374-6
“The main advantage of LLM-as-a-Judge is the ability to evaluate those various aspects of quality [21], both the accuracy and relevance of the answer, and those that cannot be formalized into metrics, for example, the appropriateness of tone, the naturalness of presentation.”
Influential
Automated bias blind spots: Examining stereotype associations in VLM-as-a-judge paradigms
Hong, R. S., Ovalle, A., Ung, M., Spiliopoulou, E., Sagun, L., Williams, A., & Ross, C. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806459 · Cited by 1
Robustness Evaluation of Graph-RAG Systems: Compound Defect Masking Effects
Yan, S., Wu, Y., & He, S. (2026) · International Symposium on Computer Technology and Information Science (ISCTIS) · doi:10.1109/isctis70043.2026.11572066
“Evaluation relies on a zero-temperature LLM-as-Judge framework powered by qwen3-max-2026-01-23 [4].”
Rollback Is Not Undo: Path-Dependent Failures in LLM-Arbitrated Network Control
Pan, W., & Liu, Z. (2026) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom59046.2026.11571400
“Networked systems increasingly incorporate AI components in the control plane [1]: telemetry is summarized by LLMs, mitigation policies are proposed by LLM agents, and learned arbiters reconcile conflicting recommendations into a single action [2], [3].”
Evaluating Frontier LLMs in Answering Game-Design Questions: A Preliminary Study
Filho, A. A. D. C., Rodriguês, M. A. F., & Mendonça, N. C. (2026) · International Workshop on Designing Software (Designing) · doi:10.1145/3786152.3788590
LLM-MHR2: Leveraging Large Language Models for Multimodal and History-Aware Hashtag Recommendation
Li, Y., Tan, Z., Feng, T., Mo, T., & Li, W. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-026-03190-6
“5 [54].”
A Generalized AI-Powered Framework for Cross-Domain Technology Readiness Assessment
Rea-Guaman, M., Almeida-Jara, A., & Masacela-Atupaña, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30488-9_30
Kamu Web Sitelerinin Rehber Tabanlı Kullanılabilirlik Değerlendirmelerinin Yapay Zekâ Destekli Otomasyonu
Öztürk, S., Altınözen, M., Yenigün, B. U., Gül, T., Odabasi, M., ŞAHİN, A. S., & ÖZTANIR, O. (2026) · International Journal of Advances in Engineering and Pure Sciences · doi:10.7240/jeps.1834806
PlayCoder: Making LLM-Generated GUI Code Playable
Peng, Z., Tao, W., Yin, X., Ying, C., Luo, Y., & Guo, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808097 · Cited by 10
Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation
Sivapiran, S., & Uddin, G. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808123 · Cited by 1
SRD: A model-agnostic semantic framework for jailbreak mitigation in large language models
Shi, C., Zhang, Z., Quan, G., Pan, J., & Yue, X. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116560
Multi-Agent LLM Architecture for Systems Engineering Automation
Padubrin, M., Kulzer, A. C., & Guerocak, E. (2026) · SAE technical papers on CD-ROM/SAE technical paper series · doi:10.4271/2026-01-0777
ISBench: Benchmarking Instruction-Following Capability and Safety of Large Speech-Language Models Across Acoustic Conditions
Wang, C., Wu, J., Ren, S., Lin, J., Wang, G., & Zhang, J. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-0199-0_4
Validating LLM-Generated SQL Queries through Metamorphic Prompting
Lin, L. G., Zhu, Q., Hong, J., Wang, C., Liu, Y., & Wu, R. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797146
Modality-Driven Search with Holistic Trace Judging for ARC-AGI-2
Land, J. (2026) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd69640.2026.11637368
“…as explicit search across multiple reasoning modalities , extending ToT-style inference [10]; (ii) tool-and program-mediated reasoning [11], [12] used to produce richer intermediate artifacts for downstream selection; and (iii) judge-based selection adapted from the LLM-as-a-judge paradigm [16].”
The Person Behind the Sound: Demystifying Audio Private Attribute Profiling Via Multimodal Large Language Models
Wang, L., Yao, K., Li, X., Yang, D., Li, H., Wang, X., & Dong, W. (2026) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp63933.2026.00150
“The phase of Scrutinization follows the LLM-as-a-judge mechanism [41].”
The Label Imitation Game:Turing Test Network for Zero-Shot Pseudo-Label Pruning
Griffin, B., & Corso, J. J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37577-3_3
Seeing Through Multiple Views: Parameter-Efficient Fine-Tuning via Selective Neurons for Consistent Radiology Report Generation
Chen, Y., Zhu, J., Yu, Y., Shi, Y., Naghshbandi, H., Liu, J., Koh, A. S.-M., Fang, F., Ong, K. E., & Yeo, S. C. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38059-3_54
TLAS-YHCT VN: Development and Evaluation of a Socratic AI Teaching Assistant for Traditional Vietnamese Medicine Education — A Two-Phase Mixed-Methods Study (Preprint)Preprint
Hồng, T. (2026) · doi:10.2196/preprints.105820
Large Language Model-Enhanced General Transportation Agent Framework for Human Mobility Forecasting and Synthetic Travel Survey Data Generation
Salvador, I., Furno, A., & Derrible, S. (2026) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/03611981261456273 · Cited by 1
Evaluating the Evaluator: LLM Evaluators for Patient-Facing AI Agents
Mastrianni, A., Andreadis, K., Chen, J., Rodriguez, D. V., & Mann, D. (2026) · ACM Interactive Health Conference · doi:10.1145/3786579.3804927 · Cited by 1
“Asaresult,recentresearchhasinvestigatedthe useoflargelanguagemodels(LLMs)asjudgestoevaluateGenAI content [10, 22].”
Fast and Robust Automated Essay Scoring with Retrieval-Augmented LLM and Ensemble Learning
Bakr, M., Elfakharany, E., & ElSeddawy, A. I. B. (2026) · ICEENG International Conference for Innovations in Intelligent Computing and Cybersecurity (IICC) · doi:10.1109/iicc69623.2026.11581980
“While comparative ranking is established in preference learning and LLM-as-a-Judge frameworks [16], its application as a mechanism to stabilize quantized, 8B-parameter LLMs in automated essay scoring is underexplored.”
Examining the Performance of Large Language Models in Health Information Quality Evaluation Tasks: An AI Post-Alignment Perspective
Zhang, S., Zhang, J., Li, P., Zuo, M., & Feng, L. (2026) · Information · doi:10.3390/info17070649
“For example, studies such as MT-Bench, Chatbot Arena, and G-Eval have shown that strong LLM evaluators can, under certain conditions, achieve substantial agreement with human preferences [10,20].”
DUA: Automated Jailbreak via Dual-Use Ambiguity and Semantic-Operational Disentanglement
Ma, S., Liu, W. T., Xie, W., Wen, L., Wang, P., Chen, R., & Chen, S. (2026) · ICEENG International Conference for Innovations in Intelligent Computing and Cybersecurity (IICC) · doi:10.1109/iicc69623.2026.11582014
“Following standard protocols [17], [19], we focus on the Attack Success Rate (ASR) as the primary efficacy measure.”
A Lightweight Vision-Language Model for Disaster Image Summarization
Yoshizaki, H., Uchiyama, A., Hiromori, A., et al. (2026) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops68308.2026.11585419
GMTW-Ro: a deterministic benchmark for evaluating large language models on grounded Romanian tasks
Bulzan, A.-Ş., Morariu, B., Joldea, A.-R., Cernăzanu-Glăvan, D., & Cernăzanu-Glăvan, C. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1831918
Improving the Efficiency of Assessing Students’ Open Ended Responses in Philosophy Courses Using LLMs with RAG
Maiatskaia, E., Анатольевна, О. Т., & Artiugin, O. (2026) · Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT) · doi:10.1109/usbereit70063.2026.11580973
“The automation of essay grading has evolved from latent semantic analysis judgments to ”LLM-as-a-Judge” paradigms, where Large Language Models act as evaluators [1].”
The road to default is paved with good intentions: Motivational values as predictors of loan securing and repayment in crowdfunded microlending
Singh, S. P., Hamby, A. M., Sarin, S., et al. (2026) · Journal of Business Venturing Insights · doi:10.1016/j.jbvi.2026.e00642
An Adaptive Multimodal End-to-End Autonomous Driving Framework Based on Behavior Discrepancy
Chen, X., Wang, X., Weihao, C., et al. (2026) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2026.3706626
ESG-SASB Label Stability: A Curated Benchmark and Reproducible Pipeline for Reusing Sentence-Level Sustainability Disclosure Labels
Li, Y., Chen, T., Ke, W., & Pang, P. C. (2026) · Informatics · doi:10.3390/informatics13070106
Safeguarding LLM Agents from Misalignment through Provenance Analysis
She, Y., Liang, Y., & Kang, E. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837535 · Cited by 8
Evaluation Criteria for Synthetic Respondents in Qualitative Research: Limits of Applicability
Zhulikov, K. (2026) · Inter · doi:10.19181/inter.2026.18.2.5
Adaptive reinforcement learning based projected gradient descent attack
Zhu, Z. D. J. D., Zhang, Y., Ye, A., Wang, X. Y., Wang, C., & Zhu, T. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08687-z · Cited by 1
ParentEval: A Multi-dimensional Age Rating Framework for Large Language Models
Schacht, S., & Lanquillon, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30846-7_35
Enhancing Jailbreak Defense with Diverse Perturbation Combinations
Wu, T., Gao, Q., & Yu, Z.-F. (2026) · International Conference on Machine Intelligence Theory and Applications (MiTA) · doi:10.1109/mita69365.2026.11582392
“5 (Vicuna, [26]), Llama-2-7B-chat (Llama-2, [27]), GPT-3.”
Can Large Language Models Evaluate Cybersecurity Detection Rules? Framework and Empirical Analysis
Meng, C., Le, W., Dong, F., et al. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582729
“LLMs present a promising direction for addressing these limitations, given their demonstrated evaluation capabilities [7], [8] and scalable nature.”
From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation
Maimon, A., Cohen, A. D. N., Vishne, G., Ravfogel, S., & Tsarfaty, R. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.745 · Cited by 14
“For generation tasks, we adopted an LLM-as-a-judge approach (Zheng et al., 2023) using Deepseek-R1 (DeepSeek-AI, 2025).”
Safe Deployment of a Generative AI Assistant for Traditional-Medicine Education: Defense-in-Depth Architecture, Domain-Calibrated Safety Evaluation, and a Proposed Minimum Safety StackPreprint
Tạ, B. H., Linh, T. T., Lê, H. O., GIANG, N. T. T., Tang, T., Tien, L. T. T., & Bui, T. N. A. (2026) · Research Square · doi:10.21203/rs.3.rs-10136391/v1
Context Collapse in Long-Horizon Agents: Benchmarking Hierarchical Memory against RAG and Summarization
Raheem, E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6976218
CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset
Libovický, J., Helcl, J., Manea, A.-A., & Vico, G. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.737 · Cited by 2
DeepSeek Under Attack: An Analysis of Jailbreak Attacks and Prompt-Level Defenses
Hayashi, V. T., Neto, M. P. P., Miers, C. C., Redígolo, F., Arakaki, R., & Junior, M. A. S. (2026) · IEEE Access · doi:10.1109/access.2026.3710427
Evidence-conditioned weak-answerer diagnosis for traceable quality control of educational quizzes generated by artificial intelligence
Han, B. G. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.116067
MovieTeller: Tool-augmented Movie Synopsis with ID Consistent Progressive Abstraction
Li, Y., Chen, X., Jiang, M., Tang, W., & Wang, G. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582204
“2) LLM-as-a-Judge [25]: We employ Gemini 2.5 Flash [26] as an automated evaluator to assess the quality of final synopses.”
Influential
MLCWStudio: a web application for Multi-LLM Consensus Weighting of social capital outreach indicators for blockchain-enabled microfinancePreprint
Shafie, N. A., Rodzi, Z. M., Sanusi, Z. M., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7067713
Building Digital Twins of Criminal Personas: When LLM Agent Responses Are Rated More Human Than Human Responses
Asadi, A. R., Kunapareddi, V., Smith, M. S., & Said, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31048-4_12
Agentic Knowledge Curation Versus Full-Context Retrieval: An Empirical Study of Retrieval Failure Topology in Long-Context LLM Systems
Martín, C. A., Torres, J., Aguilar, R. M., Alayón, S., & Bacallado, M. A. (2026) · Applied Sciences · doi:10.3390/app16136793
Enhancing Realism of Scenario-Based Experiments on Human-AI Interactions with LLM
Mouakher, A., Leszczyński, G., Gaczek, P., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29596-5_11
Radiologist Detection of Deficiencies in LLM-Generated Patient Communications About Radiology Reports: A Multi-Reader Study
Shin, C., Park, J. H., Kim, S. J., Lee, Y. H., & Lee, H. S. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02109-5
Conceptual Design and Taxonomy of a RAG-Based Attack Memory for LLM Red Teaming
Heithoff, I., Woldai, B., & Schacht, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30860-3_5
AB-JB (Jailbreaking): A Budgeted Hybrid Pipeline for Dataset-Scale Red-Teaming of 7B Aligned LLMs
Ahmad, A., Naseer, M., & Ahmad, M. (2026) · International Conference on Computing, Mathematics and Engineering Technologies (iCoMET) · doi:10.1109/icomet69771.2026.11591723
“LLM-as-a-judge is widely used for scalable evaluation but may introduce bias, motivating careful interpretation and occasional human validation [13], [14].”
Progressive Text-guided Lightweight Vision-Language Models with Decoupling-and-Alignment Learning for Autonomous DrivingPreprint
Dong, Z., Deng, W., Niu, S., & Shao, X. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7073820
Comparative testing of open-source LLMs in professional subject area tasks: a multi-domain benchmark with multi-judge and expert evaluation
Pankratov, E. R., & Yakimov, A. B. (2026) · Программные системы и вычислительные методы · doi:10.7256/2454-0714.2026.3.80458
Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?
Rouillard, A., Mundia, S., Camara, L., Dangor, Z., Gramanie, M. C., Kalla, I., Madhi, S. A., Morar, K., Ncube, M. T., Saloojee, H., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30710-1_44
“Early results suggest that strong LLM-based judges can approximate human judgment closely [7, 8].”
Debate-Driven Evaluation of LLMs: Challenges and Opportunities for Transparency and Human–AI Collaboration
Du, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30849-8_24
WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation
Dong, W., Fu, T., Yu, Z., Wang, H., Su, A., Fang, Z., Chen, Y., Wang, S., Wu, M., Jiang, P., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37035-8_28
From MIRAGE to CLEAR: Component-Level Explainable Anomaly Reasoning for Autonomous Vehicle Perception Systems
Fernandez, D., MohajerAnsari, P., Salarpour, A., et al. (2026) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn69566.2026.00072
Highway Landscape Preference Along Malaysia’s North–South Expressway: A Comparison of Multimodal Large Language Models and Human Judgments
Gao, H., Smardon, R., Bakar, S. A., Maulan, S., & Yang, J. (2026) · Land · doi:10.3390/land15071240
Constructivist Summarization Meets Realist Clustering: A Hybrid Framework for Aviation Incident Knowledge Extraction
Hatoum, C., Combes, C., Fresse, V., Gravier, C., & Orzalesi, M. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-31141-2_13
What Do AI Agents Actually Change? An Empirical Taxonomy of Mutation Patterns in Performance-Improving Pull Requests
Dovhoshliubnyi, I., Soroush, N., Sami, A., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30699-9_12
“…filter and the dual-LLM intersection err toward exclusion, so reported category counts are conservative lower bounds reflecting potential LLM bias [8]; per-category proportions may still be inflated for mutations the filter detects most reliably (e.g., explicit import or rename edits).”
Can LLMs Understand Baseball Rules? A Benchmark for Long-Context Rule Reasoning
Matsuzaki, Y., Hirao, Y., Perusquía-Hernández, M., Uchiyama, H., & Kiyokawa, K. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20897-2_43
Evidence-Bound Persona Extraction with LLMs: Theory and Algorithms
Mingkhwan, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20897-2_77
Evaluation Validity in Information Retrieval
Thomas, P., Craswell, N., Sanderson, M., Spielman, S. E., Sim, R., & White, R. W. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808538 · Cited by 1
Failing Forward: Understanding Query Failure in Retrieval, Judgment, and Generation
Hosseini, S. M. B., Arabzadeh, N., Saliminabi, M. H., Androutsos, D., Zihayat, M., & Bagheri, E. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809859 · Cited by 1
Readability Improvement with Open-Source LLMs: A Comparative Evaluation on Romanian Language
Gruia, L., & Țapu, R. (2026) · International Conference on Communications (COMM) · doi:10.1109/comm70054.2026.11591289
“The emerging LLM-as-a-Judge paradigm—where a frontier model evaluates outputs from other models—has been shown to approximate human judgments with over 80% agreement on pairwise comparisons [14], making it a scalable alternative to manual annotation.”
PersonaPlugin: A Multi-Source Persona Framework for LLM Personalization in Telecommunications
Kang, J., Lee, M. S., Kim, S., Shin, J., Lee, C., Jeon, Y., & Jo, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808416
Speak Beyond English: Multilingual Prompts Improve Query Classification in Small Language Models
Banerjee, P., Bhattacharjee, P., & Jana, A. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808528
AI Control Through Causal Agents
Kaushik, M. M., Karthik, K. N. V., & Arjan Durresi, A. (2026) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-032-23304-2_11
One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
Cai, H., Li, Y., Yu, T., Zhu, F., Wang, W., Feng, F., & Li, W. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809689 · Cited by 8
Evolution Context Gap: An Empirical Study of Agent Manifests
Chondamrongkul, N., Rueangsirarak, W., & Jakkaew, P. (2026) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse68839.2026.11597070
“A category is assigned to a section when at least two of the three models agree (majority vote ≥ 2 / 3 ), following LLM-as-Judge methodology [16] and reducing single-model bias.”
Automated Bug Frame Retrieval from Gameplay Videos Using Multimodal Large Language Models
Lu, W., Senchenko, A., Hindle, A., & Bezemer, C. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786865
“…a final, globally consistent ranking of bug-relevant keyframes, we aggregate all candidate keyframes from the batch-level rankings and their ratio-nales into a single prompt and pass it to GPT-4o again, following recent advances in which large language models can be used as judges [10, 15, 24, 54].”
TraE-VL: A Cascaded Framework for Fine-Grained Perception and Controllable Stylization in Domain-Specific Copywriting Generation
Fu, X., Fan, C., & Yu, M. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2852-2_38
Advancing Fit-for-Purpose Access to CII Best Practices Using Multimodal Large Language Models
Chen, G., Pedraza, M., Albaloul, R., Albert, A., & Jaselskis, E. J. (2026) · Journal of Management in Engineering · doi:10.1061/jmenea.meeng-7437
Evolva: A Multi-turn Contextual Attack for Long-Reasoning LLMs
Du, W., Yang, X., Liu, Z., Zhang, B., & Xue, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2852-2_46
DPLoRA: Stable Diffusion Model distillation via Dynamic Parallel LoRA branches
Zeng, W., Kong, Y., Wu, F., et al. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114420
EvidITI: Enhancing RAG Robustness Through Inference-Time Intervention on Evidence-Seeking Attention Heads
Feng, C., Li, B., Huang, R., Ma, M., Dai, Y., & Wang, C. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2862-1_14
Hard Problems, Soft Guards: Softening LLM Guards with Olympiad-Grade Problems
Zhong, T., Xu, M., & Yang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2859-1_13
Investigating Data Lakes Semantic Enrichment Using LLMs
Photiou, A., Papageorgiou, P., Pingos, M., & Andreou, A. S. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30809-2_31
Containing the Cascade: A Benchmark and Reference Mediator for Failure Propagation in Tool-Using LLM Multi-Agent Systems
Divi, V. R., Gullapalli, S., & Brahmandam, B. A. (2026) · International Conference on Electronics Representation and Algorithm (ICERA) · doi:10.1109/icera72709.2026.11666599 · Cited by 1
“Our semantic anomaly gate uses LLM-as-judge scoring [30], [31] as one detection channel.”
Influential
Dual-Loop Multi-agent Framework and Policy Optimization for Numbered Musical Notation Digitization
Lei, Z., Bai, P., & Shi, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3447-9_18
Robust Neural Retrieval and Generation for Digital Advocacy: Transitioning from Fuzzy Matching to Affective RAG
Bar, N., Abraham, S., Hay, Y., & Cohen, S. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30801-6_8
Toward Dynamic Evaluation of Factual Numeric Recall in Large Language Models via Next Prime Identification
Çulha, D. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30801-6_9
Exploring the Use of Artificial Intelligence for Automatic Text Summarisation and Evaluation in eRulemaking
Promikyridis, R., Patsouras, C. F., Zeginis, D., Tambouris, E., & Tarabanis, K. A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-24807-7_22 · Cited by 1
Development of LLM- and RAG-Based Question Answering for Submarine Combat System Manuals
Na, H.-H., Lim, J.-K., Lee, J.-G., et al. (2026) · Journal of the KNST · doi:10.31818/jknst.2026.6.9.2.329
Reflect-SQL: A Self-reflection Based Framework for Text-to-SQL
Jain, A., & Shrivastava, M. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2014-4_7
“We acknowledge that using an LLM as a judge can introduce potential biases [17].”
A Multi-Stage Adversarial Framework for Compact and Effective Jailbreaking of Large Language Models
Liu, W. T., Liu, Z. H., Xu, C., Li, C. B., Guo, J., Ma, J., & Huang, F. (2026) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2026.3712350
“Open-source targets include DeepSeek-R1 [43], DeepSeek-V3 [44], Vicuna-7B/13B [45], and LLaMA2-7B/13B [46].”
Breaking Memory Wall for Fast Edge LLM Inference Using Contextual Sparsity
Wei, Z., Zhou, Y., Zhao, J., Li, K., & Fu, X. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3712257
“Instead, users submit asynchronous or delay-tolerant tasks such as document analysis [5], data cleaning tasks [6], and synthetic data generation [7].”
From Unfinished to Done: Bridging the MDE Implementation Gap with Constrained LLMs
Kirchhof, J. C., Netz, L., Mertens, D., & Rumpe⋆, B. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3822455.3830316
AtomicGPT: A domain-adapted large language model for secure on-premises applications in nuclear engineering
Seungdon, Y., Lim, K., & Yu, Y. (2026) · Nuclear Engineering and Technology · doi:10.1016/j.net.2026.104565
Reliable Offline-Policy Selection for Multimodal Recommendation
Yan, R., & Liu, H. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-3560-5_31
Conversational trajectory degrades large language model detection of suicidal ideation relative to clinicians: a preregistered studyPreprint
Kalinich, M., Luccarelli, J., Maria, J. P. S., Flathers, M., Nguyen, A., Song, S. H., Makhoul, K. G., Criado, M. J. R., Ginapp, C. M., Hill, B., et al. (2026) · medRxiv · doi:10.64898/2026.07.10.26357132
A Cascaded Generative Approach for e-Commerce Recommendations
Hasani, M., Shahidi, H., Levinson, T., Zhong, Y., Shu, G., Gudla, V., & Tenneti, T. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808458
Predicting Compound Risk in AI Incidents: A Cross-Sector Classification Approach with Education as a Critical Case
Niu, C., Bradley, K. D., Grote-Garcia, S. A., & Love, A. S. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30805-4_20
More than a Judge: An Empirical Study of Agent-Human Interaction in Crowdsourced Testing Assessment
Wang, Y., Zhao, Y., Yu, S., Chen, Z., & Gu, Q. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3828168
When LLM Judges Inflate Scores: Exploring Overrating in Relevance Assessment
Yu, C., Li, H., Zuccon, G., Mackenzie, J., & Leelanupab, T. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809905 · Cited by 7
“LLMs are known to prefer longer responses regardless of quality [21, 31], and input length shifts LLM label distributions in model-dependent ways [19].”
SAER: Scalable Assessment of E-commerce Recommendations using Large Language Models
Liu, X., Regulagedda, M. S., Patel, S., & Wong, W. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808462
“Zheng et al. [32] established the LLM-as-a-Judgeparadigm,andsubsequent worksvalidateditacrosssearch[10, 26], translation [16], and persona-conditioned evaluation [4], along-side studies on hallucination limits [11, 14].”
NeuroSymbolic-Chain: A neuro-symbolic framework for intelligent disaster chain analysis from multi-source text
Xie, N., Fan, J., Jiang, L., Wang, X., Wang, X., Liu, H., 郑火国, Meng, Z., Wu, H., Li, X., et al. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116650
Preliminary Study of an Evaluation Benchmark for Vision–Language Models in Fashion E-Commerce
Shimizu, R., HtaungKham, S., Sakurai, S., & Shimizu, Y. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808442
“Future directions include larger, more diverse datasets, constructing a publicly available benchmark following the same protocol, alternative evaluation criteria, such as LLM-as-a-Judge [45], and integration with a continuous monitoring pipeline.”
Blast Radius Characterization of Triple-Level Poisoning Attacks in Knowledge Graph Retrieval-Augmented Generation SystemsPreprint
R, S. R. H., & S, J. K. (2026) · Research Square · doi:10.21203/rs.3.rs-10016017/v1
Scalable Multi-vector Retrieval for Policy Enforcement on E-commerce Marketplaces
Sarpal, A., Kotamraju, S., & Uppalapati, R. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808506
“This is an expensive default baseline that now acts as a ground-truth labeler on a significantly smaller volume, leveraging the ’LLM-as-a-judge’ paradigm [16] to identify new false negatives in real-time..”

Authors: Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I.
ORCID: Zheng, Lianmin
Chiang, Wei-Lin
Sheng, Ying
Zhuang, Siyuan
Li, Dacheng
Gonzalez, Joseph E.
Venue: arXiv
Publisher: arXiv
Published: 2023
Pages: 46595-46623
ISSN: 1049-5258 (ISSN-L)
DOI: 10.48550/arxiv.2306.05685
arXiv: 2306.05685
OpenAlex: W7133239970
Cited by: 1,888
References: 2
Citations by year: 2026: 1115 · 2025: 413 · 2024: 308 · 2023: 51
FWCI: 2548.93
Type: conference paper
Language: en
Open access: green
arxiv.org
Licence: arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations: neural information processing systems (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Funders: VMware
Samsung
Topics: Social Sciences › Law › Artificial Intelligence in Law
Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords: large language models, LLM-as-a-judge, automated assessment, MT-Bench, Chatbot Arena, benchmarking, chatbots

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Abstract

Overview

AI snapshot

TL;DR

Key findings

Methods

Limitations

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks