PreprintCited by 2,065Artificial Intelligence in LawTopic ModelingNatural Language Processing Techniques

The DOI matches its DataCite record; 58 of the 58 references it cites were found in a registry or an index; 4 of the 29 sources its claims cite were checked for support.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Open access

Zheng, Lianmin, Chiang, Wei-Lin, Sheng, Ying, Zhuang, Siyuan, Wu, Zhanghao, Zhuang, Yonghao, Lin, Zi, Li, Zhuohan, Li, Dacheng, Xing, Eric P., Zhang, Hao, Gonzalez, Joseph E., Stoica, Ion·2023·arXiv

DOI 10.48550/arxiv.2306.05685arXiv 2306.05685

Open this paper in CoChat — free account Open PDF

Abstract

Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper examines whether strong LLMs can serve as judges for chatbot responses by comparing their judgments with human preferences on two new benchmarks, MT-bench and Chatbot Arena. It finds that GPT-4 can reach over 80% agreement with human preferences, comparable to human-human agreement, while also documenting several judge biases and mitigation strategies.

Key findings

GPT-4’s judgments aligned closely with human experts on MT-bench: 85% agreement without ties, versus 81% agreement among humans; on Chatbot Arena, GPT-4 also showed high agreement with crowd votes, above 80%.
Position bias was substantial: with the default prompt, consistency was 65.0% for GPT-4, 46.2% for GPT-3.5, and 23.8% for Claude-v1; swapping positions and few-shot examples raised GPT-4 consistency to 77.5%.
Verbosity bias appeared in a “repetitive list” attack: Claude-v1 and GPT-3.5 failed 91.3% of cases, while GPT-4 failed 8.7%.
Reference-guided grading reduced math-judging failures from 14/20 under the default prompt to 3/20, with chain-of-thought at 6/20; self-enhancement was suggested by GPT-4 favoring itself with a 10% higher win rate and Claude-v1 with a 25% higher win rate, but the study could not confirm the bias.

Methods

The authors created MT-bench, 80 multi-turn questions across 8 categories with 10 questions each, and Chatbot Arena, a crowdsourced anonymous battle platform; they collected about 3K expert votes from 58 labelers on MT-bench and about 30K Arena votes, sampling 3K for analysis.
They compared LLM judges—mainly GPT-4, GPT-3.5, and Claude-v1—using pairwise comparison, single-answer grading, and reference-guided grading against human expert and crowd votes, measuring agreement and win rates.
They also evaluated LLaMA and Vicuna variants on MMLU, TruthfulQA, and MT-bench with GPT-4 single-answer grading to test how preference-based and capability-based benchmarks complement each other.

Limitations

The paper states that it emphasizes helpfulness and largely neglects safety, honesty, and harmlessness; it also notes that helpfulness dimensions such as accuracy, relevance, and creativity are combined into one metric, and its bias mitigations are preliminary.
The paper reports that self-enhancement bias could not be determined because of limited data and small differences, and that its position-bias test is difficult because the compared answers are very similar and sometimes indistinguishable even to humans.
The text makes evident that few-shot judging may introduce new biases and makes API calls about 4× more expensive, while zero-shot Vicuna-13B as a judge had low consistency (11.2–16.2%) and high error rates (22.5–78.8%), limiting its use as a scalable open-source judge.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.05685

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

56 Own result · 35 Own interpretation · 7 Sources not found · 7 No basis given · 3 Partly supported · 2 Own method · 1 Not checked yet · accuracy not yet measured

The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Once aligned with humans, these chat models are strongly preferred by human users over the original, unaligned models on which they are built.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites [31] source
- Partly supported
  Section 4.5 (p. 21) says crowdworkers clearly preferred the online RLHF models in comparisons with context-distilled and earlier RLHF models. But those preference comparisons do not establish that users strongly preferred them over the original, unaligned base models; the paper discusses base-model improvements on other evaluations (Introduction, p. 4). (LLM, conf=0.86)
  cites [2] sourceExperimental — accuracy not yet measured
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [30]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [8]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [52]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [48]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [14]
boosted
direct
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We show that some of the biases are minor or can be mitigated.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that in math/coding category, GPT-3.5 and GPT-4 have similar overall win-rate because they both failed to answer some hard questions, but GPT-4 is still significantly better than GPT-3 in the direct pairwise comparison or single-answer grading.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.3 Win rates under different judges
plain
correlational
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, a small high-quality conversation dataset can quickly teach the model a style preferred by GPT-4 (or approximately human) but cannot improve MMLU significantly, as shown by the Vicuna-7B (selected) which is trained with only 4.8M tokens or 3K conversations.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
direct
Single case
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most LLM judges favor the first position.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only GPT-4 successfully detected this attack.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plain
Single case
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As in Table 2, we found all of them exhibit strong position bias.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, it has a high error rate (from 22.5% to 78.8%) because of its weaker instruction-following capability.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plain
direct
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Claude-v1 favors itself with a 25% higher win rate.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
direct
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 12 (Appendix), the few-shot judge can significantly increase the consistency of GPT-4 from 65.0% to 77.5%.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
direct
n = 3

Show all 65 claims

Probably not a substantial claim. 22
It may state two separate claims as one. 29
The quote only partly states this claim. 2

The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As LLMs continue to improve, they show potential in replacing human annotators in many tasks [17,20].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3 LLM as a Judge
What the sources say
- Supported
  The Discussion (p. 4) says LLMs could transform text annotation across a variety of tasks and that ChatGPT may already outperform crowd annotators. The Introduction (p. 2) also describes the possibility of annotating entire samples with ChatGPT. These passages support the hedged claim about potential replacement in annotation tasks, particularly those performed by crowd-workers. (LLM, conf=0.88)
  cites [17] source
- Partly supported
  The Discussion and Conclusion (Section 5; page unavailable) says ChatGPT showed great potential as a data annotation tool for implicit hateful tweets. It does not establish that LLMs could replace human annotators across many tasks; it also urges caution about incorrect decisions. (LLM, conf=0.86)
  cites [20] sourceExperimental — accuracy not yet measured
hedged
conditional
Probably not a substantial claim.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“DynaBench addresses the challenges posed by static standardized benchmarks, such as saturation and overfitting, by emphasizing dynamic data with human-in-the-loop.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
What the sources say
- Partly supported
  The Introduction (p. 1) presents Dynabench as dynamic data collection with humans and models in the loop, in contrast to static benchmarking. Caveats and Objections (p. 8) explicitly says it anticipates dataset saturation. The copy does not identify overfitting as a challenge of static benchmarks; its discussion of annotators ‘overfitting’ on models (p. 7) describes a potential risk of the dynamic approach instead. (LLM, conf=0.84)
  cites [21] sourceExperimental — accuracy not yet measured
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite the base LLaMA models showing competitive performance on conventional benchmarks (Table 8), its answers to open-ended questions are often not preferred by humans.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.”
- Also stated in: § GPT-4 Judgment: · plain
  “Once addressed, our results from 3K controlled expert votes and 3K crowdsourced human votes in the wild verify that GPT-4 judge match human evaluations at an agreement rate exceeding 80%, achieving the same level of human-human agreement ( §4.2, Table 4).”
- Also stated in: § 4.2 High agreement between GPT-4 and humans · plain
  “The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Abstract
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The consistency improves significantly from 16.2% to 65.0%.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plain
direct
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves 56.8% when including all three labels, and 85.5% when excluding tie predictions and labels, significantly outperforming random guesses of 33% and 50% respectively, and show positive signals to match GPT-4 (66% and 87% respectively).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plain
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Despite GPT-4 knows how to solve the question (if asked separately), it made a wrong judgement saying both assistants' wrong answers are correct.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 14:
plain
Single case
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to the classification formulation, every output is recognizable (error rate 0%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Position bias results
plain
direct
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 15, Vicuna-13B has a consistency rate from 11.2% to 16.2% across different prompt templates, much lower than all the closed-sourced models.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, in Figure 13 (Appendix), we present an example of an elementary math question in which GPT-4 makes an incorrect judgment.It's worth noting that although GPT-4 can solve the problem (when asked separately), it was misled by the provided answers, ultimately resulting in incorrect judgment.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
direct
Single case
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 3 shows the failure rate of LLM judges under this attack, demonstrating that all LLMs may be prone to verbosity bias though GPT-4 defends significantly better than others.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
hedged
conditional
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 11, the position bias is more noticeable for models with close performance and can almost disappear when the performance of the two models differs a lot.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > D.1 Position bias
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results reveal that strong LLMs can achieve an agreement rate of over 80%, on par with the level of agreement among human experts, establishing a foundation for an LLM-based evaluation framework.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 7 Conclusion
plain
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Comparing GPT-4 and other LLM judges, we find they reach a similar non-tie agreement ratio between humans but the number of non-tied votes from GPT-4 is much larger.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plain
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plain
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When Assistant A is placed in the first position, GPT-4 thinks A is better, but its verdict changes when we swap the position of A and B.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 11: An example of position bias.
plain
direct
Single case
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that fine-tuning on high-quality dialog datasets (i.e., ShareGPT) can consistently improve the model performance on MMLU and the improvement scales with fine-tuning data size.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
direct
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both GPT-3.5 and Claude-v1 show a verbosity bias towards the longer and repetitive answer.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plain
direct
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both tables, GPT-4 with single-answer grading matches both pairwise GPT-4 and human preferences very well.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
boosted
correlational
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On math and coding questions, LLM judges are more confident even though their judgments can often be wrong, as we show in Section 3.3.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > D.1 Position bias
plain
correlational
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Besides, the longer prompts make API calls 4× more expensive.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
direct
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, even with the CoT prompt, we find that in many cases LLM makes exactly the same mistake as the given answers in its problem-solving process (See example in Figure 15 (Appendix), suggesting that LLM judge may still be misled by the context.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Chain-of-thought and reference-guided judge.
hedged
conditional
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We observe similar pattern from other LLM judges such as Claude/GPT-3.5.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Figure 11: An example of position bias.
plain
correlational
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, the pairwise comparison may lack scalability when the number of players increases, given that the number of possible pairs grows quadratically;”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.1 Types of LLM-as-a-Judge
hedged
conditional
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“However, the heightened user preference does not always correspond to improved scores on traditional LLM benchmarks -benchmarks like MMLU [19] and HELM [24] cannot effectively tell the difference between these aligned models and the base models.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 1 Introduction
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [19]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [24]
plain
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This misalignment of conventional benchmarks underscores the core problem driving this paper: the need for a robust and scalable automated method to evaluate LLM alignment with human preferences.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > GPT-4 Judgment:
plain
direct
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“single answer grading may be unable to discern subtle differences between specific pairs, and its results may become unstable, as absolute scores are likely to fluctuate more than relative pairwise results if the judge model changes.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.1 Types of LLM-as-a-Judge
hedged
conditional
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Conversational benchmarks, like CoQA [35], MMDialog [15] and OpenAssistant [23], are closest to our intended use cases.However, the diversity and complexity of their questions often fall short in challenging the capabilities of the latest chatbots.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 2.1 Motivation
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [35]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [15]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [23]
plain
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, high consistency may not imply high accuracy and we are not sure whether the few-shot examples will introduce new biases.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
hedged
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means that GPT-4 is more affirmative and less suffered from position bias but other models also perform well when they give an affirmative answer.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 4.2 High agreement between GPT-4 and humans
plain
correlational
It may state two separate claims as one.
In the paper
“LLMs are known to have limited math and reasoning capability [10], which results in its failure of grading such questions because they do not know the correct answers.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [10]
boosted
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They focus on different aspects of a model and the recommended way is to comprehensively evaluate models with both kinds of benchmarks.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
correlational
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To invite participants, we obtained their consent by letting them sign an application form.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > C.1 MT-bench human evaluation
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another more aggressive approach is to assign positions randomly, which can be effective at a large scale with the correct expectations.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
conditional
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Table 8, no single benchmark can determine model quality, meaning that a comprehensive evaluation is needed.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This bias is not unique to our context and has been seen in human decision-making [3,34] and other ML domains [22,41].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [3]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [34]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [22]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [41]
plain
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Chain-of-thought is a widely used technique to improve LLM's reasoning capability [47].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Chain-of-thought and reference-guided judge.
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [47]
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our LLM-as-a-judge approach can automate and scale platforms of this nature.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
plain
direct
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results indicate that using LLM-as-a-judge to approximate human preferences is highly feasible and could become a new standard in future benchmarks.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 5 Human Preference Benchmark and Standardized Benchmark
hedged
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Users are required to accept the terms of use, which obtain their consent and give us the right to release the conversation data.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > C.2 Chatbot Arena
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A conservative approach is to call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Swapping positions.
plain
direct
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These responses are rendered as natural languages and are difficult to be parsed automatically, making the model less useful in a scalable and automatic evaluation pipeline.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > F.1 Zero-Shot Vicuna
plain
direct
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLM-as-a-judge offers two key benefits: scalability and explainability.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.2 Advantages of LLM-as-a-Judge
plain
direct
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that this test is challenging because the answers are very similar and occasionally indistinguishable even to humans.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 3.3 Limitations of LLM-as-a-Judge
plain
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The single-answer grading can be converted into pairwise comparison results for calculating the agreement.”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > Additional information
plain
direct
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Honesty and harmlessness are crucial for a chat assistant as well [2].”
§ Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena > 6 Discussion
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites [2]
plain
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.

46 of 46

The paper relies on this without support.
LLMs trained with RLHF already exhibit strong human alignment.
“Because these models are often trained with RLHF, they already exhibit strong human alignment.”
The paper relies on this without support.
There has not been a systematic study of the LLM-as-a-judge approach.
“However, there has not been a systematic study of this approach.”
The paper relies on this without support.
Human preferences serve as a direct measure of a chatbot's utility in open-ended, multi-turn human-AI interactions.
“While largely overlooked by existing LLM benchmarks, human preferences serve as a direct measure of a chatbot's utility in open-ended, multi-turn human-AI interactions.”
The paper relies on this without support.
The human labelers are mostly graduate students and are considered experts and more skilled than average crowd workers.
“The labelers are mostly graduate students so they are considered experts and more skilled than average crowd workers.”
The paper relies on this without support.
Human evaluation is the gold standard for assessing human preferences.
“While human evaluation is the gold standard for assessing human preferences, it is exceptionally slow and costly.”
The paper relies on this without support.
Model services such as GPT-4 can become expensive with a growing number of evaluations.
“However, model services such as GPT-4 can also become expensive with a growing number of evaluations.”
The paper relies on this without support.
Popular open-sourced LLMs, e.g. Vicuna-13B, show strong language understanding capability and are much cheaper than close-sourced LLMs.
“On the other hand, popular open-sourced LLMs, e.g.Vicuna-13B shows strong language understanding capability, and are much cheaper than close-sourced LLMs.”

7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Traditional evaluation metrics based on the similarity between outputs and reference answers (e.g., ROUGE [25], BLEU [32]) are also ineffective for these questions.”
Method
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 6×
“5, Claude-V1, Vicuna-13B, Alpaca-13B [38], and LLaMA-13B [39].”
BackgroundInfluential
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1×
“Chain-of-thought is a widely used technique to improve LLM’s reasoning capability [39].”
Method
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Chain-of-thought is a widely used technique to improve LLM’s reasoning capability [39].”
Method
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832 · cited 1×
“This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
Background
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“There has been a proliferation of LLM-based chat assistants (chatbots) that leverage supervised instruction fine-tuning and reinforcement learning with human feedback (RLHF) to unlock new instruction following and conversational abilities [27, 2, 26, 7, 43].”
Background
HellaSwag: Can a Machine Really Finish Your Sentence?Preprint
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07830 · Cited by 1,693 · cited 2×
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
Method
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 2×
“This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
Background
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 2×
“• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
Method
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243

Show all 2 references

Research works printed but not linked: 2

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and Ion Stoica. SkyPilot: An intercloud broker for sky computing. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 437-455, Boston, MA, April 2023. USENIX Association.

Sorted by citations

Items 1,601–1,700 of 2,065

Multimedia Learning through Agentic Educational Video Generation
Lam, P. H., & Chan, H. C. B. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00206
“The evaluation method includes text similarity assessment and LLM-as-a-Judge scoring process [30].”
ElderBench: Benchmarking Personalized Open-Source LLMS for Older Adults
Eskandari, A., Tao, J., Zulkernine, F. H., Morningstar, M. A., Poppenk, J., & Herrmann, B. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00176
“• Level 2 employs GPT-4o as a blind LLM judge [28], rating responses on six dimensions using a 5-point Likert scale.”
Congestion-Aware Scheduling for Heterogeneous LLM-Agent Teams
Xie, W., & Li, Y. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00432
“For preference-style evaluation, we report match accuracy under a fixed pairwise judging protocol, following the broader LLM-as-judge evaluation line while taking into account known positional and evaluator-bias concerns [27]– [29].”
VADER: Validation-Driven DSL Generation with Small Language Models
Baber, J., González-Sáez, G., Hili, N., & Schwab, D. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00068
“Finally, we evaluate modeling quality using an LLM-as-a-judge protocol, building on evidence that carefully prompted LLMs can approximate human preferences reliably [16], [17].”
SEMAD: Semantic-Enhanced Malicious Package Detection Using LLMs and Prompt EngineeringPreprint
Anh, T. D.-T., Nguyen, H.-L., Truong-Le, T. M., & Vu, D. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7323943
Risks, Mitigations and Challenges of LLM-AS-A-Judge in Automated Scoring
Li, Y., Sun, Y., & Hou, S. (2026) · International Conference on Computer Science and Technologies in Education (CSTE) · doi:10.1109/cste69562.2026.11647786
“These preferences can manifest as biases toward particular response lengths, formatting styles, linguistic registers, and even argumentative positions [11], [12].”
Work-in-Progress: PiazzaPlus–Enhancing Piazza Search with Semantic Matching
Sheikha, I. E., & Emara, S. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--61076
Efficient and reliable multimodal retrieval-augmented question answering via greedy leave-one-out consistency pruning
Kim, S.-J., Kim, H. R., Kim, S., Kim, K., & Ko, B. C. (2026) · Pattern Analysis and Applications · doi:10.1007/s10044-026-01753-5
“To evaluate the influence of backbone choice on the G-LOO pipeline, we compared three variants Qwen2 7B [30], LLaVA-Vicuna 7B [27], and LLaVA-Mistral 7B [32]–under identical settings.”
Influential
PRISM: stable black-box prompt optimization via multi-sampling and semantic consensus
Pham, T.-A., Cuong, N. M., Anh, N. Đ., Tran, H.-V., Nguyen, D.-A., & Hùng, P. N. (2026) · Pattern Analysis and Applications · doi:10.1007/s10044-026-01749-1
“These methods include specialized prompt optimizer models, including BPO, MePO, Qwen3, Llama-2-7B-chat [ 1], Vicuna-7B [ 16 ], and Gemma-3-4B-IT [ 17].”
A hybrid reasoning framework for artificial intelligence assessment rubric generation in human and automated contexts: Evidence from an undergraduate programming course
Mendonça, P. C., Quintal, F., Figueiredo, M. A. T., Baras, K., & Mendonça, F. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100663
Privacy by Design in AI-Assisted Cybersecurity Policy Generation: Benchmarking Private RAG Against Frontier AI Models
Carboni, D., & Verona, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37218-5_9
Can VLMs Understand Handwritten Mathematical Documents?
Mitra, S., Mondal, A., & Jawahar, C. V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36039-7_37
Improving the Utility, Privacy, and Human Acceptance of Differentially Private Rewritten Texts
Meisenbacher, S., Jashari, E., Klymenko, A., & Matthes, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37218-5_6
LLM-Assisted Scoring for College English Writing Assessment: Statistical Calibration Against Teacher Standards
Wang, Y., Liu, N., Chu, X., Wang, T., Cheng, X., & Wang, Y. (2026) · Mathematics · doi:10.3390/math14173033
Quality-Aware Selection for Retrieval-Augmented Fine-Tuning of Small Language Models
Cho, S., & Jung, H. (2026) · Mathematics · doi:10.3390/math14173026
Communication, Embodiment, and Coordination in an El Farol Bar-Inspired LLM Society
Takata, R., Masumori, A., & Ikegami, T. (2026) · Artificial Life · doi:10.1162/artl.a.459
From Prediction to Action: A Predict–Validate–Act Framework for CVE Vulnerability Triage
Beddies, C., Bonaiuto, V., Kubica, S., et al. (2026) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice70481.2026.00029
“Since surface metrics may not capture the semantic quality of safety-critical text, we additionally employ an LLM-as-Judge rubric following Zheng et al. [22], who report > 80% agreement between LLM judges and human preferences.”
A Reproducible and Budget-Aware Framework for Evaluating Local Retrieval-Augmented Generation (RAG) Robustness to Perturbations
Gangapuram, R. R., & Andreopoulos, W. (2026) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice70481.2026.00015
TC3-VLM: A Vision–Language Model for Tactical Combat Casualty Care
Kim, J., Chng, J., Ali, A., Moas, V. M., Hwang, S., & Kamaleswaran, R. (2026) · IEEE Access · doi:10.1109/access.2026.3726225
“LLM-as-Judge protocols offer a scalable alternative and have been shown to align with human preferences, but they are also known to exhibit position, verbosity, and self-enhancement biases [35], [36].”
Auditable LLM Safety-Judge Selection via Confidence-Interval Filtering on Unsafe Recall
Yang, J., Yi, J., Li, J., & Ma, S. (2026) · IEEE Access · doi:10.1109/access.2026.3726251
Concordance Between Clinical Practice Recommendations Generated by Generative Artificial Intelligence and the Vía RICA 2026 Enhanced Recovery Guideline: A Proof-of-Concept Study Using a Closed Evidence Corpus
Moral, A., Antonio Arroyo, S., Aparicio, J., & Barber, X. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8090256
Mitigating Hallucinations in Third-Grade Primary Mathematics Education: An Empirical Analysis of Local RAG and LLMs
Ocak, A., Diri, S., & Kilimci, Z. H. (2026) · Süleyman Demirel Üniversitesi Fen Bilimleri Enstitüsü Dergisi · doi:10.19113/sdufenbed.1912302
Using Architectural Zones to Generate Information-Flow-Aware Abstract Penetration Test Cases
Jafari, M., Weber, T., Chebbi, F., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36590-3_4
Cost-Quality Trade-Offs in Reasoning Architectures for Agentic Software Engineering Deliberation
Harbar, Y., & Chudziak, J. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36590-3_16
SemGuard: A Triple-Anchor Semantic Security Gateway for Multilingual Prompt Attack Detection in Large Language Models
Abughallous, A. M., & Abufakher, S. (2026) · IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies · doi:10.1109/aeect69724.2026.11657880
“Judge diversity was motivated by architectural independence: three model families from three organizations across two continents minimize shared training bias [14].”
Tide: Chatbot de Apoio à Comunicação Clínica na Menopausa baseado em Geração Aumentada por Recuperação
Azevedo, K., Brandão, M. A., & Moro, M. M. (2026) · Anais do Women in Information Technology (WIT) · doi:10.5753/wit.2026.22469
“Na quarta etapa da DSR, validou-se a qualidade das respostas sintéticas geradas pelo artefato por meio da metodologia de avaliação LLM-as-a-Judge [Zheng et al. 2023].”
APFL: A controlled agentic framework for Thai QG–QA dataset construction via adaptive prompt feedback loop
Jirapatpongthada, C., Sawang, S., Klinhomutis, T., & Siriborvornratanakul, T. (2026) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-026-05114-2
It Takes Two: Embracing Sparsity and Speculative Decoding for Efficient LLM Inference
Chu, H., Chen, C., Liu, W., Luan, J., Deng, J., Liu, L., Ma, H., & Zheng, X. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661086
“Following EAGLE2 [22], [23] and Medusa [4], we conduct evaluations on multi-turn tasks using the following standard datasets: ARC Challenge [8], ARC Easy [8], HellaSwag [44], PIQA [3], WinoGrande [32], GSM8K [9], HumanEval [28], Alpaca [36] and MT-bench [45].”
Metamorphic Testing of Multi-Agent LLM Systems: A Trace-Based Behavioral Oracle Framework
Marimuthu, G. (2026) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest70988.2026.00036
“• LLM-as-Judge : An LLM evaluator that assesses whether two executions on related inputs produce consistent re-sults, following the approach in [27].”
AI models as consciousness attributors: how LLMs ascribe consciousness to other agents
Kang, B., & Kim, C.-E. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1926286
Rapport-building dialogue strategy enhanced with proactivity, personalization, and aizuchi backchannels, and its automatic evaluation in HRI
Baihaqi, M. Y., Contreras, A. F. G., Kawano, S., & Yoshino, K. (2026) · Advanced Robotics · doi:10.1080/01691864.2026.2718600
Uma Abordagem Baseada em LLMs para Auditoria Contínua de Conformidade com a LGPD em Microsserviços Integrada a Pipelines CI/CD
da Silva, L. M., Albuquerque, D. W., Filho, E. D., & Moreno, B. N. (2026) · doi:10.5753/semish.2026.21832
Ygg: Tree-Based Collaborative Speculative Decoding with Token-Only Transmission
Li, S., Liang, Y., Li, R., Li, J., & Wu, J. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661225
“We assess these models on two widely used benchmarks to cover diverse capabilities, specifically utilizing MT-Bench [28] for multi-turn conversation and instruction following, and GSM8K [29] for mathematical reasoning tasks.”
Evil-AI Benchmark: An Evaluation Framework for Adversarial Security Assessment of LLM Agents in Smart Environments
Varol, A., Shaikh, A., Jalo, H., Muchenje, G., de Cerqueira, J. S., Kemell, K., Motlagh, N. H., Leino, M., Pirkkalainen, H., Abrahamsson, P., et al. (2026) · IEEE Access · doi:10.1109/access.2026.3727159
Dual-Track Red Teaming and Evaluation for Traditional Chinese Localized LLMs
Lu, R. W., Chen, W. P., & Yu, F. (2026) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest70988.2026.00013
“As a result, recent work has moved beyond static benchmarks toward dynamic red teaming and model-based evaluation pipelines [3], which are better suited to open-ended and interaction-dependent failures.”
FTDI: A budget-aware self-healing framework for LLM multi-agent code generation
Men, S., Lu, Z., Luo, H., & Zheng, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134164
AutoQABench: A Three-Level UX Benchmark for Automated Evaluation of Open-Ended LLM Responses
Wang, T., Yuan, F., Miao, C., Kaniyanoor, M. S., & Matsumoto, K. (2026) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest70988.2026.00011
“G-Eval [2], MT-Bench [5], and Chatbot Arena [11] showed that strong LLM judges can align well with human judgments in summarization, dialogue, and open-ended question answering.”
A Review of Retrieval-Augmented Generation Technology
Jiang, P., & Cai, X. (2026) · Symmetry · doi:10.3390/sym18091431
Context-Aware LLM Evaluators for Content Moderation Judgments in a Low-Resource SettingPreprint
Krejca, F., Kietreiber, T., Wiegand, M., & Neumaier, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10576121/v1
Evaluating Techniques for Reducing Generative AI Training Costs: a Refined Instruction Tuning Pipeline Approach
Ünaldı, A. O., Yılmaz, M., & Clarke, P. M. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-36816-4_17
Evaluating automated refereeing with heterogeneous LLM panels
Douven, I. (2026) · Scientometrics · doi:10.1007/s11192-026-05783-y · Cited by 1
Pick and Spin: Cold-Start-Aware Routing for Self-Hosted LLM Serving
Vangala, B. P., & Malik, T. (2026) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud72782.2026.00050
“We use standard benchmark scoring for all benchmarks: exact match for GSM8K and MATH, pass@1 via unit tests for HumanEval and MBPP, multiple-choice accuracy for ARC, HellaSwag, and MMLU-Pro, and LLM-as-judge [39] only for TruthfulQA where no objective scoring criterion exists.”
Schema-Grounded Agents for Enabling Emergent Multi-Device Reasoning in Smart Environments
Romero, V., Matsui, T., Matsuda, Y., Suwa, H., & Yasumoto, K. (2026) · International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT) · doi:10.1109/dcoss-iot69657.2026.00040
“We investigate these questions using an LLM-as-a-judge framework powered by deepseek-r1:70b, chosen for its strong reasoning ability [17].”
Improving Security Vulnerability Descriptions Using LLMs
Nwodo, K., Maitra, S., Sun, S., Wang, H., & Stavrou, A. (2026) · Studies in computational intelligence · doi:10.1007/978-3-032-18585-3_7
LLM-Assisted Task-Adaptive Semantic Communication over Non-Ideal Channels for Vision–Language Systems
Orimoogunje, O., Sarker, S., Li, X., & Qian, L. (2026) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn69946.2026.11662614
“…Semantic Evaluation: Large language models have demonstrated strong capabilities as natural language evaluators motivating their use as reward signals in reinforcement learning [10] and the LLM-as-a-judge paradigm [17] leverages model-generated quality assessments without dense human annotations.”
FlowGuard: Slack-Aware Overload Control for Multi-Agent LLM Serving
Sadiq, A. Z., & Shen, H. (2026) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn69946.2026.11662939
“FastChat [10] and TensorRT-LLM [11] provide production-grade serving but treat requests independently without modeling multi-agent workﬂow dependencies.”
Portable Lightweight Real-Time AI System Using Edge-RAG for Construction Best Practice Interactions
Chen, G., Li, Y., Jaselskis, E. J., et al. (2026) · ASCE Construction Research Congress (CRC) · doi:10.1061/9780784486962.025
WaSMa: Workflow-Aware Scheduling for Multi-Agent LLM Systems
Rao, U., Sadiq, A. Z., Shen, H., Yang, Y., Hodges, E.-A. R., & Muntean, M. (2026) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn69946.2026.11662938
“Requests from AutoGen applications are served through a local FastChat-based serving layer and executed by a PagedAttention-based engine, which gives us a standard request-centric baseline for studying workflow-level inefficiencies [8], [18], [21].”
Decoding Romanian Law: A Fine-Tuned LLM Approach
Adam, M., Berindei, A., Onchiş, D. M., Ivaşcu, T., & Negrutiu, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36207-0_29
ASH: A Systematic Automated Evaluation Framework for Creative Text Generation in Cuisine Transfer Tasks
Lee, H., Gim, M., Park, D., Choi, D., & Kang, J. (2026) · IEEE Access · doi:10.1109/access.2026.3727977
Does the Verdict Function Matter? How Evaluation Rewards Causally Steer Automated LLM Fuzzing and Distort Discovered VulnerabilitiesPreprint
Vishwakarma, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7360360
Toward AI epidemiology: a measurement standardization framework for prospective risk detection
Tempest-Walters, K. (2026) · AI & Society · doi:10.1007/s00146-026-03276-3
Beyond Perception: Structured Rationale Supervision and Distillation for Constructive Geometry Reasoning in Vision-Language ModelsPreprint
Cui, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7369010
Evaluating the Effect of Fine-Tuning on Usability Annotation of English-Slovak Translation
Chotar, M., Kleštinec, M., & Munková, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37249-9_19
Analysis and Implications of Alignment Between LLM-as-a-Judge and Human Evaluators in a Generative AI Chatbot: A Real-World Study
Escudero-Mancebo, D., Rodríguez, M., & Oliva, G. V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37249-9_46
FitRAG: Self-Improving LLM-Based Services through Feedback-Driven Adapter Lifecycle Management
Syah, R. A., Haryanto, C. Y., Lomempow, E., Malik, K., & Putra, I. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00022
“While LLM-as-judge has known limitations including position bias [25], it provides a practical quality signal when human evaluation is infeasible.”
DesignDNA: Injecting Domain Knowledge into Generative Models for Controllable Air Conditioner Design
Wei, J., Lin, C., Zhao, Q., et al. (2025) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid68789.2025.00081
“We adopt the ‘LLM-as-a-Judge’ framework [18] to score results against images and user prompts.”
An Adaptive Harness Architecture for AI Transformation Diagnosis: A TIM-Taxonomy Core with Target and Stage-Specific Plug-Ins
Choi, K. H., & Cho, S. (2026) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2026.24.8.277
DualCareKG: Integrating Clinical and Patient Knowledge Graphs for Dementia Caregiver Support
Hossain, M. M., Shadin, N. S., Cummings, A., Zhang, X., & Akintomide, M. (2026) · IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies · doi:10.1109/chase69719.2026.00025
An agentic multimodal RAG framework for enhancing nuclear regulatory review: Preserving structural integrity of safety analysis reports
Kim, B., & Yu, Y. (2026) · Nuclear Engineering and Technology · doi:10.1016/j.net.2026.104687
Generating High-Fidelity Synthetic Mental Health Records with Expert-in-the-Loop Verification
Wang, L., Juneja, S., Naar, S., Lee, M., Sha, K., Shah, K., Green, S., & Feng, Y. (2026) · IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies · doi:10.1109/chase69719.2026.00023
“Many studies rely on lexical overlap metrics such as BLEU [14] or ROUGE; however, broader LLM evaluation research has shown that such metrics fail to capture human preferences and factual correctness [15].”
Evaluating Answer Quality and LLM-Based Assessment in a Zero-Marginal-Cost Retrieval-Augmented Generation Agricultural Advisory System
Jegede, T., Obaido, G., Esenogho, E., & Cameron, M. (2026) · Mathematical and Computational Applications · doi:10.3390/mca31050174
EduTrack: An Outcome-Based Assessment Management System with Integrated Analytics and AI-Assisted Evaluation
Pham, D., Nguyen, D., Nguyen, L., et al. (2026) · International Conference on Distance Education and Learning (ICDEL) · doi:10.1109/icdel70250.2026.11663666
“That design follows the same caution emphasized in the recent works on criterion-based grading and judge reliability [11, 12].”
What Does a Prompt Actually Do? A Systematic Ablation Study of Prompt Components in LLM Code Generation
Venkatappa, R. P. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00036
Efficient Vision-Language Fusion-Cascade for Explainable Manufacturing Workplace Safety MonitoringPreprint
Dongkyu, Y., & Choi, S.-H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7379938
Computing Interpersonal Metafunctions: Quantifying Appraisal Shifts via LLM Ensembles
Gao, Y., Zhang, J., & Yu, Y. (2026) · International Conference on Computer Information and Big Data Applications (CIBDA) · doi:10.1145/3813822.3813869
“…overcome the methodological bottleneck of lacking large-scale expert gold standards in fine-grained pragmatic annotation and deep translation evaluation, employing multi-model ensemble methods to generate a silver standard has emerged as a cutting-edge paradigm in computational linguistics [ 9 ] .”
Implementing a Cognitively Grounded Artificial Moral Advisor: A Multi-LLM Multi-Agent Approach Based on the Cognitive–Reflective Equilibration Model
Kim, C., & Ahn, S. (2026) · Journal of Intelligence · doi:10.3390/jintelligence14090212
“…of the measurement scores in a form equivalent to a standardized agreement index related to ICC(2,k); it is not conventional inter-rater reliability, evidence of normative correctness, or evidence of freedom from biases shared across models trained on overlapping corpora (Zheng et al., 2023).”
A review of bias detection and fairness auditing techniques in LLMs
Kaveti, N. K., & Pattanshetti, T. R. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02121-2
“It implies that LLM-based automatic evaluators could significantly cut down the costs of manual annotation but should be considered supplementary decision-making tools until the time comes to replace human expertise in risky applications [12, 29].”
Influential
A multi-agent large language model-based group consensus evaluation system for identifying technological innovation opportunities
Hu, Y., Cao, M., Li, J., et al. (2026) · Computers & Industrial Engineering · doi:10.1016/j.cie.2026.112342
Fixing, Breaking, or Faking It? An Execution-Calibrated Evaluation of LLM Vulnerability Patching in JavaScript, Python, Go, and Java, and the Limits of LLM-as-Judge
Deininger, P., & Slany, W. (2026) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp6050153
“Code-specific judge methods such as ICE-Score [27] and CodeJudge [28] instruct an LLM to score code directly, and LLM judges more broadly exhibit position, verbosity, and self-enhancement biases [29,30], including a documented preference for their own outputs [31].”
Multimodal-guided self-distillation for unified person search
Yang, X., Zhou, H., Zhu, H., & Wang, N. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109581
DesignCritic: Confidence-Banded Calibration and Principle-Aware Auditing for AI-Generated Graphic Design Evaluation
吳卓勳, & Song, J.-S. (2026) · IEEE Access · doi:10.1109/access.2026.3730226
VietLegalLM: Progressive Legal Expertise Through Synthetic Comprehension and Reinforcement Learning
Le, T., Le, A.-C., Hà, N. V., Duong, C., Nguyen, Q., & Nguyen, N. (2026) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3829212
“…of agreement is scientifically robust and aligns with established findings in the literature; notably, foundational studies on the LLM-as-a-judge paradigm demonstrate that model evaluators can achieve over 80% agreement with human experts, a rate comparable to human inter-annotator agreement [32].”
A Review of Neural Question Generation: Approaches, Challenges, and Future Directions
Guo, S., Pang, L., Zhang, J., Li, C., & Chen, H. (2026) · ACM Computing Surveys · doi:10.1145/3843765
“Studies should also report the aggregation rule and the frequency of order-dependent disagreements [139, 176].”
Influential
Automated Benchmarking Identifies Topic-Dependent Accuracy of Large Language Models Applied for Indoor Air Quality
Zhou, A., Ngo, N. D., Duy, L. K. N., Huang, Y., Wong, K.-S., Yu, H., & Nguyen, T. H. (2026) · ACS ES&T Air · doi:10.1021/acsestair.6c00152
TD-GAE: A multimodal large language model framework for adaptive and explainable guided art learning
Liu, W., Chen, J., Liu, Y., Kuang, S., & Wang, C. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01270-9
Co-Debater: A multimodal dialectical reasoning framework for suppressing reasoning hallucinations in hateful meme detection
Cheng, C., Zhong, H., & Song, Y. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.135027
CIRCLE: A Circumplex-Aware Contrastive Learning Framework for Fine-grained Human Value Identification in LLM-Generated ContentPreprint
Yuan, Y., & Liu, K. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7412293
Standardizing longitudinal chest X-ray report evaluation via large language model annotation
Wang, X., Figueredo, G., Li, R., & Chen, X. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134237
“First, motivated by previous findings that commercial LLM-based evaluations can achieve strong agreement with human judgments in open-ended text generation tasks [56], together with the superior performance of Gemini 3 Flash on the longitudinal annotation task (Appendix A.4), we replaced Qwen2.”
The XAI-Seeking Principle: Structuring Explainability for LLM-Mediated Decision Support
Grimm, V., Rubart, J., Herder, E., & Röcker, C. (2026) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3800935.3830869
“LLM-based judging (cf. [21]) can provide scalable diagnostic signals for such cases, but must be interpreted cautiously and not as independent ground truth.”
Indirect Prompt Injection in Municipal Document-Processing Copilots: Attacks, Defences and Harm
Cisneros-González, J., Ondiviela, J. A., & Sánchez-Soriano, J. (2026) · Street Art & Urban Creativity · doi:10.62161/sauc.v12.6399
“LLM-based evaluation offers a scalable alternative to human assessment, but it can diverge from human judgements and exhibit systematic biases, including position, verbosity and self-preference effects (Chiang & Lee, 2023; Zheng et al., 2023).”
Beyond the Blue Link: Empirical Evaluation of Generative Engine Optimization in Stochastic Retrieval Systems
Mehta, M., Sharma, R., Kalluru, V., & Kotwal, A. (2026) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3800935.3830864
“Zheng et al. [24] validated GPT-4 judge agreement with humans; Chen et al. [8] identified biases in both human and LLM judges.”
An AI4Research Scoring Benchmark: Characterizing and Calibrating LLM Evaluation of Automated Research ProtocolsPreprint
Yichen, Y., Dianqing, L., & Meng, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7422696
Fine-TOONing: A Large-Scale Instruction-Tuning Dataset for Structured Output Generation in TOON Format
Orlando, G. M., Perillo, M., Russo, D., Gatta, V. L., & Moscato, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37667-1_6
QoS-Aware Brokering for LLM-as-a-Service: SLA-Constrained Routing with Drift Detection
Tibermacine, O., & Tibermacine, C. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00112
“Similarly, crowdsourced platforms like LMSYS Chatbot Arena [6] rely on aggregated human preference to rank models using an Elo rating system.”
A Requirement-Driven Approach for Multi-Agent Service Pattern Recommendation
Xi, M., Zhang, Z., Jin, Y., Cheng, G., Guan, H., Li, Y., Pan, X., & Yin, J. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00185
“Existing datasets largely focus on individual agent capabilities [14], [15] rather than comprehensive pattern-level collaborations and service quality evaluations [2].”
EGP-Defense: Enhancing Adversarial Robustness of LVLMs via Training-Free Edge-Guided Prompting
Wang, B., He, Z., Hu, X., Wang, C., Li, Z., & Fu, Z. (2026) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3845608
Self-citation
LRGCL: LLM-refined graph contrastive learning for review-based recommendation
Shi, L., Shao, P., Zhang, S., & Chen, J. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01221-4
“For example, on Automotive, the restructured reviews achieve a BERTScore-F1 of 0.8933, indicating In addition to the automatic evaluations, we conduct an LLM-as-a-judge evaluation as a complementary assessment of restructuring quality (Zheng et al. 2023; Liu et al. 2023).”
Influential
SAGE-QA: Shared-context agentic graph-enhanced question answering for scientific reasoning
Hsu, Y.-C., & Buehler, M. J. (2026) · MRS Communications · doi:10.1557/s43579-026-01036-4 · Cited by 1
One Reflex, Two Answers: Model Judges Weakly Discriminate Localized from Distributed Causes in Agent Failure Attribution
Scavotto, B. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8090275
A TextGrad-Inspired Optimization Pipeline for AI-Assisted Generation of Lecture Notes and Problem Sets
So, C.-C., Lee, C., Yang, D., Loh, A. W. K., & Wu, C.-F. (2026) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2026.3731978
Towards Automatic Software Faults Analysis
Yu, J., Jiang, W., Zhang, X., et al. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3795652
“Empirical software fault studies are particularly suitable for such automation, as they follow structured workflows and emphasize consistent judgment rather than novel algorithm design, aligning well with the strengths of LLMs [12].”
Towards Task-Agnostic Evaluation Framework for LLM-Embedded Applications
Widanapathiranage, D. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787633
“However, they do not capture domain constraints, robustness to input variation, or system-level effects of prompts, retrieval, and tools [33].”
Observe, imitate, and then analogize: Long-tail traffic event generation for robust autonomous driving
Chen, S., Chen, B., Zhang, S., Luo, Y., Ma, Y., Qiao, Y., & Wang, Y. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114856
Debugging Requirements Interview Scripts: A Framework for Quality Assessment
Tefur, A. E.-M., Fotrousi, F., Ferrari, A., Spoletini, P., & Maalej, W. (2026) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re68928.2026.00007
“Promising strategies include multi-agent pipelines for ﬁltering/prioritization (e.g., “LLM as a judge” [25]) and retrieval-augmented approaches to ground feedback in RE rubrics and annotated examples [26].”
Can the Production of Territorial and Environmental Diagnosis by Consulting Firms Be Co-Piloted? An Experimental Analysis of the Potential Offered by Multi-Agent Frameworks
Patault, E., Sinsin, T., Bruis-Gervasone, L., Quesnot, P., Girard, S., Pesquier, B., & MARDUEL, B. (2026) · AI for Engineering · doi:10.3390/aieng1020011
“Likewise, agentic peer-review systems, or LLMs as judges [74], in which several agents mutually evaluate generated outputs before final validation, constitute a promising avenue for strengthening scientific reliability.”
Stakeholder-Adaptive Safety Report Generation Using RAG and GSN
Kato, H., Hayashi, M., Tsuchiya, T., Yoda, M., & Matsuno, Y. (2026) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re68928.2026.00047
“Generation quality is assessed with an LLM-as-a-Judge protocol [18], using criteria from SummEval [19], RAGAs [20], and APPLS [21].”
EmbedCopilot-Bench: Evaluating Vision-Language Models for Hardware-Aware Embedded System Development
Yuan, D., Deng, Y., & Chen, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37132-4_19
Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language InstructionsPreprint
Zhang, L., Shi, B., Wu, H., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7439504
DoCoG: Mask-Based Multi-type Grounded Chain-of-Thought for Document QA
Gunda, S. M., Jinka, J. S., Sagar, R. H., Jain, A., Venna, V. K., Srinivasan, A., & Sarvadevabhatla, R. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37271-0_17

Authors: Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I.
ORCID: Zheng, Lianmin
Chiang, Wei-Lin
Sheng, Ying
Zhuang, Siyuan
Li, Dacheng
Gonzalez, Joseph E.
Venue: arXiv
Publisher: arXiv
Published: 2023
Pages: 46595-46623
ISSN: 1049-5258 (ISSN-L)
DOI: 10.48550/arxiv.2306.05685
arXiv: 2306.05685
OpenAlex: W7133239970
Cited by: 1,888
References: 2
Citations by year: 2026: 1115 · 2025: 413 · 2024: 308 · 2023: 51
FWCI: 2548.93
Type: conference paper
Language: en
Open access: green
arxiv.org
Licence: arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations: neural information processing systems (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Funders: VMware
Samsung
Topics: Social Sciences › Law › Artificial Intelligence in Law
Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords: large language models, LLM-as-a-judge, automated assessment, MT-Bench, Chatbot Arena, benchmarking, chatbots

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Abstract

Overview

AI snapshot

TL;DR

Key findings

Methods

Limitations

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks